Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
This is actually a well-known phenomenon in ML, called "The Bitter Lesson".
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
Out of curiousity, I like security just as much as the next guy and ensure that certs are valid, ciphers are fresh, and protocols are mature, but I am also not really concerned with an informational site that runs http being browsed. Would you mind sharing what your concern would be with browsing this content over http?
If you'd like to dig more into the history, let's encrypt did so much to enable the movement to https by making certs free, and chrome with the red text and warnings in the url bar, which has since graduated to a full on page that requires clicking to still go despite the warnings
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
It's kind of sad that popular CS textbooks often focus on solving precise problems with lowest theoretical complexity bounds while ignoring more practical (but generally applicable) computation techniques.
In machine learning they call it "gradient descent", which in older days had analogies in techniques called "hill climbing", "local search" and "simulated annealing". Basically you have a function you need to optimize for, and you clumsily tweak the parameters so that you get the (locally) max/min value you wanted. These techniques were great at finding approximate, locally maximal solutions without trying all the possibilities at once (which is more akin to the kind of "brute force" in the traditional CS context).
I guess because these techniques were generally applicable yet the outputs were approximate and you couldn't analyze them much (no fancy O(n log n)), the theorists did not find them interesting and thus were not put into the spotlight of student's learning curricula.
In modern machine learning they do this gradient descent thing which is also tweaking the parameters bit by bit to optimize for the loss function, except that the parameters are now in the billions and trillions. The compute required is huge of course, but it's actually quite an "efficient" process, and it's not actually doing much of "brute forcing" at all. During training, the process is essentially, almost equivalent to, compressing the many many trillions of tokens of training data. To me it's quite amazing that they manage to complete such a process within a couple months of training, even if they have hundreds of thousands of GPUs...
It might be that what we consider a basic and very hard puzzle are extremely close together on a more absolute scale. The difference is often for us what proportion of humans can solve it. And the low end of that is still quite high up - animals that can solve things that are very basic for the vast majority of humans are pretty rare and known about, yet are capable of quite complex actions and learning and aren’t wildly different in scale of neurons to us.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in
Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model error, and that this portion of the cross entropy is reduced with scaling.
And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and memorize every combination. They call this "slingshot generalization".
There was a presentation at CS 25 Transformers United about some phenomena like chain-of-thought emerging only after training LLMs with at least 1T tokens and only in LLMs of certain size.
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset:
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
Exactly, I was going to suggest the Welch Labs videos on grokking. Especially the newer one at https://youtu.be/D8GOeCFFby4?si=yLI9zzcjsnEELUqy . They are really well done and really eye-opening.
> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.
> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them
How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.
it's certainly not a definite procedure for determining if an arbitrary mathematical statement is true or not. it's more like educated guess and check which definitely scales up
Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
The nvidia version is heavy to compute (common problem with RNN and LSTM, also noted in the paper). Moonshot's Kimi K3 replaces part of it with some function that performs better.
I dont understand the details but, that in itself might a pretty big contribution.
Anyway, I'm amazed how fast these companies improve each other's ideas and put them in new products.
I'm no expert but it seems like a descendent of LSTMs. There's a series of papers which show how to reformulate attention as RNNs which arrives at linear attention. Then they add a decay term to get mamba2. Then they add modified the decay term as like a scale to apply both to the existing state and the new update to get delta net. Then they added a gate matrix on the output to get gated delta net. Then Kimi Linear Attention seems to be gated delta net with a more expressive gate. The Gated DeltaNet paper recaptilulates this evolution decently well. But yeah, it feels like they're starting with the same lego blocks and assembling them in similar shapes to accomplish similar but slightly distinct modules.
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
Agreed. I think when it comes to light that Claude has been known to say “I’m DeepSeek” that everyone has had their hand in that cookie jar. Moreover, paying for API calls hardly seems like an attack; ToS violation to be certain but not in the same category of law as criminal activity like hacking.
wait, is there evidence of this? I've not observed it. It sounds like the kind of thing that I want to be true because it would be hilarious but that makes me suspicious.
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
> It's clearly a forcible, aggressive, non-consensual attempt to take something
Distillers are not taking anything, they are just making their model learn from better ones - isn't that the whole AI training doesn't violate IP argument?
They just aren't using the tool in compliance with the terms of service. Anthropic could ban them or take them to court maybe. Not an attack still.
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant.
The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks
All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.
If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
Snarky answer: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Realer answer: A combination of the above, plus group/bubble effect of all your coworkers saying the same thing. You as a group, conflate a bunch of concerns together (China, no-guardrails-AI, etc), decide that your group will be the responsible stewards of AI, and then anybody "stealing your work" appears dangerous - both to your livelihood and to the human race.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
No, it's perfectly reasonable once you get down to reality.
China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
We don't have the privilege to be able to hold western companies to higher ethical, legal, and environmental standards and not risk competitiveness.
That there is a whole lot of people right now who insist on doing above and still somehow praise China at every turn is something historians or news outlets will have to make sense of in some 5 years time.
Give up IP for end users and people will be pretty okay with giving up on IP for ai companies. You can't have different standards for special companies though.
> So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
Anthropic is currently angling for a "IP restrictions for China but not for me" la-la land scenario. If they were instead arguing for either of the choices you said, it wouldn't be hypocritical.
strong agreement, I've stopped using all closed weight models on principle, but the latest gemini models have increased hallucinations and now talk back, so double reason not to use them
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
I remember 15 years ago or so, one of my first student job was to evaluate Bing results compared to the same query on Google. Didn't know then that I was a distillation attacker.
It can easily be both. Also, they didn't use this innovation in K3 - K3 pre-training would have started months ago and the paper only mentions a 48B model. The people working at this level may not even be heavily involved in shipping a new iteration of K3, or at least theory contributions to it were done many months or even a year ago and after that it is all engineering.
Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
most new effort in training comes in the late phase with RL techniques
the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
This is actually a well-known phenomenon in ML, called "The Bitter Lesson".
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
The full essay is worth a read, it's pretty short http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Is that page served from a secure domain anywhere?
Out of curiousity, I like security just as much as the next guy and ensure that certs are valid, ciphers are fresh, and protocols are mature, but I am also not really concerned with an informational site that runs http being browsed. Would you mind sharing what your concern would be with browsing this content over http?
Not quite. The Wikipedia page is worth a look through if you don’t want to click on an http page. https://en.wikipedia.org/wiki/Bitter_lesson
Take your pick:
https://web.archive.org/web/20190401161916/http://www.incomp...
or
https://web.archive.org/web/20260727174324/http://www.incomp...
You could read it through internet archive if it's this important.
https://archive.ph/DQU4a
the page is nearly just a .txt file.
the reason for https is MITM injection, regardless of original content
ah, didn't know that. thanks!
If you'd like to dig more into the history, let's encrypt did so much to enable the movement to https by making certs free, and chrome with the red text and warnings in the url bar, which has since graduated to a full on page that requires clicking to still go despite the warnings
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
It's kind of sad that popular CS textbooks often focus on solving precise problems with lowest theoretical complexity bounds while ignoring more practical (but generally applicable) computation techniques.
In machine learning they call it "gradient descent", which in older days had analogies in techniques called "hill climbing", "local search" and "simulated annealing". Basically you have a function you need to optimize for, and you clumsily tweak the parameters so that you get the (locally) max/min value you wanted. These techniques were great at finding approximate, locally maximal solutions without trying all the possibilities at once (which is more akin to the kind of "brute force" in the traditional CS context).
I guess because these techniques were generally applicable yet the outputs were approximate and you couldn't analyze them much (no fancy O(n log n)), the theorists did not find them interesting and thus were not put into the spotlight of student's learning curricula.
In modern machine learning they do this gradient descent thing which is also tweaking the parameters bit by bit to optimize for the loss function, except that the parameters are now in the billions and trillions. The compute required is huge of course, but it's actually quite an "efficient" process, and it's not actually doing much of "brute forcing" at all. During training, the process is essentially, almost equivalent to, compressing the many many trillions of tokens of training data. To me it's quite amazing that they manage to complete such a process within a couple months of training, even if they have hundreds of thousands of GPUs...
In my math syllabus for Engineering there was a book "numeric analysis", it showed how you could find the solution to weird equations like x=ln(x)
I thought this is nowadays called gradient descent
It might be that what we consider a basic and very hard puzzle are extremely close together on a more absolute scale. The difference is often for us what proportion of humans can solve it. And the low end of that is still quite high up - animals that can solve things that are very basic for the vast majority of humans are pretty rare and known about, yet are capable of quite complex actions and learning and aren’t wildly different in scale of neurons to us.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model error, and that this portion of the cross entropy is reduced with scaling.
And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and memorize every combination. They call this "slingshot generalization".
There was a presentation at CS 25 Transformers United about some phenomena like chain-of-thought emerging only after training LLMs with at least 1T tokens and only in LLMs of certain size.
https://youtu.be/tVtOevLrt5U?t=923
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset:
https://en.wikipedia.org/wiki/Grokking_(machine_learning)
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
https://www.youtube.com/watch?v=NrO20Jb-hy0
so it seems like there is a benefit to giving models more space to learn in rather than forcing them to compress the knowledge from the start.
Exactly, I was going to suggest the Welch Labs videos on grokking. Especially the newer one at https://youtu.be/D8GOeCFFby4?si=yLI9zzcjsnEELUqy . They are really well done and really eye-opening.
> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.
> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them
How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.
Here's one way it could happen:
Let's say there's some circuit that does problem solving of the kind we call intelligence.
We dont know what this circuit looks like, but it exists in our brain.
Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit.
But if you try to fit it with fewer parameters than it needs, you're just going to get some linear approximation.
So basically a Nyquist rate type of concept.
https://en.wikipedia.org/wiki/Information_bottleneck_method
what is the approximation linear in?
In the output of this nonlinear model /s.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
Mote-Carlo is pretty useful still. Not sure if your statement holds
it's certainly not a definite procedure for determining if an arbitrary mathematical statement is true or not. it's more like educated guess and check which definitely scales up
Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
[0] https://arxiv.org/abs/2607.24653
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
Is it just me or does this read like a re-implementation of LSTMs?
More like RNN.
The nvidia version is heavy to compute (common problem with RNN and LSTM, also noted in the paper). Moonshot's Kimi K3 replaces part of it with some function that performs better.
I dont understand the details but, that in itself might a pretty big contribution.
Anyway, I'm amazed how fast these companies improve each other's ideas and put them in new products.
I'm no expert but it seems like a descendent of LSTMs. There's a series of papers which show how to reformulate attention as RNNs which arrives at linear attention. Then they add a decay term to get mamba2. Then they add modified the decay term as like a scale to apply both to the existing state and the new update to get delta net. Then they added a gate matrix on the output to get gated delta net. Then Kimi Linear Attention seems to be gated delta net with a more expressive gate. The Gated DeltaNet paper recaptilulates this evolution decently well. But yeah, it feels like they're starting with the same lego blocks and assembling them in similar shapes to accomplish similar but slightly distinct modules.
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
This is just awesome.
I believe this is the repo:
https://github.com/MoonshotAI/Kimi-Linear
... holy cow!
if non standard transformers like this take off are companies like etched forked?
If you want to believe that the success of Kimi is about distillation attacks, ignore this.
I'd kindly suggest that we could also stop calling them "distillation attacks".
Agreed. I think when it comes to light that Claude has been known to say “I’m DeepSeek” that everyone has had their hand in that cookie jar. Moreover, paying for API calls hardly seems like an attack; ToS violation to be certain but not in the same category of law as criminal activity like hacking.
wait, is there evidence of this? I've not observed it. It sounds like the kind of thing that I want to be true because it would be hilarious but that makes me suspicious.
LLMs can't reliably answer what they are (w/o getting that info from system prompt/tool call/etc), so yea.
https://xcancel.com/teortaxesTex/status/2026130112685416881
I think I've seen same happening with some European languages as well.
Ask it in Chinese
Agreed, "distilled variants" might be more suitable.
Calling them variants is also inaccurate when the pretrained base and architecture are completely different.
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
> It's clearly a forcible, aggressive, non-consensual attempt to take something
Distillers are not taking anything, they are just making their model learn from better ones - isn't that the whole AI training doesn't violate IP argument?
They just aren't using the tool in compliance with the terms of service. Anthropic could ban them or take them to court maybe. Not an attack still.
We could call it distillation learning
A model is teaching another here
Everyone wins
Fine, then let's refer to the initial data collection/training as 'compilation attacks' going forward.
...or maybe we stop defaulting to adversarial paradigms for every conceivable situation.
"They are using and paying for an API I am selling...But they save the data and use it for something I don't like! I'm being attacked."
False dichotomy right?
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
That Apple lawsuit is against OpenAi, just for clarity
Wow, I should never comment first thing in the morning... Thanks for the correction, you’re right. I will see if I can still edit my comment.
Did Kimi or other open source models use a different corpus? Why is your animus directed specifically to Anthropic?
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
Yep, that second part of my comment was an article I read about OpenAI and my mind mixed it up with Anthropic. My mistake.
In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant.
The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
Also possibly true: Anthropic is running Kimi locally in their hardware and "distilling" it.
If they have any sense, they should be. It would be permitted under the licence, too (unless I'm misreading the k3 licence).
Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks
All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.
If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?
“Distillation” is just indirectly pirating the largely pirated training data used to train the original model.
“You stole my warez!”
If we do it, it's training a model. When they do it, it's distillation attack. - Anthropic
"You are distilling what I have rightfully pirated."
Is there even a steelman against this?
The current steelman argument against this is "China bad, west good".
The distillation complaints to me sound like when a casino complains about card counting
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
i want china to win so that i get access to ai and not restricted and censored.
the chinese models are less censored, you'd better believe it.
try asking claude about its 'guardrails' (restrictions), very high chance anthropic will censor it.
> chinese models are less censored
Tried spicy geopolitical dispute questions?
Yes, and they aren't a problem. I get tired of people claiming this. Go download Qwen 3.6 and run it yourself and fire away.
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
How does anyone justify this? How can you argue this in good faith?
Snarky answer: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Realer answer: A combination of the above, plus group/bubble effect of all your coworkers saying the same thing. You as a group, conflate a bunch of concerns together (China, no-guardrails-AI, etc), decide that your group will be the responsible stewards of AI, and then anybody "stealing your work" appears dangerous - both to your livelihood and to the human race.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
No, it's perfectly reasonable once you get down to reality.
China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
We don't have the privilege to be able to hold western companies to higher ethical, legal, and environmental standards and not risk competitiveness.
That there is a whole lot of people right now who insist on doing above and still somehow praise China at every turn is something historians or news outlets will have to make sense of in some 5 years time.
Give up IP for end users and people will be pretty okay with giving up on IP for ai companies. You can't have different standards for special companies though.
> So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
Anthropic is currently angling for a "IP restrictions for China but not for me" la-la land scenario. If they were instead arguing for either of the choices you said, it wouldn't be hypocritical.
why should western AI companies be held to different standards than Chinese ones? Neither of them are your buddy.
They shouldn't be. But the point is, they are. OpenAI and Anthropic are paying many rights holders for access to their data (reddit, NYT, etc.).
So distillation, among other things, allows Chinese labs to indirectly benefit from these arrangements at no cost to them.
Oh really - how much are individual Redditors getting paid?
Google is apparently taking a different stance and offering distillation as a paid product
https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
You don't get to take distilled model home, it all stays with Google.
It's "optimize your costs in our garden" product.
yup, strings are certainly attached when dealing with US Big Tech / Ai
I recommend Fireworks as an alternative
But nobody wants to distill Google's models, Gemini is really bad.
I think GLM 5.2 is in part distilled from it.
strong agreement, I've stopped using all closed weight models on principle, but the latest gemini models have increased hallucinations and now talk back, so double reason not to use them
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
Reminds me of this btw:
https://www.bbc.com/news/technology-12343597
Microsoft replied that Bing uses “many different signals” —- including cribbing from Google :-)
I remember 15 years ago or so, one of my first student job was to evaluate Bing results compared to the same query on Google. Didn't know then that I was a distillation attacker.
It can easily be both. Also, they didn't use this innovation in K3 - K3 pre-training would have started months ago and the paper only mentions a 48B model. The people working at this level may not even be heavily involved in shipping a new iteration of K3, or at least theory contributions to it were done many months or even a year ago and after that it is all engineering.
This paper is from last year
Does one have to exclude the other?
Well said.
The distillation theory does not even make sense as Fable was only around for days (effectively) before Kimi was released.
Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.
Another banger from Zhang et. al
(2025)
As it's 9 months old and they just had a major model release
For K3 read this instead: https://arxiv.org/abs/2607.24653
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
Same with DeepSeek papers, they are a joy to read.
Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?
most new effort in training comes in the late phase with RL techniques
the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)
Rather under-discussed back then: https://news.ycombinator.com/item?id=45766937
I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.
It's not the same KDA as used in Kimi Linear, though.
What's the difference? They are both called Kimi Delta Attention.
The differences are explained in section 2.1.1 of the Kimi K3 technical report: https://arxiv.org/pdf/2607.24653#page=4