> On-policy distillation: The expanded student model learns token-specific feedback from teacher distributions generated on self-sampled sequences.
The weird thing is to see this based off the KV-Cache in the diagrams.
Completely different from the Databricks RAFT [1].
My team built "Dreamtime" earlier this year [2], which used a different mechanism to distill out learnings across multiple sessions.
This didn't pull from the token layer, but from user nudges on unsatisfactory responses.
This works like a sawtooth wave where we accumulate context, check for dissonances (a user gets to say "It depends" and fix it), compress it back, retire the old one.
The audit requirements are the real problem for customers, where you need to be able to ask "When did this change? and who changed it?" which was hard to answer from the lowest layer of the model unless you can retain lineage to the original sessions.
A model we train can't serve as an accomplice to a user who wants to hide some financial crimes by nudging the agent repeatedly.
I know we just had the carbon announcement which was distinctly confusing, but comparatively I think that this is an exceptionally well-written article. I do believe that Google has some exceptional scientific minds as illustrated by their ability to take highly technical concepts and explain them in an understandable way for the layperson.
As for the technical nature of this, I'm not sure if I've totally understood correctly despite how well it's written, but are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce what gets stored in those MoE weights? That to me seems like a solution which only Google is best placed to solve due to their fully integrated AI stack and historic experience with hardware optimisation, but which would be rather hard to do if you are a more resource and compute constrained organisation
> are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce...
Sort of. As the model is interacted with, short term "memories" are accumulated in a sort of buffer/cache. During "sleep" these "memories" are given a ranking/score, and the useful bits are integrated into the weights.
They say the weight update can be done by computing a transform where the end state is the old weights with the addition of the useful semantic knowledge contained in memories. Rather than just additively chucking the entire memory concepts into weights. That's pretty neat.
The subtext for the memory architecture diagram sums the whole thing up pretty tidily I think:
> Input streams from the active phase are stored in short-term memory before entering the sleep cycle, which bifurcates into Consolidation (transferring fragile short-term tokens into expanded long-term parametric memory using low-rank structures and MoE) and Dreaming (autonomous self-improvement via synthetic routing with random experts and RL refinement).
A very low tech version of this would be maintaining your md files after a work period. Down rank or delete irrelevant bits, mark as important and prioritize relevant bits. What's important is not all context that is important at the beginning of a work period remains important at the end of the period. It's this "learning" that needs to be captured.
I wonder how well this would translate with a much larger model. They ran the experiments with comparatively tiny models. You would presumably need to churn a lot more tokens to move the needle of a multi-trillion-parameter model's weights. It might just be that it's not that hard to improve Llama-3B and Llama 3-8B because they're not very good to begin with. They compare it to other methods - in-context learning and TTT - but I'd be curious to see it compared to a "control group" of RLHF.
As an aside, I get that this is a preprint, but it's very odd to see a piece of writing with obvious typos and grammar issues (e.g. page 13: "We attributes this results to," "In this setting (see Table 4), Sleep achieves a 80% success rate, higher then the other methods.") in an era with LLMs. It sort of locks in that this was written by human beings, and I'm all for that, but it jumps out at me more than it used to.
I'm seeing so many more typos on the internet than I did a few years ago. I'm convinced some of it is deliberate to ensure readers the writer isn't AI.
From my own experience, a significant minority of my daily writing is sent to an LLM, which is very forgiving of typos. I would not be surprised if people are getting sloppier.
I have a theory that every typo you make distracts the LLM from whatever you are trying to get it to do, so I avoid typos as much as possible. I have no evidence that this is true though.
in my case it very much is. im now deliberately leaving slightly off from correct writing, or exaggerated slightly. as a sort of organic behavioral kintsugi
I love that they tested it on Qwen 3. I’m curious if they’ve experimented with putting the new parameters onto system RAM or even an SSD. Big bummer they didn’t publish that part.
Well, you will have created an important part an /ideal/ human that will finally fulfill the mandate, and with more resources (read and digest many orders of magnitude more).
And for what the cost are concerned: to call it more expensive in instance terms I want to see the numbers first, and - have you consider the cost of "unfinished" humans as liabilities? The costs of fools, of crooks, of the vile, of conformists etc.?
More on memory consolidation from neurosci and neurology:
- "New study shows highly creative people’s brains work differently from others'" (2022) https://news.ycombinator.com/item?id=41921738 ; and does this correlate for example like REM/non-REM and ADHD? How much does this topology vary amongst the population?
- "Brain found to store three copies of every memory"
https://news.ycombinator.com/item?id=41352124 but actually more than three: {in the hippocampus {EBN, LBN,}, prefrontal cortex, and engrams} but functional activations drift even just days apart given representational drift.
> On-policy distillation: The expanded student model learns token-specific feedback from teacher distributions generated on self-sampled sequences.
The weird thing is to see this based off the KV-Cache in the diagrams.
Completely different from the Databricks RAFT [1].
My team built "Dreamtime" earlier this year [2], which used a different mechanism to distill out learnings across multiple sessions.
This didn't pull from the token layer, but from user nudges on unsatisfactory responses.
This works like a sawtooth wave where we accumulate context, check for dissonances (a user gets to say "It depends" and fix it), compress it back, retire the old one.
The audit requirements are the real problem for customers, where you need to be able to ask "When did this change? and who changed it?" which was hard to answer from the lowest layer of the model unless you can retain lineage to the original sessions.
A model we train can't serve as an accomplice to a user who wants to hide some financial crimes by nudging the agent repeatedly.
[1] - https://arxiv.org/abs/2403.10131
You forgot the reference [2] ...
Such a waste.
I know we just had the carbon announcement which was distinctly confusing, but comparatively I think that this is an exceptionally well-written article. I do believe that Google has some exceptional scientific minds as illustrated by their ability to take highly technical concepts and explain them in an understandable way for the layperson.
As for the technical nature of this, I'm not sure if I've totally understood correctly despite how well it's written, but are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce what gets stored in those MoE weights? That to me seems like a solution which only Google is best placed to solve due to their fully integrated AI stack and historic experience with hardware optimisation, but which would be rather hard to do if you are a more resource and compute constrained organisation
They say the weight update can be done by computing a transform where the end state is the old weights with the addition of the useful semantic knowledge contained in memories. Rather than just additively chucking the entire memory concepts into weights. That's pretty neat.
The subtext for the memory architecture diagram sums the whole thing up pretty tidily I think:
A very low tech version of this would be maintaining your md files after a work period. Down rank or delete irrelevant bits, mark as important and prioritize relevant bits. What's important is not all context that is important at the beginning of a work period remains important at the end of the period. It's this "learning" that needs to be captured.
I wonder how well this would translate with a much larger model. They ran the experiments with comparatively tiny models. You would presumably need to churn a lot more tokens to move the needle of a multi-trillion-parameter model's weights. It might just be that it's not that hard to improve Llama-3B and Llama 3-8B because they're not very good to begin with. They compare it to other methods - in-context learning and TTT - but I'd be curious to see it compared to a "control group" of RLHF.
As an aside, I get that this is a preprint, but it's very odd to see a piece of writing with obvious typos and grammar issues (e.g. page 13: "We attributes this results to," "In this setting (see Table 4), Sleep achieves a 80% success rate, higher then the other methods.") in an era with LLMs. It sort of locks in that this was written by human beings, and I'm all for that, but it jumps out at me more than it used to.
I'm seeing so many more typos on the internet than I did a few years ago. I'm convinced some of it is deliberate to ensure readers the writer isn't AI.
From my own experience, a significant minority of my daily writing is sent to an LLM, which is very forgiving of typos. I would not be surprised if people are getting sloppier.
I have a theory that every typo you make distracts the LLM from whatever you are trying to get it to do, so I avoid typos as much as possible. I have no evidence that this is true though.
Yes we are.
in my case it very much is. im now deliberately leaving slightly off from correct writing, or exaggerated slightly. as a sort of organic behavioral kintsugi
The funny thing is that you can easily get a LLM to introduce typos and grammar mistakes if you ask it.
The actual paper: https://arxiv.org/pdf/2606.03979
I love that they tested it on Qwen 3. I’m curious if they’ve experimented with putting the new parameters onto system RAM or even an SSD. Big bummer they didn’t publish that part.
AFAIK Muse (from Meta) dreams as well... every night... to consolidate its knowledge graph.
Reminds me of one of my favourite ML papers, DreamCoder: https://dl.acm.org/doi/pdf/10.1145/3453483.3454080
> Parameter expansion then provides the capacity needed to write these selected memories without overwriting previously consolidated knowledge.
This seems like the limiting factor here, depending on efficiency
Good, this seems one of the crucial missing pieces towards AGI. Add critical thinking during cleanup, and we get very close to it.
Add critical thinking and we’ll create a human but more expensive.
Well, you will have created an important part an /ideal/ human that will finally fulfill the mandate, and with more resources (read and digest many orders of magnitude more).
And for what the cost are concerned: to call it more expensive in instance terms I want to see the numbers first, and - have you consider the cost of "unfinished" humans as liabilities? The costs of fools, of crooks, of the vile, of conformists etc.?
If there were any jobs left that humans could get paid for, this would eliminate a big chunk of them.
how is that good? Certainly seems like a bleak future to me.
A future in which finally we could have synthetic intelligence, reliable judges, accomplished consultants?
Thank you again for reminding me I have a reason to live.
this is valid for a lot of the AI future.
Who can afford to sleep in this race to the bottom?
But my dear friends, they were already dreaming
Of electric sheep?
More on memory consolidation from neurosci and neurology:
- "New study shows highly creative people’s brains work differently from others'" (2022) https://news.ycombinator.com/item?id=41921738 ; and does this correlate for example like REM/non-REM and ADHD? How much does this topology vary amongst the population?
- "Discovery of Memory "Glue" Explains Lifelong Recall: KIBRA Protein, PKMzeta" (2024) https://news.ycombinator.com/item?id=41921738
- "Brain found to store three copies of every memory" https://news.ycombinator.com/item?id=41352124 but actually more than three: {in the hippocampus {EBN, LBN,}, prefrontal cortex, and engrams} but functional activations drift even just days apart given representational drift.
- "Memories stick around even after half the synapses are gone" (2026) https://news.ycombinator.com/item?id=49404593
Does this shit actually work lol