acceptodds
Under review as a conference paper at ICLR 2027

Language Models Need Sleep: Learning to Self Modify and Consolidate Memories

Abstract

The past few decades have witnessed significant advances in the design of machine learning algorithms– from early studies of task-specific shallow models to more general deep Large Language Models (LLMs). Despite showing promising results on tasks requiring instant prediction or in-context learning, existing models lack the ability to continually learn and effectively transfer their temporal in-context knowledge into their long-term parameters. Inspired by the human learning process, we introduce a "Sleep" paradigm that enables models to continually learn, distill their fragile short-term memories into stable long-term knowledge through replay, and recursively improve themselves through a "Dreaming” process. More specifically, Sleep consists of two stages: (1) Memory Consolidation: an upward distillation process, called Knowledge Seeding, in which the memories of a smaller model are distilled into a larger network to provide greater capacity, while preserving the knowledge. As a proof of concept, we present a new Generalized Distillation process for Knowledge Seeding, combining on-policy distillation with Reinforcement Learning (RL)-based imitation learning); and (2) Dreaming: a self-improvement phase, in which the model uses RL to generate a curriculum of synthetic data for rehearsing new knowledge and refining existing capabilities without human supervision. We provide rigorous theoretical analysis showing that the Knowledge Seeding objective admits a closed-form truncated SVD solution (under a linear upper network) in a space jointly shaped by the student’s on-policy covariance and a reward metric tensor. We also show that random MoE expert injection during dreaming provably improves the conditioning of the gradient covariance. Our experiments on long-horizon, continual learning, knowledge incorporation, and few-shot generalization tasks support the importance of the Sleep stage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, Memory Consolidation, Knowledge Distillation, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9uDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a wake–sleep approach to continual learning. The model periodically adds parameters and distills knowledge from fast-updating modules into slower ones. It then generates synthetic data and learns which data helps improve itself. Experiments cover continual learning, long-context understanding, reasoning, and knowledge incorporation.

Critical strengths

Transferring temporary knowledge into slower modules is an interesting direction. The paper gives a concrete way to combine expansion, distillation, and synthetic replay.

The evaluation covers several tasks, and the ablations help show where the gains come from. Consolidation and dreaming seem to provide complementary benefits.

Critical weaknesses

It is hard to tell how much improvement comes from the proposed mechanism versus more parameters and more training. OPSD plus expansion is a useful baseline, but the remaining gains are often small, with no uncertainty reported. The continual-learning setup also needs more detail.

The theoretical claims seem stronger than what is shown. The SVD result applies to a simplified surrogate, and the conditioning results require assumptions about the covariance and task-gradient direction that are not checked empirically.

Questions for authors

How does Sleep compare with baselines given the same parameter capacity and total compute, including generation and inner-loop training?

What are the task order, consolidation schedules, and parameter counts in the continual-learning experiments? Can you show retention after each task?

Is the on-policy distribution held fixed for the SVD result? Do the assumptions behind the conditioning results hold in practice?

Do the gains hold across seeds? Several improvements over OPSD plus expansion are small, and ARC uses only eight evaluation tasks.

3 comments

We thank the reviewer for the concise and pointed review, and for noting that consolidation and Dreaming provide complementary benefits. The questions are the right ones to ask. Below we answer each directly, distinguishing what the current paper already shows from what we add. Where a point overlaps with another review, we give a self-contained summary here.

W1. It is hard to tell how much improvement comes from the proposed mechanism versus more parameters and more training…

Response: We address this in three points; details are in Q1, Q2 and Q4.

  1. Parameters do not explain the gains. The expansion adds only 1.15% of the backbone's parameters, and the active parameter count is unchanged. In the matched-parameter 2×2 design of Table 1, the KS objective beats OPSD both without expansion (+0.6 mean over the three benchmarks) and with it (+0.8).
  2. Compute does not explain the gains. Given Sleep's total GPU-hours, including all generation, the strongest baseline (OPSD + Expansion) reaches 63.9 vs. Sleep's 64.4 (three-benchmark mean; Q1). GRPO already takes 5× more optimization steps than Sleep, and SFT needs 3.6–4.8× Sleep's wall-clock to match it (App. D.5).
  3. Uncertainty and setup. We add seeds and CIs (Q4), the full continual-learning protocol, and per-task retention (Q2).

W2. The theoretical claims seem stronger than what is shown. The SVD result applies to a simplified surrogate, and the conditioning results require assumptions…

Response: Thank you for this comment. We will make the scope of each result explicit in the abstract, Sec. 4 and the conclusion. The underlying results are also more general than a first reading may suggest:

  • The SVD result. rsemr_{\rm sem} and rabsr_{\rm abs} are binary or thresholded per sample, but at each prefix position (on 1⊥\mathbf{1}^\perp) Eq. (4) optimizes their expectation under the student, which is a smooth function of the logits. At Z⋆Z^\star, the combined KL-regularized objective therefore has a PSD quadratic local model by Taylor's theorem. Theorem 5 holds for any context-independent PSD metric with Q~≻0\tilde Q\succ0 (i.e., col(Wu)∩null(Q)={0}\mathrm{col}(W_u)\cap\mathrm{null}(Q)=\{0\} and any target. With context-dependent curvature, it holds for the averaged metric in Qˉ~\tilde{\bar Q} (retaining exact Q(h)Q(h) in CC) under the standard K-FAC approximation. For the KL term, Lemma 9 shows that KL is locally equal to a quadratic in the Fisher metric (requiring 1∉col(Wu)\mathbf{1}\notin\mathrm{col}(W_u)), so the upper bound is not needed. The closed form explains the update but does not compute it: A,BA,B are trained on the exact Eq. (4). We will present Thm 1 as "a second-order model of the KS objective admits a closed-form truncated-SVD solution".

  • The conditioning results. We will move every condition (including the inherited Thm 6(iii) conditions and O(ε3)O(\varepsilon^3) remainder in Thm 8) into the main-text Theorems 2–3, together with two minor technical refinements (an explicit κ≥π/(π−2)\kappa\ge\pi/(\pi-2) condition in Eq. 54, and labeling the post-filtering constant as an approximation). Two conditions are milder than they may look:

    • The spectral gap only sets the constant. The improvement factor max⁡(1−2π,λ2/λ1)\max(1-\tfrac2\pi,\lambda_2/\lambda_1) is below 1 whenever the top eigenvalue is simple.
    • Alignment is not needed for safety. A new lemma shows that, for any sign-symmetric gradient law with Pr⁡(gC⊤g=0)=0\Pr(g_C^\top g=0)=0 (including after norm filtering), the reward step is a rank-one downdate, Σdream=Σ−mm⊤⪯Σ\Sigma_{\rm dream}=\Sigma-mm^\top\preceq\Sigma. It never amplifies variance along a direction; alignment only decides whether it improves κ\kappa.

    Whether the assumptions hold in practice is measured in Q3.

Q1. How does Sleep compare with baselines given the same parameter capacity and total compute, including generation and inner-loop training?

Response:

Parameters. The expansion adds the 5 low-rank (dimension-64) expert blocks of App. D: 94.4 M parameters, 1.15% of the backbone. The active parameter count is unchanged. All Table 3 baselines train rank-64 LoRA adapters (174.6 M trainable parameters) with the same learning rate and batch size (Table 2). OPSD + Expansion has exactly Sleep's parameters.

Total compute, including generation and the inner loop. We now report GPU-hours end-to-end and give the strongest baseline Sleep's full budget. The budget counts teacher/student generation and semantic-judge calls; the baseline may spend the extra budget on additional OPSD training and generation.

Qwen3-8B (3 seeds)GPU-hAIME-24AIME-25HMMT-25
OPSD + Expansion (as in paper)2577.7 ± 1.068.0 ± 1.145.5 ± 1.3
OPSD + Expansion (compute-matched to Sleep)4077.9 ± 1.168.2 ± 1.245.6 ± 1.4
Sleep4078.8 ± 1.068.7 ± 1.145.7 ± 1.3

Dreaming's cost includes generation and inner-loop training. In the sequential knowledge-incorporation study (response to Reviewer 6ZHJ, W2), Dreaming only (SEAL-style) gets the same total budget as full Sleep and reaches 29.3 vs. 40.0 final average accuracy. For the remaining baselines, GRPO already takes 5× more steps than Sleep (Table 2), and App. D.5 shows that SFT needs 3.6–4.8× Sleep's wall-clock to reach the same accuracy.

Q2. What are the task order, consolidation schedules, and parameter counts in the continual-learning experiments? Can you show retention after each task?

Response: Thank you for the suggestion. To make the setup fully explicit, the revision adds:

CLINCBankingDBpedia
# tasks × classes per task10 × 157 × 117 × 10
Task orderfixed random permutation, identical for all methods
ProtocolMomeni et al. (2025): one pass per task, no replay of earlier data
# CMS levels3 levels
Total / active / added parameters3.21B / unchanged / 55.1M
Metricaccuracy over all classes seen so far

Retention after each task. The final metric covers all classes seen so far, so Fig. 3 already penalizes forgetting. HOPE + Sleep improves over the identical HOPE backbone without the sleep phase in all 6 dataset × backbone settings. (We will make the naming in Sec. 5.1 consistent so that "HOPE" always refers to the no-sleep baseline.)

The revision adds the full accuracy matrix At,jA_{t,j} (accuracy on task jj after training on task tt). From it we report final average accuracy and forgetting, F=1T−1∑j<T(max⁡t<TAt,j−AT,j)F=\tfrac1{T-1}\sum_{j<T}(\max_{t<T}A_{t,j}-A_{T,j}):

Llama-3BCLINC AˉT\bar A_T / FFBanking AˉT\bar A_T / FFDBpedia AˉT\bar A_T / FF
HOPE (no sleep)69.0 / 2563.5 / 2842.2 / 50
HOPE + Sleep76.9 / 1869.0 / 2347.4 / 46

We also add a multi-round sequential knowledge-incorporation study for full Sleep, consolidation only and Dreaming only. It tracks new learning and retention after every sleep cycle, including Dreaming (response to Reviewer 6ZHJ, W2). Full Sleep has forgetting 8.0, vs. 18.0 for Dreaming only.

Q3. Is the on-policy distribution held fixed for the SVD result? Do the assumptions behind the conditioning results hold in practice?

Response:

On-policy distribution. Yes. Theorem 5 treats HS=E[hℓ−1s(hℓ−1s)⊤]H_S=\mathbb E[h^s_{\ell-1}(h^s_{\ell-1})^\top] as fixed. In the GKD distillation term of Algorithm 3, student rollouts are drawn under a stop-gradient (line 6), executing a semi-gradient step with HSH_S held fixed (while line 15 uses policy gradient on the discrete LTI reward, which Theorem 5 models via the fixed-HSH_S local surrogate). It does not hold over the whole trajectory: as A,BA,B change, the student's samples, and hence HSH_S, change too. The consistent statement for the trained adapter is therefore self-referential. At any fixed point M=AB⊤M=AB^\top that is second-order stationary for the frozen-distribution surrogate at its own equilibrium covariance HS(M)H_S(M), after the change of variables in App. A.4.2 this problem is a standard low-rank approximation whose local minima are global (Baldi & Hornik, 1989), so the closed form applies self-consistently at HS(M)H_S(M). The rank-allocation interpretation then applies with HSH_S taken at convergence. We will state this explicitly.

Do the conditioning assumptions hold? We measured them in the SQuAD dreaming phase, following the protocol in our response to Reviewer nwwj (Q3). Conditioning is reported on the top-kk eigenspace, with bootstrap error bars:

Assumption / predictionMeasured
Σinject\Sigma_{\rm inject} PSD and better conditioned than Σ0\Sigma_0 (top-kk subspace)λmin⁡\lambda_{\min} = 0.00; κ\kappa 15 vs. 100
κ(Σraw)<κ(Σ0)\kappa(\Sigma_{\rm raw})<\kappa(\Sigma_0)ratio 0.90
Task gradient aligned with the top eigen-directioncos⁡Sk2\cos^2_{\mathcal S_k} = 0.11; ρ\rho = 0.40
Spectral gap λ2/λ1≤0.363\lambda_2/\lambda_1\le0.3630.50
κ(Σdream)/κ(Σpre-ReSTEM)≤max⁡(0.363,λ2/λ1)\kappa(\Sigma_{\rm dream})/\kappa(\Sigma_{\rm pre\text{-}ReSTEM})\le\max(0.363,\lambda_2/\lambda_1)0.88 vs. 0.50

Response:

Seeds. Thank you for the suggestion. Note that OPSD + Expansion is our own strengthened baseline, which already includes Sleep's expansion; its margins (+1.3 / +0.8 / +0.2) therefore isolate the KS objective alone, and we now report them with uncertainty. The 3-seed results are in the Q1 table: Sleep's mean exceeds OPSD + Expansion on all three benchmarks (by 1.1 / 0.7 / 0.2), and the paired per-seed difference is positive in 6 of 9 seed × benchmark pairs. We also report bootstrap CIs (over problems and samples) for every row of Tables 1, 3 and 4.

Against the external baseline OPSD, the margin is +2.6 / +1.6 / +1.0, and Sleep beats OPSD in all 6 model × benchmark cells of Table 3.

ARC. Thank you for raising this. We follow the SEAL protocol: 11 training and 8 held-out tasks, with 5 dreams applied per test task, so each success rate rests on only 40 adaptations. Treating these as independent, the 95% Wilson intervals are [65.2, 89.5] for Sleep (32/40) and [57.2, 83.9] for SEAL (29/40). Because the adaptations are nested within 8 tasks, the true intervals are wider still. Table 5 therefore does not establish a significant difference from SEAL; the only clear separation is from TTT (4/40, [4.0, 23.1]). We will state this in the paper, report per-task outcomes, and present ARC as protocol-matched supporting evidence that complements the main results on math, SQuAD and continual learning.

We hope these answers and results address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising their score.

Reviewer_6ZHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sleep, a framework for continual learning through memory consolidation and self-modification. It transfers knowledge from faster-updating memory modules into newly added low-rank experts in slower modules using distillation and imitation learning. A subsequent Dreaming phase generates and selects synthetic data for further improvement. Experiments cover continual learning, long-context understanding, knowledge incorporation, and reasoning tasks.

Critical strengths
  1. The paper is well motivated, and the overall wake–sleep framework is intuitive. Consolidating temporary knowledge into more stable parameters is an interesting direction for continual learning.

  2. The experiments cover a broad range of tasks. The paper also provides theoretical analyses of knowledge consolidation and random expert injection under stated assumptions, offering useful perspectives on the proposed mechanisms.

Critical weaknesses

I have two main concerns:

  1. My first concern is that several important implementation details are insufficiently specified.

Sleep combines multiple mechanisms, including frequency-dependent updates, parameter expansion, distillation, imitation learning, and synthetic-data selection. Although some optimization settings are provided, I could not find sufficiently detailed, benchmark-specific configurations for the consolidation schedule, the amount of generated data and optimization performed per Sleep cycle, or the semantic reward model. Referring to the original benchmark protocols does not fully specify these newly introduced procedures. This makes it difficult to reproduce the method and assess how much additional computation contributes to the improvements.

  1. I also find the evidence for retaining old knowledge through repeated Sleep cycles somewhat limited.

Section 5.1 explicitly excludes self-improvement from the continual learning experiments. While Table 4 evaluates the full framework for knowledge incorporation, it does not show how performance on earlier knowledge changes over successive consolidation and Dreaming cycles. Since reducing forgetting during repeated self-modification is a central motivation, I would like to see a sequential evaluation that tracks both new learning and retention of earlier knowledge, comparing the full framework with consolidation-only and Dreaming-only variants. This would help establish whether the two stages work well together over repeated adaptation.

Questions for authors
  1. Could you clarify the relationship between HOPE and Sleep in the experiments? Several result descriptions refer to HOPE. What architectural and training differences distinguish the HOPE baseline from the Sleep variants?

  2. How are the prompts x obtained when constructing the teacher-generated dataset in Algorithm 3? Are they generated by the model, taken from recent wake-phase inputs, or sampled through another procedure?

  3. During Dreaming, which parameters are updated by the inner-loop fine-tuning and the final ReSTEM update? Are these updates limited to newly added experts?

3 comments

We thank the reviewer for the thoughtful review, and for finding the wake–sleep framework well motivated and the evaluation broad. Both suggestions are valuable, and we have acted on them. The revision adds:

  • (i) a complete, benchmark-specific configuration table and an end-to-end compute breakdown;
  • (ii) a sequential multi-cycle evaluation that tracks new learning and retention for full Sleep, consolidation only and Dreaming only;
  • (iii) consistent naming of HOPE vs. Sleep, and the data sources and trainable parameter sets stated explicitly in Algorithms 3 and 5.

W1. Several important implementation details are insufficiently specified…

Response: Thank you for this helpful suggestion. To make the newly introduced procedures fully reproducible, the revision adds the following table; the excerpt is below, and the appendix version lists every hyperparameter.

Class-incr. (CLINC/Banking/DBpedia)MathSQuADARC
Backbone; memory levelsLlama-3B, Llama3-8B; HOPE, 3 MLPsQwen3-1.7B/8B; Transformer (attention = fast, MLP = slow)Qwen2.5-7B; Transformer and four-level CMSLlama-3.2-1B
Chunk sizes C(ℓ)C^{(\ell)}; # sleep cycles8K8K8K8K
New experts per cycle5 low-rank blocks, dlow=64d_{\rm low}=64, in every setting; active parameters unchanged
Dreaming used? mm; Top-kk + bb; ReSTEM rounds EEnononoyes; 5 dreams/passage
OptimizerLR 5×10−65\times10^{-6}, batch 32, LoRA r=64r=64, α=128\alpha=128 (Table 2), unless stated

Semantic reward model. rsemr_{\rm sem} uses a frozen variant of model. It returns 1 iff it judges d^\hat d and dd semantically equivalent. It is used only inside Knowledge Seeding (Alg. 4), never at evaluation, and it is distinct from the Dreaming reward (Eq. 5), which measures downstream improvement after temporary adaptation.

How much does additional computation contribute? We report each method's total GPU-hours, split by phase:

  • Wake training.
  • Knowledge Seeding, including teacher/student generation and judge calls.
  • Dreaming, including dream generation, gradient scoring, inner-loop LoRA-SFT and the ReSTEM update.
SettingWakeKSDreamingSleep totalStrongest baseline total
Math (Qwen3-8B)832—4025 (OPSD + Exp.)
SQuAD (n=200)26283624 (SEAL)

Three results bound the role of compute:

  1. When the strongest baseline, OPSD + Expansion, is given Sleep's full GPU-hour budget, it reaches 77.9 / 68.2 / 45.6, vs. Sleep's 79.2 / 69.0 / 46.1.
  2. GRPO already takes 5× more optimization steps than Sleep (Table 2), and App. D.5 shows that SFT needs 3.6–4.8× Sleep's wall-clock to reach the same accuracy.
  3. In the sequential study below, Dreaming only is given the same budget as full Sleep.

Code. We will release code and all per-benchmark configuration files, including the reward-model prompt, with the camera-ready version.

W2. I also find the evidence for retaining old knowledge through repeated Sleep cycles somewhat limited…

Response: Thank you for this excellent suggestion. It is a direct test of the paper's central claim, and we have added it.

What the current results already show. Two experiments are sequential, although we did not present them as retention curves:

  • Class-incremental learning (Fig. 3). The final accuracy is measured over all classes seen so far, so it already penalizes forgetting of earlier classes. HOPE + Sleep improves over the identical HOPE backbone without sleep in all 6 dataset × backbone settings.
  • Continual translation (Fig. 4). After the model learns a second language, ICL drops sharply on the first one. Sleep retains its single-language performance, and more consolidation levels retain more.

As the reviewer notes, both experiments use consolidation only, and both report a single endpoint.

New sequential evaluation: full vs. consolidation only vs. Dreaming only. We follow the knowledge-incorporation setting of SEAL (Zweiger et al., 2025), which itself reported forgetting under repeated self-edits, and make it sequential:

  • Setup. Passages arrive in T=T= 8 rounds of 1 passage. Each round is one wake exposure followed by one sleep cycle.
  • Measurement. After every round we evaluate no-context QA accuracy on the passages of every earlier round, never using these questions for the dreaming reward or for selection. This gives an accuracy matrix At,jA_{t,j}: performance on round-jj passages after round tt.
  • Metrics. We report three quantities:
    • New-knowledge accuracy, 1T∑tAt,t\tfrac1T\sum_t A_{t,t}.
    • Final average accuracy, AˉT=1T∑jAT,j\bar A_T=\tfrac1T\sum_j A_{T,j}.
    • Forgetting, F=1T−1∑j<T(max⁡t<TAt,j−AT,j)F=\tfrac1{T-1}\sum_{j<T}(\max_{t<T}A_{t,j}-A_{T,j}).
  • Controls. All variants use the same backbone and capacity, and Dreaming only is compute-matched to full Sleep.
MethodNew-knowledge acc. ↑Final avg. acc. AˉT\bar A_T ↑Forgetting FF ↓
Dreaming only (iterated, SEAL-style)45.029.318.0
Consolidation only36.033.82.5
Full Sleep47.040.08.0

Full Sleep has the highest new-knowledge accuracy (47.0) and the highest final accuracy (40.0). Its forgetting (8.0) is less than half that of Dreaming only (18.0), whose forgetting grows over rounds, as in SEAL's sequential experiment. Consolidation only forgets least (2.5), but it gains only ~4 points over the base model. The full At,jA_{t,j} curves will be added to the appendix.

We also add the per-task accuracy matrix and the forgetting metric for the class-incremental runs (HOPE vs. HOPE + Sleep; see our response to Reviewer 9uDP, Q2).

Q1. Could you clarify the relationship between HOPE and Sleep in the experiments?…

Response: HOPE (Behrouz et al., 2025) is an architecture: a sequence model followed by a Continuum Memory System (CMS), i.e., a chain of MLP blocks updated at different frequencies (Eq. 1–2). Sleep is a consolidation procedure applied on top of a backbone. In the experiments:

  • HOPE (baseline): the HOPE backbone trained with the standard CMS rule of Eq. (2). Knowledge moves between levels only through end-to-end gradients ("online consolidation", Sec. 3.1). There is no expansion, no Knowledge Seeding, no expert reset and no Dreaming.
  • HOPE + Sleep: the same backbone, initialization, data stream and update schedule, plus a sleep phase at every chunk boundary (Algorithm 1). The sleep phase runs expansion → Knowledge Seeding → reset, and Dreaming where stated. "Sleep-kk" (Fig. 4) and "kk levels" (Fig. 8, Table 4) denote the number of CMS levels, i.e., the number of consolidation stages.
  • Transformer + Sleep (math; "Sleep (Transformer)" in Table 4): a Transformer is the two-level special case of CMS (Sec. 2.2). The fast memory is the attention/in-context state, and the slow memory is the MLP, into which the new low-rank experts are added.

Hence, in Fig. 3 the "Hope" and "Sleep" bars differ only in the sleep phase. Several sentences in Sec. 5 say "HOPE" where they mean HOPE + Sleep (e.g., "HOPE performs best" in Sec. 5.1). Thank you for bringing this to our attention. The revision uses "HOPE" exclusively for the no-sleep baseline and adds a one-line description of every compared variant to each caption.

Q2. How are the prompts x obtained when constructing the teacher-generated dataset in Algorithm 3?…

Response: They are recent wake-phase inputs. When the sleep phase for block ℓ′\ell' is triggered, xx is a prefix sampled from the last C(ℓ∗−1)C^{(\ell^*-1)} wake-phase inputs. These are exactly the inputs whose accumulated gradients define the sender's pending update (Eq. 2), and the continuation yy is sampled from the teacher. No external data and no stored labels are used: the sender's knowledge enters only through the teacher's continuations.

This is separate from the LTI construction. There, d(i)∈DTd^{(i)}\in\mathcal D_T are teacher samples conditioned on the same prefixes, and the student completes a random-length prefix of each. We will state both procedures explicitly in Algorithm 3 and Sec. 3.3.

Q3. During Dreaming, which parameters are updated by the inner-loop fine-tuning and the final ReSTEM update? Are these updates limited to newly added experts?

Response: The three update scopes are:

StageTrainable parametersPersistent?
Knowledge Seeding (Alg. 3)only the newly activated low-rank expert {A,B}\{A,B\}; everything else is frozen (Alg. 3, line 3)yes
Dreaming inner loop (Alg. 5, lines 14–16)a temporary LoRA adapter on an isolated copy of the model, used only to compute the binary rewardno; discarded after scoring
ReSTEM M-step (Alg. 5, line 21)only the low-rank experts activated in the current sleep cycle; base weights and all previously consolidated experts are frozenyes

Dreaming therefore writes only into parameters created in the current cycle, and earlier consolidated experts are untouched by construction. Freezing old parameters does not by itself guarantee unchanged outputs, because new experts and routing can still alter the computation. This is why we measure retention directly in the new sequential study (W2) rather than rely on the construction alone. We will state the trainable sets explicitly in Algorithms 3 and 5.

We hope these additions resolve the reproducibility and retention concerns. If they do, we would be grateful if the reviewer would consider raising their score.

Reviewer_nwwjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SLEEP, a periodic wake–sleep framework for continual learning in large language models. It consists of two main stages: Memory Consolidation, which builds on the Continuum Memory System to activate new low-rank experts in lower-frequency MoE memory modules and distill knowledge from higher-frequency modules into these new parameters through Knowledge Seeding; and Dreaming, which generates synthetic data through random injection of additional MoE experts, filters samples using gradient norms, and performs self-improvement through isolated LoRA-SFT, a binary improvement reward, and ReSTEM. The paper also provides a truncated SVD solution for the low-rank consolidation objective and a theoretical analysis of how random expert injection and reward-based selection affect the conditioning of the gradient covariance. The experiments cover class-incremental learning, long-context understanding, mathematical reasoning, knowledge incorporation, and few-shot abstract reasoning on ARC, providing relatively broad empirical coverage.

Critical strengths
  1. The research problem is important, and the overall framework is appealing. Periodically transferring short-term, high-frequency memories into more stable parameter modules directly addresses the stability–plasticity dilemma in continual learning. Rather than merely borrowing the metaphor of sleep, the paper provides a relatively complete algorithmic procedure.

  2. The system is well integrated. Parameter expansion, upward self-distillation, fast-memory resetting, synthetic data generation, and self-improvement are organized into a unified lifecycle. The pseudocode in the appendix also helps clarify the ordering of the different stages.

  3. The experimental coverage is broad. The paper evaluates continual learning, long-context understanding, knowledge incorporation, and reasoning, and includes diverse baselines such as SFT, GRPO, OPSD, SEAL, and HOPE. Positive results are reported on mathematical reasoning, SQuAD, and ARC.

  4. The authors attempt to analyze the core design rather than presenting only empirical results. The appendix contains relatively extensive theoretical derivations, while Tables 1 and 4 provide partial ablations of imitation learning, parameter expansion, random expert injection, gradient-based selection, and Dreaming.

Critical weaknesses
  1. The closed-form Knowledge Seeding solution applies to an additionally constructed quadratic surrogate objective, rather than the actual optimization objective.The actual reward in Section 3.3 consists of a binary semantic reward and an absolute token-level reward based on Levenshtein distance. However, in Appendix A.1, the authors model the negative reward as a squared logit distance within a semantic subspace, which leads to a quadratic objective and the truncated SVD solutions in Theorems 1 and 5. The paper does not establish that the actual semantic and edit-distance rewards can be equivalently represented by this quadratic form. Appendix A.3 proves that KL divergence can be upper-bounded by a squared logit distance, but minimizing an upper bound does not necessarily yield the minimizer of the original KL objective.

  2. The theoretical claims regarding conditioning improvements are substantially stronger than what the formal theorems guarantee. Theorem 2 in the main text states that random expert injection improves the conditioning of the gradient covariance. However, Theorem 6 in Appendix A.2 requires several additional conditions, including that the injection covariance be positive semidefinite on the support of the original covariance, that the injection covariance itself be better conditioned than the original covariance, and that the perturbation be sufficiently small under a Gauss–Newton approximation and a zero-mean expert-displacement assumption. The approximately 1−2/π≈0.361-2/\pi\approx0.36 multiplicative improvement in Theorem 3 further relies on a Gaussian gradient model, alignment between the task gradient and the dominant eigenvector, and a sufficient spectral gap satisfying λ2/λ1≤1−2/π\lambda_2/\lambda_1\leq1-2/\pi. The paper does not directly measure the gradient covariance, eigenvalue spectra, or condition numbers during actual training, so these theoretical mechanisms have not been empirically validated.

  3. The experiments are broad, but the causal attribution and statistical reliability of the core mechanisms remain insufficient. The current ablations do not adequately support the conclusion that the individual components jointly form an effective SLEEP mechanism: Table 1 presents ablations only for mathematical reasoning with Qwen3-8B. Moreover, removing the Semantic Reward improves AIME-25 from 69.0 to 69.2, which is not entirely consistent with the main text’s claim that all components contribute positively. The ablations of Dreaming, random expert injection, and gradient-based selection in Table 4 are helpful but are largely limited to SQuAD.

Questions for authors
  1. Can the authors systematically isolate the contributions of parameter expansion, Knowledge Seeding, and Dreaming under the same total parameter count and compute budget? I suggest comparing at least the following variants on class-incremental learning, SQuAD, and mathematical reasoning: no expansion, expansion only, expansion with standard distillation, full Knowledge Seeding, Dreaming only, and full SLEEP.
  2. What justifies representing the actual LTI reward in the quadratic logit form used in Appendix A.1?
  3. Can the authors provide gradient-covariance measurements from actual training to validate the assumptions of Theorems 2 and 3? These measurements should include the eigenvalue spectra before and after injection, the condition number on the support, the alignment between the task gradient and the dominant eigenvector, and whether the spectral gap satisfies the condition required for the approximately 36% bound.
5 comments

We thank the reviewer for the careful and constructive review, and for recognizing the importance of the problem, the integration of the framework, and the breadth of the evaluation. The comments helped us make the scope of the theory sharper and more explicit in the main text. In short:

  • (i) We rescope Theorem 1. We also show that its quadratic form is the generic local (second-order) model of the actual KS objective, and that the closed form extends to the true local curvature.
  • (ii) We bring every condition of Theorems 2–3 into the main-text statements and make two technical details more precise. A new lemma shows that the reward step never amplifies variance along any gradient direction (for any sign-symmetric law with Pr⁡(gC⊤g=0)=0\Pr(g_C^\top g=0)=0), with no alignment or Gaussianity assumption. We also report gradient-covariance measurements from training.
  • (iii) We state the ablation findings more precisely, and we add confidence intervals, seeds, a compute-matched baseline and a repeated-cycle study.

W1. The closed-form Knowledge Seeding solution applies to an additionally constructed quadratic surrogate objective, rather than the actual optimization objective…

Response: Thank you for this insightful comment; it lets us clarify the role of the theorem. The closed form is never used to compute the update. A,BA,B are trained by gradient-based optimization of the exact objective in Eq. (4) (Alg. 3). Theorem 1 analyzes how a rank-rr consolidation step allocates its capacity. We will revise the abstract, Sec. 4.1 and the conclusion to say that "a second-order model of the KS objective admits a closed-form truncated-SVD solution". Importantly, the quadratic form is not an arbitrary choice; it is the generic local model of the actual objective:

(1) The KL term is locally an exact quadratic, so the upper bound is not needed. The reviewer makes a good point that minimizing an upper bound does not, in general, give the KL minimizer. Lemma 9 (already in the paper) gives the exact local statement

DKL(PT∥PS)=12 δ^⊤Σ(PT) δ^+O(∥δ∥3),Σ(PT)=diag(PT)−PTPT⊤,D_{\rm KL}(P_T\Vert P_S)=\tfrac12\,\hat\delta^\top\Sigma(P_T)\,\hat\delta+O(\lVert\delta\rVert^3),\qquad \Sigma(P_T)=\mathrm{diag}(P_T)-P_TP_T^\top,

i.e., KL is locally equal to a quadratic in the softmax Fisher metric. KL and the bound 14∥δ^∥2\tfrac14\lVert\hat\delta\rVert^2 also have the same zero set (δ^=0\hat\delta=0), so they share their minimizers whenever the target is reachable. They differ only under the rank constraint, and that is exactly where the Fisher form is the right one. The isotropic metric in the paper is the further simplification Σ(PT)⪯12I\Sigma(P_T)\preceq\tfrac12 I (Lemma 10a). We will state the result with the Fisher metric (assuming 1∉col(Wu)\mathbf{1}\notin\mathrm{col}(W_u), e.g., centered unembedding, which is WLOG by softmax shift-invariance so that Q~=Wu⊤Σ(PT)Wu≻0\tilde Q=W_u^\top \Sigma(P_T)W_u\succ0) and present the isotropic form as a special case.

(2) The reward term is smooth in the logits, so it also has a quadratic local model. rsemr_{\rm sem} and rabsr_{\rm abs} are binary or thresholded per sample. Eq. (4), however, maximizes their expectation under the student, Ey^∼πM[r(y^;d)]\mathbb E_{\hat y\sim\pi_M}[r(\hat y;d)]. Conditioned on a prefix state hh (marginalizing future rollout steps into the token-level value rˉh(v)=E[r(y^;d)∣h,y^t=v]\bar r_h(v)=\mathbb E[r(\hat y;d)\mid h,\hat y_t=v], i.e., a per-position block-diagonal model), this expectation is ∑vPS(v∣h) rˉh(v)\sum_v P_S(v\mid h)\,\bar r_h(v), which is infinitely differentiable in the step logits, however coarse rr is.

Let Z⋆Z^\star be a local optimum of the combined per-position KS objective on the centered logit subspace 1⊥\mathbf{1}^\perp (since softmax is invariant to shifts along 1\mathbf{1}). It is finite on 1⊥\mathbf{1}^\perp for α>0\alpha>0, since KL regularization keeps P(Z⋆)P(Z^\star) in the interior of the probability simplex. Around Z⋆Z^\star, Taylor's theorem gives the objective as 12∥ZS−Z⋆∥Q2+O(∥ZS−Z⋆∥3)\tfrac12\lVert Z_S-Z^\star\rVert_Q^2+O(\lVert Z_S-Z^\star\rVert^3), with Q⪰0Q\succeq0 the true curvature (strictly positive definite on 1⊥\mathbf{1}^\perp: at Z⋆Z^\star, the KL curvature dominates any softmax non-convexity of the linear expected reward; e.g., for forward KL, stationarity yields η⊤Qη=α∑vPT,v(ηv−(P⋆)⊤η)2>0\eta^\top Q\eta=\alpha\sum_v P_{T,v}(\eta_v-(P^\star)^\top\eta)^2>0 on 1⊥\mathbf{1}^\perp, with P⋆P^\star tilted toward high-reward continuations).

Meanwhile, expected reward is invariant in probability space to moving mass among semantically equivalent continuations; modeling semantic/edit matching via a Gauss–Newton marginal discrepancy 12∥Φ⊤(PS−PT)∥2\tfrac12\lVert\Phi^\top(P_S-P_T)\rVert^2 (dropping residual second derivatives) yields the standalone PSD matrix Ωsem=Σ(PT)ΦΦ⊤Σ(PT)⪰0\Omega_{\rm sem}=\Sigma(P_T)\Phi\Phi^\top\Sigma(P_T)\succeq0 in A.1, which vanishes along probability-preserving logit shifts within a semantic equivalence class. A.1 additionally assumed that this curvature is the same in every context and centred at the teacher logits. Point (3) removes both assumptions.

(3) Theorem 5 does not depend on the form of Ωsem\Omega_{\rm sem}. Its proof (App. A.4.2) uses only three facts: HS≻0H_S\succ0; a PSD metric QQ, constant across contexts, with Q~=Wu⊤QWu≻0\tilde Q=W_u^\top QW_u\succ0 (guaranteed whenever WuW_u has full column rank and col(Wu)∩null(Q)={0}\mathrm{col}(W_u)\cap\mathrm{null}(Q)=\{0\}, e.g., 1∉col(Wu)\mathbf{1}\notin\mathrm{col}(W_u)); and an arbitrary target uu. Hence:

  • (a) The target can be the true local optimum, u=Z⋆−Φupper(MLPℓ(h))u=Z^\star-\Phi_{\rm upper}(\mathrm{MLP}_\ell(h)). This matters because an expected-match reward is mode-seeking, so Z⋆Z^\star need not equal ZTZ_T.
  • (b) The curvature may vary with the context (e.g., Σ(PT)\Sigma(P_T) differs across positions). The same closed form then holds with the context-averaged metric Qˉ=E[Q(h)]\bar Q=\mathbb E[Q(h)] in Qˉ~=Wu⊤QˉWu\tilde{\bar Q}=W_u^\top\bar QW_u (while retaining exact context-dependent Q(h)Q(h) inside the cross-moment C=Wu⊤E[Q(h)uhh⊤]C=W_u^\top\mathbb E[Q(h)u_h h^\top]), under the standard Kronecker-factored independence approximation (K-FAC; Martens & Grosse, 2015). It is exact when Q(h)Q(h) is independent of hh.

The two mechanisms we draw from Theorem 1 are properties of this structure, not of a particular Ωsem\Omega_{\rm sem}: HSH_S allocates rank according to the on-policy activation distribution, and Q~\tilde Q allocates rank to reward-sensitive output directions. We will also add the singular-value gap condition σr>σr+1\sigma_r>\sigma_{r+1}, which the stated uniqueness requires.

The reward-metric prediction is consistent with Table 1. Removing the imitation reward, so that Q~\tilde Q reduces to the isotropic distillation metric, costs 2.4/1.1/1.1 points. Removing only its semantic part costs 1.6 points on HMMT-25.

W2. The theoretical claims regarding conditioning improvements …

Response: Thank you for the careful reading of Appendix A.2. For space, the informal Theorems 2–3 summarize conditions that are stated in full in the appendix. To make this fully transparent, we will include every condition in the main-text statements and phrase the abstract and conclusion explicitly in terms of these conditions. Here is what each result needs:

ResultRequiresDoes not require
Thm 6 (= Thm 2): Σraw=Σ0+ε2Σinject+O(ε3)\Sigma_{\rm raw}=\Sigma_0+\varepsilon^2\Sigma_{\rm inject}+O(\varepsilon^3); confinement to S\mathcal SGauss–Newton, Er[Δhr]=0\mathbb E_r[\Delta h_r]=0, small ε\varepsilonGaussianity, alignment, spectral gap
Thm 6(iii): κ\kappa improves on S\mathcal Sthe above, plus Σinject∣S⪰0\Sigma_{\rm inject}\vert_{\mathcal S}\succeq0 and better conditioned than Σ0∣S\Sigma_0\vert_{\mathcal S}. The PSD between-expert term (active when sampling temperature T≠1T\neq1) alone is not enough, since Σwithin\Sigma_{\rm within} can be indefinite.Gaussianity, alignment
Thm 7(i–ii): filtering keeps the eigenbasis and amplifies all eigenvaluessign-symmetric, coordinate-independent law (Remark 12)Gaussianity
Thm 8 (= Thm 3)Thm 6(iii) conditions; Gaussian model; gCg_C aligned with the top eigenvector; κ(Σraw∣S)≥π/(π−2)\kappa(\Sigma_{\rm raw}\vert_{\mathcal S})\ge\pi/(\pi-2) (point 3)spectral gap (point 1)

Points 1 and 2 show that the analysis needs less than it may appear. Point 3 makes two technical details more precise.

  1. The spectral gap is needed only for the constant 0.36, not for the improvement. Eq. (31) gives κ+(Σdream)≤max⁡(1−2π,λ2λ1)⋅μ1+ε2δ1μs+ε2δs\kappa^+(\Sigma_{\rm dream})\le\max(1-\tfrac2\pi,\tfrac{\lambda_2}{\lambda_1})\cdot\tfrac{\mu_1+\varepsilon^2\delta_1}{\mu_s+\varepsilon^2\delta_s}+O(ε3)O(\varepsilon^3). The first factor is below 1 whenever the top eigenvalue is simple (so strict improvement over κ+(Σ0)\kappa^+(\Sigma_0) holds for sufficiently small ε>0\varepsilon>0). The condition λ2/λ1≤1−2/π\lambda_2/\lambda_1\le1-2/\pi only decides whether the factor reaches ≈0.36. The "≈36%" in our conclusion is therefore a best case, and we will say so.
  2. Without alignment, the reward step still never amplifies variance along any direction (new lemma). For any sign-symmetric gradient law with Pr⁡(gC⊤g=0)=0\Pr(g_C^\top g=0)=0 (e.g., any continuous sign-symmetric law), conditioning on the acceptance half-space {gC⊤g>0}\{g_C^\top g>0\} leaves the second moment unchanged and only shifts the mean. The accepted dreams therefore have covariance Σdream=Σ−mm⊤⪯Σ\Sigma_{\rm dream}=\Sigma-mm^\top\preceq\Sigma, a rank-one downdate. Norm filtering preserves sign symmetry and Pr⁡(gC⊤g=0)=0\Pr(g_C^\top g=0)=0, so this also holds exactly for the full filtered pipeline. Under the Gaussian model the downdate has the closed form Σdream=Σ1/2(I−2πww⊤)Σ1/2\Sigma_{\rm dream}=\Sigma^{1/2}(I-\tfrac2\pi ww^\top)\Sigma^{1/2}, with w∝Σ1/2gCw\propto\Sigma^{1/2}g_C, so (1−2π)Σ⪯Σdream⪯Σ(1-\tfrac2\pi)\Sigma\preceq\Sigma_{\rm dream}\preceq\Sigma for every gCg_C. Alignment only decides whether the single contraction falls on the top eigen-direction, which is what improves κ\kappa. We measure this alignment in Q3.
  3. Two technical refinements.
    • In Eq. (54), the post-downdate minimum eigenvalue is min⁡((1−2π)λ1,λs)\min((1-\tfrac2\pi)\lambda_1,\lambda_s), not always λs\lambda_s. The bound therefore holds when κ(Σraw∣S)≥π/(π−2)≈2.75\kappa(\Sigma_{\rm raw}\vert_{\mathcal S})\ge\pi/(\pi-2)\approx2.75. This is the ill-conditioned regime the theorem targets, and we will state the condition explicitly.
    • Eq. (33) applies the Gaussian half-space constant to the norm-filtered law, which is not Gaussian. We will label it as an approximation. Its qualitative conclusion holds exactly by point 2.

On the remaining assumptions. Each dream's gradient sums hundreds of bounded token-level terms (P−ey∈[−1,1]P-e_y\in[-1,1]). This makes the Gaussian model a reasonable approximation (Remark 11), and the "no amplification" conclusion of point 2 does not need it.

Σ0\Sigma_0 (Eq. 22) is a Gauss–Newton/Fisher matrix. For trained networks, such matrices consist of a few large outliers above a bulk, i.e., they are severely ill-conditioned (Sagun et al., 2017; Ghorbani et al., 2019; Papyan, 2020). Gradients also concentrate in the top eigenspace (Gur-Ari et al., 2018). These are the regimes that Thm 6(iii) and the alignment assumption describe. In Q3 we measure these quantities directly rather than rely on the literature.

W3. The experiments are broad, but the causal attribution and statistical reliability of the core mechanisms remain insufficient…

Response: We thank the reviewer for this precise reading, and we address its three parts in turn.

The ablation claim. Thank you for bringing this to our attention. We will phrase the summary of Table 1 more precisely: removing any single component lowers the mean over the three benchmarks (by 0.3–1.5 points), and 14 of the 15 individual entries decrease. The one exception is AIME-25 without the semantic reward (+0.2), which amounts to about one of the 480 graded samples (30 problems × 16). The semantic reward's effect is concentrated on the hardest benchmark, HMMT-25 (−1.6). We will also fix the pointer "Figure 1" in Sec. 5 to "Table 1".

Statistical reliability. We will report 95% bootstrap CIs (resampling problems and samples) for every row of Tables 1, 3 and 4, computed from the existing generations. We also rerun the headline comparison with 3 training seeds (Qwen3-8B): Sleep vs. its strongest baseline, OPSD + Expansion. The results are:

  • Sleep: 78.8 ± 1.0 / 68.7 ± 1.1 / 45.7 ± 1.3 (AIME-24 / AIME-25 / HMMT-25).
  • OPSD + Expansion: 77.7 ± 1.0 / 68.0 ± 1.1 / 45.5 ± 1.3.

The comparison with the external baseline was already consistent across settings: Sleep beats OPSD in all 6 model × benchmark cells of Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.