acceptodds
Under review as a conference paper at ICLR 2027

Self-Consolidating Language Models

17 views

Abstract

Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context workflows. While longer context windows expose more evidence, they do not ensure that useful information is preserved and reused. We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information. We propose elf-nsolidating anguage Models (SCoL), a post-training framework in which, given current context, an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated. Because committed updates change the model that later generates future selections, we train SCoL with meta-reinforcement learning over an evolving model state. We instantiate SCoL with supervised QA rewards on SQuAD knowledge incorporation and intrinsic likelihood-based rewards for LongBench v2 long-context consolidation. On SQuAD, SCoL improves acquisition over the evaluated baselines and retention over sequential finetuning. On LongBench v2, we meta-train SCoL only on short contexts. It outperforms the evaluated baselines on short contexts and generalizes to longer contexts while remaining competitive with in-context methods. Analysis of learned selection patterns shows that SCoL encourages the LLM to generate sparse update locations that align with layers of high Fisher information, suggesting that the model learns to route plasticity toward loss-sensitive regions while limiting interference.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, test time training, language models, meta-learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ak9hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCoL, a post-training framework that trains LLMs via meta-RL to generate textual instructions selecting which Transformer layers to update, thereby consolidating long context streams into parameters. A composite reward balancing new-knowledge acquisition and forgetting is used to trade off plasticity and retention in continual data streams.

Critical strengths
  1. Enabling LLMs to self-regulate where to update via natural-language generation is an interesting and novel form of dynamic plasticity routing.
  2. Empirical analysis shows SCoL spontaneously produces highly sparse layer selections that overlap strongly with high-Fisher-information layers, demonstrating that preference optimization can internalize parameter importance.
Critical weaknesses
  1. SEAL baseline reports only 1.30% retention, yet Appendix results show its open-book QA acc. collapses to 0.0%. This looks more like training collapse than memory forgetting. The authors should explain the cause.
  2. To compute the forgetting penalty over historical streams, SCoL must traverse all past contexts at every step, yielding O(T2)O(T^2) complexity. The cost is prohibitive.
  3. Appendix ablations show that a hard-coded Last-10 layers control, with no inner/outer RL, reaches 20.10% retention, versus 20.46% for SCoL's meta-learned routing. A 0.36% gain indicates a severe mismatch between the cost (see W2) and benefit of the Meta-RL architecture. Why not use this stronger, simpler method as a main baseline? Also, does "Last-10" include the decoding layer? Prior work suggests skipping the decoding layer and updating middle-to-late layers may work better. If tuned, could Last-10 exceed SCoL?
  4. Evaluation data is extremely limited. In LongBench v2, only 60 passages are sampled, with just 20 testing long-context generalization. Why not use the full 503-item benchmark? Such small test sets are highly susceptible to sampling variance, undermining robustness and confidence.
  5. The paper claims continual knowledge injection, yet meta-training and evaluation are confined to the same distribution and even the same dataset (e.g., SQuAD streams). Real continual learning may involve OOD tasks. Without cross-domain injection tests, the method looks like overfitting to a single dataset distribution rather than a general continual learning paradigm.
  6. Only SQuAD and LongBench v2 are tested, ignoring degradation of base capabilities after continual updates. Without evaluation on general domains, it is hard to show the model retains its original generalization and reasoning robustness.
Questions for authors

None

Reviewer_yPpdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCoL, which learns to select Transformer layers for LoRA updates when incorporating new context. Preference training balances acquisition and forgetting. Experiments on SQuAD and LongBench v2 show improvements over sequential fine-tuning.

Critical strengths

Letting the model select its own update locations is an interesting idea. The fixed-layer controls show encouraging acquisition gains, and the appendix provides useful diagnostics on instruction following, forgetting, and selector collapse.

Critical weaknesses
  1. The contribution of learned selection needs clearer isolation. Compared with Last-10, SCoL improves immediate accuracy by 6.62 points, but its retention advantage is only 0.36 ± 1.57 points (Table 13, mean ± SE). Final retention also remains below the prompting-only reference. Meanwhile, Table 11 shows that all-layer sequential updates severely damage QA instruction following, so the large improvement over SEAL cannot be interpreted solely as better knowledge retention. Random-10 and Fisher-top-10 controls, using the same initialization, would help separate adaptive selection from the benefit of restricting updates.

  2. The evaluation does not adequately separate model selection from testing. Appendix B.11 uses the headline 100-passage validation stream to compare algorithms and hyperparameters. Granularity selection also uses this stream. An independent test stream and multiple passage orders are needed. The differing one-round results in Tables 8 and 10 should also be explained, including any differences in seeds or configuration.

  3. The long-context results provide limited evidence of generalization. Evaluation includes only 20 longer documents. SCoL scores 37.0%, compared with 36.0% for summarization and 41.5% for Batch TTT. Table 15 reports uncertainty from repeated answer sampling, which does not establish robustness across documents or training runs. A fixed-layer sequential baseline and paired document-level uncertainty estimates would make this comparison more informative.

Questions for authors
Reviewer_4TgMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCoL which enables LLMs to select suitable Transformer layers for continual knowledge consolidation through dynamic LoRA updates. Experiments on squad show the effectiveness of the method.

Critical strengths
  1. The paper addresses an important problem of continual knowledge acquisition and forgetting in LLMs.
  2. The proposed layer-selection mechanism is simple and interpretable.
  3. The experiments demonstrate improvements over sequential fine-tuning, with additional analysis of Fisher alignment.
Critical weaknesses
  1. Limited novelty. The method is closely related to SEAL. The differences from PTE (a) and TMEM (b) could also be better discussed.
  2. Missing baselines. Random and Fisher-based layer selection are not compared. The Last-10 baseline also achieves nearly the same retention as SCoL (20.10% vs. 20.46%).
  3. Limited retention. SCoL retains only 20.46% accuracy after 100 updates, raising questions about its long-term memory effectiveness.
  4. Limited experiments. The paper only evaluates two datasets using a single 7B model. The mode collapse in later training rounds is also concerning.
  5. Unclear role of meta-RL. It is not clear whether the drifting-policy formulation is necessary, since the training mainly relies on local preference comparisons.

References

(a) Zhang et al., Training Large Reasoning Models Efficiently via Progressive Thought Encoding. ICLR 2026.

(b) Ren et al., Scaling Self-Evolving Agents via Parametric Memory. arXiv:2606.04536, 2026.

Questions for authors

See weakness And

  1. How does SCoL compare with random or Fisher-based layer selection under the same update budget?
  2. Can the authors evaluate more datasets and model sizes, and further investigate the mode collapse in later training rounds?
Reviewer_EbEdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an algorithm for continual learning in large language models. The technique involves training the language model using a meta-RL approach to propose (in a textual format) specific locations in its weights that could can be updated (via low-rank updates) to hold newly ingested information. This involves a bilevel optimization loop that utilizes a reward that encourages internalizing the new context and punishes forgetting on previous contexts. The model learns a policy (that reportedly persists robustly across future updates) that is able to propose sparse updates to its own weights in a manner that ameliorates catastrophic forgetting. The authors find that the model learns a policy that proposes update locations that align somewhat with those of high Fisher information, reminiscent of classical knowledge updation methods.

Critical strengths
  1. The paper’s methodology is well-motivated and appears sound. The problem formulation is general enough (demonstrated in part by its application to both QA and long-context style settings) to be of wide interest. The paper does a good job of seperating the general framework and problem setting from the more specific choices they make for the SCoL algorithm and their empirical settings.
  2. The paper is very well-written and easy to follow despite its technical nature. I was able to understand most sections in just one pass.
  3. The paper conducts informative analyses over its results (Eg. Figure 3, Table 2, etc.)
  4. The work appears reproducible with many implementation details provided in the appendices.
Critical weaknesses
  1. The paper only experiments with one language model: Qwen2.5-7B-Instruct. It would be good to know if the SCoL method generalizes to other language models and have some intuition about how model size (e.g #layers) and other hyperparameters (eg. BB) affects SCoL.
  2. The paper discusses a wide range of continual learning, long-context, and RL-based self-adaptation prior work in section 2. However, it only compares variants of SCoL to Prompting, SEAL, and batch TTT in its experiments. Comparing to a few more recent works from continual learning and/or long context literature would help clarify SCoL’s performance in the landscape of today’s methods.
Questions for authors
  1. Line 194: What is the space the argmin is operating over? If it is the universal set of any possible weight update, wouldn’t you always be able to pick weight updates that push the perplexity of ctc_t arbitrarily low? I might be misunderstanding how equation 5 connects to lines 191-193. I understand that you are enforcing constraints on the sparsity of Δθt(a)\Delta \theta_t(a) through BB, but is a constraint on its magnitude also enforced? How does the ADAPT procedure work?

  2. Did you conduct an investigation showing how performance of SCoL and other baselines degrades at extremely long contexts? I see Table 3, but I’m just curious what the shape of the degradation looks like as a curve. Because I think continual learning of this form becomes most interesting when we’re dealing with contexts so long they don’t fit in the context window.

  3. If you found that the model learns to perform weight updates in layers that correlate somewhat (at a rate of 45% when random chance would be 36%) with layers with high Fisher information, is there any scope for a method that doesn’t source the locations of weight updates from the model itself, but runs mechanical Fisher information compuations and chooses layers in a without involving the language model? If this doesn’t work as well, could you show results on comparing SCoL to such a technique?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC (1/2)

We thank all reviewers:

  • They found the problem important (4TgM),
  • the methodology well motivated and sound and the formulation general (EbEd),
  • letting the model select its own update locations an interesting idea (yPpd),
  • a novel form of dynamic plasticity routing (ak9h),
  • the layer selection simple and interpretable (4TgM), and,
  • the analyses of the learned selections informative (EbEd, ak9h).

Our research question is whether an LLM can itself determine where in its own weights to write each incoming context at test time. The model is updated at every context, so we frame the LLM at test time as a drifting policy rather than a static model, and train it with meta-reinforcement learning to generate which of its own layers to update, with forgetting penalized in the reward.

The reviewers' main questions are:

  1. whether the gain comes from learning where to update, since SCoL's retention in the submitted version was 20.46% against 20.10% for Last-10, which always updates the last 10 layers (4TgM, yPpd, ak9h),
  2. whether SCoL holds on other model families, model sizes and update budgets (EbEd, 4TgM), and
  3. whether SCoL generalizes beyond the distribution it is meta-trained on, and whether the LongBench v2 evaluation is large enough to show it (ak9h, yPpd, EbEd).

During the rebuttal, we added the following experiments to the revised version. The new results further strengthen each claim of the paper. The tables are in our responses to each reviewer and in the revised PDF.

1. SCoL alone forms the Pareto frontier between immediate accuracy and retention (Section 4.2, Tables 1, 5 and 17, Figure 3)\textcolor{blue}{\text{(Section 4.2, Tables 1, 5 and 17, Figure 3)}}. For a fair comparison, the submitted version used SEAL's inner adaptation hyperparameters, the LoRA update applied at each context, for every method. We tuned this inner adaptation hyperparameters on a held-out set outside the evaluation stream and found a setting that is twice as fast and better for every method. We also added Random-10, which updates 10 random layers. On the 100-passage SQuAD stream, SCoL outperforms every baseline on both metrics, immediate accuracy right after each update and retention after the whole stream. Against Last-10, SCoL reaches 40.66% vs. 35.07% in immediate accuracy and 31.00% vs. 26.98% in retention, a paired difference of +5.59 ± 1.60 and +4.02 ± 1.54 points (mean ± SE), and its lead over every other baseline is also larger than the standard error. With the tuned hyperparameters, SEAL's retention rises from 1.30% to 21.36% and is still far below SCoL's 31.00%. Random-10 updates as many layers as SCoL and stays below it on both metrics, so the gain comes from learning where to update. Random selection falls further behind when one document is chunked and consolidated over many updates. On LongBench v2, each document is consolidated chunk by chunk and the question is asked only at the end, and Random-10 is 10.75 ± 4.47 (short) and 15.02 ± 5.18 (long) points below SCoL in paired difference.

2. SCoL holds on another model family and on a 27B model (Appendix B.9, Tables 8 to 10)\textcolor{blue}{\text{(Appendix B.9, Tables 8 to 10)}}. We ran SQuAD on Llama-3.1-8B-Instruct (32 layers) and Qwen3.8-27B (64 layers), against 28 layers for Qwen2.5-7B. SCoL outperforms every baseline on both metrics on both models, so SCoL alone forms the Pareto frontier on all three models. On Qwen3.8-27B, we also meta-trained SCoL at budgets of B=5B=5, 10 and 20 layers, and it outperforms updating the last BB layers or BB random layers on both metrics at every budget. Llama-3.1-8B-Instruct is meta-trained for 4 rounds without mode collapse, and it still emits 93 distinct selections over the 100 passages at round 4.

3. SCoL generalizes beyond what it is meta-trained on (Section 5.2, Appendices B.9 and C.3, Tables 3, 11, 12 and 17)\textcolor{blue}{\text{(Section 5.2, Appendices B.9 and C.3, Tables 3, 11, 12 and 17)}}.

  • We took the SQuAD-trained Qwen2.5-7B checkpoints and asks it to consolidated 100 New York Times paragraphs from SQuADShifts. SCoL again outperforms every baseline on both metrics.
  • After the 100-passage SQuAD stream, SCoL keeps 5-shot MMLU at 73.56%, against 74.19% before any update, while every baseline falls to between 70.61% and 71.75%.
  • On LongBench v2, SCoL is meta-trained only on short documents (16k to 32k tokens) and is also tested on long documents (32k to 64k tokens), beyond the 32,768-token context window of Qwen2.5-7B. We now report 3 seed runs, add Last-10, and give paired document-level uncertainty estimates. SCoL outperforms every baseline on short documents, and on long documents it outperforms every sequential and in-context method. SCoL makes 469 sequential updates over the 40 short documents and 476 sequential updates over the 20 long documents, so each of the two experiments tests SCoL on about 5x as many updates as the 100-passage SQuAD stream.
AuthorsOpenReview ↗

General response (2/2): changes in the revised PDF

In the revised PDF, new and rewritten content is in blue. The table lists each change, where it is in the revised version, and the reviewer weaknesses (W)/questions (Q) it answers.

ChangeIn the revised versionResponse to reviewer
Inner adaptation tuned on held-out passages outside the evaluation stream, and used by every method. The SEAL baseline no longer collapses.Section 4.1, Appendix B.3, Table 54TgM W2, yPpd W1 and W2, ak9h W1 and W3
Last-10, Middle-10 and Random-10 as baselines under the same 10-layer budget.Sections 4.1 and 4.2, Table 1, Appendix B.84TgM W2, yPpd W1, ak9h W3, EbEd W2
SQuAD results over 3 seed runs, and the plot of immediate accuracy against retention with the Pareto frontierSection 4.2, Table 1, Figure 34TgM W2 and W3, yPpd W1, ak9h W1 and W3
Fisher alignment of the learned selections for SCoL and SCoL$_{\lambda=0}$ (42.9% and 37.6%, against 35.7% for a random selection)Section 4.2, Table 2EbEd Q3, 4TgM W2, yPpd W1
LongBench v2 over 3 seed runs, with Last-10 and the number of sequential updates on short and long documentsSections 5.1 and 5.2, Table 3yPpd W3, ak9h W4, EbEd W2 and Q2
Llama-3.1-8B-Instruct and Qwen3.8-27B on SQuADAppendix B.9, Table 8EbEd W1, 4TgM W4
Llama-3.1-8B-Instruct over 4 rounds of meta-training, with the number of distinct selections after each roundAppendix B.9, Table 94TgM W4, yPpd W2
Update budgets B=5B=5, 10 and 20 on Qwen3.8-27BAppendix B.9, Table 10EbEd W1
Out-of-distribution stream of New York Times paragraphs from SQuADShiftsAppendix B.9, Table 11ak9h W5, yPpd W2, 4TgM W4
5-shot MMLU before any update and after the SQuAD streamAppendix B.9, Table 12ak9h W6
QA replay with the swept inner adaptationAppendix B.9, Table 13yPpd W1
Outer-loop algorithm and granularity comparisons over 3 seed runsAppendices B.11 and B.12, Tables 14 and 15yPpd W2
Paired differences between SCoL and each method, on SQuAD and on LongBench v2 over documentsAppendix C.3, Table 17yPpd W1 and W3, ak9h W3 and W4, 4TgM W2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.