acceptodds
Under review as a conference paper at ICLR 2027

Toward Self-Referential Continual Learning via a CREB-Inspired Selective Consolidation Mechanism

Abstract

Continuous parameter updates in streaming Large Language Models (LLMs) can exhibit representational “smearing,” where sequential learning across disjoint formal domains degrades previously learned reasoning structure. We introduce Self-Referential Continual Learning (SRCL), a dual-memory architecture that gates long-term structural consolidation through a discrete, CREB-inspired epistemic threshold. Instead of continuous uncertainty-weighted replay, SRCL processes streaming inputs through a transient Working Memory (WM) adapter and computes a consistency score over successive WM adapter updates, S(x_t) = E[cos(Δφ_t, Δφ_t-1)]; in experiments, this quantity is estimated online with an EMA-based cosine-similarity approximation. When this score exceeds a discrete threshold τ, the WM adapter update is projected into Long-Term Memory (LTM) via orthogonal rank expansion; otherwise, the transient WM weights decay. Principal Component Analysis of parameter trajectories over 50 sequential steps suggests that SRCL more often confines updates to distinct, approximately orthogonal subspaces, reducing the geometric overlap and high-variance interference observed under continuous Prioritized Experience Replay (PER). On the reported sequential math and code streams, SRCL preserves formal reasoning performance more effectively than continuous baselines, with cumulative interference plateauing at 0.244 after 50 steps while the baselines continue to degrade. While this discrete gating mechanism mitigates catastrophic forgetting in our deterministic reasoning tasks, performance degrades sharply under high injected aleatoric noise, suggesting that discrete epistemic thresholds are comparatively effective for the tested formal domain shifts but brittle under irreducible data stochasticity.

Primary area:
reinforcement learning
Keywords:
continual learning, large language models, catastrophic forgetting, memory consolidation, selective consolidation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZRaEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes SRCL, a continual learning method that decouples short-term adaptation from long-term consolidation. The approach incorporates a LoRA module—acting as working memory—onto a frozen backbone model, alongside a long-term memory component that expands incrementally. Gradient consistency is used to determine the timing of consolidation, while projection operations—which steer new information away from previously consolidated input subspaces—dictate the directions in which this information is stored. Experimental results demonstrate that this method yields significant improvements in both sequential learning accuracy and knowledge retention.

Critical strengths
  1. SRCL separates when to consolidate knowledge from where to store it. Table 3 supports the importance of both components: with six consolidation events and the same rank budget, the full method achieves 68.4% math accuracy, compared with 60.3% for periodic consolidation and 61.7% without projection. These controls provide evidence that the gains reflect consolidation timing and storage geometry, beyond simply allocating additional memory.
  2. Across the tested baselines, SRCL consistently improves over prioritized replay on both math and code and performs comparably to InfLoRA. Achieving this performance without boundary annotations or stored training examples supports the practical value of the proposed consolidation mechanism within the evaluated settings.
Critical weaknesses
  1. Unclear treatment of working memory in the main evaluations. Section 3.1 states that “At evaluation the WM is cleared,” whereas Section 4.5 states that “Every method is evaluated with its full state at the end of each domain—for SRCL, Θ + Φ + ϕ.” These descriptions appear inconsistent and leave uncertainty about the memory state used to obtain the main CRP and BWT results. Full-state evaluation is a reasonable measure of overall continual-learning performance.
  2. Limited evidence for sustained learning under a fixed budget. The selected math configuration uses six of the 32 available consolidation events. Although the paper studies budget exhaustion in ablations and contamination experiments, it remains unclear how the recommended configuration balances new-rule acquisition and old-rule retention on longer, clean streams approaching capacity. The interference bound does not establish continued learning, since the LTM stops accepting updates once the budget is exhausted.
Questions for authors
  1. Could the authors reconcile Section 3.1 (“At evaluation the WM is cleared”) with Section 4.5 (“Every method is evaluated with its full state at the end of each domain—for SRCL, Θ + Φ + ϕ”)? Was WM cleared or retained when computing the main CRP and BWT results, particularly those in Table 1? If WM was retained, could the authors provide paired evaluations of Θ + Φ + ϕ and Θ + Φ at the same checkpoints, including per-domain accuracy and the corresponding BWT? The comparison should leave the training trajectory unchanged and remove WM only for the additional evaluation, so that it isolates the contribution of consolidated memory.
  2. Under the fixed memory budget and recommended configuration, how does performance evolve on longer, clean streams that introduce additional rules or domains? Could the authors characterize this progression by reporting cumulative consolidation events and rank usage, accuracy on newly encountered domains, and retention of those domains after subsequent learning? Of particular interest is when acquisition or retention begins to deteriorate as the budget is approached and eventually exhausted. This would help establish how much continuing novelty the method can accommodate and the practical range over which it maintains both acquisition and retention.
  3. The experiments do not include CLS-ER or SYNERgy, which also use multiple memories and learn without task boundaries. Although these methods require storing past examples, comparison with at least one would help assess whether SRCL’s advantage over PER extends to replay methods with a more similar memory design. Could the authors compare SRCL with CLS-ER or SYNERgy, or explain why these methods are unsuitable for this setting? Reporting accuracy alongside memory and computing costs would help assess the trade-offs fairly.
Reviewer_DtEDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Referential Continual Learning (SRCL), which separates two decisions in continual learning: when an update should become permanent and where it should be stored. A transient working-memory (WM) adapter is updated on each batch. A self-referential gate consolidates the WM into long-term memory (LTM) when the current gradient is consistent with recent gradient history. Each consolidation is projected onto the orthogonal complement of the subspace spanned by previous commits. The paper provides an interference bound and derives a critical contamination rate for coherent corruption. Experiments on counterfactual Lean theorem proving and code generation, across several model sizes and architectures, report improved accuracy and less negative backward transfer than prioritized replay and other baselines. Ablations examine the gate, projection, threshold, and contamination behavior.

Critical strengths
  1. The paper presents a clear conceptual separation between when to consolidate and where to store an update. Table 3 provides a useful ablation indicating that the consistency-based gate improves the selection of updates to retain, while the projection reduces interference from later consolidations.

  2. The evaluation covers Lean theorem proving and code-generation streams across several model architectures and scales, using mechanically checked or hidden-test-based probes. The contamination sweep is also valuable because it characterizes a concrete failure mode of the self-referential gate, rather than presenting SRCL as universally robust.

Critical weaknesses
  1. Appendix C.2 uses a single rank-16 adapter for SFT and PER, whereas SRCL has a rank-16 working memory in addition to accumulated long-term memory. For example, the 8B math setting in Table 4 ends with 48 LTM ranks. Thus, SRCL’s advantage over these baselines may come partly from its consolidation/projection mechanism and partly from a larger effective adapter capacity. The current experiments do not fully separate these effects. A comparison under a matched total adapter parameter budget would help isolate the contribution of the proposed mechanism. This issue does not necessarily invalidate the method, but it affects how the source of the gains should be interpreted.

  2. Section 3.1 states that the WM is cleared at evaluation, but Section 4.5 says that the main experiments use the full state Θ+Φ+ϕ\Theta+\Phi+\phi, and Appendix C.3 notes that the final domain may still depend on WM. Therefore, the main tables can reflect final task performance and resistance to forgetting under the full state, but they do not by themselves establish persistent retention in pure LTM. Proposition 1 bounds interference from subsequent LTM updates for a fixed input, but does not by itself establish persistent retention when the reported performance may also depend on WM. The paper should state the actual evaluation state used for the main tables and report per-domain accuracy after clearing WM.

Questions for authors
  1. Do the main-table and BWT results use the full state, or are they obtained after clearing WM?
  2. If WM is cleared, what are the per-domain accuracies and BWT?
  3. After matching total adapter parameters or effective rank, how much of SRCL’s advantage over PER and SFT remains?
Reviewer_MaB7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LLMs can learn new rules while minimizing interference with previously acquired knowledge in continual learning settings without explicit task boundaries. The authors propose Self-Referential Continual Learning (SRCL), which decomposes memory consolidation into two decisions: (1) WHEN to write updates into long-term memory, and (2) WHERE to store these updates. Specifically, the method employs a working-memory (WM) LoRA adapter with decay to continuously learn from incoming data. A hard gate is implemented based on the cosine similarity between the current gradient and an exponential moving average of historical gradients. When consolidation is triggered, updates are written into long-term memory (LTM) through orthogonal projection and truncated SVD. The theoretical analysis establishes an upper bound on LTM consolidation interference under a finite rank budget, which is independent of the stream length, and examines the conditions under which the gating mechanism fails in the presence of coherent data corruption. Experiments introduce counterfactual rules in Lean 4 mathematical theorem proving and code generation across Python, C++, and Rust, evaluating counterfactual reasoning probe (CRP) accuracy, backward transfer (BWT), and interference across multiple model scales and architectures. Compared with Continuous PER, SRCL significantly improves accuracy and reduces forgetting, achieving performance comparable to InfLoRA, which requires task boundaries, without relying on task boundaries or replaying historical samples.

Critical strengths
  1. The paper provides a clear problem formulation by separating update selection and parameter interference in continual learning into two questions, WHEN and WHERE, establishing a relatively unified and intuitive methodological framework. The gradient-consistency gate and orthogonal subspace constraints address these two questions, respectively, while the theoretical analysis also explains how the latter protects historical inputs. This clear division of responsibilities between the two mechanisms is a meaningful contribution, particularly because the method does not require prior knowledge of task boundaries.

  2. The experimental design is well targeted to the research problem. The authors construct counterfactual rules that conflict with pretrained priors, requiring models to genuinely adapt to new rules rather than relying solely on pretrained knowledge to complete the evaluations. The Lean kernel and code execution tests provide objective correctness verification. In addition, the paper compares against multiple continual learning baselines and provides evidence for the effectiveness of the proposed mechanisms through WHEN and WHERE ablation studies. These design choices strengthen the credibility of the experimental conclusions.

Critical weaknesses
  1. The current experiments are insufficient to fully establish the necessity and practical benefits of SRCL relative to replay-based methods. Although the paper highlights being data-free as an advantage, this does not mean that replay-based approaches lack practical competitiveness. The existing Continuous PER baseline uses a fixed buffer of 2,048 samples, a 1:1 replay ratio, and a priority strategy based on gradient consistency. Prior work, such as Gradient Based Sample Selection for Online Continual Learning (GSS, NeurIPS 2019), has already investigated more effective replay sample selection without task boundaries. Meanwhile, SRCL requires tuning multiple mechanism-specific hyperparameters and has additional long-term memory capacity, whereas the PER baseline uses a fixed rank-16 adapter. Therefore, the currently observed large performance gap cannot fully rule out the effects of replay strategy and parameter capacity configurations. Given the widespread practical use of replay, I suggest that the authors at least consider a more representative task-free replay method, or provide a more thorough discussion of the trade-offs between optimized replay methods and SRCL in terms of accuracy, computational overhead, storage requirements, and hyperparameter tuning complexity, without necessarily conducting an extensive hyperparameter search.

  2. There is an inconsistency in the long-term memory evaluation protocol, which affects the assessment of the core mechanism. Section 3.1 states that WM is cleared during evaluation, leaving LTM as the only retained memory. However, Section 4.5 explicitly states that evaluation after each domain uses the full state Θ+Φ+ϕ\Theta+\Phi+\phi, and Appendix C.3 also notes that knowledge from the last domain remains in WM. Since the final CRP accuracy includes the last domain, while BWT excludes the last domain from the forgetting calculation, it is currently difficult to determine how much of the reported performance comes from permanently consolidated LTM and how much comes from WM that has not yet decayed. This issue is directly related to the paper's central claim of selective long-term memory consolidation. The authors should clarify the actual evaluation protocol and report LTM-only performance after clearing WM.

Questions for authors
  1. Why can gradient consistency serve as a reliable criterion for long-term memory consolidation? High cosine similarity directly reflects the alignment between the current update direction and recent historical gradients, but does not necessarily indicate the importance or long-term retention value of the knowledge. Although Table 5 already provides evidence of a correlation between gradient consistency and the recurrence of new rules, I would appreciate further clarification on why this signal is preferable to other consolidation criteria and how the method handles important new rules that occur infrequently.

  2. Table 3 separately evaluates the effects of gating and projection, but the combination of "periodic consolidation + no projection" is missing. Adding this setting would provide a more complete WHEN × WHERE ablation study and help determine whether the benefits of the two mechanisms are independent of each other.

  3. The necessity of hard gating and WM decay requires further explanation. Table 4 demonstrates sensitivity to the gating threshold, and Table 10 evaluates different decay coefficients, but neither includes a no-decay baseline with λ=1\lambda=1. In particular, when λ\lambda increases from 0.95 to 0.99, the decay actually becomes weaker, yet accuracy decreases. Is this phenomenon caused by the mixing of historical updates, changes in consolidation frequency, or other mechanisms? Could the authors clarify the independent contribution of decay and explain why hard gating is necessary compared with continuous or soft gating?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.