acceptodds
Under review as a conference paper at ICLR 2027

When Does Continual Learning Require Learning

Abstract

As large language models (LLMs) become increasingly capable, the next question is how to enable them to continually learn. Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes: space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. We introduce a common framework that treats continual learning methods as update operators acting on model state, allowing weight updates, prompt optimization, retrieval, and context compression to be studied together. We operationalize this framework through sequential evaluations built from existing benchmarks and public datasets, comparing supervised learning, reinforcement learning, prompt-based methods, context compression, and retrieval. Our experiments with Qwen3-8B reveal distinct requirements for adaptation across settings. Under domain shift, self-distillation achieves higher final average performance than prompt-based methods. For changing factual knowledge, retrieval that replaces outdated records outperforms the evaluated learning methods without updating model weights, but this advantage degrades when conflicting versions accumulate. Under financial temporal drift, SDPO improves economic performance over the frozen model, while accuracy alone obscures differences in useful adaptation. These findings make the distinction between storing new information and learning from it central to continual learning. More broadly, our framework connects the choice of update mechanism to the structure of how data changes. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.

Primary area:
datasets and benchmarks
Keywords:
continual learning, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QatyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper reformulates continual learning in LLMs as increasing model competence over time and proposes a unified evaluation framework covering domain shift, factual updates, temporal drift, and agentic accumulation. The authors compare existing adaptation methods, including supervised fine-tuning, reinforcement learning, prompt optimization, and context compression. The results highlight the strengths and limitations of different update mechanisms across these settings.

Critical strengths
  1. The paper covers diverse continual learning settings and evaluates a broad range of adaptation methods.
  2. The empirical comparisons provide an overview of the trade-offs among existing adaptation methods under different types of distributional changes.
Critical weaknesses
  1. Limited conceptual novelty. The proposed competence-based formulation and space/time taxonomy appear to overlap with prior work, including Continual Learning of Large Language Models: A Comprehensive Survey (Shi et al., ACM CSUR, 2025) and Towards Lifelong Learning of Large Language Models: A Survey (Zheng et al., ACM CSUR, 2025). These surveys already cover adaptation across time and domains, knowledge acquisition, and parametric/non-parametric learning. A clearer distinction from these established perspectives would help highlight the conceptual contribution of the proposed framework.
  2. Limited novelty of empirical findings. Several key findings appear consistent with known properties of existing methods, such as SDFT's ability to mitigate forgetting (Shenfeld et al., Self-Distillation Enables Continual Learning, 2026), the instability of RL under noisy rewards, and the stability–plasticity trade-off. While comparing these methods across different settings is useful, further analysis of why certain methods succeed or fail would strengthen the empirical contribution. For instance, the observation that SDFT performs well under domain shift but struggles with factual updates appears consistent with the known stability–plasticity trade-off. It would be helpful to clarify what additional insights emerge from these comparisons.
Questions for authors

Minor suggestions for improvement:

  1. Figure 1, Table 1, and the surrounding text are somewhat crowded and could be simplified to improve readability.
  2. The description in Lines 93–103 could be clarified, as the authors initially introduce two sub-regimes but subsequently refer to a third sub-regime.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Qaty

Thank you for the review and the presentation suggestions.

W1: Overlap with Shi et al. (2025) and Zheng et al. (2025). Both surveys discuss adaptation over time and across domains, as well as parametric and non-parametric learning, and we will cite and discuss them. The taxonomy is not our main contribution. The surveys categorize existing work (vertical vs. horizontal continuity in Shi et al., internal vs. external knowledge in Zheng et al.) but do not compare methods experimentally. Our framework is an evaluation protocol, with a common update operator U_k, a fixed per-stage budget, and a shared stage sequence, so that a prompt edit, a full-weight update, and a KV-cache adapter can be compared directly. We also split the time axis into slow noisy drift, discrete fact change, and agentic accumulation, each with its own metric (e.g., the stable-fact probe for catastrophic memorizing). The experiments show that these regimes favor different mechanisms, which the surveys do not establish.

W2: Findings consistent with known properties; why methods succeed or fail. Some results agree with prior work, which we take as a consistency check. Others do not follow from the individual papers (see the general response): prompt optimization forgets the most and does not transfer forward, compression is stable but learns little, the best weight-update method changes with the regime, and accuracy and PnL disagree on 10-K. The paper gives mechanism-level explanations for these results, which we will collect into a dedicated analysis subsection:

  • GEPA/ACE overwrite stable facts because the optimizer is rewarded only on drift facts in the validation set and edits playbook content that the stable facts also depend on (Sec. 4.2). Under noisy drift they add generic, time-invariant heuristics (Fig. 5).
  • SDFT/SDPO resist fact changes because at stage k the teacher is θ_{k-1}, so the loss pulls the model back toward its previous answer. This helps when new data should add to existing competence (domain shift, slow drift) and hurts when it should replace it.
  • GRPO learns changed facts but fails on 10-K because group-normalized advantages amplify any within-group reward difference. With clean binary rewards this gives sharp updates. With labels that are only about 50% predictable, the differences are mostly noise, and wrong answers get reinforced.

The stability-plasticity trade-off names the tension. Our results show where each mechanism falls on it in each regime, and that this position is not fixed for a given method.

Minor. We will simplify Fig. 1 and Table 1, and rewrite lines 93-103 so that all three time sub-regimes (slow drift, discrete fact change, agentic accumulation) are introduced up front.

We would appreciate it if the reviewer could tell us whether this addresses the novelty concern, and if not, which specific comparison or result would.

Reviewer_MWXvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates different adaptation methods for changes in tasks and knowledge over time. It proposes an evaluation framework and compares multiple methods. The experiments cover aspects such as domain shifts, factual updates, etc. The authors find that the relative performance of these methods depends on the type of change.

Critical strengths

The empirical study is useful. Understanding how LLMs adapt to changing tasks and knowledge is an important practical problem. Comparing different adaptation methods across several settings could provide a useful reference for researchers choosing these methods.

Critical weaknesses

Towards Continual Knowledge Learning of Language Models (ICLR 2022) already distinguished between retaining stable knowledge, updating outdated knowledge, and acquiring new knowledge. Self-Distillation Enables Continual Learning already demonstrated that SDFT could accumulate skills across sequential tasks while reducing forgetting compared with SFT. Although this paper adapts SDFT and provides broader comparisons, it is unclear what substantial new understanding these comparisons provide beyond the reported performance differences. I therefore find the contribution incremental.

The main-text and appendix results need to be discussed together. The main text suggests that SDFT damages existing knowledge, based on declining performance on stable facts. However, filtering the data and changing the scoring protocol in Appendix A.2.3 largely removes this decline. These results come from different experimental conditions, but the paper should explain how the appendix findings affect the main-text interpretation and which conclusions remain supported.

The authors use Qwen3-8B, released in 2025, with adaptation data from 2015-2020. Even if the model has not seen the specific 10-K filings, it may know subsequent company developments or economic events unavailable at the prediction date. Chronologically ordering the adaptation data does not exclude this information from the pretrained model. It is therefore unclear whether the experiment measures adaptation to genuinely unseen future conditions.

Questions for authors

What are the main new findings beyond those reported in Towards Continual Knowledge Learning of Language Models (ICLR 2022) and Self-Distillation Enables Continual Learning?

How do the filtered results in Appendix A.2.3 affect the main-text explanation of SDFT’s degradation?

How do you control for relevant information in Qwen3-8B’s pretraining that comes from after the prediction dates in the 10-K experiment?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MWXv

Thank you for the detailed review, including the appendix.

Q1: What is new relative to CKL and SDFT?

CKL introduced the retain/update/acquire split and evaluated continued-pretraining methods (regularization, rehearsal, parameter expansion) on T5 and GPT-2. Our TempWiki drift/stable probe follows the same idea, and we will credit CKL explicitly. CKL does not test whether prompt optimization, RL-based updates, or context compression can update facts. These families behave very differently in our results: GEPA/ACE get the largest drift gains but lose stable facts, compression barely moves either metric, and GRPO is the only weight-update method that improves drift F1 while holding stable F1 flat. We also compare the same methods under domain shift and noisy drift, where their ranking changes.

SDFT showed reduced forgetting relative to SFT in sequential skill learning, and our domain-shift results agree with this. What we add is the other side of the trade-off: SDFT does not learn changed facts (drift F1 stays flat in both TempWiki variants), whereas GRPO does. This cannot be seen from the SDFT paper alone, and it matters when choosing an update method for knowledge that changes over time.

Q2: How do the TempWiki-Easy results (App. A.2.3) affect the main text? The reviewer is right that the main text overstates SDFT's damage to stable facts. In the revision we will:

  • keep the conclusions supported by both variants: SFT and SDFT do not learn changed facts, while GRPO (and to a lesser degree SDPO) does, with stable F1 unchanged;
  • qualify the stable-F1 drop for SDFT/SDPO. TempWiki-Easy indicates that much of it comes from noisy Wikipedia diffs (formatting and annotation edits) and the thresholded string-level metric, so we will no longer present it as evidence that SDFT corrupts unchanged knowledge;
  • move Fig. 7 next to Fig. 3.

The SDFT conclusion becomes "stable but not plastic under discrete fact changes". The main claim of the paper, that the right update mechanism depends on the type of change, is unaffected.

Q3: Pretraining contamination in the 10-K experiment. We agree that chronological ordering controls the adaptation data but not the backbone's prior knowledge. We believe the effect on our conclusions is limited, for three reasons:

  1. All methods start from the same backbone and are compared to the same frozen per-year baseline (stars in Fig. 4). Prior knowledge of later events would raise the baseline and all methods equally, while our claims are about differences between methods and relative to that baseline (e.g., GRPO and GEPA lose forward accuracy while SDFT gains).
  2. Zero-shot accuracy is close to 50% for every year, so the backbone does not appear to exploit memorized outcomes.
  3. This concern is the reason TempWiki uses snapshots from after the Qwen3 cutoff (Nov 2025 to Feb 2026).

In the revision we will add a control with company names, tickers, and dates masked, and state the limitation in Sec. 4.3.

Please let us know if any of these points remain unclear.

Reviewer_mjTXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that the traditional definition of continual learning for LLMs, which focuses primarily on mitigating catastrophic forgetting, is incomplete. To address this, the authors propose a unified framework that defines continual learning as the ongoing enhancement of a model's capabilities as the world evolves across two dimensions, i.e., space (domain shifts) and time (temporal drift, factual updates, and the accumulation of agent states). The authors evaluate eight methods—spanning four categories (prompt optimization, supervised learning, reinforcement learning, and in-context compression), across a series of sequential benchmarks. The study finds that different types of changes require specific update strategies, and no single method performs optimally across all scenarios.

Critical strengths
  1. The paper introduces a novel, unified framework that categorizes the challenges of continual learning along spatial and temporal dimensions. This taxonomy clarifies the specific requirements for various continual learning scenarios.

  2. The authors have developed a robust evaluation framework covering a wide range of task types, including domain shifts (ToolUse, FinQA, SciKE-Bio), factual knowledge updates (TempWiki), temporal drift (10-K sentiment analysis), and sequential agent tasks (WebArena-Infinity). This comprehensive suite enables a nuanced comparison of different methods.

  3. The experimental results offer valuable insights into the strengths and weaknesses of various update mechanisms. For instance, the study shows that prompt-based methods allow for rapid adaptation but are prone to catastrophic forgetting, while distillation-based methods facilitate stable knowledge accumulation but struggle to incorporate new factual updates. Furthermore, the paper highlights the limitations of reinforcement learning (RL) approaches when faced with noise or weak reward signals.

Critical weaknesses
  1. The primary experiments are based on a single model family (Qwen3-8B). The authors also acknowledge this limitation, noting that the performance of different methods may vary when applied to larger-scale models or models operating in inference mode.

  2. Although the evaluation framework encompasses four specific types of change—domain shifts, factual updates, noisy temporal drift, and agent-based sequences—the environments encountered by large language models (LLMs) deployed in the real world are far more complex, open-ended, and unpredictable. These findings should be interpreted as evidence regarding the update behaviors required in specific contexts, rather than as a definitive ranking of continual learning (CL) algorithms.

  3. Evaluating "backward transfer" and "forgetting" requires extensive retrospective inference on historical tasks. This imposes a significant computational overhead and limits the framework's scalability when handling exceptionally long sequences.

Questions for authors

refer to weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer mjTX

Thank you for the review and the positive assessment.

W1: Single model family. We agree this is a limitation and will add a second backbone family for the domain-shift and TempWiki chains. The agentic experiment already covers two scales (Qwen3-8B with SFT, Qwen3-32B with ACE), and both improve over their zero-shot baselines at every chain length (Fig. 6). Our claims also concern the relative ordering of methods within each regime and how that ordering changes across regimes, rather than absolute accuracies.

W2: Real-world change is more open-ended. Agreed. We will revise the abstract and conclusion to say that the results characterize which update behaviors each regime requires and should not be read as a ranking of CL algorithms. Each of the four regimes isolates one requirement: learning a new task without overwriting old ones (domain shift), overwriting specific facts selectively (fact update), keeping only the persistent part of a noisy signal (temporal drift), and conditioning on self-generated state (agentic). We see deployment settings as mixtures of these.

W3: Cost of backward/forward evaluation. A full evaluation matrix has K x K cells. In our chains K is between 3 and 6, and each cell has 50 to 128 prompts, so evaluation takes at most a few thousand generations per method. This is small compared to training, which took under one 8xH100 node-hour per stage (App. A.1). For long sequences the full matrix is not needed. Evaluating the current stage, a fixed random subset of past stages, and the next stage brings the cost down to O(K) and still measures forgetting and forward transfer. We will add this discussion and a cost table to Sec. 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback. Reviewers found the coverage of settings and methods broad (mjTX, MWXv, Qaty) and the comparison useful as a reference (MWXv, Qaty). The main shared concern is novelty relative to CKL (Jang et al., 2022), SDFT (Shenfeld et al., 2026), and recent surveys (Shi et al., 2025; Zheng et al., 2025). We clarify the contributions here and respond to each reviewer below.

Contributions. The surveys organize the literature, and CKL and SDFT each study one class of weight-update method. We are not aware of prior work that runs prompt optimization (GEPA, ACE), supervised updates (SFT, SDFT), RL (GRPO, SDPO), and context compression (Cartridges, In-place TTT) under one protocol, with the same backbone, stage sequence, and per-stage budget (Sec. 3, Table 1). Several of the resulting findings cannot be inferred from the individual method papers:

  1. Prompt optimization forgets the most under domain shift. GEPA gains close to 40 points during the FinQA stage and then drops back to roughly zero-shot after the Bio stage (Fig. 2). It also shows no forward transfer on 10-K (Fig. 4).
  2. Context compression is the most stable family but learns very little in any setting (Figs. 2-4).
  3. The best weight-update method depends on the regime. SDFT is strongest under domain shift and noisy drift but does not learn changed facts. GRPO is the only weight-update method that learns changed facts while keeping stable F1 flat (Figs. 3 and 7), yet it is the weakest on 10-K (Fig. 4).
  4. On 10-K, accuracy and PnL disagree. A simple section-filtering rule yields positive PnL, while the GEPA-optimized prompt reduces forward PnL (Fig. 8).

Planned revisions.

  • Sec. 4.2: rewrite the TempWiki discussion around conclusions that hold on both the original split and TempWiki-Easy, and move Fig. 7 into the main text (MWXv).
  • Related work: discuss Shi et al. (2025), Zheng et al. (2025), and CKL. We will also correct the sentence in App. A.2.2 stating that earlier temporal-knowledge benchmarks do not probe unchanged facts; CKL and TemporalWiki both do.
  • Sec. 4.3: discuss pretraining contamination in the 10-K setting and add an anonymized-filing control (MWXv).
  • Add a second backbone family for the domain-shift and TempWiki chains (mjTX).
  • Add an evaluation cost analysis (mjTX), simplify Fig. 1 and Table 1, and fix the sub-regime description in Sec. 1 (Qaty).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.