acceptodds
Under review as a conference paper at ICLR 2027

Protecting Continual Reasoning with Depth-Informed Trust Regions

Abstract

Continual reinforcement learning with verifiable rewards requires reasoning language models to acquire new capabilities without erasing behaviors learned from earlier tasks. This setting is especially challenging when replay is disallowed: after a task ends, the learner must preserve useful prior behavior using only compact statistics rather than old prompts or trajectories. We introduce **DToP** (Depth-informed Trust Region over Parameters), a replay-free continual adaptation method that summarizes past reasoning behavior through complementary functional sensitivities and constrains future optimization within the resulting anisotropic parameter-space geometry. At each task boundary, DToP first identifies *late-settling* token states, whose intermediate-layer next-token distributions approach the final-layer prediction only at relatively deep layers. For these selected states, DToP measures the parameter sensitivity of the local decision margin between the sampled token and its strongest alternative, and compresses the resulting gradients into a low-rank second-moment memory. On the remaining candidate states, DToP estimates diagonal policy Fisher information to capture broader distributional sensitivity. The two memories are scale-normalized and combined into a joint trust-region metric. During subsequent tasks, DToP does not constrain optimizer steps independently. Instead, each proposed update is projected so that the *cumulative* parameter displacement from the previous task endpoint remains inside the boundary-centered trust region. This explicitly controls the accumulation of many individually small updates along historically sensitive directions. We evaluate DToP on a heterogeneous continual reasoning benchmark spanning multiple reasoning domains. The results provide encouraging evidence that DToP can better balance the acquisition of new capabilities with the retention of previously learned ones, while retaining only compact parameter statistics and requiring no inference-time modification.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, continual reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2xtQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DToP aims to reduce forgetting during continual reinforcement learning with verifiable rewards, without replaying old data. After each task, it selects token positions whose predictions settle late across the model's layers. It stores a low-rank summary of their decision-margin gradients and a diagonal policy-Fisher estimate from other positions. These memories define a trust region around the adapter parameters at the end of the task. During the next task, the method adjusts optimizer proposals to keep the total displacement inside that region. On six algorithmic tasks, Qwen3-8B achieves better final accuracy and retention than replay-free baselines in two task orders. A ten-task Qwen3-4B experiment adds a longer sequence, with one seed. The theory guarantees the quadratic parameter constraint and its connection to retained behavior depends on local approximations and the stated assumptions.

Critical strengths
  1. The method protects sensitive combinations of parameters without storing old trajectories. Its low-rank margin memory complements the diagonal Fisher estimate, and the constraint covers total movement within a task. This addresses the drift that can accumulate over many individually small steps. Appendix A.7 states what the stored-sensitivity bounds establish and where further assumptions are needed to connect them to policy behavior or task accuracy.
  2. The reverse-order results give a useful comparison at almost equal acquisition: DToP scores 70.6% and EWC 70.7%, while BWT improves from −9.9 to −1.4 points and final accuracy from 62.5% to 69.4% (Table 1). Both task orders use three seeds. The paper also reports component ablations, selection comparisons, and compute diagnostics.
Critical weaknesses
  1. The code and manuscript use different margin gradients. Equation 3 and Appendix A.2 explicitly specify raw gradients without a per-state normalizer. In the supplement, vjp.py calls relative_margin_gradient, and trust.py divides each gradient by max(margin, 1.0)\texttt{max(margin,\ 1.0)}. These rows enter the low-rank memory through base.py. The normalization changes the relative weight of states before compression, so the later shared calibration scalar does not generally cancel it. The authors need to identify which version produced the tables before the results can be attributed to the method analyzed in the paper.
  2. The experiments do not fully isolate which design choices produce the gains. Relative to EWC, DToP changes the sensitivity estimator, state selection, calibration, and constraint enforcement together (Appendix B.1). Applying a penalty or per-step constraint to the same stored metric would clarify the benefit of cumulative projection. Fisher-only is already close in canonical order, and the strength-matched control uses one seed (Appendix B.3). Table 2 supports depth-based allocation overall, but changing the selected margin states also changes the complementary Fisher pool. Compared with random selection, depth selection improves final accuracy and acquisition while giving more negative backward transfer. This leaves its specific contribution to retention unclear. The paper reports no sensitivity study of the number of selected states or memory rank (Table 5). Since consolidation draws on a small final-batch sample, its coverage of earlier task behavior also needs testing. Small evaluation sets and the one-seed ten-task extension further limit conclusions about robustness.
Questions for authors
  1. Which margin-gradient definition produced Tables 1-3? Please reconcile the normalizer in the supplied boundary code with Equation 3 and Appendix A.2, identify the code version used for the reported results, and explain whether any results or sensitivity bounds change.
  2. With the same sensitivity estimates and comparable tuning access, how does cumulative projection compare with a quadratic penalty or per-step projection? How was the EWC penalty tuned, and does the matched-strength control hold across the reported seeds?
  3. Can you isolate the effect of depth-based margin selection by holding the Fisher reference sample fixed, and compare it with selection by small margin? How do acquisition and retention change with the number of selected states, sampled responses, and memory rank? These tests would help explain whether depth is the useful signal and how well the small contextualized sample covers a task.
Reviewer_X7kwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DToP, a replay-free continual reinforcement learning method designed to mitigate catastrophic forgetting in reasoning language models. The method selects late-settling token states based on intermediate-layer predictions and constructs two types of historical sensitivity memory: a low-rank decision-margin gradient memory and a diagonal policy-Fisher memory. These are combined to define an anisotropic trust region that constrains cumulative parameter displacement during subsequent tasks.

The authors evaluate DToP on six-task and ten-task algorithmic reasoning benchmarks using Qwen3 models. The reported results show improvements in final accuracy and backward transfer compared with several continual learning baselines. Component and token-selection ablations are also provided, although the experimental analysis is limited in isolating the contributions of individual design choices.

Critical strengths
  1. Intuitively motivated framework. The combination of historical parameter sensitivity and cumulative trust-region constraints provides a reasonable approach to mitigating catastrophic forgetting without retaining previous training data.

  2. Extensive theoretical analysis. The paper provides detailed mathematical formulations and theoretical analysis of the proposed framework, including the cumulative trust-region projection and bounds on parameter sensitivity. These analyses help establish the mathematical properties of the method.

  3. Consistent improvements on the evaluated benchmarks. DToP achieves higher final accuracy than the replay-free baselines across both task orders in the six-task and ten-task settings. The results provide some empirical support for the overall approach.

Critical weaknesses
  1. Insufficient ablation studies to justify the method's complexity.

    The method introduces multiple components and hyperparameters, but the experiments do not sufficiently isolate their individual contributions. In particular, the paper lacks comparisons with stronger token-selection baselines, systematic sensitivity analyses for important hyperparameters, and a direct comparison between cumulative and per-step trust-region projection under the same sensitivity metric. As a result, it remains unclear which components are necessary and whether the added complexity is justified.

  2. Limited experimental validation and robustness analysis.

    DToP constructs its historical sensitivity memory from at most eight correct responses in the final training batch, without investigating whether such a small sample can adequately capture the sensitivity of an entire task. Furthermore, the evaluation uses only 64 held-out examples per task in the main experiments, which may be insufficient to reliably assess the reported performance gains, particularly the relatively small improvements over some baselines. Given that the tasks are procedurally generated, a larger evaluation set would provide more convincing evidence. Additionally, much of the theoretical analysis establishes properties of parameter constraints and local sensitivity bounds rather than directly guaranteeing the preservation of previously learned reasoning capabilities. The connection between these theoretical results and actual task-level retention remains insufficiently validated.

Questions for authors
  1. Token selection. Why is depth-informed selection only compared with random and lowest-entropy selection? Could the authors include highest-entropy and smallest-logit-margin baselines to better establish the benefit of settling depth?

  2. Settling depth. How reliable is settling depth as an indicator of reasoning importance? Since the layer-wise JSD is not necessarily monotonic, why is the first threshold crossing used rather than a more stable convergence criterion?

  3. Memory construction. How are the eight verifier-correct responses selected from the final training batch? Can multiple responses come from the same prompt? More importantly, are eight responses sufficient to capture the sensitivity of an entire task? How do the results change with larger sampling budgets and different memory ranks?

  4. Sensitivity measures. Why are decision-margin gradients used for late-settling states and diagonal Fisher information for the remaining states? Have the authors compared alternative sensitivity measures on the same selected states to isolate the benefit of this design?

  5. Cumulative trust region. Does cumulative projection outperform conventional per-step trust-region projection when using the same historical sensitivity matrix and comparable constraint strength? This comparison seems important for establishing the contribution of cumulative displacement control.

  6. Initial performance. Could the authors report the accuracy of the base models before continual RLVR training? Without initial accuracy, it is difficult to assess how much new capability is actually acquired during training.

  7. Replay-free setting. Given that the benchmark tasks are procedurally generated and historical prompts can potentially be regenerated at low cost, what practical scenarios motivate the strict replay-free constraint?

Reviewer_Ef2rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DToP for continual RLVR without replay of historical examples. It combines two sensitivity summaries estimated at task boundaries with a cumulative trust region centered at the preceding task endpoint. A low-rank matrix of margin gradients protects token states whose intermediate-layer predictions approach the final prediction relatively late, while a diagonal policy Fisher protects the remaining candidate states. Updates on the next task are projected to constrain cumulative displacement. Experiments compare two task orders on a six-task Qwen3-8B stream and a ten-task Qwen3-4B stream from Algorithmic-10, with EWC, TreeLoRA, the replay method CPR, and a multi-task reference. The geometric projection has a clear conditional derivation, and the supplementary package provides substantial implementation material.

Critical strengths
  1. DToP constrains the cumulative displacement of successive updates instead of only individual steps. This is an appealing and readily stated design relative to Fisher regularization and per-step constraints.

  2. The main experiments cover canonical and reverse orders, several baselines, and component and state-selection controls. The paper reports single-GPU overhead, and the supplementary package includes a frozen benchmark, run configurations, pinned model revisions, and locked dependencies, providing a useful starting point for verification.

Critical weaknesses

1.The supplementary implementation appears to divide each margin gradient by max, although the paper explicitly specifies no per-state normalization. This changes the weight of each state in the second moment and weakens the stated interpretation in terms of absolute logit-gap changes.

2.The evidence is limited in scale: the six-task experiments use three training seeds, while the ten-task experiments use one seed. Together with the lack of reported systematic tuning for some baseline hyperparameters, these results support preliminary gains under the evaluated protocol, but do not establish robust algorithmic superiority or generalization to broader reasoning tasks.

3.Tuning is incompletely documented. The paper fixes the EWC coefficient at “λ = 1” and adapts TreeLoRA through a gradient-norm constraint, without reporting searches over these strengths. Sensitivity to key DToP parameters is also missing, so the ranking under comparable tuning budgets remains unverified.

Questions for authors
  1. Are the tables generated by the exact code version in the supplement? How do you reconcile the per-state margin scaling with Appendix A.2, and how does it affect the main results?

  2. Can you provide multiple-seed ten-task results and paired uncertainty for method differences across training seeds and evaluation items?

  3. How did you choose each method's parameters? Does the ranking persist under comparable tuning budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.