- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DToP aims to reduce forgetting during continual reinforcement learning with verifiable rewards, without replaying old data. After each task, it selects token positions whose predictions settle late across the model's layers. It stores a low-rank summary of their decision-margin gradients and a diagonal policy-Fisher estimate from other positions. These memories define a trust region around the adapter parameters at the end of the task. During the next task, the method adjusts optimizer proposals to keep the total displacement inside that region. On six algorithmic tasks, Qwen3-8B achieves better final accuracy and retention than replay-free baselines in two task orders. A ten-task Qwen3-4B experiment adds a longer sequence, with one seed. The theory guarantees the quadratic parameter constraint and its connection to retained behavior depends on local approximations and the stated assumptions.
Critical strengths
- The method protects sensitive combinations of parameters without storing old trajectories. Its low-rank margin memory complements the diagonal Fisher estimate, and the constraint covers total movement within a task. This addresses the drift that can accumulate over many individually small steps. Appendix A.7 states what the stored-sensitivity bounds establish and where further assumptions are needed to connect them to policy behavior or task accuracy.
- The reverse-order results give a useful comparison at almost equal acquisition: DToP scores 70.6% and EWC 70.7%, while BWT improves from −9.9 to −1.4 points and final accuracy from 62.5% to 69.4% (Table 1). Both task orders use three seeds. The paper also reports component ablations, selection comparisons, and compute diagnostics.
Critical weaknesses
- The code and manuscript use different margin gradients. Equation 3 and Appendix A.2 explicitly specify raw gradients without a per-state normalizer. In the supplement, vjp.py calls relative_margin_gradient, and trust.py divides each gradient by . These rows enter the low-rank memory through base.py. The normalization changes the relative weight of states before compression, so the later shared calibration scalar does not generally cancel it. The authors need to identify which version produced the tables before the results can be attributed to the method analyzed in the paper.
- The experiments do not fully isolate which design choices produce the gains. Relative to EWC, DToP changes the sensitivity estimator, state selection, calibration, and constraint enforcement together (Appendix B.1). Applying a penalty or per-step constraint to the same stored metric would clarify the benefit of cumulative projection. Fisher-only is already close in canonical order, and the strength-matched control uses one seed (Appendix B.3). Table 2 supports depth-based allocation overall, but changing the selected margin states also changes the complementary Fisher pool. Compared with random selection, depth selection improves final accuracy and acquisition while giving more negative backward transfer. This leaves its specific contribution to retention unclear. The paper reports no sensitivity study of the number of selected states or memory rank (Table 5). Since consolidation draws on a small final-batch sample, its coverage of earlier task behavior also needs testing. Small evaluation sets and the one-seed ten-task extension further limit conclusions about robustness.
Questions for authors
- Which margin-gradient definition produced Tables 1-3? Please reconcile the normalizer in the supplied boundary code with Equation 3 and Appendix A.2, identify the code version used for the reported results, and explain whether any results or sensitivity bounds change.
- With the same sensitivity estimates and comparable tuning access, how does cumulative projection compare with a quadratic penalty or per-step projection? How was the EWC penalty tuned, and does the matched-strength control hold across the reported seeds?
- Can you isolate the effect of depth-based margin selection by holding the Fisher reference sample fixed, and compare it with selection by small margin? How do acquisition and retention change with the number of selected states, sampled responses, and memory rank? These tests would help explain whether depth is the useful signal and how well the small contextualized sample covers a task.