- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates where forgetting-related damage occurs within a model during RL post-training, particularly when multiple domains are trained together. In particular, the authors find that performance degradation occurs even though the gradients of different domains are nearly orthogonal, that the parameters changed by RL are relatively sparse, and that although the neurons modified by domain differ significantly, the activation paths used in actual inference overlap. Furthermore, they provided the theoretical validity of this hypothesis and restored performance through experiments using short refresh and conflict subspace proxy for training-free rollback.
Critical strengths
- The issue of forgetting in RL is important and timely. The paper provides insights through a structural analysis of this issue. The theoretical analysis also provides conditional support for the hypothesis.
- The experimental evaluation is comprehensive, providing rich analyses. It goes beyond the hypothesis and demonstrates it through actual experiments.
- The rationale, reasons, and design for each experiment are clear, making them relatively easy to follow. The training-free rollback experiment is particularly compelling.
Critical weaknesses
- Table 1 shows clear motivation, but one question remains regarding the effect of Re-math's short refresh. For example, RL sequential training can be order-sensitive, and if Math is performed last, the forgetting issue may not actually occur or may be minimal.
- In Figure 3 (b), Math-Code also exhibits substantial overlap. This suggests that such overlap may arise much more simply due to domain-specific characteristics (e.g., the relationship between science and math) than the authors claim. It is unclear whether such metrics fully explain why performance degradation occurs much more severely in the Math domain compared to other domains.
- In Sections 4.4 and 4.5, the authors measure related but different things. It seems natural to simply use the overlapping regions of activated and edited neurons. Additionally, the analysis in each section uses models trained in the single domain, which may produce results and observations different from those initially claimed.
Questions for authors
- In the short refresh experiments, does the effect come from a specific parameter selection? If the authors' hypothesis is correct, are the short refresh updates highly relevant to the identified parameters such as selected rollback coordinates?
- If forgetting occurs in multiple domains simultaneously, can the proposed method still be applied?
9 comments
Author's Comment [2]
- In Figure 3 (b), Math-Code also exhibits substantial overlap. This suggests that such overlap may arise much more simply due to domain-specific characteristics (e.g., the relationship between science and math) than the authors claim. It is unclear whether such metrics fully explain why performance degradation occurs much more severely in the Math domain compared to other domains.
(1) Domain relatedness and what we formalize. We agree that domains such as math and science are intrinsically related. Relatedness, however, is an abstract property of the data. We map it onto the parameters RL actually changes: which units each domain uses (Fig. 3b), which units it edits (Fig. 3a), and in which direction it edits the units both domains share (Fig. 4). We then write the earlier domain's loss change exactly in terms of the later displacement (Section 5). Relatedness alone does not determine the outcome. Training QA from the base model raises Math from 43.19 to 55.31 (, Table 2). Training QA from lowers Math from 66.49 to 59.90. The domains and their data are the same in both runs, and the effect has opposite sign; what differs is the parameter state. Reasoning traces show the same dependence on training. On the same four domains, CGPO converges toward one structured, verification-heavy style. JT polarizes, with Math more compact and symbolic and QA more exploratory. Sequential training keeps the base style but shifts toward the most recent stage.
(2) A small subspace can carry most of the effect. RL updates are sparse yet nearly full-rank (Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. Mukherjee et al., 2025), and the rank-1 subspace of the update recovers over 99% of single-domain reasoning gains (On Predictability of Reinforcement Learning Dynamics for Large Language Models. Cai et al., 2025). A later domain can therefore affect an earlier one through a small shared subspace. We test this by intervention. Reverting 9.8% of the QA displacement recovers 57.4% of the QA-induced Math drop, and QA changes by only −0.13. At a 1.3% budget, the selected coordinates recover 20.4%, while an equal-budget random rollback gives −6.3% (Table 3).
(3) What the metrics explain (a fuller discussion is in our response to Reviewer 13uU, Q3). Our mechanistic claim is that damage requires the later update to enter directions to which the earlier domain is sensitive. The rollback above tests this claim directly. Overlap, direction, and curvature are signals that locate those directions. We do not claim that any single one of them explains the damage.
- Overlap is symmetric, but damage is not. In QA→Math, Math training leaves QA nearly unchanged (63.31→63.00). In Math→QA, QA training lowers Math by 6.59 points (Appendix D.3). Removing activation from the selector costs little: M×C recovers 18.3%, against 20.4% for A×M×C.
- Direction separates the two high-overlap pairs. Code and Math edit shared units in aligned directions, and Code→Math transfer is positive. Math and QA edit them in opposite directions in layers L3–6. Removing the conflict term halves recovery (A×M 9.6%).
- Curvature confirms that the located direction is sensitive for Math. Math's curvature density along it is 4.5–7.4× that of equal-budget random directions (Table 4).
Math is damaged most because QA's update falls on units that Math relies on, in conflicting directions along which Math's loss is highly curved. The overlap is only the precondition for this.
Author's Comment [3]
- In Sections 4.4 and 4.5, the authors measure related but different things. It seems natural to simply use the overlapping regions of activated and edited neurons. Additionally, the analysis in each section uses models trained in the single domain, which may produce results and observations different from those initially claimed.
We agree with both points and ran the analyses the reviewer suggests. Everything else is the same as in the paper: one neuron is one MLP intermediate channel, each set is the top 10% of neurons per layer, and every metric is averaged over the 36 layers.
(1) Restricting the analysis to neurons that are both edited and active. Take two domains and . In each layer, let be the top-10% changed neurons of domain (Sec. 4.3) and its top-10% active neurons (Sec. 4.4). We define the overlap region
and the overlap rate . is the fraction of co-edited neurons that lie on routes both domains use. If where a domain edits had nothing to do with which neurons are active, would equal .
| Pair | Overlap rate$R$ | Chance |
|---|---|---|
| Math–Code | 0.458 | 0.064 |
| Math–QA | 0.427 | 0.067 |
| Code–QA | 0.393 | 0.061 |
| Math–CW | 0.365 | 0.042 |
| Code–CW | 0.361 | 0.042 |
| QA–CW | 0.307 | 0.045 |
The edited and active sets are not identical. is 0.31–0.46, so more than half of the co-edited neurons lie outside the shared active routes. The two sets still share a substantial intersection, 6–9× what chance predicts. This matches the claim in the paper that sparse edits reach another domain through the routes both domains rely on at inference time.
We then recompute the Sec. 4.5 directional cosine on only. The pattern is the same as in Sec. 4.5:
- Code–Math is predominantly aligned, with the average cosine staying positive across depth.
- Math–QA is negative in layers L3–6 and positive in layers L14–21.
- The CW pairs stay near zero.
The synergistic pairs are more aligned on than on the neuron set used in Sec. 4.5. Restricting to therefore separates synergy from conflict more sharply.
(2) Repeating Secs. 4.3–4.4 and the overlap rate on the sequential RL checkpoints. We use the Math–QA pair, which shows the largest sequential damage in our task-level results. The edits are the stage-wise differences and , and all metrics are computed exactly as in (1).
| Metric | Single-domain experts | Sequential |
|---|---|---|
| Changed-neuron Jaccard (Sec. 4.3) | 0.134 | 0.169 |
| Active-neuron Jaccard (Sec. 4.4) | 0.515 | 0.521 |
| Overlap rate$R$ (chance in brackets) | 0.427 [0.067] | 0.474 [0.068] |
The findings carry over to sequential training:
- Edits remain sparse and overlap weakly.
- Active routes remain strongly shared.
- Co-edits concentrate on shared routes slightly more than they do for independent experts.
(3) Do the experts and the sequential run identify the same neurons?
| Set (expert vs. sequential), mean Jaccard | Math | QA |
|---|---|---|
| Active neurons | 0.921 | 0.924 |
| Changed neurons | 0.269 | 0.168 |
| Math–QA overlap region$O$ | 0.299 |
Random sets of the same size give a Jaccard of 0.053.
- The active routes are nearly identical (). The domain's data decides them, not the checkpoint that processes the data.
- The changed sets differ more, but they are still 3–5× above chance.
- 51% of the experts' overlap neurons reappear in the sequential overlap region. Agreement rises in deeper layers, up to 0.62 at layer 28.
The single-domain analysis therefore finds the same routes that sequential training uses.
Revisions
- Sec. 4.4 adds the overlap rate between edited and active neurons, with its heatmap.
- Sec. 4.5 computes the directional cosine on the overlap region .
- A new appendix repeats the analyses on the sequential checkpoints and adds the expert-vs-sequential comparison.
Author's Comment [1]
We thank the reviewer for the specific and actionable suggestions. We ran the analyses the reviewer proposed:
We agree that sequential RL is order-sensitive. This is one of our findings, and the paper already tests a reversed ordering (Appendix D.3). In QA→Math, later Math training leaves QA nearly unchanged (63.31→63.00). In Math→QA, later QA training lowers Math from 66.49 to 59.90. Cross-domain interference is therefore directional rather than symmetric, and this asymmetry is consistent with our local perturbation analysis (Proposition 1): the damage to an earlier domain depends on whether the later update enters that domain's curvature-sensitive shared directions. Math updates perturb QA-sensitive directions only weakly, whereas QA updates perturb Math-sensitive directions more strongly. Placing Math last would therefore largely avoid its degradation, for the same reason our analysis gives.
In practice, however, which domains interfere with which is not known in advance, either when several domains are first trained together or when a model is updated iteratively with new domains. Sequential training followed by evaluation gives two things at once: a checkpoint with expert-level performance on the current domain, and a direct diagnosis of which earlier domain was damaged. A short refresh on that domain (Re-Math, 57.66→66.04 with limited changes elsewhere) then repairs the damage. There is no need to first map the pairwise interference between domains and retrain the whole chain in a different order. Order sensitivity is therefore what makes targeted refresh useful, not an alternative explanation for its effect.