acceptodds
Desk rejected at ICLR 2027

Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) post-training improves large language models on individual domains but can degrade previously acquired capabilities. We study this interference in mathematical reasoning, code generation, question answering (QA), and creative writing (CW). Full-model domain gradients can be nearly orthogonal despite substantial selective degradation. Our structural analyses show sparse, weakly overlapping edits on computation routes that reasoning domains share, with directional alignment associated with transfer and conflict. A local perturbation model separates loss changes into a first-order term and a path-averaged curvature term, decomposes the latter along the actual rollback intervention, and provides a conditional explanation for recovery by short refresh. The experiments support this local sensitivity account. After Code Math QA CW training, a short Re-Math refresh raises Math from 57.66 to 66.04 and achieves the highest average score among the evaluated baselines. A training-free rollback at a budget recovers of the QA-induced Math accuracy drop with a small QA change. Curvature probes at the Math checkpoint further show higher sensitivity along the selected direction than along matched random directions. Together, these results support sparse intervention and targeted refresh as effective ways to mitigate cross-domain degradation.

Primary area:
reinforcement learning
Keywords:
multi-domain RL, cross-domain interference, model analysis, interpretability, continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BPsWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates where forgetting-related damage occurs within a model during RL post-training, particularly when multiple domains are trained together. In particular, the authors find that performance degradation occurs even though the gradients of different domains are nearly orthogonal, that the parameters changed by RL are relatively sparse, and that although the neurons modified by domain differ significantly, the activation paths used in actual inference overlap. Furthermore, they provided the theoretical validity of this hypothesis and restored performance through experiments using short refresh and conflict subspace proxy for training-free rollback.

Critical strengths
  • The issue of forgetting in RL is important and timely. The paper provides insights through a structural analysis of this issue. The theoretical analysis also provides conditional support for the hypothesis.
  • The experimental evaluation is comprehensive, providing rich analyses. It goes beyond the hypothesis and demonstrates it through actual experiments.
  • The rationale, reasons, and design for each experiment are clear, making them relatively easy to follow. The training-free rollback experiment is particularly compelling.
Critical weaknesses
  • Table 1 shows clear motivation, but one question remains regarding the effect of Re-math's short refresh. For example, RL sequential training can be order-sensitive, and if Math is performed last, the forgetting issue may not actually occur or may be minimal.
  • In Figure 3 (b), Math-Code also exhibits substantial overlap. This suggests that such overlap may arise much more simply due to domain-specific characteristics (e.g., the relationship between science and math) than the authors claim. It is unclear whether such metrics fully explain why performance degradation occurs much more severely in the Math domain compared to other domains.
  • In Sections 4.4 and 4.5, the authors measure related but different things. It seems natural to simply use the overlapping regions of activated and edited neurons. Additionally, the analysis in each section uses models trained in the single domain, which may produce results and observations different from those initially claimed.
Questions for authors
  • In the short refresh experiments, does the effect come from a specific parameter selection? If the authors' hypothesis is correct, are the short refresh updates highly relevant to the identified parameters such as selected rollback coordinates?
  • If forgetting occurs in multiple domains simultaneously, can the proposed method still be applied?
9 comments
AuthorsOpenReview ↗

Author's Comment [1]

We thank the reviewer for the specific and actionable suggestions. We ran the analyses the reviewer proposed:

  • Overlap region (Q3). On neurons that are both edited and active, 31–46% of co-edited neurons lie on shared routes (6–9× chance), and the Sec. 4.5 direction pattern holds. The same metrics on the sequential checkpoints reproduce the single-domain findings.
  • Where the refresh updates (Q4). The unconstrained refresh puts its largest updates on the identified units: 85.4% of the Math–QA shared edited-and-active neurons (8.5× chance), and 4.0× chance on the rollback coordinates.
  • Multi-domain repair (Q5). Sequential Math→Code refresh recovers 92.2% / 40.0% of the two drops.
  • Order (Q1). The paper's reversed order (Appendix D.3) shows the interference is directional, which is why targeted refresh is useful.
  1. Table 1 shows clear motivation, but one question remains regarding the effect of Re-math's short refresh. For example, RL sequential training can be order-sensitive, and if Math is performed last, the forgetting issue may not actually occur or may be minimal.

We agree that sequential RL is order-sensitive. This is one of our findings, and the paper already tests a reversed ordering (Appendix D.3). In QA→Math, later Math training leaves QA nearly unchanged (63.31→63.00). In Math→QA, later QA training lowers Math from 66.49 to 59.90. Cross-domain interference is therefore directional rather than symmetric, and this asymmetry is consistent with our local perturbation analysis (Proposition 1): the damage to an earlier domain depends on whether the later update enters that domain's curvature-sensitive shared directions. Math updates perturb QA-sensitive directions only weakly, whereas QA updates perturb Math-sensitive directions more strongly. Placing Math last would therefore largely avoid its degradation, for the same reason our analysis gives.

In practice, however, which domains interfere with which is not known in advance, either when several domains are first trained together or when a model is updated iteratively with new domains. Sequential training followed by evaluation gives two things at once: a checkpoint with expert-level performance on the current domain, and a direct diagnosis of which earlier domain was damaged. A short refresh on that domain (Re-Math, 57.66→66.04 with limited changes elsewhere) then repairs the damage. There is no need to first map the pairwise interference between domains and retrain the whole chain in a different order. Order sensitivity is therefore what makes targeted refresh useful, not an alternative explanation for its effect.

AuthorsOpenReview ↗

Author's Comment [2]

  1. In Figure 3 (b), Math-Code also exhibits substantial overlap. This suggests that such overlap may arise much more simply due to domain-specific characteristics (e.g., the relationship between science and math) than the authors claim. It is unclear whether such metrics fully explain why performance degradation occurs much more severely in the Math domain compared to other domains.

(1) Domain relatedness and what we formalize. We agree that domains such as math and science are intrinsically related. Relatedness, however, is an abstract property of the data. We map it onto the parameters RL actually changes: which units each domain uses (Fig. 3b), which units it edits (Fig. 3a), and in which direction it edits the units both domains share (Fig. 4). We then write the earlier domain's loss change exactly in terms of the later displacement (Section 5). Relatedness alone does not determine the outcome. Training QA from the base model raises Math from 43.19 to 55.31 (QAs\mathrm{QA}_s, Table 2). Training QA from Matho\mathrm{Math}_o lowers Math from 66.49 to 59.90. The domains and their data are the same in both runs, and the effect has opposite sign; what differs is the parameter state. Reasoning traces show the same dependence on training. On the same four domains, CGPO converges toward one structured, verification-heavy style. JT polarizes, with Math more compact and symbolic and QA more exploratory. Sequential training keeps the base style but shifts toward the most recent stage.

(2) A small subspace can carry most of the effect. RL updates are sparse yet nearly full-rank (Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. Mukherjee et al., 2025), and the rank-1 subspace of the update recovers over 99% of single-domain reasoning gains (On Predictability of Reinforcement Learning Dynamics for Large Language Models. Cai et al., 2025). A later domain can therefore affect an earlier one through a small shared subspace. We test this by intervention. Reverting 9.8% of the QA displacement recovers 57.4% of the QA-induced Math drop, and QA changes by only −0.13. At a 1.3% budget, the selected coordinates recover 20.4%, while an equal-budget random rollback gives −6.3% (Table 3).

(3) What the metrics explain (a fuller discussion is in our response to Reviewer 13uU, Q3). Our mechanistic claim is that damage requires the later update to enter directions to which the earlier domain is sensitive. The rollback above tests this claim directly. Overlap, direction, and curvature are signals that locate those directions. We do not claim that any single one of them explains the damage.

  • Overlap is symmetric, but damage is not. In QA→Math, Math training leaves QA nearly unchanged (63.31→63.00). In Math→QA, QA training lowers Math by 6.59 points (Appendix D.3). Removing activation from the selector costs little: M×C recovers 18.3%, against 20.4% for A×M×C.
  • Direction separates the two high-overlap pairs. Code and Math edit shared units in aligned directions, and Code→Math transfer is positive. Math and QA edit them in opposite directions in layers L3–6. Removing the conflict term halves recovery (A×M 9.6%).
  • Curvature confirms that the located direction is sensitive for Math. Math's curvature density along it is 4.5–7.4× that of equal-budget random directions (Table 4).

Math is damaged most because QA's update falls on units that Math relies on, in conflicting directions along which Math's loss is highly curved. The overlap is only the precondition for this.

AuthorsOpenReview ↗

Author's Comment [3]

  1. In Sections 4.4 and 4.5, the authors measure related but different things. It seems natural to simply use the overlapping regions of activated and edited neurons. Additionally, the analysis in each section uses models trained in the single domain, which may produce results and observations different from those initially claimed.

We agree with both points and ran the analyses the reviewer suggests. Everything else is the same as in the paper: one neuron is one MLP intermediate channel, each set is the top 10% of neurons per layer, and every metric is averaged over the 36 layers.

(1) Restricting the analysis to neurons that are both edited and active. Take two domains AA and BB. In each layer, let SdS_d be the top-10% changed neurons of domain dd (Sec. 4.3) and AdA_d its top-10% active neurons (Sec. 4.4). We define the overlap region

OA,B=SA∩SB∩AA∩AB,O_{A,B}=S_A\cap S_B\cap A_A\cap A_B,

and the overlap rate RA,B=∣OA,B∣/∣SA∩SB∣R_{A,B}=|O_{A,B}|/|S_A\cap S_B|. RR is the fraction of co-edited neurons that lie on routes both domains use. If where a domain edits had nothing to do with which neurons are active, RR would equal ∣AA∩AB∣/9728|A_A\cap A_B|/9728.

PairOverlap rate$R$Chance
Math–Code0.4580.064
Math–QA0.4270.067
Code–QA0.3930.061
Math–CW0.3650.042
Code–CW0.3610.042
QA–CW0.3070.045

The edited and active sets are not identical. RR is 0.31–0.46, so more than half of the co-edited neurons lie outside the shared active routes. The two sets still share a substantial intersection, 6–9× what chance predicts. This matches the claim in the paper that sparse edits reach another domain through the routes both domains rely on at inference time.

We then recompute the Sec. 4.5 directional cosine on OO only. The pattern is the same as in Sec. 4.5:

  • Code–Math is predominantly aligned, with the average cosine staying positive across depth.
  • Math–QA is negative in layers L3–6 and positive in layers L14–21.
  • The CW pairs stay near zero.

The synergistic pairs are more aligned on OO than on the neuron set used in Sec. 4.5. Restricting to OO therefore separates synergy from conflict more sharply.

(2) Repeating Secs. 4.3–4.4 and the overlap rate on the sequential RL checkpoints. We use the Math–QA pair, which shows the largest sequential damage in our task-level results. The edits are the stage-wise differences Matho−Codeo\mathrm{Math}_o-\mathrm{Code}_o and QAo−Matho\mathrm{QA}_o-\mathrm{Math}_o, and all metrics are computed exactly as in (1).

MetricSingle-domain expertsSequential
Changed-neuron Jaccard (Sec. 4.3)0.1340.169
Active-neuron Jaccard (Sec. 4.4)0.5150.521
Overlap rate$R$ (chance in brackets)0.427 [0.067]0.474 [0.068]

The findings carry over to sequential training:

  • Edits remain sparse and overlap weakly.
  • Active routes remain strongly shared.
  • Co-edits concentrate on shared routes slightly more than they do for independent experts.

(3) Do the experts and the sequential run identify the same neurons?

Set (expert vs. sequential), mean JaccardMathQA
Active neurons0.9210.924
Changed neurons0.2690.168
Math–QA overlap region$O$0.299

Random sets of the same size give a Jaccard of 0.053.

  • The active routes are nearly identical (J>0.92J>0.92). The domain's data decides them, not the checkpoint that processes the data.
  • The changed sets differ more, but they are still 3–5× above chance.
  • 51% of the experts' overlap neurons reappear in the sequential overlap region. Agreement rises in deeper layers, up to 0.62 at layer 28.

The single-domain analysis therefore finds the same routes that sequential training uses.

Revisions

  • Sec. 4.4 adds the overlap rate between edited and active neurons, with its heatmap.
  • Sec. 4.5 computes the directional cosine on the overlap region OO.
  • A new appendix repeats the analyses on the sequential checkpoints and adds the expert-vs-sequential comparison.
Reviewer_mpUFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how training on one domain can hurt another. It uses a local loss analysis, selective weight rollback, and short refresh training to study and repair this loss of ability. The tests cover four domains, compare refresh training with replay and joint training, and measure recovery after weight rollback.

Critical strengths
  • A short math refresh raises the reported score from 57.66 to 66.04. This shows useful recovery after training across domains.
  • Weight rollback directly tests whether selected weight changes cause damage, going beyond gradient comparisons.
  • The study includes multi models and states the limits of its local theory. This helps readers judge the scope of the findings.
Critical weaknesses
  • Rollback needs earlier checkpoints and domain signals; refresh needs data from the damaged domain. Repair is therefore harder when old data are unavailable or the damaged ability is not already known.
  • The main case repairs math after QA and writing training. The model and order checks do not show how to repair several damaged domains without making them interfere again.
  • The analysis explains why a short refresh can work, but does not choose when to refresh or which weights to restore. Its main value is diagnosing a known failure, rather than guiding repair of a new one.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Author's Comment [1]

We thank the reviewer for the questions on applicability. This response adds:

  • New experiment (Q2). Repairing two damaged domains, Math and Code, in one model. Refreshing one domain at a time recovers 92.2% of the Math drop and 40.0% of the Code drop, against 74.0% and 35.5% for a joint 1:1 refresh, with QA unchanged.
  • Clarifications. Neither rollback nor refresh needs the original training data, and the per-stage validation identifies which domain degraded (Q1). Refresh needs no weight selection and no knowledge of the cause; the theory favors a short refresh over a long one (Q3).
  1. Rollback needs earlier checkpoints and domain signals; refresh needs data from the damaged domain. Repair is therefore harder when old data are unavailable or the damaged ability is not already known.

We thank the reviewer for raising the question of applicability. Below we state what each component requires.

(1) Rollback is a training-free mitigation; targeted refresh is the full repair. Rollback recovers part of the damage without any further training: 57.4% of the QA-induced Math drop at a 9.8% budget, with QA changing by only −0.13. It also serves as the paper's mechanistic test of whether the damage is concentrated in a sparse set of coordinates (Section 6.2), and it fixes the direction used in the curvature probes (Section 6.3). Refresh recovers more (Math 57.66→66.04) but requires training.

(2) Neither method needs the original training data. The two methods need different inputs:

  • Rollback needs the checkpoints before and after the damaging stage (Matho\mathrm{Math}_o and QAo\mathrm{QA}_o, plus Codeo\mathrm{Code}_o to define the Math edit direction) and unlabeled inputs from the two domains, used only in forward passes for the activation statistics. It needs no answers, no rewards and no gradient steps.
  • Refresh optimizes the domain objective LAL_A. Any prompts from the damaged domain with verifiable answers define the same objective, so the original training data is not required, and GRPO needs no reference responses.

(3) Sequential training makes the damaged domain easy to identify. Each stage ends at the checkpoint with the best validation score on its own domain, which gives a reference score for that domain. Evaluating earlier domains at later checkpoints then shows which domain degraded and by how much. For example, Math falls from 66.49 to 57.66 (Table 1). Mixed training provides no such per-domain reference unless separate experts are trained. Once the damaged domain is known, repair amounts to a short refresh on it.

(4) The recipe does not depend on how the checkpoint was obtained. Theorem 1 and Proposition 3 take the starting checkpoint as given; they do not assume it came from sequential training. The same recipe therefore applies to checkpoints from sequential, mixed, or other training. One use is a short refresh to strengthen a target domain. Another is a short refresh after strengthening, to restore a domain that degraded.

(5) Scope. Refresh cannot be applied when no data from the damaged domain is available and the damaged ability is not monitored. Every data-based mitigation we compare against shares this boundary, and each needs more data than refresh does. Replay-all, JT, and CGPO use data from all four domains, whereas Re-Math uses only Math data.

AuthorsOpenReview ↗

Author's Comment [2]

  1. The main case repairs math after QA and writing training. The model and order checks do not show how to repair several damaged domains without making them interfere again.

We thank the reviewer for this question. We repaired two damaged domains in one model: Math and Code, both below their own-stage scores at CWo\mathrm{CW}_o.

(1) Math and Code are largely synergistic but interfere with each other to a small degree. Their edits on shared units are predominantly aligned (Section 4.5), and from the base model each domain's training improves the other: Code training raises Math from 43.19 to 59.63, and Math training raises Code from 29.57 to 34.65 (Table 1). Small interference remains in both directions. In our chain, the Math stage lowered Code from 52.67 to 50.69 (−1.98), which accounts for almost all of the Code drop at CWo\mathrm{CW}_o; the QA and CW stages together add only −0.22. In the other direction, the Code refresh below lowers Math by 0.24.

(2) Two repair schemes. Each run is trained to convergence, and its checkpoint is selected with the same rule as every other stage (Section 3).

  • Sequential refresh, one domain at a time: Re-Math from CWo\mathrm{CW}_o, then a Code refresh from Re-Math (Code-hard). The original Code training data is saturated at this checkpoint, so Code-hard uses 2,560 harder problems from Klear-15k.
  • Joint refresh: Re-{Math,Code}, one refresh from CWo\mathrm{CW}_o on a 1:1 mixture of Math and Code.

Recovery is the fraction of the drop that is regained, measured against each domain's own-stage score: (x−57.66)/(66.49−57.66)(x-57.66)/(66.49-57.66) for Math and (x−50.47)/(52.67−50.47)(x-50.47)/(52.67-50.47) for Code.

SettingStartMathMath rec.CodeCode rec.QA
Codeo\mathrm{Code}_oBase59.63–52.67ref.60.89
Matho\mathrm{Math}_oCodeo\mathrm{Code}_o66.49ref.50.69–60.52
CWo\mathrm{CW}_oQAo\mathrm{QA}_o57.660%50.470%62.34
Re-MathCWo\mathrm{CW}_o66.0494.9%51.0526.4%62.49
Re-Math → Code-hardRe-Math65.8092.2%51.3540.0%62.63
Re-{Math,Code} (1:1)CWo\mathrm{CW}_o64.1974.0%51.2535.5%62.56
Replay-all (4 domains)CWo\mathrm{CW}_o63.5466.6%51.0928.2%62.56

(3) Repairing one domain costs the other little.

  • Re-Math recovers 94.9% of the Math drop. Code does not drop again: it rises from 50.47 to 51.05, although the original Math stage lowered it by 1.98.
  • Code-hard improves Code from 51.05 to 51.35 (recovery 26.4% → 40.0%). It costs Math 0.24 points (66.04 → 65.80), so 92.2% of the Math drop stays recovered. QA stays at 62.63.

After the two refreshes, Math has recovered 92.2% of its drop and Code 40.0%, and QA is unchanged.

(4) This is what our analysis predicts. A later update damages an earlier domain to the extent that it has an opposing component along that domain's sensitive directions.

  • The interference is small in both directions. Math–Code edits are predominantly aligned, so only a small part of either update opposes the other domain. This explains why the mutual interference (1.98 and 0.24 points) is far below the 6.59 points QA training cost Math.
  • Re-Math is a local correction. It removes the displacement that QA and CW introduced along Math-sensitive directions (Theorem 1), instead of learning Math anew from Codeo\mathrm{Code}_o. Proposition 3 bounds its collateral change on other domains by the distance the refresh travels.

(5) Repairing one domain at a time is better than repairing them jointly. The joint refresh recovers 74.0% of the Math drop, against 92.2% sequentially, and 35.5% of the Code drop, against 40.0%. QA is comparable. In a 1:1 mixture, each domain's correction is weighted by 1/2, and at convergence mixed training settles at a compromise between the two objectives instead of restoring each (Appendix C.6). The four-domain Replay-all recovers even less (66.6% for Math, 28.2% for Code).

(6) Recipe. Refresh the damaged domains one at a time. After each refresh, validate all earlier domains, as at the end of every stage. This check shows directly whether a refresh has disturbed a domain that was already repaired. We will add this experiment and the table to the appendix.

AuthorsOpenReview ↗

Author's Comment [3]

  1. The analysis explains why a short refresh can work, but does not choose when to refresh or which weights to restore. Its main value is diagnosing a known failure, rather than guiding repair of a new one.

We thank the reviewer for this comment. We clarify the structure of the paper, the separate roles of rollback and refresh, and how the analysis guides repair.

(1) Repair is one of three contributions. The paper's central claim is about where interference comes from: a later domain damages an earlier one to the extent that its sparse update falls into directions to which the earlier domain's objective is sensitive; these directions lie on computation routes the two domains share, so interference is selective and can arise even when full-model gradients are nearly orthogonal. Our first contribution locates this structure: sparse, weakly overlapping edits act on shared active routes (Section 4). Our second contribution formalizes it: an exact local decomposition of the damage into first-order drift and curvature contributions along the actual intervention, and conditions under which a short refresh contracts the harmful displacement (Section 5). The third contribution is the repair, targeted refresh, which follows from the first two.

(2) Rollback chooses which weights to restore; refresh does not need to. Rollback serves both as a training-free mitigation and as the mechanistic test. As noted in our reply to Q1, it tests whether the damage is concentrated in a sparse, structurally selected set of coordinates (20.4% recovery at a 1.3% budget vs. −6.3% for an equal-budget random set; 57.4% at a 9.8% budget), and its direction is the one used in the curvature probes (4.5–7.4× the curvature of matched random directions). Refresh only resumes training on the degraded domain and selects no weights.

(3) Refresh repairs a new failure without knowing its cause. Refresh needs only the observation that a domain has degraded (Q1, point 3). It does not need to know which later domain caused the damage, which routes are shared, or which coordinates conflict. Our main experiment already works this way. Re-Math starts from CWo\mathrm{CW}_o, where Math has been degraded by the QA and CW stages together, and restores Math from 57.66 to 66.04 without separating their contributions. The same procedure, applied without modification to Phi-4-mini-reasoning (Appendix D.4), again restores Math with QA essentially unchanged. The analysis is therefore not specific to the failure we diagnose. It predicts that damage of this local kind is recoverable by a short refresh, wherever it arises.

(4) When to refresh and for how long. Refresh starts once degradation is detected, which in a staged pipeline is the validation at the end of a stage. Its length does not need to be chosen precisely, and the theory indicates why. In Theorem 1 the term carrying the initial harmful displacement decays geometrically with refresh steps, while in Proposition 3 the bound on collateral change grows with them. Together they favor a short refresh and caution against a long one. This matches our observations: Re-Math improves rapidly and then saturates, and over-long Re-Code visibly degrades Math (Appendix D.2). In practice, refresh stops at the damaged domain's validation plateau, using the same checkpoint selection as every other stage (Section 3).

Reviewer_13uUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-domain interference in multi-domain RL post-training for LLMs across mathematical reasoning, code generation, question answering, and creative writing. The authors analyze a sequential training curriculum, mainly Code → Math → QA → CW, and argue that full-model gradient similarity does not explain the observed selective degradation. They propose a local perturbation-based analysis involving sparse edits, shared active routes, directional alignment, rollback, and short targeted refresh. Experiments are conducted mainly on Qwen3-4B-Thinking-2507, with evaluations on domain-specific benchmarks and additional appendix results on a Phi-family model.

Critical strengths
  1. The paper targets an important practical issue in multi-domain RL post-training. Understanding why later domain training can damage earlier capabilities is relevant for building general-purpose models.

  2. The analysis goes beyond aggregate benchmark scores. By examining sparse edits, active-route overlap, and layer/module-level directional patterns, the paper provides a more detailed view of where interference may occur.

  3. The proposed repair experiments are concrete and easy to interpret. The short Math refresh and rollback experiments give measurable recovery results, rather than only reporting diagnostic statistics.

Critical weaknesses
  1. The empirical support is still narrow relative to the breadth of the claims. Most core results are built around one main model and one fixed sequential curriculum, Code → Math → QA → CW, so it is unclear how general the conclusions are.

  2. The repair-coordinate construction is still fairly heuristic. The selector combines shared activation, edit magnitude, and directional conflict, but the paper does not make clear why this particular combination should be preferred over alternative scoring rules or whether each component is necessary for the rollback intervention.

  3. Several conclusions rely more on diagnostic correlation than direct causal evidence. Shared routes, directional alignment, and curvature probes are plausible explanations, but the experiments do not fully isolate whether these quantities are necessary for the observed degradation and recovery.

  4. The local perturbation theory has assumptions that may not closely match the RL training setting. The analysis is useful for organizing the observations, but the paper itself notes that the theory studies fixed local objectives while the RL experiments use adaptive optimization and a moving policy-gradient surrogate.

  5. The repair strategy is not compared against enough simple alternatives. The paper compares with joint training, CGPO, replay-all, and targeted refresh, but stronger or more systematic baselines such as different replay budgets, mixed refresh schedules, or repeated domain balancing would make the recovery claim more convincing.

Questions for authors

Please see Critical Weaknesses. I would be happy to reconsider my rating if the authors can provide convincing responses to these concerns.

8 comments
AuthorsOpenReview ↗

Author's Comment [1]

We thank the reviewer for the detailed and constructive review. This response adds:

  • Theory (Q4). Theorem 1 and Proposition 3 extended to diagonally preconditioned (Adam-type) updates; the Limitations wording "moving surrogate" is corrected.
  • New repair baselines (Q5). A 1:1 Math–Code mixed refresh and a sequential Math→Code refresh from CWo\mathrm{CW}_o. Math recovery follows the Math mixing weight (94.9% / 74.0% / 66.6%), and the sequential schedule gives the best recovery on both domains.
  • Clarifications. The causal test is the rollback intervention: 57.4% of the Math drop at a 9.8% budget, and 20.4% vs. −6.3% for random at 1.3% (Q3). The selector is an analysis tool, and the existing ablation shows what each factor does (Q2). The paper already covers a reversed order, a second refresh target and a second model (Q1).
  1. The empirical support is still narrow relative to the breadth of the claims. Most core results are built around one main model and one fixed sequential curriculum, Code → Math → QA → CW, so it is unclear how general the conclusions are.

(1) The structural findings match prior work on other models and algorithms. Apart from the gradient analysis (§4.1, joint training), Section 4 uses four single-domain experts trained independently from the base model (Codes,Maths,QAs,CWs\mathrm{Code}_s,\mathrm{Math}_s,\mathrm{QA}_s,\mathrm{CW}_s), so it does not depend on the curriculum. Prior studies report the same structure beyond our setting:

  • RL updates are sparse yet nearly full-rank, across 10 LLMs and 7 RL algorithms (Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. Mukherjee et al., 2025).
  • RL largely preserves the base model's representations (Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models. Shi et al., 2026).
  • The reasoning gains of an RL update concentrate in a very small subspace: its rank-1 component recovers over 99% of them (On Predictability of Reinforcement Learning Dynamics for Large Language Models. Cai et al., 2025).

Our observations, sparse and weakly overlapping edits acting on strongly shared active routes (§4.2–4.4), are consistent with these results. These works do not study where cross-domain conflict arises, nor how to repair it, which is the question we address.

(2) The paper covers more than one training order and more than one model. Beyond Code→Math→QA→CW, we report the reverse order QA→Math (Appendix D.3), in which Math training does not damage QA to a comparable degree. This shows that interference is directional, consistent with our account that damage depends on whether the later domain's update enters the earlier domain's sensitive directions. In Code→Math→Re-Code (Appendix D.2), refresh recovers Code while largely preserving Math, and its effect on other domains grows with the number of refresh steps, consistent with Proposition 3. On Phi-4-mini-reasoning, Math→QA→Re-Math (Appendix D.4) reproduces the pattern of damage from a later domain followed by recovery through a short refresh. In addition, four-domain joint training on Phi shows that full-model gradients are likewise nearly orthogonal.

AuthorsOpenReview ↗

Author's Comment [2]

  1. The repair-coordinate construction is still fairly heuristic. The selector combines shared activation, edit magnitude, and directional conflict, but the paper does not make clear why this particular combination should be preferred over alternative scoring rules or whether each component is necessary for the rollback intervention.

(1) On the selector design. We do not claim that the selector is an optimal scoring rule; in the paper it is an analysis tool. It supports the following claim. The QA displacement contains a sparse, conflict-aware subset whose rollback recovers part of the Math performance (20.4% at a 1.3% budget), while an equal-budget random rollback gives a recovery of −6.3%. Along this subset, the Math curvature density is also 4.5–7.4× that of random directions (Table 4).

(2) Why these three signals. Each signal comes directly from an observation in Section 4:

  • A reflects shared active routes. In Section 4.4, Math–QA has the highest activation overlap of all domain pairs (0.515).
  • M reflects sparse, small-magnitude edits (Sections 4.2–4.3).
  • C reflects directional conflict on shared edited neurons. In Section 4.5, the Math–QA cosine is negative in layers L3–6.

The signals can also be read as cheap proxies for Qv=v⊤HˉvQ_v=\mathbf v^\top\bar{\mathbf H}\mathbf v in Proposition 2.

  • A×M. For an MLP neuron, the gradient with respect to its down-projection column is hi⋅∂L/∂yh_i\cdot\partial L/\partial y, so its Gauss–Newton block scales with E[hi2]\mathbb E[h_i^2]. A×M therefore ranks neurons approximately by their diagonal contribution δi⊤Giiδi\delta_i^\top G_{ii}\delta_i.
  • M×C. M×C approximately corresponds to the length of the QA update projected onto the reverse of the Math edit. Under a local quadratic model with Matho\mathrm{Math}_o as its minimizer, undoing a fraction cc of the Math edit raises the loss by c2c^2 times the loss reduction achieved by Math training. C therefore selects directions with known Math sensitivity.

We take the product because rolling back a neuron helps only if three conditions hold together: Math uses the neuron, QA training changed it substantially, and the change opposes the Math edit. If any factor is close to zero, rolling back that neuron has little effect on the Math loss. We have not shown that the product outperforms other ways of combining the factors.

(3) What each component does. Section 6.2 (Table 3) reports the main results, and Appendix D.5 gives the full ablation and the layer-wise analysis (Fig. 14). The selector works at two levels: it first allocates the global budget across layers by a layer score, then ranks neurons within each layer. The ablations show that the three factors play different roles at these two levels.

  • C sets the layer allocation. Every selector that contains C produces a strongly layered, U-shaped budget that concentrates neurons in shallow and late layers; selectors without C give a nearly flat budget (Fig. 14, middle). Dropping C halves recovery (A×M 9.6%). C is a weaker ranker inside a layer: under the full selector's layer budget, ranking by C alone recovers 12.7%.
  • A selects neurons within a layer. Under the full selector's layer budget, ranking by A alone recovers 20.4%, the same as the full selector. When A also sets its own budget, recovery falls to 16.6%. A's strength therefore depends on the layer allocation that C provides.
  • M calibrates the conflict term within a layer. C is a clipped cosine, max⁡(0,−cos⁡(ΔvM,ΔvQA))\max(0,-\cos(\Delta\mathbf v_{\mathrm{M}},\Delta\mathbf v_{\mathrm{QA}})), so it says only that QA moved a neuron against the Math edit, not by how much. M×C measures the amount of QA displacement that opposes the Math edit, which is what rollback actually removes. Without M, C dominates the within-layer ranking and the selector can pick neurons whose opposing displacement is small. A×C recovers only 9.9%, although its budget is also shaped by C, while adding M raises it to 20.4% (A×M×C), and M×C alone reaches 18.3%.

No single factor reproduces the full selector on its own: when each factor sets its own budget, A, M and C recover 16.6%, 13.9% and 16.3%, against 20.4% for A×M×C. The three factors together give the best recovery, and every ablated selector recovers more than an equal-budget random rollback (−6.3%). Different sparse subsets reach similar recovery, so the support is not unique. This fits our use of the selector as an analysis tool that shows the damage is sparse and locatable, not as an optimal selection rule.

AuthorsOpenReview ↗

Author's Comment [3]

  1. Several conclusions rely more on diagnostic correlation than direct causal evidence. Shared routes, directional alignment, and curvature probes are plausible explanations, but the experiments do not fully isolate whether these quantities are necessary for the observed degradation and recovery.

Our answer on necessity is different for the mechanism and for the signals.

  • Mechanism. We claim that most of the damage requires the later-domain update to have a component in the earlier domain's high-sensitivity subspace, and we test this necessity by intervention.
  • Signals. Shared routes and directional alignment are how we locate that subspace, and curvature checks that it is high-sensitivity. We do not claim that any one of these quantities is by itself a necessary condition for damage. Each carries information about where the damage lies, and the ablations measure that information by intervention.

(1) Mechanism: necessity is tested by intervention. Rollback removes the QA displacement on the located coordinates and leaves the rest of QAo\mathrm{QA}_o unchanged.

  • At a 9.8% budget, rollback recovers 57.4% of the QA-induced Math drop, while QA changes by only −0.13.
  • At a 1.3% budget, the selector recovers 20.4%. An equal-budget random rollback gives −6.3%.

Removing the located component therefore removes most of the damage, and removing the same amount of displacement elsewhere does not.

(2) Signals: each helps locate the damage, and none is necessary on its own. The ablations (Table 3, Appendix D.5) measure what each signal contributes.

  • Shared routes. Dropping activation from the selector costs little: M×C recovers 18.3%, close to the full selector's 20.4%. Overlap alone also cannot determine damage. Math–QA activation overlap is symmetric, yet in the reverse order QA→Math (Appendix D.3), Math training does not damage QA to a comparable degree.
  • Directional alignment. Section 4 relates edit direction to transfer and conflict through observations across domain pairs. Code–Math edits are aligned, and Code→Math transfer is positive. Math–QA edits have a negative cosine in layers L3–6, and QA training damages Math. The rollback ablation tests this relation by intervention: without the conflict term, recovery roughly halves (A×M 9.6% vs. 20.4% for A×M×C). The conflict term also sets the layer budget. Ranking by activation recovers 20.4% under the conflict-aware layer budget, but only 16.6% when activation sets its own budget.

Every ablated selector recovers more than random, and the full product recovers the most. Each signal therefore carries part of the localization. Because different sparse subsets reach similar recovery, the damage is concentrated in a sparse region that several signals can find, not in one unique set of coordinates.

(3) Curvature probes check the located subspace and are not a separate causal test. The mechanism requires the located subspace to be high-sensitivity for Math, and the probe checks this property. At Matho\mathrm{Math}_o, the Math curvature density along the located direction is 4.5–7.4× that of equal-budget random directions (Table 4). The probe shows what kind of direction the selector finds. The causal evidence comes from the rollback in (1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

We thank all reviewers and the Area Chair for their time and engagement. Below we summarize the discussion with each reviewer and where it ended.

Our claims. The paper studies cross-domain interference in multi-domain RL post-training and makes three claims. First, a later domain damages an earlier one when its sparse update falls on computation routes the two domains share, along directions to which the earlier domain is sensitive. Second, a local perturbation analysis decomposes this damage exactly and gives conditions under which a short refresh contracts it. Third, a short targeted refresh repairs the damaged domain with limited effect on the others.

Reviewer 13uU asked about the generality of the setting, the design of the rollback selector, causal versus correlational evidence, the match between the theory and RL training, and the strength of the repair baselines. We pointed to the existing results on a reversed order, a second refresh target and a second model. We explained the role of each selector factor using the ablation. We also separated the causal test from the diagnostic signals: rollback recovers 20.4% of the Math drop at a 1.3% budget, against −6.3% for an equal-budget random set. We extended Theorem 1 and Proposition 3 to diagonally preconditioned (Adam-type) updates and corrected the "moving surrogate" wording. We ran two new repair baselines, a 1:1 Math–Code mixed refresh and a sequential Math→Code refresh. Math recovery follows the Math mixing weight (94.9% / 74.0% / 66.6%), as the first-order analysis predicts. Reviewer 13uU raised their score.

Reviewer mpUF asked what rollback and refresh require, how to repair several damaged domains, and whether the analysis can guide the repair of a new failure. We clarified three points. Neither method needs the original training data. Per-stage validation identifies the damaged domain. Refresh needs neither weight selection nor knowledge of the cause. In a new experiment we repaired two damaged domains in one model. Refreshing Math and then Code recovers 92.2% and 40.0% of the two drops, against 74.0% and 35.5% for a joint 1:1 refresh, with QA unchanged. In a later comment, the reviewer raised the importance of the research problem. We replied with reports from Qwen3, DeepSeek-V3.2, Nemotron-Cascade and MiMo-V2-Flash, each of which designs its RL pipeline around cross-domain interference. We also explained why repairing a model after it has degraded matters when a deployed model is iterated with new domains.

Reviewer BPsW asked about six things: order sensitivity, whether overlap simply reflects domain relatedness, the difference between the metrics of Sections 4.4 and 4.5, the use of single-domain experts, whether refresh is related to the identified parameters, and the repair of several damaged domains. We showed that interference is directional (reversed order, Appendix D.3) and that relatedness alone does not set the sign of transfer. We then ran the analyses the reviewer suggested. On neurons that are both edited and active, 31–46% of co-edited neurons lie on shared routes (6–9× chance), and the direction pattern of Section 4.5 holds. The same metrics computed on the sequential checkpoints reproduce the single-domain findings. The unconstrained refresh puts its largest updates on the identified units: 85.4% of the Math–QA shared edited-and-active neurons fall among its most-updated neurons (8.5× chance). The reviewer then asked whether short refresh still works when a non-Math domain suffers large forgetting. We tested this on the public checkpoints of Nemotron-Cascade-8B. Its RLHF stage lowers IFEval from 69.4 to 61.9, and a 60-step IF-RL refresh restores it to 70.7, with SuperGPQA and MMLU-Pro staying within 0.6 points. Reviewer BPsW raised their score.

What the discussion established.

  • The damage is localized. Two independent tests support this. Rolling back a sparse selected set recovers the damage (57.4% of the Math drop at a 9.8% budget, and 20.4% at 1.3%), while an equal-budget random set does not (−6.3% at 1.3%). The unconstrained refresh also concentrates its updates on the same units.
  • Targeted refresh works beyond the main case. It repairs two damaged domains in one model, a second model family (Phi-4-mini-reasoning), and a non-Math domain with large forgetting in a model trained by an independent team.
  • One dedicated refresh per damaged domain gives the highest recovery on each domain. Mixed refresh at different weights trades the domains against each other.
  • The contraction result also holds for Adam-type preconditioned updates, not only for gradient descent.

All clarifications and new experiments from the discussion will be included in the revised paper.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.