acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Recoverability: Suppression-Like vs. Overwrite-Like Capability Loss in LLM Post-Training

Abstract

Post-training improves task performance but can degrade general capabilities such as instruction following, factuality, and safety. The same accuracy drop can conceal two different states: a capability may remain accessible but difficult to elicit, or it may require substantial relearning. We call these suppression-like and overwrite-like forgetting. We introduce TRIAGE, a diagnostic and repair framework that combines recovery dynamics, hidden-state readability, and lightweight-repair operability into a continuous Forgetting Typing Score (FTS). FTS achieves AUROC under leave-one-injection-type-out evaluation and transfers to new suppression mechanisms, a second model family, and a second training objective. At matched mathematical-reasoning gains, RLVR leaves more accessible losses than SFT, with FTS-weighted suppression indices of and . A controlled study reproduces of this gap within supervised training by combining on-policy data with proximity-to-base regularization; realized model–base distance strongly predicts forgetting type across supervised and RL runs. FTS-guided repair turns this distinction into a practical advantage, reaching a common recovery target at – lower cost than uniform replay. The type of forgetting thus connects how post-training changes a model to the intervention that restores its capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
catastrophic forgetting, LLM post-training, RLVR, forgetting diagnosis, capability recovery, model repair

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UrXkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TRAGE, a framework that diagonse and fix ability loss after LLM post-training. It distinguishes two types of forgetting: suppression-like forgetting and overwrite-like forgetting. LLM is likely to recover the abilities lost from suppression like forgetting, whereas needs relearning for ovrewrite-like forgetting. The method combines recovery dynamics, hidden-state readability, and lightweight- repair effectiveness into a Forgetting Typing Score, which helps with repair. Experiments demonstrate that RLVR produces more recoverable losses than SFT, and TRIAGE succeeds in reducing the repair costs compared with uniform replay, and largely preserves new-task performance.

Critical strengths

Useful perspective on post-training forgetting: The paper studies both how much performance LLMs lose after post-training and how easily the lost capabilities can be recovered. Although previous work has explored related ideas, the authors clearly position their contribution by introducing the Forgetting Typing Score (FTS). This perspective helps determine when lightweight repair is sufficient and when relearning is needed.

Effective and efficient capability repair: As reported in Section 4.4, TRIAGE reaches approximately 90% recovery at 1.7–2.4× lower computational cost, measured in GPU-hours, than uniform replay while largely preserving new-task performance. The cost comparison is thorough, including diagnosis, repair, validation, and safeguard costs.

Critical weaknesses

In Section 3.3, the overwrite-like reference criterion requires recovery below 50% after replay with 4,096 examples for two epochs. However, Table 7 reports a median half-recovery cost of 312 examples and a median fitted recovery ceiling of ρ∞ = 0.83 for these references. Appendix D.7 also indicates that only 11 of the 52 overwrite-labeled references have ρ∞ < 0.5. Could the authors explain this difference and clarify whether the same training protocol was used for these two measurements?

Questions for authors
  1. Please refer to the concern raised in the Critical Weaknesses section.

  2. Could the authors include a LoRA-based replay baseline? Since Axis 1 already uses LoRA for recovery, this comparison would help determine whether TRIAGE provides cost savings over parameter-efficient replay methods.

  3. The paper includes single-axis ablations, evaluating Axes 1, 2, and 3 separately in Table 9. It would also be helpful to evaluate two-axis combinations (1+2, 2+3, and 1+3) and compare them with the full TRIAGE framework.

Reviewer_t9VCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TRIAGE, which scores each capability degraded by post-training with a Forgetting Typing Score (FTS) that fuses LoRA recovery speed, linear-probe readability and the success of gradient-free light repairs, calibrated on injected reference states labelled by causal patching and a replay ceiling. FTS reaches AUROC 0.92–0.94 and transfers with frozen weights across injection mechanisms, a second model family and a second objective; at matched math gain, RLVR losses on Qwen3-8B are far more suppression-like than SFT losses (74% vs 39%), a 2×2 factorial attributes most of the gap to self-generated traces plus KL regularization, and routing light repair versus replay by FTS reaches ≈90% recovery at 1.7–2.4× lower cost than uniform replay.

Critical strengths

The main strength is experimental rigor. The reference set is built from mechanism-matched injections with labels derived from patching and replay measurements rather than from recipe intent; the fused score is evaluated leave-one-mechanism-out and, with weights frozen, on 18 suppressions with no operator inverse, on Llama-3.1-8B and on an instruction-following objective, with 36 of 39 externally verified real-checkpoint instances agreeing; nearly every threshold, operator class, patching window and budget is subjected to a sensitivity analysis; costs are itemized to the GPU-hour; and the repair evaluation is a closed loop with seven ablated routers, including the ones that make the authors’ own method look least necessary (Table 25, Appendix H.1.1). The central empirical finding, that RLVR and SFT losses differ in recoverability at matched gain and also at matched severity (49.3% vs 74.2%), is established with three seeds, matched-gain and matched-severity controls and a factorial decomposition, and is reported conservatively.

Critical weaknesses

My concern is that the practical value of the findings appears modest in the regimes tested, with the caveat that the paper is dense and I may have missed evidence that would change this reading. As far as I can tell, the recoverability signal is shown to change only one decision, how to repair a finished checkpoint, and there it adds about three recovery points over simply trying the light operators first (Axis-3-only router, Table 25; a 3% cost difference at matched recovery), while a near-free statistic, KL to the base, already predicts forgetting type with R² = 0.96 (Section 4.3); post-hoc repair is moreover a narrow workflow with small absolute savings (about 35 GPU·h against the 161 GPU·h run being repaired). The SFT-versus-RLVR result seems largely to confirm what RL’s Razor and Retaining by Doing predict, and the diagnostic’s AUROC is measured against labels built from near-duplicates of its own axes, so neither appears to add significance beyond the repair experiments. What I believe would strengthen this work is a demonstration of a decision that loss magnitude or KL gets wrong and the type signal gets right: for instance, monitoring FTS along a training trajectory to intervene before losses turn overwrite-like (Appendix F.1 suggests the signal exists), or auditing safety and unlearning interventions for superficial, easily unmasked removal. Both look feasible but are, as far as I can see, only hinted at. The main text is hard to follow without the appendix and would benefit from one capability followed from measurement through typing to repair.

I am on the fence between a score of 2 and 3; as it stands, I have decided for 2 because the significance of the findings is not sufficiently obvious and the presentation of the work should be improved.

Based on the additional results in the rebuttal I decided to raise my score to 3.

Questions for authors

(1) Does per-capability diagnosis beat a near-free alternative? Please add to the closed loop of Table 25 a router that assigns types from arm-level KL to the base (Section 4.3) rather than from FTS, and report, on the 58 real-checkpoint instances, whether FTS predicts realized repair cost beyond loss magnitude and KL. (2) Is there already a decision that the type signal gets right and magnitude or KL gets wrong? For instance, on the dense SFT and GRPO checkpoints of Appendix F.1, does stopping or intervening on the FTS trajectory improve the gain versus repair-cost trade-off relative to stopping on loss alone; or, on your O-v unlearning and safety instances, does FTS flag superficial removal that accuracy does not? (3) Is the base model θ0\theta_0 the released, already post-trained Qwen3-8B (and Llama-3.1-8B-Instruct) in all experiments? If so, please state this and discuss whether the suppression-versus-overwrite findings are expected to hold when post-training starts from a pretrained base model. (4) At matched recovery, what is the cost of the Axis-3-only router versus full TRIAGE in the mixed-band, small-delta and cross-family settings of Table 27, where the fusion margin is said to be largest? (5) What AUROC does FTS reach against a label that shares no measurement with its axes, for example full-parameter relearning cost to 90% recovery, and does the R² = 0.96 proximity relation survive when the parameter-space operators (interpolation, layer scaling) are removed from Axis 3 and the repair loop?

Suggestions: describe the sample-wise forgetting and CKA rows of Table 9 as reference points for endpoint-only and representation-only signals rather than as competing diagnostics, since neither was designed to predict recoverability; state explicitly that Section 4.3 extends the KL/on-policy account of Shenfeld et al. (2026) and Chen et al. (2026) from forgetting magnitude to forgetting type, and reconcile your one-shot self-generated arm with Chen et al.’s finding that such data forgets much more than RL in magnitude.

Reviewer_DkgeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates types of skill forgetting/capability loss when LLMs are post-trained and introduces a repair framework based on the diagnosis. Specifically, they hypothesize that capability losses can be of two types - suppression-like and overwrite-like. The former can be reverted easier than the latter. The paper finds that RLVR leaves more accessible losses compared to SFT and demonstrates repair through combining on-policy data with special regularization.

Critical strengths
  1. Tackles an important problem - diagnosing and repairing catastrophic forgetting when LLMs are post-trained with SFT or RLVR
  2. The two categories of forgetting proposed and the mitigations generally seem solid.
Critical weaknesses
  1. The experiments could be executed better. For one, the results seem mostly based on synthetically constructed tasks for post-training. It would be great to have validation that FTS and the repair methods work on actual post-training pipelines.
  2. The paper's experimentation details in sections 3 and 4 are quite difficult to parse - very dense text with jargon (likely due to it being written by AI without human editing). This makes it hard to understand what exactly was done.
Questions for authors
  1. Do you think that the 3 signal score can do better than a single signal/plain KL on the base model?
  2. Can you run the entire pipeline on naturally occuring forgetting (e.g. between a qwen base and qwen post-trained model)? This would make the method a lot more ready for real-world use.
  3. Can you compare with stronger baselines such as replay with KL regularization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.