- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes TRAGE, a framework that diagonse and fix ability loss after LLM post-training. It distinguishes two types of forgetting: suppression-like forgetting and overwrite-like forgetting. LLM is likely to recover the abilities lost from suppression like forgetting, whereas needs relearning for ovrewrite-like forgetting. The method combines recovery dynamics, hidden-state readability, and lightweight- repair effectiveness into a Forgetting Typing Score, which helps with repair. Experiments demonstrate that RLVR produces more recoverable losses than SFT, and TRIAGE succeeds in reducing the repair costs compared with uniform replay, and largely preserves new-task performance.
Critical strengths
Useful perspective on post-training forgetting: The paper studies both how much performance LLMs lose after post-training and how easily the lost capabilities can be recovered. Although previous work has explored related ideas, the authors clearly position their contribution by introducing the Forgetting Typing Score (FTS). This perspective helps determine when lightweight repair is sufficient and when relearning is needed.
Effective and efficient capability repair: As reported in Section 4.4, TRIAGE reaches approximately 90% recovery at 1.7–2.4× lower computational cost, measured in GPU-hours, than uniform replay while largely preserving new-task performance. The cost comparison is thorough, including diagnosis, repair, validation, and safeguard costs.
Critical weaknesses
In Section 3.3, the overwrite-like reference criterion requires recovery below 50% after replay with 4,096 examples for two epochs. However, Table 7 reports a median half-recovery cost of 312 examples and a median fitted recovery ceiling of ρ∞ = 0.83 for these references. Appendix D.7 also indicates that only 11 of the 52 overwrite-labeled references have ρ∞ < 0.5. Could the authors explain this difference and clarify whether the same training protocol was used for these two measurements?
Questions for authors
-
Please refer to the concern raised in the Critical Weaknesses section.
-
Could the authors include a LoRA-based replay baseline? Since Axis 1 already uses LoRA for recovery, this comparison would help determine whether TRIAGE provides cost savings over parameter-efficient replay methods.
-
The paper includes single-axis ablations, evaluating Axes 1, 2, and 3 separately in Table 9. It would also be helpful to evaluate two-axis combinations (1+2, 2+3, and 1+3) and compare them with the full TRIAGE framework.