acceptodds
Under review as a conference paper at ICLR 2027

Two Checkpoints Are Enough: Data-Free Post-Hoc Repair of Catastrophic Forgetting

Abstract

Fine-tuning a language model for a target task degrades capabilities that the training data never explicitly threatened, yet it also improves other held-out skills as a useful side effect. Low-rank fine-tuning methods limit the forgetting but forfeit these incidental gains. Furthermore, recovering the damaged capabilities usually requires the data the model was originally trained on, which practitioners who fine-tune open-weight models rarely have. We study post-hoc repair of this catastrophic forgetting using only the pre-fine-tuned checkpoint and its fine-tuned descendant . The goal is to recover the damaged capabilities while preserving the target-task gains and incidental held-out improvements that reversion toward the base discards. We show that a single fine-tuning update separates in singular-value space into a small set of task-aligned spikes and a broad bulk that matches the Marchenko-Pastur prediction for an IID noise matrix. DG-Hard applies the Gavish-Donoho optimal hard singular-value threshold to each weight-delta matrix, keeps the spikes, and reverts the bulk. The cutoff is closed-form, so the repair needs no further training, gradients, data, or hyperparameters. To evaluate such methods, we also introduce a partition-conditional metric that separately tracks healing, preservation, non-damage, and target-task retention. Across nine held-out benchmarks, DG-Hard attains both the highest combined score and the highest mean held-out accuracy among non-spectral training-time and post-hoc baselines. We show that the retained signal carries the incidental held-out gains, and a model rebuilt from the discarded bulk alone keeps only a third of the incidental held-out gains. Alternative truncation rules such as fixed-rank and fixed-energy truncation reach or exceed DG-Hard only when their retained fraction is tuned against old-task performance, while DG-Hard's closed-form cutoff needs no tuning. The same repair also recovers most of the safety alignment eroded by benign fine-tuning, without alignment data. The code is available at https://anonymous.4open.science/r/dghard-repair-2D84/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic forgetting, LLMs, LoRA, finetuning, continual learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fzCSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a post-hoc approach for mitigating catastrophic forgetting using the pretrained and fine-tuned checkpoints. Given the weights of the pretrained model and the fine-tuned one, they model their differences as signal + noise, then using the optimal hard threshold for singular values, they trim the spectrum of the difference to remove the noise.

The authors evaluate the approach on Qwen3.5-4B and Llama-3.2-3B-Instruct across seven fine-tuning setups and nine held-out benchmarks. Their method reports a favorable balance between restoring degraded base capabilities and preserving fine-tuning gains.

Critical strengths
  1. The idea of modeling the forgetting problem as removing noise from signal in the spectrum is interesting and seems to be novel to me. This also justifies the spectral analysis and the correction story.

  2. The algorithm is extremely simple, you just need the two checkpoints that you usually have and no additional data or hyper-parameter tuning.

  3. The experimental part is strong. Evaluating on two language models, seven fine-tuning tasks, and nine benchmarks. The results are pretty consistent, and the method seems to work.

Critical weaknesses
  1. It seems that the results are not as strong as the paper frame. DG-Hard doesn't beat the other methods on all metrics, V-SoftMask is better in preservation and on-task retention, and FAPM is better in healing and non-damage. DG-Hard gives a reasonable balance between them, but I don't think this necessarily means it is a better method.

  2. The evaluation metric seems strange. Harmonic mean puts a lot of weight on having balanced results, which is where DG-Hard has the advantage. Another thing I find questionable is setting negative scores to zero. I don't see a clear reason for doing this, and it might affect the comparison with the baselines. I would like to see the results without this clamping and with the Pareto trade-offs.

Questions for authors

See weaknesses, plus:

  1. Can the authors explain why the nested harmonic mean is the appropriate primary ranking criterion here?

  2. How well does the method generalize beyond the two language models considered here? I would be interested in seeing results on additional model families and scales, since the spectral structure of fine-tuning updates may depend strongly on architecture and optimization regime.

6 comments

We thank the reviewer for the careful reading, and for finding the signal-plus-noise view of forgetting novel, the algorithm simple, and the experiments strong and consistent. We answer the two weaknesses and the two questions below.

Weakness 1. DG-Hard does not beat the other methods on every metric.

We agree with the observation, and the paper makes no claim that DG-Hard leads every sub-statistic. The claim is that, among the non-spectral methods, DG-Hard gives the best balance across them, measured by a score built to reward balance. The paper states its ranking claim on that score in the abstract (p. 1, ll. 27-28), contribution 3 (p. 2, ll. 95-96), the opening of Sec. 4 (p. 6, ll. 303-304), and the conclusion (p. 9, l. 483). Sec. 3.4 defines the score as "demoting methods that win one axis by sacrificing another" (p. 6, ll. 294-295), and App. G says it "is designed precisely to demote single-axis extremes" (p. 29, ll. 1540-1541).

We built the score this way because of what a repair has to do. It has to heal the damaged capabilities, leave the untouched ones alone, keep the incidental held-out gains, and keep the target-task gain, all at the same time.

A method that heals most of the damage but discards the incidental and target-task gains has undone the fine-tuning it was meant to repair. A method that keeps every gain but heals little has left the forgetting in place. Both extremes are available without any method. Reverting to Base heals everything, and keeping the fine-tuned model preserves everything.

In our view, a lead on one sub-statistic therefore says little about whether a method repairs forgetting, and the sub-statistics have to be read together, which our evaluation metric attempts to do.

Tab. 1 shows this split. FAPM, WiSE-FT, L1-reg, and LoRA each have a higher Clean-up than DG-Hard, and V-SoftMask has a higher Retention. DG-Hard is the only non-spectral method that scores 80 or more on both (Tab. 1 caption, p. 7, ll. 325-326), and among the non-spectral methods it is the one closest to the ideal corner of % healed and % preserved in all five cohorts (Fig. 1 caption, p. 3, ll. 121-122). Three further observations support the balance claim without relying on the Combined score.

  1. No non-spectral method dominates DG-Hard. On the four sub-statistics of Tab. 1, no baseline is at least as good as DG-Hard on all four, and each sub-statistic leader pays on another axis. FAPM heals the most damage (93.6%) but keeps none of the incidental gains (% preserved −4.4) and only 77.8% of the on-task accuracy. LoRA also heals more than DG-Hard (87.3%) and keeps 6.0% of the incidental gains. V-SoftMask keeps the most gains (95.8%) and heals 23.2% of the damage. Fig. 4 names FAPM and V-SoftMask the cleanup and retention extremes (caption, p. 31, from l. 1633).

  2. Among the non-spectral methods, DG-Hard has the strongest weakest axis. Its lowest sub-statistic is 78.5. The next best non-spectral method's lowest is 59.4 (WiSE-FT), and every other one falls to 44.5 or below (Tab. 5, "Minimum" column, and Tab. 1).

  3. The flat average agrees. Mean held-out accuracy involves no metric design choice. DG-Hard is highest among the non-spectral methods at 64.4, ahead of V-SoftMask (63.8), the fine-tuned model (63.3), and WiSE-FT (63.2), while keeping on-task accuracy at 81.6 against 83.7 for the fine-tuned model (Tab. 1).

We address the choice of the harmonic mean itself under Weakness 2 and Question 1.

We thank the reviewer for pointing out that this scope can be read more strongly than intended. The extended and revised PDF will state it directly in Sec. 4. DG-Hard leads none of the four sub-statistics on its own, and it is the only non-spectral method that scores 80 or more on both Clean-up and Retention.

Weakness 2. The harmonic mean favours balance, negative scores are set to zero, and the reviewer asks for unclamped results and Pareto trade-offs.

2a. Does the harmonic mean create DG-Hard's lead, and why did we choose the harmonic mean?

DG-Hard's lead among the non-spectral methods holds under other aggregators, including one that gives balance no extra weight. The table below scores all seven methods on the Overall cohort. Tab. 5 (App. C, p. 23, from l. 1189) already reports the four strongest non-spectral methods under these aggregators, and the table below adds the remaining three and the unclamped column. The arithmetic mean of the four sub-statistics gives no extra weight to balance, and DG-Hard is first among the non-spectral methods under it (86.6 against 83.0 for WiSE-FT). The order of all seven non-spectral methods is the same in every column, including the geometric mean.

MethodCombined (paper)Combined, no floorGeometric meanArithmetic meanMinimum
DG-Hard86.186.186.486.678.5
WiSE-FT80.180.181.683.059.4
L1-reg72.072.076.079.444.5
V-SoftMask54.254.268.279.123.2
CoFi-Tune42.742.757.569.317.0
LoRA20.020.045.468.46.0
FAPM0.0−20.60.0*66.1−4.4
Tuned spectral controls (§4.4)
Fixed energy-percentage 15% (tuned)87.987.988.188.381.6
Fixed-rank 5% (tuned)86.986.987.187.479.6

Bold marks the best non-spectral method in each column.

* The geometric mean is undefined for a negative input, so FAPM's negative % preserved is floored at 0 in that column.

The tuned spectral controls behave the same way under every aggregator. The lower block repeats the two tuned controls from Tab. 5. Once their retained fraction is tuned against old-task performance, they reach or slightly exceed DG-Hard under every aggregator, as they do under the harmonic mean (App. C, p. 22, ll. 1166-1167). Neither has a negative sub-statistic, so the floor never applies to them. DG-Hard reaches the same level with a closed-form cutoff and no fraction to select (Sec. 4.4).

The submitted paper already addresses this concern, and tests the Combined score against three of its design choices, namely the partition threshold (Sec. 3.4, p. 5, l. 269; Tab. 4), the aggregator (Tab. 5), and the ratio-of-sums definition (Sec. 3.4, p. 6, ll. 296-297; Tab. 6). DG-Hard is first among the non-spectral methods in every one of these checks.

The ratio of sums does not create the lead either. % healed and % preserved add up the gains and losses over all damaged or improved triples before dividing. Tab. 6 (App. C, p. 23, from l. 1209) recomputes Combined with a mean of per-triple ratios, which gives every triple equal weight. DG-Hard is first among the non-spectral methods under both definitions (86.1 and 83.3, against 80.1 and 76.4 for WiSE-FT), and the only change in order is that V-SoftMask and CoFi-Tune exchange fourth and fifth place (p. 23, ll. 1206-1207).

Why we chose the harmonic mean. The choice follows from what a repair has to do, which we described under Weakness 1.

A repair has to heal the damage and keep the gains at the same time, and each side alone is available without any method. Reverting to Base heals everything, and keeping the fine-tuned model preserves everything.

A ranking score should therefore place both of these trivial strategies at the bottom. The harmonic mean does this, because it is pulled toward its smaller input. A method that scores near zero on one side scores near zero overall, however strong its other side is. A flat average of the two sides does the opposite and gives each trivial strategy 50, the same score as a method that recovers half of the damage and keeps half of the gains (App. C, p. 21, l. 1131 to p. 22, l. 1137).

The nesting applies the same rule at two levels. Within each side, both of its sub-statistics must be high, and across the two sides, both sides must be high. The reviewer is right that this rewards balance, and that is its purpose. The table above shows that DG-Hard's first place among the non-spectral methods does not depend on it.

2b. Clamping of negative scores, unclamped results, and Pareto trade-offs

In the Overall cohort the clamping changes only FAPM's scores and does not change any rankings. The "Combined, no floor" column of the table in 2a gives the unclamped results. The extended and revised PDF will add this column to Tab. 5. Every reported sub-statistic is uncapped and unfloored (App. C, p. 22, ll. 1185-1187), and Tab. 1 prints FAPM's % preserved of −4.4.

Only the inputs to the harmonic mean are floored at 0 (Sec. 3.4, p. 6, ll. 277-278). On the Overall cohort the floor applies only to FAPM. Without it, FAPM's Retention becomes −9.3 and its Combined −20.6, and FAPM is last either way. Across the four sub-cohorts the floor applies five more times (LoRA and FAPM on Cognition and on Reasoning, V-SoftMask on Non-reasoning), and in each case the method is in the bottom two of its cohort with or without the floor.

We floor the inputs because the harmonic mean is not monotone when its inputs have opposite signs. For example, HM(−6,5)=60\mathrm{HM}(-6, 5) = 60, which is higher than HM(5,5)=5\mathrm{HM}(5, 5) = 5. Without the floor, a method could raise its score by doing more damage on one axis. With the floor, a negative sub-statistic always counts as zero.

Pareto view. Among the seven non-spectral methods, five are non-dominated on the four unclamped sub-statistics, namely DG-Hard, WiSE-FT, L1-reg, V-SoftMask, and FAPM. CoFi-Tune and LoRA are dominated by L1-reg. Fig. 4 (App. I) plots the Clean-up vs Retention frontier and the Knowledge vs Cognition frontier, and in both panels DG-Hard is the non-spectral method closest to the upper-right corner. Fig. 1 plots % healed against % preserved per cohort. Membership in the non-dominated set does not rank the five methods by itself, and every aggregator in the table in 2a selects DG-Hard among them.

The extended and revised PDF will report all seven methods under every aggregator in Tab. 5 (App. C), including the unclamped harmonic mean, so both points can be checked directly in the paper. App. C will also state that in the Overall cohort removing the floor changes only FAPM's Retention and Combined and no ranking, and give the reason for the floor shown above.

Question 1. Why is the nested harmonic mean the appropriate primary ranking criterion?

The metric has two sides. Clean-up covers the damage side (% healed and non-damage), and Retention covers the gain side (% preserved and on-task retention). Two trivial strategies need no repair method, namely reverting fully to Base and keeping the fine-tuned model unchanged. Each one is perfect on one side and zero on the other. The harmonic mean ranks both at zero.

A flat average of the two sides gives the above two strategies a score of 50, the same score as a method that recovers half of the damage and keeps half of the gains (App. C, p. 21, l. 1131 to p. 22, l. 1137). The nesting gives each side one equal vote, and the harmonic mean at each level requires both of its inputs to be high. This is the same reason F1 combines precision and recall with a harmonic mean.

DG-Hard's first place among the non-spectral methods does not depend on this choice. DG-Hard stays first among the non-spectral methods under the geometric mean, the arithmetic mean, and the minimum (Tab. 5, p. 23, ll. 1191-1192, and the table above). It also stays first at every partition threshold from 2 to 6 pp (Tab. 4, p. 22, l. 1173), and under a mean of per-triple ratios in place of the ratio of sums (Tab. 6, p. 23, l. 1206).

Question 2. How well does the method generalize beyond the two language models?

We agree that this is the main open limitation, and the paper states it (Sec. 5, p. 9, l. 473: two models under 5B parameters).

The scope reflects the compute budget of the study. The paper notes that the cutoff was not tracked across more settings "due to limited computational resources" (Sec. 5, p. 9, ll. 474-475), and the size of the evaluation matrix explains why.

For each model, we fine-tune on seven tasks under five training regimes (Full SFT and the four training-time baselines), derive three post-hoc repairs from each Full SFT checkpoint, and evaluate ten spectral controls, which gives 126 fine-tuned or repaired checkpoints per model.

With the two base models, the study evaluates 254 checkpoints. Each one is scored on the nine held-out benchmarks with full-length generation and no max_tokens cap, each fine-tuned or repaired checkpoint is also scored on its target task, and six cells are also scored on the safety benchmarks of App. D.3.

Each additional model family adds another 127 checkpoints to this matrix. With the budget we had, we chose depth on two models from different families (seven tasks, nine benchmarks, ten spectral controls, and safety evaluations) over a broader but shallower set of models.

Reviewer_vFcUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a computationally inexpensive approach to repairing catastrophic forgetting in fine-tuned language models. The approach calculates the difference between the fine-tuned and base model checkpoint’s weights, and performs SVD on each weight-difference matrix. The large singular components are retained, which they hypothesise contain task-relevant information. The smaller components are discarded, which they hypothesise to represent noise contributing to capability degradation. The repaired model is then reconstructed without needing additional training or calibration data. They demonstrate that their closed-form spectral filtering approach can achieve a competitive trade-off between recovery–preservation, without needing additional data or optimisation. The appendix provides a fairly comprehensive set of experiments supporting the contribution.

Critical strengths

• Addresses an important problem in fine-tuning language models. Being able to recover lost capability without retraining or needing access ot the original training data is very useful.
• The proposed method is relatively straightforward and computationally efficient, using a closed-form spectral threshold without requiring training, data, or hyperparamter tuning.
• The authors present a good consideration of the recovery-preservation tradeoff.
• Comprehensive empirical evaluation and ablation studies are provided.

Critical weaknesses
  • Robustness & generalisability of the approach: The results across the 14 model–task combinations are promising, but the experiments use a relatively limited range of models and fine-tuning conditions. Appendix M includes some useful synthetic experiments looking at different signal ranks and noise levels, but I'd still be interested in seeing how the method holds up under different fine-tuning conditions, such as learning rates, training duration, and batch sizes. This would help to establish whether the same spectral structure, and therefore the proposed threshold, would work as well in these other settings.

  • Signal-plus-noise assumption: The additional spectral analyses and ablations are useful and provide good evidence that separating the large and small singular-value components can help recover lost capabilities while preserving fine-tuning gains. However, the paper could be clearer about what the experiments actually demonstrate versus what remains a hypothesis about the underlying mechanism. I'm not fully convinced that the results establish the underlying assumption that the discarded components are IID-like noise from the fine-tuning process. A good fit to the Marchenko–Pastur distribution doesn't necessarily mean the underlying residual is IID noise. Also, the finding that both the spike-only and bulk-only reconstructions recover damaged capabilities suggests that forgetting might be due to interactions between these components, rather than the noise component being harmful on its own.

Questions for authors
  • How sensitive is DG-Hard to different fine-tuning conditions, particularly learning rate, training duration, and batch size? Do you expect these to affect the spectral structure of the updates and the effectiveness of the proposed threshold?
  • Since both the spike-only and bulk-only reconstructions recover damaged capabilities, could forgetting be driven primarily by interactions between these components rather than by the noise residual itself?
4 comments

We thank the reviewer for the careful and accurate summary of our work and for the encouraging assessment of the importance of recovering lost capabilities without retraining or the original data. We also appreciate the recognition of the simplicity and low cost of a closed-form repair that needs no training, data, or hyperparameter tuning, of our treatment of the recovery and preservation trade-off, and of the breadth of the evaluation and ablations.

We are equally grateful for the two weaknesses raised. Both are fair and well argued, and both concern the parts of the paper that we could support least at submission time. They prompted new experiments, which we report below, and the paper is stronger due to them.

Weakness 1 and Question 1. Sensitivity to learning rate, training duration, and batch size.

The submitted paper lists this as a limitation and states that "due to limited computational resources, the cutoff has not been tracked across learning rates, epochs, batch sizes, or larger models" (Sec. 5, p. 9, ll. 474-475). That constraint has not changed, and a sweep over all 14 cells remains out of our reach. The reviewer's request is a fair one, however, so we funded the sweep for this ablation on one model. It required eight new full-parameter fine-tunes and 32 complete evaluations, 10 of them on checkpoints saved during training, which was a considerable expense for us.

We took Llama-3.2-3B on StrategyQA, changed one fine-tuning setting at a time from the paper's configuration (learning rate 1×10−51 \times 10^{-5}, effective batch 64, 3 epochs, seed 42, AdamW), ran a new fine-tune from the Base model for each, and evaluated the fine-tuned and the repaired model on the nine held-out benchmarks and the target task. Every row uses the same evaluation driver, together with fresh Base, fine-tuned, and DG-Hard anchors for the paper setting.

SettingDirections kept by the cutoffEnergy keptHeld-out mean, FT → DG-Hard (Base 56.0)Target task, FT → DG-Hard% healed% preserved
Paper setting1.5%12.3%58.4 → 57.772.1 → 69.184.949.8
Seed 431.6%12.4%58.7 → 57.373.8 → 68.176.745.4
1 epoch1.4%14.9%56.9 → 56.268.7 → 67.274.020.8
6 epochs1.8%9.5%60.2 → 58.571.9 → 69.083.849.5
Learning rate 5×10−65 \times 10^{-6}1.3%8.1%57.0 → 55.670.5 → 65.647.20.6
Learning rate 3×10−53 \times 10^{-5}2.0%25.9%11.2 → 20.972.2 → 72.521.6n/a
Adafactor4.1%27.7%58.2 → 56.373.7 → 70.348.524.9
Effective batch 321.7%10.4%59.2 → 57.672.8 → 69.473.948.2
Effective batch 1281.5%13.0%57.7 → 56.372.6 → 68.687.321.2

% healed and % preserved use the paper's definitions (Sec. 3.4) with the 3 pp partition threshold. "n/a" marks a row with no improved benchmark. Directions kept and energy kept are pooled over all repaired matrices. For the paper setting, the pooled energy equals the value in Tab. 3 (12.3%).

The cutoff keeps a small share of the directions in every setting. In every row the cutoff keeps between 1.3% and 4.1% of the singular directions of the update, pooled over all matrices. As a mean over matrices, Tab. 3 (p. 20) reports 1.7% for the paper's checkpoint of this cell, and the average over the 14 checkpoints is about 3% (Sec. 4.4, p. 8, l. 415). The threshold is computed from each matrix, so it follows the update as the training setting changes, with no adjustment on our side.

Healing is stable across seeds, training duration, and batch size. With a second seed, and at 1, 3, and 6 epochs, DG-Hard heals 74 to 85% of the damage and keeps 92 to 98% of the fine-tuned target-task accuracy. At half and at twice the batch size it heals 74% and 87%.

Epoch by epoch. We also evaluated the checkpoint saved after each epoch of the 6-epoch run, with DG-Hard applied to each one. These are snapshots within one run, where the learning rate has not yet decayed, so they differ from the separate 1-epoch and 3-epoch runs in the table above.

Epoch of the 6-epoch runHeld-out mean, FT → DG-Hard (Base 56.0)Target task, FT → DG-Hard% healed% preserved
157.1 → 56.669.1 → 69.182.434.8
258.8 → 57.273.1 → 69.689.541.5
358.7 → 57.872.5 → 69.0109.650.1
459.7 → 58.471.0 → 68.781.748.6
560.0 → 58.572.2 → 69.691.751.4
660.2 → 58.571.9 → 69.083.849.5

Weakness 1 and Question 1. Sensitivity to learning rate, training duration, and batch size. (continued)

DG-Hard heals 82 to 110% of the damage at every epoch, from the first one on. The share of the incidental gains that it keeps rises from 35% after the first epoch to about 50% from the third epoch on, as the fine-tune accumulates more gains (three improved benchmarks after the first epoch, five from the fourth epoch on).

The learning rate and the optimizer are the sensitive settings. The paper fine-tunes every cell at a peak learning rate of 1×10−51 \times 10^{-5} (Tab. 10, App. D.4, p. 25, l. 1301). At half that rate (5×10−65 \times 10^{-6}) the fine-tuned model has a single damaged benchmark, so there is little to repair, and the repair gives back the incidental gains (% preserved 0.6). At three times that rate (3×10−53 \times 10^{-5}) the fine-tuned model collapses on all nine held-out benchmarks (mean 11.2), and DG-Hard recovers about a fifth of the loss (mean 20.9) while keeping the target task. With Adafactor in place of AdamW the repair heals about half of the damage.

We read these rows as the operating range of the method on this cell. Healing is reliable at the paper's learning rate and optimizer under changes of seed, duration, and batch size. It is weaker when the learning rate moves away from that value in either direction or when the optimizer changes.

The extended and revised PDF will include these tables as a new appendix section, and the limitation sentence in Sec. 5 will state the measured range.

Weakness 2 and Question 2. What the experiments establish about the noise assumption, and whether forgetting comes from the interaction of the two components.

We agree with the reviewer on both points, and we thank them for stating the distinction so clearly.

The fit to the Marchenko-Pastur law is a consistency check. The paper states that "IID entries are our modeling assumption" (Sec. 3.1, p. 4, l. 212) and that the bulk-fit statistic "remains a consistency check and not a test of the IID assumption" (App. A, p. 19, ll. 1014-1015).

The spike-only and bulk-only result. Sec. 4.4 reports that "either one of them, on its own, leaves the damaged benchmarks near base and keeps the target task, and the damage appears only when both are present" (p. 8, ll. 399-400). We agree that this observation does not show the residual to be harmful by itself, but rather the interactions between the two are the likely culprit. We plan to study this in depth to develop a deeper understanding for our future work. The extended and revised PDF will mention this as future work.

What the repair changes in the outputs. The origin of the residual is an assumption, as stated above. The effect of the repair can be observed directly, so we examined the raw completions behind the paper's largest forgetting case, Qwen3.5-4B fine-tuned on MedQA and tested on GSM8K, where accuracy falls from 92.9 to 1.1. No output was truncated and no correct answer was lost in scoring. The fine-tuned model stops reasoning, and only 0.2% of its replies contain a reasoning trace, against 94% for Base. After DG-Hard, 89% of the replies reason again and accuracy returns to 81.6. The same holds on GSM8K in all seven Qwen cells, where every fine-tune lowers the share of replies with a reasoning trace and DG-Hard raises it. In this case the forgetting is a concrete change in model behaviour, and the repair reverses it.

Weakness 2 and Question 2. What the experiments establish about the noise assumption, and whether forgetting comes from the interaction of the two components. (continued)

How the two components combine. To examine the reviewer's question directly, we compared the output distributions of the fine-tuned model, the spike-only model (DG-Hard), and the bulk-only model with those of Base. Base continues each prompt for 16 tokens, and every checkpoint then scores that same token sequence, so the next-token distributions are compared position by position. We used 80 prompts per benchmark on four cells, with the target task and five held-out benchmarks (GSM8K, ARC-Challenge, MMLU, TriviaQA, IFEval). The table reports the KL divergence from Base in nats, averaged over positions and prompts. The same checkpoint scored twice differs by less than 0.003.

CellPromptsFine-tunedSpikes only (DG-Hard)Bulk only
Qwen3.5-4B, MedQATarget task1.721.030.58
Held-out0.450.140.06
Qwen3.5-4B, RTETarget task1.721.250.92
Held-out0.650.330.14
Llama-3.2-3B, MedQATarget task1.411.151.30
Held-out0.420.090.16
Llama-3.2-3B, StrategyQATarget task2.131.461.86
Held-out0.500.110.28

On held-out prompts, each component alone moves the next-token distributions about 2 to 8 times less than the full update, in every cell. On target-task prompts all three remain far from Base, so each component keeps most of the task behaviour. The two components also push in the same direction. On held-out prompts the cosine between their logit shifts is 0.78 to 0.93, and their sum has a cosine of 0.85 to 0.96 with the shift of the full update.

The generated replies show how the two shifts combine. In the same Qwen3.5-4B MedQA case, the spike-only model starts 96.7% of its GSM8K replies with an answer letter, the MedQA answer format, and then goes on to solve the problem (81.6% correct). The bulk-only model keeps Base's opening in most replies and reasons (89.7% correct). The full update makes the model stop after the letter on 94.2% of the items (1.1% correct), against 8.6% for the spike-only model and 2.1% for the bulk-only model.

These results support the reviewer's reading that forgetting comes from the two components together. Each one moves the outputs in the same direction as the full update, and on held-out prompts their combined shift is larger than either one alone.

What is established and what is assumed. Taken together, the experiments establish three observations. The update separates at the Gavish-Donoho cutoff into a few large directions and a broad bulk. The repaired model recovers most of the damaged held-out performance (85.4% healed across the 14 cells). It also keeps most of the incidental gains (78.5% preserved, against 35.5% for the bulk-only variant, p. 8). The reading of the bulk as IID optimization noise is an assumption that motivates the cutoff, and we do not claim that the experiments prove it.

The extended and revised PDF will separate these observations from the modelling assumption in Secs. 1, 3.1, 4.4, and 5, and Sec. 4.4 will include the output-level analysis.

Thanks to the authors for the response and additional experiments. They have strengthened the paper and largely addressed my concerns, although the preservation results are more variable. I will keep my score.

Reviewer_sMtFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies post-hoc repair of catastrophic forgetting using only a base source checkpoint and a finetuned checkpoint. The proposed method, DG-Hard, applies truncated SVD to each fine-tuning weight delta, using the Gavish–Donoho hard threshold to select the retained singular components. This repair requires no calibration data or additional training. The experiments cover two language models, seven fine-tuning tasks and nine held-out benchmarks, with additional safety evaluations.

The main findings include: (1) spectral truncation recovers much of the performance lost (forgetting) during fine-tuning while retaining most target-task accuracy and incidental held-out gains; (2) DG-Hard achieves a better adaptation-retention tradeoff than the reported non-spectral baselines, while tuned fixed-rank and fixed-energy truncation achieve comparable results; (3) the retained spikes preserve more incidental gains than the discarded bulk. The paper also analyzes spectral distributions and layer-wise update energy to support its signal-plus-noise interpretation of fine-tuning updates.

Critical strengths
  1. The method is simple and potentially useful. The repair only requires two checkpoints, without additional training or calibration data. The reported results show a promising adaptation-retention tradeoff. The comparison with several spectral truncation variants also helps assess the practical value of the proposed default threshold.

  2. The spectral interventions provide interesting observations. Both spike-only and bulk-only updates recover much of the damaged performance, but the spikes preserve substantially more incidental gains. The layer-mask experiments further examine which modules contribute to repair. These observations are worth investigating, although their underlying mechanism remains unclear.

Critical weaknesses
  1. Limited methodological novelty. As acknowledged in Section 2, truncated SVD of task vectors already appears in TSV-C. The hard threshold and median-based noise estimation are also directly adopted from Gavish–Donoho. The main technical contribution is therefore applying an existing adaptive truncation rule to forgetting repair. This combination has practical value, but fixed-rank and fixed-energy controls already match or exceed its aggregate performance. Stronger evidence that the adaptive threshold transfers reliably across settings would help establish a more substantial contribution.

  2. The noise interpretation is insufficiently supported. A Marchenko–Pastur-like singular-value distribution does not establish that the discarded component is IID optimization noise or functionally uninformative. The bulk-only model also retains substantial target-task performance and some incidental gains, which complicates this interpretation. The analyses mainly concern weight spectra and benchmark scores, without directly connecting the components to activation or output changes. Moreover, the experiments use AdamW with BF16/FP32 Adam, but do not investigate how optimizer dynamics or numerical precision affect the proposed decomposition. The results support useful spectral filtering more clearly than the claimed explanation of forgetting

Questions for authors

Q1. How sensitive are the spectral bulk, estimated noise scale and repair performance to optimizer, batch size and training precision? Could the authors clarify whether FP32 master weights are maintained, and the precision used for checkpoint storage, delta computation and SVD?

Q2. Could the authors examine activation or logit changes under spike-only and bulk-only updates on target and held-out tasks? This would help explain why both components separately recover performance, while their combination causes forgetting.

Q3. Could the authors compare against a WiSE-FT interpolation sweep and fixed spectral fractions selected on other tasks? Would DG-Hard retain an advantage when these baselines are transferred without checkpoint-specific calibration?

Q4. Have the authors tested an RL (e.g., GRPO or PPO) fine-tuning checkpoint? It would be interesting to check whether useful RL updates remain above the threshold, or whether filtering the bulk removes meaningful gains. This would help clarify the scope of the signal-plus-noise assumption.

Q5. Could the authors clarify the WikiQA evaluation metric and report a majority-class baseline if classification accuracy is used? For severe drops such as Qwen/MedQA on GSM8K, reporting truncation and answer-extraction failure rates would also help distinguish capability loss from changes in output behavior.

8 comments

We thank the reviewer for the precise summary of our work and for a thorough and constructive review. We appreciate the recognition that the repair is simple and potentially useful, since it needs only two checkpoints and no additional training or calibration data, and that the comparison with several spectral truncation variants helps to assess the default threshold. We are also glad that the reviewer found the spike-only, bulk-only, and layer-mask observations worth investigating.

We are equally grateful for the two weaknesses and the five questions. They are well posed, and several of them point at exactly the places where the submitted paper was lacking. Our compute budget is small, as the paper states (Sec. 5, p. 9, ll. 474-475), but the questions deserved to be answered with experiments. We therefore funded 9 new full-parameter fine-tunes (one of them with reinforcement learning) and 34 complete evaluations for this response, which offered insightful results, despite being a considerable expense.

Weakness 1. Methodological novelty, and whether the adaptive threshold transfers.

We agree with the description of the method. The paper credits truncated SVD of task vectors to TSV-C and the threshold to Gavish and Donoho (Sec. 1, p. 2, ll. 86-88), and it reports that tuned fixed-rank and fixed-energy truncation match DG-Hard (Sec. 4.4, p. 8, ll. 410-421; Sec. 5, p. 9, ll. 470-473). The contribution we claim is the closed-form, per-matrix cutoff in this repair setting (Sec. 1, p. 2, ll. 87-88) and what the experiments show in it, namely a forgetting repair from two checkpoints with no data and no selected/tuned fraction, and the recovery and preservation it reaches across 14 cells.

Known techniques have repeatedly carried new value through the objective they serve and the understanding they yield. Magnitude pruning, an established compression heuristic, was reused by the Lottery Ticket Hypothesis (Frankle and Carbin, ICLR 2019) to reveal trainable sparse subnetworks within dense ones. In the same way, truncated SVD of a task vector, a compression tool in TSV-C, serves a different objective here, which is the repair of forgetting after a single fine-tune.

The reviewer asks for evidence that the adaptive threshold transfers across settings. We add two pieces.

Across training settings. For one model (Llama-3.2-3B on StrategyQA) we retrained with one setting changed at a time. The cutoff keeps 1.3 to 4.1% of the directions in all nine settings, and DG-Hard heals 74 to 87% of the damage across seeds, training durations, and batch sizes. It is weaker when the learning rate or the optimizer changes. The full table is given below, under Weakness 2 and Question 1. This cell (model, task) was selected randomly.

Against fractions selected on other tasks. This is the reviewer's Question 3, and we answer it in full below. In short, a fixed fraction transferred from other tasks reaches a similar average and has a much lower worst-cell score than the closed-form cutoff.

Weakness 2 and Question 1. The noise interpretation, and sensitivity to the training settings.

Precision used in the paper. The fine-tunes kept no FP32 master weights. The model weights were BF16 throughout training, and checkpoints are stored in BF16. The "fp32 Adam" entry of Tab. 10 refers to the optimizer arithmetic. The fused AdamW kernel computes each update in FP32 and stores the weights and its moment estimates in BF16. The delta and its SVD are computed in FP32. The repaired weights are stored and evaluated in BF16. The extended and revised PDF will list the precision of each of these stages separately in Tab. 10 (App. D.4, p. 25, l. 1308).

New runs. We retrained Llama-3.2-3B on StrategyQA with one setting changed at a time, and evaluated every fine-tuned and repaired model on the nine held-out benchmarks and the target task, in one batch with fresh anchors. The paper setting is AdamW, learning rate 1×10−51 \times 10^{-5}, effective batch 64, 3 epochs, seed 42.

SettingDirections keptEnergy keptHeld-out mean, FT → DG-Hard (Base 56.0)Target task, FT → DG-Hard% healed% preserved
Paper setting1.5%12.3%58.4 → 57.772.1 → 69.184.949.8
Seed 431.6%12.4%58.7 → 57.373.8 → 68.176.745.4
1 epoch1.4%14.9%56.9 → 56.268.7 → 67.274.020.8
6 epochs1.8%9.5%60.2 → 58.571.9 → 69.083.849.5
Learning rate 5×10−65 \times 10^{-6}1.3%8.1%57.0 → 55.670.5 → 65.647.20.6
Learning rate 3×10−53 \times 10^{-5}2.0%25.9%11.2 → 20.972.2 → 72.521.6n/a
Adafactor4.1%27.7%58.2 → 56.373.7 → 70.348.524.9
Effective batch 321.7%10.4%59.2 → 57.672.8 → 69.473.948.2
Effective batch 1281.5%13.0%57.7 → 56.372.6 → 68.687.321.2

% healed and % preserved use the paper's definitions (Sec. 3.4) with the 3 pp partition threshold. "n/a" marks a row with no improved benchmark. Directions kept and energy kept are pooled over all repaired matrices. For the paper setting, the pooled energy equals the value in Tab. 3 (12.3%).

Optimizer, batch size, seed, duration, learning rate. The bulk is present in every setting, and the repair works best at the paper's fine-tuning learning rate (1×10−51 \times 10^{-5}). It heals 74 to 87% of the damage across seeds, 1 to 6 epochs, and batch sizes of 32 to 128. It heals about half with Adafactor or at half the learning rate (5×10−65 \times 10^{-6}), and about a fifth at three times the learning rate (3×10−53 \times 10^{-5}), where the fine-tuned model collapses on all nine benchmarks.

Why the repair weakens in three settings. The repair removes the part of the update below the cutoff. Its effect therefore depends on two quantities in the table, namely how much damage the fine-tune caused and how much of the update lies below the cutoff. Seed, duration, and batch size change neither. The cutoff removes 85 to 91% of the update energy in those six rows, and healing is stable.

At three times the learning rate (3×10−53 \times 10^{-5}) the damage sits in the large directions. The share of energy above the cutoff doubles to 25.9%, and the repaired model, which holds only those directions, is still collapsed (held-out mean 20.9). A truncation at the cutoff keeps the dominant directions by design, so it cannot repair damage that they carry.

At half the learning rate (5×10−65 \times 10^{-6}) there is little to repair. The held-out mean rises with fine-tuning (56.0 to 57.0) and a single benchmark falls, by 3.1 points, so % healed rests on a very small loss. The update is also weak relative to the cutoff (8.1% of the energy above it, the lowest value in the sweep), and the repaired model returns close to Base (55.6) and gives back the incidental gains. This is the regime that the paper lists as a limitation (Sec. 5, p. 9, ll. 468-469).

With Adafactor the damage is also small (two benchmarks, by 3.3 and 4.8 points), and the spectrum itself changes. More directions lie above the cutoff (4.1% against 1.5%) and they hold 27.7% of the energy, so the truncation removes less of the update and heals about half of the damage.

The extended and revised PDF will include this table as a new appendix section, and the limitation sentence in Sec. 5 will state the measured range.

Weakness 2 and Question 1. The noise interpretation, and sensitivity to the training settings. (continued)

Hypothesis and future work. The repair deletes the directions below the cutoff and keeps the others unchanged, and the paper shows that this selection does more than reduce the size of the update. DG-Hard keeps less of the update than WiSE-FT (11% of the energy against 25%) and preserves more of the incidental gains, and at comparable rollback it leads uniform interpolation by 22 points (Sec. 4.4, p. 8, ll. 400-403). The sweep shows where one cutoff per matrix reaches its limit. At a high learning rate the damage is carried by directions above the cutoff, so the cutoff keeps it. At a low learning rate part of the task gain lies below the cutoff, so the cutoff deletes it. In both cases one boundary per matrix cannot place the damage on one side and the gain on the other. We hypothesize that the damage and the gain are carried by largely different directions of the update, also where one boundary cannot separate them. A repair that identifies the directions carrying the damage and deletes them would then remove the damage without giving up part of the gain, which a single cutoff does in these settings. The test we propose is a finer learning-rate sweep on several models and tasks that measures the effect of individual directions of each update on the target task and on held-out benchmarks, and checks whether the two effects separate. The extended and revised PDF will state this hypothesis and the proposed test as future work.

The noise interpretation. We agree with the reviewer on each point, and we thank them for stating the concern so precisely.

The fit to the Marchenko-Pastur law is a consistency check. The paper states that "IID entries are our modeling assumption" (Sec. 3.1, p. 4, l. 212) and that the bulk-fit statistic "remains a consistency check and not a test of the IID assumption" (App. A, p. 19, ll. 1014-1015). It also notes that "a task signal spread thinly inside the noise bulk would be lost" (Sec. 5, p. 9, ll. 468-469), so the paper does not treat the discarded component as empty by construction. The attribution of the bulk to mini-batch gradient noise (Sec. 1, p. 1, l. 53) is a hypothesis that motivates the cutoff, and the paper does not test it.

The bulk-only result. The reviewer notes that the bulk-only model keeps substantial target-task performance and some incidental gains. The paper reports this result in Sec. 4.4 (p. 8), where the bulk-only variant preserves 35.5% of the gains against 78.5% for DG-Hard. We agree that it complicates a pure noise reading. How the two components interact is a question that we plan to study in depth in future work. The extended and revised PDF will include this as future work.

Optimizer dynamics and output changes. The split into a few large directions and a broad bulk is present in all nine settings of the sweep above, with 1.3 to 4.1% of the directions above the cutoff. The output-level analysis under Question 2 connects the repair to a concrete change in model behaviour.

What is established and what is assumed. The experiments establish three observations. The update separates at the Gavish-Donoho cutoff into a few large directions and a broad bulk. The repaired model recovers most of the damaged held-out performance (85.4% healed across the 14 cells). It also keeps most of the incidental gains (78.5% preserved). The reading of the bulk as IID optimization noise is an assumption that motivates the cutoff, and we do not claim that the experiments prove it. The extended and revised PDF will separate these observations from the modelling assumption in Secs. 1, 3.1, 4.4, and 5. This agrees with the reviewer's reading that the results support spectral filtering more clearly than an explanation of forgetting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.