- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a post-hoc approach for mitigating catastrophic forgetting using the pretrained and fine-tuned checkpoints. Given the weights of the pretrained model and the fine-tuned one, they model their differences as signal + noise, then using the optimal hard threshold for singular values, they trim the spectrum of the difference to remove the noise.
The authors evaluate the approach on Qwen3.5-4B and Llama-3.2-3B-Instruct across seven fine-tuning setups and nine held-out benchmarks. Their method reports a favorable balance between restoring degraded base capabilities and preserving fine-tuning gains.
Critical strengths
-
The idea of modeling the forgetting problem as removing noise from signal in the spectrum is interesting and seems to be novel to me. This also justifies the spectral analysis and the correction story.
-
The algorithm is extremely simple, you just need the two checkpoints that you usually have and no additional data or hyper-parameter tuning.
-
The experimental part is strong. Evaluating on two language models, seven fine-tuning tasks, and nine benchmarks. The results are pretty consistent, and the method seems to work.
Critical weaknesses
-
It seems that the results are not as strong as the paper frame. DG-Hard doesn't beat the other methods on all metrics, V-SoftMask is better in preservation and on-task retention, and FAPM is better in healing and non-damage. DG-Hard gives a reasonable balance between them, but I don't think this necessarily means it is a better method.
-
The evaluation metric seems strange. Harmonic mean puts a lot of weight on having balanced results, which is where DG-Hard has the advantage. Another thing I find questionable is setting negative scores to zero. I don't see a clear reason for doing this, and it might affect the comparison with the baselines. I would like to see the results without this clamping and with the Pareto trade-offs.
Questions for authors
See weaknesses, plus:
-
Can the authors explain why the nested harmonic mean is the appropriate primary ranking criterion here?
-
How well does the method generalize beyond the two language models considered here? I would be interested in seeing results on additional model families and scales, since the spectral structure of fine-tuning updates may depend strongly on architecture and optimization regime.
6 comments
Weakness 2. The harmonic mean favours balance, negative scores are set to zero, and the reviewer asks for unclamped results and Pareto trade-offs.
2a. Does the harmonic mean create DG-Hard's lead, and why did we choose the harmonic mean?
DG-Hard's lead among the non-spectral methods holds under other aggregators, including one that gives balance no extra weight. The table below scores all seven methods on the Overall cohort. Tab. 5 (App. C, p. 23, from l. 1189) already reports the four strongest non-spectral methods under these aggregators, and the table below adds the remaining three and the unclamped column. The arithmetic mean of the four sub-statistics gives no extra weight to balance, and DG-Hard is first among the non-spectral methods under it (86.6 against 83.0 for WiSE-FT). The order of all seven non-spectral methods is the same in every column, including the geometric mean.
| Method | Combined (paper) | Combined, no floor | Geometric mean | Arithmetic mean | Minimum |
|---|---|---|---|---|---|
| DG-Hard | 86.1 | 86.1 | 86.4 | 86.6 | 78.5 |
| WiSE-FT | 80.1 | 80.1 | 81.6 | 83.0 | 59.4 |
| L1-reg | 72.0 | 72.0 | 76.0 | 79.4 | 44.5 |
| V-SoftMask | 54.2 | 54.2 | 68.2 | 79.1 | 23.2 |
| CoFi-Tune | 42.7 | 42.7 | 57.5 | 69.3 | 17.0 |
| LoRA | 20.0 | 20.0 | 45.4 | 68.4 | 6.0 |
| FAPM | 0.0 | −20.6 | 0.0* | 66.1 | −4.4 |
| Tuned spectral controls (§4.4) | |||||
| Fixed energy-percentage 15% (tuned) | 87.9 | 87.9 | 88.1 | 88.3 | 81.6 |
| Fixed-rank 5% (tuned) | 86.9 | 86.9 | 87.1 | 87.4 | 79.6 |
Bold marks the best non-spectral method in each column.
* The geometric mean is undefined for a negative input, so FAPM's negative % preserved is floored at 0 in that column.
The tuned spectral controls behave the same way under every aggregator. The lower block repeats the two tuned controls from Tab. 5. Once their retained fraction is tuned against old-task performance, they reach or slightly exceed DG-Hard under every aggregator, as they do under the harmonic mean (App. C, p. 22, ll. 1166-1167). Neither has a negative sub-statistic, so the floor never applies to them. DG-Hard reaches the same level with a closed-form cutoff and no fraction to select (Sec. 4.4).
The submitted paper already addresses this concern, and tests the Combined score against three of its design choices, namely the partition threshold (Sec. 3.4, p. 5, l. 269; Tab. 4), the aggregator (Tab. 5), and the ratio-of-sums definition (Sec. 3.4, p. 6, ll. 296-297; Tab. 6). DG-Hard is first among the non-spectral methods in every one of these checks.
The ratio of sums does not create the lead either. % healed and % preserved add up the gains and losses over all damaged or improved triples before dividing. Tab. 6 (App. C, p. 23, from l. 1209) recomputes Combined with a mean of per-triple ratios, which gives every triple equal weight. DG-Hard is first among the non-spectral methods under both definitions (86.1 and 83.3, against 80.1 and 76.4 for WiSE-FT), and the only change in order is that V-SoftMask and CoFi-Tune exchange fourth and fifth place (p. 23, ll. 1206-1207).
Why we chose the harmonic mean. The choice follows from what a repair has to do, which we described under Weakness 1.
A repair has to heal the damage and keep the gains at the same time, and each side alone is available without any method. Reverting to Base heals everything, and keeping the fine-tuned model preserves everything.
A ranking score should therefore place both of these trivial strategies at the bottom. The harmonic mean does this, because it is pulled toward its smaller input. A method that scores near zero on one side scores near zero overall, however strong its other side is. A flat average of the two sides does the opposite and gives each trivial strategy 50, the same score as a method that recovers half of the damage and keeps half of the gains (App. C, p. 21, l. 1131 to p. 22, l. 1137).
The nesting applies the same rule at two levels. Within each side, both of its sub-statistics must be high, and across the two sides, both sides must be high. The reviewer is right that this rewards balance, and that is its purpose. The table above shows that DG-Hard's first place among the non-spectral methods does not depend on it.
2b. Clamping of negative scores, unclamped results, and Pareto trade-offs
In the Overall cohort the clamping changes only FAPM's scores and does not change any rankings. The "Combined, no floor" column of the table in 2a gives the unclamped results. The extended and revised PDF will add this column to Tab. 5. Every reported sub-statistic is uncapped and unfloored (App. C, p. 22, ll. 1185-1187), and Tab. 1 prints FAPM's % preserved of −4.4.
Only the inputs to the harmonic mean are floored at 0 (Sec. 3.4, p. 6, ll. 277-278). On the Overall cohort the floor applies only to FAPM. Without it, FAPM's Retention becomes −9.3 and its Combined −20.6, and FAPM is last either way. Across the four sub-cohorts the floor applies five more times (LoRA and FAPM on Cognition and on Reasoning, V-SoftMask on Non-reasoning), and in each case the method is in the bottom two of its cohort with or without the floor.
We floor the inputs because the harmonic mean is not monotone when its inputs have opposite signs. For example, , which is higher than . Without the floor, a method could raise its score by doing more damage on one axis. With the floor, a negative sub-statistic always counts as zero.
Pareto view. Among the seven non-spectral methods, five are non-dominated on the four unclamped sub-statistics, namely DG-Hard, WiSE-FT, L1-reg, V-SoftMask, and FAPM. CoFi-Tune and LoRA are dominated by L1-reg. Fig. 4 (App. I) plots the Clean-up vs Retention frontier and the Knowledge vs Cognition frontier, and in both panels DG-Hard is the non-spectral method closest to the upper-right corner. Fig. 1 plots % healed against % preserved per cohort. Membership in the non-dominated set does not rank the five methods by itself, and every aggregator in the table in 2a selects DG-Hard among them.
The extended and revised PDF will report all seven methods under every aggregator in Tab. 5 (App. C), including the unclamped harmonic mean, so both points can be checked directly in the paper. App. C will also state that in the Overall cohort removing the floor changes only FAPM's Retention and Combined and no ranking, and give the reason for the floor shown above.
Question 1. Why is the nested harmonic mean the appropriate primary ranking criterion?
The metric has two sides. Clean-up covers the damage side (% healed and non-damage), and Retention covers the gain side (% preserved and on-task retention). Two trivial strategies need no repair method, namely reverting fully to Base and keeping the fine-tuned model unchanged. Each one is perfect on one side and zero on the other. The harmonic mean ranks both at zero.
A flat average of the two sides gives the above two strategies a score of 50, the same score as a method that recovers half of the damage and keeps half of the gains (App. C, p. 21, l. 1131 to p. 22, l. 1137). The nesting gives each side one equal vote, and the harmonic mean at each level requires both of its inputs to be high. This is the same reason F1 combines precision and recall with a harmonic mean.
DG-Hard's first place among the non-spectral methods does not depend on this choice. DG-Hard stays first among the non-spectral methods under the geometric mean, the arithmetic mean, and the minimum (Tab. 5, p. 23, ll. 1191-1192, and the table above). It also stays first at every partition threshold from 2 to 6 pp (Tab. 4, p. 22, l. 1173), and under a mean of per-triple ratios in place of the ratio of sums (Tab. 6, p. 23, l. 1206).
Question 2. How well does the method generalize beyond the two language models?
We agree that this is the main open limitation, and the paper states it (Sec. 5, p. 9, l. 473: two models under 5B parameters).
The scope reflects the compute budget of the study. The paper notes that the cutoff was not tracked across more settings "due to limited computational resources" (Sec. 5, p. 9, ll. 474-475), and the size of the evaluation matrix explains why.
For each model, we fine-tune on seven tasks under five training regimes (Full SFT and the four training-time baselines), derive three post-hoc repairs from each Full SFT checkpoint, and evaluate ten spectral controls, which gives 126 fine-tuned or repaired checkpoints per model.
With the two base models, the study evaluates 254 checkpoints. Each one is scored on the nine held-out benchmarks with full-length generation and no max_tokens cap, each fine-tuned or repaired checkpoint is also scored on its target task, and six cells are also scored on the safety benchmarks of App. D.3.
Each additional model family adds another 127 checkpoints to this matrix. With the budget we had, we chose depth on two models from different families (seven tasks, nine benchmarks, ten spectral controls, and safety evaluations) over a broader but shallower set of models.
We thank the reviewer for the careful reading, and for finding the signal-plus-noise view of forgetting novel, the algorithm simple, and the experiments strong and consistent. We answer the two weaknesses and the two questions below.
Weakness 1. DG-Hard does not beat the other methods on every metric.
We agree with the observation, and the paper makes no claim that DG-Hard leads every sub-statistic. The claim is that, among the non-spectral methods, DG-Hard gives the best balance across them, measured by a score built to reward balance. The paper states its ranking claim on that score in the abstract (p. 1, ll. 27-28), contribution 3 (p. 2, ll. 95-96), the opening of Sec. 4 (p. 6, ll. 303-304), and the conclusion (p. 9, l. 483). Sec. 3.4 defines the score as "demoting methods that win one axis by sacrificing another" (p. 6, ll. 294-295), and App. G says it "is designed precisely to demote single-axis extremes" (p. 29, ll. 1540-1541).
We built the score this way because of what a repair has to do. It has to heal the damaged capabilities, leave the untouched ones alone, keep the incidental held-out gains, and keep the target-task gain, all at the same time.
A method that heals most of the damage but discards the incidental and target-task gains has undone the fine-tuning it was meant to repair. A method that keeps every gain but heals little has left the forgetting in place. Both extremes are available without any method. Reverting to Base heals everything, and keeping the fine-tuned model preserves everything.
In our view, a lead on one sub-statistic therefore says little about whether a method repairs forgetting, and the sub-statistics have to be read together, which our evaluation metric attempts to do.
Tab. 1 shows this split. FAPM, WiSE-FT, L1-reg, and LoRA each have a higher Clean-up than DG-Hard, and V-SoftMask has a higher Retention. DG-Hard is the only non-spectral method that scores 80 or more on both (Tab. 1 caption, p. 7, ll. 325-326), and among the non-spectral methods it is the one closest to the ideal corner of % healed and % preserved in all five cohorts (Fig. 1 caption, p. 3, ll. 121-122). Three further observations support the balance claim without relying on the Combined score.
No non-spectral method dominates DG-Hard. On the four sub-statistics of Tab. 1, no baseline is at least as good as DG-Hard on all four, and each sub-statistic leader pays on another axis. FAPM heals the most damage (93.6%) but keeps none of the incidental gains (% preserved −4.4) and only 77.8% of the on-task accuracy. LoRA also heals more than DG-Hard (87.3%) and keeps 6.0% of the incidental gains. V-SoftMask keeps the most gains (95.8%) and heals 23.2% of the damage. Fig. 4 names FAPM and V-SoftMask the cleanup and retention extremes (caption, p. 31, from l. 1633).
Among the non-spectral methods, DG-Hard has the strongest weakest axis. Its lowest sub-statistic is 78.5. The next best non-spectral method's lowest is 59.4 (WiSE-FT), and every other one falls to 44.5 or below (Tab. 5, "Minimum" column, and Tab. 1).
The flat average agrees. Mean held-out accuracy involves no metric design choice. DG-Hard is highest among the non-spectral methods at 64.4, ahead of V-SoftMask (63.8), the fine-tuned model (63.3), and WiSE-FT (63.2), while keeping on-task accuracy at 81.6 against 83.7 for the fine-tuned model (Tab. 1).
We address the choice of the harmonic mean itself under Weakness 2 and Question 1.
We thank the reviewer for pointing out that this scope can be read more strongly than intended. The extended and revised PDF will state it directly in Sec. 4. DG-Hard leads none of the four sub-statistics on its own, and it is the only non-spectral method that scores 80 or more on both Clean-up and Retention.