- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether changes to previously acquired capabilities can be predicted before fine-tuning. It expresses old-task logit drift through an exact gradient-flow identity involving the cross-task tangent kernel, then develops local predictors evaluated at the pre-adaptation model. Extensions incorporate cross-entropy Gauss–Newton curvature and an optimizer-dependent metric. Experiments investigate spectral concentration, capability ranking across language-model families, prediction decay during adaptation, and protection through an existing spectral output penalty. The strongest empirical result is prospective ranking of short-horizon, signed cross-entropy changes; the paper explicitly distinguishes this from predicting absolute forgetting magnitude.
Critical strengths
- Do with substantial cross-model evaluation. Predicting vulnerable capabilities before adaptation is useful.
- Clear separation of exact identities and empirical observations.
Critical weaknesses
-
Section 6.3 defines the transport score as . For predicting a loss increase, its normalized counterpart is negative cosine similarity. Under a small SGD step, positive gradient alignment predicts a loss decrease. Consequently, the negative correlations of the unflipped cosine baseline in Figure 3 and Table 7 do not, by themselves, show that normalization destroys useful information or reverses the correct ordering. The sign-aligned baseline should appear in the main ranking comparison, with paired uncertainty on the remaining advantage.
-
The validity-horizon argument does not establish forecast accuracy. Lemma 1 bounds how much realized drift leaves an initially selected eigenspace. It does not bound the difference between predicted and realized drift, nor errors in capability rankings. Drift may remain entirely within a fixed subspace while its direction and relative components change substantially. Even with constant kernels, evolution of the task-B loss gradient can invalidate the initial raw-transport forecast. Thus, the small instantaneous tail and large subspace rotation measured in the vision experiment do not establish that rotation is the main cause of forecast expiry, especially for the language-model horizon results. The paper should distinguish subspace capture from predictive accuracy and separate changes in geometry from changes in forcing. There is also a problem in Proposition 3: the displayed spectral factors are correct, but their multiplication by an arbitrary does not guarantee that the full lazy operator has a flatter spectrum or larger stable rank. The observed stable-rank ordering should be presented as empirical unless additional assumptions are supplied.
-
The incremental contribution needs sharper positioning. The function-space interpretation is useful, but the distinction from prior work is too categorical. [1] already formulate output drift and a residual-dependent spectral expression. The cited [2] explicitly develop a pre-adaptation output-drift predictor and old-task eigenspace interpretation, beyond the spectral penalty credited here. The scalar score also has the acknowledged first-order gradient-product form of TracIn. The potentially new value therefore lies primarily in the CE extension, optimizer-specific empirical controls, and prospective cross-family evaluation. These extensions should be distinguished explicitly from the inherited predictor and structural interpretation. The experiments themselves suggest that the new curvature component contributes only modestly in frozen vision and little in the principal language-model results. A further relevant connection is [3], which compares unlearning locally grokked and ungrokked examples in Phi-1.5 and relates selective forgetting to reduced forget–retain gradient alignment. This is relevant to the broader interpretation of representation geometry and interference. The distinction is important: that work selects example groups using loss histories and evaluates intentional unlearning, whereas this submission predicts signed collateral changes from a specified incoming fine-tune using the current checkpoint. Discussing this connection would sharpen the conceptual positioning; it does not establish equivalence of the methods.
-
The paper omit essential experimental details, learning rates, batch sizes, probe‑sample counts, MMLU subject splits. This is especially problematic because the paper’s main claim concerns short‑horizon adaptation, yet no corresponding analysis of time complexity, space complexity, or actual runtime/memory costs is provided. A quantitative comparison across the raw score, the curvature‑aware forecast, and adaptation‑plus‑evaluation is necessary, particularly since the raw‑score method nearly matches the full forecast at scale.
[1] A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (AISTATS, 2021)
[2] Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation (ICML workshop, 2026)
[3] Grokked Models are Better Unlearners (UAI, 2026)
Questions for authors
see weakness.
If the autho can answer my question, I will improve my score.