acceptodds
Under review as a conference paper at ICLR 2027

Tangent Geometry Structures Fine-Tuning Forgetting

Abstract

Fine-tuning can degrade capabilities that a model acquired before adaptation, but it is usually unclear before training which capabilities will be affected. We study this problem in the model's local tangent space and show that forgetting has a structured geometry there. Under gradient flow, the change in old-task outputs admits an exact decomposition: the new-task loss gradient is transported into the old-task function space through the cross-task tangent kernel . This identity separates the forcing induced by the incoming task from the geometry through which it affects existing outputs. In small-output regimes, the resulting forgetting drift is strongly anisotropic and concentrates along leading eigendirections of the old-task kernel . This structure suggests a pre-adaptation diagnostic: evaluate the tangent geometry once, before any parameter update, and use it to rank which outputs or capabilities are most likely to move. We test this prediction across vision models and four language-model families. For LoRA fine-tuning of language models, the resulting unnormalised cross-task gradient score ranks held-out capability changes before training, including on three model families not used during development. In contrast, cosine-normalised gradient similarity is negatively correlated with the same realised changes across all tested configurations, indicating that the gradient magnitude removed by normalization contains information relevant to interference. The tangent-space formulation also predicts an optimizer dependence: incorporating the AdamW preconditioner improves the one-step prediction under AdamW but not under SGD, as confirmed by permutation controls. For frozen representations, we further derive a Gauss–Newton correction for cross-entropy. The curvature-aware operator predicts realised output drift better than a curvature-free predictor even when the latter is given an oracle scalar ridge parameter. When representations are allowed to change, the tangent kernel rotates and the quality of a pre-adaptation predictor degrades with training, giving it a measurable validity horizon. Finally, constraining the vulnerable eigenspace identified by the tangent geometry reduces forgetting in vision models and cross-entropy degradation in language models. Our results characterize forgetting as a structured, locally predictable phenomenon in function space. The proposed diagnostic predicts relative vulnerability rather than forgetting magnitude, and is most reliable over short adaptations for which the local tangent geometry remains informative.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic forgetting, fine-tuning, neural tangent kernel, LoRA, parameter-efficient fine-tuning, continual learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_uZgSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether changes to previously acquired capabilities can be predicted before fine-tuning. It expresses old-task logit drift through an exact gradient-flow identity involving the cross-task tangent kernel, then develops local predictors evaluated at the pre-adaptation model. Extensions incorporate cross-entropy Gauss–Newton curvature and an optimizer-dependent metric. Experiments investigate spectral concentration, capability ranking across language-model families, prediction decay during adaptation, and protection through an existing spectral output penalty. The strongest empirical result is prospective ranking of short-horizon, signed cross-entropy changes; the paper explicitly distinguishes this from predicting absolute forgetting magnitude.

Critical strengths
  1. Do with substantial cross-model evaluation. Predicting vulnerable capabilities before adaptation is useful.
  2. Clear separation of exact identities and empirical observations.
Critical weaknesses
  1. Section 6.3 defines the transport score as si=−⟨∇θLAi,∇θLB⟩s_i=-\langle\nabla_\theta L_{A_i},\nabla_\theta L_B\rangle. For predicting a loss increase, its normalized counterpart is negative cosine similarity. Under a small SGD step, positive gradient alignment predicts a loss decrease. Consequently, the negative correlations of the unflipped cosine baseline in Figure 3 and Table 7 do not, by themselves, show that normalization destroys useful information or reverses the correct ordering. The sign-aligned baseline should appear in the main ranking comparison, with paired uncertainty on the remaining advantage.

  2. The validity-horizon argument does not establish forecast accuracy. Lemma 1 bounds how much realized drift leaves an initially selected eigenspace. It does not bound the difference between predicted and realized drift, nor errors in capability rankings. Drift may remain entirely within a fixed subspace while its direction and relative components change substantially. Even with constant kernels, evolution of the task-B loss gradient can invalidate the initial raw-transport forecast. Thus, the small instantaneous tail and large subspace rotation measured in the vision experiment do not establish that rotation is the main cause of forecast expiry, especially for the language-model horizon results. The paper should distinguish subspace capture from predictive accuracy and separate changes in geometry from changes in forcing. There is also a problem in Proposition 3: the displayed spectral factors are correct, but their multiplication by an arbitrary JAVJ_AV ​ does not guarantee that the full lazy operator has a flatter spectrum or larger stable rank. The observed stable-rank ordering should be presented as empirical unless additional assumptions are supplied.

  3. The incremental contribution needs sharper positioning. The function-space interpretation is useful, but the distinction from prior work is too categorical. [1] already formulate output drift and a residual-dependent spectral expression. The cited [2] explicitly develop a pre-adaptation output-drift predictor and old-task eigenspace interpretation, beyond the spectral penalty credited here. The scalar score also has the acknowledged first-order gradient-product form of TracIn. The potentially new value therefore lies primarily in the CE extension, optimizer-specific empirical controls, and prospective cross-family evaluation. These extensions should be distinguished explicitly from the inherited predictor and structural interpretation. The experiments themselves suggest that the new curvature component contributes only modestly in frozen vision and little in the principal language-model results. A further relevant connection is [3], which compares unlearning locally grokked and ungrokked examples in Phi-1.5 and relates selective forgetting to reduced forget–retain gradient alignment. This is relevant to the broader interpretation of representation geometry and interference. The distinction is important: that work selects example groups using loss histories and evaluates intentional unlearning, whereas this submission predicts signed collateral changes from a specified incoming fine-tune using the current checkpoint. Discussing this connection would sharpen the conceptual positioning; it does not establish equivalence of the methods.

  4. The paper omit essential experimental details, learning rates, batch sizes, probe‑sample counts, MMLU subject splits. This is especially problematic because the paper’s main claim concerns short‑horizon adaptation, yet no corresponding analysis of time complexity, space complexity, or actual runtime/memory costs is provided. A quantitative comparison across the raw score, the curvature‑aware forecast, and adaptation‑plus‑evaluation is necessary, particularly since the raw‑score method nearly matches the full forecast at scale.

[1] A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (AISTATS, 2021)

[2] Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation (ICML workshop, 2026)

[3] Grokked Models are Better Unlearners (UAI, 2026)

Questions for authors

see weakness.

If the autho can answer my question, I will improve my score.

Reviewer_5SfpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether changes to existing model capabilities can be predicted before fine tuning begins. The authors formulate forgetting from a function space perspective, using the cross task tangent kernel to connect gradients from the incoming task with changes in existing outputs. Based on this formulation, they develop a predictor of capability vulnerability and analyze the effects of curvature, optimizer preconditioning, and kernel evolution. Experiments on vision models and four language model families demonstrate promising results for predicting early capability changes, while spectral regularization experiments further show that protecting selected geometric directions can reduce forgetting.

Critical strengths
  1. The paper provides an interesting and relatively novel perspective on forgetting by studying interference in function space. The tangent kernel connecting different tasks provides a principled basis for predicting capability vulnerability before adaptation. The paper also makes a valuable effort to connect its theoretical analysis with empirical observations and forgetting mitigation. In particular, the improvement from Adam preconditioning under AdamW provides convincing support for the transport interpretation.

  2. The empirical evaluation provides strong evidence for predicting early capability changes, with encouraging generalization across different model families.

Critical weaknesses
  1. The forgetting mitigation experiments lack comparisons with recent LoRA methods such as KeepLoRA[1] and EWC-LoRA[2]. KeepLoRA protects existing knowledge through constrained updates without requiring old task samples, while EWC-LoRA uses parameter importance estimates to regularize shared LoRA updates with storage and inference costs that remain constant across tasks. These methods are particularly relevant since the proposed spectral intervention requires stored anchors and additional geometric computations.

  2. The main language model experiments evaluate capability changes after only four optimization steps of LoRA, where many capabilities actually improve rather than deteriorate. This raises concerns about whether the reported results demonstrate the prediction of substantial forgetting or mainly capture small local loss changes. Prediction performance also decreases considerably with longer adaptation, and full parameter fine tuning of language models is not systematically evaluated.

[1] KeepLoRA: Continual Learning with Residual Gradient Adaptation. ICLR 2026

[2] Revisiting Weight Regularization for Low-Rank Continual Learning. ICLR 2026

Questions for authors

Could the authors provide a more detailed comparison of the computational cost of tangent-geometry estimation and direct fine-tuning? In particular, under what conditions could the cost of the proposed diagnostic be amortized across multiple candidate adaptations?

Reviewer_6GScOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether fine-tuning-induced changes to existing capabilities can be predicted before adaptation. It develops a vulnerability score from cross-task tangent-kernel transport and examines curvature, optimizer preconditioning, and kernel movement. Vision and language-model experiments evaluate ranking accuracy, forecast validity, and protection of vulnerable directions.

Critical strengths
  1. The paper turns a chain-rule identity into a testable prospective prediction task. At four steps, mean Spearman is 0.776 across 90 evaluation cells (18 configurations × five seeds), positive in 88/90, including held-out model families. Precision@5 is 0.711 versus 0.25 chance.

  2. The controls and limitations are carefully presented. Optimizer-specific permutations and matched-rank subspace comparisons support the interpretation. The measured validity horizon is informative: MMLU correlation falls from 0.907 at one step to 0.184 at sixteen steps.

Critical weaknesses
  1. Novelty and efficiency remain concerns. At language-model scale, the score resembles a signed inner product of old- and new-task gradients, close to existing interference measures. Evaluation can cost more than the short fine-tune being predicted, yet no direct one- or two-step adaptation probe is compared. The curvature benefit is modest and mainly isolated in frozen-head tests.

  2. Some interpretations need care. The contrast between positive transport correlations and negative cosine correlations appears partly attributable to sign convention; sign-corrected cosine is the clearer baseline (precision@5 0.536 versus 0.711). Protection mainly reduces cross-entropy deterioration rather than improving accuracy, while full-output anchoring already removes 75.2% of damage versus 79.8% for the top eigenspace.

Questions for authors
  1. How do forecast cost and ranking accuracy compare with a direct one- or two-step adaptation probe under the same evaluation budget?

  2. Can the authors clarify the cosine sign convention, report sign-corrected baselines, and account for shared configurations when estimating uncertainty?

Reviewer_UCMiOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a model's vulnerability to catastrophic forgetting can be predicted before fine-tuning by analyzing its local tangent geometry. The paper presents a function-space perspective that connects the geometry of fine-tuning interference with prospective vulnerability prediction and forgetting mitigation. Its main limitations are the local nature of the approximation, the computational cost of evaluating tangent geometry, and the limited evidence for the practical ranking benefits of the method when representations change substantially or adaptation proceeds beyond the early training regime.

Critical strengths
  1. A principled theoretical formulation of forgetting. The paper derives an exact transport identity under gradient flow, connecting old-task output changes to the cross-task tangent kernel and the incoming task's loss gradient.
  2. Prospective prediction with promising cross-family generalization. Unlike approaches that characterize forgetting after observing fine-tuning outcomes, the proposed diagnostic ranks vulnerable capabilities before any parameter update.
Critical weaknesses
  1. The computational overhead and practical utility remain insufficiently established. Evaluating tangent geometry and cross-task interactions for large language models may require substantial Jacobian-related computation and memory.
  2. Predictive reliability is strongly dependent on the adaptation regime. The method performs best for short adaptations with relatively small output changes, while its ranking quality deteriorates as the tangent kernel evolves.
Questions for authors
  1. The paper acknowledges that evaluating the tangent-geometry diagnostic may cost more than the short fine-tuning run it predicts. Could the authors quantify the computational and memory overhead of the method for the reported language models, relative to ordinary fine-tuning and alternative pilot-based diagnostics?
  2. The predictive ranking degrades as the tangent kernel evolves, and the MMLU results indicate that prior competence eventually becomes a stronger predictor than the proposed forecast. Can the authors identify measurable, pre-adaptation criteria that reliably estimate the validity horizon for a new fine-tuning task?
Reviewer_mupQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether fine-tuning-induced forgetting can be predicted before adaptation using tangent-space geometry. The authors derive a cross-task transport identity based on the neural tangent kernel and use it to predict changes in old-task outputs. They further consider Gauss–Newton curvature, optimizer preconditioning, and kernel evolution, and evaluate the predictions on vision models and several language model families.

Critical strengths

Predicting which existing capabilities will be affected before fine-tuning is a meaningful research problem.

The experiments cover multiple vision and language models, with several prediction baselines and analyses of the prediction horizon.

Critical weaknesses

Limited methodological novelty. The central identity, (\dot{z}_A=-J_AJ_B^\top g_B), follows directly from the chain rule and is closely related to existing gradient interference, influence function, and NTK overlap analyses. The practical language-model predictor essentially reduces to the negative inner product between old-task and new-task gradients. It is unclear what fundamentally new mechanism is introduced beyond applying this existing quantity to pre-adaptation ranking.

The proposed theoretical extensions provide limited empirical benefits. The Gauss–Newton predictor performs almost identically to raw gradient transport in the language-model experiments (Table 7). Even in the frozen-backbone experiments, the improvement over an oracle scalar-curvature baseline is only around 0.012–0.016 in drift cosine. Much of the theoretical development therefore appears to have limited impact on the main empirical results.

The practical setting is restrictive. The prediction is most reliable during very short adaptations and deteriorates as fine-tuning proceeds. Figure 3(b) shows that its predictive advantage disappears after approximately 8–16 steps in the MMLU experiments. This substantially limits its applicability to realistic fine-tuning and continual learning scenarios.

Limited practical utility relative to computational cost. The authors acknowledge that computing the prediction operator can cost more than the short fine-tuning process being predicted. Without demonstrating how this cost can be amortized or reduced, the benefit of performing such predictions instead of directly evaluating a short fine-tuning run remains unclear.

The connection between predicted output changes and actual forgetting is insufficiently established. Most language-model experiments evaluate changes in cross-entropy rather than substantial capability degradation. Several configurations even show improvements after adaptation. Similarly, the proposed protection mainly reduces cross-entropy degradation without meaningful accuracy recovery. These results support predicting local loss changes, but provide weaker evidence for predicting or preventing catastrophic forgetting.

The forgetting mitigation experiments do not establish a strong practical advantage. The spectral regularization strategy is adopted from existing work, and standard approaches such as replay and GPM remain competitive or better in several settings. Moreover, random or full-output anchoring already provides substantial protection, suggesting that much of the improvement comes from output regularization rather than the specifically identified tangent eigenspace.

Questions for authors

Beyond evaluating a gradient inner product before fine-tuning, what is the fundamental methodological contribution compared with existing gradient interference, TracIn, and influence-based approaches?

Since raw gradient transport performs almost identically to the full Gauss–Newton predictor in the language-model experiments, what practical benefit does the additional curvature modeling provide?

How can the proposed predictor be useful in realistic fine-tuning settings when its accuracy rapidly deteriorates and its computational cost may exceed that of the adaptation being predicted?

Can the authors demonstrate reliable prediction of substantial accuracy degradation, rather than primarily short-horizon cross-entropy changes?

How much of the forgetting reduction comes specifically from identifying the leading tangent eigenspace, rather than from generic output anchoring? Can the authors demonstrate a more substantial advantage over matched regularization baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.