acceptodds
Under review as a conference paper at ICLR 2027

CurvDelta: Low-Curvature Delta Modeling For Surgical Learning and Unlearning

Abstract

A central challenge in adapting large language models (LLMs) is capability preservation: an adaptation can achieve its target while degrading general capabilities. We introduce CurvDelta, a scalable framework that incorporates the preservation constraint into the parameterization of the model update. By analyzing the local geometry of the preservation constraint around the pre-adaptation model, CurvDelta constructs a low-curvature space in which parameter updates leave the preservation loss nearly invariant. We make its construction practical for LLMs through Kronecker-factored approximate curvature (K-FAC) and a factorized projection. We also introduce CurvLoRA, a low-rank variant within the same preservation geometry, designed for more efficient adaptation. Both methods apply to preservation-constrained adaptation tasks such as capability learning and selective unlearning. We evaluate our methods on mathematical reasoning, code generation, and knowledge unlearning. For mathematical reasoning and code generation, our approaches achieves competitive downstream performance while better preserving general capabilities, offering a favorable acquisition–retention trade-off relative to the baselines. For knowledge unlearning, our main results on TOFU and MUSE show a better balance between targeted forgetting and knowledge retention than the baselines, with retention performance close to the from-scratch retraining reference.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Ja7FOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CurvDelta, a delta-based adaptation framework that aims to learn new capabilities while preserving the original model’s knowledge. The authors first define a preservation loss between the original and adapted models and use a second-order Taylor expansion to show that parameter updates along low-curvature directions cause less preservation loss under the local quadratic assumption. To make the curvature tractable for large models, they approximate the preservation Hessian with K-FAC, perform spectral truncation on its Kronecker factors, and constrain the parameter delta to the resulting low-curvature subspace. They further combine this constraint with LoRA to obtain CurvLoRA, which simultaneously enforces low curvature and low-rank updates. Theoretical analysis provides a local upper bound on preservation loss under the stated quadratic and K-FAC assumptions. Experiments on mathematical reasoning and unlearning show that CurvDelta generally achieves a better trade-off between acquiring new capabilities and retaining existing ones. Ablation studies further examine the projection basis, spectral-energy threshold, and amount of preservation data used for curvature estimation.

Critical strengths
  1. The proposed framework is theoretically well-grounded. The paper provides a fairly complete derivation connecting the preservation loss, local curvature, K-FAC approximation, spectral truncation, and the resulting low-curvature delta space. In particular, the local preservation bound explicitly relates the preservation cost to the retained curvature and the magnitude of parameter updates under clearly stated local quadratic and K-FAC assumptions, providing a principled explanation for why the proposed delta space can improve the acquisition-retention trade-off.

  2. The parameterization is meaningful and empirically supported. Compared with methods such as CrispEdit, the key distinction is that the curvature constraint is directly incorporated into the parameterization, ensuring that the resulting updates remain within the prescribed low-curvature space throughout optimization, including when using adaptive or other optimizers. Experiments show that CurvDelta maintains competitive target-task performance while better reducing the degradation of existing capabilities than standard fine-tuning and several adaptation baselines. The dimension-matched low-/high-curvature and random-subspace ablations further support the importance of selecting curvature-informed update directions rather than merely restricting the update dimension.

Critical weaknesses

Limited evaluation across model scales and settings. The experiments cover models up to 8B parameters, with the main capability-learning experiments using only Gemma-3-1B and Llama-2-7B. Since the effectiveness of the proposed low-curvature delta space may depend on the backbone, task, and training setting, it is unclear whether the observed acquisition-retention trade-off generalizes consistently across different model scales and architectures. More experiments or analysis across a broader range of models would strengthen the evidence for the method's robustness.

Questions for authors
  1. The downstream-task results do not report zero-shot performance of the base models. Could the authors provide the zero-shot GSM8K and MATH results for the base models to better quantify the actual capability gains from adaptation?

  2. Since the effectiveness of the low-curvature delta space may depend on the model and training setting, could the authors provide additional analysis or evidence on how the alignment between task-relevant updates and the low-curvature subspace changes across different model scales or backbones? If possible, could the authors also provide an anonymous code repository or supplementary implementation to facilitate reproducibility?

4 comments

Thank you for your thoughtful review. We address your concerns below.


Weakness 1. Limited evaluation across model scales and settings.

Response: We agree that evaluation beyond 8B parameters would strengthen the evidence for generalization across scales. Such experiments are currently beyond our computational budget. Within these constraints, we designed our evaluation to cover variation in model size, backbone, adaptation task, and training objective.

For capability learning, we evaluate Gemma-3-1B and Llama-2-7B following prior work on LLM adaptation, covering different model families and parameter scales. Both backbones are evaluated on mathematical reasoning and code generation. To support fair comparisons, we retrain LoRA, PiSSA, and MiLoRA using configurations selected through extensive hyperparameter searches reported in prior work. For the remaining methods, including ours, we tune the learning rate over a shared grid, with the other training settings held fixed across methods.

For selective unlearning, our evaluation covers four backbones: Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct-2507 on TOFU, and Llama-2-7B on MUSE Books. We evaluate each method with three unlearning objectives: GradDiff, NPO, and SimNPO. The complete results are provided in the appendix.

Taken together, these experiments cover five distinct model backbones from four model families (Gemma-3, Llama-2, Llama-3 and Qwen-3), spanning 1B–8B parameters, across both capability learning and selective unlearning. We hope this clarifies the breadth of the existing evaluation and helps address your concern.


Question 1. Could the authors provide the zero-shot GSM8K and MATH results for the base models to better quantify the actual capability gains from adaptation?

Response: Thank you for this suggestion. We agree that reporting the base models' downstream performance helps quantify the gains from adaptation. Both pretrained base models score 0.0% on GSM8K and MATH in our zero-shot evaluation. We additionally evaluate them with 1-shot and 3-shot prompting to provide context beyond the zero-shot results. For completeness, we also report their code-generation performance. All scores below are percentages.

Base modelShotsGSM8KMATHHumanEvalMBPP
Gemma-3-1B00.00.01.213.0
Gemma-3-1B10.50.10.011.6
Gemma-3-1B30.50.13.75.0
Llama-2-7B00.00.01.829.9
Llama-2-7B11.10.00.020.9
Llama-2-7B36.70.05.528.3

For these pretrained checkpoints without instruction tuning, few-shot prompting improves performance on some benchmarks, although additional examples do not consistently yield higher scores.

For reference, the following table reports the results used in the manuscript:

BackboneMethodGSM8K (%)MATH (%)HumanEval (%)MBPP (%)
Gemma-3-1BCurvDelta36.95.517.122.2
Gemma-3-1BCurvLoRA33.74.716.520.6
Llama-2-7BCurvDelta58.411.039.641.3
Llama-2-7BCurvLoRA56.610.036.041.8

Both CurvDelta and CurvLoRA substantially exceed the highest base-model score among the three tested prompting settings on every benchmark for both backbones. On GSM8K and MATH, CurvDelta exceeds these base-model references by 36.4 and 5.4 percentage points for Gemma-3-1B, and by 51.7 and 11.0 percentage points for Llama-2-7B, respectively. These comparisons show substantially higher measured downstream performance after adaptation. We will include base-model results in the revised manuscript.

Question 2. Could the authors provide additional analysis or evidence on how the alignment between task-relevant updates and the low-curvature subspace changes across different model scales or backbones?

Response: Thank you for raising this question. In our gradient diagnostics, we observe substantial differences between the original and projected task gradients. This is consistent with the purpose of the preservation constraint: it redirects optimization away from directions with high preservation curvature. High alignment with the unconstrained task gradient is therefore not a prerequisite for effective adaptation. Gradient alignment describes local task descent, whereas the practical question is whether optimization within the retained subspace can achieve competitive task performance while preserving existing capabilities.

Our existing experiments provide complementary evidence for this adaptation capacity across backbones. Capability-learning experiments cover Gemma-3-1B and Llama-2-7B, while selective-unlearning experiments additionally cover Llama-3.2-3B, Qwen3-4B, and Llama-3.1-8B. Although these comparisons do not directly measure cross-backbone differences in alignment, the reported acquisition–retention and forgetting–retention results support the usefulness of the retained subspace across the evaluated models and tasks.


Question 3. Could the authors also provide an anonymous code repository or supplementary implementation to facilitate reproducibility?

Response: We provide the implementations in two anonymous repositories:

We will also release the code publicly after the review process.


We hope these clarifications address your concerns. We would be happy to provide further clarification if needed.

Thanks for the authors' responses, I think my score is enough for this paper.

Thank you for your time, constructive feedback, and supportive assessment of our work.

Reviewer_KJNSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CurvDelta and its low-rank variant, CurvLoRA, for preservation-constrained learning and unlearning. The methods parameterize updates within a fixed low-curvature subspace estimated using K-FAC. The paper provides a local preservation bound and evaluates the approach on mathematical reasoning, code generation, and selective unlearning.

Critical strengths
  1. Incorporating the projection into the delta parameterization ensures that the applied update remains in the prescribed subspace independently of the optimizer. The factorwise construction supports both dense and low-rank adaptation while preserving the latter’s rank constraint.

  2. The theoretical analysis connects retained curvature and update magnitude to preservation loss. Experiments cover multiple tasks and models, with encouraging code-generation results and dimension-matched ablations supporting the choice of low-curvature directions.

Critical weaknesses

The benefits of the distinctive design choices are not sufficiently isolated. CrispEdit[1] already derives low-curvature gradient projections using preservation-loss geometry and K-FAC, while OPLoRA[2] applies two-sided projections to LoRA updates using pretrained-weight singular subspaces. CurvDelta introduces a factorwise, rank-preserving projection based on preservation curvature and incorporates it into the delta parameterization. However, its comparison with CrispEdit changes both the subspace construction and the object being projected. Although these differences are discussed, a controlled ablation holding the projection space fixed would better establish the independent benefit of delta parameterization.

The practical applicability of the preservation guarantee remains unclear. Theorem 1 establishes a conditional bound under a local quadratic model and block-diagonal K-FAC assumptions. It also requires control of the update magnitude, which Algorithm 1 does not explicitly enforce. Since curvature is estimated only at initialization, the paper should establish whether the bound remains informative throughout the reported training runs.

[1] CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing [2] OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting During Parameter-Efficient Fine-Tuning

Questions for authors

The same as weaknesses.

3 comments

Thank you for your careful review and for recognizing the strengths of our method and theoretical analysis. We address your concerns below.


Weakness 1. The benefits of the distinctive design choices are not sufficiently isolated, and a controlled ablation holding the projection space fixed would better establish the independent benefit of delta parameterization.

Response: Thank you for suggesting this controlled comparison. We agree that holding the projection space fixed is necessary to isolate the contribution of delta parameterization. We have conducted an additional ablation on TOFU Forget 10% using Llama-3.2-3B-Instruct and SimNPO, comparing CurvDelta with a gradient-projection variant, denoted CurvDeltaGrad. This variant uses exactly the same fixed low-curvature projectors as CurvDelta, applying them to task gradients rather than incorporating them into the delta parameterization. Both methods use Adam. At each learning rate, the remaining training settings are matched; other configurations follow the manuscript. The experiments below were conducted on different machines from those used for the submission, so the absolute results differ from the reported values.

LRMethodFQ ↑FP (%) ↓MU (%) ↑
2e-5CurvDelta-1.10811.366.3
2e-5CurvDeltaGrad-1.69911.866.1
5e-5CurvDelta-0.19310.265.6
5e-5CurvDeltaGrad-0.67610.065.1

With the projection space fixed, CurvDelta achieves higher FQ and MU at both reported learning rates. At LR = 2e-5, it improves all three metrics. At LR = 5e-5, which gives the highest FQ for both methods, CurvDelta improves FQ from -0.676 to -0.193 and MU from 65.1% to 65.6%, with nearly unchanged FP (10.0% versus 10.2%). These results support the contribution of delta parameterization to the forgetting–utility trade-off even when the low-curvature space is held fixed.

CurvDelta directly constrains the applied delta through ΔW=PoutδPin\Delta W=P^{\mathrm{out}}\delta P^{\mathrm{in}}, keeping it in the prescribed low-curvature space throughout optimization, independently of the optimizer. The controlled comparison above provides empirical evidence for the value of this parameterization beyond subspace selection alone. We will include this ablation in the revised manuscript.

Weakness 2. The practical applicability of the preservation guarantee remains unclear.

Response: Thank you for this suggestion. We evaluated existing CurvLoRA checkpoints on TOFU Forget 10% with Llama-3.2-3B-Instruct and SimNPO, comparing empirical retain-set KL with the K-FAC quadratic approximation and the spectral bound in Eq. (8). We also added a preservation comparison with baseline methods.

1. Checkpoint comparison and relationship to the theoretical analysis.

The table below reports all three quantities across three learning rates and three spectral-energy thresholds γ\gamma:

LRγ\gammaEmpirical KLK-FAC quadratic approximationSpectral bound
1e-40.50.038925.82e-91.72e-6
1e-40.70.030972.44e-92.60e-7
1e-40.90.012965.17e-101.24e-8
2e-40.50.048031.30e-84.73e-6
2e-40.70.038035.03e-96.48e-7
2e-40.90.018539.14e-102.56e-8
5e-40.50.073989.12e-83.04e-5
5e-40.70.058993.56e-84.36e-6
5e-40.90.040723.62e-91.18e-7

Our theorem bounds the local quadratic preservation cost under the block-diagonal K-FAC model. The quadratic approximation satisfies this bound in all nine configurations. We notice that actual KL is substantially higher, indicating a gap between the local approximation and the empirical loss at these checkpoints. Nevertheless, the comparison also reveals consistent trends across the theoretical quantities and actual KL.

Stronger curvature constraints consistently improve empirical preservation, in line with the theoretical analysis. At every learning rate, increasing γ\gamma excludes more leading curvature directions and reduces all three quantities. At each fixed γ\gamma, all three increase with the learning rate. Although the local model does not reproduce the numerical scale of actual KL, it captures the direction of these changes across the evaluated checkpoints. This agreement supports its use in motivating the low-curvature constraint and understanding its effect on preservation. We will include this comparison and clarify the scope of the analysis in the revised manuscript.

2. Both variants improve preservation relative to their respective baselines.

For an additional comparison, we evaluate all methods:

Update typeMethodRetain KL ↓
DenseFFT0.62394
DenseCrispEdit0.75187
DenseCurvDelta0.14866
Low-rankLoRA0.04302
Low-rankVILA0.47349
Low-rankOPLoRA0.04222
Low-rankCurvLoRA0.01296

CurvDelta has substantially lower KL than FFT and CrispEdit, while CurvLoRA has the lowest KL among the evaluated low-rank methods. This comparison complements the threshold experiment: the preservation benefit appears both when strengthening the constraint within our method and when comparing against other methods at the same learning rate. These KL measurements assess preservation; the accompanying forgetting results are needed to assess the overall forgetting–utility trade-off.

3. Control of update magnitude.

Algorithm 1 constrains the update directions but does not explicitly cap the delta norm. A simple optional extension is to impose a norm limit RR on the total applied delta. After each optimization step, if ∑l∈T∥ΔWl∥F2>R\sqrt{\sum_{l\in\mathcal{T}}\lVert\Delta W_l\rVert_F^2}>R, we rescale the layer deltas by a common factor:

ΔWl←R∑j∈T∥ΔWj∥F2ΔWl,l∈T.\Delta W_l \leftarrow \frac{R}{\sqrt{\sum_{j\in\mathcal{T}}\lVert\Delta W_j\rVert_F^2}}\Delta W_l, \qquad l\in\mathcal{T}.

This directly controls update magnitude while preserving the selected subspace and the rank constraint. For CurvLoRA, the rescaling can be implemented by multiplying one LoRA factor in each layer by the common scaling factor. We leave this extension to future work as a simple way to combine curvature constraints with explicit control of update magnitude.


We hope these responses address your concerns and would be happy to clarify any remaining points.

Dear Reviewer KJNS,

Thank you again for your thoughtful review. As the discussion period is drawing to a close, we wanted to kindly follow up and ask whether our previous response—including the controlled fixed-subspace ablation and the checkpoint-based verification of the preservation bound—has addressed your concerns, or if there are any remaining points we could further clarify.

Reviewer_4TjEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CrispEdit-like curvature null-space projection method for LLM editing. Their method, CurveLoRA, extends it for low-rank parameters.

Critical strengths

The idea is sound. Presentation of the work is solid. The experiments are carried out properly and are quite extensive.

Critical weaknesses

The novelty of this work is modest. It appears to be almost identical to CrispEdit [1] in its section 2, 3.1, 3.2. CurveDelta, in the presented way, appears indistinguishable from CrispEdit, which makes the result make sense: most of the row scores for CurveDelta and CrispEdit appears to be within noise.

  1. CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing, ICML 2026
Questions for authors
  1. Can the authors explain the difference between CurveDelta and CrispEdit?
  2. What is \Delta W_\ell used for in Algorithm 1?
3 comments

Thank you for recognizing the soundness, presentation, and experimental scope of our work. Your concern about novelty and the relationship to CrispEdit, one of the baselines evaluated in our paper, is closely connected to your question about ΔWℓ\Delta W_\ell, which is both the core concept of our method and a key distinction from CrispEdit. We address these points together below.


What is ΔWℓ\Delta W_\ell? It is the actual weight update applied to layer ℓ\ell relative to the frozen pretrained weight Wℓ,0W_{\ell,0}. At each training step, the layer uses Wℓ,0+ΔWℓW_{\ell,0}+\Delta W_\ell, and the collection of these layer updates forms Δθ\Delta\theta in Algorithm 1, so that the adapted parameters are θ0+Δθ\theta_0+\Delta\theta, where θ0\theta_0 denotes the frozen pretrained parameters. We construct it as (Equation 7 in the paper):

ΔWℓ=Pℓout δℓ(ϕℓ) Pℓin,\Delta W_\ell=P_\ell^{\mathrm{out}}\,\delta_\ell(\phi_\ell)\,P_\ell^{\mathrm{in}},

where δℓ(ϕℓ)\delta_\ell(\phi_\ell) is the pre-projection delta represented by trainable parameters ϕℓ\phi_\ell, and the fixed projectors define the low-curvature space. In CurvDelta, δℓ\delta_\ell is a trainable dense matrix; in CurvLoRA, it is BℓAℓB_\ell A_\ell, where BℓB_\ell and AℓA_\ell are trainable low-rank factors with rank parameter rr. We evaluate the task loss using the projected updates ΔWℓ\Delta W_\ell and optimize ϕℓ\phi_\ell by differentiating through the projection. After training, we merge the projected delta Pℓout δℓ(ϕℓ) PℓinP_\ell^{\mathrm{out}}\,\delta_\ell(\phi_\ell)\,P_\ell^{\mathrm{in}} into the pretrained weight.


How does CurvDelta differ from CrispEdit? The two methods share foundations in low-curvature preservation, but differ in the object they constrain, the construction of the projection, and their formulation and analysis.

1. Update parameterization.

  • CrispEdit motivates low-curvature weight changes and projects the task gradient.
  • CurvDelta directly parameterizes the realized delta in a fixed low-curvature space.

This distinction matters because projecting a gradient does not in general constrain the update: for example, Adam's coordinate-wise rescaling can take an increment out of the projected space. By contrast, our applied ΔWℓ\Delta W_\ell lies in the prescribed space at every step.

2. Projection construction.

  • CrispEdit projects gradients using a mask based on the products of K-FAC factor eigenvalues.
  • CurvDelta instead uses a separable approximation to the product-thresholded low-curvature eigenspace: it truncates each factor spectrum separately, yielding a two-sided projection of the model delta.

This approximation is nontrivial: it substantially reduces computational and storage costs, and also allows us to apply the projection directly to low-rank factors, yielding CurvLoRA. It therefore makes the preservation geometry applicable to efficient low-rank adaptation as well as dense adaptation.

3. Formulation and analysis.

We respectfully disagree that the similarities in Sections 2, 3.1, and 3.2 make the papers almost identical. These sections share concepts and mathematical tools, but differ in their formulation and analysis:

  • Section 2: Problem formulation. Both use constrained optimization for capability preservation. CrispEdit formulates model editing in terms of changes in capability loss. We define preservation through output deviation from the original model and use task and preservation sets to cover both capability learning and selective unlearning.
  • Section 3.1: Local curvature. Both use a second-order approximation and motivate low-curvature directions. We make explicit how the local preservation cost depends jointly on the update's direction and magnitude. This analysis motivates our delta-space construction and the subsequent preservation bound.
  • Section 3.2: Curvature estimation. Both use K-FAC, but our factorwise truncation constructs a two-sided delta space rather than CrispEdit’s product-thresholded gradient projection.

Building on these foundations, we introduce projected delta parameterizations, a low-rank variant, and a bound that relates preservation loss to both retained curvature and update magnitude.

Empirical comparison with CrispEdit. In addition to the methodological and theoretical differences above, we would like to address the concern that most scores are within noise. The reported results include differences in adaptation–preservation trade-offs across both capability learning and selective unlearning. We highlight two examples from the main paper below.

Capability learning. On code generation with Llama-2-7B, CurvDelta improves both downstream performance and retention over CrispEdit (Table 2).

MethodHumanEval ↑MBPP ↑Downstream Avg. ↑TriviaQA ↑NQ-Open ↑MMLU ↑Retention Avg. ↑ALL ↑
CrispEdit37.840.739.357.819.543.540.339.9
CurvDelta39.641.340.563.124.945.144.442.8

Selective unlearning. On TOFU Forget 10% with Llama-3.1-8B-Instruct and SimNPO, CurvDelta achieves lower forget-answer probability and higher model utility than CrispEdit (Table 3).

MethodFQ ↑FP ↓MU ↑
CrispEdit−1.916.942.0
CurvDelta−1.16.563.2

These examples show differences in the reported scores. We therefore believe that the comparison warrants consideration beyond the characterization that the reported gains are generally within noise.


Taken together, these methodological and theoretical differences, along with the empirical improvements over CrispEdit, demonstrate that CurvDelta makes distinct contributions. We hope this clarification addresses the reviewer’s concerns about novelty and answers their questions. If there are any remaining concerns, we would be happy to provide further clarifications.

Dear Reviewer 4TjE,

Thank you again for your review and feedback. As the discussion period is approaching its end, we wanted to kindly check whether our earlier response (clarifying ΔWℓ\Delta W_\ell in Algorithm 1, the methodological distinctions from CrispEdit, and our empirical comparisons against CrispEdit) has addressed your concerns, or if there are any remaining questions we can clarify.

Reviewer_RaScOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to mitigate the degradation of existing capabilities during model adaptation and proposes CurvDelta, a framework for low-curvature parameter updates. CurvDelta uses K-FAC to approximate the local curvature of the preservation loss and applies two-sided projections to constrain parameter updates to a fixed low-curvature subspace. The framework supports both dense updates and low-rank adaptation (CurvLoRA). The authors also derive a local bound on the preservation loss. Experiments cover mathematical reasoning, code generation, and selective unlearning on TOFU and MUSE. The results show that CurvDelta achieves competitive performance across multiple settings while improving the trade-off between target task performance and retention of existing capabilities.

Critical strengths

1. CurvDelta incorporates low-curvature constraints directly into the delta parameterization, ensuring that the applied updates remain within the prescribed subspace throughout optimization. The framework supports both dense and low-rank adaptation, offering flexibility and reusability across different tasks.

2. The evaluation covers both capability learning and selective unlearning, with comparisons against relevant methods such as CrispEdit and OPLoRA, as well as ablation studies. CurvDelta achieves competitive results across multiple settings, demonstrating a favorable trade-off between task performance and retention of existing capabilities.

Critical weaknesses

The paper includes comparisons with relevant methods such as CrispEdit and OPLoRA, along with ablations on the projection subspace. However, the current experiments do not fully isolate the benefit of the proposed delta parameterization from that of low-curvature subspace selection. Given that the parameterization is a key contribution of the paper, a more direct comparison would help clarify its advantages over existing projection-based methods.

Questions for authors

Q1. Could the authors compare gradient projection and delta parameterization under the same low-curvature subspace and training setup, particularly when using Adam? This would help better isolate the benefit of the proposed parameterization.

Q2. Table 4 provides a theoretical complexity analysis. Could the authors also report the one-time cost of curvature estimation, wall-clock training time, and peak GPU memory usage? These measurements would help assess the practical efficiency of the method, especially when the projection subspace is reused across tasks.

Q3. Could the authors use existing checkpoints to compare the empirical preservation loss with its local quadratic approximation and the corresponding theoretical bound? This would help clarify how well the theoretical analysis reflects the behavior observed during training.

4 comments

Thank you for your thoughtful review and positive assessment of our method and experiments. We address your concerns below.


Weakness 1 and Question 1. The current experiments do not fully isolate the benefit of the proposed delta parameterization from that of low-curvature subspace selection.

Response: Thank you for suggesting this controlled comparison. We have conducted an additional ablation on TOFU Forget 10% using Llama-3.2-3B-Instruct and SimNPO, comparing CurvDelta with a gradient-projection variant, denoted CurvDeltaGrad. This variant uses exactly the same fixed low-curvature projectors as CurvDelta. Both methods use Adam. The experiments below were conducted on a different machine, so the absolute results differ from the submission.

LRMethodFQ ↑FP (%) ↓MU (%) ↑
2e-5CurvDelta-1.10811.366.3
2e-5CurvDeltaGrad-1.69911.866.1
5e-5CurvDelta-0.19310.265.6
5e-5CurvDeltaGrad-0.67610.065.1

With the projection space fixed, CurvDelta achieves higher FQ and MU at both learning rates. At LR = 2e-5, it improves all three metrics. At LR = 5e-5, which gives the highest FQ for both methods, CurvDelta improves FQ from -0.676 to -0.193 and MU from 65.1% to 65.6%, with nearly unchanged FP (10.0% versus 10.2%). These results support the contribution of delta parameterization to the forgetting–utility trade-off even when the low-curvature space is held fixed.

CurvDelta directly constrains the applied delta through ΔW=PoutδPin\Delta W=P^{\mathrm{out}}\delta P^{\mathrm{in}}, keeping it in the prescribed low-curvature space throughout optimization, independently of the optimizer. The controlled comparison above provides empirical evidence for the value of this parameterization beyond subspace selection alone. We will include this ablation in the revised manuscript.


Question 2. Could the authors also report the one-time cost of curvature estimation, wall-clock training time, and peak GPU memory usage?

Response: Thank you for suggesting these practical efficiency measurements. We benchmark all methods under the same setting: TOFU Forget 10% with Llama-3.2-3B-Instruct and SimNPO, using an NVIDIA L40 GPU. We fix per_device_batch_size=4 across methods to make the memory comparison consistent. The runtimes below are total wall-clock times, including curvature estimation for methods that require it and subsequent task training.

MethodTotal runtime (s) ↓Peak GPU memory (GB) ↓
Full fine-tuning (FFT)53727.5
CrispEdit248240.8
LoRA67417.9
OPLoRA88218.0
CurvLoRA86624.4
CurvDelta100438.4

In this setting, CurvDelta reduces total runtime by 59.5% relative to CrispEdit (1004 vs. 2482 s), with slightly lower peak memory (38.4 vs. 40.8 GB). CurvLoRA further reduces memory to 24.4 GB, a 36.5% reduction relative to dense CurvDelta, and takes 866 s, close to OPLoRA's 882 s.

The table includes the initial curvature-estimation cost in the total runtime. Regarding the one-time curvature cost and reuse, we estimate the K-FAC factors and compute their eigendecompositions once before task training, then cache the selected leading eigenvector bases. With the base model, preservation set, target weights, and spectral-energy threshold fixed, this cache can be reused across task objectives, optimization hyperparameters, and both dense and low-rank delta parameterizations. Thus, subsequent runs using the same cache avoid repeating curvature estimation and eigendecomposition.

Question 3. Could the authors use existing checkpoints to compare the empirical preservation loss with its local quadratic approximation and the corresponding theoretical bound?

Response: Thank you for this suggestion. We evaluated existing CurvLoRA checkpoints on TOFU Forget 10% with Llama-3.2-3B-Instruct and SimNPO, comparing empirical retain-set KL with the K-FAC quadratic approximation and the spectral bound in Eq. (8). We also added a preservation comparison with baseline methods.

1. Checkpoint comparison and relationship to the theoretical analysis.

The table below reports all three quantities across three learning rates and three spectral-energy thresholds γ\gamma:

LRγ\gammaEmpirical KLK-FAC quadratic approximationSpectral bound
1e-40.50.038925.82e-91.72e-6
1e-40.70.030972.44e-92.60e-7
1e-40.90.012965.17e-101.24e-8
2e-40.50.048031.30e-84.73e-6
2e-40.70.038035.03e-96.48e-7
2e-40.90.018539.14e-102.56e-8
5e-40.50.073989.12e-83.04e-5
5e-40.70.058993.56e-84.36e-6
5e-40.90.040723.62e-91.18e-7

Our theorem bounds the local quadratic preservation cost under the block-diagonal K-FAC model. The quadratic approximation satisfies this bound in all nine configurations. We notice that actual KL is substantially higher, indicating a gap between the local approximation and the empirical loss at these checkpoints. Nevertheless, the comparison also reveals consistent trends across the theoretical quantities and actual KL.

Stronger curvature constraints consistently improve empirical preservation, in line with the theoretical analysis. At every learning rate, increasing γ\gamma excludes more leading curvature directions and reduces all three quantities. At each fixed γ\gamma, all three increase with the learning rate. Although the local model does not reproduce the numerical scale of actual KL, it captures the direction of these changes across the evaluated checkpoints. This agreement supports its use in motivating the low-curvature constraint and understanding its effect on preservation. We will include this comparison and clarify the scope of the analysis in the revised manuscript.

2. Both variants improve preservation relative to their respective baselines.

For an additional comparison, we evaluate all methods:

Update typeMethodRetain KL ↓
DenseFFT0.62394
DenseCrispEdit0.75187
DenseCurvDelta0.14866
Low-rankLoRA0.04302
Low-rankVILA0.47349
Low-rankOPLoRA0.04222
Low-rankCurvLoRA0.01296

CurvDelta has substantially lower KL than FFT and CrispEdit, while CurvLoRA has the lowest KL among the evaluated low-rank methods. This comparison complements the threshold experiment: the preservation benefit appears both when strengthening the constraint within our method and when comparing against other methods at the same learning rate. These KL measurements assess preservation; the accompanying forgetting results are needed to assess the overall forgetting–utility trade-off.


We hope these responses address your concerns and would be happy to clarify any remaining points.

The additional experiments and clarifications help address my concerns and provide further support for the proposed method. I appreciate the authors' efforts in responding to the review.

Thank you for your time, constructive feedback, and supportive assessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment to Area Chairs

Dear Area Chair,

We thank the reviewers for their careful reviews. Below we summarize the key issues resolved during the rebuttal and our manuscript revisions.

1. Key Issues Resolved

  • Isolating Delta Parameterization vs. Gradient Projection (RaSc, KJNS, 4TjE): We added a controlled ablation comparing our CurvDelta with CurvDeltaGrad, which applies the same fixed projectors to task gradients under Adam (Section 4.3, Figure 3). Holding the subspace fixed, CurvDelta consistently improves performance, confirming the independent benefit of delta parameterization under adaptive optimization.
  • Distinction from CrispEdit (4TjE, KJNS): Beyond parameterizing the realized delta rather than projecting gradients, CurvDelta uses separable factorwise truncation instead of CrispEdit's non-separable product mask (Section 3.2–3.3, Algorithm 1). This substantially reduces storage and compute costs by caching only thin leading eigenbases, allows projecting low-rank factors directly without materializing a dense matrix (yielding CurvLoRA), and outperforms CrispEdit in capability learning (Table 2), unlearning (Table 3), retain KL (Figure 4b), and runtime (Figure 4c).
  • Empirical Verification of the Preservation Bound (RaSc, KJNS): Across learning rates and thresholds γ\gamma (Section 4.4, Figure 4a, Appendix Table 13), the K-FAC quadratic approximation strictly satisfies Theorem 1's spectral bound, and both theoretical quantities and empirical retain KL share identical monotonic trends. Moreover, CurvDelta and CurvLoRA achieve the lowest retain KL among dense and low-rank baselines (Figure 4b).
  • Practical Efficiency & Base-Model Evaluations (RaSc, Ja7F): CurvDelta cuts wall-clock runtime (including curvature estimation) by 59.5% vs. CrispEdit with lower peak memory, while CurvLoRA reduces peak memory by 36.5% (Section 4.4, Figure 4c). We also added zero-shot base-model downstream evaluations (Tables 1–2).

2. Manuscript Revisions

  • Section 3.2–3.3 & Algorithm 1: Clarified ΔWl\Delta W_l and differences from CrispEdit.
  • Section 4.1 (Tables 1–2): Added zero-shot base-model downstream results.
  • Section 4.3 (Figure 3): Added the controlled CurvDelta vs. CurvDeltaGrad ablation.
  • Section 4.4 (Figure 4) & Appendix F.4 (Tables 13–14): Added plots and numerical values of the preservation bound, retain KL, and efficiency analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.