- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CurvDelta, a delta-based adaptation framework that aims to learn new capabilities while preserving the original model’s knowledge. The authors first define a preservation loss between the original and adapted models and use a second-order Taylor expansion to show that parameter updates along low-curvature directions cause less preservation loss under the local quadratic assumption. To make the curvature tractable for large models, they approximate the preservation Hessian with K-FAC, perform spectral truncation on its Kronecker factors, and constrain the parameter delta to the resulting low-curvature subspace. They further combine this constraint with LoRA to obtain CurvLoRA, which simultaneously enforces low curvature and low-rank updates. Theoretical analysis provides a local upper bound on preservation loss under the stated quadratic and K-FAC assumptions. Experiments on mathematical reasoning and unlearning show that CurvDelta generally achieves a better trade-off between acquiring new capabilities and retaining existing ones. Ablation studies further examine the projection basis, spectral-energy threshold, and amount of preservation data used for curvature estimation.
Critical strengths
-
The proposed framework is theoretically well-grounded. The paper provides a fairly complete derivation connecting the preservation loss, local curvature, K-FAC approximation, spectral truncation, and the resulting low-curvature delta space. In particular, the local preservation bound explicitly relates the preservation cost to the retained curvature and the magnitude of parameter updates under clearly stated local quadratic and K-FAC assumptions, providing a principled explanation for why the proposed delta space can improve the acquisition-retention trade-off.
-
The parameterization is meaningful and empirically supported. Compared with methods such as CrispEdit, the key distinction is that the curvature constraint is directly incorporated into the parameterization, ensuring that the resulting updates remain within the prescribed low-curvature space throughout optimization, including when using adaptive or other optimizers. Experiments show that CurvDelta maintains competitive target-task performance while better reducing the degradation of existing capabilities than standard fine-tuning and several adaptation baselines. The dimension-matched low-/high-curvature and random-subspace ablations further support the importance of selecting curvature-informed update directions rather than merely restricting the update dimension.
Critical weaknesses
Limited evaluation across model scales and settings. The experiments cover models up to 8B parameters, with the main capability-learning experiments using only Gemma-3-1B and Llama-2-7B. Since the effectiveness of the proposed low-curvature delta space may depend on the backbone, task, and training setting, it is unclear whether the observed acquisition-retention trade-off generalizes consistently across different model scales and architectures. More experiments or analysis across a broader range of models would strengthen the evidence for the method's robustness.
Questions for authors
-
The downstream-task results do not report zero-shot performance of the base models. Could the authors provide the zero-shot GSM8K and MATH results for the base models to better quantify the actual capability gains from adaptation?
-
Since the effectiveness of the low-curvature delta space may depend on the model and training setting, could the authors provide additional analysis or evidence on how the alignment between task-relevant updates and the low-curvature subspace changes across different model scales or backbones? If possible, could the authors also provide an anonymous code repository or supplementary implementation to facilitate reproducibility?
4 comments
Question 2. Could the authors provide additional analysis or evidence on how the alignment between task-relevant updates and the low-curvature subspace changes across different model scales or backbones?
Response: Thank you for raising this question. In our gradient diagnostics, we observe substantial differences between the original and projected task gradients. This is consistent with the purpose of the preservation constraint: it redirects optimization away from directions with high preservation curvature. High alignment with the unconstrained task gradient is therefore not a prerequisite for effective adaptation. Gradient alignment describes local task descent, whereas the practical question is whether optimization within the retained subspace can achieve competitive task performance while preserving existing capabilities.
Our existing experiments provide complementary evidence for this adaptation capacity across backbones. Capability-learning experiments cover Gemma-3-1B and Llama-2-7B, while selective-unlearning experiments additionally cover Llama-3.2-3B, Qwen3-4B, and Llama-3.1-8B. Although these comparisons do not directly measure cross-backbone differences in alignment, the reported acquisition–retention and forgetting–retention results support the usefulness of the retained subspace across the evaluated models and tasks.
Question 3. Could the authors also provide an anonymous code repository or supplementary implementation to facilitate reproducibility?
Response: We provide the implementations in two anonymous repositories:
- Capability learning: anonymous repository.
- Selective unlearning: anonymous repository.
We will also release the code publicly after the review process.
We hope these clarifications address your concerns. We would be happy to provide further clarification if needed.
Thanks for the authors' responses, I think my score is enough for this paper.
Thank you for your time, constructive feedback, and supportive assessment of our work.
Thank you for your thoughtful review. We address your concerns below.
Weakness 1. Limited evaluation across model scales and settings.
Response: We agree that evaluation beyond 8B parameters would strengthen the evidence for generalization across scales. Such experiments are currently beyond our computational budget. Within these constraints, we designed our evaluation to cover variation in model size, backbone, adaptation task, and training objective.
For capability learning, we evaluate Gemma-3-1B and Llama-2-7B following prior work on LLM adaptation, covering different model families and parameter scales. Both backbones are evaluated on mathematical reasoning and code generation. To support fair comparisons, we retrain LoRA, PiSSA, and MiLoRA using configurations selected through extensive hyperparameter searches reported in prior work. For the remaining methods, including ours, we tune the learning rate over a shared grid, with the other training settings held fixed across methods.
For selective unlearning, our evaluation covers four backbones: Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct-2507 on TOFU, and Llama-2-7B on MUSE Books. We evaluate each method with three unlearning objectives: GradDiff, NPO, and SimNPO. The complete results are provided in the appendix.
Taken together, these experiments cover five distinct model backbones from four model families (Gemma-3, Llama-2, Llama-3 and Qwen-3), spanning 1B–8B parameters, across both capability learning and selective unlearning. We hope this clarifies the breadth of the existing evaluation and helps address your concern.
Question 1. Could the authors provide the zero-shot GSM8K and MATH results for the base models to better quantify the actual capability gains from adaptation?
Response: Thank you for this suggestion. We agree that reporting the base models' downstream performance helps quantify the gains from adaptation. Both pretrained base models score 0.0% on GSM8K and MATH in our zero-shot evaluation. We additionally evaluate them with 1-shot and 3-shot prompting to provide context beyond the zero-shot results. For completeness, we also report their code-generation performance. All scores below are percentages.
For these pretrained checkpoints without instruction tuning, few-shot prompting improves performance on some benchmarks, although additional examples do not consistently yield higher scores.
For reference, the following table reports the results used in the manuscript:
Both CurvDelta and CurvLoRA substantially exceed the highest base-model score among the three tested prompting settings on every benchmark for both backbones. On GSM8K and MATH, CurvDelta exceeds these base-model references by 36.4 and 5.4 percentage points for Gemma-3-1B, and by 51.7 and 11.0 percentage points for Llama-2-7B, respectively. These comparisons show substantially higher measured downstream performance after adaptation. We will include base-model results in the revised manuscript.