- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LessEdit, a capability-aware method for sequential knowledge editing. The authors note that existing null-space methods restrict updates to feasible directions without considering their varying impacts on general capabilities. To address this, LessEdit combines dynamic null-space constraints with anisotropic regularization based on the second-order moments of reference activations to reduce interference. Overall, the motivation is clear, the method and derivations are complete, and updates are computed efficiently in closed form. Experiments across multiple benchmarks use the WILD protocol, reflecting actual generation ability better than teacher-forcing. Ablations and directional analyses further verify the regularization, showing improved sequential editing while maintaining base capabilities.
Critical strengths
-
Realistic evaluation setup. Instead of relying solely on traditional teacher-forcing, the paper uses the WILD framework to better capture open-ended post-edit performance and avoid probability-based evaluation bias. Evaluating reliability and generalization across diverse prompt formats also makes the results more comprehensive and convincing.
-
Clear motivation and well-validated core mechanism. Prior null-space methods use threshold truncation to form an approximate null space, so updates within the editable subspace can still perturb original activations and outputs—a directional impact largely overlooked by existing work. To minimize such perturbations, the authors use the second-order moments of reference activations to quantify directional sensitivity and introduce capability-aware anisotropic regularization. The method is well-motivated with complete derivations, and the proposed CapCov mechanism is sufficiently validated through ablations, fixed-state directional analyses, and comparisons with alternative regularizers
Critical weaknesses
-
Lack of evaluation on smaller-scale models. Current experiments focus primarily on LLaMA-3-8B and Qwen2.5-7B, leaving the method's applicability to smaller models underexplored. Since smaller models typically have lower-dimensional FFN weight matrices and LessEdit continuously removes previously used directions during sequential editing, the editable subspace may shrink more rapidly as edits accumulate, potentially limiting subsequent knowledge injection. It is recommended to evaluate smaller models and analyze how the editable subspace dimensionality evolves across model scales, as well as how this affects editing success rates and base capability preservation, to better establish the method's scalability.
-
Unclear link between CapCov's local perturbation metric and actual capability retention. Using reference activation second-order moments to measure FFN output perturbation serves only as a local proxy; smaller local shifts do not necessarily ensure preserved final task performance due to subsequent layers and non-linearities. Although Appendix D.1 examines decoder-block and logit-level shifts, it lacks direct validation against actual capability retention. Correlating CapCov metrics with downstream performance changes (e.g., MMLU, GSM8K) across editing stages would better justify this proxy objective.
Questions for authors
See weaknesses.
4 comments
Rebuttal-Part 2
W2. Does the local perturbation metric track actual downstream capability across editing stages?
We measured benchmark performance alongside reference-output cost at successive checkpoints. Applying the paper's reference quadratic to cumulative effective weight changes gives
Here remains fixed. Cost measures reference-output change, not computation time.
(1) Across editing stages. The following CounterFact results use the original LLaMA-3-8B runs, including the historical AlphaEdit ridge-1 configuration. Cost is estimated with 64 Hutchinson probes.
Table N3. Reference-output cost and all five capability tasks across six stages.
| Method | Edits | Total cost | MMLU | IFEval | TruthfulQA-MC2 | ARC-Challenge | GSM8K | Five-task mean |
|---|---|---|---|---|---|---|---|---|
| Unedited | 0 | 0 | 69.47 | 69.26 | 50.72 | 56.00 | 73.50 | 63.79 |
| AlphaEdit | 100 | 0.1528 | 69.33 | 70.64 | 50.21 | 56.50 | 73.00 | 63.94 |
| EvoEdit | 100 | 0.1123 | 69.35 | 69.99 | 50.38 | 56.50 | 72.50 | 63.74 |
| LessEdit | 100 | 0.0598 | 69.28 | 69.83 | 50.42 | 56.00 | 73.00 | 63.71 |
| AlphaEdit | 500 | 0.8581 | 68.62 | 69.01 | 47.43 | 55.00 | 75.50 | 63.11 |
| EvoEdit | 500 | 0.5926 | 68.92 | 68.54 | 49.18 | 56.00 | 73.50 | 63.23 |
| LessEdit | 500 | 0.3383 | 69.03 | 71.55 | 50.77 | 55.00 | 71.50 | 63.57 |
| AlphaEdit | 1,000 | 1.9369 | 66.83 | 66.19 | 46.35 | 54.50 | 74.00 | 61.57 |
| EvoEdit | 1,000 | 1.2268 | 67.83 | 70.42 | 48.73 | 54.50 | 73.50 | 63.00 |
| LessEdit | 1,000 | 0.7449 | 68.50 | 72.02 | 50.78 | 54.50 | 71.50 | 63.46 |
| AlphaEdit | 1,500 | 3.2539 | 64.46 | 64.00 | 47.28 | 54.00 | 69.50 | 59.85 |
| EvoEdit | 1,500 | 1.8972 | 67.29 | 69.67 | 48.30 | 55.00 | 71.50 | 62.35 |
| LessEdit | 1,500 | 1.2205 | 67.68 | 67.32 | 50.97 | 55.50 | 73.00 | 62.89 |
| AlphaEdit | 2,000 | 4.8355 | 61.12 | 60.43 | 44.68 | 52.00 | 65.00 | 56.65 |
| EvoEdit | 2,000 | 2.6202 | 66.61 | 68.64 | 47.59 | 54.00 | 71.00 | 61.57 |
| LessEdit | 2,000 | 1.7705 | 67.05 | 70.11 | 49.28 | 55.00 | 71.00 | 62.49 |
| AlphaEdit | 3,000 | 8.4803 | 48.33 | 33.41 | 37.79 | 43.50 | 41.50 | 40.91 |
| EvoEdit | 3,000 | 4.0668 | 64.52 | 65.22 | 45.80 | 54.00 | 71.00 | 60.11 |
| LessEdit | 3,000 | 2.9858 | 64.76 | 64.25 | 47.64 | 54.50 | 69.00 | 60.03 |
Table N4. Correlation between cost and capability decline from the unedited baseline. These are descriptive correlations across 18 checkpoints from three trajectories with varying edit counts.
| Capability decline | Spearman correlation | Pearson correlation |
|---|---|---|
| MMLU | 0.986 | 0.960 |
| IFEval | 0.781 | 0.903 |
| TruthfulQA-MC2 | 0.827 | 0.942 |
| ARC-Challenge | 0.897 | 0.911 |
| GSM8K | 0.670 | 0.870 |
| Five-task mean | 0.981 | 0.927 |
Cost correlates with decline in every benchmark, with Spearman correlations of 0.986 for MMLU and 0.981 for the five-task mean. At each matched stage from 500 to 3,000 edits, lower cost also agrees with higher MMLU.
(2) At a fixed edit count. We additionally computed exact costs after 10,000 ZsRE edits under two random orders, using shared initial weights and reference statistics. The three configurations were selected before these order runs. The isotropic variant replaces CapCov by the identity; its total coefficient is .
Table N5. New LLaMA-3-8B comparisons at 10,000 edits.
| Order seed | Configuration | Total cost | MMLU | IFEval | TruthfulQA-MC2 | ARC-Challenge | GSM8K | Five-task mean |
|---|---|---|---|---|---|---|---|---|
| 20261121 | LessEdit: ridge 4 + CapCov | 7.6568 | 59.11 | 58.23 | 44.07 | 49.50 | 58.00 | 53.78 |
| 20261121 | EvoEdit: ridge 4 | 14.1458 | 51.60 | 44.44 | 40.72 | 48.50 | 28.50 | 42.75 |
| 20261121 | Isotropic: coefficient 1.999 | 30.2958 | 27.54 | 25.04 | 46.33 | 30.00 | 2.00 | 26.18 |
| 20261122 | LessEdit: ridge 4 + CapCov | 7.6666 | 59.83 | 58.95 | 40.86 | 47.50 | 54.50 | 52.33 |
| 20261122 | EvoEdit: ridge 4 | 13.9933 | 52.19 | 51.87 | 41.72 | 46.00 | 30.50 | 44.46 |
| 20261122 | Isotropic: coefficient 1.999 | 28.4870 | 30.36 | 28.99 | 44.91 | 32.00 | 3.50 | 27.95 |
LessEdit lowers cost relative to EvoEdit by 45.87% and 45.21%, alongside 11.03- and 7.87-point gains in mean capability. These comparisons hold edit count fixed, complementing the stage correlations.
(3) On the smaller model. We also computed exact costs for all eight Qwen1.5B checkpoints in Table N1a.
Table N6. New 1.5B experiment: exact reference-output cost.
| Edits | LessEdit | EvoEdit |
|---|---|---|
| 500 | 35.2403 | 41.3733 |
| 1,000 | 73.6158 | 87.3493 |
| 2,000 | 162.6978 | 195.2864 |
| 3,000 | 294.0071 | 391.1105 |
At 3,000 edits, LessEdit has 24.83% lower cost, together with the editing and capability gains in Tables N1a–b. These measurements support reference sensitivity as a useful guide to capability retention.
Official Review of Submission23723 by Reviewer NzDL
Thank you for the clarifications and additional experiments. These address my main concerns, and I maintain my original rating.
Thank you for taking the time to review our response and maintaining your original rating. We are glad our additional experiments addressed your main concerns.
Rebuttal-Part 1
We thank the reviewer for the positive assessment and helpful suggestions.
(1) Editing and capability on a 1.5B model. Following your suggestion, we evaluated Qwen2.5-1.5B-Instruct on ZsRE at 500, 1,000, 2,000 and 3,000 edits. LessEdit retains its default ridge 1 with CapCov, EvoEdit uses ridge 1.
Table N1a. New 1.5B experiment: all four WILD metrics and stage capability scores.
Table N1b. All five capability tasks at 3,000 edits, with the unedited baseline.
At 3,000 edits, LessEdit improves all four WILD metrics over EvoEdit, raising mean WILD by 15.46 points and mean capability by 6.36 points. These results extend the paper's editing–capability balance to 1.5B.
(2) Editable space across model sizes. We recorded the free dimensions before each 100-edit batch. The 1.5B model retains free directions in every edited layer through its final solve and completes all 3,000 edits.
Table N2a. New 1.5B run: remaining dimensions after the stated number of completed edits. The 2,900 row is measured before the final 100-edit batch.
Table N2b. Original paper, Table 6: Qwen2.5-7B, layers 7–11, for comparison. The 9,900 row is measured before the final 100-edit batch.
The smaller model starts with fewer free directions in these configurations. In a separate extended 1.5B test using layers 7–9, layers 7 and 8 exhausted their free space after 6,500 committed edits, and the next batch was rejected. These trajectories quantify the available space for each configuration; CapCov guides update selection within that space, as in our original formulation.