acceptodds
Under review as a conference paper at ICLR 2027

Editing Where It Matters Less: Capability-Aware Sequential Model Editing

18 views

Abstract

Sequential model editing offers a practical way to update outdated knowledge in large language models, but many targeted edits applied over time can accumulate interference and degrade general behavior. Null-space-based editors improve stability by restricting parameter updates to directions that approximately preserve protected key–value associations, yet they typically select among the remaining writable directions with isotropic penalties. This creates a metric mismatch: admissible directions can still differ greatly in their effect on natural activations. We propose LessEdit, a capability-aware free-subspace editor for sequential model editing. At each edit, LessEdit excludes protected and previously used directions, projects the uncentered second moment of general-domain activations into the remaining free subspace, and uses the projected covariance as an anisotropic regularizer that favors low-drift updates, while keeping the reduced optimization closed form. Across 3,000 sequential edits on LLaMA-3 (8B) using ZsRE, CounterFact, and WikiBigEdit, LessEdit outperforms AlphaEdit by 13.9 points on average in LLM-as-a-judge reliability and generalization. The edited models retain stronger base capabilities, with a mean score of 60.3 compared to 46.7 for AlphaEdit. Our code is available at https://anonymous.4open.science/r/LessEdit.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language models, sequential model editing, null-space optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NzDLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LessEdit, a capability-aware method for sequential knowledge editing. The authors note that existing null-space methods restrict updates to feasible directions without considering their varying impacts on general capabilities. To address this, LessEdit combines dynamic null-space constraints with anisotropic regularization based on the second-order moments of reference activations to reduce interference. Overall, the motivation is clear, the method and derivations are complete, and updates are computed efficiently in closed form. Experiments across multiple benchmarks use the WILD protocol, reflecting actual generation ability better than teacher-forcing. Ablations and directional analyses further verify the regularization, showing improved sequential editing while maintaining base capabilities.

Critical strengths
  1. Realistic evaluation setup. Instead of relying solely on traditional teacher-forcing, the paper uses the WILD framework to better capture open-ended post-edit performance and avoid probability-based evaluation bias. Evaluating reliability and generalization across diverse prompt formats also makes the results more comprehensive and convincing.

  2. Clear motivation and well-validated core mechanism. Prior null-space methods use threshold truncation to form an approximate null space, so updates within the editable subspace can still perturb original activations and outputs—a directional impact largely overlooked by existing work. To minimize such perturbations, the authors use the second-order moments of reference activations to quantify directional sensitivity and introduce capability-aware anisotropic regularization. The method is well-motivated with complete derivations, and the proposed CapCov mechanism is sufficiently validated through ablations, fixed-state directional analyses, and comparisons with alternative regularizers

Critical weaknesses
  1. Lack of evaluation on smaller-scale models. Current experiments focus primarily on LLaMA-3-8B and Qwen2.5-7B, leaving the method's applicability to smaller models underexplored. Since smaller models typically have lower-dimensional FFN weight matrices and LessEdit continuously removes previously used directions during sequential editing, the editable subspace may shrink more rapidly as edits accumulate, potentially limiting subsequent knowledge injection. It is recommended to evaluate smaller models and analyze how the editable subspace dimensionality evolves across model scales, as well as how this affects editing success rates and base capability preservation, to better establish the method's scalability.

  2. Unclear link between CapCov's local perturbation metric and actual capability retention. Using reference activation second-order moments to measure FFN output perturbation serves only as a local proxy; smaller local shifts do not necessarily ensure preserved final task performance due to subsequent layers and non-linearities. Although Appendix D.1 examines decoder-block and logit-level shifts, it lacks direct validation against actual capability retention. Correlating CapCov metrics with downstream performance changes (e.g., MMLU, GSM8K) across editing stages would better justify this proxy objective.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal-Part 1

We thank the reviewer for the positive assessment and helpful suggestions.

W1. Evaluation on smaller models, shrinking editable space, and its effect on editing quality and capability retention.

(1) Editing and capability on a 1.5B model. Following your suggestion, we evaluated Qwen2.5-1.5B-Instruct on ZsRE at 500, 1,000, 2,000 and 3,000 edits. LessEdit retains its default ridge 11 with CapCov, EvoEdit uses ridge 1.

Table N1a. New 1.5B experiment: all four WILD metrics and stage capability scores.

MethodEditsQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalizationMMLUGSM8K
LessEdit50065.6055.6067.2042.8061.3752.00
EvoEdit50067.6056.4063.0044.0061.3654.00
LessEdit1,00065.0053.7063.2046.7060.2454.00
EvoEdit1,00062.5054.1062.0042.8060.4456.00
LessEdit2,00061.0049.7555.3537.7057.6656.50
EvoEdit2,00066.2555.5541.8028.4056.1647.00
LessEdit3,00058.6347.0050.9732.3352.3147.50
EvoEdit3,00029.3023.7043.9330.1748.6029.50

Table N1b. All five capability tasks at 3,000 edits, with the unedited baseline.

MethodEditsMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KFive-task mean
Unedited062.3443.8445.4745.5061.5051.73
LessEdit3,00052.3132.4045.3237.0047.5042.91
EvoEdit3,00048.6019.2447.3938.0029.5036.55

At 3,000 edits, LessEdit improves all four WILD metrics over EvoEdit, raising mean WILD by 15.46 points and mean capability by 6.36 points. These results extend the paper's editing–capability balance to 1.5B.

(2) Editable space across model sizes. We recorded the free dimensions before each 100-edit batch. The 1.5B model retains free directions in every edited layer through its final solve and completes all 3,000 edits.

Table N2a. New 1.5B run: remaining dimensions after the stated number of completed edits. The 2,900 row is measured before the final 100-edit batch.

Completed editsLayer 4Layer 5Layer 6
04,5855,5705,542
5004,0855,0705,042
1,0003,5854,5704,542
2,0002,5863,5713,542
2,9001,6862,6712,642

Table N2b. Original paper, Table 6: Qwen2.5-7B, layers 7–11, for comparison. The 9,900 row is measured before the final 100-edit batch.

Completed editsLayer 7Layer 8Layer 9Layer 10Layer 11Mean retained fraction (%)
013,47113,00111,82013,33714,350100.00
1,00012,47112,00110,82012,33713,35092.39
5,0008,4748,0036,8228,3389,35061.97
9,0004,4884,0152,8294,3445,35231.60
9,9003,5903,1191,9323,4454,45424.77

The smaller model starts with fewer free directions in these configurations. In a separate extended 1.5B test using layers 7–9, layers 7 and 8 exhausted their free space after 6,500 committed edits, and the next batch was rejected. These trajectories quantify the available space for each configuration; CapCov guides update selection within that space, as in our original formulation.

AuthorsOpenReview ↗

Rebuttal-Part 2

W2. Does the local perturbation metric track actual downstream capability across editing stages?

We measured benchmark performance alongside reference-output cost at successive checkpoints. Applying the paper's reference quadratic to cumulative effective weight changes Dl=Wl,t−Wl,0D_l=W_{l,t}-W_{l,0} gives

Ct=∑ltr⁡(DlΣ0,lDl⊤).C_t=\sum_l\operatorname{tr}(D_l\Sigma_{0,l}D_l^\top).

Here Σ0,l\Sigma_{0,l} remains fixed. Cost measures reference-output change, not computation time.

(1) Across editing stages. The following CounterFact results use the original LLaMA-3-8B runs, including the historical AlphaEdit ridge-1 configuration. Cost is estimated with 64 Hutchinson probes.

Table N3. Reference-output cost and all five capability tasks across six stages.

MethodEditsTotal costMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KFive-task mean
Unedited0069.4769.2650.7256.0073.5063.79
AlphaEdit1000.152869.3370.6450.2156.5073.0063.94
EvoEdit1000.112369.3569.9950.3856.5072.5063.74
LessEdit1000.059869.2869.8350.4256.0073.0063.71
AlphaEdit5000.858168.6269.0147.4355.0075.5063.11
EvoEdit5000.592668.9268.5449.1856.0073.5063.23
LessEdit5000.338369.0371.5550.7755.0071.5063.57
AlphaEdit1,0001.936966.8366.1946.3554.5074.0061.57
EvoEdit1,0001.226867.8370.4248.7354.5073.5063.00
LessEdit1,0000.744968.5072.0250.7854.5071.5063.46
AlphaEdit1,5003.253964.4664.0047.2854.0069.5059.85
EvoEdit1,5001.897267.2969.6748.3055.0071.5062.35
LessEdit1,5001.220567.6867.3250.9755.5073.0062.89
AlphaEdit2,0004.835561.1260.4344.6852.0065.0056.65
EvoEdit2,0002.620266.6168.6447.5954.0071.0061.57
LessEdit2,0001.770567.0570.1149.2855.0071.0062.49
AlphaEdit3,0008.480348.3333.4137.7943.5041.5040.91
EvoEdit3,0004.066864.5265.2245.8054.0071.0060.11
LessEdit3,0002.985864.7664.2547.6454.5069.0060.03

Table N4. Correlation between cost and capability decline from the unedited baseline. These are descriptive correlations across 18 checkpoints from three trajectories with varying edit counts.

Capability declineSpearman correlationPearson correlation
MMLU0.9860.960
IFEval0.7810.903
TruthfulQA-MC20.8270.942
ARC-Challenge0.8970.911
GSM8K0.6700.870
Five-task mean0.9810.927

Cost correlates with decline in every benchmark, with Spearman correlations of 0.986 for MMLU and 0.981 for the five-task mean. At each matched stage from 500 to 3,000 edits, lower cost also agrees with higher MMLU.

(2) At a fixed edit count. We additionally computed exact costs after 10,000 ZsRE edits under two random orders, using shared initial weights and reference statistics. The three configurations were selected before these order runs. The isotropic variant replaces CapCov by the identity; its total coefficient is 22.

Table N5. New LLaMA-3-8B comparisons at 10,000 edits.

Order seedConfigurationTotal costMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KFive-task mean
20261121LessEdit: ridge 4 + CapCov7.656859.1158.2344.0749.5058.0053.78
20261121EvoEdit: ridge 414.145851.6044.4440.7248.5028.5042.75
20261121Isotropic: coefficient 1.99930.295827.5425.0446.3330.002.0026.18
20261122LessEdit: ridge 4 + CapCov7.666659.8358.9540.8647.5054.5052.33
20261122EvoEdit: ridge 413.993352.1951.8741.7246.0030.5044.46
20261122Isotropic: coefficient 1.99928.487030.3628.9944.9132.003.5027.95

LessEdit lowers cost relative to EvoEdit by 45.87% and 45.21%, alongside 11.03- and 7.87-point gains in mean capability. These comparisons hold edit count fixed, complementing the stage correlations.

(3) On the smaller model. We also computed exact costs for all eight Qwen1.5B checkpoints in Table N1a.

Table N6. New 1.5B experiment: exact reference-output cost.

EditsLessEditEvoEdit
50035.240341.3733
1,00073.615887.3493
2,000162.6978195.2864
3,000294.0071391.1105

At 3,000 edits, LessEdit has 24.83% lower cost, together with the editing and capability gains in Tables N1a–b. These measurements support reference sensitivity as a useful guide to capability retention.

Reviewer_NzDLOpenReview ↗

Official Review of Submission23723 by Reviewer NzDL

Thank you for the clarifications and additional experiments. These address my main concerns, and I maintain my original rating.

Thank you for taking the time to review our response and maintaining your original rating. We are glad our additional experiments addressed your main concerns.

Reviewer_fVQPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the accumulation of interference in sequential model editing: null-space-based editors select among the remaining writable directions with isotropic penalties, overlooking how differently these directions affect general-domain activations. The authors propose LessEdit, which excludes protected and previously used directions, projects the second moment of general-domain activations into the remaining free subspace, and uses it as an anisotropic regularizer while keeping the closed-form solution. Experiments with 3,000 sequential edits on three datasets with LLaMA-3-8B show that LessEdit improves edit reliability and generalization while retaining the model's general capabilities.

Critical strengths
  1. The proposed method has a closed-form solution. The anisotropic regularizer acts only on the remaining free subspace, so the reduced problem still has a unique minimizer; the dual-branch structure can be merged at deployment, adding no inference overhead.
  2. This paper designs broad and targeted experiments: on LLaMA-3-8B, it compares against eleven baselines on three datasets under two prompt formats, and isolates the role of the anisotropic regularizer with an isotropic control that replaces the covariance with an identity matrix; the results also hold on Qwen2.5-7B.
Critical weaknesses
  1. Both the novelty and the gain of the proposed method are limited. It keeps EvoEdit's null-space constraints and replaces the isotropic penalty with a regularizer built from the activation second moment that AlphaEdit already uses to construct its null space; the paper itself notes that the objective reduces to the isotropic EvoEdit formulation when β=0\beta=0 and λeff=1\lambda_{\text{eff}}=1. Relative to EvoEdit, WILD scores improve by only 4.3 points on average and by 0.4 to 3.3 points on CounterFact, with no variance reported.

Updated (17 Nov 2026): The authors clarify the relation to AlphaEdit and EvoEdit: the latter two define and shrink the writable space, while LessEdit selects updates within it by directional sensitivity. This locates the increment, which I still consider limited; however, the cross-host and equal-trace controls show that directional weighting is not specific to a single host, so this concern is addressed.

  1. The paper's claim of capability preservation lacks support. Capability awareness is the core motivation of the method, yet its average base-capability score equals EvoEdit's at 60.3. In the ablation, removing the capability covariance lowers the base-capability score to 47.6, well below EvoEdit's roughly 60.0 on ZsRE, and this gap is not explained.

Updated (17 Nov 2026): The authors attribute the 47.6 ablation result to a ridge mismatch and explain the gap I cited with matched-ridge controls; the correlation between reference-output cost and capability decline further supports the core mechanism. This concern is addressed.

Questions for authors
  1. Can you repeat multiple editing trajectories on the three datasets, using different edit orders or random seeds, and report the mean and variance of each WILD metric for LessEdit and EvoEdit?

Updated (17 Nov 2026): Answered.

  1. In the ablation, removing the capability covariance while retaining the original norm penalty yields an average base-capability score of 47.6, whereas EvoEdit reaches about 60.0 on ZsRE in Table 1. What exactly differs between these two configurations, and what accounts for this gap?

Updated (17 Nov 2026): Answered.

5 comments
AuthorsOpenReview ↗

Rebuttal-Part 1

We thank the reviewer for the constructive comments.

W1. Limited novelty and gains over EvoEdit; the activation second moment is already used by AlphaEdit.

AlphaEdit uses reference statistics to define a near-null space, EvoEdit further restricts it as edits accumulate. LessEdit uses the remaining directional sensitivity to choose updates within this space, while retaining the closed-form solution. The original results show 4.3 points higher mean WILD than EvoEdit, with 60.3 mean capability for both, stronger editing at comparable retained capability.

Table F1a. Original Table 1: editing after 3,000 edits, one order.

DatasetOriginal-paper variantQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalization
ZsRELessEdit with CapCov82.466.574.355.7
ZsREEvoEdit77.464.960.248.1
CounterFactLessEdit with CapCov83.469.961.750.7
CounterFactEvoEdit82.566.660.250.3
WikiBigEditLessEdit with CapCov83.976.887.879.3
WikiBigEditEvoEdit81.973.282.173.1

To test whether directional weighting adds value beyond projection, we compared qI+Σ^qI+\widehat\Sigma with (q+1)I(q+1)I inside AlphaEdit, using q∈1,10q\in\\{1,10\\} on all three datasets. Here Σ^\widehat\Sigma is CapCov's normalized projected second moment. Its mean eigenvalue is one, giving equal-trace penalties. Both retain AlphaEdit's fixed projector and historical-key penalty, other settings match within each pair.

Table F2a. Transfer to AlphaEdit: 3,000 edits, one run per configuration.

DatasetEditing variantRidge coefficientRidge + metric termQA Context: ReliabilityQA Context: GeneralizationNo Context: ReliabilityNo Context: GeneralizationMean WILD score
ZsREAlphaEdit with CapCov11·I + Σ̂77.2368.0062.5049.3064.26
ZsREAlphaEdit with isotropic metric12·I (isotropic)73.8063.8757.2347.5360.61
ZsREAlphaEdit with CapCov1010·I + Σ̂72.2362.2060.0051.8061.56
ZsREAlphaEdit with isotropic metric1011·I (isotropic)70.1361.4031.3328.5347.85
CounterFactAlphaEdit with CapCov11·I + Σ̂86.5370.0759.1349.9066.41
CounterFactAlphaEdit with isotropic metric12·I (isotropic)83.3366.9053.1747.5362.73
CounterFactAlphaEdit with CapCov1010·I + Σ̂81.2062.0057.7347.7362.17
CounterFactAlphaEdit with isotropic metric1011·I (isotropic)82.2363.6055.0746.5061.85
WikiAlphaEdit with CapCov11·I + Σ̂83.4775.3387.2379.0381.27
WikiAlphaEdit with isotropic metric12·I (isotropic)73.0068.5075.9368.9071.58
WikiAlphaEdit with CapCov1010·I + Σ̂78.0371.6780.7372.6375.77
WikiAlphaEdit with isotropic metric1011·I (isotropic)78.4772.2378.8068.6374.53

CapCov improves mean WILD in all six pairs by 0.32–13.71 points, supporting directional weighting beyond the choice of writable space.

AuthorsOpenReview ↗

Rebuttal-Part 2

W2. Does capability awareness help if both methods average 60.3, and why does removing CapCov reduce the score to 47.6?

Tables F1a–b show better editing at comparable mean capability. Q2 explains the 47.6 ablation result.

Table F1b. Original Table 1: capability scores for the same runs.

DatasetOriginal-paper variantMMLUIFEval (as reported)TruthfulQA-MC2ARC-ChallengeGSM8K
ZsRELessEdit with CapCov63.265.548.053.069.0
ZsREEvoEdit64.768.848.952.065.5
CounterFactLessEdit with CapCov64.864.447.654.569.0
CounterFactEvoEdit64.565.645.854.071.0
WikiBigEditLessEdit with CapCov64.569.049.553.569.0
WikiBigEditEvoEdit65.471.148.554.065.0

Table F2b. Capability scores for the twelve transfer runs in Table F2a.

DatasetEditing variantRidge coefficientRidge + metric termMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KMean capability score
ZsREAlphaEdit with CapCov11·I + Σ̂63.9365.5347.1853.0071.0060.13
ZsREAlphaEdit with isotropic metric12·I (isotropic)58.9356.4543.5347.5057.0052.68
ZsREAlphaEdit with CapCov1010·I + Σ̂67.1871.8047.7152.5071.0062.04
ZsREAlphaEdit with isotropic metric1011·I (isotropic)66.3569.0447.5353.5070.5061.39
CounterFactAlphaEdit with CapCov11·I + Σ̂63.7464.6643.9054.5068.0058.96
CounterFactAlphaEdit with isotropic metric12·I (isotropic)56.7247.2939.9345.0055.0048.79
CounterFactAlphaEdit with CapCov1010·I + Σ̂67.0268.6448.6454.5073.5062.46
CounterFactAlphaEdit with isotropic metric1011·I (isotropic)66.3568.3947.2155.5070.5061.59
WikiAlphaEdit with CapCov11·I + Σ̂64.7669.8646.7052.5070.0060.76
WikiAlphaEdit with isotropic metric12·I (isotropic)61.1360.5838.6254.0057.5054.37
WikiAlphaEdit with CapCov1010·I + Σ̂67.6470.4249.0554.5071.5062.62
WikiAlphaEdit with isotropic metric1011·I (isotropic)67.2268.7047.9855.0068.5061.48

Mean capability improves by 0.65–10.17 points in all six pairs, alongside the WILD gains. Benefits vary by task.

Q1. Repeat editing runs on all three datasets and report the mean and variance of each WILD metric.

We ran different 3,000-edit orders per method and datase. Seeds are 20261101, 20261102, and 20261103 for ZsRE, CounterFact, and WikiBigEdit, respectively. These runs are separate from the original-paper results.

Table F3a. Mean [sample variance] across orders. Variance uses unrounded scores and denominator n−1=1n-1=1, in squared percentage points.

DatasetEditing variantQA Context: ReliabilityQA Context: GeneralizationNo Context: ReliabilityNo Context: GeneralizationMean WILD scoreMean capability score
ZsRELessEdit with CapCov; ridge 180.42 [4.6006]64.62 [4.2050]77.72 [4.8050]59.38 [2.9606]70.53 [0.0050]59.69 [1.1253]
ZsREEvoEdit; ridge 477.47 [2.8800]65.60 [5.3356]68.78 [159.0139]55.37 [79.3800]66.80 [19.1684]59.77 [0.0458]
CounterFactLessEdit with CapCov; ridge 187.63 [0.0800]70.95 [0.6050]58.93 [0.4356]47.98 [0.4672]66.38 [0.0672]60.34 [1.1001]
CounterFactEvoEdit; ridge 486.67 [2.1356]67.98 [2.3472]59.70 [0.0800]49.58 [0.0139]65.98 [0.7200]59.77 [0.0183]
WikiLessEdit with CapCov; ridge 181.72 [1.4450]73.37 [2.5689]85.83 [0.5689]76.13 [8.8200]79.26 [2.6642]61.01 [0.4143]
WikiEvoEdit; ridge 479.85 [15.1250]71.27 [27.8756]84.28 [17.8006]73.62 [13.3472]77.25 [18.1503]60.08 [0.2315]

Table F3b. Mean-WILD difference on each order: LessEdit minus EvoEdit (points).

DatasetOriginal orderShared shuffled order
ZsRE+6.88+0.58
CounterFact+1.18−0.39
Wiki+6.18−2.16

Mean WILD gains across the two tested orders are 3.73, 0.39, and 2.01 points on ZsRE, CounterFact, and WikiBigEdit. Rankings reverse on the CounterFact and WikiBigEdit shuffles.

AuthorsOpenReview ↗

Rebuttal-Part 3

Q2. What differs between LessEdit without CapCov and EvoEdit, and what accounts for the 47.6-versus-about-60 capability gap?

Removing CapCov leaves LessEdit's effective ridge at ρ=1\rho=1, while EvoEdit uses 4. Their implementations also differ: LessEdit tracks used directions with a basis and solves reduced normal equations; EvoEdit updates a dense projector and uses a Woodbury solve. They share the initial projector, reference statistics, and target-optimization settings.

We varied the ridge within EvoEdit and compared both implementations at ridge 4. All configurations below exclude CapCov.

Table F4a. Original references and new controls: editing after 3,000 ZsRE edits.

Result originEditing variantRidge coefficientQA Context: ReliabilityQA Context: GeneralizationNo Context: ReliabilityNo Context: GeneralizationMean WILD score
Original paper, Table 2LessEdit without CapCovρ =182.165.560.944.563.3
Original paper, Table 1EvoEdit477.464.960.248.162.65
New rebuttal experimentEvoEditρ =180.3064.2368.2750.7765.89
New rebuttal experimentEvoEdit478.6767.2359.8749.0763.71
New rebuttal experimentLessEdit without CapCov478.2766.6064.6752.8065.58

Table F4b. Capability scores for the same configurations.

Result originEditing variantRidge coefficientMMLUIFEval (origin-specific)TruthfulQA-MC2ARC-ChallengeGSM8KMean capability score
Original paper, Table 2LessEdit without CapCovρ =154.952.342.043.046.047.6
Original paper, Table 1EvoEdit464.768.848.952.065.559.98
New rebuttal experimentEvoEditρ =154.3854.3543.1247.0048.0049.37
New rebuttal experimentEvoEdit464.4168.2345.9852.5067.0059.62
New rebuttal experimentLessEdit without CapCov465.0168.5446.1651.0066.5059.44

Increasing EvoEdit’s ridge to 4 raises mean capability by 10.25 points, with a 2.18-point WILD decrease. At ridge 4, the implementations have similar capability (59.44 versus 59.62), with 1.88 higher WILD for LessEdit without CapCov. These controls establish a large ridge effect. Without a matched no-CapCov run at ρ, we cannot separate ridge and implementation contributions to the historical gap.

Reviewer_eSykOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LessEdit augments evolving null-space constraints with CapCov, an anisotropic penalty obtained by projecting and normalizing the reference second moment within the free subspace. It computes a closed-form effective update and allocates it between anchored and editable FFN branches. Experiments on three sequential editing benchmarks and two model families assess WILD reliability and generalization together with base-capability preservation.

Critical strengths
  1. The WILD evaluation tests autoregressive answers under both QA Context and No Context. The comparison with teacher-forced evaluation exposes substantial differences for several baselines. Five base-capability benchmarks provide additional evidence about behavior beyond the edited facts.
  2. The reduced objective and closed-form solution are clearly specified. The analysis states the conditions for a unique effective update and distinguishes near-null projection from exact null-space preservation. It also limits the historical-key preservation claim to stored activations.
  3. Appendix D.1 compares updates at 30 fixed layer states using equal-trace isotropic, spectrum-permutation, and same-response minimum-norm controls. These comparisons support a directional benefit from CapCov beyond shrinking the update norm.
Critical weaknesses
  1. The need for two branches remains unclear. With identical frozen key maps, WAk+WEk=(WA+WE)kW_Ak+W_Ek=(W_A+W_E)k, and the optimized branch allocation reduces to λeff∥ΔLessEdit,t∥F2\lambda_{\mathrm{eff}}\|\Delta_{\mathrm{LessEdit},t}\|_F^2. Redistributing the same effective update cannot strengthen preservation of K0K_0 or stored edit keys. The paper acknowledges this equivalence, yet Table 2 reports mean WILD scores of 69.7 for Full and 65.1 for Single. The merge test in Appendix E.1 checks a fixed checkpoint; it does not establish that these two editing trajectories use matched targets, keys, projectors, regularization, and numerical settings. The ablation therefore does not isolate a benefit of branch allocation. The additional parameters and peak memory reported for the editing stage in Table 13 make this distinction practically relevant.

  2. CapCov has a direct algebraic connection to the existing preservation penalty. For a nonempty free subspace, ct>10−12c_t>10^{-12}, and the same reference keys as in Eq. (6), the identity fallback is inactive and

    =\frac{\beta}{n_0c_t}\|\Delta_tP_{t-1}K_0\|_F^2.$$ Thus, the method reinstates the preservation penalty of Eq. (3) on the projected effective update, with a coefficient that changes across editing steps. This can help because near-null projection retains residual reference energy. However, the contribution appears to be the combination of this existing penalty with evolving constraints and normalization. The directional controls establish useful local effects, but they do not resolve this novelty concern. The paper appropriately calls the penalty a local surrogate; its mathematical analysis does not establish a quantitative connection to task-level capability preservation.
  3. The 10K results do not fully separate CapCov from changes in regularization strength. Table 2 and Appendix D.1 already provide isotropic and fixed-state controls. However, Appendix B.4 raises λeff\lambda_{\mathrm{eff}} to 4 for LLaMA ZsRE10K, while Table 5 does not report EvoEdit and isotropic controls tuned under the same search budget. Appendix C.4 also reports that LessEdit's aggregate total reference-output cost at the LLaMA 10K checkpoint is 1.28 times EvoEdit's. Lower cost per squared update norm therefore cannot, by itself, explain the capability gain in this setting. Most evaluations use a single editing trajectory, leaving the sensitivity to edit order unclear.

Questions for authors
  1. Which settings differ between Full and Single in Table 2? With identical initial effective weights, targets, keys, projector updates, CapCov, λeff\lambda_{\mathrm{eff}}, and numerical settings, do their stepwise effective updates agree? What explains the WILD gap?
  2. What does CapCov add beyond applying the preservation penalty in Eq. (3) to the projected update? Does normalization by ctc_t improve results over a fixed coefficient on that same penalty?
  3. Under equal hyperparameter search budgets, how much of the 10K advantage remains over tuned EvoEdit and isotropic controls? Please report variation across edit orders and apply Appendix C.4's total reference-output cost diagnostic to these new controls.
6 comments
AuthorsOpenReview ↗

Rebuttal-Part 1

We thank the reviewer for the careful comments and suggestions.

W1. Why use two branches when their sum is equivalent to one projection? Does the Full–Single ablation isolate a branch benefit?

With identical frozen key maps, the two branches yield one effective projection. Their separate penalties reduce to the effective ridge in Eq. (10), allowing a single tensor to implement the same CapCov objective and constraints.

Table E1. 300 CounterFact edits, matched code and effective ridge. Each repeat starts unedited. Timing includes the first editing call and excludes loading, evaluation, and saving.

LessEdit implementationRepeat 1 time (s)Repeat 2 time (s)Repeat 3 time (s)Peak allocated memory in each repeat (GiB)
Two output branches1223.021229.851228.7447.28
Single output tensor1176.791170.271100.8842.69

The single-tensor form uses 4.59 GiB less peak memory. Merging the two-branch form at deployment also restores the original FFN structure.

W2. CapCov reuses the preservation quadratic in Eq. (3). What is the contribution, and what connects it to capability retention?

We agree with the algebraic connection. For Σ0=K0K0⊤/n0\Sigma_0=K_0K_0^\top/n_0 and a nonempty free space with the identity fallback inactive,

βct∣ΔtPt−1Σ01/2∣F2=βn0ct∣ΔtPt−1K0∣F2.\frac{\beta}{c_t}\\|\Delta_tP_{t-1}\Sigma_0^{1/2}\\|_F^2 =\frac{\beta}{n_0c_t}\\|\Delta_tP_{t-1}K_0\\|_F^2.

The contribution is to use this sensitivity within the evolving writable space. Near-null thresholding leaves directions with unequal reference-output costs. CapCov weights these directions as the space changes, while retaining the closed-form solution.

We tested 4I+Σ^4I+\widehat\Sigma, 4I4I, and 5I5I in the same implementation on the same 3,000-edit ZsRE order. Each setting follows its own trajectory. Since tr⁡(Σ^)=rt\operatorname{tr}(\widehat\Sigma)=r_t, 5I5I is the equal-trace control at a given free rank.

Table E2a. Direction and strength controls: editing scores.

LessEdit regularizerQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalizationWILD mean
CapCov with ridge 4: 4I+Σ^4I+\widehat\Sigma77.4765.1368.9756.4066.99
Isotropic, same scalar ridge: 4I4I78.2766.6064.6752.8065.58
Isotropic, equal trace: 5I5I72.9761.6058.8748.0760.38

Table E2b. Capability scores for the same runs. New evaluations use the mean of prompt-level and instruction-level strict IFEval accuracy.

LessEdit regularizerMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KCapability mean
CapCov with ridge 4: 4I+Σ^4I+\widehat\Sigma66.4967.9847.3852.5072.0061.27
Isotropic, same scalar ridge: 4I4I65.0168.5446.1651.0066.5059.44
Isotropic, equal trace: 5I5I63.4966.6043.4949.5066.0057.81

CapCov improves all four editing metrics and all five capability scores over 5I5I, with mean gains of 6.62 and 3.45 points. The objective remains a local surrogate for capability retention.

W3. Separate CapCov from coefficient tuning at 10,000 edits, report order variation, and apply the total-cost diagnostic.

Q3 reports the six candidates, three orders, and total costs. Across orders, selected LessEdit gains 7.87–11.29 capability points over EvoEdit with 45.21–46.08% lower total reference-output cost, editing differences vary by order.

Q1. Which Full–Single settings differ? Do matched effective updates agree, and what explains the WILD gap?

With identical effective weights, targets, keys, free basis, and regularization, the effective updates agree in exact arithmetic. Applying the same reduced solution through one or two branches at 30 fixed layer states gave a maximum weight difference of 1.06×10−61.06\times10^{-6} relative to the 64-bit update norm.

The historical runs share requests and nominal regularization but differ in execution paths, GPUs, and generation/scoring procedures. We will clarify this scope in the ablation discussion.

AuthorsOpenReview ↗

Rebuttal-Part 2

Q2. Does dynamic normalization improve on a fixed coefficient on the same preservation term?

On the same 3,000-edit ZsRE order, we compared dynamic β/ct,l\beta/c_{t,l} with each layer's first-batch coefficient β/c1,l\beta/c_{1,l} held fixed. Both retain CapCov, with ridge ρ\rho and β=1\beta=1.

Table E3a. Dynamic versus fixed scaling: editing scores.

CapCov coefficientQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalizationWILD mean
Dynamic: β/ct,l\beta/c_{t,l}81.9366.0776.1758.1770.58
Fixed after the first batch: β/c1,l\beta/c_{1,l}84.1067.6376.5758.1771.62

Table E3b. Capability scores for the same runs.

CapCov coefficientMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KCapability mean
Dynamic: β/ct,l\beta/c_{t,l}63.5168.5744.6052.5065.5058.94
Fixed after the first batch: β/c1,l\beta/c_{1,l}63.4265.7244.2754.5068.5059.28

Fixed scaling scores 1.03 WILD and 0.35 capability points higher in this run. Both versions use directional weighting; this comparison shows no mean gain from updating its scale at every step.

Q3. What remains under equal search budgets, across edit orders, and under the total reference-output-cost diagnostic?

(1) Candidate selection. We test two candidates per family. The isotropic control replaces CapCov with II in LessEdit; 1 and 2 abbreviate ρ\rho and ρ+1\rho+1.

Within each family, we select the largest (WILD mean + capability mean)/2, breaking ties by capability. Candidate counts are matched. We select on the original 10,000-edit ZsRE order and keep the settings fixed for both new orders.

Table E4a. Candidate selection for the new orders.

Method familyPenalty on the writable updateWILD editing meanCapability meanSelection scoreSelected
LessEdit1I+Σ^1I+\widehat\Sigma60.7031.3246.01No
LessEdit4I+Σ^4I+\widehat\Sigma68.2153.8761.04Yes
EvoEdit1I1I7.7123.5815.65No
EvoEdit4I4I66.3942.5854.49Yes
LessEdit, isotropic control2I2I49.9725.1237.55Yes
LessEdit, isotropic control5I5I27.5730.9929.28No

The isotropic control selects 2I2I: its higher editing score gives a joint score of 37.55 versus 29.28 for 5I5I.

Table E4b. Editing scores; bold labels mark selected configurations.

Method familyPenaltyQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalizationWILD mean
LessEdit1I+Σ^1I+\widehat\Sigma68.1956.0866.7851.7560.70
LessEdit (selected)4I+Σ^4I+\widehat\Sigma73.7961.6476.7560.6668.21
EvoEdit1I1I4.753.6112.939.557.71
EvoEdit (selected)4I4I74.7361.3172.7356.8066.39
LessEdit, isotropic control (selected)2I2I54.4543.1258.4343.8849.97
LessEdit, isotropic control5I5I30.4624.8831.1723.7627.57

Table E4c. Capability scores for the same candidates.

Method familyPenaltyMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KCapability mean
LessEdit1I+Σ^1I+\widehat\Sigma36.2537.9837.8735.009.5031.32
LessEdit (selected)4I+Σ^4I+\widehat\Sigma58.6658.2642.9149.0060.5053.87
EvoEdit1I1I25.4117.3949.6225.500.0023.58
EvoEdit (selected)4I4I50.7748.4941.1541.5031.0042.58
LessEdit, isotropic control (selected)2I2I24.9924.4441.6731.503.0025.12
LessEdit, isotropic control5I5I41.3537.8036.2831.508.0030.99

Selected LessEdit exceeds selected EvoEdit by 1.82 WILD and 11.29 capability points. The equal-trace comparison is 4I+Σ^4I+\widehat\Sigma versus 5I5I, the selected 2I2I control answers the separate comparison between the best candidates in each family.

AuthorsOpenReview ↗

Rebuttal-Part 3

(2) New edit orders. Methods share requests and order within each seed.

Table E5a. Editing scores by order, summaries give mean ± sample standard deviation.

Editing orderConfigurationQA Context reliabilityQA Context generalizationNo Context reliabilityNo Context generalizationWILD mean
Original orderLessEdit with CapCov, ridge 473.7961.6476.7560.6668.21
Original orderEvoEdit, ridge 474.7361.3172.7356.8066.39
Original orderIsotropic LessEdit, coefficient ρ+154.4543.1258.4343.8849.97
Shuffle 20261121LessEdit with CapCov, ridge 476.5064.1668.7555.9166.33
Shuffle 20261121EvoEdit, ridge 475.1661.6877.1060.2168.54
Shuffle 20261121Isotropic LessEdit, coefficient ρ+158.4547.8554.5042.0750.72
Shuffle 20261122LessEdit with CapCov, ridge 474.5362.6776.2160.8068.55
Shuffle 20261122EvoEdit, ridge 475.5163.2971.8656.3066.74
Shuffle 20261122Isotropic LessEdit, coefficient ρ+165.7653.4062.0647.6557.22
Three-order summaryLessEdit with CapCov, ridge 474.94 ± 1.4062.82 ± 1.2773.90 ± 4.4759.12 ± 2.7867.70 ± 1.20
Three-order summaryEvoEdit, ridge 475.13 ± 0.3962.09 ± 1.0573.90 ± 2.8157.77 ± 2.1367.22 ± 1.15
Three-order summaryIsotropic LessEdit, coefficient ρ+159.55 ± 5.7448.12 ± 5.1558.33 ± 3.7844.53 ± 2.8552.64 ± 3.99

Table E5b. Capability scores, with the same summary convention.

Editing orderConfigurationMMLUIFEvalTruthfulQA-MC2ARC-ChallengeGSM8KCapability mean
Original orderLessEdit with CapCov, ridge 458.6658.2642.9149.0060.5053.87
Original orderEvoEdit, ridge 450.7748.4941.1541.5031.0042.58
Original orderIsotropic LessEdit, coefficient ρ+124.9924.4441.6731.503.0025.12
Shuffle 20261121LessEdit with CapCov, ridge 459.1158.2344.0749.5058.0053.78
Shuffle 20261121EvoEdit, ridge 451.6044.4440.7248.5028.5042.75
Shuffle 20261121Isotropic LessEdit, coefficient ρ+127.5425.0446.3330.002.0026.18
Shuffle 20261122LessEdit with CapCov, ridge 459.8358.9540.8647.5054.5052.33
Shuffle 20261122EvoEdit, ridge 452.1951.8741.7246.0030.5044.46
Shuffle 20261122Isotropic LessEdit, coefficient ρ+130.3628.9944.9132.003.5027.95
Three-order summaryLessEdit with CapCov, ridge 459.20 ± 0.5958.48 ± 0.4142.61 ± 1.6348.67 ± 1.0457.67 ± 3.0153.33 ± 0.86
Three-order summaryEvoEdit, ridge 451.52 ± 0.7248.27 ± 3.7241.20 ± 0.5045.33 ± 3.5530.00 ± 1.3243.26 ± 1.04
Three-order summaryIsotropic LessEdit, coefficient ρ+127.63 ± 2.6926.16 ± 2.4744.30 ± 2.3931.17 ± 1.042.83 ± 0.7626.42 ± 1.43

LessEdit retains more capability on every order. On the two new orders, its mean capability gain is 9.45 points, with similar WILD means (67.44 versus 67.64). Editing rankings vary by order.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your constructive feedback and for the discussion so far. Following your suggestions, we conducted additional experiments and clarified several points in our responses.

  1. Directional weighting. Across three datasets and two ridge settings in AlphaEdit, CapCov achieved higher mean WILD and base-capability scores than the equal-trace isotropic control in all six comparisons. We also clarified that, under matched settings, a single tensor implements the same effective update as the two branches.
  2. Robustness at 10,000 edits. We tested two settings per method on the original ZsRE order and kept the selected settings fixed for two new orders. Averaged over those two orders, LessEdit scored 9.45 points higher than EvoEdit on the five-task capability mean, with similar mean WILD scores (67.44 vs. 67.64). Its total reference-output cost was 45–46% lower on each order.
  3. Model scale and capability retention. After 3,000 ZsRE edits on Qwen2.5-1.5B, LessEdit exceeded EvoEdit by 15.5 points in mean WILD and 6.4 points in mean base capability. Across 18 LLaMA CounterFact checkpoints, higher cumulative reference-output cost was associated with a larger decline in the five-task capability mean.

As the discussion deadline approaches, we would be grateful to hear whether these results address your concerns. We would be happy to clarify any remaining points. Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.