acceptodds
Under review as a conference paper at ICLR 2027

Protect What Predictions Rely On: Decision-Aware Preservation for Massive Model Editing

Abstract

Editing thousands of facts in a language model requires deciding which aspects of its existing behavior to protect. Closed-form editors commonly make this decision using background activation statistics, which measure representation usage but do not directly measure predictive sensitivity. We propose Decision-Aware Preservation (DAP), a reusable preservation metric that augments activation statistics with prediction-gradient information. DAP weights Wikipedia probe keys by squared loss-gradient norms and adds the resulting matrix to the host editor's preservation term. This construction follows from an upper bound on loss derivatives at selected probe positions and retains the host's closed-form solution without changing target optimization. Experiments on GPT2-XL, GPT-J (6B), and Qwen2.5-7B across two benchmarks show improved locality at comparable edit success with up to 10,000 edits. The evaluated compositions gain up to 9.4 percentage points in locality success and 4.3 points in overall editing score over their hosts. Matched-probe controls on Qwen/PRUNE and GPT-J/RECT at 10,000 edits show that correct gradient–key pairing improves preservation beyond uniform or shuffled weights, with gains persisting after independent strength tuning. Fixed-spectrum interventions further characterize the role of protection orientation. Together, these results identify sensitivity-guided allocation as a practical way to improve preservation in closed-form model editing. Code is available at https://anonymous.4open.science/r/dap-editing-46F4/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model editing, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TH5NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decision-Aware Preservation (DAP), a method that augments activation-based preservation metrics with gradient-weighted probe statistics for large-scale knowledge editing. DAP incorporates prediction-sensitivity information into the preservation term while retaining the closed-form editing procedure. Experiments on three language models and two benchmarks show improvements in locality with comparable editing efficacy, supported by additional controlled analyses of gradient–key assignment, protection geometry, and hyperparameter sensitivity.

Critical strengths
  1. The paper proposes a coherent and technically motivated approach to improving locality in large-scale knowledge editing by incorporating prediction-sensitivity information into the preservation metric.
  2. Experiments on three models and two benchmarks demonstrate improvements across multiple editing metrics, with edit batches of up to 10,000 facts.
  3. The paper includes extensive controlled experiments, including gradient–key assignment comparisons, protection-geometry interventions, and hyperparameter sensitivity analyses.
Critical weaknesses
  1. Although several existing editors are discussed, comprehensive comparisons with standalone methods are primarily reported on Qwen2.5-7B. Broader comparisons across models and datasets would better establish the general advantage of DAP over existing editing methods.
  2. The main benchmark experiments cover three models, but several controlled analyses are restricted to particular model–dataset–editor combinations. Additional cross-model validation would strengthen the claim that the observed benefits arise from DAP's general design rather than specific configurations.
  3. The related work section does not adequately cover the recent advances in knowledge editing, particularly methods published in the past two years. Several relevant approaches to preservation, regularization, and large-scale editing are mentioned but not sufficiently discussed or compared. The authors should provide a more up-to-date and comprehensive review to better contextualize DAP and clarify its novelty relative to existing methods.
  4. The experimental evaluation covers a limited range of standalone editing methods, while most main results focus on combining DAP with MEMIT, RECT, and PRUNE. More comparisons with recent state-of-the-art editors and additional method compositions are needed to demonstrate the general applicability of DAP and establish whether its benefits consistently extend beyond the selected host methods.
Questions for authors
  1. Please clarify the hyperparameter-selection protocols for DAP and its host editors, including the search budgets and whether the baselines receive comparable tuning opportunities.
  2. Please define all evaluation metrics and abbreviations clearly at their first occurrence, especially when multiple locality metrics are reported.
  3. The paper would benefit from a more detailed discussion of the cases in which DAP provides limited gains, particularly when the host editor already achieves strong locality.
4 comments
AuthorsOpenReview ↗

Response to Reviewer TH5N (1/3): Cross-model contribution and controlled results

Thank you for asking for broader comparisons and clearer tuning protocols. DAP contributes a reusable preservation metric within existing editors. We apologize that the paper did not clearly separate compositional evidence from standalone comparisons. We report the complete available controlled results below and distinguish their scope.

Evidence correction. We withdraw the unverifiable non-host rows of Tables 3–4, the Uniform/Permuted rows of Table 9, and the original Table 10 values, together with claims that rely on them (abstract, §5.3 and §6). The controls and layerwise diagnostic below replace those sources; they do not reproduce the withdrawn 10,000-edit claims.

Weaknesses 1, 2 and 4: cross-model controls.

SettingProtectionγScoreESGStop-1top-5top-10DKL ↓
GPT-J / MEMIT / MCFHost083.8899.5490.2768.2963.7063.080.6904
Uniform7,50083.8099.5089.9468.3563.9263.320.6836
DAP7,50083.8899.4890.1268.4263.7863.270.6845
Qwen2.5-7B / MEMIT / zsREHost068.1189.0682.1848.4244.9248.742.8479
Uniform60,00069.1989.7082.3449.8346.1549.682.8316
DAP60,00068.9489.6082.2449.5145.8149.472.8379
Qwen2.5-7B / PRUNE / zsREHost073.5088.4483.2457.1552.6655.622.6367
Uniform300,00073.6588.2283.4857.4153.1855.932.6348
Loss600,00073.6388.2283.3257.4553.1855.962.6320
DAP300,00073.5788.2883.1657.3953.1755.962.6271
GPT-J / RECT / MCFHost / all selected arms079.0693.0069.1178.5975.0874.770.2841

The controls separate adding protection from choosing its weights. DAP improves all four locality readouts over the corresponding MEMIT hosts, but the gain over Uniform is mixed on GPT-J; on Qwen/MEMIT Uniform has the higher Score (69.19 vs 68.94). On Qwen/PRUNE DAP has the lowest DKL, while Uniform/Loss have slightly higher Scores. GPT-J/RECT selects the host endpoint for every weighting. These results support useful protection at fixed host settings, while stable superiority of gradient weighting over simpler weights remains an empirical question.

Fixed-host contribution at 10,000 edits. In the six MEMIT comparisons in submitted Tables 1–2, Table 7 lists unchanged host parameters. The displayed differences below are MEMIT+DAP minus MEMIT, in percentage points. They are transcribed from the submitted tables; the new 5k controls above have separate run-level verification.

Model / dataESGStop-1top-5top-10
GPT2-XL / zsRE0.0+0.8+3.2+2.4+2.1
GPT-J / zsRE+0.50.0+0.4+0.1+0.1
Qwen2.5-7B / zsRE0.0+0.3+1.0+0.6+0.4
GPT2-XL / MCF+0.5+0.5+1.4+1.6+1.5
GPT-J / MCF0.0−0.10.0+0.2+0.2
Qwen2.5-7B / MCF0.0+0.2+0.3+0.7+0.8

Top-1/5/10 overlap improves in 17 of 18 cells, with one tie; ES does not decrease. This isolates adding the DAP term at those host settings. It does not by itself isolate gradient weighting from uniform protection.

The standalone comparisons in Table 8 cover Qwen2.5-7B. Historical GPT-J baseline numbers lack a verified shared manifest/evaluator binding, so they are not presented as matched comparisons. A non-Qwen standalone comparison and a new host composition remain distinct missing experiments.

AuthorsOpenReview ↗

Response to Reviewer TH5N (2/3): Tuning, metrics and limited-gain cases

Matched protocol. These are supplementary 5,000-edit tests, with 2,000 development edits and 2,000 fixed Wikipedia probes. Within each setting, weightings share probes, targets, host parameters and layerwise trace calibration. Uniform uses wp=1w_p=1, Loss uses next-token surprisal wp=−log⁡p(yp∣xp)w_p=-\log p(y_p\mid x_p), and DAP uses wp=∥gp∥2w_p=\lVert g_p\rVert^2. Each matrix is symmetrized and scaled to the trace of the same host covariance. There is no loss-weight clipping. Selection maximizes development Score; ties select smaller γ\gamma. The grid, including the shared host endpoint, is {0, 7,500, 15,000, 30,000, 60,000} for both MEMIT settings and GPT-J/RECT, and {0, 75,000, 150,000, 300,000, 600,000} for Qwen/PRUNE.

Host μ\mu is 15,000 for MEMIT/RECT and 30,000 for matched Qwen/PRUNE, whose clip fraction is 0.5 and α=20\alpha=20. RECT keeps 40% of the update entries. Development and test case IDs are disjoint; the recorded construction also excludes development subjects from the main evaluation set. GPT-J edits layers 3–8 and Qwen layers 4–8. All arms start from the base model. These runs use PyTorch 2.9.1 / Transformers 4.51.3 and retained covariance assets; they are supplementary controlled runs, not a recovery of the original 10k execution.

ES/GS are case means of new-answer preference, using mean token NLL, on edit/paraphrase prompts. Locality pools prompts: top-kk is pre/post set overlap divided by kk, and DKL is DKL(ppre∥ppost)D_{KL}(p^{\mathrm{pre}}\Vert p^{\mathrm{post}}). Score is their harmonic mean using ES, GS and top-1. Percentages are shown except DKL.

Question 1: main-table tuning opportunities. Hosts in the 10k tables use default settings; +DAP selects on a disjoint 2k development slice. The original complete candidate logs have not been recovered, so the 10k tables are not described as equal-budget tuning. In 14/18 comparisons only the DAP term changes. The four exceptions are GPT-J/PRUNE/zsRE (clip 0.5→0.75), Qwen/RECT/MCF (μ 15k→30k), and Qwen/PRUNE on both datasets (μ 15k→30k). These settings are not independent-effect controls. The 5k table has explicit, equal grids for its compared weightings.

Question 2: metrics. ES is new-answer preference on edit prompts; GS uses paraphrases, averaging within a case and then over cases. Both compare teacher-forced mean token NLLs. Top-k and DKL use prompt pooling as defined above. LS instead asks about preservation against reference answers: token correctness for zsRE and old-versus-new answer preference for MCF. The submitted LS aggregation cannot yet be recovered, so it is excluded from our numerical argument. Existing full-answer diagnostic recomputations give Qwen/PRUNE case means 49.72/49.75/49.78 and token-pooled values 55.17/55.27/55.25 for Uniform/Loss/DAP. They illustrate why aggregation matters, rather than supplying recovered original LS values. The reviewer-cited 73.2−71.4 difference is 1.8, not 3.1; we withdraw that original gain claim rather than certify the unresolved GPT-J/RECT row. The submitted zsRE RECT+DAP row has a separate arithmetic mismatch: 76.9/67.5/79.2 imply Score 74.17, not 74.3.

Question 3: limited gains. GPT-J/MEMIT/MCF is close to its efficacy ceiling; the 5k matched controls remain near the host's Score. GPT-J/RECT selects γ=0 for every weighting, exactly recovering its host. On Qwen/MEMIT Uniform outperforms DAP at the selected points. Thus the added sensitivity metric is not the best operating point in every setting. Probe-count sensitivity and conditional probe bootstrap intervals do not establish independent-run stability.

Strength controls. The submitted GPT2-XL frontier tests host μ ∈ {1, 2, 2.5, 3, 4, 6, 8} × 10⁴ and DAP γ ∈ {10³, 10⁴, 3 × 10⁴} at host μ=2×104\mu=2\times10^4. One directly comparable point has ES/GS/DKL of 76.8/71.9/0.497 for DAP (γ=30,000\gamma=30,000), versus 74.3/69.1/0.521 for the host (μ=30,000\mu=30,000). This supports a favorable operating point within the tested grid, not dominance over every possible host tuning. Figure 6 varies probe count from 2k to 10k; it does not measure repeated sampling at a fixed count.

AuthorsOpenReview ↗

Response to Reviewer TH5N (3/3): Related work and diagnostic correction

Weakness 3: relation to recent work. DAP precomputes sensitivity in key space and inserts that metric into the closed-form solve. The intervention points below clarify the distinction.

MethodWhere it intervenesRelation to DAP
MEMIT, EMMETClosed-form solve with the activation metric Wiki-C; EMMET uses equality constraintsDAP adds a sensitivity-weighted metric inside the same solve; EMMET is compared in Table 8
AlphaEdit (2025)Projects the update onto the null space of preserved keysA hard constraint built from activation statistics; DAP is a soft metric built from prediction sensitivity. Compared in Table 8
ENCORE (2025)Early stopping of target optimization and a Frobenius-norm constraint on the updateActs on targets and on update size; DAP acts on the allocation of protection across keys
EAMET (2026), PMET, MEMIT-MergeEdit targets: key–residual alignment, or the distribution of targets across layersTarget-side changes; DAP changes the preservation metric
PRUNE, RECTPost-process the solved updateApplied after the solve; composed with DAP in Tables 1–2
RoSE (2026)Replaces covariance with identity and aggregates target gradients hierarchically for same-subject editingAddresses instruction-dependent generalization; DAP allocates soft protection from prediction sensitivity
Fisher regularization, Fisher mergingParameter-space sensitivityDAP relaxes sensitivity into a key-space matrix that fits the closed-form solve

RoSE is discussed from its primary paper; its focus is same-subject editing and instruction-dependent generalization. This conceptual comparison does not assert untested compatibility or competitive performance.

Replacement Table 10 protocol. Qwen2.5-7B/MEMIT; one batch of 100 zsRE edits; all edited layers 4–8, reported separately. The 512 held-out Wikipedia probes come from 512 articles and are distinct from the 2,000 construction probes. There is no cross-layer pooling. Each median and Spearman correlation uses the 512 probes for that layer. With base-model gradient gg, key kk and layer update Δ\Delta,

E=(g⊤Δk)2,B=∥g∥2∥Δk∥2,A=(ℓpost−ℓpre)2.E=(g^\top\Delta k)^2,\qquad B=\lVert g\rVert^2\lVert\Delta k\rVert^2,\qquad A=(\ell^{\mathrm{post}}-\ell^{\mathrm{pre}})^2.

AA uses the complete five-layer edit; B>0B>0 for all probes. The original 0.18–0.10 ratios and scale-dependent correlation rows are withdrawn, not retained as additional observations.

Layer (MEMIT update)Median E/Bρ(E,A)ρ(B,A)
41.57 × 10⁻⁴0.5620.655
51.25 × 10⁻⁴0.5960.654
61.35 × 10⁻⁴0.5690.659
71.39 × 10⁻⁴0.5820.686
81.35 × 10⁻⁴0.5840.692

The corrected ratios are approximately 10−410^{-4}. Since E/BE/B is squared cosine, this indicates a loose local bound, consistent in scale with the reviewer's check. The dimension-only isotropic reference is about 0.455/3584=1.3×10−40.455/3584=1.3\times10^{-4}; it is a geometric reference, not a model of the actual gradient/update distribution. The positive correlations evaluate risk information for these observed updates. They do not guarantee a ranking of arbitrary candidate updates or bound the finite multi-layer loss change.

Thank you for these experiments and candid clarifications. but weaknesses 1,2 still hasn't been well resolved, so I remain my opinion

Reviewer_tuPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Decision-Aware Preservation (DAP), a gradient-weighted preservation metric for closed-form model editing. DAP weights background activation keys by squared next-token loss-gradient norms and adds the resulting matrix to the host editor’s preservation term. Experiments cover three language models, two benchmarks, and up to 10,000 edits. The paper also includes matched-probe controls, independent strength tuning, and geometric interventions to investigate the source of the locality improvements.

Critical strengths

DAP offers a simple, compatible extension to existing editors. It introduces sensitivity information while retaining the host’s closed-form solve and target optimization, making the approach potentially useful across several editing methods.

Critical weaknesses

The incremental benefit of gradient weighting needs stronger practical validation. After independent strength tuning, Table 4 shows LS gains of 0.8–0.9 points over uniform weighting, with slightly lower ES/GS. These are encouraging results, but their stability across probe samples is unclear, and simpler loss-based weighting is not evaluated. The additional preprocessing cost is also unquantified. The MQuAKE retention experiment provides useful answer-level evidence, but broader evidence is needed to establish that the modest locality gains justify the additional computation.

Questions for authors
  1. Could you compare gradient-norm weighting with a simpler weighting based on next-token loss (surprisal), using the same probes and tuning budget?
  2. Could you clarify the main reason for preferring the isotropic relaxation? In particular, how should we understand the trade-off between preserving the host’s reusable solve and penalizing output directions that do not affect the selected loss to first order?
  3. Does reuse of Wiki-M across edit batches assume that each batch starts from the original model, or is it also intended for cumulative editing?
  4. Could you evaluate DAP on additional base-model families and larger parameter scales to assess whether its preservation benefits generalize across architectures and model sizes?
  5. Numerical consistency. Section 5.2 reports a 3.1-point Score improvement for RECT+DAP on GPT-J/Multi-CounterFact, whereas Table 2 gives 73.2 versus 71.4, corresponding to 1.8 points. Could you clarify which values are correct and check the related summaries for consistency?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tuPB (1/3): Matched loss-weighting control

Thank you for suggesting the loss-based control and asking us to quantify its cost. DAP measures sensitivity at the edited layer, whereas surprisal measures prediction error at the output. We apologize that the original discussion did not distinguish their information and practical effects clearly. We now give the complete controls and their protocol.

Evidence correction. We withdraw the unverifiable non-host rows of Tables 3–4, the Uniform/Permuted rows of Table 9, and the original Table 10 values, together with claims that rely on them (abstract, §5.3 and §6). The controls and layerwise diagnostic below replace those sources; they do not reproduce the withdrawn 10,000-edit claims.

Weakness 1 / Question 1.

SettingProtectionγScoreESGStop-1top-5top-10DKL ↓
GPT-J / MEMIT / MCFHost083.8899.5490.2768.2963.7063.080.6904
Uniform7,50083.8099.5089.9468.3563.9263.320.6836
DAP7,50083.8899.4890.1268.4263.7863.270.6845
Qwen2.5-7B / MEMIT / zsREHost068.1189.0682.1848.4244.9248.742.8479
Uniform60,00069.1989.7082.3449.8346.1549.682.8316
DAP60,00068.9489.6082.2449.5145.8149.472.8379
Qwen2.5-7B / PRUNE / zsREHost073.5088.4483.2457.1552.6655.622.6367
Uniform300,00073.6588.2283.4857.4153.1855.932.6348
Loss600,00073.6388.2283.3257.4553.1855.962.6320
DAP300,00073.5788.2883.1657.3953.1755.962.6271
GPT-J / RECT / MCFHost / all selected arms079.0693.0069.1178.5975.0874.770.2841

The controls separate adding protection from choosing its weights. DAP improves all four locality readouts over the corresponding MEMIT hosts, but the gain over Uniform is mixed on GPT-J; on Qwen/MEMIT Uniform has the higher Score (69.19 vs 68.94). On Qwen/PRUNE DAP has the lowest DKL, while Uniform/Loss have slightly higher Scores. GPT-J/RECT selects the host endpoint for every weighting. These results support useful protection at fixed host settings, while stable superiority of gradient weighting over simpler weights remains an empirical question.

Preprocessing cost and reuse. Retained float32 gradient-diagnostic wall times, including loading, are 96 s for GPT2-XL/L20X, 155 s for GPT-J/L20X, and 145 s for Qwen2.5-7B/H20. These include gradient extraction and diagnostic statistics, but exclude full matrix construction/saving. They are not matched incremental timings against Uniform/Loss. The complete cost comparison still needs matched timing. The practical reuse applies to batches reset to the same original model and to multiple γ\gamma choices; the solve retains its size and inference is unchanged.

AuthorsOpenReview ↗

Response to Reviewer tuPB (2/3): Protocol and sensitivity diagnostic

Matched protocol. These are supplementary 5,000-edit tests, with 2,000 development edits and 2,000 fixed Wikipedia probes. Within each setting, weightings share probes, targets, host parameters and layerwise trace calibration. Uniform uses wp=1w_p=1, Loss uses next-token surprisal wp=−log⁡p(yp∣xp)w_p=-\log p(y_p\mid x_p), and DAP uses wp=∥gp∥2w_p=\lVert g_p\rVert^2. Each matrix is symmetrized and scaled to the trace of the same host covariance. There is no loss-weight clipping. Selection maximizes development Score; ties select smaller γ\gamma. The grid, including the shared host endpoint, is {0, 7,500, 15,000, 30,000, 60,000} for both MEMIT settings and GPT-J/RECT, and {0, 75,000, 150,000, 300,000, 600,000} for Qwen/PRUNE.

Host μ\mu is 15,000 for MEMIT/RECT and 30,000 for matched Qwen/PRUNE, whose clip fraction is 0.5 and α=20\alpha=20. RECT keeps 40% of the update entries. Development and test case IDs are disjoint; the recorded construction also excludes development subjects from the main evaluation set. GPT-J edits layers 3–8 and Qwen layers 4–8. All arms start from the base model. These runs use PyTorch 2.9.1 / Transformers 4.51.3 and retained covariance assets; they are supplementary controlled runs, not a recovery of the original 10k execution.

ES/GS are case means of new-answer preference, using mean token NLL, on edit/paraphrase prompts. Locality pools prompts: top-kk is pre/post set overlap divided by kk, and DKL is DKL(ppre∥ppost)D_{KL}(p^{\mathrm{pre}}\Vert p^{\mathrm{post}}). Score is their harmonic mean using ES, GS and top-1. Percentages are shown except DKL.

Replacement Table 10 protocol. Qwen2.5-7B/MEMIT; one batch of 100 zsRE edits; all edited layers 4–8, reported separately. The 512 held-out Wikipedia probes come from 512 articles and are distinct from the 2,000 construction probes. There is no cross-layer pooling. Each median and Spearman correlation uses the 512 probes for that layer. With base-model gradient gg, key kk and layer update Δ\Delta,

E=(g⊤Δk)2,B=∥g∥2∥Δk∥2,A=(ℓpost−ℓpre)2.E=(g^\top\Delta k)^2,\qquad B=\lVert g\rVert^2\lVert\Delta k\rVert^2,\qquad A=(\ell^{\mathrm{post}}-\ell^{\mathrm{pre}})^2.

AA uses the complete five-layer edit; B>0B>0 for all probes. The original 0.18–0.10 ratios and scale-dependent correlation rows are withdrawn, not retained as additional observations.

Layer (MEMIT update)Median E/Bρ(E,A)ρ(B,A)
41.57 × 10⁻⁴0.5620.655
51.25 × 10⁻⁴0.5960.654
61.35 × 10⁻⁴0.5690.659
71.39 × 10⁻⁴0.5820.686
81.35 × 10⁻⁴0.5840.692

What information does the gradient add? For the MEMIT+DAP update on the same held-out probes, compare U=∥Δk∥2U=\lVert\Delta k\rVert^2, loss-weighted ℓpreU\ell^{\mathrm{pre}}U, and gradient-weighted ∥g∥2U\lVert g\rVert^2U against AA:

Layer (MEMIT+DAP update)UnweightedLossGradientGradient − loss [95% CI]
4−0.1170.5820.623+0.041 [0.003, 0.079]
5−0.1130.5720.625+0.054 [0.015, 0.090]
6−0.1520.5770.635+0.058 [0.020, 0.094]
7−0.1230.6440.687+0.042 [0.004, 0.079]
8−0.0860.6520.695+0.044 [0.007, 0.081]

The intervals are paired-probe bootstrap intervals (2,000 resamples, RNG seed 0), pointwise without multiplicity adjustment, conditional on this batch and probe pool. The MEMIT-host update has the same point-estimate ordering, but its gradient-minus-loss intervals include zero. Thus the diagnostic supports gradient magnitude as a risk signal; it does not substitute for independent repeats of matrix construction, tuning and editing.

AuthorsOpenReview ↗

Response to Reviewer tuPB (3/3): Design, practical scope and consistency

Question 2: why the isotropic relaxation. The exact local penalty ∑p(gp⊤Δkp)2\sum_p(g_p^\top\Delta k_p)^2 couples each key to an output direction. The formulation in Section B needs a batch-dependent correction and an additional N×NN\times N system per layer. DAP instead uses

gpgp⊤⪯∥gp∥2I,M∝N−1∑p∥gp∥2kpkp⊤,g_pg_p^\top\preceq\lVert g_p\rVert^2I,\qquad M\propto N^{-1}\sum_p\lVert g_p\rVert^2 k_pk_p^\top,

within Δ=RK⊤(KK⊤+μC+γM)−1\Delta=RK^\top(KK^\top+\mu C+\gamma M)^{-1}. The smallest scalar cc with cI⪰gpgp⊤cI\succeq g_pg_p^\top is ∥gp∥2\lVert g_p\rVert^2. This preserves each probe's sensitivity magnitude and the reusable solve, while charging directions orthogonal to gpg_p as well. The layerwise diagnostic above measures the resulting risk information. Cross-layer and higher-order effects lie outside the selected-position linearization.

Question 3: reset-batch reuse. Wiki-M is computed on the original model. Every evaluated batch starts there; the scaling study also restores base weights between chunks. Cumulative editing changes the sensitivity reference and is outside that reuse claim.

Question 4: model scope and stability. The paper covers GPT-2, GPT-J and Qwen2.5, 1.5B–7B, and the controls above cover GPT-J and Qwen. Fixed-budget independent probe repeats and an additional larger family are not present in these records. The probe-count sweep measures budget sensitivity, not independent-repeat variance.

Fixed-host contribution at 10,000 edits. In the six MEMIT comparisons in submitted Tables 1–2, Table 7 lists unchanged host parameters. The displayed differences below are MEMIT+DAP minus MEMIT, in percentage points. They are transcribed from the submitted tables; the new 5k controls above have separate run-level verification.

Model / dataESGStop-1top-5top-10
GPT2-XL / zsRE0.0+0.8+3.2+2.4+2.1
GPT-J / zsRE+0.50.0+0.4+0.1+0.1
Qwen2.5-7B / zsRE0.0+0.3+1.0+0.6+0.4
GPT2-XL / MCF+0.5+0.5+1.4+1.6+1.5
GPT-J / MCF0.0−0.10.0+0.2+0.2
Qwen2.5-7B / MCF0.0+0.2+0.3+0.7+0.8

Top-1/5/10 overlap improves in 17 of 18 cells, with one tie; ES does not decrease. This isolates adding the DAP term at those host settings. It does not by itself isolate gradient weighting from uniform protection.

MQuAKE adds answer-level evidence: at 10k edits original-answer retention is 18.4% for MEMIT and 22.0% for MEMIT+DAP (p = 9.1 × 10⁻⁴); at 2k it is 32.8% and 32.2% (p = 0.55). These are answer-retention results, separate from the gradient-versus-loss comparison.

Question 5: numerical consistency. The reviewer-cited values give 73.2 − 71.4 = 1.8 points, so the 3.1-point claim cannot be justified from that submitted pair. We withdraw that gain claim. The GPT-J/RECT rows also have unresolved source issues. In the submitted zsRE row, ES/GS/top-1 = 76.9/67.5/79.2 imply Score 74.17 (74.2 at one decimal), rather than the printed 74.3. This arithmetic correction does not certify the underlying row. The contribution evidence here uses the fixed-host MEMIT comparisons and the separately audited 5k records. The submitted LS aggregation is also unresolved, so no LS values are used as evidence in this response.

Reviewer_XQmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies preservation in large-scale model editing: existing closed-form editors allocate protection according to activation statistics, overlooking the effect of updates on predictions. The authors propose DAP, a plug-in preservation method for these editors, which allocates protection using prediction sensitivity and adds the resulting term to the host editor's preservation term, retaining its closed-form solution. Experiments on three models, two benchmarks, and three host editors show that DAP improves the preservation of existing knowledge while keeping edit success at a similar level, and the authors attribute this gain to a change in the direction of protection rather than merely its strength.

Critical strengths
  1. The proposed method is plug-in by design. Its preservation metric depends only on the unedited model, so it can be precomputed and reused across edit batches. It adds a single term to the host editor's preservation term, reduces exactly to the host when γ=0\gamma=0, and can be combined with methods such as PRUNE and RECT.
  2. This paper designs controlled experiments that isolate the source of the gain. Uniform weights and shuffled pairings, with fixed probes, trace, and strength, test the role of gradient-key pairing; independent strength tuning under an equal budget rules out gains from strength alone. In addition, fixed-spectrum interventions test the role of protection orientation itself.
Critical weaknesses
  1. The independent gain of the proposed method is limited. Its core addition is weighting the background second-moment statistics by gradient magnitude. The gains in the main tables include effects of the host combination; under independent strength tuning with an equal budget, locality success improves by only 0.8 and 0.9 points, with a slight decrease in edit success.

Updated (17 Nov 2026): The authors have withdrawn the rows of Tables 3–4 that supported the original gain figures. The new equal-budget controls do not resolve this weakness. On Qwen2.5-7B/MEMIT, Uniform weighting attains higher top-1/5/10 than DAP; on Qwen2.5-7B/PRUNE, loss weighting matches or exceeds DAP; on GPT-J/RECT every weighting selects γ=0\gamma=0. The authors describe the gain over Uniform as "mixed", which does not support the paper's central claim.

  1. This paper's surrogate objective and key diagnostic lack verifiable support. The upper-bound relaxation discards the gradient direction and does not guarantee the ranking among candidate updates. Table 10 reports a median E/B of 0.18, whereas my independent check, at reduced scale and under a protocol that could not be fully matched because the paper does not specify the layer selection and aggregation rule, obtains about 10−410^{-4}, raising a reproducibility concern that cannot be resolved without the paper's protocol and raw data.

Updated (17 Nov 2026): The authors have withdrawn the original Table 10 values and conducted new experiments with results also showing that the E/B value is around 10−410^{-4}, confirm that this weakness is a design flaw: the quantity actually optimized is a Cauchy relaxation whose magnitude exceeds the true loss response by about four orders of magnitude, so it exerts little constraint on the objective the paper cares about. The authors' new evidence that gradient weighting outperforms loss weighting in ρ(B,A)\rho(B,A) shows only that, within this already-loose surrogate, gradient information is slightly more informative; it does not address that the surrogate itself is too loose to justify the protection allocation. Therefore, this weakness remains critical.

Questions for authors
  1. The independent strength tuning in Table 4 covers only two configurations. Can you report the same comparison between DAP and uniform weights on the other host-dataset combinations in the main tables, together with the variance across multiple runs?

Updated (17 Nov 2026): The cross-configuration comparisons have been provided, while the variance remains unprovided. This question is partially open.

  1. Can you provide the layer selection rule, cross-layer aggregation rule, edit list, and probe list for Table 10, as well as the diagnostic script and per-probe results used to produce it? In my reduced-scale check, the rank correlations are close to those in Table 10, but the median E/B differs by about three orders of magnitude; the source of this gap remains to be identified.

Updated (17 Nov 2026): The authors supplied the replacement Table 10 protocol in full, together with the results. This question is answered.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XQmy (1/3): Independent gain and matched controls

Thank you for asking us to isolate the DAP term and verify the diagnostic. DAP adds a reusable prediction-sensitive metric inside the host solve. We apologize that the original presentation mixed the supporting evidence and left the diagnostic protocol incomplete. The tables and protocols below specify both comparisons.

Evidence correction. We withdraw the unverifiable non-host rows of Tables 3–4, the Uniform/Permuted rows of Table 9, and the original Table 10 values, together with claims that rely on them (abstract, §5.3 and §6). The controls and layerwise diagnostic below replace those sources; they do not reproduce the withdrawn 10,000-edit claims.

Weakness 1 / Question 1: equal-budget controls.

SettingProtectionγScoreESGStop-1top-5top-10DKL ↓
GPT-J / MEMIT / MCFHost083.8899.5490.2768.2963.7063.080.6904
Uniform7,50083.8099.5089.9468.3563.9263.320.6836
DAP7,50083.8899.4890.1268.4263.7863.270.6845
Qwen2.5-7B / MEMIT / zsREHost068.1189.0682.1848.4244.9248.742.8479
Uniform60,00069.1989.7082.3449.8346.1549.682.8316
DAP60,00068.9489.6082.2449.5145.8149.472.8379
Qwen2.5-7B / PRUNE / zsREHost073.5088.4483.2457.1552.6655.622.6367
Uniform300,00073.6588.2283.4857.4153.1855.932.6348
Loss600,00073.6388.2283.3257.4553.1855.962.6320
DAP300,00073.5788.2883.1657.3953.1755.962.6271
GPT-J / RECT / MCFHost / all selected arms079.0693.0069.1178.5975.0874.770.2841

The controls separate adding protection from choosing its weights. DAP improves all four locality readouts over the corresponding MEMIT hosts, but the gain over Uniform is mixed on GPT-J; on Qwen/MEMIT Uniform has the higher Score (69.19 vs 68.94). On Qwen/PRUNE DAP has the lowest DKL, while Uniform/Loss have slightly higher Scores. GPT-J/RECT selects the host endpoint for every weighting. These results support useful protection at fixed host settings, while stable superiority of gradient weighting over simpler weights remains an empirical question.

AuthorsOpenReview ↗

Response to Reviewer XQmy (2/3): Selection protocol and fixed-host evidence

Matched protocol. These are supplementary 5,000-edit tests, with 2,000 development edits and 2,000 fixed Wikipedia probes. Within each setting, weightings share probes, targets, host parameters and layerwise trace calibration. Uniform uses wp=1w_p=1, Loss uses next-token surprisal wp=−log⁡p(yp∣xp)w_p=-\log p(y_p\mid x_p), and DAP uses wp=∥gp∥2w_p=\lVert g_p\rVert^2. Each matrix is symmetrized and scaled to the trace of the same host covariance. There is no loss-weight clipping. Selection maximizes development Score; ties select smaller γ\gamma. The grid, including the shared host endpoint, is {0, 7,500, 15,000, 30,000, 60,000} for both MEMIT settings and GPT-J/RECT, and {0, 75,000, 150,000, 300,000, 600,000} for Qwen/PRUNE.

Host μ\mu is 15,000 for MEMIT/RECT and 30,000 for matched Qwen/PRUNE, whose clip fraction is 0.5 and α=20\alpha=20. RECT keeps 40% of the update entries. Development and test case IDs are disjoint; the recorded construction also excludes development subjects from the main evaluation set. GPT-J edits layers 3–8 and Qwen layers 4–8. All arms start from the base model. These runs use PyTorch 2.9.1 / Transformers 4.51.3 and retained covariance assets; they are supplementary controlled runs, not a recovery of the original 10k execution.

ES/GS are case means of new-answer preference, using mean token NLL, on edit/paraphrase prompts. Locality pools prompts: top-kk is pre/post set overlap divided by kk, and DKL is DKL(ppre∥ppost)D_{KL}(p^{\mathrm{pre}}\Vert p^{\mathrm{post}}). Score is their harmonic mean using ES, GS and top-1. Percentages are shown except DKL.

Fixed-host contribution at 10,000 edits. In the six MEMIT comparisons in submitted Tables 1–2, Table 7 lists unchanged host parameters. The displayed differences below are MEMIT+DAP minus MEMIT, in percentage points. They are transcribed from the submitted tables; the new 5k controls above have separate run-level verification.

Model / dataESGStop-1top-5top-10
GPT2-XL / zsRE0.0+0.8+3.2+2.4+2.1
GPT-J / zsRE+0.50.0+0.4+0.1+0.1
Qwen2.5-7B / zsRE0.0+0.3+1.0+0.6+0.4
GPT2-XL / MCF+0.5+0.5+1.4+1.6+1.5
GPT-J / MCF0.0−0.10.0+0.2+0.2
Qwen2.5-7B / MCF0.0+0.2+0.3+0.7+0.8

Top-1/5/10 overlap improves in 17 of 18 cells, with one tie; ES does not decrease. This isolates adding the DAP term at those host settings. It does not by itself isolate gradient weighting from uniform protection.

Strength controls. The submitted GPT2-XL frontier tests host μ ∈ {1, 2, 2.5, 3, 4, 6, 8} × 10⁴ and DAP γ ∈ {10³, 10⁴, 3 × 10⁴} at host μ=2×104\mu=2\times10^4. One directly comparable point has ES/GS/DKL of 76.8/71.9/0.497 for DAP (γ=30,000\gamma=30,000), versus 74.3/69.1/0.521 for the host (μ=30,000\mu=30,000). This supports a favorable operating point within the tested grid, not dominance over every possible host tuning. Figure 6 varies probe count from 2k to 10k; it does not measure repeated sampling at a fixed count.

Independent probe-pool repeats at fixed budget remain necessary to quantify run-to-run variability.

AuthorsOpenReview ↗

Response to Reviewer XQmy (3/3): Replacement Table 10 and surrogate evidence

Weakness 2 / Question 2.

Replacement Table 10 protocol. Qwen2.5-7B/MEMIT; one batch of 100 zsRE edits; all edited layers 4–8, reported separately. The 512 held-out Wikipedia probes come from 512 articles and are distinct from the 2,000 construction probes. There is no cross-layer pooling. Each median and Spearman correlation uses the 512 probes for that layer. With base-model gradient gg, key kk and layer update Δ\Delta,

E=(g⊤Δk)2,B=∥g∥2∥Δk∥2,A=(ℓpost−ℓpre)2.E=(g^\top\Delta k)^2,\qquad B=\lVert g\rVert^2\lVert\Delta k\rVert^2,\qquad A=(\ell^{\mathrm{post}}-\ell^{\mathrm{pre}})^2.

AA uses the complete five-layer edit; B>0B>0 for all probes. The original 0.18–0.10 ratios and scale-dependent correlation rows are withdrawn, not retained as additional observations.

Layer (MEMIT update)Median E/Bρ(E,A)ρ(B,A)
41.57 × 10⁻⁴0.5620.655
51.25 × 10⁻⁴0.5960.654
61.35 × 10⁻⁴0.5690.659
71.39 × 10⁻⁴0.5820.686
81.35 × 10⁻⁴0.5840.692

The corrected ratios are approximately 10−410^{-4}. Since E/BE/B is squared cosine, this indicates a loose local bound, consistent in scale with the reviewer's check. The dimension-only isotropic reference is about 0.455/3584=1.3×10−40.455/3584=1.3\times10^{-4}; it is a geometric reference, not a model of the actual gradient/update distribution. The positive correlations evaluate risk information for these observed updates. They do not guarantee a ranking of arbitrary candidate updates or bound the finite multi-layer loss change.

What information does the gradient add? For the MEMIT+DAP update on the same held-out probes, compare U=∥Δk∥2U=\lVert\Delta k\rVert^2, loss-weighted ℓpreU\ell^{\mathrm{pre}}U, and gradient-weighted ∥g∥2U\lVert g\rVert^2U against AA:

Layer (MEMIT+DAP update)UnweightedLossGradientGradient − loss [95% CI]
4−0.1170.5820.623+0.041 [0.003, 0.079]
5−0.1130.5720.625+0.054 [0.015, 0.090]
6−0.1520.5770.635+0.058 [0.020, 0.094]
7−0.1230.6440.687+0.042 [0.004, 0.079]
8−0.0860.6520.695+0.044 [0.007, 0.081]

The intervals are paired-probe bootstrap intervals (2,000 resamples, RNG seed 0), pointwise without multiplicity adjustment, conditional on this batch and probe pool. The MEMIT-host update has the same point-estimate ordering, but its gradient-minus-loss intervals include zero. Thus the diagnostic supports gradient magnitude as a risk signal; it does not substitute for independent repeats of matrix construction, tuning and editing.

Thank you sincerely for the detailed responses and the additional matched controls. Your corrected Table 10 protocol and the new equal-budget comparisons clarify the situation considerably: they pin down the scale of the surrogate objective, and they show how DAP's weighting compares with uniform and loss weighting under a matched budget.

However, these results confirm rather than resolve my proposed two weaknesses. The surrogate's looseness is a property of the method's design, and the new controls show mixed or unfavorable comparisons against simpler weightings on several configurations. Across-run variability also remains unquantified.

Given these weaknesses, I prefer to remain at the original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.