- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Decision-Aware Preservation (DAP), a method that augments activation-based preservation metrics with gradient-weighted probe statistics for large-scale knowledge editing. DAP incorporates prediction-sensitivity information into the preservation term while retaining the closed-form editing procedure. Experiments on three language models and two benchmarks show improvements in locality with comparable editing efficacy, supported by additional controlled analyses of gradient–key assignment, protection geometry, and hyperparameter sensitivity.
Critical strengths
- The paper proposes a coherent and technically motivated approach to improving locality in large-scale knowledge editing by incorporating prediction-sensitivity information into the preservation metric.
- Experiments on three models and two benchmarks demonstrate improvements across multiple editing metrics, with edit batches of up to 10,000 facts.
- The paper includes extensive controlled experiments, including gradient–key assignment comparisons, protection-geometry interventions, and hyperparameter sensitivity analyses.
Critical weaknesses
- Although several existing editors are discussed, comprehensive comparisons with standalone methods are primarily reported on Qwen2.5-7B. Broader comparisons across models and datasets would better establish the general advantage of DAP over existing editing methods.
- The main benchmark experiments cover three models, but several controlled analyses are restricted to particular model–dataset–editor combinations. Additional cross-model validation would strengthen the claim that the observed benefits arise from DAP's general design rather than specific configurations.
- The related work section does not adequately cover the recent advances in knowledge editing, particularly methods published in the past two years. Several relevant approaches to preservation, regularization, and large-scale editing are mentioned but not sufficiently discussed or compared. The authors should provide a more up-to-date and comprehensive review to better contextualize DAP and clarify its novelty relative to existing methods.
- The experimental evaluation covers a limited range of standalone editing methods, while most main results focus on combining DAP with MEMIT, RECT, and PRUNE. More comparisons with recent state-of-the-art editors and additional method compositions are needed to demonstrate the general applicability of DAP and establish whether its benefits consistently extend beyond the selected host methods.
Questions for authors
- Please clarify the hyperparameter-selection protocols for DAP and its host editors, including the search budgets and whether the baselines receive comparable tuning opportunities.
- Please define all evaluation metrics and abbreviations clearly at their first occurrence, especially when multiple locality metrics are reported.
- The paper would benefit from a more detailed discussion of the cases in which DAP provides limited gains, particularly when the host editor already achieves strong locality.
4 comments
Response to Reviewer TH5N (2/3): Tuning, metrics and limited-gain cases
Matched protocol. These are supplementary 5,000-edit tests, with 2,000 development edits and 2,000 fixed Wikipedia probes. Within each setting, weightings share probes, targets, host parameters and layerwise trace calibration. Uniform uses , Loss uses next-token surprisal , and DAP uses . Each matrix is symmetrized and scaled to the trace of the same host covariance. There is no loss-weight clipping. Selection maximizes development Score; ties select smaller . The grid, including the shared host endpoint, is {0, 7,500, 15,000, 30,000, 60,000} for both MEMIT settings and GPT-J/RECT, and {0, 75,000, 150,000, 300,000, 600,000} for Qwen/PRUNE.
Host is 15,000 for MEMIT/RECT and 30,000 for matched Qwen/PRUNE, whose clip fraction is 0.5 and . RECT keeps 40% of the update entries. Development and test case IDs are disjoint; the recorded construction also excludes development subjects from the main evaluation set. GPT-J edits layers 3–8 and Qwen layers 4–8. All arms start from the base model. These runs use PyTorch 2.9.1 / Transformers 4.51.3 and retained covariance assets; they are supplementary controlled runs, not a recovery of the original 10k execution.
ES/GS are case means of new-answer preference, using mean token NLL, on edit/paraphrase prompts. Locality pools prompts: top- is pre/post set overlap divided by , and DKL is . Score is their harmonic mean using ES, GS and top-1. Percentages are shown except DKL.
Question 1: main-table tuning opportunities. Hosts in the 10k tables use default settings; +DAP selects on a disjoint 2k development slice. The original complete candidate logs have not been recovered, so the 10k tables are not described as equal-budget tuning. In 14/18 comparisons only the DAP term changes. The four exceptions are GPT-J/PRUNE/zsRE (clip 0.5→0.75), Qwen/RECT/MCF (μ 15k→30k), and Qwen/PRUNE on both datasets (μ 15k→30k). These settings are not independent-effect controls. The 5k table has explicit, equal grids for its compared weightings.
Question 2: metrics. ES is new-answer preference on edit prompts; GS uses paraphrases, averaging within a case and then over cases. Both compare teacher-forced mean token NLLs. Top-k and DKL use prompt pooling as defined above. LS instead asks about preservation against reference answers: token correctness for zsRE and old-versus-new answer preference for MCF. The submitted LS aggregation cannot yet be recovered, so it is excluded from our numerical argument. Existing full-answer diagnostic recomputations give Qwen/PRUNE case means 49.72/49.75/49.78 and token-pooled values 55.17/55.27/55.25 for Uniform/Loss/DAP. They illustrate why aggregation matters, rather than supplying recovered original LS values. The reviewer-cited 73.2−71.4 difference is 1.8, not 3.1; we withdraw that original gain claim rather than certify the unresolved GPT-J/RECT row. The submitted zsRE RECT+DAP row has a separate arithmetic mismatch: 76.9/67.5/79.2 imply Score 74.17, not 74.3.
Question 3: limited gains. GPT-J/MEMIT/MCF is close to its efficacy ceiling; the 5k matched controls remain near the host's Score. GPT-J/RECT selects γ=0 for every weighting, exactly recovering its host. On Qwen/MEMIT Uniform outperforms DAP at the selected points. Thus the added sensitivity metric is not the best operating point in every setting. Probe-count sensitivity and conditional probe bootstrap intervals do not establish independent-run stability.
Strength controls. The submitted GPT2-XL frontier tests host μ ∈ {1, 2, 2.5, 3, 4, 6, 8} × 10⁴ and DAP γ ∈ {10³, 10⁴, 3 × 10⁴} at host . One directly comparable point has ES/GS/DKL of 76.8/71.9/0.497 for DAP (), versus 74.3/69.1/0.521 for the host (). This supports a favorable operating point within the tested grid, not dominance over every possible host tuning. Figure 6 varies probe count from 2k to 10k; it does not measure repeated sampling at a fixed count.
Response to Reviewer TH5N (3/3): Related work and diagnostic correction
Weakness 3: relation to recent work. DAP precomputes sensitivity in key space and inserts that metric into the closed-form solve. The intervention points below clarify the distinction.
| Method | Where it intervenes | Relation to DAP |
|---|---|---|
| MEMIT, EMMET | Closed-form solve with the activation metric Wiki-C; EMMET uses equality constraints | DAP adds a sensitivity-weighted metric inside the same solve; EMMET is compared in Table 8 |
| AlphaEdit (2025) | Projects the update onto the null space of preserved keys | A hard constraint built from activation statistics; DAP is a soft metric built from prediction sensitivity. Compared in Table 8 |
| ENCORE (2025) | Early stopping of target optimization and a Frobenius-norm constraint on the update | Acts on targets and on update size; DAP acts on the allocation of protection across keys |
| EAMET (2026), PMET, MEMIT-Merge | Edit targets: key–residual alignment, or the distribution of targets across layers | Target-side changes; DAP changes the preservation metric |
| PRUNE, RECT | Post-process the solved update | Applied after the solve; composed with DAP in Tables 1–2 |
| RoSE (2026) | Replaces covariance with identity and aggregates target gradients hierarchically for same-subject editing | Addresses instruction-dependent generalization; DAP allocates soft protection from prediction sensitivity |
| Fisher regularization, Fisher merging | Parameter-space sensitivity | DAP relaxes sensitivity into a key-space matrix that fits the closed-form solve |
RoSE is discussed from its primary paper; its focus is same-subject editing and instruction-dependent generalization. This conceptual comparison does not assert untested compatibility or competitive performance.
Replacement Table 10 protocol. Qwen2.5-7B/MEMIT; one batch of 100 zsRE edits; all edited layers 4–8, reported separately. The 512 held-out Wikipedia probes come from 512 articles and are distinct from the 2,000 construction probes. There is no cross-layer pooling. Each median and Spearman correlation uses the 512 probes for that layer. With base-model gradient , key and layer update ,
uses the complete five-layer edit; for all probes. The original 0.18–0.10 ratios and scale-dependent correlation rows are withdrawn, not retained as additional observations.
| Layer (MEMIT update) | Median E/B | ρ(E,A) | ρ(B,A) |
|---|---|---|---|
| 4 | 1.57 × 10⁻⁴ | 0.562 | 0.655 |
| 5 | 1.25 × 10⁻⁴ | 0.596 | 0.654 |
| 6 | 1.35 × 10⁻⁴ | 0.569 | 0.659 |
| 7 | 1.39 × 10⁻⁴ | 0.582 | 0.686 |
| 8 | 1.35 × 10⁻⁴ | 0.584 | 0.692 |
The corrected ratios are approximately . Since is squared cosine, this indicates a loose local bound, consistent in scale with the reviewer's check. The dimension-only isotropic reference is about ; it is a geometric reference, not a model of the actual gradient/update distribution. The positive correlations evaluate risk information for these observed updates. They do not guarantee a ranking of arbitrary candidate updates or bound the finite multi-layer loss change.
Thank you for these experiments and candid clarifications. but weaknesses 1,2 still hasn't been well resolved, so I remain my opinion
Response to Reviewer TH5N (1/3): Cross-model contribution and controlled results
Thank you for asking for broader comparisons and clearer tuning protocols. DAP contributes a reusable preservation metric within existing editors. We apologize that the paper did not clearly separate compositional evidence from standalone comparisons. We report the complete available controlled results below and distinguish their scope.
Evidence correction. We withdraw the unverifiable non-host rows of Tables 3–4, the Uniform/Permuted rows of Table 9, and the original Table 10 values, together with claims that rely on them (abstract, §5.3 and §6). The controls and layerwise diagnostic below replace those sources; they do not reproduce the withdrawn 10,000-edit claims.
Weaknesses 1, 2 and 4: cross-model controls.
The controls separate adding protection from choosing its weights. DAP improves all four locality readouts over the corresponding MEMIT hosts, but the gain over Uniform is mixed on GPT-J; on Qwen/MEMIT Uniform has the higher Score (69.19 vs 68.94). On Qwen/PRUNE DAP has the lowest DKL, while Uniform/Loss have slightly higher Scores. GPT-J/RECT selects the host endpoint for every weighting. These results support useful protection at fixed host settings, while stable superiority of gradient weighting over simpler weights remains an empirical question.
Fixed-host contribution at 10,000 edits. In the six MEMIT comparisons in submitted Tables 1–2, Table 7 lists unchanged host parameters. The displayed differences below are MEMIT+DAP minus MEMIT, in percentage points. They are transcribed from the submitted tables; the new 5k controls above have separate run-level verification.
Top-1/5/10 overlap improves in 17 of 18 cells, with one tie; ES does not decrease. This isolates adding the DAP term at those host settings. It does not by itself isolate gradient weighting from uniform protection.
The standalone comparisons in Table 8 cover Qwen2.5-7B. Historical GPT-J baseline numbers lack a verified shared manifest/evaluator binding, so they are not presented as matched comparisons. A non-Qwen standalone comparison and a new host composition remain distinct missing experiments.