acceptodds
Desk rejected at ICLR 2027

Rethinking Covariance in Knowledge Editing: Estimation, Update Geometry, and Mechanisms

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Many locate-then-edit knowledge editing (KE) methods like ROME and AlphaEdit promise efficient factual updates by modifying a small set of model weights, and they rely on precomputed covariance statistics to preserve existing knowledge during parameter updates. However, the cost of estimating these statistics is generally overlooked and the mechanisms through which they influence editing remain poorly understood. In this paper, we present a systematic investigation connecting covariance estimation, update geometry, and editor-specific sensitivity. We find that extending calibration contexts increases estimation costs but yields no consistent gains in editing performance or general ability, revealing a mismatch between computational overhead and downstream benefits. We further examine how editing performance varies across calibration sequence length distributions. Beyond estimation, we show that covariance defines an anisotropic geometry whose effects on editing depend on how each method translates it into parameter updates. Mechanism analysis identifies update magnitude and directional structure as distinct factors in covariance sensitivity, with their relative importance varying across editors: MEMIT's sensitivity to calibration length is largely attributable to update magnitude, whereas AlphaEdit's behavior depends on the subspace it selects. Together, these findings show that covariance's role in knowledge editing is governed by the interaction between its geometry and the editor's update rule.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge Editing, Covariance Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_HSJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how covariance estimation and geometry affect knowledge editing, finding that longer calibration contexts increase cost without consistently improving performance. Mechanistic analyses suggest that MEMIT’s sensitivity is largely related to update magnitude, while AlphaEdit’s scale sensitivity depends on projection subspace selection.

Critical strengths
  1. It examines covariance, an often-overlooked component of knowledge editing, and investigates the relationship between its estimation cost and editing performance.
  2. Using MEMIT and AlphaEdit as case studies, it investigates the roles of update magnitude and subspace selection through norm-matching and fixed-projector controls.
Critical weaknesses
  1. At 1,000 edits, AlphaEdit’s ES/Rewrite values across L=512/1024/2048 are 34.95/99.10/96.60 in Table 16 versus 97.60/97.60/97.60 in Table 19. Different aggregation formulas alone cannot explain these discrepancies.

  2. Acknowledged token-budget and sequence-construction confounds limit causal attribution, while large AlphaEdit changes on C4 challenge the stability claim. Main mechanistic controls use one model, 100 edits, and a fixed editing seed, limiting generalization.

Questions for authors
  1. Could the authors reconcile the configurations, covariance caches, and evaluation protocols underlying Tables 16 and 19? An overview figure connecting the analyses would also improve clarity.

  2. Given the derived scale invariance, which implementation factors cause EMMET’s observed sensitivity, and could controlled experiments verify their contributions beyond the possibilities already discussed?

  3. Given the explanation for nonzero scores without updates, could the authors explicitly report no-edit baselines and verify whether the main conclusions hold for individual metrics?

Reviewer_e1tWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the role of calibration second-moment statistics in locate-then-edit knowledge editing. It studies how maximum calibration length, corpus choice, and sequence construction affect estimation cost and editing performance, then intervenes on covariance scale, spectrum, and directional structure. It further analyzes realized parameter updates and conducts mechanism experiments on MEMIT's residual allocation/update magnitude and AlphaEdit's covariance-derived projection subspaces. The central argument is that covariance sensitivity depends on how each editor converts preservation statistics into actual weight updates.

Critical strengths

The MEMIT experiment preserves the original remaining-layer allocation while matching each request's joint update norm to the uniform-allocation reference. For AlphaEdit, the matched-rank random-projector and norm-matching controls in Tables 33–34 show that matching update size does not remove the locality difference from the covariance-derived projector. These are meaningful controls, and the distinction between per-request norm matching and final-checkpoint norms is appropriately documented.

Critical weaknesses

W1. The calibration cost/performance evidence is not sufficiently reconciled or controlled to support the main explanation.

Appendix C.3 states that the original baseline and Prefix should produce identical token sequences under matching configurations, but that equivalence of the historical configurations and caches has not been fully verified. The associated results support substantially different conclusions. For example, at 1,000 edits, Table 19 reports Llama-3.1-8B/AlphaEdit rewrite accuracy of 97.60 at every calibration length, whereas Table 16 reports ES of 34.95/99.10/96.60. These may reflect different historical protocols, but their provenance and relevance to the central comparison need to be reconciled before attributing the behavior to calibration length.

Even within the current tables, the main characterization is problematic. Table 11 reports AlphaEdit S4 of 17.80/66.11/59.44 on Llama-3.1-8B-Instruct, which does not support broad length invariance. Section 4.3 also claims that the non-PG19-native settings remain largely stable and that the short-sample distribution explains insensitivity. However, Table 14 reports C4-match scores of 5.98/62.57/29.56 and PG19-match scores of 33.50/9.19/12.60. The controls establish an interaction between construction and length, but do not identify short-sequence prevalence as the cause of invariance. Included-token budgets and sampling also vary with length, as acknowledged in C.3.

The cost evidence has related reporting problems. Table 3 labels entries as “Real Memory,” while its caption and C.1 describe analytical estimates rather than measured device-wide peaks. Tables 3 and 18 also disagree on analytical memory at shorter lengths, and no wall-clock measurements are reported. Since batch_tokens is set to 3L, the resource comparison changes batching capacity along with context length. Consequently, the quantitative cost and workstation-feasibility conclusions need a clearly specified measurement protocol.

W2. The scale experiments do not adequately separate covariance geometry from numerical and implementation effects.

For the stated equality-constrained ROME and EMMET updates, multiplying covariance by a positive scalar cancels exactly. Appendix D.1, Equations 34–36 correctly establishes this invariance. Nevertheless, Table 21 reports EMMET edit success dropping from 99% at baseline to 55%, 4%, and 0% at scales 10, 100, and 1000, respectively; ROME produces nonfinite updates at the tested scales below one.

The appendix acknowledges possible effects of regularization, precision, or implementation choices, but does not identify which operation causes these departures or establish whether the conclusions survive a scale-consistent implementation. Uniform scalar rescaling also preserves the covariance's relative condition number, so a generic appeal to conditioning is insufficient. These observations could be useful implementation findings, but their interpretation requires distinguishing the stated objectives from the computations actually executed.

This is particularly important because Appendix F.3 points to EasyEdit's changing hyperparameter directory without specifying an immutable implementation, exact solver tolerances, regularization rules, or the corrected EMMET changes mentioned elsewhere. Stable higher-precision reference solves, explicit regularization/threshold controls, and first-edit update comparisons would determine which effects precede sequential error amplification. Until then, the empirical basis for several editor-specific scale conclusions remains uncertain.

Questions for authors
  1. Experimental provenance: Can you reconcile the original, Prefix, and current default results using verified token sequences, covariance normalization, cache provenance, edit order, and frozen configurations? Please also clarify the different aggregate-score definitions in Table 19 and the S4 tables. Do the central length-sensitivity claims survive a matched rerun?

  2. Scale invariance: Which precise implementation operations break the ROME/EMMET invariance derived in Equations 34–36? Can the scale sweeps be reproduced with stable, scale-consistent reference solves, reporting solver precision, regularizers, thresholds, and first-edit update discrepancies?

  3. Cost and length controls: Which Table 3 values are measured, and how were they obtained? Can you reconcile Tables 3 and 18 and provide runtime/device-memory measurements with controlled batching, alongside a fixed real-token-budget comparison that separates context length from calibration amount?

  4. Mechanism scope: Do the MEMIT norm-matching and AlphaEdit projector findings persist beyond the 100-edit Llama-3.1-8B-Instruct/CounterFact setting, for another model or a longer trajectory with a different edit order? This would help establish whether these controls explain the broader sensitivity patterns reported elsewhere.

Reviewer_U21QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how the preprocessing step of covariance estimation affects the performances of locate-then-edit knowledge-editing methods. It examines varying length of tokens in the corpus used to estimate the covariance and finds that there are no consistent gains in editing performances when increasing the corpus context length for five editing methods. The effects of preprocessing covariance estimation on editing performances vary by editing methods via different characteristics of the covariance matrix.

Critical strengths

Understanding the effect and limitations of preprocessing covariance estimation in locate-then-edit KE methods is indeed an important and interesting research question.

Critical weaknesses
  1. The paper has low readability. The Introduction contains repetitive motivations and objectives, and dense experimental summaries without clear takeaways or actionable insights. The introduction also has insufficient literature positioning and unclear novelty. Throughout the paper, many terms are introduced without clear definitions, explanations, or references, and they do not appear to be commonly used as standard terminology in the related KE literature, such as "calibration corpus", "inverse-key alignment", "donor checkpoint's behavior", "post-edit weight rollback".

  2. The experiments have several major issues and appear to insufficiently support the main claims. Also, the experimental sections have low readability. Only one aggregated table is presented in the main text, whereas most results and tables are deferred to the appendix, making the experimental findings very difficult to evaluate. In Figure 1, the only figure in the paper, why is cosine similarity ranging 0-0.04 is considered "highly similar"? Other identified issues: (1) Evaluation metrics are not clearly defined in the main text. (2) The effect of calibration sequence length is not isolated, because varying the maximum length also changes the number and set of tokens used for covariance estimation. (3) The paper states that (line 193) AlphaEdit is nearly invariant to length, but its scores vary substantially across lengths in Section 4.3. (4) Several methods exhibit near-zero editing performance after 1000 sequential edits, making comparisons uninformative under such degenerate conditions. (5) Three out of four evaluated LLMs come from the Llama-3 family, raising questions about generalizability to other LLM architectures. Due to the challenging readability, please clarify if any of these points is due to misunderstanding.

  3. Likely implementation artifacts, incomplete results (e.g., Table 14, 15, 16), and "unverified" experiments appear in the results. Identified examples:

  • Appendix C.3: “...equivalence of the configurations and caches actually used in those runs has not been fully verified.”
  • Table 24: “...recomputed from the selected August runs and corrected EMMET.”
  • Table 8: “External-corpus cache provenance remains unresolved in the source audit.” I don’t understand this sentence.
  • Tables 14: “GA is the five-task mean, joined only by exact checkpoint name.” What does this mean?
  • Table 4: “The corresponding implementation names are prefix, sequential_chunk, within_doc_shuffle, global_sentence_shuffle, fixed_budget_uniform, and padded_chunk with padbudget.” Why is it important to document these implementation names?

These sentences suggest that some results may come from unverified or mismatched runs. The paper could benefit from careful proofreading before submission.

Questions for authors

Please see weaknesses.

Reviewer_xbzCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies covariance in locate-then-edit knowledge editing, primarily focusing on how covariance is estimated, how it affects editing, and what differs across editing methods. First, it tests whether longer calibration contexts help, finding that these add compute but do not improve editing results reliably. It then perturbs covariance scale, eigenspectrum, and directional structure, showing that covariance defines an anisotropic geometry that depends on how updates are made in each editing algorithm. Mechanistic experiments link MEMIT’s covariance sensitivity mainly to update magnitude, while AlphaEdit’s behavior appears driven by the projection subspace derived from covariance. The paper tries to connect covariance estimation and geometry to the parameter updates that different editing methods produce.

Critical strengths
  1. The paper studies a widely used yet not well-explored topic. Covariance statistics are important for many editing methods. This paper systematically studies the effects and reliability of covariance to different methods. These can help future knowledge-editing research and methods to understand the properties of the updates.
  2. The paper shows wide range of analysis on covariance properties and controlled experiments. Without limiting to only downstream editing scores, thorough analysis provides insights and better interpretation rather than observation.
  3. The paper covered ROME, r-ROME, MEMIT, EMMET, and AlphaEdit, with multiple Llama and Qwen. Extensive experiments and analysis provide additional evidence to the explanations.
Critical weaknesses
  1. The paper studies and claims to analyze the computation costs of estimating statistics, but primarily reports only memory estimations and out-of-memory. Other important metrics like end-to-end processing time are not reported, which weakens the claim.
  2. Experiments and claims about AlphaEdit self-contradicts. In section 4.1, the authors claim “AlphaEdit is nearly invariant to length”, but section 4.3 instead reports score drops significantly (62.57 -> 29.56 for C4-matched and 54.92 -> 10.74 for PG19-native). Although these experiments use different calibration settings, the paper should clarify better when the invariance holds.
  3. While the paper systematically studies empirical evidence and properties of covariance statistics in editing methods, it is unclear how these can be used concretely to improve the methods.
  4. The paper mentions multiple closely related works like [11], [16], and [33], but this paper’s differences to them are not well explained.
Questions for authors
  1. Can the authors provide more efficiency analysis, especially across different calibration lengths?
  2. Can the authors isolate the effect of calibration sequence length distribution while controlling the effective token budget and sampled documents? The current experiments do not sufficiently isolate the effect of calibration length, since other variables exist.
  3. How much do results vary across runs? Are multiple seeds/runs made to understand stability and statistical significance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.