- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how covariance estimation and geometry affect knowledge editing, finding that longer calibration contexts increase cost without consistently improving performance. Mechanistic analyses suggest that MEMIT’s sensitivity is largely related to update magnitude, while AlphaEdit’s scale sensitivity depends on projection subspace selection.
Critical strengths
- It examines covariance, an often-overlooked component of knowledge editing, and investigates the relationship between its estimation cost and editing performance.
- Using MEMIT and AlphaEdit as case studies, it investigates the roles of update magnitude and subspace selection through norm-matching and fixed-projector controls.
Critical weaknesses
-
At 1,000 edits, AlphaEdit’s ES/Rewrite values across L=512/1024/2048 are 34.95/99.10/96.60 in Table 16 versus 97.60/97.60/97.60 in Table 19. Different aggregation formulas alone cannot explain these discrepancies.
-
Acknowledged token-budget and sequence-construction confounds limit causal attribution, while large AlphaEdit changes on C4 challenge the stability claim. Main mechanistic controls use one model, 100 edits, and a fixed editing seed, limiting generalization.
Questions for authors
-
Could the authors reconcile the configurations, covariance caches, and evaluation protocols underlying Tables 16 and 19? An overview figure connecting the analyses would also improve clarity.
-
Given the derived scale invariance, which implementation factors cause EMMET’s observed sensitivity, and could controlled experiments verify their contributions beyond the possibilities already discussed?
-
Given the explanation for nonzero scores without updates, could the authors explicitly report no-edit baselines and verify whether the main conclusions hold for individual metrics?