- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits sequential knowledge editing and re-examines AlphaEdit, finding that its default null-space projector is nearly the identity and contributes little, while update regularization is the main driver of preservation. Based on this diagnosis, the authors propose FisherEdit, which uses a diagonal Fisher estimate to apply importance-aware element-wise update penalties, with a row-wise closed-form solver based on the Woodbury identity; Appendix H shows this regularizer is a diagonal approximation of the second-order Taylor expansion of the locality KL objective. Experiments on CounterFact, Ripple, and EVOKE across three backbones show that FisherEdit achieves the best average editing scores, better preserves downstream capabilities and WikiText perplexity, and reduces preprocessing time by 36.6× and end-to-end runtime by 12.8× with nearly unchanged peak GPU memory.
Critical strengths
-
The re-diagnosis of AlphaEdit's mechanism is supported by direct, reproducible evidence. Fig. 2a shows the default projector is close to the identity, Fig. 3 shows disabling the projector leaves results nearly unchanged, and Fig. 4 and Fig. 7 show that $\lambda$ is the real control knob. This reframes preservation in sequential editing toward a more principled design of update regularization. I have also observed this point in my own experiments, which is why I consider this diagnosis to be of practical value to the field.
-
FisherEdit is simple, effective, and substantially more efficient. Diagonal Fisher weighting provides importance-aware regularization, and the Woodbury closed-form solver avoids the inverse, reducing preprocessing time by 36.6× and end-to-end runtime by 12.8× with nearly unchanged peak GPU memory. Ablations show the gains come from the structural information in the Fisher diagonal, and improvements are consistent across three backbones and three benchmarks.
Critical weaknesses
-
The claim that the projector contributes little is only strictly established under the default $\tau$, and no full sweep of $\tau$ is provided. The authors state that strengthening the projection trades edit efficacy for locality, but do not show this trade-off curve. If some $\tau$ makes the projection genuinely effective, the conclusion should be qualified as holding under the default setting. This matters because the paper's central diagnosis rests on the projector being inconsequential.
-
The efficiency comparison is not clearly controlled. Table 11 reports 17.7 minutes for FisherEdit versus 648 minutes for AlphaEdit, but the two use different calibration sets (1,000 MMLU examples vs. 100k Wikipedia samples with layerwise SVD). Readers may mistake this for a pure algorithmic speedup rather than a consequence of different design choices. The authors should clarify whether the comparison is under matched conditions.
-
From a K-V perspective, AlphaEdit orthogonalizes K while O-Edit [1] orthogonalizes V, and the latter achieves an advantage through orthogonality without any regularization. Please analyze why O-Edit works. This comparison bears directly on the paper's central claim that preservation comes from regularization rather than projection: if O-Edit achieves an advantage through V-side orthogonality alone without regularization, then the conclusion that regularization is the main control knob may need to be revisited. The authors should explain the mechanism behind O-Edit and how it relates to the Fisher-weighted regularization in FisherEdit. This question is critical: if the response is reasonable and convincing, I will raise my score from 2 to 3, or even 4.
[1] O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing
Questions for authors
See Critical Weaknesses
3 comments
Response to Reviewer xyuT (2/N)
W3. Why O-Edit works and how it relates to Fisher-weighted regularization.
Thank you for raising this important question. We clarify O-Edit's directional mechanism and its relationship to our analysis of AlphaEdit from the following prespectives:
(1) O-Edit's directional constraints in the MEMIT update.
When applied to MEMIT, O-Edit retains its closed-form weight-update rule. For a single knowledge edit at step , let denote the MLP down-projection matrix before the edit, the input key vector, and the optimized target value vector at that layer. We write the MEMIT update as
where is the scaled second-moment matrix of the preserved keys defined in Section 3.1, with the preservation weight and normalization absorbed into . Here, is the desired output correction, is the input-side factor computed from and , and is the resulting weight update.
O-Edit changes the preceding optimization of : it adds similarity penalties involving the historical-update and background-gradient subspaces to the original value-optimization loss. During gradient-based optimization, these penalties act on the residual . The resulting value is then passed to the same MEMIT closed-form update. Thus, the unchanged solver does not imply an unconstrained parameter update: shaping also shapes the output direction of .
(2) Connecting update direction and magnitude through cumulative weight changes.
For one edited weight block, let denote the accumulated update from previous edits. Then
O-Edit's historical constraint is designed to suppress the cross term through orthogonality. Let be an orthonormal basis for the historical update column space, so that . If target optimization achieves , then
AlphaEdit's penalty directly penalizes the current update norm , while FisherEdit uses a Fisher-weighted quadratic penalty on the update. Appendix E (Figure 7) shows that weak regularization can lead to rapidly growing batch updates in AlphaEdit, whereas the tested settings with keep their magnitudes bounded over the evaluated sequence.
(3) Experimental comparison within a shared MEMIT pipeline.
We examined this connection using the Llama-3-8B-Instruct/CounterFact setting in Table 1. All three variants share the native MEMIT pipeline, retain its preservation statistic , and update layers 4–8. We compare native MEMIT with variants that add either an update penalty with or O-Edit's soft V constraint reproduced from the paper. The V constraint is applied at layer 8, where the shared value target is optimized, and the added penalty applies to all five layers.
The rank-one derivation above concerns individual edits, but the cumulative-norm identity also holds for batch updates. Below, indexes edit batches, and norms and inner products are aggregated across the five edited layers. We record the final cumulative weight change and the cumulative cross term:
| Variant | Cumulative weight change | Cumulative cross term |
|---|---|---|
| Native MEMIT | 1,483.04 | 541,639.86 |
| + update penalty | 1,229.17 | 358,035.18 |
| + O-Edit soft V constraint (layer 8) | 760.84 | 94,574.72 |
Relative to native MEMIT, the added penalty reduces the final cumulative norm by 17.12%. The soft V constraint reduces the positive cumulative cross term by 82.54% and the final cumulative norm by 48.70%. These observations support the view that V-side constraints can limit cross-update accumulation, alongside changes in individual update magnitudes. O-Edit's useful directional constraints are therefore compatible with our finding that explicit update regularization drives much of the observed preservation in the evaluated AlphaEdit configurations.
Dear Reviewer xyuT,
Thank you again for your time and insightful questions. As the discussion period is nearing its end, we kindly invite you to review our response above, especially our new theoretical and experimental analysis of O-Edit's mechanism. We hope this addresses your concerns and would be very glad to discuss further if needed.
Best,
Authors
Response to Reviewer xyuT (1/N)
We thank the reviewer for the careful reading and constructive feedback. We appreciate the thoughtful comments and suggestions, which help us improve the clarity and completeness of our work. We address each comment below.
Thank you for suggesting a more detailed examination of the threshold dependence. The AlphaEdit curve in the submitted Figure 1 was constructed by varying τ over six values: 0.0002,0.0004,0.0006,0.001,0.002,0.02. To provide denser evidence, we have additionally evaluated five thresholds, 0.0005,0.0008,0.003,0.006,0.06, using the same Figure 1 configuration. The original measurements and the five additions are reported together below. Bold denotes the new experiments.
The new experiments make the trade-off clearer: in this setting, stronger projection improves specificity while reducing efficacy and generalization.
Thank you for this helpful suggestion. We agree that the basis of the timing comparison could have been stated more clearly. Table 11 measures the preprocessing cost of the configurations used to obtain the reported editing results: FisherEdit estimates its diagonal Fisher from 1,000 MMLU examples, while AlphaEdit constructs second-moment statistics from 100,000 Wikipedia samples and performs layerwise SVD. With this substantially smaller calibration set, FisherEdit already achieves a higher average editing score than AlphaEdit on a wide range of settings, therefore FisherEdit's ability to achieve strong editing performance with fewer calibration samples is part of this efficiency benefit. Following your advice, we have clarified the calibration settings and the basis of this comparison in the revised PDF.