acceptodds
Under review as a conference paper at ICLR 2027

FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models

29 views

Abstract

Sequential knowledge editing in Large Language Models (LLMs) often leads to a degradation of general capabilities due to cumulative parameter interference. In this paper, we revisit prominent projection-based methods like AlphaEdit and reveal a surprising mismatch: their robustness is primarily driven by update regularization rather than the intended null-space projection. Based on this insight, we propose FisherEdit, a theoretically grounded method built on Fisher geometry, which replaces coarse activation-space projections with importance-aware regularization in parameter space. It penalizes changes to sensitive parameters more strongly while retaining flexibility for new facts, and admits an efficient row-wise closed-form solution. Experiments on Llama-3, Qwen2.5, and Olmo-3 across multiple benchmarks demonstrate that FisherEdit achieves state-of-the-art performance while better preserving model stability and downstream task accuracy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge editing, model editing, large language models, sequential editing, Fisher information, parameter-space regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_xyuTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits sequential knowledge editing and re-examines AlphaEdit, finding that its default null-space projector is nearly the identity and contributes little, while update regularization is the main driver of preservation. Based on this diagnosis, the authors propose FisherEdit, which uses a diagonal Fisher estimate to apply importance-aware element-wise update penalties, with a row-wise closed-form solver based on the Woodbury identity; Appendix H shows this regularizer is a diagonal approximation of the second-order Taylor expansion of the locality KL objective. Experiments on CounterFact, Ripple, and EVOKE across three backbones show that FisherEdit achieves the best average editing scores, better preserves downstream capabilities and WikiText perplexity, and reduces preprocessing time by 36.6× and end-to-end runtime by 12.8× with nearly unchanged peak GPU memory.

Critical strengths
  1. The re-diagnosis of AlphaEdit's mechanism is supported by direct, reproducible evidence. Fig. 2a shows the default projector is close to the identity, Fig. 3 shows disabling the projector leaves results nearly unchanged, and Fig. 4 and Fig. 7 show that $\lambda$ is the real control knob. This reframes preservation in sequential editing toward a more principled design of update regularization. I have also observed this point in my own experiments, which is why I consider this diagnosis to be of practical value to the field.

  2. FisherEdit is simple, effective, and substantially more efficient. Diagonal Fisher weighting provides importance-aware regularization, and the Woodbury closed-form solver avoids the inverse, reducing preprocessing time by 36.6× and end-to-end runtime by 12.8× with nearly unchanged peak GPU memory. Ablations show the gains come from the structural information in the Fisher diagonal, and improvements are consistent across three backbones and three benchmarks.

Critical weaknesses
  1. The claim that the projector contributes little is only strictly established under the default $\tau$, and no full sweep of $\tau$ is provided. The authors state that strengthening the projection trades edit efficacy for locality, but do not show this trade-off curve. If some $\tau$ makes the projection genuinely effective, the conclusion should be qualified as holding under the default setting. This matters because the paper's central diagnosis rests on the projector being inconsequential.

  2. The efficiency comparison is not clearly controlled. Table 11 reports 17.7 minutes for FisherEdit versus 648 minutes for AlphaEdit, but the two use different calibration sets (1,000 MMLU examples vs. 100k Wikipedia samples with layerwise SVD). Readers may mistake this for a pure algorithmic speedup rather than a consequence of different design choices. The authors should clarify whether the comparison is under matched conditions.

  3. From a K-V perspective, AlphaEdit orthogonalizes K while O-Edit [1] orthogonalizes V, and the latter achieves an advantage through orthogonality without any regularization. Please analyze why O-Edit works. This comparison bears directly on the paper's central claim that preservation comes from regularization rather than projection: if O-Edit achieves an advantage through V-side orthogonality alone without regularization, then the conclusion that regularization is the main control knob may need to be revisited. The authors should explain the mechanism behind O-Edit and how it relates to the Fisher-weighted regularization in FisherEdit. This question is critical: if the response is reasonable and convincing, I will raise my score from 2 to 3, or even 4.

[1] O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing

Questions for authors

See Critical Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer xyuT (1/N)

We thank the reviewer for the careful reading and constructive feedback. We appreciate the thoughtful comments and suggestions, which help us improve the clarity and completeness of our work. We address each comment below.

W1. Projection-threshold sweep and the efficacy–locality trade-off.

Thank you for suggesting a more detailed examination of the threshold dependence. The AlphaEdit curve in the submitted Figure 1 was constructed by varying τ\tau over six values: 0.0002,0.0004,0.0006,0.001,0.002,0.02\\{0.0002,0.0004,0.0006,0.001,0.002,0.02\\}. To provide denser evidence, we have additionally evaluated five thresholds, 0.0005,0.0008,0.003,0.006,0.06\\{0.0005,0.0008,0.003,0.006,0.06\\}, using the same Figure 1 configuration. The original measurements and the five additions are reported together below. Bold denotes the new experiments.

τ\tauEff.Gen.Spe.
0.000231.79526.35984.708
0.000490.87261.53881.744
0.000595.38567.74480.954
0.000697.02671.12880.010
0.000898.56475.28279.087
0.00198.97477.48778.287
0.00299.28280.66776.205
0.00399.48782.25675.436
0.00699.28282.41074.595
0.0299.38583.48773.549
0.0699.69284.46273.292

The new experiments make the trade-off clearer: in this setting, stronger projection improves specificity while reducing efficacy and generalization.

W2. Interpretation of the preprocessing-time comparison.

Thank you for this helpful suggestion. We agree that the basis of the timing comparison could have been stated more clearly. Table 11 measures the preprocessing cost of the configurations used to obtain the reported editing results: FisherEdit estimates its diagonal Fisher from 1,000 MMLU examples, while AlphaEdit constructs second-moment statistics from 100,000 Wikipedia samples and performs layerwise SVD. With this substantially smaller calibration set, FisherEdit already achieves a higher average editing score than AlphaEdit on a wide range of settings, therefore FisherEdit's ability to achieve strong editing performance with fewer calibration samples is part of this efficiency benefit. Following your advice, we have clarified the calibration settings and the basis of this comparison in the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer xyuT (2/N)

W3. Why O-Edit works and how it relates to Fisher-weighted regularization.

Thank you for raising this important question. We clarify O-Edit's directional mechanism and its relationship to our analysis of AlphaEdit from the following prespectives:

(1) O-Edit's directional constraints in the MEMIT update.

When applied to MEMIT, O-Edit retains its closed-form weight-update rule. For a single knowledge edit at step tt, let Wt−1W_{t-1} denote the MLP down-projection matrix before the edit, ktk_t the input key vector, and vtv_t the optimized target value vector at that layer. We write the MEMIT update as

Et=vt−Wt−1kt,at⊤=kt⊤(C0+ktkt⊤)−1,Δt=Etat⊤,E_t=v_t-W_{t-1}k_t,\qquad a_t^\top=k_t^\top(C_0+k_tk_t^\top)^{-1},\qquad \Delta_t=E_ta_t^\top,

where C0=λK0K0⊤C_0=\lambda K_0K_0^\top is the scaled second-moment matrix of the preserved keys K0K_0 defined in Section 3.1, with the preservation weight and normalization absorbed into λ\lambda. Here, EtE_t is the desired output correction, ata_t is the input-side factor computed from ktk_t and C0C_0, and Δt\Delta_t is the resulting weight update.

O-Edit changes the preceding optimization of vtv_t: it adds similarity penalties involving the historical-update and background-gradient subspaces to the original value-optimization loss. During gradient-based optimization, these penalties act on the residual EtE_t. The resulting value is then passed to the same MEMIT closed-form update. Thus, the unchanged solver does not imply an unconstrained parameter update: shaping EtE_t also shapes the output direction of Δt\Delta_t.

(2) Connecting update direction and magnitude through cumulative weight changes.

For one edited weight block, let St−1=∑s<tΔsS_{t-1}=\sum_{s < t}\Delta_s denote the accumulated update from previous edits. Then

∣St−1+Δt∣F2=∣St−1∣F2+∣Δt∣F2+2⟨St−1,Δt⟩F⏟cross-update accumulation.\\|S_{t-1}+\Delta_t\\|_F^2 =\\|S_{t-1}\\|_F^2+\\|\Delta_t\\|_F^2 +2\underbrace{\langle S_{t-1},\Delta_t\rangle_F}_{\text{cross-update accumulation}}.

O-Edit's historical constraint is designed to suppress the cross term through orthogonality. Let UU be an orthonormal basis for the historical update column space, so that St−1=UBS_{t-1}=UB. If target optimization achieves U⊤Et=0U^\top E_t=0, then

⟨St−1,Δt⟩F=Et⊤St−1at=(U⊤Et)⊤Bat=0.\langle S_{t-1},\Delta_t\rangle_F =E_t^\top S_{t-1}a_t =(U^\top E_t)^\top Ba_t=0.

AlphaEdit's ℓ2\ell_2 penalty directly penalizes the current update norm ∣Δt∣F2\\|\Delta_t\\|_F^2, while FisherEdit uses a Fisher-weighted quadratic penalty on the update. Appendix E (Figure 7) shows that weak regularization can lead to rapidly growing batch updates in AlphaEdit, whereas the tested settings with λ≥10−3\lambda\ge10^{-3} keep their magnitudes bounded over the evaluated sequence.

(3) Experimental comparison within a shared MEMIT pipeline.

We examined this connection using the Llama-3-8B-Instruct/CounterFact setting in Table 1. All three variants share the native MEMIT pipeline, retain its preservation statistic C0C_0, and update layers 4–8. We compare native MEMIT with variants that add either an ℓ2\ell_2 update penalty with λ=1\lambda=1 or O-Edit's soft V constraint reproduced from the paper. The V constraint is applied at layer 8, where the shared value target is optimized, and the added ℓ2\ell_2 penalty applies to all five layers.

The rank-one derivation above concerns individual edits, but the cumulative-norm identity also holds for batch updates. Below, tt indexes edit batches, and norms and inner products are aggregated across the five edited layers. We record the final cumulative weight change and the cumulative cross term:

CT=2∑t=1T⟨St−1,Δt⟩F,∣ST∣F2=∑t=1T∣Δt∣F2+CT.C_T=2\sum_{t=1}^{T}\langle S_{t-1},\Delta_t\rangle_F, \qquad \\|S_T\\|_F^2=\sum_{t=1}^{T}\\|\Delta_t\\|_F^2+C_T.

VariantCumulative weight change ∥ST∥F\lVert S_T\rVert_FCumulative cross term CTC_T
Native MEMIT1,483.04541,639.86
+ ℓ2\ell_2 update penalty1,229.17358,035.18
+ O-Edit soft V constraint (layer 8)760.8494,574.72

Relative to native MEMIT, the added ℓ2\ell_2 penalty reduces the final cumulative norm by 17.12%. The soft V constraint reduces the positive cumulative cross term by 82.54% and the final cumulative norm by 48.70%. These observations support the view that V-side constraints can limit cross-update accumulation, alongside changes in individual update magnitudes. O-Edit's useful directional constraints are therefore compatible with our finding that explicit update regularization drives much of the observed preservation in the evaluated AlphaEdit configurations.

Dear Reviewer xyuT,

Thank you again for your time and insightful questions. As the discussion period is nearing its end, we kindly invite you to review our response above, especially our new theoretical and experimental analysis of O-Edit's mechanism. We hope this addresses your concerns and would be very glad to discuss further if needed.

Best,

Authors

Reviewer_DvECOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the classic setting of knowledge editing for LLMs. Based on the analysis of the well-known method AlphaEdit, the authors find that its preservation mainly comes from regularization instead of null-space projection. They proposed FisherEdit method. It replaced the projection with empirical Fisher-weighted regularization. Experiments on several LLMs and benchmarks FisherEdit method performs slightly better and AlphaEdit.

Critical strengths
  1. The paper studies an important problem of sequential knowledge editing, and improves the performance over the classic AlphaEdit method.

  2. The analysis and findings of AlphaEdit is interesting, and give the motivation for this paper.

  3. The method is simple and easy to understand.

  4. Experiments on several LLMs and benchmarks demonstrate the potential of this paper.

Critical weaknesses
  1. Major concern: I have a concern about the baselines. As the authors mention in the related work, CrispEdit also uses K-FAC to approximate the Fisher information matrix of the model to protect general abilities, which is close to the intuition of FisherEdit. However, CrispEdit is not compared in the experiments. Since the authors claim that their soft Fisher penalty is better than such hard projections, I think this comparison is needed to support the claim.

  2. Major concern: The analysis of the projection in AlphaEdit needs further justification. The projector looking close to the identity matrix does not mean it has little effect. Since only about 10 directions are removed, the matrix will naturally look close to the identity. However, if the keys of general knowledge mainly lie in these few directions, removing them can already reduce most of the effect on general knowledge. I think a more convincing measure is how much the change on the preserved keys is reduced by the projection.

  3. Minor Concern: The novelty of the method is not significant. Using a diagonal Fisher to weight a quadratic penalty is the same idea as EWC in continual learning. The main difference is applying this penalty in the MEMIT-style closed-form editing objective and solving it row by row. Thus, I feel the method is mainly an application of EWC to the MEMIT framework. Feel free to correct me if I miss anything.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DvEC (1/N)

We appreciate the reviewer's careful consideration of our work and the helpful feedback. Thank you for highlighting points that would benefit from further explanation. We respond to the comments below.

W1. Comparison with CrispEdit.

Thank you for highlighting CrispEdit as a closely related method. We agree that a direct comparison helps assess the two preservation approaches, and have included results from its official implementation below. We retain CrispEdit's sequential editing and K-FAC procedures and use the same datasets and evaluation protocol as our main tables. Since there is no official Olmo configuration, we transfer its Llama configuration without additional tuning.

Table 1 settings: CounterFact across three backbones.

BackboneMethodEff.Gen.Spe.Avg.
Llama-3-8B-InstructCrispEdit85.5468.0053.1768.90
Llama-3-8B-InstructFisherEdit99.9090.9275.6888.83
Qwen2.5-7B-InstructCrispEdit74.4648.2156.0359.57
Qwen2.5-7B-InstructFisherEdit99.9071.0866.6679.21
Olmo-3-7B-InstructCrispEdit75.6960.9754.9063.86
Olmo-3-7B-InstructFisherEdit99.3879.5373.3384.08

Table 2 settings: Olmo-3-7B-Instruct across three benchmarks.

BenchmarkMethodEff.Gen.Spe.Avg.
CounterFactCrispEdit75.4961.0854.6863.75
CounterFactFisherEdit99.3879.5373.3384.08
RippleCrispEdit89.6086.5525.7367.29
RippleFisherEdit97.3074.6642.2871.41
EVOKECrispEdit78.0073.2848.8266.70
EVOKEFisherEdit99.9092.9471.0887.97

FisherEdit achieves higher Average scores than CrispEdit across all settings, with consistent improvements in Efficacy and Specificity. Although CrispEdit obtains higher Generalization on Ripple, FisherEdit offers a better overall balance between editing quality and knowledge preservation. We have added the CrispEdit results to Tables 1 and 2 of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer DvEC (2/N)

W2. Further justification of the projection analysis in AlphaEdit.

Thank you for suggesting this direct measurement. We agree that the appearance of PP alone does not establish its preservation effect. We have added direct measurements in Appendix K.

1. The preservation effect of projection depends on regularization.

Appendix K.1, Figure 9 compares complete AlphaEdit trajectories with and without the default projector on Llama-3-8B-Instruct / CounterFact. The two branches share the initial model, edit order, and regularization coefficient, and each computes its own keys, targets, and history. On fixed held-out Wikipedia keys from the initial model, we measure cumulative changes as

Jt=∑ℓ∣(Wℓ,t−Wℓ,0)Kℓ,0∣F2.J_t=\sum_{\ell}\\|(W_{\ell,t}-W_{\ell,0})K_{\ell,0}\\|_F^2.

At the larger coefficients λ=10−3\lambda=10^{-3} and 10−210^{-2}, PP reduces the final preserved-key change by about one third, confirming its protection of existing knowledge. However, when regularization is weakened to λ=10−5\lambda=10^{-5} or 10−410^{-4}, deviations grow faster with PP and reach much larger values than without it. This reversal indicates that, PP needs to work together with sufficient regularization: regularization keeps the updates stable, enabling projection to provide additional protection.

2. Hard projection creates a trade-off between preservation and editing.

Our findings are not limited to whether the default PP is effective or resembles the identity matrix. Finding 3 highlights a more general limitation of hard projection: directions excluded to preserve existing knowledge may also be needed to write new facts. This creates a trade-off between protecting existing knowledge and incorporating new knowledge, consistent with the decrease in Efficacy and Generalization as projection becomes stronger.

To examine this effect directly, Appendix K.2, Figure 10 measures new-target fitting along a common reference trajectory. Before each batch, we fix the model state, current keys, target residuals, and historical keys. We solve AlphaEdit at different τ\tau values and FisherEdit on these same inputs, using a scalar regularization coefficient of 10 for both methods, and measure

Ledit(t)=∑ℓ∣Δℓ,tKℓ,1,t−Eℓ,t∣F2.L_{\mathrm{edit}}(t)=\sum_{\ell}\\|\Delta_{\ell,t}K_{\ell,1,t}-E_{\ell,t}\\|_F^2.

Lower error means more complete fitting of the fixed new targets. Stronger projection increases this error. FisherEdit's curve lies below all six AlphaEdit curves at every stage, including the default τ=0.02\tau=0.02, which uses the weakest projection among the tested settings. These results support the trade-off highlighted in Finding 3: hard projection can protect existing knowledge while suppressing directions needed for new edits. FisherEdit replaces this hard exclusion with a soft Fisher-weighted penalty, allowing those directions to adapt when needed.

W3. Relationship to EWC and the contribution of FisherEdit.

Thank you for this perspective. We agree that the Fisher-weighted quadratic penalty shares the regularization principle of EWC. We would like to explain how this formulation arose in our study and how we view its contribution.

Our starting point was reproducing AlphaEdit. We observed that removing its default projector made little difference to the editing metrics, which led us to examine the effect of strengthening the projection. This investigation highlighted a difficulty with hard constraints: the excluded directions can also contain information needed to implement the desired edit. We therefore sought a soft constraint that would protect important parameters while allowing them to change when necessary. Such a constraint requires a measure of parameter importance, for which Fisher information was a natural choice. Incorporating this importance measure into the MEMIT-style editing objective led to the Fisher-weighted final term in Eq. (5).

The resemblance to EWC reflects a shared requirement of knowledge editing and continual learning: both must accommodate new information while preserving previously acquired knowledge and general capabilities. We do not regard Fisher-weighted quadratic regularization itself as a new principle. Our contribution lies in the analysis of preservation in sequential knowledge editing, the resulting formulation within the editing objective, and its efficient row-wise closed-form solution. We appreciate the opportunity to make this motivation and connection clearer.

Dear Reviewer DvEC,

Thank you again for your time and thoughtful feedback. As the discussion period is nearing its end, we kindly invite you to review our response above, including the CrispEdit baselines and direct trajectory measurements on projection. Please let us know if these additions address your concerns, and we would be glad to discuss further if needed.

Best,

Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer m91K (1/N)

We thank the reviewer for the thorough and balanced assessment of our work. We appreciate the constructive suggestions and the opportunity to provide further clarification. Our responses are given below.

W1. Connection between the theoretical Fisher metric and the implemented empirical Fisher.

Thank you for highlighting the distinction between the theoretical Fisher metric and its practical approximation. The expansion in Section 4.4 and Appendix H identifies the model Fisher as the local curvature of predictive KL. Following[1], [2], [3], FisherEdit makes this principle tractable by estimating sensitivities from labeled calibration examples and retaining only the diagonal. The derivation motivates this quadratic regularizer, while its empirical relationship with predictive change requires a direct test. We therefore examine two questions: whether the implemented empirical Fisher tracks predictive KL, and whether it remains useful during sequential editing.

(1) Relationship between the implemented Fisher penalty and theoretical Fisher metric

We generate 20 independent CounterFact edits on Qwen2.5-7B-Instruct. Each starts from the original weights W0W_0, updates layers 4-8, and is evaluated at its original update magnitude. On the same 1,000 MMLU examples, we compute three quadratic forms for each update: the full model Fisher quadratic, which averages the quadratic contributions of all possible next tokens, weighted by their probabilities under the unedited model; the full empirical Fisher quadratic, which uses correct-answer gradients while retaining parameter cross terms; and the diagonal empirical Fisher quadratic Q(ΔW)=12∑jF^jj(ΔWj)2Q(\Delta W)=\tfrac12\sum_j\widehat F_{jj}(\Delta W_j)^2. Each is paired with the full-vocabulary next-token DKL(pW0∣pW0+ΔW)D_{\mathrm{KL}}(p_{W_0}\\|p_{W_0+\Delta W}), averaged over those examples.

Fisher quadraticPearson rr with predictive KL
Full model Fisher (theoretical)0.987
Full empirical Fisher0.666
Diagonal empirical Fisher0.583

This comparison makes the approximation costs explicit. Replacing the model-output expectation with labeled empirical gradients reduces the correlation from 0.987 to 0.666; discarding cross terms further reduces it to 0.583. Both full Fisher variants are costly to store explicitly and couple updates across weight rows, preventing direct use of our efficient row-wise solver. Nevertheless, the diagonal empirical statistic retains a positive association with the actual distributional change across these edits. The results therefore support its use as an informative sensitivity proxy, with lower storage and computational requirements, rather than establishing equivalence to the model Fisher or an exact estimate of predictive KL.

(2) Utility of the initial Fisher statistic during sequential editing.

For sequential editing, a further question is whether that sensitivity proxy remains useful as the parameters move away from their initial values. Appendix C.4, Tables 9-10, examines this on Llama-3-8B-Instruct / CounterFact over the complete sequential edit stream, editing layers 4-8 in batches of 100. We re-estimate the diagonal after editing on the same held-out MMLU calibration set and compare it with the initial statistic. Across the five layers, Spearman correlation remains 0.86-0.90, log-Pearson correlation 0.87-0.91, and cosine similarity 0.82-0.98; 72-75% of the initial top-10% coordinates remain in that set. Thus, the parameter-importance structure remains substantially aligned over the evaluated sequence.

We then test whether adapting the statistic during editing improves its practical utility. After each batch, an online variant recomputes a local diagonal from per-example cross-entropy gradients on the just-edited prompt/target pairs and incorporates it through Ft+1=γFt+(1−γ)Flocal,t\mathbf F_{t+1}=\gamma\mathbf F_t+(1-\gamma)\mathbf F_{\mathrm{local},t}. With other settings unchanged, the fixed statistic (γ=1\gamma=1) achieves an Average of 88.83, compared with 88.31 and 88.41 for γ=0.99\gamma=0.99 and 0.90.9, and has the highest scores on all four editing metrics. These online variants also add approximately 95-97 seconds per 100-edit batch, about 45% of the reported per-batch editing time. Together with the measured alignment, this supports reusing the initial Fisher in the tested setting.

[1] Overcoming catastrophic forgetting in neural networks. PNAS 2017.

[2] Training Neural Networks with Fixed Sparse Masks. NeurIPS 2021.

[3] Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge. 2025.

AuthorsOpenReview ↗

Response to Reviewer m91K (2/N)

W2. Separating parameter importance from effective regularization strength.

Thank you for this helpful suggestion. Following your recommendation, we conducted a layer-wise Fisher-shuffling experiment under the Table 3 setting. Within each edited layer, we randomly permute the entries of the Fisher diagonal, changing the correspondence between parameter coordinates and their Fisher values while preserving the complete distribution of values, including their sum and scale. The regularization coefficient and the other editing settings are unchanged. This control therefore preserves the aggregate penalty-weight mass while disrupting which parameters receive stronger protection.

MethodEff.Gen.Spe.Avg.
Fisher99.8091.4475.2188.81
Shuffled Fisher99.4990.6271.5087.20

Shuffling reduces Specificity by 3.71 points and Average by 1.61 points, despite preserving the distribution and overall scale of the regularization weights. This supports the importance of the correspondence between Fisher values and parameter coordinates, beyond the use of nonuniform penalty values alone. We have updated Table 3 with the matched Fisher result and this shuffled-Fisher control.

W3. Missing baseline CrispEdit.

Following your suggestion, we have added CrispEdit as an additional baseline for the settings in Tables 1 and 2. We use the authors' official implementation, retaining its sequential editing and K-FAC procedures, and report results on the same datasets and evaluation protocol as our main tables. For Olmo, which does not have an official CrispEdit configuration, we transfer the official Llama configuration without additional tuning.

Table 1 settings: CounterFact across three backbones.

BackboneMethodEff.Gen.Spe.Avg.
Llama-3-8B-InstructCrispEdit85.5468.0053.1768.90
Llama-3-8B-InstructFisherEdit99.9090.9275.6888.83
Qwen2.5-7B-InstructCrispEdit74.4648.2156.0359.57
Qwen2.5-7B-InstructFisherEdit99.9071.0866.6679.21
Olmo-3-7B-InstructCrispEdit75.6960.9754.9063.86
Olmo-3-7B-InstructFisherEdit99.3879.5373.3384.08

Table 2 settings: Olmo-3-7B-Instruct across three benchmarks.

BenchmarkMethodEff.Gen.Spe.Avg.
CounterFactCrispEdit75.4961.0854.6863.75
CounterFactFisherEdit99.3879.5373.3384.08
RippleCrispEdit89.6086.5525.7367.29
RippleFisherEdit97.3074.6642.2871.41
EVOKECrispEdit78.0073.2848.8266.70
EVOKEFisherEdit99.9092.9471.0887.97

FisherEdit achieves higher Average scores than CrispEdit across all settings, with consistent improvements in Efficacy and Specificity. Although CrispEdit obtains higher Generalization on Ripple, FisherEdit offers a better overall balance between editing quality and knowledge preservation.

We have added these CrispEdit results to Tables 1 and 2 of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer m91K (3/N)

Q2. Joint tuning and the effect of projection.

(1) Joint tuning of AlphaEdit's projection threshold and regularization coefficient.

Thank you for suggesting this joint comparison. We have now evaluated AlphaEdit with all 24 combinations of the six projection thresholds in Figure 1 and the four regularization coefficients in Table 8 on Qwen2.5-7B-Instruct / CounterFact. Each cell below reports the Avg. score of Eff, Gen and Spe:

Projection threshold τ\tauℓ2=0.001\ell_2=0.001ℓ2=0.01\ell_2=0.01ℓ2=0.1\ell_2=0.1ℓ2=1\ell_2=1
0.000249.7750.2149.4948.16
0.000453.7355.7654.5751.49
0.000655.3256.6855.9553.07
0.00159.5859.0758.5256.34
0.00270.3969.4068.7567.16
0.0275.7677.0977.3276.84

The best AlphaEdit configuration in this grid is τ=0.02\tau=0.02, ℓ2=0.1\ell_2=0.1. Its detailed scores are compared below with the FisherEdit result:

MethodEff.Gen.Spe.Avg.
AlphaEdit (best joint configuration)99.8065.8066.3877.32
FisherEdit99.9071.0866.6679.21

FisherEdit retains a 1.89-point Average advantage over the best AlphaEdit configuration in this joint grid, with improvements in all three component metrics.

(2) Effect of projection on actual updates.

Following your suggestion, we added a complementary experiment in Appendix K.2 (Figure 10) to examine how projection affects new-target fitting. We compute AlphaEdit and FisherEdit updates using the same model weights and editing inputs at each stage. Stronger projection increases fitting error, while FisherEdit achieves lower error than all six AlphaEdit variants at every stage, including the default τ=0.02\tau=0.02. These results indicate that hard projection can restrict new-target fitting, motivating our soft Fisher-weighted penalty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.