acceptodds
Under review as a conference paper at ICLR 2027

Fisher-LoRA: Per-Edit Rank-Adaptive Knowledge Editing via Causal Fisher Information

Abstract

Knowledge editing of large language models patches individual factual associations through low-rank weight updates, and each such editor must commit in advance to how many weight-space directions the update may occupy. However, existing editors do not answer how many a particular edit fundamentally requires: each fixes a footprint decoupled from the edit, ROME imposing rank one, MELO rank eight, and ELDER rank thirty-two, so they over-pay on easy edits and silently fail on hard ones. We propose Fisher-LoRA, a per-edit rank-adaptive editor that sets LoRA rank from a measurement on the model rather than a method-level constant: the eigenrank of an attribution-patching estimate of the Causal Fisher Information Matrix on the edit's causal subspace, obtained from one backward pass on the unedited model and read as an estimate of the edit's minimum rank. A Cram\'er-Rao-style heuristic motivates an energy estimate for the Fisher-weighted update along this subspace, and throughout the evaluation we observe an empirical rank floor at less a slack. The estimate is diagonal in the component basis, so its spectrum is read by sorting component scores and the LoRA factors start at its top eigenvectors. On COUNTERFACT and zsRE across three backbones, Fisher-LoRA reaches reliability and locality on LLaMA-2-7B, locality points above ELDER at an average rank of against its fixed , cuts ELDER's hard-slice failure rate from to , edits at lower cost, and predicts which edits defeat ROME, MEMIT and ELDER at Spearman , and at when the joint failure also covers the memory editors GRACE and WISE. Edit difficulty is, to first order, a property of the (model, edit) pair, not of the editor, and readable before any edit is attempted.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge editing, adaptive LoRA rank, Fisher information matrix, attribution patching, edit difficulty prediction, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v7TTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Fisher-LoRA, a knowledge editing method that uses contrastive-gradient scores to determine the LoRA rank and initialize adapters for each factual edit. Evaluation on COUNTERFACT and zsRE across three language model backbones examines editing reliability, locality, and computational efficiency. The paper further investigates the estimated rank as a predictor of editing difficulty.

Critical strengths
  1. Editing performance. Fisher-LoRA achieves higher reliability and locality than ELDER.
  2. Computational efficiency. The method requires lower average rank and less editing time than ELDER.
  3. Predictive analysis. Evaluation on 5,000 edits supports the estimated rank as a predictor of editing difficulty.
Critical weaknesses
  1. Minimum-rank claim. The paper interprets failures at lower ranks as evidence that an edit requires a higher rank. However, these failures may also reflect unsuccessful optimization. The experiments do not separate these explanations, leaving the claimed minimum rank insufficiently supported.

  2. Missing comparisons. The paper cites AlphaEdit [1] and RoseLoRA [2] but does not compare their editing performance. These omissions leave the advantage over closely related methods unclear.

  3. Unclear contribution of rank adaptation. Fisher-LoRA changes the rank, update directions, and initialization together. The paper does not evaluate a fixed-rank version of Fisher-LoRA, so the improvements cannot be attributed specifically to adapting the rank.

  4. Evaluation thresholds lack justification. The paper defines successful editing as achieving 95% reliability and 90% locality, but its headline failure rate counts only reliability below 50%. The choice of the 95% and 90% thresholds is not justified, and their effect on the estimated minimum ranks is not evaluated.

  5. Limited statistical validation. The main comparisons use 100 edits per benchmark and one random seed, leaving the stability of the gains unclear.

  6. Incomplete locality evaluation. Locality is reported only for LLaMA-2-7B, leaving preservation of unrelated knowledge on the other backbones unverified.

[1] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR, 2025.

[2] Wang et al. RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning. EMNLP, 2024.

Questions for authors
  1. Can the authors distinguish insufficient rank from optimization failure by testing lower ranks with multiple initializations and longer training?
  2. Can the authors compare Fisher-LoRA with AlphaEdit and RoseLoRA under the same evaluation protocol?
  3. Can the authors evaluate fixed-rank Fisher-LoRA with identical initialization and training settings at a matched average parameter budget to isolate the contribution of per-edit rank adaptation?
  4. Can the authors justify the 95% reliability and 90% locality thresholds, assess sensitivity to these choices, and report failures under this joint criterion?
  5. Can the authors repeat the main comparisons with more edits and multiple seeds, and report confidence intervals for the gains?
  6. Can the authors report locality on Mistral-7B and GPT-2-XL?
4 comments
AuthorsOpenReview ↗

Response to Reviewer v7TT

Dear Reviewer v7TT:

Thank you for recognizing the method effectiveness, the efficiency, and the thorough analysis. We have responded point by point to your concerns regarding the source of failures, more baselines, an ablation study, threshold sensitivity, more seeds, and evaluation on more models. In the marked copy appended to the revised PDF (pages 38–74), the revisions made in response to your comments are tagged with @Reviewer v7TT. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q1: Insufficient rank versus unsuccessful optimisation

A1: We respectfully clarify that the searched minimum rank is defined at a fixed optimisation budget (i.e., 25 steps and one initialisation), and a failure below it can come from the rank or from the budget, as you point out. Following your suggestion, we further search the minimum under a 16 times longer schedule with the best of three initialisations on the 100 LLaMA-2-7B COUNTERFACT test edits. Table R1 shows that the minimum equals its value at the deployed budget on 40% of the edits and lies within one of it on 67%, with a mean of 10.53 against 11.58, and the ordering of edits by difficulty is kept (Spearman 0.99). Where the minimum falls, part of it was set by optimisation, and where it stays, the result is consistent with a rank limit without proving one.

Table R1: Match between two per-edit rank readings.

Pair of readingsequal (%)within 1 (%)within 2 (%)Spearmanmean (first)mean (second)
r-hat vs r* (T = 25, one initialisation)6395981.0011.9411.58
r-hat vs count of the full Causal FIM of Eq. (5) from 64 samples per edit prompt, same V_e and tau4577920.9911.9412.64
r* (T = 25, one init.) vs r* (T = 400, best of three inits.)4067900.9911.5810.53
r-hat vs r* (T = 400, best of three inits.)3056780.9811.9410.53
r* at 95/90 vs r* at 90/853667860.9911.5810.42
r* at 95/90 vs r* at 99/952858860.9911.5812.90
r* (MELO and ELDER search) vs the same search over Fisher-LoRA6890960.9911.5811.92

Following your suggestion, we have added these results to Appendix A.24 as Table 16 (Lines 1890–1898) and to Appendix A.19 (Lines 1698–1706), stated the fixed budget of the search in Section 3 (Lines 146–147), Appendix A.14 (Lines 1381–1389), Appendix A.16 (Lines 1494–1498) and the Conclusion (Lines 484–485).


Summarized W2 & Q2: Comparison with AlphaEdit and RoseLoRA

A2: Following your suggestion, we further compare Fisher-LoRA with AlphaEdit [1] at its released configuration and RoseLoRA [2] at rank 8, the total rank MELO runs at, under the protocol of Table 1 on the same 100 test edits on the three backbones over three seeds. Table R3 (under A5) shows that Fisher-LoRA has the highest mean Reliability and Locality of the seven editors on every backbone, with a mean Reliability of at least 95.1%.

Following your suggestion, we have added these results to Appendix A.24 as Table 18 (Lines 1945–1967), and pointed to them from Section 5.2 (Lines 320–321).


Summarized W3 & Q3: Contribution of rank adaptation with a fixed-rank version

A3: Following your suggestion, we have added a fixed-rank version of Fisher-LoRA to Table 6, which keeps the component selection, the Fisher-eigenvector initialisation and the training of Fisher-LoRA and fixes the rank of every edit at 12, the average allocated rank over the admitted test edits rounded to an integer (1% more parameters per edit). It reaches 91.2% Reliability and 83.6% Locality against 95.9% and 91.2% for the full method, and both rows use the same screen, which rejects no test edit, with the comparison over the same 100 edits. At approximately the same average budget, the per-edit allocation adds 4.7 Reliability and 7.6 Locality points, and on the test edits the gain comes from the per-edit allocation, not from the average budget. On the 1,500 COUNTERFACT-HARD edits the same control reaches 76.5% Reliability against 91.2%, where the budgets are not matched, since rank 12 lies below the floor of every slice edit, and the slice shows what a fixed average rank costs on hard edits rather than a matched comparison.

Following your suggestion, we have added this row to Table 6 (Lines 394–404) with its reading in Section 5.3 (Lines 411–417), and described it in Appendix A.22 (Lines 1831–1849).

(Continued)

Summarized W4 & Q4: Justification and sensitivity of the 95% and 90% thresholds

A4: We respectfully clarify that the 95% and 90% thresholds are operating points rather than values the paper shows to be optimal, namely 1−ϵ1 - \epsilon and 1−δ1 - \delta at the tolerances ϵ=0.05\epsilon = 0.05 and δ=0.1\delta = 0.1 at which the slack of Eq. (7) is evaluated, where the slack is 1.27 dimensions, about one integer rank, the resolution at which the floor is read. On sensitivity, Table R1 (under A1) shows that the searched minimum lies within one of its 95/90 value on 67% of the edits under the looser 90/85 and on 58% under the stricter 99/95, and the ordering of edits is kept at both (Spearman 0.99). Under the joint criterion on LLaMA-2-7B, Table R2 shows that Fisher-LoRA misses 95/90 on 34.4% of the COUNTERFACT-HARD edits against 71.1% for ELDER, and stays below ROME, MEMIT and ELDER at 90/85 and 99/95, with rejected edits counted as failures.

Table R2: Criterion failure (%, count in brackets) and ES, PS and NS (%).

EditorFail 95/90, testFail 95/90, HARDFail 90/85, HARDFail 99/95, HARDFail Eq. (2), HARDESPSNS
ROME5585.7 (1,285)74.5 (1,118)90.4 (1,356)85.1 (1,276)100.098.577.0
MEMIT4480.0 (1,200)68.6 (1,029)85.1 (1,277)79.9 (1,198)98.087.075.4
MELOn/an/an/an/an/a94.083.582.9
ELDER5271.1 (1,067)60.6 (909)79.8 (1,197)70.5 (1,057)90.095.082.7
Fisher-LoRA (Ours)2334.4 (516)29.1 (436)47.8 (717)33.9 (508)98.097.090.9

Following your suggestions, we have added these results to Appendix A.24 as Tables 16 and 17 (Lines 1890–1906), stated the two thresholds and the treatment of rejected edits in Appendix A.16 (Lines 1463–1475), added the criterion-failure rate of Fisher-LoRA to Appendix A.17 (Lines 1616–1646), and revised the caption of Table 2 to define its 50% Failure column as a separate statistic (Lines 343–348) together with the hard-slice paragraph of Section 5.2 (Lines 342–359).


Summarized W5 & Q5: More edits, multiple seeds and confidence intervals

A5: Following your suggestion, we further repeat the comparison over three seeds (i.e., 42, 43 and 44) with a 95% t-interval for each editor and for the paired gain of Fisher-LoRA over ELDER. Table R3 shows that the gain over ELDER is positive on every seed and backbone (Table R4 gives each seed), with a Reliability gain of 3.0 [1.9, 4.1] on LLaMA-2-7B, 2.8 [1.4, 4.2] on Mistral-7B and 6.8 [3.5, 10.0] on GPT-2-XL. A seed changes none of the entries of ROME, MEMIT and AlphaEdit, which write closed-form updates, while it changes the random initialisation of MELO, ELDER and RoseLoRA and, for every editor trained on the loss of Eq. (10), Fisher-LoRA included, the unrelated prompts drawn at each step. Rejected edits of Fisher-LoRA score zero. Beyond the 100 test edits, the stratified analyses use the 5,000-edit pool and the 1,500-edit COUNTERFACT-HARD slice, with exact Clopper-Pearson intervals for the per-decile failure rates in Table 11. These analyses, zsRE, AdaLoRA and GaLore stay at seed 42, and a larger set of test edits is not supplied in this revision.

(Continued)

Table R3: Reliability and Locality (%) on the 100 COUNTERFACT test edits over three seeds, with 95% t-intervals.

EditorBackboneRel. mean [95% CI]Loc. mean [95% CI]
ROMELLaMA-2-7B74.0 [74.0, 74.0]82.1 [82.1, 82.1]
ROMEMistral-7B71.5 [71.5, 71.5]77.9 [77.9, 77.9]
ROMEGPT-2-XL69.8 [69.8, 69.8]77.1 [77.1, 77.1]
MEMITLLaMA-2-7B80.6 [80.6, 80.6]84.7 [84.7, 84.7]
MEMITMistral-7B78.2 [78.2, 78.2]87.9 [87.9, 87.9]
MEMITGPT-2-XL76.3 [76.3, 76.3]74.3 [74.3, 74.3]
MELOLLaMA-2-7B86.2 [85.7, 86.8]79.2 [76.2, 82.1]
MELOMistral-7B84.1 [80.3, 87.8]78.5 [69.9, 87.0]
MELOGPT-2-XL84.6 [83.1, 86.1]76.4 [71.5, 81.2]
ELDERLLaMA-2-7B93.1 [92.8, 93.5]77.3 [72.6, 82.0]
ELDERMistral-7B92.3 [91.1, 93.6]78.4 [75.7, 81.2]
ELDERGPT-2-XL89.1 [85.6, 92.6]74.9 [60.0, 89.8]
AlphaEdit [1]LLaMA-2-7B84.2 [84.2, 84.2]87.3 [87.3, 87.3]
AlphaEdit [1]Mistral-7B78.1 [78.1, 78.1]90.3 [90.3, 90.3]
AlphaEdit [1]GPT-2-XL77.6 [77.6, 77.6]78.9 [78.9, 78.9]
RoseLoRA [2]LLaMA-2-7B86.4 [84.3, 88.6]78.4 [66.8, 90.1]
RoseLoRA [2]Mistral-7B84.6 [74.3, 94.9]75.7 [65.8, 85.7]
RoseLoRA [2]GPT-2-XL84.2 [76.7, 91.7]73.3 [62.6, 84.0]
Fisher-LoRA (Ours)LLaMA-2-7B96.1 [95.3, 96.9]91.0 [90.3, 91.6]
Fisher-LoRA (Ours)Mistral-7B95.1 [94.8, 95.5]91.3 [91.1, 91.5]
Fisher-LoRA (Ours)GPT-2-XL95.9 [95.6, 96.2]88.9 [86.9, 91.0]
Fisher-LoRA minus ELDERLLaMA-2-7B3.0 [1.9, 4.1]13.7 [9.2, 18.2]
Fisher-LoRA minus ELDERMistral-7B2.8 [1.4, 4.2]12.9 [10.3, 15.5]
Fisher-LoRA minus ELDERGPT-2-XL6.8 [3.5, 10.0]14.0 [0.4, 27.6]

Table R4: Reliability and Locality (%) of each training seed on the 100 COUNTERFACT test edits.

EditorBackboneRel. 42Rel. 43Rel. 44Loc. 42Loc. 43Loc. 44
ROMELLaMA-2-7B74.074.074.082.182.182.1
ROMEMistral-7B71.571.571.577.977.977.9
ROMEGPT-2-XL69.869.869.877.177.177.1
MEMITLLaMA-2-7B80.680.680.684.784.784.7
MEMITMistral-7B78.278.278.287.987.987.9
MEMITGPT-2-XL76.376.376.374.374.374.3
MELOLLaMA-2-7B86.486.086.380.578.278.8
MELOMistral-7B84.082.685.677.882.275.4
MELOGPT-2-XL84.983.985.075.175.478.6
ELDERLLaMA-2-7B93.193.393.079.275.477.3
ELDERMistral-7B92.092.192.978.079.777.6
ELDERGPT-2-XL90.487.689.368.279.776.9
AlphaEdit [1]LLaMA-2-7B84.284.284.287.387.387.3
AlphaEdit [1]Mistral-7B78.178.178.190.390.390.3
AlphaEdit [1]GPT-2-XL77.677.677.678.978.978.9
RoseLoRA [2]LLaMA-2-7B86.285.787.476.974.783.7
RoseLoRA [2]Mistral-7B80.684.388.980.172.274.9
RoseLoRA [2]GPT-2-XL87.284.281.270.271.578.2
Fisher-LoRA (Ours)LLaMA-2-7B95.996.096.591.291.090.7
Fisher-LoRA (Ours)Mistral-7B95.395.095.191.291.491.3
Fisher-LoRA (Ours)GPT-2-XL96.095.895.888.589.988.4

Following your suggestion, we have added these results to Appendix A.24 as Tables 18 and 19 (Lines 1945–1990), and pointed to them from Section 5.1 (Lines 298–302) and Appendix A.14 (Lines 1381–1389).


Summarized W6 & Q6: Locality on Mistral-7B and GPT-2-XL

A6: As reported in Table R3 of our response to your W5, Fisher-LoRA has the highest mean Locality of the seven editors on Mistral-7B and GPT-2-XL, with 91.3% and 88.9% against 78.4% and 74.9% for ELDER. Locality on zsRE is reported on LLaMA-2-7B, and the revision does not add it for the other two backbones.

Following your suggestion, we have added these results to Appendix A.24 as Table 18 (Lines 1945–1967).


References

[1] Junfeng Fang et al. AlphaEdit: Null-space constrained knowledge editing for language models. ICLR, 2025.

[2] Haoyu Wang et al. RoseLoRA: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. EMNLP, 2024.


Best Wishes,

The Authors of Paper 53337

Dear Reviewer v7TT,

Thank you for your time in reviewing our work, and for recognizing its editing performance, computational efficiency, and predictive analysis.

We believe that our rebuttal has addressed your concerns, with the longer-budget search of the minimum rank, the comparison with AlphaEdit and RoseLoRA, the fixed-rank version of Fisher-LoRA, the threshold sensitivity, and the results over three seeds with Locality on all three backbones. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 53337

Reviewer_jEa7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fisher-LoRA, which assigns a LoRA rank to each knowledge edit using a diagonal approximation to a causal Fisher matrix derived from one backward pass. The estimated spectrum also guides initialization and budget screening before optimization. Experiments on COUNTERFACT and zsRE across three models report improvements in reliability and locality with a lower average rank than ELDER. The paper includes difficulty-stratified comparisons, component ablations, and sensitivity analyses.

Critical strengths

1.The paper contains a substantial amount of experimental work, including evaluations across three models and two benchmarks, a difficulty analysis, component ablations, and sensitivity studies.

2.The proposed method achieves competitive results. On LLaMA-2-7B, it improves reliability and locality over ELDER while using an lower rank.

Critical weaknesses

1.The evaluation is limited to independent single-fact edits. Existing methods can achieve strong reliability and locality in batch and sequential editing settings, so the gains reported in the paper don’t give a practical advantage. The paper does not show whether its rank estimates remain useful after repeated updates or whether the method preserves earlier edits. Therefore, the practicality of this method may be limited.

2.Most of the methods used for comparing papers are old, with ELDER being the only method from 2025 in the main comparison. Comparisons with more recent methods, such as AlphaEdit, are missing, making it difficult to assess how Fisher-LoRA compares with the current state of the art.

Questions for authors

1.Evaluation method. Why replace COUNTERFACT’s original ES, PS, and NS with teacher-forced token accuracy and pre/post-edit argmax agreement? Please explain the reason or report the original metrics on the same edited models and clarify whether main conclusions remain consistent.

2.Robustness and resource costs. The paper includes sensitivity studies, but these mainly cover one model. How stable are the results across models, seeds, or rank budgets? Please also report peak GPU memory and additional storage alongside runtime, including setup costs separately.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jEa7

Dear Reviewer jEa7:

Thank you for recognizing the extensive experiments and the method effectiveness. We have responded point by point to your concerns regarding the editing setting, recent baselines, evaluation metrics, robustness, and resource costs. In the marked copy appended to the revised PDF (pages 38–74), the revisions made in response to your comments are tagged with @Reviewer jEa7. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Single-fact edits versus batch and sequential editing

A1: We respectfully clarify that Fisher-LoRA is defined for single-fact edits, one at a time, and the revised paper makes no claim about batch or sequential editing and adds no experiment in those regimes. The rank is read from one fact on the model at hand, and a rank under repeated updates has to be read from the edited model and account for the earlier edits, a separate problem in Appendix A.0. Within the single-edit scope, Fisher-LoRA reaches 95.9% Reliability and 91.2% Locality on the 100 LLaMA-2-7B test edits, none of them rejected by its screen, against 93.1% and 79.2% for ELDER (Table 1, Lines 270–277).


Summarized W2: Comparison with recent methods such as AlphaEdit

A2: Following your suggestion, we further compare Fisher-LoRA with two recent methods (i.e., AlphaEdit [1] at its released configuration and RoseLoRA [2] at rank 8, the total rank MELO runs at) on the 100 test edits of Table 1 on the three backbones over three seeds. Table R2 (under A4) shows that Fisher-LoRA has the highest mean Reliability and Locality of the seven editors on every backbone, with a mean Reliability of at least 95.1%.

Following your suggestion, we have added these results to Appendix A.24 as Table 18 (Lines 1945–1967), and pointed to them from Section 5.2 (Lines 320–321).


Summarized Q1: Why replace ES, PS and NS with token accuracy?

A3: We respectfully clarify that we replace ES, PS and NS with token accuracy and argmax preservation because the rank search needs a per-edit pass or fail at a level close to one, which the share of target tokens ranked first gives for multi-token targets, while ES, PS and NS [3] test an ordering of the new and old target probabilities. Following your advice, we additionally report ES, PS and NS on the same 100 LLaMA-2-7B test edits, with a rejected edit scoring zero and n/a marking the criterion failure the paper does not report for MELO. Table R1 shows that Fisher-LoRA also leads on NS, as on Locality in Table 1, with 90.9 against at most 82.9 for the baselines (a margin of 8.0 on NS and 5.3 on Locality), while Fisher-LoRA is below ROME on ES and PS (98.0 and 97.0 against 100.0 and 98.5).

Table R1: Criterion failure (%, count in brackets) and ES, PS and NS (%).

EditorFail 95/90, testFail 95/90, HARDFail 90/85, HARDFail 99/95, HARDFail Eq. (2), HARDESPSNS
ROME5585.7 (1,285)74.5 (1,118)90.4 (1,356)85.1 (1,276)100.098.577.0
MEMIT4480.0 (1,200)68.6 (1,029)85.1 (1,277)79.9 (1,198)98.087.075.4
MELOn/an/an/an/an/a94.083.582.9
ELDER5271.1 (1,067)60.6 (909)79.8 (1,197)70.5 (1,057)90.095.082.7
Fisher-LoRA (Ours)2334.4 (516)29.1 (436)47.8 (717)33.9 (508)98.097.090.9

Following your advice, we have added these results to Appendix A.24 as Table 17 (Lines 1900–1906).

(Continued)

Summarized Q2: Robustness across models, seeds and budgets, and resource costs

A4: Following your suggestion, we further report Reliability and Locality over three seeds (i.e., 42, 43 and 44) on the three backbones, the rank-budget sweep, and the resource costs (including peak GPU memory, one-time setup and storage). Table R2 shows that Fisher-LoRA is ahead of ELDER on every seed and backbone (Table R4 gives each seed), with a paired Reliability gain of 3.0 [1.9, 4.1] on LLaMA-2-7B, 2.8 [1.4, 4.2] on Mistral-7B and 6.8 [3.5, 10.0] on GPT-2-XL. Rejected edits score zero, and AdaLoRA and GaLore are not repeated over seeds. A seed changes none of the entries of ROME, MEMIT and AlphaEdit, which write closed-form updates. On the rank budget, Table R5 shows that from a budget of 16 to the deployed 40 the Reliability of Fisher-LoRA rises from 73.9% to 95.9% and the rejected test edits fall from 24 to 0.

Table R2: Reliability and Locality (%) on the 100 COUNTERFACT test edits over three seeds, with 95% t-intervals.

EditorBackboneRel. mean [95% CI]Loc. mean [95% CI]
ROMELLaMA-2-7B74.0 [74.0, 74.0]82.1 [82.1, 82.1]
ROMEMistral-7B71.5 [71.5, 71.5]77.9 [77.9, 77.9]
ROMEGPT-2-XL69.8 [69.8, 69.8]77.1 [77.1, 77.1]
MEMITLLaMA-2-7B80.6 [80.6, 80.6]84.7 [84.7, 84.7]
MEMITMistral-7B78.2 [78.2, 78.2]87.9 [87.9, 87.9]
MEMITGPT-2-XL76.3 [76.3, 76.3]74.3 [74.3, 74.3]
MELOLLaMA-2-7B86.2 [85.7, 86.8]79.2 [76.2, 82.1]
MELOMistral-7B84.1 [80.3, 87.8]78.5 [69.9, 87.0]
MELOGPT-2-XL84.6 [83.1, 86.1]76.4 [71.5, 81.2]
ELDERLLaMA-2-7B93.1 [92.8, 93.5]77.3 [72.6, 82.0]
ELDERMistral-7B92.3 [91.1, 93.6]78.4 [75.7, 81.2]
ELDERGPT-2-XL89.1 [85.6, 92.6]74.9 [60.0, 89.8]
AlphaEdit [1]LLaMA-2-7B84.2 [84.2, 84.2]87.3 [87.3, 87.3]
AlphaEdit [1]Mistral-7B78.1 [78.1, 78.1]90.3 [90.3, 90.3]
AlphaEdit [1]GPT-2-XL77.6 [77.6, 77.6]78.9 [78.9, 78.9]
RoseLoRA [2]LLaMA-2-7B86.4 [84.3, 88.6]78.4 [66.8, 90.1]
RoseLoRA [2]Mistral-7B84.6 [74.3, 94.9]75.7 [65.8, 85.7]
RoseLoRA [2]GPT-2-XL84.2 [76.7, 91.7]73.3 [62.6, 84.0]
Fisher-LoRA (Ours)LLaMA-2-7B96.1 [95.3, 96.9]91.0 [90.3, 91.6]
Fisher-LoRA (Ours)Mistral-7B95.1 [94.8, 95.5]91.3 [91.1, 91.5]
Fisher-LoRA (Ours)GPT-2-XL95.9 [95.6, 96.2]88.9 [86.9, 91.0]
Fisher-LoRA minus ELDERLLaMA-2-7B3.0 [1.9, 4.1]13.7 [9.2, 18.2]
Fisher-LoRA minus ELDERMistral-7B2.8 [1.4, 4.2]12.9 [10.3, 15.5]
Fisher-LoRA minus ELDERGPT-2-XL6.8 [3.5, 10.0]14.0 [0.4, 27.6]

On resource costs, Table R5 shows that Fisher-LoRA peaks at 15.8 GB of GPU memory, against 18.2 GB for ELDER and 15.1 GB for MELO. The Edit Cost of Table 1 already includes the backward pass that reads the rank (10.1 s against 19.7 s for ELDER), and adding the one-time setup of 81 s spread over the 100 edits gives 10.91 s per edit against 20.57 s. Table R3, computed from the factor shapes and the public model configurations, shows that an edit at the average allocated rank stores 0.90 million parameters on LLaMA-2-7B against 2.42 million for a rank-32 adapter on the same layers.

Table R3: Parameters an edit stores on each backbone.

Backboned_in + d_outlayers in Sparameters per unit of rankat rank 32 (millions)at the average rank (millions)
LLaMA-2-7B11008 + 4096575,5202.420.90 (rank 11.9)
Mistral-7B14336 + 4096592,1602.951.04 (rank 11.3)
GPT-2-XL6400 + 1600648,0001.540.58 (rank 12.0)

(Continued)

Table R4: Reliability and Locality (%) of each training seed on the 100 COUNTERFACT test edits.

EditorBackboneRel. 42Rel. 43Rel. 44Loc. 42Loc. 43Loc. 44
ROMELLaMA-2-7B74.074.074.082.182.182.1
ROMEMistral-7B71.571.571.577.977.977.9
ROMEGPT-2-XL69.869.869.877.177.177.1
MEMITLLaMA-2-7B80.680.680.684.784.784.7
MEMITMistral-7B78.278.278.287.987.987.9
MEMITGPT-2-XL76.376.376.374.374.374.3
MELOLLaMA-2-7B86.486.086.380.578.278.8
MELOMistral-7B84.082.685.677.882.275.4
MELOGPT-2-XL84.983.985.075.175.478.6
ELDERLLaMA-2-7B93.193.393.079.275.477.3
ELDERMistral-7B92.092.192.978.079.777.6
ELDERGPT-2-XL90.487.689.368.279.776.9
AlphaEdit [1]LLaMA-2-7B84.284.284.287.387.387.3
AlphaEdit [1]Mistral-7B78.178.178.190.390.390.3
AlphaEdit [1]GPT-2-XL77.677.677.678.978.978.9
RoseLoRA [2]LLaMA-2-7B86.285.787.476.974.783.7
RoseLoRA [2]Mistral-7B80.684.388.980.172.274.9
RoseLoRA [2]GPT-2-XL87.284.281.270.271.578.2
Fisher-LoRA (Ours)LLaMA-2-7B95.996.096.591.291.090.7
Fisher-LoRA (Ours)Mistral-7B95.395.095.191.291.491.3
Fisher-LoRA (Ours)GPT-2-XL96.095.895.888.589.988.4

Table R5: Rank budget with the average rank over the admitted edits, peak memory and one-time setup on LLaMA-2-7B.

Rank budgetRejectedAverage rankRel.Loc.
16247.573.971.0
24119.685.882.1
32211.494.089.6
40 (deployed)011.995.991.2
EditorPeak memory (GB)Setup (s)Edit Cost with setup (s)
ROME16.01,38525.05
MEMIT17.71,67130.71
MELO15.18110.21
ELDER18.28720.57
AdaLoRA19.85623.06
GaLore17.06320.73
Fisher-LoRA (Ours)15.88110.91

Following your suggestions, we have added these results to Appendix A.24 as Tables 18, 19 and 20 (Lines 1945–1967, 1970–1990 and 1992–1997) and to Appendix A.20 as Tables 14 and 15 (Lines 1728–1737) with their two paragraphs (Lines 1764–1781), and pointed to them from Section 5.1 (Lines 298–302), Section 5.3 (Lines 447–450) and Appendix A.14 (Lines 1381–1389).


References

[1] Junfeng Fang et al. AlphaEdit: Null-space constrained knowledge editing for language models. ICLR, 2025.

[2] Haoyu Wang et al. RoseLoRA: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. EMNLP, 2024.

[3] Kevin Meng et al. Locating and editing factual associations in GPT. NeurIPS, 2022.


Best Wishes,

The Authors of Paper 53337

Dear Reviewer jEa7,

Thank you for your time in reviewing our work, and for recognizing its substantial experimental work and competitive results.

We believe that our rebuttal has addressed your concerns, with the comparison with AlphaEdit and RoseLoRA, the original ES, PS and NS metrics, the results over three seeds and rank budgets, and the resource costs. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 53337

Reviewer_1JxcOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Fisher-LoRA addresses per-edit rank allocation for single-fact knowledge editing. Its attribution-patching single-pass estimator approximates the Causal Fisher Information Matrix (Causal FIM), estimates edit-intrinsic complexity through a relative eigenvalue cutoff, and supplies directions for LoRA initialisation. The empirical rank floor supports a feasibility screen that rejects an edit when r^⋆−ϕ\hat r^\star-\phi exceeds the rank budget. Experiments evaluate Reliability on COUNTERFACT and zsRE across three backbones; Locality and the COUNTERFACT Edit Cost comparison are reported for LLaMA-2-7B.

Critical strengths
  1. The paper addresses a practical limitation of fixed-rank editors: different edits may require different capacity. Estimating that allocation before optimisation, with one additional backward pass, is an appealing approach under a limited editing budget.
  2. Table 1 reports a favourable trade-off on the 100 COUNTERFACT edits with LLaMA-2-7B. Fisher-LoRA achieves 96.4% Reliability and 91.3% Locality at an average allocated rank of 11.4, exceeding ELDER on both metrics despite ELDER's fixed rank of 32. Edit Cost includes the rank-estimation pass.
  3. The separate development split and reuse of the COUNTERFACT configuration on zsRE reduce concerns about evaluation-specific tuning. Results by difficulty decile and the Fisher-eigenvector initialisation ablation also reveal patterns that aggregate scores would conceal.
Critical weaknesses
  1. The interpretation as edit-intrinsic complexity remains stronger than the evidence establishes. Equations (8)–(9) construct a matrix of squared contrastive-gradient component scores that is diagonal in the component basis, and Equation (6) counts eigenvalues satisfying λi≥τλmax⁡\lambda_i\geq\tau\lambda_{\max}. This count is neither the matrix rank of the weight update nor necessarily the effective rank of the Causal FIM in Equation (5). Appendix A.6 acknowledges a diagonal-approximation bias that persists as the linearisation error vanishes; Appendix A.4 also notes that a rank-one update can have nonzero components along multiple eigendirections. The heuristic energy argument therefore does not establish the required connection. The rank agreement in Appendices A.19–A.20 is useful empirical evidence, but the reference search still identifies the smallest successful rank under selected editors and optimisation budgets, rather than an editor-independent minimum.
  2. The mapping from the component basis to the per-layer LoRA factors is underspecified. Section 4.2 and Appendix A.11 give each parameter-space basis vector support in one module, while Algorithm 1 restricts the global top-rr directions to each layer and sets Bℓ=0B_\ell=0. If these are literal coordinate restrictions, without supplying additional directions for zero rows, directions supported in other layers produce zero rows of AℓA_\ell paired with zero columns of BℓB_\ell. Such pairs would retain zero gradients under the stated optimisation, so some layers would have fewer than rr active directions. Appendix A.4 already defines the factor shapes, but the conversion into those shapes remains unclear. The paper also does not specify how subsequent optimisation maintains the selected subspace. These details affect the interpretation of the effective update rank, parameter budgets, and the subspace ablations.
  3. The headline failure reduction does not measure the success criterion used in rank search. Table 2's 7.6% failure rate on LLaMA-2-7B/COUNTERFACT-HARD counts edits with Reliability below 50%; rank search requires at least 95% Reliability and 90% Locality. Appendix A.16 discloses this distinction, but Fisher-LoRA's criterion-failure rate on the hard slice is missing, although Appendix A.17 provides it for several baselines. This matters because, in the hardest decile of the 5,000-edit LLaMA-2-7B pool, Fisher-LoRA's Locality is 81.2%, versus ELDER's 88.7%. Furthermore, teacher-forced token accuracy and argmax preservation do not directly verify Equation (2)'s probability-mass and KL constraints. Halving the reported failure rate therefore does not establish more frequent satisfaction of the editing specification.
Questions for authors
  1. Beyond the rank agreement already reported in Appendices A.19–A.20, how closely does the estimated rank match the effective rank of the full Causal FIM in Equation (5), using the same causal subspace and relative cutoff? Does agreement with the searched minimum rank persist under a larger optimisation budget for the reference search?
  2. How are global parameter-space directions converted into rows of each AℓA_\ell? Does the implementation select directions independently for each layer, or supply additional directions where the global vectors restrict to zero? Please specify each layer's number of trainable directions and the operations that maintain the subspace constraint.
  3. Please report Fisher-LoRA's criterion-failure rates—Reliability below 95% or Locality below 90%—on the main test set and COUNTERFACT-HARD alongside the existing baseline results. Are edits rejected by the feasibility screen included in the denominator? Do direct probability-mass and KL measurements support the same conclusion?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Jxc

Dear Reviewer 1Jxc:

Thank you for recognizing the practical motivation, the method effectiveness, and the thorough analysis. We have responded point by point to your concerns regarding the meaning of the estimated rank, the mapping from components to the per-layer LoRA factors, and the success criterion. In the marked copy appended to the revised PDF (pages 38–74), the revisions made in response to your comments are tagged with @Reviewer 1Jxc. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q1: Estimated rank versus full Causal FIM and longer-budget minimum

A1: We respectfully clarify that at the same causal subspace and cutoff the count of the full Causal FIM lies within one of the estimated rank r^⋆\hat{r}^\star on 77% of the 100 LLaMA-2-7B COUNTERFACT test edits (Spearman 0.99), and under a larger optimisation budget the rank order of edits persists (Spearman 0.98) while the share of edits with r^⋆\hat{r}^\star within one of the searched minimum falls from 95% to 56%, as shown in Table R1. As in your reading, r^⋆\hat{r}^\star counts eigenvalues of the estimator in Eq. (6), which is neither the matrix rank of the update nor, in general, the effective rank of the full Causal FIM of Eq. (5), and the two counts part where the order-one term of Eq. (16), which the estimator drops, moves eigenvalues across the cutoff. The searched minimum r⋆r^\star is defined over the procedures of MELO and ELDER at a fixed budget, and at that budget and to first order it is a property of the (model, edit) pair rather than of the editors searched, with the same search over Fisher-LoRA within one of it on 90% of the edits, while part of a minimum found at T=25T = 25 is set by optimisation.

Table R1: Match between two per-edit rank readings.

Pair of readingsequal (%)within 1 (%)within 2 (%)Spearmanmean (first)mean (second)
r-hat vs r* (T = 25, one initialisation)6395981.0011.9411.58
r-hat vs count of the full Causal FIM of Eq. (5) from 64 samples per edit prompt, same V_e and tau4577920.9911.9412.64
r* (T = 25, one init.) vs r* (T = 400, best of three inits.)4067900.9911.5810.53
r-hat vs r* (T = 400, best of three inits.)3056780.9811.9410.53
r* at 95/90 vs r* at 90/853667860.9911.5810.42
r* at 95/90 vs r* at 99/952858860.9911.5812.90
r* (MELO and ELDER search) vs the same search over Fisher-LoRA6890960.9911.5811.92

Following your suggestions, we have added these results to Appendix A.24 as Table 16 (Lines 1890–1898) and to Appendix A.19 (Lines 1698–1706), and revised the definition of the searched minimum in Section 3 (Lines 146–147) and Appendix A.16 (Lines 1494–1498).


Summarized W2 & Q2: Mapping from components to the per-layer LoRA factors

A2: We respectfully clarify that directions are selected independently for each layer, every layer of SS trains rr rows of AℓA_\ell and rr columns of BℓB_\ell, and no operation maintains a subspace constraint. Each component is the unit vector along the contrastive gradient geg_e on the row of input weights of one output neuron of the down-projection, and it belongs to one layer. The ii-th row of AℓA_\ell, for i≤r=min⁡(r^⋆,rbudget)i \le r = \min(\hat{r}^\star, r^{budget}), is set to the ii-th highest-scoring component of layer ℓ\ell, inside or outside VeV_e, with Bℓ=0B_\ell = 0, and a column of BℓB_\ell receives a non-zero gradient whenever its row of AℓA_\ell responds to the input and the output gradient is non-zero, and no pair is held at zero by construction. Phase 3 then updates both factors freely by Adam, and at the first step the rows with a non-zero score span at most min⁡(r,npos)\min(r, n_{pos}) directions (nposn_{pos} the token positions of the backward pass). The subspace rows of Table 6 differ, as they hold the rows of AℓA_\ell in a fixed subspace, and the step to the full method releases that constraint together with the initialisation.

Following your suggestions, we have revised Section 4.2 (Lines 184–188), Section 4.5 (Lines 257–258 and 262–269), Appendix A.11 (Lines 1273–1283 and 1292–1303), Appendix A.12 with Algorithm 1 (Lines 1325–1377, line 13 at Lines 1371–1372) and the ablation-ladder rows in Appendix A.22 (Lines 1831–1849) to define this mapping.

(Continued)

Summarized W3 & Q3: Criterion-failure rates, rejected edits, and probability-mass and KL measurements

A3: Following your suggestion, we report the criterion-failure rates of Fisher-LoRA next to the baselines on LLaMA-2-7B (the paper reports none for MELO, marked n/a), where the edits the screen rejects stay in the denominator and count as failures, and score zero in the Reliability and Locality means. Table R2 shows that Fisher-LoRA misses the 95/90 criterion on 34.4% of the 1,500 COUNTERFACT-HARD edits against 71.1% for ELDER, and on 23% of the 100 test edits against 52%. Under the per-edit form of Eq. (2) itself (i.e., target probability at least 95% and mean KL divergence over the unrelated prompts at most 0.1), of which token accuracy and argmax preservation are surrogates, the failure on the slice is 33.9% against 70.5% for ELDER, with Fisher-LoRA below ROME, MEMIT and ELDER as under 95/90. On the hardest decile the Locality of Fisher-LoRA falls to 80.3% (81.2% in the reviewed version) against 88.7% for ELDER, where the locality half of the criterion binds for Fisher-LoRA and the criterion-failure rate is the statistic to read. Of the Fisher-LoRA failures, 6 on the slice are rejected edits, and the screen rejects no test edit.

Table R2: Criterion failure (%, count in brackets) and ES, PS and NS (%).

EditorFail 95/90, testFail 95/90, HARDFail 90/85, HARDFail 99/95, HARDFail Eq. (2), HARDESPSNS
ROME5585.7 (1,285)74.5 (1,118)90.4 (1,356)85.1 (1,276)100.098.577.0
MEMIT4480.0 (1,200)68.6 (1,029)85.1 (1,277)79.9 (1,198)98.087.075.4
MELOn/an/an/an/an/a94.083.582.9
ELDER5271.1 (1,067)60.6 (909)79.8 (1,197)70.5 (1,057)90.095.082.7
Fisher-LoRA (Ours)2334.4 (516)29.1 (436)47.8 (717)33.9 (508)98.097.090.9

Following your suggestions, we have added these results to Appendix A.24 as Table 17 (Lines 1900–1906) and to Appendix A.17 (Lines 1616–1646), revised the caption of Table 2 to define its 50% Failure column (Lines 343–348) together with the hard-slice paragraph of Section 5.2 (Lines 342–359), and stated in Section 5.2 (Lines 319–320) and Appendix A.16 (Lines 1463–1475) how rejected edits are scored and that the accuracy criterion is a surrogate for Eq. (2).


Best Wishes,

The Authors of Paper 53337

Dear Reviewer 1Jxc,

Thank you for your time in reviewing our work, and for recognizing that per-edit rank allocation addresses a practical limitation of fixed-rank editors.

We believe that our rebuttal has addressed your concerns, with the comparison of the estimated rank against the full Causal FIM and a longer-budget search, the clarified mapping to the per-layer LoRA factors, and the criterion-failure rates of Fisher-LoRA. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 53337

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Line references refer to the clean copy, PDF pages 1–37.

Summary

Knowledge editors fix the rank of the update before the edit is seen, while the rank an edit needs differs from edit to edit. Fisher-LoRA sets the LoRA rank of each edit from one backward pass on the unedited model, as the eigenrank of an attribution-patching estimate of the Causal Fisher Information Matrix. The revision adds AlphaEdit and RoseLoRA, against which Fisher-LoRA has the highest mean Reliability and Locality of the seven editors on every backbone over three seeds (Table 18), together with a fixed-rank version of Fisher-LoRA, criterion-failure rates, and a longer-budget rank search.

Strengths

  • Practical problem. Addresses a practical limitation of fixed-rank editors (1Jxc).
  • Strong performance. A favourable trade-off (1Jxc); competitive results (jEa7); higher Reliability and Locality than ELDER (v7TT).
  • Efficiency. Lower average rank and less editing time than ELDER (v7TT).
  • Broad experiments. Substantial experimental work (jEa7); analyses by difficulty decile (1Jxc); predictive analysis on 5,000 edits (v7TT).

Weaknesses and Revisions

  1. Recent baselines (jEa7, v7TT). Added AlphaEdit and RoseLoRA on three backbones to Appendix A.24 (Table 18, Lines 1945–1967), pointed to from Section 5.2 (Lines 320–321).
  2. Seeds and confidence intervals (jEa7, v7TT). Repeated the comparison over seeds 42, 43 and 44 with 95% t-intervals, where the paired Reliability gain over ELDER is positive on every backbone (Tables 18 and 19, Lines 1945–1990).
  3. Criterion-failure rates (1Jxc, v7TT). Added the criterion failure of Fisher-LoRA, with rejected edits counted as failures, at 95/90, 90/85, 99/95 and the per-edit form of Eq. (2) (Table 17, Lines 1900–1906; Appendix A.17, Lines 1616–1646).
  4. Locality on Mistral-7B and GPT-2-XL (v7TT). Added to Table 18 (Lines 1945–1967).
  5. Fixed-rank version (v7TT). Added a fixed-rank row to Table 6 (Lines 394–404) with its reading in Section 5.3 (Lines 411–417) and Appendix A.22 (Lines 1831–1849).
  6. Original ES, PS and NS (jEa7). Added to Table 17 (Lines 1900–1906).
  7. Rank budget and resource costs (jEa7). Added the rank-budget sweep, peak memory and one-time setup to Appendix A.20 (Tables 14 and 15, Lines 1728–1737), and the parameters an edit stores (Table 20, Lines 1992–1997).
  8. Estimated rank versus searched minimum (1Jxc, v7TT). Compared the estimate with the full Causal FIM and with a 16 times longer search (Table 16, Lines 1890–1898; Appendix A.19, Lines 1698–1706), and defined the searched minimum at a fixed budget (Section 3, Lines 146–147).
  9. Mapping to the LoRA factors (1Jxc). Revised Section 4.2 (Lines 184–188), Section 4.5 (Lines 257–258 and 262–269), Appendix A.11 (Lines 1273–1303) and line 13 of Algorithm 1 (Lines 1371–1372).
  10. Thresholds (v7TT). Stated the 95% and 90% thresholds and the scoring of rejected edits in Appendix A.16 (Lines 1463–1475).
  11. Single-edit scope (jEa7). Clarified that the framework is defined for one edit at a time, as stated in Appendix A.0 (Lines 724–730).

In the marked copy (PDF pages 38–74), every revision made in response to a reviewer is tagged with that reviewer's ID.

All experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 53337

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.