- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Fisher-LoRA, a knowledge editing method that uses contrastive-gradient scores to determine the LoRA rank and initialize adapters for each factual edit. Evaluation on COUNTERFACT and zsRE across three language model backbones examines editing reliability, locality, and computational efficiency. The paper further investigates the estimated rank as a predictor of editing difficulty.
Critical strengths
- Editing performance. Fisher-LoRA achieves higher reliability and locality than ELDER.
- Computational efficiency. The method requires lower average rank and less editing time than ELDER.
- Predictive analysis. Evaluation on 5,000 edits supports the estimated rank as a predictor of editing difficulty.
Critical weaknesses
-
Minimum-rank claim. The paper interprets failures at lower ranks as evidence that an edit requires a higher rank. However, these failures may also reflect unsuccessful optimization. The experiments do not separate these explanations, leaving the claimed minimum rank insufficiently supported.
-
Missing comparisons. The paper cites AlphaEdit [1] and RoseLoRA [2] but does not compare their editing performance. These omissions leave the advantage over closely related methods unclear.
-
Unclear contribution of rank adaptation. Fisher-LoRA changes the rank, update directions, and initialization together. The paper does not evaluate a fixed-rank version of Fisher-LoRA, so the improvements cannot be attributed specifically to adapting the rank.
-
Evaluation thresholds lack justification. The paper defines successful editing as achieving 95% reliability and 90% locality, but its headline failure rate counts only reliability below 50%. The choice of the 95% and 90% thresholds is not justified, and their effect on the estimated minimum ranks is not evaluated.
-
Limited statistical validation. The main comparisons use 100 edits per benchmark and one random seed, leaving the stability of the gains unclear.
-
Incomplete locality evaluation. Locality is reported only for LLaMA-2-7B, leaving preservation of unrelated knowledge on the other backbones unverified.
[1] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR, 2025.
[2] Wang et al. RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning. EMNLP, 2024.
Questions for authors
- Can the authors distinguish insufficient rank from optimization failure by testing lower ranks with multiple initializations and longer training?
- Can the authors compare Fisher-LoRA with AlphaEdit and RoseLoRA under the same evaluation protocol?
- Can the authors evaluate fixed-rank Fisher-LoRA with identical initialization and training settings at a matched average parameter budget to isolate the contribution of per-edit rank adaptation?
- Can the authors justify the 95% reliability and 90% locality thresholds, assess sensitivity to these choices, and report failures under this joint criterion?
- Can the authors repeat the main comparisons with more edits and multiple seeds, and report confidence intervals for the gains?
- Can the authors report locality on Mistral-7B and GPT-2-XL?
4 comments
(Continued)
Summarized W4 & Q4: Justification and sensitivity of the 95% and 90% thresholds
A4: We respectfully clarify that the 95% and 90% thresholds are operating points rather than values the paper shows to be optimal, namely and at the tolerances and at which the slack of Eq. (7) is evaluated, where the slack is 1.27 dimensions, about one integer rank, the resolution at which the floor is read. On sensitivity, Table R1 (under A1) shows that the searched minimum lies within one of its 95/90 value on 67% of the edits under the looser 90/85 and on 58% under the stricter 99/95, and the ordering of edits is kept at both (Spearman 0.99). Under the joint criterion on LLaMA-2-7B, Table R2 shows that Fisher-LoRA misses 95/90 on 34.4% of the COUNTERFACT-HARD edits against 71.1% for ELDER, and stays below ROME, MEMIT and ELDER at 90/85 and 99/95, with rejected edits counted as failures.
Table R2: Criterion failure (%, count in brackets) and ES, PS and NS (%).
| Editor | Fail 95/90, test | Fail 95/90, HARD | Fail 90/85, HARD | Fail 99/95, HARD | Fail Eq. (2), HARD | ES | PS | NS |
|---|---|---|---|---|---|---|---|---|
| ROME | 55 | 85.7 (1,285) | 74.5 (1,118) | 90.4 (1,356) | 85.1 (1,276) | 100.0 | 98.5 | 77.0 |
| MEMIT | 44 | 80.0 (1,200) | 68.6 (1,029) | 85.1 (1,277) | 79.9 (1,198) | 98.0 | 87.0 | 75.4 |
| MELO | n/a | n/a | n/a | n/a | n/a | 94.0 | 83.5 | 82.9 |
| ELDER | 52 | 71.1 (1,067) | 60.6 (909) | 79.8 (1,197) | 70.5 (1,057) | 90.0 | 95.0 | 82.7 |
| Fisher-LoRA (Ours) | 23 | 34.4 (516) | 29.1 (436) | 47.8 (717) | 33.9 (508) | 98.0 | 97.0 | 90.9 |
Following your suggestions, we have added these results to Appendix A.24 as Tables 16 and 17 (Lines 1890–1906), stated the two thresholds and the treatment of rejected edits in Appendix A.16 (Lines 1463–1475), added the criterion-failure rate of Fisher-LoRA to Appendix A.17 (Lines 1616–1646), and revised the caption of Table 2 to define its 50% Failure column as a separate statistic (Lines 343–348) together with the hard-slice paragraph of Section 5.2 (Lines 342–359).
Summarized W5 & Q5: More edits, multiple seeds and confidence intervals
A5: Following your suggestion, we further repeat the comparison over three seeds (i.e., 42, 43 and 44) with a 95% t-interval for each editor and for the paired gain of Fisher-LoRA over ELDER. Table R3 shows that the gain over ELDER is positive on every seed and backbone (Table R4 gives each seed), with a Reliability gain of 3.0 [1.9, 4.1] on LLaMA-2-7B, 2.8 [1.4, 4.2] on Mistral-7B and 6.8 [3.5, 10.0] on GPT-2-XL. A seed changes none of the entries of ROME, MEMIT and AlphaEdit, which write closed-form updates, while it changes the random initialisation of MELO, ELDER and RoseLoRA and, for every editor trained on the loss of Eq. (10), Fisher-LoRA included, the unrelated prompts drawn at each step. Rejected edits of Fisher-LoRA score zero. Beyond the 100 test edits, the stratified analyses use the 5,000-edit pool and the 1,500-edit COUNTERFACT-HARD slice, with exact Clopper-Pearson intervals for the per-decile failure rates in Table 11. These analyses, zsRE, AdaLoRA and GaLore stay at seed 42, and a larger set of test edits is not supplied in this revision.
(Continued)
Table R3: Reliability and Locality (%) on the 100 COUNTERFACT test edits over three seeds, with 95% t-intervals.
| Editor | Backbone | Rel. mean [95% CI] | Loc. mean [95% CI] |
|---|---|---|---|
| ROME | LLaMA-2-7B | 74.0 [74.0, 74.0] | 82.1 [82.1, 82.1] |
| ROME | Mistral-7B | 71.5 [71.5, 71.5] | 77.9 [77.9, 77.9] |
| ROME | GPT-2-XL | 69.8 [69.8, 69.8] | 77.1 [77.1, 77.1] |
| MEMIT | LLaMA-2-7B | 80.6 [80.6, 80.6] | 84.7 [84.7, 84.7] |
| MEMIT | Mistral-7B | 78.2 [78.2, 78.2] | 87.9 [87.9, 87.9] |
| MEMIT | GPT-2-XL | 76.3 [76.3, 76.3] | 74.3 [74.3, 74.3] |
| MELO | LLaMA-2-7B | 86.2 [85.7, 86.8] | 79.2 [76.2, 82.1] |
| MELO | Mistral-7B | 84.1 [80.3, 87.8] | 78.5 [69.9, 87.0] |
| MELO | GPT-2-XL | 84.6 [83.1, 86.1] | 76.4 [71.5, 81.2] |
| ELDER | LLaMA-2-7B | 93.1 [92.8, 93.5] | 77.3 [72.6, 82.0] |
| ELDER | Mistral-7B | 92.3 [91.1, 93.6] | 78.4 [75.7, 81.2] |
| ELDER | GPT-2-XL | 89.1 [85.6, 92.6] | 74.9 [60.0, 89.8] |
| AlphaEdit [1] | LLaMA-2-7B | 84.2 [84.2, 84.2] | 87.3 [87.3, 87.3] |
| AlphaEdit [1] | Mistral-7B | 78.1 [78.1, 78.1] | 90.3 [90.3, 90.3] |
| AlphaEdit [1] | GPT-2-XL | 77.6 [77.6, 77.6] | 78.9 [78.9, 78.9] |
| RoseLoRA [2] | LLaMA-2-7B | 86.4 [84.3, 88.6] | 78.4 [66.8, 90.1] |
| RoseLoRA [2] | Mistral-7B | 84.6 [74.3, 94.9] | 75.7 [65.8, 85.7] |
| RoseLoRA [2] | GPT-2-XL | 84.2 [76.7, 91.7] | 73.3 [62.6, 84.0] |
| Fisher-LoRA (Ours) | LLaMA-2-7B | 96.1 [95.3, 96.9] | 91.0 [90.3, 91.6] |
| Fisher-LoRA (Ours) | Mistral-7B | 95.1 [94.8, 95.5] | 91.3 [91.1, 91.5] |
| Fisher-LoRA (Ours) | GPT-2-XL | 95.9 [95.6, 96.2] | 88.9 [86.9, 91.0] |
| Fisher-LoRA minus ELDER | LLaMA-2-7B | 3.0 [1.9, 4.1] | 13.7 [9.2, 18.2] |
| Fisher-LoRA minus ELDER | Mistral-7B | 2.8 [1.4, 4.2] | 12.9 [10.3, 15.5] |
| Fisher-LoRA minus ELDER | GPT-2-XL | 6.8 [3.5, 10.0] | 14.0 [0.4, 27.6] |
Table R4: Reliability and Locality (%) of each training seed on the 100 COUNTERFACT test edits.
| Editor | Backbone | Rel. 42 | Rel. 43 | Rel. 44 | Loc. 42 | Loc. 43 | Loc. 44 |
|---|---|---|---|---|---|---|---|
| ROME | LLaMA-2-7B | 74.0 | 74.0 | 74.0 | 82.1 | 82.1 | 82.1 |
| ROME | Mistral-7B | 71.5 | 71.5 | 71.5 | 77.9 | 77.9 | 77.9 |
| ROME | GPT-2-XL | 69.8 | 69.8 | 69.8 | 77.1 | 77.1 | 77.1 |
| MEMIT | LLaMA-2-7B | 80.6 | 80.6 | 80.6 | 84.7 | 84.7 | 84.7 |
| MEMIT | Mistral-7B | 78.2 | 78.2 | 78.2 | 87.9 | 87.9 | 87.9 |
| MEMIT | GPT-2-XL | 76.3 | 76.3 | 76.3 | 74.3 | 74.3 | 74.3 |
| MELO | LLaMA-2-7B | 86.4 | 86.0 | 86.3 | 80.5 | 78.2 | 78.8 |
| MELO | Mistral-7B | 84.0 | 82.6 | 85.6 | 77.8 | 82.2 | 75.4 |
| MELO | GPT-2-XL | 84.9 | 83.9 | 85.0 | 75.1 | 75.4 | 78.6 |
| ELDER | LLaMA-2-7B | 93.1 | 93.3 | 93.0 | 79.2 | 75.4 | 77.3 |
| ELDER | Mistral-7B | 92.0 | 92.1 | 92.9 | 78.0 | 79.7 | 77.6 |
| ELDER | GPT-2-XL | 90.4 | 87.6 | 89.3 | 68.2 | 79.7 | 76.9 |
| AlphaEdit [1] | LLaMA-2-7B | 84.2 | 84.2 | 84.2 | 87.3 | 87.3 | 87.3 |
| AlphaEdit [1] | Mistral-7B | 78.1 | 78.1 | 78.1 | 90.3 | 90.3 | 90.3 |
| AlphaEdit [1] | GPT-2-XL | 77.6 | 77.6 | 77.6 | 78.9 | 78.9 | 78.9 |
| RoseLoRA [2] | LLaMA-2-7B | 86.2 | 85.7 | 87.4 | 76.9 | 74.7 | 83.7 |
| RoseLoRA [2] | Mistral-7B | 80.6 | 84.3 | 88.9 | 80.1 | 72.2 | 74.9 |
| RoseLoRA [2] | GPT-2-XL | 87.2 | 84.2 | 81.2 | 70.2 | 71.5 | 78.2 |
| Fisher-LoRA (Ours) | LLaMA-2-7B | 95.9 | 96.0 | 96.5 | 91.2 | 91.0 | 90.7 |
| Fisher-LoRA (Ours) | Mistral-7B | 95.3 | 95.0 | 95.1 | 91.2 | 91.4 | 91.3 |
| Fisher-LoRA (Ours) | GPT-2-XL | 96.0 | 95.8 | 95.8 | 88.5 | 89.9 | 88.4 |
Following your suggestion, we have added these results to Appendix A.24 as Tables 18 and 19 (Lines 1945–1990), and pointed to them from Section 5.1 (Lines 298–302) and Appendix A.14 (Lines 1381–1389).
Summarized W6 & Q6: Locality on Mistral-7B and GPT-2-XL
A6: As reported in Table R3 of our response to your W5, Fisher-LoRA has the highest mean Locality of the seven editors on Mistral-7B and GPT-2-XL, with 91.3% and 88.9% against 78.4% and 74.9% for ELDER. Locality on zsRE is reported on LLaMA-2-7B, and the revision does not add it for the other two backbones.
Following your suggestion, we have added these results to Appendix A.24 as Table 18 (Lines 1945–1967).
References
[1] Junfeng Fang et al. AlphaEdit: Null-space constrained knowledge editing for language models. ICLR, 2025.
[2] Haoyu Wang et al. RoseLoRA: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. EMNLP, 2024.
Best Wishes,
The Authors of Paper 53337
Dear Reviewer v7TT,
Thank you for your time in reviewing our work, and for recognizing its editing performance, computational efficiency, and predictive analysis.
We believe that our rebuttal has addressed your concerns, with the longer-budget search of the minimum rank, the comparison with AlphaEdit and RoseLoRA, the fixed-rank version of Fisher-LoRA, the threshold sensitivity, and the results over three seeds with Locality on all three backbones. As suggested, we will keep these experiments in the final version.
Thank you again for your constructive feedback.
Best Wishes,
The Authors of Paper 53337
Response to Reviewer v7TT
Dear Reviewer v7TT:
Thank you for recognizing the method effectiveness, the efficiency, and the thorough analysis. We have responded point by point to your concerns regarding the source of failures, more baselines, an ablation study, threshold sensitivity, more seeds, and evaluation on more models. In the marked copy appended to the revised PDF (pages 38–74), the revisions made in response to your comments are tagged with @Reviewer v7TT. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1 & Q1: Insufficient rank versus unsuccessful optimisation
A1: We respectfully clarify that the searched minimum rank is defined at a fixed optimisation budget (i.e., 25 steps and one initialisation), and a failure below it can come from the rank or from the budget, as you point out. Following your suggestion, we further search the minimum under a 16 times longer schedule with the best of three initialisations on the 100 LLaMA-2-7B COUNTERFACT test edits. Table R1 shows that the minimum equals its value at the deployed budget on 40% of the edits and lies within one of it on 67%, with a mean of 10.53 against 11.58, and the ordering of edits by difficulty is kept (Spearman 0.99). Where the minimum falls, part of it was set by optimisation, and where it stays, the result is consistent with a rank limit without proving one.
Table R1: Match between two per-edit rank readings.
Following your suggestion, we have added these results to Appendix A.24 as Table 16 (Lines 1890–1898) and to Appendix A.19 (Lines 1698–1706), stated the fixed budget of the search in Section 3 (Lines 146–147), Appendix A.14 (Lines 1381–1389), Appendix A.16 (Lines 1494–1498) and the Conclusion (Lines 484–485).
Summarized W2 & Q2: Comparison with AlphaEdit and RoseLoRA
A2: Following your suggestion, we further compare Fisher-LoRA with AlphaEdit [1] at its released configuration and RoseLoRA [2] at rank 8, the total rank MELO runs at, under the protocol of Table 1 on the same 100 test edits on the three backbones over three seeds. Table R3 (under A5) shows that Fisher-LoRA has the highest mean Reliability and Locality of the seven editors on every backbone, with a mean Reliability of at least 95.1%.
Following your suggestion, we have added these results to Appendix A.24 as Table 18 (Lines 1945–1967), and pointed to them from Section 5.2 (Lines 320–321).
Summarized W3 & Q3: Contribution of rank adaptation with a fixed-rank version
A3: Following your suggestion, we have added a fixed-rank version of Fisher-LoRA to Table 6, which keeps the component selection, the Fisher-eigenvector initialisation and the training of Fisher-LoRA and fixes the rank of every edit at 12, the average allocated rank over the admitted test edits rounded to an integer (1% more parameters per edit). It reaches 91.2% Reliability and 83.6% Locality against 95.9% and 91.2% for the full method, and both rows use the same screen, which rejects no test edit, with the comparison over the same 100 edits. At approximately the same average budget, the per-edit allocation adds 4.7 Reliability and 7.6 Locality points, and on the test edits the gain comes from the per-edit allocation, not from the average budget. On the 1,500 COUNTERFACT-HARD edits the same control reaches 76.5% Reliability against 91.2%, where the budgets are not matched, since rank 12 lies below the floor of every slice edit, and the slice shows what a fixed average rank costs on hard edits rather than a matched comparison.
Following your suggestion, we have added this row to Table 6 (Lines 394–404) with its reading in Section 5.3 (Lines 411–417), and described it in Appendix A.22 (Lines 1831–1849).