- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the degradation of batch knowledge editing when multiple relations of the same subject are edited simultaneously. It identifies weight internalization, rather than residual optimization, as the main failure source and provides a spectral analysis of the fitting–stability trade-off in soft- and hard-constrained updates. Based on this analysis, the paper proposes SCRO, which jointly optimizes residuals under feasibility and spectral-norm constraints. Experiments on four LLMs show improved performance on the proposed HetionetEdit benchmark while remaining competitive on CounterFact and ZsRE.
Critical strengths
-
The paper develops a unified theoretical framework connecting residual fitting errors and non-target perturbations. Section 3.3 reveals the fitting–stability trade-off of soft and hard constraint updates along weak singular directions through spectral analysis, and Section 4 derives the feasibility and stability conditions that motivate SCRO.
-
The paper strengthens its empirical evidence through multi-model evaluation and diagnostic analyses. Table 3 shows that SCRO outperforms the compared baselines on HetionetEdit across four 7B-scale models, while Section 3.2 and Figure 2 provide failure-mode analysis beyond final performance metrics by examining residual injection, weight internalization, fitting error, and non-key-state perturbations.
Critical weaknesses
1.Table 4 shows a substantial performance drop when the feasibility constraint is removed. Section 5.4 attributes this drop to residual components outside the key matrix’s row space being truncated during internalization, but does not report the fitting error for this ablation. Thus, the specific explanation for this performance drop lacks direct diagnostic evidence.
2.The computational cost of SCRO is not quantified. Joint residual optimization introduces additional spectral projection steps during optimization, yet the paper reports neither editing time nor peak memory usage relative to baselines. This leaves the performance–cost trade-off at the 10,000-edit scale unclear.
Questions for authors
1.Table 6 lists prompt-specific neighborhood answers, whereas Eq. (35) uses (o_i). Does the evaluation use each prompt’s own reference answer? What fraction of neighborhood facts answered correctly before editing remain correct afterward?
2.Beyond the dataset-level statistics in Figure 1 and Table 5, how are the 10,000-edit batches in Table 3 sampled, and what is the distribution of relations per subject within each batch?
3.How sensitive is SCRO to the stability bound , and how does varying it affect editing success and knowledge preservation?
5 comments
Response to Weaknesses 2
Computational cost and memory usage
We thank the reviewer for raising this important point regarding the computational cost and memory usage. To address this, we conducted experiments using Qwen2.5-7B as the base model. We measured the average time (in seconds) required to edit a single sample across different datasets and methods, as well as the peak memory usage (in GB) under a batch size of 10,000. The average performance (Avg.) across the three datasets is also reported.
| Method | Avg. | HetionetEdit | ZsRE | CounterFact |
|---|---|---|---|---|
| FT | 0.42 / 35.03 | 0.51 / 34.32 | 0.52 / 36.63 | 0.24 / 34.13 |
| MEMIT | 9.17 / 49.63 | 10.18 / 49.63 | 9.69 / 49.63 | 7.64 / 49.63 |
| EMMET | 8.59 / 48.14 | 9.66 / 48.14 | 8.80 / 48.14 | 7.30 / 48.14 |
| PMET | 7.42 / 44.74 | 9.06 / 44.74 | 7.01 / 44.74 | 6.19 / 44.74 |
| ALPHAEDIT | 10.72 / 35.99 | 11.48 / 35.99 | 10.96 / 35.99 | 9.71 / 35.99 |
| MEMIT-MERGE | 6.66 / 44.81 | 8.69 / 43.16 | 6.96 / 44.61 | 4.32 / 46.66 |
| EAMET | 9.59 / 50.01 | 10.47 / 50.01 | 10.18 / 50.01 | 8.12 / 50.01 |
| SCRO | 8.54 / 49.53 | 9.34 / 49.34 | 8.86 / 49.62 | 7.42 / 49.64 |
We provide the following analysis based on the table:
- Time Efficiency: Although SCRO introduces additional spectral projection steps, its average editing time (8.54s) is comparable to the strong baseline EAMET (9.59s).
- Memory Usage: The peak memory of SCRO (49.53 GB) is well-controlled and similar to MEMIT (49.63 GB) and EAMET (50.01 GB), which is acceptable at the 10,000-edit scale.
- Performance-Cost Trade-off: FT is fast and memory-light, but its editing performance is poor. MEMIT-MERGE is the cheapest strong baseline, yet under SCMR (HetionetEdit, Table 3), its average performance over four model backbones falls 14.03% behind SCRO. SCRO therefore offers a better performance-cost trade-off than the baselines.
We have added these detailed computational cost and memory usage analyses to Appendix D.7 in the revised manuscript.
Response to Questions 1
1. Notation Clarification
We thank the reviewer for the careful reading. To clarify, the evaluation does use each prompt's own reference answer. Specifically, for a neighborhood prompt , the reference answer is the entry of associated with (e.g., C6 and OMP in Table 6), rather than a single answer shared across all neighborhood prompts. In the original Spe. this quantity was denoted by , which is also used to denote the original answer of the edited sample and thus creates ambiguity. We have therefore renamed the neighborhood reference answer as in the revised manuscript.
2. Statistics on Neighborhood Fact Retention
Taking Qwen2.5-7B on HetionetEdit (batch size 10,000) as an example, we further report the transition rates of neighborhood facts before and after editing. In addition to the proportion of facts that are correct both before and after editing (Correct Correct), we report the other three related metrics. The results are summarized below:
| Method | Correct Correct | Correct Wrong | Wrong Correct | Wrong Wrong |
|---|---|---|---|---|
| FT | 52.22 | 6.21 | 18.63 | 22.95 |
| MEMIT | 38.92 | 19.51 | 10.96 | 30.61 |
| EMMET | 31.88 | 26.55 | 17.38 | 24.20 |
| PMET | 43.52 | 14.92 | 8.99 | 32.58 |
| ALPHAEDIT | 35.36 | 23.07 | 14.17 | 27.41 |
| MEMIT-MERGE | 38.82 | 19.61 | 6.01 | 35.56 |
| EAMET | 41.38 | 17.05 | 6.56 | 35.01 |
| SCRO | 42.49 | 15.94 | 5.21 | 36.36 |
As shown, SCRO achieves 42.49% on Correct Correct while keeping Correct Wrong at a low 15.94%, indicating that SCRO effectively performs knowledge editing while well preserving the neighborhood facts that were originally correct. Compared with the strong baselines MEMIT-MERGE and EAMET, SCRO leads by 3.67 and 1.11 points on Correct Correct, respectively, with an identical reduction on Correct Wrong. Although FT and PMET attain a higher Correct Correct (52.22%/43.52%) and a lower Correct Wrong (6.21%/14.92%) than SCRO, their overall editing performance is poor, as shown in the Table 3. Overall, compared with all baselines, SCRO achieves a better trade-off between knowledge updating and neighborhood preservation.
The above content and analysis have been added to Appendix D.8 of the revised manuscript.
Response to Weaknesses 1
Extended Ablation: Fitting Error and Spillover Analysis
We thank the reviewer for this valuable comment. To illustrate the mechanism of each constraint more directly, we extend Table 4 with the fitting error (ϵfit) and spillover (ϵsp) of each variant:
Analysis of the results:
The above content has been added to the revised manuscript accordingly.