acceptodds
Under review as a conference paper at ICLR 2027

Shape the Residuals, Not the Update: Spectrally Constrained Batch Editing for Subject-Centric Knowledge

Abstract

Locate-then-edit methods dominate batch model editing, yet they are evaluated almost exclusively on subject-diverse benchmarks where edits concern unrelated entities. Real-world updates are often subject-centric: many facts about the same entity change together. On HetionetEdit, a benchmark we construct from a biomedical knowledge graph, representative methods lose roughly 30 Efficacy points. We trace the failure not to the optimized residuals, which remain near-perfect, but to their internalization into the weights. A spectral analysis shows why: soft- and hard-constraint updates are one operator with different filters, and on directions where keys are nearly indistinguishable the soft filter discards the residual while the hard filter amplifies it without bound. Same-subject edits create exactly such directions while demanding different residuals. We therefore constrain the residuals rather than the update: a single reparameterization keeps them in the subspace the keys can express and bounds their separation by that of the keys. The resulting method, Spectrally Constrained Residual Optimization (SCRO), improves the average editing score on HetionetEdit from to across four LLMs, remains highly competitive on CounterFact and ZsRE, and preserves general language understanding. Code is available at https://anonymous.4open.science/status/SCRO-ED97.

Primary area:
interpretability and explainable AI
Keywords:
model editing, batch editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LXztOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the degradation of batch knowledge editing when multiple relations of the same subject are edited simultaneously. It identifies weight internalization, rather than residual optimization, as the main failure source and provides a spectral analysis of the fitting–stability trade-off in soft- and hard-constrained updates. Based on this analysis, the paper proposes SCRO, which jointly optimizes residuals under feasibility and spectral-norm constraints. Experiments on four LLMs show improved performance on the proposed HetionetEdit benchmark while remaining competitive on CounterFact and ZsRE.

Critical strengths
  1. The paper develops a unified theoretical framework connecting residual fitting errors and non-target perturbations. Section 3.3 reveals the fitting–stability trade-off of soft and hard constraint updates along weak singular directions through spectral analysis, and Section 4 derives the feasibility and stability conditions that motivate SCRO.

  2. The paper strengthens its empirical evidence through multi-model evaluation and diagnostic analyses. Table 3 shows that SCRO outperforms the compared baselines on HetionetEdit across four 7B-scale models, while Section 3.2 and Figure 2 provide failure-mode analysis beyond final performance metrics by examining residual injection, weight internalization, fitting error, and non-key-state perturbations.

Critical weaknesses

1.Table 4 shows a substantial performance drop when the feasibility constraint is removed. Section 5.4 attributes this drop to residual components outside the key matrix’s row space being truncated during internalization, but does not report the fitting error ϵfit\epsilon_{\mathrm{fit}} for this ablation. Thus, the specific explanation for this performance drop lacks direct diagnostic evidence.

2.The computational cost of SCRO is not quantified. Joint residual optimization introduces additional spectral projection steps during optimization, yet the paper reports neither editing time nor peak memory usage relative to baselines. This leaves the performance–cost trade-off at the 10,000-edit scale unclear.

Questions for authors

1.Table 6 lists prompt-specific neighborhood answers, whereas Eq. (35) uses (o_i). Does the evaluation use each prompt’s own reference answer? What fraction of neighborhood facts answered correctly before editing remain correct afterward?

2.Beyond the dataset-level statistics in Figure 1 and Table 5, how are the 10,000-edit batches in Table 3 sampled, and what is the distribution of relations per subject within each batch?

3.How sensitive is SCRO to the stability bound α\alpha, and how does varying it affect editing success and knowledge preservation?

5 comments
AuthorsOpenReview ↗

Response to Weaknesses 1

Extended Ablation: Fitting Error and Spillover Analysis

We thank the reviewer for this valuable comment. To illustrate the mechanism of each constraint more directly, we extend Table 4 with the fitting error (ϵfit\epsilon_{\mathrm{fit}}) and spillover (ϵsp\epsilon_{\mathrm{sp}}) of each variant:

VariantAvg.Eff.Gen.Spe.ϵfit\epsilon_{\mathrm{fit}}ϵsp\epsilon_{\mathrm{sp}}
Feasibility only17.420.750.7050.810.000.12
Stability only24.7111.978.4553.700.720.01
SCRO (both)50.8068.5036.2047.700.000.01

Analysis of the results:

  • Feasibility only: The fitting error is forced to ϵfit=0\epsilon_{\mathrm{fit}} = 0, but without the stability constraint the residual energy along small-singular-value directions is unbounded, driving the spillover error ϵsp\epsilon_{\mathrm{sp}} up to 0.12. This ultimately causes the weight update to blow up and the efficacy (Eff.) to collapse to 0.75%.
  • Stability only: Spillover is effectively controlled (ϵsp=0.01\epsilon_{\mathrm{sp}} = 0.01), but since rank deficiency is not addressed, the residuals contain non-expressible components, yielding a fitting error ϵfit\epsilon_{\mathrm{fit}} as high as 0.72 and limiting efficacy to 11.97%.
  • SCRO (both): Constraining both the residual subspace and its magnitude simultaneously attains ϵfit=0\epsilon_{\mathrm{fit}} = 0 together with low spillover (ϵsp=0.01\epsilon_{\mathrm{sp}} = 0.01). The two constraints act in concert, ultimately achieving an efficacy of 68.50%.

The above content has been added to the revised manuscript accordingly.

AuthorsOpenReview ↗

Response to Weaknesses 2

Computational cost and memory usage

We thank the reviewer for raising this important point regarding the computational cost and memory usage. To address this, we conducted experiments using Qwen2.5-7B as the base model. We measured the average time (in seconds) required to edit a single sample across different datasets and methods, as well as the peak memory usage (in GB) under a batch size of 10,000. The average performance (Avg.) across the three datasets is also reported.

MethodAvg.HetionetEditZsRECounterFact
FT0.42 / 35.030.51 / 34.320.52 / 36.630.24 / 34.13
MEMIT9.17 / 49.6310.18 / 49.639.69 / 49.637.64 / 49.63
EMMET8.59 / 48.149.66 / 48.148.80 / 48.147.30 / 48.14
PMET7.42 / 44.749.06 / 44.747.01 / 44.746.19 / 44.74
ALPHAEDIT10.72 / 35.9911.48 / 35.9910.96 / 35.999.71 / 35.99
MEMIT-MERGE6.66 / 44.818.69 / 43.166.96 / 44.614.32 / 46.66
EAMET9.59 / 50.0110.47 / 50.0110.18 / 50.018.12 / 50.01
SCRO8.54 / 49.539.34 / 49.348.86 / 49.627.42 / 49.64

We provide the following analysis based on the table:

  1. Time Efficiency: Although SCRO introduces additional spectral projection steps, its average editing time (8.54s) is comparable to the strong baseline EAMET (9.59s).
  2. Memory Usage: The peak memory of SCRO (49.53 GB) is well-controlled and similar to MEMIT (49.63 GB) and EAMET (50.01 GB), which is acceptable at the 10,000-edit scale.
  3. Performance-Cost Trade-off: FT is fast and memory-light, but its editing performance is poor. MEMIT-MERGE is the cheapest strong baseline, yet under SCMR (HetionetEdit, Table 3), its average performance over four model backbones falls 14.03% behind SCRO. SCRO therefore offers a better performance-cost trade-off than the baselines.

We have added these detailed computational cost and memory usage analyses to Appendix D.7 in the revised manuscript.

AuthorsOpenReview ↗

Response to Questions 1

1. Notation Clarification

We thank the reviewer for the careful reading. To clarify, the evaluation does use each prompt's own reference answer. Specifically, for a neighborhood prompt qq, the reference answer is the entry of NiN_i associated with qq (e.g., C6 and OMP in Table 6), rather than a single answer shared across all neighborhood prompts. In the original Spe. this quantity was denoted by oio_i, which is also used to denote the original answer of the edited sample and thus creates ambiguity. We have therefore renamed the neighborhood reference answer as ai,qa_{i,q} in the revised manuscript.

2. Statistics on Neighborhood Fact Retention

Taking Qwen2.5-7B on HetionetEdit (batch size 10,000) as an example, we further report the transition rates of neighborhood facts before and after editing. In addition to the proportion of facts that are correct both before and after editing (Correct →\to Correct), we report the other three related metrics. The results are summarized below:

MethodCorrect →\to CorrectCorrect →\to WrongWrong →\to CorrectWrong →\to Wrong
FT52.226.2118.6322.95
MEMIT38.9219.5110.9630.61
EMMET31.8826.5517.3824.20
PMET43.5214.928.9932.58
ALPHAEDIT35.3623.0714.1727.41
MEMIT-MERGE38.8219.616.0135.56
EAMET41.3817.056.5635.01
SCRO42.4915.945.2136.36

As shown, SCRO achieves 42.49% on Correct →\to Correct while keeping Correct →\to Wrong at a low 15.94%, indicating that SCRO effectively performs knowledge editing while well preserving the neighborhood facts that were originally correct. Compared with the strong baselines MEMIT-MERGE and EAMET, SCRO leads by 3.67 and 1.11 points on Correct →\to Correct, respectively, with an identical reduction on Correct →\to Wrong. Although FT and PMET attain a higher Correct →\to Correct (52.22%/43.52%) and a lower Correct →\to Wrong (6.21%/14.92%) than SCRO, their overall editing performance is poor, as shown in the Table 3. Overall, compared with all baselines, SCRO achieves a better trade-off between knowledge updating and neighborhood preservation.

The above content and analysis have been added to Appendix D.8 of the revised manuscript.

Reviewer_QfeBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies subject-centric multi-relation batch knowledge editing, where multiple facts about the same entity are updated together. It introduces the biomedical benchmark HetionetEdit and attributes existing methods’ failures primarily to internalizing optimized residuals into model weights. A unified spectral analysis shows that soft-constraint updates can underfit residuals along weak key directions, while hard-constraint updates can amplify perturbations to other hidden states. The proposed SCRO method jointly optimizes residuals under feasibility and stability constraints before applying a hard-constraint weight update. Across four language models with 10,000 edits, SCRO raises the average editing score on HetionetEdit from 36.5 to 50.5 while remaining competitive on CounterFact and ZsRE. Additional experiments examine general capability preservation and the contributions of both constraints.

Critical strengths
  1. This paper is well-written with clear structure.
  2. The topic is timing and interesting in the community of model editing.
Critical weaknesses
  1. The methodological contribution over prior work needs to be clarified. MEMIT-Merge already addresses key–value conflicts in same-subject editing, while EAMET constrains the relationship between keys and residuals. SCRO’s contribution therefore appears to lie primarily in its spectral analysis and the resulting constraints. Substituting the residual parameterization into the update formula gives Δ=ZUr⊤C0−1/2\Delta = ZU_r^\top C_0^{-1/2}, which can also be interpreted as a subspace-constrained weight update with a spectral-norm bound in whitened coordinates. A comparison with simpler approaches to stabilizing the update would help establish whether joint residual optimization is essential to the reported gains.

  2. The feasibility ablation requires further explanation. When the key matrix has full column rank, VrVr⊤=IV_rV_r^\top = I, so the condition R=RVrVr⊤R = RV_rV_r^\top automatically holds for any residual matrix. This makes the actual rank important for interpreting the substantial gap between the stability-only variant and the full method in Table 4. The paper does not report this information or clarify whether numerical truncation makes the feasibility constraint active. The implementation of the stability-only variant also remains unclear. Since small singular values do not necessarily imply rank deficiency, the current ablation does not provide enough evidence to conclude that both constraints are individually necessary.

  3. The proposed spectral mechanism needs more direct empirical validation. The analysis attributes editing failures to large residual components along directions with small singular values, but the experiments do not directly measure this mismatch as subject repetition increases. Showing how SCRO reduces it would strengthen the connection between the theory and the results. Figure 2 also compares residual optimization and weight internalization only for MEMIT and EMMET. Including SCRO would reveal whether it actually closes this gap and whether the constraints reduce success under direct residual injection. At present, the final performance gains alone do not establish that the proposed mechanism explains the improvement.

  4. The experiments do not sufficiently isolate the effect of subject concentration. Moving from CounterFact or ZsRE to HetionetEdit also changes the domain and difficulty of the editing task, so the performance gap cannot be attributed solely to subject structure. The B1B_1–B4B_4 experiments partially address this concern, but they still draw from different subject populations. It is unclear whether those populations are comparable in editing difficulty. A stronger control would keep the fact pool fixed and change how facts are grouped into batches. This would vary within-batch subject repetition while reducing the influence of differences in the underlying data.

Questions for authors
  1. What additional benefit does joint residual optimization provide over a simpler regularized weight update under matched optimization settings?

  2. How is the stability-only variant implemented in Table 4, and what makes the feasibility constraint nontrivial given that VrVr⊤=IV_rV_r^\top = I when the key matrix has full column rank?

  3. Can the authors use a fixed fact pool with different batching schemes to verify whether increasing subject repetition produces the predicted spectral mismatch, and whether SCRO reduces the resulting gap between residual optimization and weight internalization?

8 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 (1/2)

1. Differences from Prior Work

We thank the reviewer for highlighting the relationship to prior work. We first clarify the core contribution of our method. We provide a unified explanation from a novel perspective (A Spectral View) regarding why both the soft-constraint and hard-constraint formulations in the locate-then-edit paradigm fail under the SCMR scenario. Based on this analysis, we propose a corresponding method to resolve the identified issues. Since both MEMIT-MERGE (optimizes a shared value for same-subject edits) and EAMET(encourages alignment between pairwise key similarities and residual similarities) retain MEMIT's soft-constraint formulation, we conduct a more detailed analysis of them from our proposed spectral perspective.

  • Soft-constraint formulation (MEMIT-MERGE and EAMET). Since they adopt the soft-constraint formulation, their fitting and stability properties are respectively given as follows (based on Eqs. 9 and 10):

    • Fitting: ΔSK−R=−R(I−VrVr⊤)−∑i=1r11+σi2(Rvi)vi⊤.\Delta_S K-R=-R(I-V_rV_r^\top)-\sum_{i=1}^{r}\frac{1}{1+\sigma_i^2}(Rv_i)v_i^\top. The first term, −R(I−VrVr⊤)-R(I-V_rV_r^\top), is the truncation term. It vanishes when VrVr⊤=IV_rV_r^\top=I or R=RVrVr⊤R=RV_rV_r^\top, where VrVr⊤=IV_rV_r^\top=I holds if and only if the whitened key matrix K~=UrΣrVr⊤\tilde{K}=U_r\Sigma_rV_r^\top has full column rank. However, as the number of same-subject samples edited in a batch increases, K~\tilde{K} in practice typically does not have full column rank (in Response to Weaknesses 2, we further present the rank of K~\tilde{K} across different models and datasets to illustrate this point). Meanwhile, MEMIT-MERGE and EAMET do not explicitly constrain RR to satisfy R=RVrVr⊤R=RV_rV_r^\top, so this term is generally nonzero. The second term, −∑i=1r11+σi2(Rvi)vi⊤-\sum_{i=1}^{r}\frac{1}{1+\sigma_i^2}(Rv_i)v_i^\top, is the shrinkage term, which always persists and cannot be eliminated under the soft-constraint formulation. Therefore, MEMIT-MERGE and EAMET essentially reduce the fitting error through heuristic approaches, but cannot make it zero.

    • Stability: ∥ΔSHnk∥2≤∥RVrΛS∥2∥Ur⊤C0−1/2Hnk∥2,∥RVrΛS∥2≤12∥RVr∥2.\lVert \Delta_S H_{\mathrm{nk}}\rVert_2 \le \lVert R V_r\Lambda_S\rVert_2 \lVert U_r^\top C_0^{-1/2}H_{\mathrm{nk}}\rVert_2, \qquad \lVert R V_r\Lambda_S\rVert_2 \le \frac{1}{2}\lVert R V_r\rVert_2. For a fixed RR, the soft filter’s amplification factor remains bounded as σi→0\sigma_i\to0, whereas the hard filter can amplify weak directions. MEMIT-Merge and EAMET inherit this property from MEMIT, but neither explicitly constrains the amplification factor.

  • Hard-constraint formulation (Our method SCRO). SCRO builds upon the hard-constraint formulation, and its fitting and stability properties are respectively given as follows (based on Eqs. 9 and 10):

    • Fitting: ΔSK−R=−R(I−VrVr⊤).\Delta_S K-R=-R(I-V_rV_r^\top). Only the truncation term remains. As described in Section 4 (Method), SCRO enforces the feasibility condition R=RVrVr⊤R=RV_rV_r^\top during residual optimization. Consequently, the fitting error is zero.

    • Stability: ∥ΔHHnk∥2≤∥RVrΣr−1∥2∥Ur⊤C0−1/2Hnk∥2,∥RVrΣr−1∥2≥max⁡i∥Rvi∥2σi.\lVert \Delta_H H_{\mathrm{nk}}\rVert_2 \le \lVert R V_r\Sigma_r^{-1}\rVert_2 \lVert U_r^\top C_0^{-1/2}H_{\mathrm{nk}}\rVert_2, \qquad \lVert R V_r\Sigma_r^{-1}\rVert_2 \ge \max_i\frac{\lVert Rv_i\rVert_2}{\sigma_i}. As described in Section 4 (Method), since the hard filter may become unstable when a weak direction σi\sigma_i retains substantial residual energy ∥Rvi∥2\lVert Rv_i\rVert_2, we introduce a stability condition: the RR obtained through constrained optimization must satisfy: ∥RVrΣr−1∥2≤α.\lVert R V_r\Sigma_r^{-1}\rVert_2 \le \alpha.

  • Conclusion: MEMIT-MERGE and EAMET are built upon the soft-constraint formulation; they mitigate the fitting error through heuristic approaches but cannot reduce it to zero. In contrast, our method, SCRO, is built upon the hard-constraint formulation and optimizes RR under two constraints, thereby reducing the fitting error to zero while ensuring a certain level of stability. The above content has been added to Appendix H of the revised manuscript.

AuthorsOpenReview ↗

Response to Weaknesses 1 (2/2)

2. Supplementary Experiments for Weight Internalization Error Analysis

Furthermore, to further validate the explanations in 1. Differences from Prior Work, we conducted supplementary experiments using Qwen2.5-7B on HetionetEdit with a batch size of 10,000. This experiment compares the fitting error ϵfit\epsilon_{\mathrm{fit}} and spillover ϵsp\epsilon_{\mathrm{sp}} of SCRO against MEMIT, EMMET, MEMIT-MERGE, and EAMET. The results are summarized in the table below:

Methodϵfit\epsilon_{\mathrm{fit}}ϵsp\epsilon_{\mathrm{sp}}
MEMIT0.590.04
EMMET0.180.16
MEMIT-MERGE0.050.01
EAMET0.280.01
SCRO0.000.01

The results demonstrate that SCRO simultaneously achieves zero fitting error and low spillover, whereas representative methods exhibit a clear trade-off. MEMIT maintains low spillover (ϵsp=0.04\epsilon_{\mathrm{sp}} = 0.04) but suffers from severe fitting error (ϵfit=0.59\epsilon_{\mathrm{fit}} = 0.59); conversely, EMMET reduces the fitting error to 0.180.18 (limited by the truncation term in Eq. 9) at the cost of higher spillover (0.160.16), which aligns with the analysis in Section 3.3. The two improved methods well-suited to the SCMR scenario, MEMIT-MERGE and EAMET, are soft-constraint methods built upon MEMIT. They further reduce fitting errors to 0.050.05 and 0.280.28, respectively, while maintaining low spillover; however, neither fully eliminates the bottleneck (as analyzed in 1. Differences from Prior Work). By jointly optimizing RR to satisfy Eq. 12, SCRO eliminates the fitting error (ϵfit=0.00\epsilon_{\mathrm{fit}} = 0.00) while maintaining low spillover (ϵsp=0.01\epsilon_{\mathrm{sp}} = 0.01).

Furthermore, this section provides a detailed decomposition of the fitting error. We present the parallel fitting error and orthogonal fitting error for each individual edit sample in a scatter plot. We find that SCRO eliminates both parallel under-writing and orthogonal directional distortion, whereas other methods exhibit varying degrees of under-writing and orthogonal directional deviation.

Even a small aggregate fitting error can undermine editing efficacy by weakening or distorting the intervention intended to elicit the target fact. Figure 5b shows that MEMIT-MERGE’s fitting error of 0.05 reflects systematic under-writing and directional deviation across individual edits. SCRO preserves the magnitude and direction of its learned residuals at the edit keys, achieving a reported fitting error of 0.00 at the same spillover of 0.01. In this setting, Efficacy rises from 53.79% to 68.50% on Qwen2.5-7B, a gain of 14.71 percentage points (Table 3).

Details of the above experimental results can be found in the newly added Section 5.4, Weight Internalization Error Analysis, in the revised PDF.

3. Necessity of Joint Residual Optimization

To investigate whether joint residual optimization is essential, we conducted this specific comparison on Qwen2.5-7B / HetionetEdit (10,000 edits). Specifically, we first obtain independent residuals R′=[r1,…,r10000]\mathbf{R}' = [r_1, \dots, r_{10000}] using the standard MEMIT RO approach. We then compute the unconstrained parameter Z=R′VrΣr−1\mathbf{Z} = \mathbf{R}'\mathbf{V}_r\Sigma_r^{-1} and apply a spectral-norm clipping operator to obtain Z^\widehat{\mathbf{Z}} such that ∣Z^∣2≤α\\|\widehat{\mathbf{Z}}\\|_2 \le \alpha. The regularized weight update is then given by Δ=Z^Ur⊤C0−1/2\Delta = \widehat{\mathbf{Z}}\mathbf{U}_r^\top \mathbf{C}_0^{-1/2}. The results are summarized below:

VariantEff.Gen.Spe.ϵfit\epsilon_{\mathrm{fit}}ϵsp\epsilon_{\mathrm{sp}}
SCRO w/o joint optimization0.00070.00040.59030.91090.0062
SCRO w/ joint optimization0.68500.36200.47700.00000.0063

Analysis: As shown, applying a simpler regularized weight update without joint residual optimization causes a catastrophic failure; despite maintaining low spillover (ϵsp=0.0062\epsilon_{\mathrm{sp}}=0.0062), Efficacy drops to 0.0007 and ϵfit\epsilon_{\mathrm{fit}} surges to 0.91. While Specificity is high (0.5903), it is meaningless since the model fails to successfully perform the edit. Independent RO does not penalize residual energy along weak directions (small σi\sigma_i), which is then amplified by 1/σi1/\sigma_i when computing Z=R′VrΣr−1\mathbf{Z} = \mathbf{R}'\mathbf{V}_r\Sigma_r^{-1}. Consequently, post-hoc clipping to enforce stability inevitably discards these components. Thus, clipping Δ\Delta compromises this learned semantic information, degrading the model's ability to generate correct targets and leading to near-zero Efficacy. By contrast, joint residual optimization consistently operates within the constraints during training, thereby ensuring that no learned semantic information is lost. This confirms that joint residual optimization is essential.

AuthorsOpenReview ↗

Response to Weaknesses 2

We thank the reviewer for the careful reading and valuable comments. The reviewer pointed out that the necessity of the Feasibility constraint is closely tied to the rank of the whitened key matrix K~\tilde{K}, while the Stability constraint relates to its small singular values, and that additional statistics on the rank and small singular values are required to justify the necessity of each ablation component. In response, we conducted the following additional statistics and analyses.

1. Rank and Small Singular Values of the Whitened Key Matrix K~\tilde{K}

We computed the rank of K~\tilde{K} and the fraction of small singular values (σi<1\sigma_i < 1) on Qwen2.5-7B (batch size = 10,000) across datasets. The results are summarized below:

DatasetRank of the Whitened Key Matrix (rr)Fraction of Small Singular Values (σi<1\sigma_i<1)
HetionetEdit5,79745.77%
ZsRE9,50633.84%
CounterFact9,7835.53%

The statistics show that:

  • Rank deficiency and the necessity of Feasibility: K~\tilde{K} is not of full column rank. Because HetionetEdit (the SCMR setting) contains many same-subject multi-relation samples, its rank deficiency is far more severe than that of ZsRE and CounterFact. Under rank deficiency, VrVr⊤=IV_r V_r^\top = I no longer holds automatically. If only the stability constraint is imposed without feasibility, the residual matrix will contain non-expressible components, resulting in a non-zero fitting error. This confirms the necessity of the Feasibility condition (R=RVrVr⊤R=RV_rV_r^\top), which ensures that the residuals lie in the expressible subspace.
  • Small singular values and the necessity of Stability: The fraction of small singular values in HetionetEdit is as high as 45.77%. Under the hard-constraint filter, directions with small singular values are severely amplified, causing spillover to blow up. This confirms the necessity of the Stability condition (constraining ∥Rvi∥2\|R v_i\|_2 to decay as σi\sigma_i decreases).

2. Implementation of Feasibility-only and Stability-only Variants

  • SCRO (both): Implements the two constraints (Feasibility and Stability) in Eq. 12 via the optimization in Eq. 14, i.e., R=ZΣrVr⊤R=Z\Sigma_rV_r^\top with ∣∥Z∣2≤α|\|Z\\|_2\le\alpha. The optimization variable is ZZ, from which RR is recovered.
  • Feasibility-only: Retains only the Feasibility constraint, R=ZΣrVr⊤R=Z\Sigma_rV_r^\top. The optimization variable is ZZ, from which RR is recovered.
  • Stability-only: Retains only the Stability constraint, ∣RVrΣr−1∣2≤α\\|RV_r\Sigma_r^{-1}\\|_2\le\alpha. Here the optimization variable is RR itself.

3. Extended Ablation: Fitting Error and Spillover Analysis

To illustrate the mechanism of each constraint more directly, we extend Table 4 with the fitting error (ϵfit\epsilon_{\mathrm{fit}}) and spillover (ϵsp\epsilon_{\mathrm{sp}}) of each variant:

VariantAvg.Eff.Gen.Spe.ϵfit\epsilon_{\mathrm{fit}}ϵsp\epsilon_{\mathrm{sp}}
Feasibility only17.420.750.7050.810.000.12
Stability only24.7111.978.4553.700.720.01
SCRO (both)50.8068.5036.2047.700.000.01

Analysis of the results:

  • Feasibility only: The fitting error is forced to ϵfit=0\epsilon_{\mathrm{fit}} = 0, but without the stability constraint the residual energy along small-singular-value directions is unbounded, driving the spillover error ϵsp\epsilon_{\mathrm{sp}} up to 0.12. This ultimately causes the weight update to blow up and the efficacy (Eff.) to collapse to 0.75%.
  • Stability only: Spillover is effectively controlled (ϵsp=0.01\epsilon_{\mathrm{sp}} = 0.01), but since rank deficiency is not addressed, the residuals contain non-expressible components, yielding a fitting error ϵfit\epsilon_{\mathrm{fit}} as high as 0.72 and limiting efficacy to 11.97%.
  • SCRO (both): Constraining both the residual subspace and its magnitude simultaneously attains ϵfit=0\epsilon_{\mathrm{fit}} = 0 together with low spillover (ϵsp=0.01\epsilon_{\mathrm{sp}} = 0.01). The two constraints act in concert, ultimately achieving an efficacy of 68.50%.

In summary, the additional statistics and analyses verify the independent necessity and complementary roles of the Feasibility and Stability constraints in controlling fitting and spillover. These statistics, implementation details, and extended ablations have been added to Section 5.5 and Appendix E of the revised manuscript.

Reviewer_yYu3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies batch knowledge editing when multiple requests concern different relations of the same subject. The authors construct HetionetEdit from a biomedical knowledge graph and find substantial degradation in representative locate-then-edit methods. By comparing direct residual injection with the edited model, they identify a large performance loss during weight internalization. Their analysis separates failure to reproduce the optimized residuals from perturbations to other token states. The paper expresses soft- and hard-constraint updates through a common spectral form. Weak key directions cause soft updates to discard residual components and can cause hard updates to amplify them. SCRO jointly optimizes residuals through a parameterization that guarantees their realizability by the keys and bounds the amplification factor. With 10,000 edits across four models, SCRO improves the average HetionetEdit editing score from 36.46 to 50.49. It remains competitive on CounterFact and ZsRE, and downstream evaluations on qwen2.5-7b suggest limited capability degradation.

Critical strengths
  1. The analysis explains two distinct failures within the same editing framework. Figure 2 separates residual injection from weight internalization and shows contrasting fitting and spillover behavior for MEMIT and EMMET. Equations (8)–(11) explain how these behaviors arise from different spectral filters. This gives a useful account of why exact residual fitting alone can be insufficient for successful editing.
  2. SCRO turns the analysis into a meaningful constraint on residual optimization. Equations (12)–(14) couple residual choices to the geometry of the entire key batch and bound their amplification under the hard update. This extends shared-value optimization beyond exact key collisions and provides guarantees beyond encouraging similarity between key and residual structures. Table 3 shows substantial HetionetEdit gains across four models, including comparisons with MEMIT-Merge and EAMET, which already address related interference problems.
Critical weaknesses
  1. The source of the key–residual mismatch is not fully explained. Section 3.3 argues that same-subject edits have close keys but distant residuals because they target different objects. However, equation 18 optimizes an intervention within a full prompt, so different target objects do not by themselves require distant intervention vectors. Section 4 also relies on residual non-uniqueness to find more compatible solutions, and MEMIT-Merge already jointly optimizes a shared value for different relations. Figure 2 shows that performance is lost during internalization, while appendix c explains the consequences of a given key–residual mismatch. Neither establishes why different target objects should produce the residual separation asserted in section 3.3. Without this step, the paper explains the consequences of the mismatch more convincingly than its origin. Explaining why independent residual optimization selects incompatible solutions, despite residual non-uniqueness, would connect the failure analysis more directly to SCRO’s design.
Questions for authors
  1. In section 3.3, what justifies inferring large residual separation from different target objects? Is this a property of the independently optimized solutions, or a requirement of every solution that successfully realizes the edits?
  2. When two keys are identical, SCRO forces their residuals to be identical. How does the resulting joint optimization relate mathematically to MEMIT-Merge’s shared-value objective, and what additional restriction becomes consequential when the keys are close but distinct?
4 comments
AuthorsOpenReview ↗

Reponse to Weaknesses 1/2

We thank the reviewer for raising this point. We agree that the original Section 3.3 did not state the connection between “different target objects” and “relatively large residual energy along weak directions” with sufficient precision. The revised manuscript has clarified this point as follows: their residuals are optimized independently without constraining the decomposition of RR in the key spectral coordinates, so ∣Rvi∣2\\|Rv_i\\|_2 need not shrink with σi\sigma_i and the optimizer can retain relatively large weak-direction residual components without penalty.

We have also added a scatter plot of (σi,∣Rvi∣2)(\sigma_i, \\|Rv_i\\|_2) to directly verify the phenomenon that the residual norm along weak directions is relatively large with respect to the singular values, as well as the mitigation of this mismatch by SCRO. In addition, we further explain the difference between the joint optimization of SCRO and MEMIT-MERGE, and why independent residual optimization selects incompatible solutions.

1. Detailed Elaboration of the Key–Residual Mismatch

In Section 3.3, we reveal that when small singular values σi\sigma_i exist and ∣Rvi∣2\\|\mathbf{R}\mathbf{v}_i\\|_2 does not shrink with σi\sigma_i, both the soft and hard updates exhibit problems along weak directions. The two are the same operator equipped with different filters: the soft filter ΛS=diag(σi/(1+σi2))\Lambda_S=\mathrm{diag}(\sigma_i/(1+\sigma_i^2)) discards Rvi\mathbf{R}\mathbf{v}_i along weak directions, leading to increased fitting error; the hard filter ΛH=diag(1/σi)\Lambda_H=\mathrm{diag}(1/\sigma_i) amplifies Rvi\mathbf{R}\mathbf{v}_i along weak directions, causing spillover and instability. Therefore, the reason why SCMR exacerbates internalization failure should be stated precisely in two points, rather than as “different target objects require large residuals.” Specifically, SCMR exacerbates this situation for two reasons:

  • (1) Multiple edit samples for the same subject read their keys at the same subject token, so k~i−k~j\tilde{\mathbf{k}}_i-\tilde{\mathbf{k}}_j is short and therefore easily lies in the span of weak directions; as the number of edit samples per subject increases, the number of small singular values in the whitened key matrix increases.
  • (2) The ri\mathbf{r}_i obtained through Eq. 18 is oriented toward producing the target answer after injection into the frozen model. The objective and locality regularizer of independent RO constrain the effect of the injected intervention within the full prompt, but do not constrain the decomposition of R\mathbf{R} in the key spectral coordinates. Consequently, ∣Rvi∣2\\|\mathbf{R}\mathbf{v}_i\\|_2 is not required to shrink with σi\sigma_i; within the non-unique solution set, the optimizer can retain relatively large residual components along weak directions without penalty.

We emphasize that this does not mean that semantic differences among target objects “necessarily” cause the residual vectors to be far apart from each other. Rather, along the weak directions induced by SCMR, independent RO has no incentive to suppress residual energy, and may therefore select a solution that is “effective for injection but incompatible with internalization”—one whose weak-direction residual component ∣Rvi∣2\\|\mathbf{R}\mathbf{v}_i\\|_2 is large relative to the singular value σi\sigma_i of that direction, i.e., ∥Rvi∥2\|\mathbf{R}\mathbf{v}_i\|_2 does not shrink accordingly with σi\sigma_i.

According to your suggestion, we have added Figure 3 (a scatter plot of (σi,∥Rvi∥2)(\sigma_i, \lVert Rv_i\rVert_2)) to the revised PDF in Section 3.3 for direct verification. As shown, as the number of repetitions per subject increases (from B1B_1 to B4B_4), the number of small singular values (σi<1\sigma_i < 1) grows accordingly (from 1 to 245). However, Figure 3 (top row) shows that the corresponding residual norms ∥Rvi∥2\lVert Rv_i\rVert_2 obtained by MEMIT/EMMET remain almost at a consistently high level, exhibiting a clear "small singular value, large residual components" mismatch. In contrast, Figure 3 (bottom row) presents the results for SCRO. Under SCRO, ∥Rvi∥2\lVert Rv_i\rVert_2 decays as σi\sigma_i decreases (enforced by the stability constraint). This confirms that SCRO is capable of mitigating this mismatch.

AuthorsOpenReview ↗

Response to Weaknesses 2/2

2. Difference between the Joint Optimization of SCRO and MEMIT-MERGE

We use a batch size 3 example to illustrate the difference between the two. Suppose we edit the following three samples simultaneously:

  • Sample 1: Prompt 1 “What university did Watts Humphrey attend?” Target 1: Illinois Institute of Technology
  • Sample 2: Prompt 2 “What role does Watts Humphrey play in football?” Target 2: Defender
  • Sample 3: Prompt 3 “What university did Denny Herzig attend?” Target 3: Harvard University

A single forward pass yields k1,k2,k3\mathbf{k}_1,\mathbf{k}_2,\mathbf{k}_3 at the subject token, respectively, where Samples 1 and 2 share the same subject.

MEMIT: The three samples are optimized independently to obtain r1,r2,r3\mathbf{r}_1,\mathbf{r}_2,\mathbf{r}_3, which are then concatenated into R=[r1,r2,r3]\mathbf{R}=[\mathbf{r}_1,\mathbf{r}_2,\mathbf{r}_3]. Each ri\mathbf{r}_i can steer the frozen model toward Target ii after injection, but the three residuals are not coordinated with one another.

MEMIT-MERGE: Samples 1 and 2 are jointly optimized into a shared residual r12r_{12} according to their shared subject, so that r12r_{12} contains information for both Target 1 and Target 2; Sample 3 is still optimized as r3r_3, yielding R=[r12,r3]R = [r_{12}, r_3]. This alleviates semantic conflict within the same-subject group, but r12r_{12} and r3r_3 remain free residuals and do not explicitly satisfy the feasibility and stability conditions in Eq. 12; when different subjects nevertheless have close keys, the same mismatch may still arise along weak directions.

SCRO: A shared spectral coefficient ZZ is jointly optimized over the entire batch, and the residual for each edit is read out individually as ri=ZΣrVr⊤eir_i=Z\Sigma_rV_r^\top e_i, rather than merging Samples 1 and 2 into a single r12r_{12}. All rir_i share the same set of key spectral bases and satisfy R=RVrVr⊤R=RV_rV_r^\top and ∣Z∥2≤α\\|Z\|_2\le\alpha by construction.

In summary, the two forms of joint optimization differ fundamentally: SCRO does not group by subject and merge semantics; it constrains the spectral geometry of the residuals across the entire batch. MEMIT-MERGE mitigates semantic conflict among same-subject samples, whereas SCRO directly eliminates the weak-direction residual-energy mismatch underlying both soft and hard internalization failures. Although SCRO adopts the hard closed-form solution because feasibility permits exact internalization, its constraints are not specific to the hard filter: they also suppress the weak-direction residual components that the soft filter discards and the hard filter amplifies. Moreover, SCRO requires no same-subject grouping and imposes spectral constraints on both same-subject samples and different-subject samples with close keys.

3. Why Independent Residual Optimization Selects Incompatible Solutions

The reason is that compatible solutions exist, but independent RO has no preference for selecting them. Eq. 18 optimizes each ri\mathbf{r}_i as a free variable on a per-sample basis; its objective only requires a high probability of the target token after injection and imposes a locality regularizer at the injection level. It contains neither Vr\mathbf{V}_r, Σr−1\Sigma_r^{-1}, nor any batch-level key spectral structure, and therefore neither rewards R\mathbf{R} for lying in the row space of K~\tilde{\mathbf{K}} nor penalizes ∣RVrΣr−1∣2\\|\mathbf{R}\mathbf{V}_r\Sigma_r^{-1}\\|_2. Within the non-unique solution set, the optimizer is determined by the per-sample objective, initialization, and gradient path, and thus tends to select a solution that is “effective for injection” rather than one that is “compatible with internalization.”

The design of SCRO moves this selection criterion forward into RO: feasibility is enforced through R=ZΣrVr⊤\mathbf{R}=\mathbf{Z}\Sigma_r\mathbf{V}_r^\top, stability is enforced through ∣Z∣2≤α\\|\mathbf{Z}\\|_2\le\alpha, and Z\mathbf{Z} is jointly optimized over the entire batch, so that a residual that can be internalized exactly and stably is selected from the non-unique solution set. Feasibility guarantees exact fitting under the hard update, while stability directly constrains the residual energy along weak directions, addressing the common source of both the soft and hard failure modes. In this way, the failure analysis and the design of SCRO are directly connected.

AuthorsOpenReview ↗

Response to Questions 1

Different target objects alone are insufficient to infer large residual separation. Even when two edits have nearly identical keys, their full prompts can induce different gradients with respect to the intervention. Independent optimization can therefore select different residuals even when a shared intervention would realize both edits. Please refer to 1. Detailed Elaboration of the Key–Residual Mismatch in the Response to Weakness for the relevant explanation.

AuthorsOpenReview ↗

Response to Questions 2

We thank the reviewer for this question. It should first be clarified that the benchmarks involved in the SCMR setting considered in this paper do not presuppose that two keys are exactly identical, nor do they rely on this rare special case. In SCMR, although multiple edit samples for the same subject share the subject, their keys are read at the subject token; owing to contextual processing in LLMs, when the prefixes preceding the subject token differ, the resulting keys are not exactly identical but highly similar. This highly similar but non-identical case is exactly the more common scenario on which this paper focuses, and the treatment of SCRO is also designed for this scenario.

In the extreme case where the prefix and the subject token are exactly identical, so that the keys are exactly identical, the feasibility constraint R=RVrVr⊤\mathbf{R}=\mathbf{R}\mathbf{V}_r\mathbf{V}_r^\top of SCRO naturally forces the residuals corresponding to identical keys to be identical; in this case, the residual accommodates the different target information to the maximum extent feasible. It should be noted that, in this special case, SCRO and the shared-residual objective of MEMIT-MERGE are similar in that residuals within the same group are identical, but their levels of constraint differ:

  • (1) MEMIT-MERGE groups by identical subject and binds a free shared residual within each group, with groups decoupled from one another.
  • (2) SCRO shares a spectral coefficient Z\mathbf{Z} across the entire batch, automatically binds the residuals of identical keys, and additionally imposes the global spectral constraint ∣Z∣2≤α\\|\mathbf{Z}\\|_2\le\alpha, so the groups remain coupled through the key spectral geometry.

To avoid exactly identical keys, a common practice in the model-editing literature can be adopted: read the key at the last token of the entire sentence, thereby avoiding exactly identical keys caused by identical prefixes and subject tokens.

For a more detailed explanation of the difference between the joint optimization of SCRO and MEMIT-MERGE, please refer to “2. Difference between the Joint Optimization of SCRO and MEMIT-MERGE” in the Response to Weakness.

Comments on the paper

3 comments
Area_Chair_X8xeOpenReview ↗

The details of the benchmark

Hi Authors,

Since you are using a self-constructed benchmark, please provide more details of the benchmark. At lease some example are needed.

Best,
AC

We sincerely thank the AC for this constructive suggestion. We fully agree that providing the details and examples is essential for a self-constructed benchmark. The detailed construction process, statistics, and examples of HetionetEdit are provided in Appendix B. For clarity, we summarize the key aspects and provide a representative example below:

  • Data Source & Construction: HetionetEdit is constructed from the Hetionet biomedical knowledge graph, following the construction paradigm of traditional model editing benchmarks such as ZsRE and CounterFact. We parse the graph into factual triples and aggregate objects sharing the same subject and fine-grained relation. Each editing sample is instantiated via predefined templates, comprising one main editing prompt and three semantically equivalent paraphrases, with the target answer derived from the aggregated object list. To evaluate the preservation of unrelated knowledge, we additionally construct two types of neighborhood prompts: same-subject/different-relation and same-relation/different-subject.
  • Dataset Statistics: The benchmark contains 56,774 editing instances across 17,555 unique subjects, averaging 3.2 relations per subject.
  • Example: A representative instance from HetionetEdit is as follows (Table 5): | Field | Example | | :--- | :--- | | sample_id | 5 | | Subject | heart | | Subject ID | Anatomy:UBERON:0000948 | | Relation Type | Anatomy - expresses - Gene | | Main Prompt | Which genes have measurable expression within the anatomical structure heart? | | Target Object | F9 | | Target Object ID | Gene:2158 | | Paraphrase Prompts | 1. What expressed genes are found in the anatomical structure heart? 2. Which genes are expressed in the anatomical structure heart? 3. Which genes can be detected in the anatomical structure heart? | | Same-subject Different-relation Neighborhood | Prompt: Which genes are linked to higher expression in the anatomical structure heart? Answer: C6 | | Same-relation Different-subject Neighborhood | Prompt: Which genes have measurable expression within the anatomical structure nose? Answer: OMP |

The Main Prompt and Target Object serve as the input for the editing process. Paraphrase Prompts are used to evaluate Generalization (Gen.), testing the model's ability to generalize to semantic variations of the main prompt. Neighborhood Prompts evaluate Specificity (Spe.), measuring the model's ability to retain unrelated knowledge when presented with related but distinct facts.

1. Core Contribution

This work introduces a novel spectral perspective to analyze the failure of batch model editing. Specifically, we provide a unified explanation for both soft- and hard-constraint paradigms in subject-centric multi-relation (SCMR) batch editing: a key-residual spectral mismatch along weak directions leads to inaccurate fitting under the soft constraint and uncontrolled spillover under the hard constraint. Based on this diagnosis, we propose a spectrally constrained residual optimization method (SCRO) alongside the HetionetEdit benchmark, achieving superior editing performance across multiple LLMs.

2. Response Summary

We have rigorously and directly responded to all weaknesses and questions raised by the reviewers, with the corresponding revisions incorporated into the updated PDF. This is validated by the positive feedback from Reviewer QfeB: "Thanks for the detailed responses, which address all of my concerns."

3. Revisions Summary

To facilitate the review process, the revised text in the updated PDF is highlighted in blue. The main additions are summarized below:

  1. Section 3 (Why Batch Editing Fails under SCMR): We incorporated our method, SCRO, into the comparative experiments of Figure 2 for direct visual comparison with the baseline methods. Furthermore, we added a new Figure 3 to further substantiate the failure mode under the SCMR scenario: the mismatch between small singular values and residual energy.
  2. Section 5 (Experiments): We added a new subsection, Section 5.4 (Weight Internalization Error Analysis), to provide an in-depth analysis of the fitting error and spillover across SCRO and the baselines. This section also includes scatter plots illustrating the parallel and orthogonal fitting errors for individual edit samples, further elucidating the reasons for our method's performance improvements. Additionally, in Section 5.5 (Ablation Study), we expanded Table 4 to further report the fitting error and spillover across different ablation variants.
  3. Appendix: We have added multiple new sections to further validate the effectiveness and robustness of our method:
    • D.5: Hyperparameter sensitivity analysis.
    • D.6: Sampling and distribution of edit samples.
    • D.7: Computational cost and peak memory usage.
    • D.8: Statistics on neighborhood fact retention.
    • E: Additional statistics and implementation details for the ablation studies.
    • F: Further controlled analysis with a fixed fact pool.
    • G: Discussion and comparison with strong baseline methods from our proposed spectral perspective.
  4. Abstract: Updated the anonymous repository URL from https://anonymous.4open.science/status/SCRO-ED97 to https://anonymous.4open.science/r/SCRO-ED97 (status →\to r).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.