acceptodds
Under review as a conference paper at ICLR 2027

RUR-Edit: Instance-Aware Residual Allocation and Utility-Aware Update Reconstruction for Knowledge Editing

Abstract

Knowledge editing aims to update specific factual knowledge in large language models while preserving unrelated knowledge. Existing multi-layer locate-then-edit methods typically distribute the remaining editing residual uniformly across candidate layers and directly apply each resulting parameter update as a whole. This coarse-grained editing overlooks heterogeneity at two levels: different layers vary in their ability to realize a target edit, while components within a layer-wise update differ in editing utility and interference with preserved knowledge. To address these limitations, we propose RUR-Edit, a fine-grained knowledge editing framework for both inter-layer and intra-update heterogeneity. At the inter-layer level, RUR-Edit introduces instance-aware residual allocation, which estimates a request-specific layer-importance profile and distributes the remaining target residual accordingly. At the intra-update level, it introduces utility-aware update reconstruction, which decomposes each layer-wise update into low-rank components and reconstructs it by balancing target-editing fidelity against interference with preserved knowledge. Experiments on CounterFact and ZsRE across LLaMA3, GPT-J, and Qwen2.5 show that RUR-Edit consistently achieves a stronger balance among editing efficacy, generalization, and specificity. On CounterFact, it achieves the highest overall editing score across all nine model–batch-size settings. On GPT-J, RUR-Edit improves the average overall score by 6.3 points over the strongest baseline while reducing editing time by 56.0%. Further analyses demonstrate the complementary effects of the two components, supporting fine-grained control over residual allocation and update composition for reliable and efficient knowledge editing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Editing, Large Language Models, Instance-Aware Residual Allocation, Update Reconstruction, Sequential Editing, Model Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WZNnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on two aspects of multi-layer locate-then-edit knowledge editing: residual allocation across editing layers and reconstruction of layer-wise parameter updates. Particularly, the authors propose FastGate, a predictor trained with an analytic teacher signal to estimate request-specific layer importance for residual allocation. They also introduce UAR, which decomposes parameter updates into low-rank components using SVD and optimizes their reconstruction coefficients through a regularized objective.

Critical strengths
  1. The motivation is natural. The paper focuses on the coarse-grained nature of traditional knowledge editing methods and investigates heterogeneity across editing layers and within parameter updates. The proposed FastGate predictor avoids the need to perform full editing independently at different layers to estimate their importance. By designing an analytic teacher score, the predictor can directly estimate the importance weights of multiple editing layers.

  2. The paper reports strong performance across multiple models, datasets, and editing batch sizes.

Critical weaknesses
  1. Insufficient justification of the proposed methods.
  • Several previous works have investigated residual distribution in knowledge editing, such as BLUE. Although BLUE does not directly focus on learning layer-importance weights, it addresses the broader problem of residual allocation in multi-layer editing. However, this paper does not provide a direct comparison with BLUE.
  • The design of the UAR loss requires further justification. The loss function is relatively complex and contains multiple terms. However, the paper does not sufficiently justify the necessity of each term. For example, an ablation that completely removes djd_j would be valuable, especially considering that the reported results are relatively insensitive to changes in βpre\beta_{\mathrm{pre}} and βnull\beta_{\mathrm{null}}.
  1. Some experimental results may require further clarification.
  • In Figure 4, RUR-Edit shows approximately a 50% reduction in runtime compared with AlphaEdit. However, if I understand correctly, RUR-Edit introduces two additional steps (FastGate weight prediction and UAR reconstruction) compared with AlphaEdit. The paper does not appear to introduce any specific optimization to accelerate the original AlphaEdit solver. Therefore, the source of this runtime improvement is unclear.
  • The specificity scores of AlphaEdit on CounterFact (1,000 edits) appear substantially lower than those reported in the original AlphaEdit paper (2,000 edits) under the same batch size. Did the authors change the specificity evaluation protocol, or are there other differences in the experimental settings that explain this discrepancy?
Questions for authors
  1. Could the authors compare RUR-Edit with related methods, such as BLUE?
  2. Please refer to Weakness 2.
  3. Could the authors show the distribution of the optimized coefficients cjc_j? I am curious whether they exhibit meaningful component-wise differences or just correspond to uniform shrinkage.
  4. How long does it take to train FastGate? Could the authors report the total wall-clock time?
Reviewer_i346OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two limitations of existing multi-layer locate-then-edit knowledge editing methods: (1) they typically distribute the remaining target residual uniformly across subsequent editing layers, overlooking differences in how individual layers contribute to an edit; and (2) they generally apply the entire parameter update at each layer, implicitly assuming that all components of the update are equally beneficial. To address these limitations, this paper extends AlphaEdit with Instance-Aware Residual Allocation and Utility-Aware Update Reconstruction. Authors demonstrate improved editing performance and provide empirical evidence that both forms of heterogeneity are important for effective model editing.

Critical strengths

The paper clearly identifies the limitations of existing locate-then-edit methods and presents its contributions in a well-organized structure.

The experimental design is well-structured to evaluate the individual contributions of the proposed components and provide empirical justification for the underlying design choices.

Critical weaknesses
  1. RUR-Edit appears to be an AlphaEdit-specific extension rather than a general framework for addressing inter-layer and intra-update heterogeneity. However, the paper presents the method as more broadly applicable than what is supported by its implementation and experiments.

  2. The preliminary section needs clearer definitions of (W_l), (k_l), (v_l), (Z^*), and (H_{l_i}) to avoid confusion, particularly for readers unfamiliar with locate-then-edit methods. For example, the authors should specify that (W_l) is the down-projection matrix and clarify the token positions at which (k_l) and (v_l) are computed .. etc

  3. The connection between Eq. (2) and Eqs. (3)–(4) is ambiguous.

  4. I think it would be better to discuss the training process of FastGate in greater detail in the main text, as it is an important component of Instance-Aware Residual Allocation that directly determines how editing residuals are distributed based on predicted layer importance. Equations (7) and (8) only describe how the predicted weights are applied to residual allocation, without sufficiently explaining how FastGate is trained or why this allocation strategy is justified.

  5. AlphaEdit already restricts parameter updates to a specific subspace through null-space projection. However, UAR further measures the deviation of each SVD component from the same subspace. It is unclear what additional benefit this constraint provides, given that the updates are already restricted to the null space.

  6. ZsRE and CounterFact are widely used benchmarks for model editing. Although the authors mention some differences in their evaluation protocol, they do not sufficiently explain why the reported performance of AlphaEdit and other baselines differs so substantially from the results in the original papers. These discrepancies are not limited to a single metric but are observed across multiple evaluation metrics. This raises concerns about whether the comparisons are fair and reliable.

  7. The proposed method appears to share similarities with CAKE: Causal-Guided Adaptive Knowledge Editing for LLMs (ACL 2026). It would be helpful to include an experimental comparison with CAKE and clarify how the proposed method differs from it, particularly in terms of its methodological contributions.

Questions for authors
  1. Could Utility-Aware Update Reconstruction unnecessarily modify an already optimal update, potentially degrading editing performance? Have the authors analyzed such potential adverse effects?

  2. Furthermore, according to Equations (18) and (24), if Equation (40) holds, the null-space interference should be zero. If so, why does varying “beta_null” lead to different editing performance in Figure 5?

Reviewer_cGm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RUR-Edit, a knowledge editing framework that updates specific facts in large language models while preserving other knowledge and model capabilities. The authors argue that existing multi-layer editing methods typically distribute editing residuals uniformly and apply parameter updates as a whole, without sufficiently accounting for differences in editing capability across layers or the benefits and interference of individual update components. Built on AlphaEdit, RUR-Edit introduces two components: Instance-Aware Residual Allocation (IRA) uses a lightweight predictor, FastGate, to estimate layer importance for each editing request, adaptively allocate residuals, and recompute the remaining residual after each layer update; Utility-Aware Update Reconstruction (UAR) decomposes parameter updates into rank-one components through truncated SVD and reweights them by jointly considering target residual fitting, knowledge preservation, and deviation from the original update. Experiments cover LLaMA3, GPT-J, and Qwen2.5 on CounterFact and ZsRE. The authors report a better balance among editing efficacy, generalization, and knowledge preservation, along with improved stability during long-term sequential editing.

Critical strengths
  1. This paper is well structured with solid experimental results.
  2. The idea of combining inter-layer and intra-updates into model editing mechanisms is interesting.
Critical weaknesses
  1. The motivation for using FastGate in IRA is not sufficiently clear. Although its architecture and training procedure are described, the paper does not adequately explain why this particular predictor is needed or how sensitive IRA is to its design. Could simpler predictors or alternative architectures achieve comparable results? The authors should also clarify whether FastGate is introduced in this work or adopted from prior research, providing an appropriate citation if applicable. Additional explanation and ablations would make the design easier to understand.
  2. The positioning of UAR should better acknowledge related work. EAMET [1] studies key–residual relationships across facts within a batch update, while REVIVE [2] explicitly decomposes parameter updates into spectral components and filters potentially harmful directions. These works are relevant to the paper’s motivation concerning intra-update structure, but neither is cited or discussed. The authors should explain how UAR differs from these approaches and clarify its specific contribution beyond existing work.
  3. Table 3 shows a relatively modest incremental improvement from adding UAR to IRA. Without results across multiple random seeds or uncertainty estimates, it is difficult to determine whether this improvement is statistically reliable. The authors should report the variability of the results and discuss UAR’s additional contribution more carefully, including its effects on individual metrics and its computational overhead.
  4. The null-space deviation penalty appears mathematically redundant under the stated formulation. Equation (18) defines an orthogonal projector P=U~U~⊤P=\widetilde U\widetilde U^\top, satisfying P⊤=PP^\top=P and P2=PP^2=P. The base update in Equation (24) satisfies Δ=ΔP\Delta=\Delta P. Specifically, letting A=(KpKp⊤+KK⊤)P+λIA=(K_pK_p^\top+KK^\top)P+\lambda I, we obtain A(I−P)=λ(I−P),A(I-P)=\lambda(I-P), and consequently
=DK^\top PA^{-1}(I-P) =\frac{1}{\lambda}DK^\top P(I-P) =0.$$ For any singular triplet with $\sigma_j>0$, the identity $\Delta^\top u_j=\sigma_jv_j$ therefore implies $$Pv_j =\frac{1}{\sigma_j}P\Delta^\top u_j =\frac{1}{\sigma_j}\Delta^\top u_j =v_j.$$ Thus, the proposed penalty in Equation (40) is identically zero for every nonzero update component: $$d_j^{\mathrm{null}}=\|v_j-Pv_j\|_2^2=0.$$ Truncating the SVD or reweighting its components cannot change this property, since the reconstructed update still satisfies $\Delta'=\Delta'P$. Selecting eigenvectors with small rather than strictly zero eigenvalues does not resolve this issue: it makes the selected subspace an approximate null space of the preservation covariance, but $P$ remains an exact orthogonal projector onto that subspace. Could the authors clarify what aspect of the implementation breaks these identities and makes this penalty nontrivial? [1] EAMET: Robust Massive Model Editing via Embedding Alignment Optimization. [2] Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse.
Questions for authors

see my comments above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.