- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies a failure in knowledge editing: a direct factual edit may succeed without changing answers to questions that depend on it. A layer-wise intervention places the desired object’s representation at the subject position and increases support for the downstream answer. Motivated by this result, the authors propose IERE. It aligns subject representations with desired-object representations through parameter updates, processes dependent edits from downstream to upstream, and finishes with batched factual editing. The method uses only single-hop edit requests. On MQuAKE, IERE achieves the highest forward multi-hop accuracy among the tested single-hop-only editors in all six model–hop settings. The paper also reports method ablations, subject-level locality on RippleEdits, and broader capability results on CSQA and MMLU.
Critical strengths
- Strong multi-hop gains using only single-hop edit requests. On Llama-3-8B-Instruct, IERE reaches 39.76%, 34.37%, and 21.41% accuracy from two to four hops, substantially outperforming the tested single-hop editors under the same information constraint. This is important because IERE does not require edit-specific multi-hop questions, reasoning paths, next-hop relations, or target answers.
- The method is tied to a concrete internal representation hypothesis. The layer-wise intervention shows that placing the desired-object representation at the subject position can move the prediction toward direct next-hop recall. This gives IERE a specific representational target instead of treating multi-hop improvement as a purely behavioral objective. The evidence does not fully establish the mechanism in the final edited model, but the connection between the intervention and the editing rule makes the method easier to interpret and analyze.
Critical weaknesses
- The representation mechanism is not verified in the final edited model. Section 4 inserts the desired-object representation into the unedited model and improves the downstream prediction. IERE then approximates that intervention with a parameter update (eq. 6), followed by a factual edit to overlapping parameters (eq. 7). Table 2 shows the behavioral effects, but does not establish whether the final model develops the intended subject-to-object representation shift or whether that shift drives the multi-hop gains. This limits the mechanistic claim, though the performance results stand. Evidence of the shift in the final model and its contribution to the gain would support the proposed explanation.
- The cross-model difference is not well explained. IERE brings large multi-hop gains on Llama-3, while the improvements on GPT-J are much smaller in several settings. Since the same editing framework is applied to both models, this gap suggests that the effectiveness of representation alignment may depend on model-specific properties. The paper reports the difference but offers little analysis of why it occurs. A brief analysis of the intervention behavior or representation alignment across the two models would help clarify when IERE is likely to work well and would make the cross-model claim easier to interpret.
Questions for authors
- After both stages of IERE, does an unpatched multi-hop input produce the intended subject-to-object representation shift? Is that signature associated with the cases IERE solves beyond factual editing alone?
- What do the intervention results and the failure analysis suggest about the Llama and GPT-J’s gap? Is GPT-J failing to establish the alignment, losing it during factual editing, or showing a different limitation?
- The NoSched ablation processes all edits in one round. Which part of the current evidence identifies the downstream-to-upstream order itself as the source of the longer-hop gain, apart from using multiple rounds and refreshed representations?
Response to Reviewer fAsv
We thank the reviewer for the constructive feedback.
Q1. Representation Shift and Prediction Improvements
IERE induces the intended representation shift during normal inference, and its alignment updates contribute to multi-hop predictions where MEMIT fails but IERE succeeds.
Representation Shift. We measured subject–desired-object cosine similarity across editing stages without activation patching (Table 1). Similarity increases substantially after representation alignment but partially decreases after factual editing, demonstrating that the intended representation shift can arise through parameter editing.
Contribution to Multi-Hop Predictions. For cases where MEMIT fails but IERE succeeds, we removed the representation-alignment parameter updates while keeping the factual-editing update fixed. The resulting answer change rates (Table 1) show that many previously correct predictions become incorrect, indicating that these updates contribute to IERE's additional multi-hop successes.
Table 1. Stage-wise cosine similarity and answer change rate after removing representation-alignment updates.
Q2. Cross-Model Performance Gap
GPT-J-6B can establish and benefit from intermediate-entity representation alignment, but subsequent factual editing may weaken this alignment, contributing to its smaller multi-hop gains.
Intervention and Failure Analysis. The original intervention results demonstrate the benefit of intermediate-entity representations, while Appendix A.4 shows that GPT-J-6B has higher proportions of Partial propagation and Negative/ineffective update among failed cases, suggesting additional propagation limitations.
Investigating the Performance Gap. Activation patching after MEMIT yields comparable performance across both models and substantially improves GPT-J-6B over IERE (Table 2), indicating that GPT-J-6B can utilize intermediate-entity information.
Stage-wise cosine similarity (Table 1) shows that both models establish high representation similarity after alignment, but factual editing partially weakens it, particularly on GPT-J-6B at 4 hops. This suggests that incomplete preservation of representation alignment may contribute to the performance gap.
Table 2. Multi-hop generation accuracy with intermediate-entity activation patching after MEMIT.
Q3. Contribution of Dependency Order
Downstream-to-upstream scheduling enables upstream edits to use updated desired-object representations, contributing to longer-hop propagation.
Dependency-Aware Scheduling. As explained in Section 5.1, downstream edits update the representations required as alignment targets by upstream edits. Therefore, processing downstream edits first ensures that upstream alignment uses the updated desired-object representations.
Additional Experiment. Since NoSched does not isolate dependency direction, we introduced a reverse-order IERE variant while preserving multi-round alignment and representation refreshing (Table 3). Reversing the order reduces multi-hop accuracy in all six settings, confirming the contribution of downstream-to-upstream scheduling.
Table 3. Multi-hop generation accuracy under default and reversed dependency orders.