acceptodds
Under review as a conference paper at ICLR 2027

StructKE: Mitigating Structural Degradation in Multi-Hop Knowledge Editing

Abstract

Knowledge editing aims to update specific facts in large language models efficiently. Although non-parametric in-context editing is robust for single-fact correction, it degrades severely on complex reasoning tasks such as multi-hop question answering. Through fine-grained empirical analysis, this paper identifies structural degradation as a central bottleneck. Under parametric priors, a model often deviates from the required factual-transition structure by taking shortcuts or stopping early, preventing injected knowledge from propagating along a valid reasoning skeleton. To address this problem, we propose StructKE, a structure-aware in-context editing framework. StructKE reformulates multi-hop editing as a controllable four-stage search process with structure prediction, structure-constrained generation, conflict-aware filling, and SDPR selection. It first trains StructEst, a lightweight estimator, to predict the target hop count, then filters candidate reasoning paths under the target-hop constraint, performs conflict-aware filling, and finally selects the answer through a Structure-Diagnostic Pairwise Ranker (SDPR) over structural and consistency signals. Experiments on multiple benchmarks show that StructKE mitigates structural degradation in multi-hop knowledge editing and achieves strong gains over the compared baselines by preserving structural integrity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, knowledge editing, multi-hop question answering, in-context editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a8p4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies structural degradation in multi-hop knowledge editing: even when a model has access to edited facts, it may skip steps, terminate early, or deviate from the required relation chain, preventing the updated knowledge from influencing the final answer along the intended path. The authors propose StructKE, a four-stage framework. First, a LoRA-tuned StructEst predicts the target hop count. Second, the framework generates candidate paths with entity masks and type constraints, then filters and recovers candidates according to their hop alignment. Third, it performs conflict-aware knowledge filling. Finally, a Structure-Diagnostic Pairwise Ranker (SDPR) selects an answer using structural signals, type consistency, generation certainty, and candidate statistics. The paper reports experiments across three language models and four dataset settings, together with hop-consistency analyses, component ablations, and efficiency comparisons.

Critical strengths
  • Multi-hop editing requires not only retrieving updated facts but also allowing them to affect subsequent dependencies. Using the target hop count as a predictable and verifiable control signal, while progressively relaxing candidate constraints when exact matches are insufficient, is a concrete and implementable design with potential value for improving reasoning reliability after editing.

  • The paper covers three models and four dataset settings, and reports performance across reasoning depths, numbers of edited relations, paraphrases, and latency conditions. In Table 5, removing StructEst, SDPR, or exact-hop candidates consistently reduces performance, providing preliminary evidence that the components contribute to the framework.

  • SDPR formulates candidate selection as pairwise preference learning within each question, combining hop alignment, candidate recovery states, entity-type consistency, and generation certainty. This design can learn interactions among different quality signals and provides a flexible mechanism for resolving conflicts between candidates, such as a structurally aligned but low-confidence path and a high-confidence path that skips steps. It is a useful extension of fixed-rule filtering.

Critical weaknesses
  • The technical relationship to DecKER and the independent evidence for the new contributions are insufficiently clear. DecKER already adopts a masked-path planning followed by knowledge injection strategy, and evaluates candidates using entity-type consistency and generation entropy. StructKE's masked planning, conflict-aware filling, and KI/RPP signals have direct counterparts in DecKER. Its main incremental contributions should therefore lie in StructEst, hop control and recovery, and SDPR, rather than primarily in the plan-then-fill framework itself. The paper cites and compares against DecKER and provides internal ablations, but these experiments do not sufficiently disentangle the effects of the new modules, candidate search budgets, and additional supervision. A direct comparison between DecKER and variants that progressively incorporate the new modules, under the same retrieval access and matched generation budgets, is needed to establish the incremental contributions. This concern is about contribution attribution and validation, not an allegation of research misconduct. (Sections 3.2–3.4 and Table 5; Sections 4.1–4.3 of DecKER.)

  • Unexplained differences in baseline results across key tables limit their comparability. For Qwen2.5-7B-Instruct on MQuAKE-CF-3k-v2, both Section 4.1.1 and Appendix J.1 define case-level success as answering at least one of the three paraphrases correctly. However, Table 3 and the Case Any column of Table 14 report substantially different results: 38.3% versus 11.27% for RAE, 26.2% versus 10.83% for EditCoT, and 12.3% versus 2.73% for DeepEdit. By contrast, the results for DecKER and StructKE agree within rounding precision. I do not conclude from these discrepancies that either set of results is incorrect; rather, I ask the authors to explain their sources. For example, do the tables use previously reported results versus local reproductions, or different retrieval access, prompts, model versions, or decoding settings? Appendix H describes an adjustment to RAE's knowledge access, but does not clarify how it relates to the two tables. Until the sources and settings are explained, it remains unclear whether the main results and structural diagnostics jointly support the same conclusion. (Tables 3 and 14, Section 4.1.1, and Appendices H and J.1.)

  • SDPR's feature differencing removes some of the context it is claimed to use, creating a mismatch between its mathematical definition and functional explanation. Equation (4) includes shared information, such as candidate-pool size and target hop count, whereas Equations (8) and (11) feed only the difference between two candidates' features from the same question into the ranker. If the shared information is uqu_q and the candidate-specific information is viv_i, then

    x(ci)=[uq,vi],x(cj)=[uq,vj],x(ci)−x(cj)=[0,vi−vj].x(c_i)=[u_q,v_i],\qquad x(c_j)=[u_q,v_j],\qquad x(c_i)-x(c_j)=[0,v_i-v_j].

    Only the shared dimensions cancel, not all candidate features; candidate-specific information such as hop-match status may remain. Under the current equations, however, pool size and target hop count themselves cannot directly participate in ranking. The authors should clarify whether additional contextual inputs or precomputed interaction features are used. The FiF_i-only result in Table 15 also requires an explanation based on the complete feature definition. Furthermore, differencing removes a common additive offset but does not automatically provide the scale invariance claimed in the paper. These issues affect the understanding of what information SDPR actually uses, rather than merely its notation. (Equations (4), (8), and (11), line 282, and Table 15.)

  • The structural diagnostics primarily measure length consistency and do not sufficiently support the mechanistic claim that correct factual dependencies are recovered. The paper acknowledges that hop consistency is only a proxy, which is an appropriate qualification. Nevertheless, paths with the same hop count can follow different relation chains, and KI checks entity types without distinguishing an outdated answer from an edited answer of the same type. In Table 4, 72.81% of StructKE's incorrect outputs still match the target hop count, highlighting the gap between length consistency and correct propagation of edited knowledge. Correlations conditioned on correctness also do not, by themselves, establish a causal explanation. The current evidence supports improved hop consistency and answer accuracy, but is insufficient to establish that edited knowledge propagates because the correct factual-transition structure is preserved. Additional verification of relation chains or intermediate facts, or interventions that replace edited facts under controlled candidate budgets, would help distinguish genuine dependency-chain recovery from gains due to length filtering, search, and ranking. (Sections 2.2 and 4.3–4.4, Tables 4 and 9, and Appendix B.)

Questions for authors

The following questions correspond to the four concerns above and could materially affect my assessment. The requested validation is intended to support the paper's current claims, rather than extend it to a new research task.

  1. Which modules are directly inherited or adapted from DecKER? Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions? Please clarify the use and impact of additional supervision.

  2. Which runs or publications are the sources of each method's results in Tables 3 and 14? Please explain the discrepancies and identify the configurations underlying the outputs used in Tables 1, 4, and 9 and Figure 3. If the settings differ, please provide sufficient configuration details to reconcile the two sets of results, without presupposing that either set is incorrect.

  3. Please list all fields in FiF_i and provide a real candidate-pair example showing the features before and after differencing. Is shared context retained through additional inputs or interaction terms? Is there any normalization omitted from the equations? What information is retained in the FiF_i-only setting in Table 15?

  4. Is there existing validation of intermediate relation-chain correctness, actual use of edited facts, or counterfactual fact replacement? If not, could the mechanistic claims be limited to the measured improvements in hop consistency and answer accuracy, with the explanations concerning dependency-chain recovery explicitly identified as requiring further validation?

7 comments
AuthorsOpenReview ↗

Response to Reviewer a8p4 - Question 1 (1/2): Technical relationship and candidate control

Question 1(a): Which modules are directly inherited or adapted from DecKER?

Thank you for asking us to clarify the technical relationship. StructKE builds on DecKER’s masked-plan-then-fill paradigm. Masked-path planning, knowledge injection and filling, entity-type consistency (KI), and entropy-derived path certainty (RPP) are inherited or adapted components.

Our additions are target-hop-driven candidate control and progressive recovery, supported by StructEst, and learned candidate selection through SDPR. StructEst supplies an explicit target-hop prediction. Candidate control prioritizes paths matching this prediction, while progressive recovery admits fallback paths when strict candidates are insufficient. SDPR then selects the final answer using structural and consistency diagnostics.

Question 1(b): Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions?

We evaluate predicted-hop filtering, progressive recovery, and SDPR in three controlled comparisons.

1. Directly adding predicted-hop filtering to original DecKER outputs.

We compare original DecKER with DecKER augmented by predicted-hop filtering on the full MQuAKE-CF-3k-v2 benchmark, using the same candidate generation, retrieval, knowledge filling, and RPP-top-3/raw-KI selection rule.

We add only a filtering step before selection: if the DecKER pool contains candidates matching the predicted hop count, selection is restricted to those candidates; otherwise, the original pool is retained.

Table: Adding predicted-hop filtering to DecKER on MQuAKE-CF-3k-v2 with Qwen2.5-7B-Instruct.

ConfigurationCase-level accuracy (%)
Original DecKER49.80
DecKER + predicted-hop filtering51.03

The improvement of 1.23 percentage points, corresponding to 37 additional correct cases, demonstrates that the predicted structural prior improves DecKER’s existing selection without increasing generation or knowledge filling.

2. Separately evaluating strict filtering and progressive recovery under shared generation attempts.

We use MQuAKE-CF-300-v2, a fixed diagnostic subset containing 300 cases with three paraphrases per case. For each question, all configurations share three batches of six generation attempts (18 attempts in total), the same backbone, prompts, retrieval access, and knowledge-filling outputs. Each configuration retains at most six candidate paths. The final selector is fixed to the RPP-top-3/raw-KI rule; SDPR is not used.

This controlled comparison isolates candidate admission and recovery by applying each policy to the same generated paths. Full StructKE uses the complete end-to-end generation, resampling, recovery, and SDPR selection configuration. The two evaluations therefore measure component effects and complete-system performance, respectively.

Only the policy for retaining generated paths changes:

  • Without hop filtering: retain the first six available parseable paths without checking their hop counts.
  • Strict predicted-hop filtering: retain only paths matching the predicted hop count. If none match, the retained pool is empty.
  • Predicted-hop filtering + recovery: prioritize matching paths and progressively relax the admission conditions when strict candidates are insufficient.

Table: Effect of predicted-hop filtering and recovery with shared generation attempts and a fixed RPP-top-3/raw-KI selector on MQuAKE-CF-300-v2.

Candidate-control configurationCase-level accuracy (%)
Without hop filtering53.00
Strict predicted-hop filtering58.33
Predicted-hop filtering + recovery59.67

Case-level accuracy counts a case as correct when at least one of its three paraphrases is answered correctly.

Strict filtering improves accuracy from 53.00% to 58.33%, showing that retaining structurally matched candidates improves final answers under the same raw-generation budget.

Recovery further increases accuracy to 59.67% with the same selector. This result is 179/300 cases under the fixed RPP/KI rule; the Full StructKE result of 59.67% reported below is 1,790/3,000 cases with SDPR. Recovery also reduces the number of questions with no retained candidate from 50 to zero. Thus, strict filtering improves candidate selectivity, while recovery supplies alternatives when the strict constraint leaves insufficient candidates.

AuthorsOpenReview ↗

Response to Reviewer a8p4 - Question 1 (2/2): SDPR selection and additional supervision

Continued from our response to Question 1 (1/2).

Question 1(b) (continued): Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions?

3. Evaluating SDPR on identical candidate outputs.

To isolate the selection procedure, we compare the raw-KI selection rule adapted from DecKER with frozen SDPR on the full MQuAKE-CF-3k-v2 benchmark under the paper's evaluation configuration. Both selectors receive the same StructKE candidate inputs and filling diagnostics.

Table: Selector comparison on identical StructKE candidate outputs on MQuAKE-CF-3k-v2 with Qwen2.5-7B-Instruct.

SelectorCase-level accuracy (%)
Raw-KI selection58.10
SDPR59.67

On the same candidate inputs, SDPR improves case accuracy by 1.57 percentage points, corresponding to 47 additional correct cases, supporting the contribution of learned candidate selection.

Together, these experiments provide separate evidence for our additions: predicted-hop filtering improves original DecKER outputs, recovery maintains candidate availability under strict structural constraints, and SDPR improves candidate selection.

Question 1(c): Please clarify the use and impact of additional supervision.

StructKE introduces two supervised components with distinct roles:

  • StructEst learns target-hop prediction from MuSiQueAns followed by MQuAKE-CF structural adaptation. Cases overlapping MQuAKE-CF-3k-v2 are removed from its MQuAKE-CF training data.
  • SDPR learns candidate preferences from a separate 500-case MQuAKE-CF training split, expanded into 1,500 questions. Preference labels are derived from candidate-answer correctness, yielding 374 candidate pairs and their reversals, or 748 directional training examples.

Both models remain frozen throughout the reported comparisons. Candidate filtering uses predicted hop counts rather than annotated test hop counts, and no condition-specific retraining or test-set tuning is performed.

The experiments evaluate their effects separately. The StructEst-derived hop prior improves selection on original DecKER outputs, and SDPR improves selection when candidate generation and filling are fixed. With the RPP/KI selector fixed, strict filtering and recovery improve case accuracy from 53.00% to 59.67%, providing separate evidence for the contribution of candidate control.

AuthorsOpenReview ↗

Response to Reviewer a8p4 - Question 2: Result sources and consistency

Question 2: Which runs or publications are the sources of each method's results in Tables 3 and 14? Please explain the discrepancies and identify the configurations underlying the outputs used in Tables 1, 4, and 9 and Figure 3. If the settings differ, please provide sufficient configuration details to reconcile the two sets of results, without presupposing that either set is incorrect.

Thank you for pointing this out. Tables 3 and 14 use results from different sources: the baseline values in Table 3 are quoted from the DecKER paper, whereas Table 14 reports our local experiments conducted for paraphrase-level evaluation and reasoning-trace analysis. The original manuscript did not clearly distinguish these sources, making the tables appear to report different statistics computed from the same outputs.

You are correct that Case Any has the same definition in both tables: a case is counted as correct if at least one of its three paraphrases is answered correctly. The differences between the two sets of results arise from different implementation settings, not different definitions of Case Any. In the revised paper, we will unify the implementations, answer-evaluation rules, and result aggregation so that the main table and related analyses use the same set of experimental outputs. We will also document the implementation configurations in the appendix to eliminate confusion arising from the different sources.

This difference in sources does not change StructKE's main accuracy advantage over its closest competitor, DecKER. The Case Any results for DecKER and StructKE are individually consistent across the two tables, apart from rounding. Their comparison therefore does not depend on the discrepancies in other baselines across the tables.

Table 14 further compares the two methods across three evaluation metrics:

Table: Comparison of StructKE and DecKER across paraphrase-evaluation metrics.

MetricDecKERStructKEGain (percentage points)
Case Any49.8059.67+9.87
Question31.1641.17+10.01
All-314.2022.90+8.70

The advantage holds for case-level evaluation requiring at least one correct paraphrase, question-level evaluation, and the stricter criterion requiring all three paraphrases to be correct. In addition, Table 3 reports gains over DecKER of 4.5, 9.9, and 6.2 percentage points on MQuAKE-CF-3k-v2 across the three backbone models, respectively.

Both sets of results thus support StructKE's accuracy advantage over DecKER on MQuAKE-CF-3k-v2, while the paraphrase-level evaluation provides further evidence of its greater robustness to paraphrasing.

Reviewer_sPT6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why an edited fact often fails to affect the answer to a multi-hop question. The authors argue that the model may skip or distort the intermediate factual steps. StructKE trains an estimator to predict the required hop count, generates masked reasoning paths under that constraint, fills them using edited facts, and ranks the candidate answers. The method is tested on three language models across MQuAKE and RippleEdits settings. Its largest gains are on MQuAKE-CF-3k-v2, where it beats DecKER on all three models. The paper also reports a failure analysis, hop-match statistics, and ablations of the main components.

Critical strengths
  1. Gains over the closest baseline. On MQuAKE-CF-3k-v2, StructKE beats DecKER by 4.5, 9.9, and 6.2 points across the three models. The qwen-7b advantage also holds when all three paraphrases of a case must be answered correctly. The result survives a more demanding measure than the main case-level score.
  2. Evidence for the hop controls. On the diagnostic subset, accuracy drops by about five points without the hop estimator and six points without exact-hop candidates. These results give a concrete reason to include hop-guided search, the clearest addition to DecKER’s pipeline.
Critical weaknesses
  1. The advance over DecKER needs a clearer account. DecKER already generates and fills masked paths and evaluates candidate answers. StructKE adds hop prediction, constrained search, and a learned ranker. The ablations show that these additions help, but figure 4 also shows substantially higher latency for the full system. I am persuaded by the accuracy gains; I am less sure which DecKER failures the hop controls solve and how much of the improvement comes from the stronger selection pipeline. The authors could address this through a closer discussion of the existing ablations and accuracy–latency results.
  2. Hop alignment leaves the path-repair explanation open. The failure analysis largely identifies structural errors through chain-length mismatch, and it uses a different diagnostic source for DecKER. In table 4, 72.81% of StructKE’s wrong answers still match the target hop count. The exact-hop ablation establishes that length control helps accuracy, but the evidence does not yet show that newly correct answers follow the intended intermediate relations and entities. If the existing candidate records show this for cases missed by DecKER, they would support the paper’s explanation of the gain.
Questions for authors
  1. Which failure mode in DecKER is addressed specifically by predicted hop control? In the existing candidate traces, does the selected path change at the missing or skipped factual transition?
  2. Among cases solved by StructKE and missed by DecKER, how often do the selected paths use the edited fact and follow the intended intermediate relations and entities?
  3. How should the accuracy gain be weighed against the additional estimator and ranker training and the latency shown in figure 4? Which configuration best represents the paper’s intended use?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sPT6 - Question 1: Hop control and candidate selection

Thank you for recognizing StructKE's accuracy gains and the ablation evidence for hop control. We provide additional comparisons and analyses addressing the specific failure mode, edited-fact use and relation-chain validity, and the accuracy-cost trade-off.

Question 1: Which failure mode in DecKER is addressed specifically by predicted hop control? In the existing candidate traces, does the selected path change at the missing or skipped factual transition?

Predicted hop control addresses premature compression of a masked plan, where a necessary intermediate entity has no separate slot and the next edited fact is consequently applied to the wrong relation. It supplies a depth prior for candidate admission; SDPR subsequently chooses among the admitted candidates. We provide controlled comparisons and a paired trace to distinguish these roles.

First, we added predicted-hop filtering directly to the original DecKER candidate outputs on all 3,000 MQuAKE-CF-3k-v2 cases with Qwen2.5-7B-Instruct. Generation, retrieval, filling, and the original RPP-top-3/raw-KI selector remain unchanged. If a pool contains a path matching the predicted depth, selection is restricted to matching paths; otherwise, the original pool is retained.

Table: Predicted-hop filtering applied to original DecKER outputs on MQuAKE-CF-3k-v2.

ConfigurationCorrect cases / 3,000Case accuracy (%)
Original DecKER1,49449.80
DecKER + predicted-hop filtering1,53151.03

The gain is 1.23 percentage points without additional generation or filling and without SDPR. This directly establishes that the predicted depth helps select from DecKER's existing candidates.

We also separated admission policy from the selector on MQuAKE-CF-300-v2, a fixed 300-case diagnostic subset with three paraphrases per case. All conditions share 18 raw generation attempts per question, the same prompts, retrieval and filling outputs, and a maximum of six admitted paths. The two selectors are a fixed RPP-top-3/raw-KI rule and frozen SDPR.

Table: Candidate-control and selector comparisons under shared generation attempts on MQuAKE-CF-300-v2. All values are case accuracy (%).

Candidate admissionFixed RPP/KI ruleFrozen SDPR
Without hop filtering53.0053.67
Predicted-hop filtering + recovery59.6760.33

Under shared generation attempts, predicted-hop filtering with recovery improves accuracy by 6.67 percentage points with the fixed RPP/KI rule and by 6.66 points with frozen SDPR. The benefit of structural candidate control holds under both selectors and therefore cannot be attributed solely to stronger final ranking.

SDPR's additional contribution is measured separately on the identical prefiltered candidate pools used by the main Full configuration across all 3,000 cases. Replacing raw-KI selection with frozen SDPR increases case accuracy from 58.10% to 59.67%, a gain of 1.57 points. Thus, candidate control and learned selection each have an independently measured benefit. These comparisons use different interventions and should not be added into an arithmetic decomposition of the overall gain.

The existing traces also show the missing transition directly. Consider: What country does the spouse of the author of “Barnaby Rudge” hold citizenship in? The updated facts specify Céline Dion as the author, Joe Manganiello as her spouse, and Argentina as his citizenship country.

Table: Paired selected paths for the same question. Facts are counterfactual benchmark edits.

MethodSelected filled pathFinal answer
DecKERAuthor of Barnaby Rudge = Céline Dion; “Céline Dion's spouse holds citizenship in Joe Manganiello.”Joe Manganiello
StructKE FullAuthor = Céline Dion; Céline Dion's spouse = Joe Manganiello; Joe Manganiello's citizenship country = Argentina.Argentina

DecKER's two-step plan combines the spouse and citizenship relations into one country-valued slot. The filling log shows that the spouse edit inserts a person, Joe Manganiello, into that slot. StructKE's selected three-step plan gives the spouse an explicit intermediate slot and then applies the citizenship edit to that person. All three edited facts are injected at their corresponding transitions. The explicit spouse slot preserves the intermediate entity needed to apply the citizenship edit correctly.

We will use these comparisons and paired traces to clarify the respective roles of hop-guided admission, recovery, and SDPR in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer sPT6 - Question 2: Edited-fact use and reference-chain verification

Question 2: Among cases solved by StructKE and missed by DecKER, how often do the selected paths use the edited fact and follow the intended intermediate relations and entities?

We supplement the analysis with checks of actual edited-fact use and complete reference-chain coverage, examining execution logs and factual transitions in the selected paths.

Across all 9,000 questions in MQuAKE-CF-3k-v2 with Qwen2.5-7B-Instruct, StructKE Full answers 1,393 questions correctly that DecKER does not. We trace filling operations in the actual selected executions: the logs must record a replacement from the edit store, and the inserted value must match the edited object. In 1,305 questions, or 93.68%, the selected execution actually injects all edits belonging to the evaluated case.

We further apply the same explicit relation-verification rules to both methods' final selected paths across all 9,000 questions. Passing requires the updated reference chain's subjects, predicates, objects, and intermediate connections to be explicitly supported, with the returned answer matching the chain endpoint.

Table: Complete reference-chain verification across all 9,000 questions in MQuAKE-CF-3k-v2.

MethodQuestions passing verificationComplete reference-chain verification rate
DecKER1,04511.61%
StructKE Full1,61817.98%
Net increase over DecKER573+6.37 percentage points

StructKE yields 573 more selected paths that pass complete reference-chain verification. This comparison directly checks updated factual relations and their connections, complementing the hop-match statistics.

The paired example in Question 1 shows how preserving an intermediate entity allows the next edited fact to be applied correctly. The full-set comparison provides broader evidence that StructKE produces more selected paths with complete reference-chain support.

We will add these results to the revised paper and clearly distinguish hop consistency, edited-fact injection, and complete reference-chain verification.

AuthorsOpenReview ↗

Response to Reviewer sPT6 - Question 3: Accuracy-cost trade-off

Question 3: How should the accuracy gain be weighed against the additional estimator and ranker training and the latency shown in figure 4? Which configuration best represents the paper’s intended use?

StructKE Full is our intended accuracy-oriented configuration; StructKE-Lite is the budget-constrained option. With Qwen2.5-7B-Instruct, Full improves MQuAKE-CF-3k-v2 case accuracy from DecKER's 49.80% to 59.67%. Candidate generation accounts for 77.4% of Full's steady inference time, making the generation and resampling budget the main control for the accuracy-latency trade-off.

Table: Accuracy and inference cost on MQuAKE-CF-300-v2 with Qwen2.5-7B-Instruct.

MethodCase accuracy (%)Latency per case (s)Tokens per case
DecKER-N653.0022.788,027.6
StructKE-Lite58.0032.668,451.4
StructKE Full65.67101.5930,150.2

Full achieves the highest accuracy among the three configurations. By reducing the generation budget, StructKE-Lite (n=2n=2, xmax⁡=2x_{\max}=2) lowers latency from 101.59 to 32.66 seconds per case. Compared with DecKER-N6, Lite gains 5.00 percentage points in accuracy with a 5.28% increase in tokens. We will state this Lite configuration consistently in the text and Figure 4.

Reviewer_KFuxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on multi-hop question answering after knowledge editing with in-context editing methods. The authors argue that the main bottleneck is "structural degradation": the generated reasoning chain has a different length than the number of hops the question needs (shortcuts, early stopping, or redundant expansion), so the edited fact is not used. They propose StructKE, which first predicts the target hop count with a fine-tuned 7B estimator (StructEst), generates candidate reasoning chains with masked slots under this hop constraint, fills the slots with edited facts when a conflict with the parametric answer is detected, and picks the final answer with a learned pairwise ranker (SDPR). Experiments on MQuAKE-CF-3k-v2, MQuAKE-T and two RippleEdits sets with three 7B to 14B backbones show gains over parametric and in-context editing baselines, mostly on MQuAKE-CF-3k-v2.

Critical strengths
  1. The observation is interesting. Looking at why in-context editing fails on multi-hop questions through the length of the effective fact chain (Section 2.2) is an intuitive angle, and if the claim is confirmed it would explain a lot about the behavior of methods such as MeLLo and EditCoT.
  2. The method is a reasonable use of this observation, and the experiments cover three backbones, four datasets and a breakdown by number of hops and edits (Figure 3). The gain on MQuAKE-CF-3k-v2 is clear (e.g., 59.7 vs 49.8 for DecKER with Qwen2.5-7B), and the margin grows with the number of hops.
Critical weaknesses
  1. The core claim of the paper is not supported by enough analysis. The whole motivation is that structural errors (chain-length mismatch) are the main reason for failure, but Table 1 only reports the share of structural errors among failed samples, where "structural" means a recovered chain whose length differs from the target. This does not show that the mismatch causes the failure. We would need to know how many of the failed samples actually have a mismatch, how many mismatched samples fail, and what happens if the mismatch is fixed for the baselines. Table 4 even points in the other direction for some methods: only 6.77% of EditCoT's correct outputs match the target hop count, and MeLLo has almost the same match rate for correct and wrong outputs (44.12 vs 41.54), so a length mismatch does not seem to prevent a correct answer. The chain length is also recovered in a different way for each method (subquestions, graph paths, or DecKER's own diagnostic category), so the numbers in Table 1 are not comparable across methods. Since this is the central idea of the paper, it deserves much more investigation (e.g., giving the baselines the oracle hop count, or a controlled intervention on the chain length).
  2. The comparison and the added gains need more care. First, StructKE adds two trained components (StructEst and SDPR) that the baselines do not have. StructEst is trained on MuSiQue and MQuAKE-CF data and tested on MQuAKE-CF-3k-v2, and the comparison with general-purpose LLMs in Table 2 (93.34 vs 60 to 72) is between a fine-tuned model and prompting, so it does not say much. Second, with Llama-3.1-8B the gain over DecKER is small or negative on three of the four datasets (81.8 vs 81.3 on MQuAKE-T, 46.2 vs 47.4 on Ripple-Pop, 45.9 vs 44.9 on Ripple-Rand), and with Qwen2.5-14B it is below 1 point on MQuAKE-T and Ripple-Pop, so "strong gains" is mainly true for MQuAKE-CF-3k-v2. Third, the ablation (Table 5) uses a 300-case subset, where the 3 to 6 point drops are about 9 to 18 cases, with no variance, and the oracle hop only gives +1.66 over the predicted hop. The latency in Figure 4 is also much higher than most baselines.

Overall, I like the observation and the method is reasonable, but the empirical analysis behind the main claim needs to be improved. I lean toward weak acceptance.

Questions for authors
  • What is the difference between StructKE and EditCoT? EditCoT also uses a small model to analyze and revise the chain of thought, so it should be discussed more clearly, in addition to being a baseline.
  • Among the failed samples of each baseline, how many have a hop mismatch, and among the samples with a hop mismatch, how many fail? Does fixing the hop count (e.g., the oracle hop for the baselines) improve them?
  • Does the hop estimator generalize beyond the MQuAKE-style questions? What is its accuracy on RippleEdits, and is any MQuAKE data used to train SDPR?
  • Can you report variance or a significance test for the main results and run the ablation on the full 3k set?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KFux - Question 1: How StructKE differs from EditCoT

Thank you for highlighting the analysis of fact-chain structure and the gains on longer chains. EditCoT and StructKE both use trained auxiliary models, but they intervene at different points. EditCoT conditions on the question, the edited knowledge, and an existing chain of thought to revise that chain and then answer. StructKE predicts the target hop count and uses it to build and screen masked reasoning skeletons before filling in entities.

This separation gives necessary intermediate entities their own slots before edited facts are inserted. StructEst supplies the target depth, which guides candidate admission and recovery; SDPR then ranks candidates using structural and consistency signals. We will make this distinction explicit in the related-work and method comparisons.

AuthorsOpenReview ↗

Response to Reviewer KFux - Question 2: Hop mismatch and controlled interventions

Hop mismatch is a useful warning sign across the baselines we evaluated. For DecKER, the case error rate is 30.94% when the recovered chain matches the target depth and 79.30% when it does not. The observational comparison motivates the controlled filtering experiments below.

Hop mismatch and answer errors on MQuAKE-CF-3k-v2 (Qwen2.5-7B-Instruct). We use the first correct paraphrase output for a successful case and the first answered paraphrase otherwise. Let MM denote a hop mismatch and WW an incorrect case.

MethodP(M∣W)P(M\mid W): mismatch among errorsP(W∣M)P(W\mid M): error among mismatches
MeLLo1,382/2,966 (46.59%)1,382/1,401 (98.64%)
PokeMQA2,131/2,851 (74.75%)2,131/2,237 (95.26%)
DeepEdit2,902/2,918 (99.45%)2,902/2,984 (97.25%)
RAE2,313/2,662 (86.89%)2,313/2,418 (95.66%)
EditCoT2,468/2,675 (92.26%)2,468/2,771 (89.07%)
DecKER858/1,506 (56.97%)858/1,082 (79.30%)

We compare matched and mismatched outputs within each method, since chain recovery differs across methods. The mismatch-group error rate ranges from 79.30% to 98.64%. For MeLLo and EditCoT, the length signal is best interpreted alongside factual consistency.

Controlled intervention on DecKER. On all 3,000 cases, we add predicted-hop filtering to DecKER while keeping candidate generation, retrieval, filling, and its RPP-top-3/raw-KI selector fixed. If any candidates match the predicted hop count, we select among them; otherwise, we keep the original pool. No SDPR is added.

ConfigurationCorrect cases / 3,000Case accuracy (%)
DecKER1,49449.80
DecKER + predicted-hop filtering1,53151.03

Filtering improves case accuracy by 1.23 percentage points (paired bootstrap 95% CI: [0.60, 1.90]; exact McNemar p<0.001p<0.001). The comparison shows that predicted depth helps DecKER choose among candidates it already produced.

Candidate admission on the fixed MQuAKE-CF-300-v2 subset. The subset contains 300 cases, with three paraphrases per case as in the full benchmark. All settings share 18 generation attempts per question, the same prompts and retrieval and filling outputs, and the same RPP-top-3/raw-KI selector. Each setting can admit up to six candidates.

Admission ruleCorrect cases / 300Case accuracy (%)Gain vs. no filtering (95% CI, pp)
No hop filtering15953.00—
Strict predicted-hop match17558.33+5.33 [1.67, 9.00]
Predicted-hop filtering + staged recovery17959.67+6.67 [3.00, 10.33]

With generation attempts and the selector held fixed, filtering with staged recovery improves case accuracy by 6.67 points. These two interventions use predicted hops. The full-set oracle-hop diagnostic for StructKE is reported in our response to Question 4.

Fact transitions beyond hop count. Across all 9,000 questions, we apply the same check to the paths selected by DecKER and StructKE. The output must explicitly support the updated reference chain's entities, relations, and intermediate links, and its answer must agree with the chain endpoint.

MethodQuestions passing the check / 9,000Pass rate (%)
DecKER1,04511.61
StructKE Full1,61817.98

StructKE yields 573 more explicitly verified chains (+6.37 points). This check measures what can be confirmed from the output; omitted or rephrased steps can prevent verification. It tests the edited facts and how they connect, beyond whether the chain has the expected length. We will add these statistics and controlled comparisons to the revision.

AuthorsOpenReview ↗

Response to Reviewer KFux - Question 3: RippleEdits and training-test separation

StructEst achieves 92.40% (462/500) hop prediction accuracy on RippleEdits-Pop and 92.00% (1,046/1,137) on RippleEdits-Random. These results evaluate the estimator on two-hop questions outside MQuAKE.

We separate training and test cases for both learned components. Before constructing supervision for StructEst adaptation or SDPR preference training, we remove cases that overlap with the MQuAKE-CF-3k-v2 test set. Both modules are frozen at evaluation, and test answers are not used to train or update either module.

StructEst first learns multi-hop structure from MuSiQueAns and is then adapted on the disjoint MQuAKE-CF training split. We will clarify in Table 2 that the 93.34% result is from a task-tuned estimator, whereas the general-purpose LLM results use prompting. SDPR learns from the disjoint MQuAKE-CF training split: 500 cases yield 1,500 paraphrased questions, and candidate answers are matched against their training references to form 748 directed preference pairs. We will document both training sources, the overlap check, and the frozen evaluation protocol together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.