- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies structural degradation in multi-hop knowledge editing: even when a model has access to edited facts, it may skip steps, terminate early, or deviate from the required relation chain, preventing the updated knowledge from influencing the final answer along the intended path. The authors propose StructKE, a four-stage framework. First, a LoRA-tuned StructEst predicts the target hop count. Second, the framework generates candidate paths with entity masks and type constraints, then filters and recovers candidates according to their hop alignment. Third, it performs conflict-aware knowledge filling. Finally, a Structure-Diagnostic Pairwise Ranker (SDPR) selects an answer using structural signals, type consistency, generation certainty, and candidate statistics. The paper reports experiments across three language models and four dataset settings, together with hop-consistency analyses, component ablations, and efficiency comparisons.
Critical strengths
-
Multi-hop editing requires not only retrieving updated facts but also allowing them to affect subsequent dependencies. Using the target hop count as a predictable and verifiable control signal, while progressively relaxing candidate constraints when exact matches are insufficient, is a concrete and implementable design with potential value for improving reasoning reliability after editing.
-
The paper covers three models and four dataset settings, and reports performance across reasoning depths, numbers of edited relations, paraphrases, and latency conditions. In Table 5, removing StructEst, SDPR, or exact-hop candidates consistently reduces performance, providing preliminary evidence that the components contribute to the framework.
-
SDPR formulates candidate selection as pairwise preference learning within each question, combining hop alignment, candidate recovery states, entity-type consistency, and generation certainty. This design can learn interactions among different quality signals and provides a flexible mechanism for resolving conflicts between candidates, such as a structurally aligned but low-confidence path and a high-confidence path that skips steps. It is a useful extension of fixed-rule filtering.
Critical weaknesses
-
The technical relationship to DecKER and the independent evidence for the new contributions are insufficiently clear. DecKER already adopts a masked-path planning followed by knowledge injection strategy, and evaluates candidates using entity-type consistency and generation entropy. StructKE's masked planning, conflict-aware filling, and KI/RPP signals have direct counterparts in DecKER. Its main incremental contributions should therefore lie in StructEst, hop control and recovery, and SDPR, rather than primarily in the plan-then-fill framework itself. The paper cites and compares against DecKER and provides internal ablations, but these experiments do not sufficiently disentangle the effects of the new modules, candidate search budgets, and additional supervision. A direct comparison between DecKER and variants that progressively incorporate the new modules, under the same retrieval access and matched generation budgets, is needed to establish the incremental contributions. This concern is about contribution attribution and validation, not an allegation of research misconduct. (Sections 3.2–3.4 and Table 5; Sections 4.1–4.3 of DecKER.)
-
Unexplained differences in baseline results across key tables limit their comparability. For Qwen2.5-7B-Instruct on MQuAKE-CF-3k-v2, both Section 4.1.1 and Appendix J.1 define case-level success as answering at least one of the three paraphrases correctly. However, Table 3 and the Case Any column of Table 14 report substantially different results: 38.3% versus 11.27% for RAE, 26.2% versus 10.83% for EditCoT, and 12.3% versus 2.73% for DeepEdit. By contrast, the results for DecKER and StructKE agree within rounding precision. I do not conclude from these discrepancies that either set of results is incorrect; rather, I ask the authors to explain their sources. For example, do the tables use previously reported results versus local reproductions, or different retrieval access, prompts, model versions, or decoding settings? Appendix H describes an adjustment to RAE's knowledge access, but does not clarify how it relates to the two tables. Until the sources and settings are explained, it remains unclear whether the main results and structural diagnostics jointly support the same conclusion. (Tables 3 and 14, Section 4.1.1, and Appendices H and J.1.)
-
SDPR's feature differencing removes some of the context it is claimed to use, creating a mismatch between its mathematical definition and functional explanation. Equation (4) includes shared information, such as candidate-pool size and target hop count, whereas Equations (8) and (11) feed only the difference between two candidates' features from the same question into the ranker. If the shared information is and the candidate-specific information is , then
Only the shared dimensions cancel, not all candidate features; candidate-specific information such as hop-match status may remain. Under the current equations, however, pool size and target hop count themselves cannot directly participate in ranking. The authors should clarify whether additional contextual inputs or precomputed interaction features are used. The -only result in Table 15 also requires an explanation based on the complete feature definition. Furthermore, differencing removes a common additive offset but does not automatically provide the scale invariance claimed in the paper. These issues affect the understanding of what information SDPR actually uses, rather than merely its notation. (Equations (4), (8), and (11), line 282, and Table 15.)
-
The structural diagnostics primarily measure length consistency and do not sufficiently support the mechanistic claim that correct factual dependencies are recovered. The paper acknowledges that hop consistency is only a proxy, which is an appropriate qualification. Nevertheless, paths with the same hop count can follow different relation chains, and KI checks entity types without distinguishing an outdated answer from an edited answer of the same type. In Table 4, 72.81% of StructKE's incorrect outputs still match the target hop count, highlighting the gap between length consistency and correct propagation of edited knowledge. Correlations conditioned on correctness also do not, by themselves, establish a causal explanation. The current evidence supports improved hop consistency and answer accuracy, but is insufficient to establish that edited knowledge propagates because the correct factual-transition structure is preserved. Additional verification of relation chains or intermediate facts, or interventions that replace edited facts under controlled candidate budgets, would help distinguish genuine dependency-chain recovery from gains due to length filtering, search, and ranking. (Sections 2.2 and 4.3–4.4, Tables 4 and 9, and Appendix B.)
Questions for authors
The following questions correspond to the four concerns above and could materially affect my assessment. The requested validation is intended to support the paper's current claims, rather than extend it to a new research task.
-
Which modules are directly inherited or adapted from DecKER? Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions? Please clarify the use and impact of additional supervision.
-
Which runs or publications are the sources of each method's results in Tables 3 and 14? Please explain the discrepancies and identify the configurations underlying the outputs used in Tables 1, 4, and 9 and Figure 3. If the settings differ, please provide sufficient configuration details to reconcile the two sets of results, without presupposing that either set is incorrect.
-
Please list all fields in and provide a real candidate-pair example showing the features before and after differencing. Is shared context retained through additional inputs or interaction terms? Is there any normalization omitted from the equations? What information is retained in the -only setting in Table 15?
-
Is there existing validation of intermediate relation-chain correctness, actual use of edited facts, or counterfactual fact replacement? If not, could the mechanistic claims be limited to the measured improvements in hop consistency and answer accuracy, with the explanations concerning dependency-chain recovery explicitly identified as requiring further validation?
7 comments
Response to Reviewer a8p4 - Question 1 (2/2): SDPR selection and additional supervision
Continued from our response to Question 1 (1/2).
Question 1(b) (continued): Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions?
3. Evaluating SDPR on identical candidate outputs.
To isolate the selection procedure, we compare the raw-KI selection rule adapted from DecKER with frozen SDPR on the full MQuAKE-CF-3k-v2 benchmark under the paper's evaluation configuration. Both selectors receive the same StructKE candidate inputs and filling diagnostics.
Table: Selector comparison on identical StructKE candidate outputs on MQuAKE-CF-3k-v2 with Qwen2.5-7B-Instruct.
| Selector | Case-level accuracy (%) |
|---|---|
| Raw-KI selection | 58.10 |
| SDPR | 59.67 |
On the same candidate inputs, SDPR improves case accuracy by 1.57 percentage points, corresponding to 47 additional correct cases, supporting the contribution of learned candidate selection.
Together, these experiments provide separate evidence for our additions: predicted-hop filtering improves original DecKER outputs, recovery maintains candidate availability under strict structural constraints, and SDPR improves candidate selection.
Question 1(c): Please clarify the use and impact of additional supervision.
StructKE introduces two supervised components with distinct roles:
- StructEst learns target-hop prediction from MuSiQueAns followed by MQuAKE-CF structural adaptation. Cases overlapping MQuAKE-CF-3k-v2 are removed from its MQuAKE-CF training data.
- SDPR learns candidate preferences from a separate 500-case MQuAKE-CF training split, expanded into 1,500 questions. Preference labels are derived from candidate-answer correctness, yielding 374 candidate pairs and their reversals, or 748 directional training examples.
Both models remain frozen throughout the reported comparisons. Candidate filtering uses predicted hop counts rather than annotated test hop counts, and no condition-specific retraining or test-set tuning is performed.
The experiments evaluate their effects separately. The StructEst-derived hop prior improves selection on original DecKER outputs, and SDPR improves selection when candidate generation and filling are fixed. With the RPP/KI selector fixed, strict filtering and recovery improve case accuracy from 53.00% to 59.67%, providing separate evidence for the contribution of candidate control.
Response to Reviewer a8p4 - Question 2: Result sources and consistency
Question 2: Which runs or publications are the sources of each method's results in Tables 3 and 14? Please explain the discrepancies and identify the configurations underlying the outputs used in Tables 1, 4, and 9 and Figure 3. If the settings differ, please provide sufficient configuration details to reconcile the two sets of results, without presupposing that either set is incorrect.
Thank you for pointing this out. Tables 3 and 14 use results from different sources: the baseline values in Table 3 are quoted from the DecKER paper, whereas Table 14 reports our local experiments conducted for paraphrase-level evaluation and reasoning-trace analysis. The original manuscript did not clearly distinguish these sources, making the tables appear to report different statistics computed from the same outputs.
You are correct that Case Any has the same definition in both tables: a case is counted as correct if at least one of its three paraphrases is answered correctly. The differences between the two sets of results arise from different implementation settings, not different definitions of Case Any. In the revised paper, we will unify the implementations, answer-evaluation rules, and result aggregation so that the main table and related analyses use the same set of experimental outputs. We will also document the implementation configurations in the appendix to eliminate confusion arising from the different sources.
This difference in sources does not change StructKE's main accuracy advantage over its closest competitor, DecKER. The Case Any results for DecKER and StructKE are individually consistent across the two tables, apart from rounding. Their comparison therefore does not depend on the discrepancies in other baselines across the tables.
Table 14 further compares the two methods across three evaluation metrics:
Table: Comparison of StructKE and DecKER across paraphrase-evaluation metrics.
| Metric | DecKER | StructKE | Gain (percentage points) |
|---|---|---|---|
| Case Any | 49.80 | 59.67 | +9.87 |
| Question | 31.16 | 41.17 | +10.01 |
| All-3 | 14.20 | 22.90 | +8.70 |
The advantage holds for case-level evaluation requiring at least one correct paraphrase, question-level evaluation, and the stricter criterion requiring all three paraphrases to be correct. In addition, Table 3 reports gains over DecKER of 4.5, 9.9, and 6.2 percentage points on MQuAKE-CF-3k-v2 across the three backbone models, respectively.
Both sets of results thus support StructKE's accuracy advantage over DecKER on MQuAKE-CF-3k-v2, while the paraphrase-level evaluation provides further evidence of its greater robustness to paraphrasing.
Response to Reviewer a8p4 - Question 1 (1/2): Technical relationship and candidate control
Question 1(a): Which modules are directly inherited or adapted from DecKER?
Thank you for asking us to clarify the technical relationship. StructKE builds on DecKER’s masked-plan-then-fill paradigm. Masked-path planning, knowledge injection and filling, entity-type consistency (KI), and entropy-derived path certainty (RPP) are inherited or adapted components.
Our additions are target-hop-driven candidate control and progressive recovery, supported by StructEst, and learned candidate selection through SDPR. StructEst supplies an explicit target-hop prediction. Candidate control prioritizes paths matching this prediction, while progressive recovery admits fallback paths when strict candidates are insufficient. SDPR then selects the final answer using structural and consistency diagnostics.
Question 1(b): Are there results that separately add hop control and SDPR to DecKER under the same retrieval access, total candidate-generation budget, and knowledge-filling conditions?
We evaluate predicted-hop filtering, progressive recovery, and SDPR in three controlled comparisons.
1. Directly adding predicted-hop filtering to original DecKER outputs.
We compare original DecKER with DecKER augmented by predicted-hop filtering on the full MQuAKE-CF-3k-v2 benchmark, using the same candidate generation, retrieval, knowledge filling, and RPP-top-3/raw-KI selection rule.
We add only a filtering step before selection: if the DecKER pool contains candidates matching the predicted hop count, selection is restricted to those candidates; otherwise, the original pool is retained.
Table: Adding predicted-hop filtering to DecKER on MQuAKE-CF-3k-v2 with Qwen2.5-7B-Instruct.
The improvement of 1.23 percentage points, corresponding to 37 additional correct cases, demonstrates that the predicted structural prior improves DecKER’s existing selection without increasing generation or knowledge filling.
2. Separately evaluating strict filtering and progressive recovery under shared generation attempts.
We use MQuAKE-CF-300-v2, a fixed diagnostic subset containing 300 cases with three paraphrases per case. For each question, all configurations share three batches of six generation attempts (18 attempts in total), the same backbone, prompts, retrieval access, and knowledge-filling outputs. Each configuration retains at most six candidate paths. The final selector is fixed to the RPP-top-3/raw-KI rule; SDPR is not used.
This controlled comparison isolates candidate admission and recovery by applying each policy to the same generated paths. Full StructKE uses the complete end-to-end generation, resampling, recovery, and SDPR selection configuration. The two evaluations therefore measure component effects and complete-system performance, respectively.
Only the policy for retaining generated paths changes:
Table: Effect of predicted-hop filtering and recovery with shared generation attempts and a fixed RPP-top-3/raw-KI selector on MQuAKE-CF-300-v2.
Case-level accuracy counts a case as correct when at least one of its three paraphrases is answered correctly.
Strict filtering improves accuracy from 53.00% to 58.33%, showing that retaining structurally matched candidates improves final answers under the same raw-generation budget.
Recovery further increases accuracy to 59.67% with the same selector. This result is 179/300 cases under the fixed RPP/KI rule; the Full StructKE result of 59.67% reported below is 1,790/3,000 cases with SDPR. Recovery also reduces the number of questions with no retained candidate from 50 to zero. Thus, strict filtering improves candidate selectivity, while recovery supplies alternatives when the strict constraint leaves insufficient candidates.