acceptodds
Under review as a conference paper at ICLR 2027

Reasoning or Retrieval? Diagnosing LLM Failures on Temporal Knowledge Graphs with Ground-Truth Evidence

Abstract

When an LLM fails at multi-hop reasoning over an event history, is the model unable to reason or did retrieval fail to provide the evidence it needed? Existing benchmarks cannot disentangle these explanations because the events that caused an outcome are unknown. We introduce a controllable synthetic temporal knowledge graph (TKG) generator that constructs event histories from explicit multi-hop temporal patterns, calibrates them to real event statistics, and records the ground- truth evidence behind every event. This lets us directly separate retrieval from reasoning failures and measure how well retrieval reaches the evidence a query requires. Benchmarking seven LLMs against graph- and rule-based forecasters under a range of retrieval strategies, we find that strong LLMs can overcome increasing hop length when given the right context (the pattern’s ground-truth evidence together with one solved example) while neither suffices alone. For example, on 3-hop patterns, GPT-5.6-terra reaches 3.7 the Hits@1 of the GNN forecaster RE-Net. The dominant obstacle is instead retrieval. Under realistic retrieval methods, 82–98% of errors on 2- and 3-hop patterns occur because necessary evidence never reaches the model, and the best existing and proposed retrievers close at most 39% of the gap to the right context. Models remain vulnerable to frequency and recency shortcuts even when the correct evidence is present. Together, these results recast multi-hop temporal reasoning as primarily a retrieval problem. Strong LLMs can often perform the reasoning, but realizing that capability requires retrieval methods that reliably surface the evidence that matters.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
temporal knowledge graphs, large language models, multi-hop reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1eQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a synthetic temporal knowledge graph generator with per-event ground-truth evidence. Event histories are generated from twelve 1–3 hop temporal motifs and calibrated to ICEWS14 statistics with Bayesian optimization, and each event is stored with its causal source. Prediction errors can then be decomposed cleanly into retrieval failures and reasoning failures, and the paper provides an evidence reachability analysis that requires no model runs. The evaluation covers seven LLMs plus GNN/rule predictors under several retrieval strategies: with ground-truth evidence and one solved example of the same pattern (OICL+ex), the strongest model reaches 3.7 times RE-Net on 3-hop; under realistic retrieval, however, 82% to 98% of 2- and 3-hop errors trace to missing evidence, the best retriever closes only 39% of the gap, and frequency/recency shortcuts remain.

Critical strengths
  1. Relative to iTemporal and Test of Time, this is the first generator to record a causal source for every event and to quantify how well each retrieval strategy covers the twelve motif shapes, making the “retrieval or reasoning” question attributable and reproducible for the first time. Perturbed datasets and a second-seed robustness check add to the credibility of the conclusions.
  2. The diagnostic experiments are well designed: bootstrap confidence intervals, random-order baselines, and randomized, near-miss, frequency-removed, and dual-shortcut adversarial tests together make a systematic case for shortcut dependence. Candidate-anchored retrieval and EvRank give the community concrete directions to follow.
Critical weaknesses
  1. The headline conclusion does not generalize far beyond the synthetic setup. Every event is generated from a pattern, entities and relations are numeric IDs, and there is no background noise; calibration is only moderate, so the claim that “multi-hop reasoning is mainly a retrieval problem (82% to 98%)” holds only under the synthetic distribution. The OICL+ex upper bound also includes a solved example of the same pattern, which is almost unavailable in a real system, and the title’s either/or framing is easy to misread as retrieval being solvable.
  2. The comparison has fairness and statistical-power problems. There are only 603 queries at 3 hops, and the LLM runs use 200 to 500 seed queries, so the point estimates have wide intervals. GNNs score over all entities while LLMs score over an oracle context with only 1.9 to 3.5 entities on average, so the candidate sets are not symmetric and “3.7 times” is not an apples-to-apples comparison. Treating every FullEv=1 case as a reasoning failure also conflates “no pattern hint” with “cannot reason.”
Questions for authors
  1. Could you build evidence chains on a real TKG, even at small scale, using manual annotation or weak supervision, to test whether the ranked finding that 82% to 98% of errors come from retrieval still holds?
  2. Could you re-compare GNNs and LLMs under a unified candidate set? Does the random baseline being as high as 0.55 on 1-hop suggest the candidate set is too small and the task too easy?
  3. Can error attribution be split into “evidence present but no pattern hint” and “evidence present with a hint but composition fails”? What fractions do these two categories occupy in the Q1 intervention?
  4. 8B models remain near random under OICL+ex: is this a context-length issue, instruction following, or pattern-composition ability? Please provide a controlled analysis.
3 comments
AuthorsOpenReview ↗

Response (part 1)

Thank you for your review and criticisms. We tried to address the weaknesses (W) you pointed out and your questions (Q) below:

Re: W1. Generalization beyond our synthetic setup.

  • Background noise. We regenerated the 2- and 3-hop datasets with 0%, 50% and 75% unlabelled background facts. Noise uses the patterns' entity and relation popularity, so frequency cannot filter it. As noise rises, retrieval gets harder (best realistic full-evidence rate at 2 hops .28, .22, .17). The ground-truth context stays solvable (GPT-4.1 Hits@1 .64, .60, .62). The context lacks evidence for 84 - 96% of the errors.
  • The calibrated dataset itself. On ICEWS14-Synth's own consequence queries, grouped by hop, the findings match the hop datasets. GPT-4.1 reaches Hits@1 .89, .53, .48 with the evidence and one solved example, and .43, .14, .19 with entity history.
  • Numeric IDs. Appendix B already replaces identifiers with real ICEWS entity and relation names. This changes Hits@3 by at most .05 for GPT-4.1 and .08 for Qwen3-32B, and the cue reliance persists.
  • Calibration. Excluding the heuristic baselines, which were calibration targets, LLM scores on ICEWS14 and ICEWS14-Synth agree in level (chance-corrected Kendall tau = .58, r = .78). After removing each model's mean, method-level agreement is moderate (tau = .30, CI [.07, .44]). We will report this precisely going forward.
  • "A solved example is rarely available." It does not need to be given, it can be retrieved from history. MotifCase does this without labels and already identifies the generating pattern for 35–58% of queries (Sec. 5, Q4).
  • "The either/or framing of the title." We take the point. Our new counterfactual analysis (detailed in our responses to reviewers 1 and 2's first question/weakness if that's of interest) answers the question per model instead of calling the task "primarily a retrieval problem". For the strongest models, a better context repairs most multi-hop errors. For the 8B models, most errors persist even with the right context. Our revised abstract will state this.

Re: W2. Fairness and statistical power.

  • Power. All of our results carry 95% bootstrap CIs. A second seeded sample falls inside the first sample's CI for 151 of 155 conditions, and restricting all models to a shared query set changes none of our claims from the paper. The headline comparisons are far larger than their CIs. For example, at 3-hops GPT-5.6-terra's Hits@1 with the ground-truth context exceeds RE-Net's on the same candidates by .56 (CI [.48, .64]).
  • Asymmetric candidate sets. We restricted RE-Net's scores to exactly the candidates the LLM sees. On the candidates of the ground-truth context (evidence plus one solved example), RE-Net reaches Hits@1 .47, .40, .21 at 1-, 2-, 3-hops, against terra's .96, .76, .77 (3.6x at 3 hops). We will revise our statement of the comparison as conditional on that context. All new error attributions are chance-corrected against a random ranking over the same candidates for the same reason.
  • FullEv = 1 conflates "no hint" with "cannot reason". The 2×2 of Table 3 already separates the two directly. OICL is evidence without a hint, and OICL+ex is evidence with a hint (one solved example). Under realistic retrieval, almost no FullEv = 1 error has a hint in its context. At 2 - 3 hops, errors with both the evidence and a hint are at most 1.2% of all errors. The revision labels FullEv = 1 errors without a hint separately.
AuthorsOpenReview ↗

Response (part 2)

Re: Q1. Evidence chains on a real TKG.

  • We did this with weak supervision on ICEWS14. For each test query, the weak evidence is the body of the most confident grounding, from the query head, of a TLogic rule learned on the training split that predicts the true answer within 29 days. 286 of the 500 evaluated queries have such a chain (202 single facts, 84 multi-fact chains). For the multi-fact chains, the paper's eight label-free retrievers include the full chain in only 16 - 17% of contexts. The chain is missing for 96% (GPT-4.1) and 93% (Qwen3-32B) of the errors on these queries. When it is present, the models miss the top-3 only 10% and 19% of the time.
  • We also ran the counterfactual. Injecting the chain into the retriever's own 25-fact budget repairs only 8 - 13% of GPT-4.1's errors beyond chance. The chain alone in a clean context repairs a further 21 - 22%. This mirrors our synthetic result that missing evidence is necessary but, inside a realistic context, far from sufficient. The chains are rule-derived, so they are a proxy for causes.

Re: Q2. A unified candidate set and random baseline performance of .55 on 1-hop.

  • The .55 random Hits@1 is for OICL, the evidence alone, which names 1.9 entities on average at 1-hop. With the solved example (OICL+ex) it is .28. With entity history, contexts have 14 - 18 candidates and random Hits@1 is .06 - .10.
  • Re-ranking the same OICL+ex candidates, RE-Net reaches .47, .40, .21 at 1-, 2-, 3-hops, against GPT-5.6-terra's .96. .76, .77. The task is not easy on these candidate sets. At 3-hops RE-Net gets .21 and the 8B models stay at chance.

Re: Q3. Evidence without a hit vs failed composition.

  • In the 2×2 of Table 3, OICL is evidence without a hint and OICL+ex is evidence with a hint. The hint adds +.29 to +.53 Hits@1 at 2 - 3 hops for GPT-4.1, gpt-oss and terra, and nothing for the 8B models. Under realistic retrieval, errors with the evidence but no hint are 2 - 17% of all errors at 2- 3 hops, and errors with both are at most 1.2%.

  • We also measured the attribution counterfactually. We injected the evidence and a solved example into each realistic retriever's own 25-fact context, and compared with the clean OICL+ex context. For chance-corrected shares of 3-hop errors (Hits@3), we get the following: | model | repaired inside the budget | repaired only when clean | persists | |---|---|---|---| | GPT-5.6-terra | .27 | .47 | .25 | | GPT-4.1 | .09 | .48 | .43 | | gpt-oss-20b | .17 | .38 | .45 | | Qwen3-32B | .10 | .20 | .71 | | Llama-3.1-8B, Qwen3-8B | ≤ .04 | ≤ .02 | ≥ .96 |

  • The last column is explicitly the category you asked about. There, composition fails even with the evidence and a hint in a clean context. It is 25 - 45% of the errors for the strongest models and nearly all errors of the 8B models.

Re: Q4. Why the 8B models stay near chance on OICL+ex.

  • Context length is not likely to be the cause. OICL+ex contexts hold 3–7 facts.
  • Instruction following is not the cause either. Open models are scored exactly by the log-probability of each candidate's label, so there is no output parsing.
  • Re: what they pick. The 8B models mostly copy the solved example's own answer entity instead of mapping it onto the query. Qwen3-8B does so in 52, 29, 21% of queries at 1-, 2-, 3-hops, and Llama-3.1-8B in 38, 24, 24%. GPT-4.1 does so in 1 - 4% and terra in 0 - 12%. Without the example, no model picks that entity (<=2%).
  • More examples (2 and 4) do not help them. Qwen3-8B goes .22, .14, .13 at 3-hops while chance falls from .16 to .07, and Llama-3.1-8B stays at .18. They do help Qwen3-32B (.34, .44, .51).
  • A relation-renamed example leaves them at chance as well.

We believe the 8B failure is therefore most attributable to analogical mapping, the step that applies a solved example to the query. Our OICL vs. OICL+ex contrast isolates this step, and it is why their errors persist in the table above.

Area_Chair_UGcCOpenReview ↗

Please engage in the discussion with authors

Please engage in the discussion with authors, thank you.

AC

Reviewer_6riYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper generates temporal knowledge graphs with recorded antecedent evidence to study retrieval and reasoning failures separately. It compares realistic retrieval, oracle evidence, solved demonstrations, and adversarial distractors across several LLMs, and introduces retrievers designed around the resulting diagnostics. The generator controls pattern depth and other graph properties, allowing evidence availability to be measured.

Critical strengths
  1. Recording the generating evidence enables direct retrieval evaluation. The separate interventions on evidence and demonstrations, followed by distractor tests, offer useful insights beyond end-task accuracy alone.

  2. The experiments cover multiple model families and retrieval strategies, include candidate-matched random baselines, and show substantial sensitivity to frequency and recency cues even with complete evidence.

Critical weaknesses
  1. The error partition is operational rather than a causal attribution: every incorrect answer with incomplete generating evidence is labeled a retrieval failure. These cases may also involve reasoning failures or alternative sufficient evidence. The binary labels conceal possible overlap between the two failure modes and should be interpreted with that limitation in mind.

  2. Generalization beyond the calibrated generator remains limited. Appendix B reports 28% self-loops in ICEWS14-Synth versus none in ICEWS14, and 60% of 1-hop query answers equal the query head. Calibration and rank correlation are useful checks, but it does not mean that the same error proportions or hop effects hold on real graphs.

Questions for authors

Do the main hop-dependent findings persist on a non-self-loop subset, with results stratified by pattern shape?

4 comments

Thank you for your review and criticism. We'll try to address your weaknesses (W) and question (Q).

Re: W1.

  • We agree that the error partition is operational rather than causal. We ran a new experiment to try to measure the attribution counter-factually to address this. For six realistic retrievers and all seven models, we took each retriever's own 25-fact context, injected the missing antecedents and the solved example, and dropped the retriever's lowest-ranked other facts to keep the budget constant. We reran the models on these contexts and compared with the clean ground-truth context (the evidence plus one solved example, OICL+ex). Each error of a realistic retriever then falls into one of three parts: (1) Coverage, repaired by the evidence and example inside the retriever's budget; (2) Precision, repaired only when the retriever's other facts are also removed (i.e., clean OICL+ex); (3) Residual, not repaired even by OICL+ex.

  • We chance-corrected each part, because the clean context has fewer candidates. Pooled over each model's retrievers (Hits@3, 95% bootstrap CIs +-.03 - .10), we get: | 3-hop errors (2-hop) | coverage | precision | residual | |---|---|---|---| | GPT-5.6-terra | .27 (.29) | .47 (.20) | .25 (.50) | | GPT-4.1 | .09 (.08) | .48 (.26) | .43 (.66) | | gpt-oss-20b | .17 (.16) | .38 (.26) | .45 (.58) | | Qwen3-32B | .10 (.15) | .20 (.19) | .71 (.65) | | Llama-3.1-8B, Qwen3-8B | ≤ .04 (≤ .08) | ≤ .02 (≤ .13) | ≥ .96 (≥ .81) |

  • The two failure modes overlap, as you pointed out. For the strongest models, a better context repairs 55 - 75% of the 3-hop errors, but mostly when the retriever's other facts are removed. Adding the missing evidence inside a realistic context helps much less. This links the error analysis to the frequency and recency cue reliance of our distractor experiments. For the 8B models almost nothing is repaired, because they fail even with the sufficient context.

  • The binary labels we had previously therefore overstated the retrieval share. In our revision we will report this more accurate three-way split per model and retriever.

Re: W2 and your question (~"Do hop findings persist without self-loops, by shape?")

  • At the query level, reflexive patterns generate 60, 59, 55% of the queries at 1-, 2-, 3-hops, and the answer equals the query head in 60, 40, 26%. We recomputed the main results on non-reflexive queries (n = 209, 201, 223 of 500) and on answer != head queries.

    • The main hop finding persists. With the ground-truth evidence and one solved example, strong models stay high across hop lengths. On non-reflexive queries, GPT-5.6-terra reaches Hits@1 1.00, .72, .75 and GPT-4.1 .96, .61, .58 (random .26, .17, .14).
    • By shape, the exceptions are bridges (.11 - .26) and bridging compositions (.38 - .53), which stay ambiguous even for a reasoner that knows the rule. Most other shapes reach .77 or more for terra, though several 3-hop shapes have fewer than 10 queries.
    • With entity history, all models are near the floor (Hits@1 <= .13) at every hop length on these subsets. The drop with hops on all queries mostly reflects the falling share of answer = head queries.
    • The same evidence statistic holds. Context lacks evidence for 84 - 92% of the 2-hop and 93 - 99% of the 3-hop errors on non-reflexive queries.
    • One of our statements needs to be qualified. On non-reflexive queries, evidence alone (without an example) is above chance for strong models (terra .47, .39 at 2-, 3-hops vs. random .32, .27).
  • Moving beyond the generator, we ran three new checks that show the same trends.

    • Background noise. We regenerated the 2- and 3-hop datasets with 50% and 75% unlabelled background facts, drawn with the patterns' entity and relation popularity. The best realistic full-evidence rate falls (2 hops .28, .22, .17 at 0, 50, 75% noise). The sufficient context is unaffected (GPT-4.1 .64, .60, .62). The share of errors with missing evidence stays at 84 - 96%.
    • ICEWS14-Synth by hop. On its own consequence queries, GPT-4.1 reaches .89, .53, .48 with the sufficient context and .43, .14, .19 with entity history. The context lacks evidence for 94 - 97% of the 2 - 3-hop errors.
    • Real ICEWS14. As weak evidence chains we use TLogic rule groundings (rules learned on the training split) that predict the true answer. Multi-fact chains reach the context in only 16 - 17% of cases, and they are missing for 93 - 96% of the errors of GPT-4.1 and Qwen3-32B on these queries.
  • We agree that equal error proportions on real graphs are not implied, and we will say so in our revised limitations, together with the self-loop difference (28% in ICEWS14-Synth, none in ICEWS14). A new appendix will report the composition and every main table on both subsets and by shape.

Thank you again for your response and positive review. In our answer to W1, coverage injected the evidence and the solved example together. Precision came from switching to the clean context, which also has fewer candidates. We have since run a stricter test (motivated by reviewer 7EQv's comments). It is in the revised PDF (new App. G.9, Tables 28 and 29). Numbers are Hits@1 (because Hits@k for higher k saturates on these experiments), with 95% bootstrap intervals within +-.05.

We made three interventions and applied them independently. For N-E, BFS-3 and MotifCase at 2- and 3- hops, and six models, we apply three interventions to each retriever's own context, in all eight combinations. We inject the missing evidence, add the solved example, and remove the retriever's other facts. Shapley values attribute the repaired errors to each intervention, averaged over all orders of applying them. We also rescore every context on matched candidates, so removing other facts never removes a candidate.

The overlap you suspected is real for the strongest models. On matched candidates, removing the other facts repairs 18–22% of the errors of GPT-5.6-terra, gpt-oss-20b and GPT-4.1, and 14–18% after chance correction. These models are misled by the facts around the evidence, not only by a larger candidate set. The solved example helps mainly once those facts are gone. It raises Hits@1 by .03–.11 inside the retriever's context and by .29–.52 in the clean context. For the 8B models, the distractor share falls to about zero after chance correction.

What remains with the sufficient context. We also gave the generating pattern as an explicit rule (placeholders, relations and lag windows) next to the evidence. GPT-5.6-terra then reaches .99 and 1.00 at 2- and 3-hops, gpt-oss-20b .98 and .98 and Qwen3-32B .99 and .97, against .76 and .77, .66 and .60 and .42 and .34 with one solved example. For these models, the remaining errors are mostly failures to identify the pattern from one example. The 8B models stay near chance even with the rule (.35–.47, random .31–.39), so their errors are failures to apply the pattern. The rule also makes GPT-5.6-terra and gpt-oss-20b robust to 50 cue-adversarial distractors (.82–.98, against .12–.26 without it).

In our revision, the new Sec. 5 (Q3) summarizes these results. Our abstract, introduction and conclusion now say specifically that models must identify and apply the patterns. And our attribution claim remains specific to the synthetic benchmark, where the causes are known by construction.

Area_Chair_UGcCOpenReview ↗

Please engage in the discussion with authors

Dear reviewer,

Please engage in the discussion with authors, thank you.

AC

Reviewer_6riYOpenReview ↗

Thanks for your response

Thanks for your response, which addresses my main concerns. I will maintain the original positive score.

Reviewer_7EQvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why LLMs fail at multi-hop forecasting on temporal knowledge graphs: because the model cannot compose the evidence, or because retrieval never supplied it. Real benchmarks cannot separate these, since the causal antecedents of an outcome are unknown. The authors therefore build a synthetic TKG generator that instantiates events from explicit 1–3 hop temporal patterns (12 shapes), calibrates it to ICEWS14 with Bayesian optimization, and stores each generated consequence together with the antecedent facts that produced it. That provenance record yields three measures — evidence recall (EvRec), full evidence present (FullEv), answer in context (AnsIn) — and lets every error be labelled a retrieval failure (FullEv = 0) or a reasoning failure (FullEv = 1). The authors give an analytical account of which pattern shapes each retrieval family can reach, and benchmark seven LLMs, RE-Net, CyGNet, TLogic and frequency/recency heuristics against about fifteen retrievers, three of them new (MotifCase, BalBundle, EvRank). They report that evidence and a worked example are each insufficient alone and jointly sufficient; that under realistic retrieval 82–98% of multi-hop errors are retrieval failures; and that cue-adversarial distractors cut 3-hop Hits@3 by 67–96% even with full evidence present. They conclude that multi-hop temporal reasoning is "primarily a retrieval problem". Validation combines calibration to ICEWS14 statistics, proofs of reachability, a 2×2 evidence/example design, distractor manipulations, context-budget sweeps, a second seeded sample, and a real-data check.

Critical strengths
  1. The question and a resource that makes it answerable. Recording the antecedent facts that generated each event inside a generator where hop length, lag spread, and entity/relation skew can be varied independently is what allows "did the retriever deliver the evidence?" to be measured directly instead of inferred from end-task accuracy. This is the part of the paper I expect others to reuse, and it is the reason the work is worth publishing in some form even though I cannot accept the current interpretation.
  2. The reachability analysis and the fact that its predictions survive. I checked every entry of Table 2 against the canonical shapes in Table 5 and the proofs of Props. 1–3 and Lemma 4/Cor. 5; all are correct. Empirically, TLogic recovers 0% of bridge, bridging-composition, and cyclic patterns but 79% of transitive closures, exactly as predicted, and 45 of the 49 cells marked unreachable are precisely zero, with the other four explained by background paths. Falsifiable, pre-registered predictions that hold are the strongest evidence in the paper and the main reason the contribution is real.
  3. Control and reporting discipline. A random-ranking baseline in every condition, entity labels permuted per prompt, scoring by label index, log-probability scores cross-checked against written ranked lists, a second seeded sample falling inside the first sample's CI in 151 of 155 conditions, a context-budget sweep, and Table 7 stating precisely what each re-implemented "-style" baseline keeps and omits. This is rarer than it should be, and it is why the paper's negative results are credible.
Critical weaknesses
  1. The error split is a priority rule, not a counterfactual, so "82–98% retrieval failures" overstates what better retrieval would fix. Every error with FullEv = 0 is counted as a retrieval failure regardless of whether the model would have succeeded given the evidence. The paper's own numbers bound the fixable share: (E − e_f)/E at 3 hops, with e_f from Fig. 3 and E from Table 11, runs from 14% for the two 8B models to 52% for GPT-5.6-terra. Those 8B models stay near chance even on OICL+ex, yet Fig. 3 attributes 82–98% of their multi-hop errors to retrieval, and the bound is optimistic because retrieved-evidence queries skew toward easier shapes. The abstract's central claim rests on this number.
  2. FullEv is neither necessary nor sufficient, so the retrieval/reasoning boundary is misplaced. OICL alone is at chance, and Definition 1 accordingly requires evidence and a solved example for sufficiency—yet §4.1 calls any FullEv = 1 error a reasoning failure, so many are really missing-example failures. Conversely, Table 8's "sufficient rule recalled" is 1.00 for premise re-derivation, where the consequence repeats an antecedent that entity history retrieves directly; that shape is 38 of 150 3-hop patterns, and its FullEv = 0 errors are labelled retrieval failures although the answer was identifiable. The 2×2 also fixes much of its own result: both arms are retrieval interventions, so reasoning ability varies only by model choice.
  3. The oracle condition is neither a ceiling nor a like-for-like comparison, and both headline numbers depend on it. OICL+ex shares every relation ID with the query's pattern, so a lookup suffices, and lookup depth barely grows with hops—consistent with the flat 2→3 hop trend. A frequency/recency ranker already reaches about 0.66 Hits@3 there against 0.48 for random. It is not an upper bound either: at 1 hop several conditions exceed it, and averaging in two 8B models (chance-corrected ≈0.01 and −0.11) leaves the gap-closure denominator near zero. Finally, "3.7× the Hits@1 of RE-Net" compares an oracle-informed ranking over a handful of candidates with full-entity ranking; under matched conditions, the order reverses (0.13 versus 0.21 at 3-hop Hits@1; every LLM at most 0.321 versus 0.368 and 0.384 at 3-hop Hits@3).
  4. Hop length is confounded with shape mix, self-loops, and answer = head. Reflexive patterns are 91/149, 88/150, and 81/150 of the three datasets, and in 674 of 1,128 1-hop queries the answer is the query head. The 3-hop set contains 81 reflexive and 38 premise-re-derivation patterns against only 22 of 150 multi-step shapes. These are pattern counts; the query-level composition the accuracies average over is never reported. The stated reason for keeping self-loops—preserving the ICEWS14-Synth calibration—does not apply to hop datasets that were never calibrated against ICEWS14.
  5. External validity is demonstrated on a dataset the main claims do not use. Fig. 7 validates ICEWS14-Synth, but the claims come from the hop datasets, calibrated only to "similar descriptive statistics" among themselves. ICEWS14-Synth departs from ICEWS14 by more than that suggests: 3,282 versus 7,128 entities, average degree 50.6 versus 25.5, relation Gini 0.46 versus 0.88, and 28% self-loops against none. The agreement statistics are inflated by level differences and non-independent pairs, and the calibration target included the frequency/recency heuristics that reappear in the validation, so that check is not independent. Only 8 of 735 3-hop ICEWS14-Synth test queries have head–answer distance 3.

Other issues: The 67–96% distractor drops are measured from clean OICL+ex, bundling cue reversal with the cost of fifty extra facts; against +50 random facts, the cue-specific drops are 44–80%, and the 96% extreme is a model already at chance. The paper should state whether adversarial facts are sampled or synthesized and whether any complete alternative valid pattern instance exists. The three new retrievers are modest (MotifCase 0.14→0.16; BalBundle has no gain; EvRank needs labels absent from real TKGs and is "Statistical" in Table 6 but "Learned" in Fig. 4), and the 34–39% gap closures are post-hoc maxima without CIs. The "-style" baselines are component rather than system comparisons, and equal fact budgets do not equalize token or latency cost. The candidate set varies with condition (random Hits@3 0.962 for OICL versus 0.174 for N-E), the metric switches between Hits@1 and Hits@3, scoring mixes log-probability with written lists (0.01 versus 0.51 on one cell), and Fig. 3 pools fifteen retrievers for four models but three to seven for others. Lemma 4 holds under uniform sampling and should not be generalized to all retrievers lacking pattern signals.

Questions for authors
  1. For queries that fail under a given retriever with FullEv = 0, if the missing antecedents are injected into that retriever's own 25-fact budget by dropping its lowest-ranked facts, what fraction of errors are repaired, per model and per retriever? Without this the 82–98% figure is a coverage statistic, not an attribution.
  2. §4.1 labels every FullEv = 1 error a reasoning failure, but Definition 1 requires evidence and a solved example for sufficiency, and OICL alone is at chance. How are FullEv = 1 errors with no example labelled, and how do the reported shares change when they are separated out?
  3. Does a symbolic relation-ID subgraph matcher solve OICL+ex, and what is accuracy when the worked example is isomorphic in shape but relation-disjoint from the query? This determines whether the oracle measures reasoning or lookup.
  4. Can you report RE-Net re-ranking the same oracle candidate set, or the LLM restricted to the full entity set, so that the comparison behind "3.7×" is like-for-like?
  5. What is the query-level (not pattern-level) shape composition, and do the hop-length trend and the error split survive recomputation excluding reflexive and answer = head queries?
7 comments
AuthorsOpenReview ↗

Response (part 1)

Thank you for your review, we'll try to address your questions (Q)/weaknesses (W) and other issues (OI).

Re: W1/Q1. What does better retrieval repair?

  • We ran the experiment you proposed ("missing antecedents are injected into that retriever's own 25-fact budget by dropping its lowest-ranked facts") for all 7 models and 6 retrievers (N-E, 2-hop neighborhood, BFS-3, RECIPE-style, HippoRAG-style, MotifCase). We took each retriever's own 25-fact context, injected missing antecedents, dropped the retriever's lowest-ranked non-evidence facts to keep the budget constant at 25, and reran the models.
  • At 3 hops, this repairs 12-35% (Hits@3) of the labelled retrieval failures for GPT-4.1, 23–42% for GPT-5.6-terra, 24–42% for Qwen3-32B and 16–33% for the 8B models, depending on the retriever. As a baseline, a random ranking over the same injected contexts would already repair 16-30%. So the 82-98% number is indeed an overstatement for being "fixed by better retrieval".
  • We next tried to attribute each error more precisely. We added an experimental setting where we inject evidence and a solved example and compare it to our clean OICL+ex context. Then, each error belongs to one of three categories: (1) Coverage, repaired by the evidence and example inside the retriever's budget; (2) Precision, repaired only when the retriever's other facts are also removed (i.e., clean OICL+ex); (3) Residual, not repaired even by OICL+ex. This clean context has fewer candidates, so we chance-correct each part against a random ranking over the same candidates. Pooled over each model's retriever's (Hits@3, bootstrap CIs +- 0.03-0.1): | 3-hop errors (2-hop) | coverage | precision | residual | |---|---|---|---| | GPT-5.6-terra | .27 (.29) | .47 (.20) | .25 (.50) | | GPT-4.1 | .09 (.08) | .48 (.26) | .43 (.66) | | gpt-oss-20b | .17 (.16) | .38 (.26) | .45 (.58) | | Qwen3-32B | .10 (.15) | .20 (.19) | .71 (.65) | | Llama-3.1-8B, Qwen3-8B | ≤ .04 (≤ .08) | ≤ .02 (≤ .13) | ≥ .96 (≥ .81) |

Therefore, the effect depends on the model. For the strongest models, better context repairs 55-75% of 3-hop errors, mostly by removing distracting facts rather than by adding missing evidence. For Qwen3-32B and the 8B models, errors persist even with sufficient context. We will revise the paper's claims such that the "82-98%" figure is only claimed as the share of errors whose context lacks necessary evidence. And we will use this new 3-way split as a more concrete, defensible headline.

Re: W2/Q2. FullEv being neither necessary nor sufficient.

  • "Not sufficient". We agree. Under realistic retrieval, almost no FullEv=1 error has a solved instance of the query's pattern in its context. At 2-3 hops, errors with both are at most 1.2% of all errors. When both are present, GPT-5.6-terra, for example, misses the top 3 on 8% of 2-hop queries and on 0 of 10 at 3-hops. We will instead rely on the "Residual" column of our previous response to measure sufficiency directly.
  • "Not necessary". We added a "relaxed" flag to our code. An error would count as a retrieval failure only if the context lacks an antecedent on the head-to-answer path of the pattern instance, and answer=head queries count as having the evidence. This lower bound givces 58-79% at 2-hops and 66-79% at 3-hops and 74-83% and 75-86% on non-reflexive queries.
  • "Both 2x2s are retrieval interventions". This is fair. We newly added 2 variations to keep evidence fixed. First, we renamed examples' relations to relations absent from the query's pattern. GPT-5.6-terra still reaches .45 Hits@1 at 3-hops (random baseline at .16), so it transfers pattern structure. Second, we gave 2 and 4 solved examples. Qwen3-32B improves from .34 to .44 to .51 at 3-hops (random .16, .11, .07), while the 8B scale models stay at chance performance. We will call the example arm "pattern identification" instead of "reasoning" to better reflect the critique.
AuthorsOpenReview ↗

Response (part 2)

Re: W3/Q3. Is OICL+ex a lookup?

  • Yes, a symbolic relation-ID matcher solves it. The matcher induces a rule from any context fact with the query relation and grounds that rule from the query head. It reaches Hits@1 .98, .86, .9 at 1-, 2-, and 3-hops, above GPT-5.6-terra (.96, .76, .77). We agree that OICL+ex tests in-context pattern identification and application, and we will correct our phrasing in the paper to say so.
  • We ran 2 new experiments to show that OICL+ex still separates models:
    • Isomorphic, relation-disjoint example: (Same experiment we referred to in our response to your W2/Q2) We took the same solved example and rename its relations to relations absent from the query's pattern, keeping its entities, times and structure. The matcher falls to chance (.15 at 3 hops, random .16). GPT-5.6-terra keeps .76, .55, .45; GPT-4.1 .53, .36, .34 and gpt-oss-20b .65, .51, .37 at 1-, 2-, 3-hops (random .28, .20, .16). OICL alone gives terra .29, .24 at 2-, 3-hops. So the example's structure alone adds ~.2 for terra, and relation identity adds a further .15 to .33 across these models.
    • Cue reversal. You're right that a recency ranker is strong on clean OICL+ex, because the last evidence fact often touches the answer. With 50 cue-adversarial facts added, the matcher still reaches .50, .44 at 2-, 3-hops, while terra falls to .26, .12 and GPT-4.1 to .14, .05.
  • We also agree that OICL+ex is not an upper bound in general. Some 1-hop conditions exceed it because they have fewer candidates, which is why every cell reports a random baseline. We will call it "sufficient context" in future, not a ceiling.

Re: W3/Q4. "3.7x" and gap closure.

  • We restricted RE-Net's scores to exactly the candidates our LLMs see. On the OICL+ex candidates, RE-Net reaches Hits@1 .47, .40, .21 at 1-, 2-, 3-hops, against terra's .96, .76, .77. At 3 hops the paired difference is .56 (CI [.48, .64]), i.e. 3.6×. We will phrase the claim as conditional on the sufficient context. You are right that under realistic retrieval RE-Net wins on all queries (.21 vs .13 Hits@1 at 3-hops). And this reverses on non-reflexive queries. There RE-Net reaches .03 Hits@1 and .23 Hits@3 over all entities, while terra with MotifCase reaches .21 and .38.
  • For the gap closure, the 4-LLM-mean includes two 8B models with near-zero denominators. Per model, with the MotifCase retriever and 95% bootstrap CIs, the gap closed at 3-hops is 23% (CI [9, 36]) for GPT-4.1, 27% (CI [13, 41]) for terra, 33% (CI [11, 60]) for gpt-oss-20b and 58% (CI [36, 94]) for Qwen3-32B. Our revision will report these numbers and drops "at most 39%".

Re: W4/Q5. Hop length and composition.

  • At the query level, reflexive patterns generate 60, 59, 55% of the queries at 1-, 2-, 3-hops, and the answer is the query head in 60, 40, 26%. We recomputed the main results on non-reflexive queries (n = 209, 201, 223 of 500) and on answer != head queries.
    • Your concern holds for the N-E hop trend. On these subsets every model is near the floor with N-E (Hits@1 <= .13) at every hop length. The drop with hops on all queries mostly reflects the falling share of answer = head queries. On the rest, head-anchored retrieval cannot reach the answer at any hop length, as Table 2 predicts.
    • The OICL+ex result holds. On non-reflexive queries, terra reaches 1.00, .72, .75; GPT-4.1 .95, .61, .58 and gpt-oss-20b .86, .65, .66 (random baseline .26, .17, .14). By shape, the exceptions are bridges (.11 - .26) and bridging compositions (.38 - .53), which stay ambiguous even for a reasoner that knows the rule.
    • The evidence statistic holds. The context lacks the evidence for 84 - 92% of the 2-hop and 93 - 99% of the 3-hop errors on non-reflexive queries.
    • We found one other statement that we need to qualify. On non-reflexive queries, OICL alone is above chance for strong models (terra .47, .39 at 2-, 3-hops vs. random .32, .27). "OICL alone leaves every model at chance" holds for the full mix only.
  • On self-loops, we agree that the calibration argument applies to ICEWS14-Synth, not to the hop datasets. As we wrote in our Appendix B, ICEWS14-Synth needed self-loops to match ICEWS14. And turning self-loops off only for the hop dataset would add a second difference between the datasets behind the claims and the one validated against ICEWS14. In such a situation, we believe we made the correct choice to include self-loops and fully disclose them. In future work using our generator, researchers may or may not choose to include self-loops, as allowed by our generator framework. Our subset results show that the conclusions do not depend on them. We will also add a new appendix reporting the composition and every main table on both subsets and by shape.
AuthorsOpenReview ↗

Response (part 3)

Re: W5. External validity.

  • We agree that the pooled statistic mixed model-level and method-level agreement and included the calibration targets. Excluding the heuristic baselines, LLM scores on ICEWS14 and ICEWS14-Synth agree in level (chance-corrected Kendall tau = .58, r = .78, mean |\Delta| = .05 over 32 model-method pairs). After removing each model's mean, method-level agreement is positive but moderate (tau = .30, p = .015, cluster-bootstrap CI over methods [.07, .44]). We will report these numbers instead of the pooled tau. The differences you list (3,282 vs 7,128 entities, degree, relation Gini, 28% self-loops) are real, and we will state them in Table 1 and the limitations. The 8 of 735 distance-3 queries (App. B) show that background paths shorten head-answer distances, which is why distance only lower-bounds hop length.
  • Our claims will also no longer rest on the hop datasets alone.
    • ICEWS14-Synth itself. On its own consequence queries, grouped by hop, we see that same trends as on the hop datasets. GPT-4.1 reaches Hits@1 .89, .53, .48 with OICL+ex and .43, .14, .19 with N-E. Realistic retrievers include the full evidence in at most 15% of 2-hop and 8% of 3-hop contexts.
    • Background noise. We regenerated the 2- and 3-hop datasets with 0%, 50% and 75% unlabelled background facts, drawn with the patterns' entity and relation popularity. As noise rises, the best realistic full-evidence rate falls (2-hops .28, .22, .17). OICL+ex is unaffected (GPT-4.1 .64, .60, .62). The share of errors with missing evidence stays at 84-96%.
    • Real ICEWS14. As weak evidence we use the body of the most confident TLogic rule grounding (rules learned on the training split) that predicts the true answer. Multi-fact chains reach the context in only 16-17% of cases, and they are missing for 93-96% of the errors of GPT-4.1 and Qwen3-32B on these queries.

Re: Other issues.

  • Re: Distractor drops. Against +50 random facts, the cue-adversarial drop at 3 hops is 51% (terra), 44% (gpt-oss), 78% (GPT-4.1), 64% (Qwen3-32B) and 67% (Llama-3.1-8B), the last two with exact scoring. We will report both baselines and drop the 96% extreme. This rules out a candidate count explanation. The adversarial contexts have fewer candidates than the +50 random ones (random Hits@3 .086 vs .051), and the symbolic matcher still solves 44% of them.
  • Re: Sampled/synthesized question. Distractors are real history facts sampled before the query time, never synthesized. A complete, lag-valid alternative instance of the query's own pattern appears in 4.6% of the 3-hop adversarial contexts, against 3.6% of clean OICL+ex contexts (13.4% vs 7.6% at 2 hops). We will state this in App. E.
  • Re: New retrievers. We agree they are modest and will narrow the claim. EvRank is "learned" because it is trained on evidence labels. Table 6 mislabels it, and we will fix that. We present it as a probe of what evidence supervision can do, not as a deployable method.
  • Re: Component vs. system comparisons/budgets. Agreed. Table 7 lists what each re-implementation keeps. We will add token counts per condition, so that equal fact budgets are not read as equal cost.
  • Re: Candidate sets, metrics, scoring. Each condition reports its random baseline over the same candidates (Table 12). We use Hits@1 only where Hits@3 saturates, and we will label the figures to make that more clear. Cross-model claims use the single-scoring parity table (Table 13). The 0.01 vs 0.51 cell is Llama-3.1-8B, which often answers with code in list mode, so we rely on its exact scores (App. E).
  • Re: Unequal retriever pools in Fig. 3. On the three retrievers every model ran (N-E, N-P, 2-hop neighborhood), the share of errors with missing evidence is 89–93% at 2 hops and 96–98% at 3 hops for all seven models.
  • Re: Lemma 4. Agreed. It holds for uniform selection, and we will restrict the sentence in Sec. 4.2 accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revision

We thank our reviewers for their detailed and constructive reviews. We have just uploaded a revised PDF. Every change since the submission is in blue. All new experiments use the paper's evaluation code unchanged. Our new analyses are mostly collected in App. G (ten subsections, 16 new tables). Their code, result summaries and new datasets have also been included in our released anonymized repository (folder rebuttal/).

Here we summarize the changes we've made in response to the reviewer comments:

Re: Error attribution (all reviewers). The submission labelled 82–98% of multi-hop errors as retrieval failures. That figure is now reported only as a coverage statistic, and the attribution comes from interventions on each retriever's own context.

  • A counterfactual analysis splits each error into coverage, precision and residual, chance-corrected (new Table 4, App. G.3).
  • A full factorial analysis applies evidence injection, the solved example and distractor removal independently. It scores every context on matched candidates and attributes repairs with Shapley values (App. G.9).
  • For the strongest models, removing facts around the evidence corrects about a fifth of the errors even on matched candidates. For the 8B models, most errors persist with any context. The abstract, Sec. 5 and the conclusion state this per-model result.

Re: What the sufficient context measures (reviewers 7EQv, 1eQg). We now describe OICL+ex as a setting to test pattern identification and application, not general multi-hop reasoning.

  • A symbolic relation-ID matcher solves it. With the example's relations renamed, the matcher falls to chance while the strongest models keep much of their accuracy (App. G.5).
  • Given the pattern as an explicit rule, Qwen3-32B and the frontier models reach Hits@1 of .94–1.00. The 8B models stay at chance and often copy the example's answer (App. G.5, G.9).

Re: Query composition and self-loops (reviewers 7EQv, 6riY). The main results table is repeated on non-reflexive queries and on answer != head queries, and OICL+ex is broken down by shape (App. G.1). The hop trend of entity history reflects the falling share of answer = head queries, and Sec. 5 now says so. The OICL+ex findings hold on these subsets.

Re: External validity (all reviewers).

  • Background noise at 50% and 75% (App. G.8).
  • ICEWS14-Synth's own queries by hop (App. G.8).
  • Weak evidence chains on real ICEWS14 from TLogic groundings, presented as consistency evidence, not as validation of the error proportions on real data (App. G.8).
  • Corrected validity statistics (level tau = .58, within-model tau = .30) in Sec. 3.2 and App. B, and the remaining differences in the limitations.

Re: Fair comparisons (reviewers 7EQv, 1eQg).

  • We experiment with graph-based method RE-Net re-ranking exactly the LLM's candidates. At 3-hops GPT-5.6-terra reaches 3.6x its Hits@1, a paired difference of .56 [.48, .64] (App. G.4).
  • All new attributions are chance-corrected against a random ranking over the same candidates.
  • Gap closure is reported per model with bootstrap intervals instead of a four-model mean (Sec. 5, App. G.7).
  • Distractor drops are reported relative to 50 random facts (App. G.6), and prompt lengths in tokens (App. G.10).

Re: Corrections. Lemma 4 is restricted to uniform selection. EvRank is labelled as trained on evidence. Distractors are stated to be sampled real facts. The "at most 39%" claim has been removed.

We believe the revision addresses every concern raised, either with a new experiment or with a narrowed claim, and that the paper is ready for publication. The remaining limitations are stated in the conclusion. Patterns span up to three hops, and error proportions on real graphs cannot be validated without annotated causes (but we believe such annotations are beyond the scope of this work, due to the expense and domain-expertise that would be required to generate them to a sufficiently high standard).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.