- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a synthetic temporal knowledge graph generator with per-event ground-truth evidence. Event histories are generated from twelve 1–3 hop temporal motifs and calibrated to ICEWS14 statistics with Bayesian optimization, and each event is stored with its causal source. Prediction errors can then be decomposed cleanly into retrieval failures and reasoning failures, and the paper provides an evidence reachability analysis that requires no model runs. The evaluation covers seven LLMs plus GNN/rule predictors under several retrieval strategies: with ground-truth evidence and one solved example of the same pattern (OICL+ex), the strongest model reaches 3.7 times RE-Net on 3-hop; under realistic retrieval, however, 82% to 98% of 2- and 3-hop errors trace to missing evidence, the best retriever closes only 39% of the gap, and frequency/recency shortcuts remain.
Critical strengths
- Relative to iTemporal and Test of Time, this is the first generator to record a causal source for every event and to quantify how well each retrieval strategy covers the twelve motif shapes, making the “retrieval or reasoning” question attributable and reproducible for the first time. Perturbed datasets and a second-seed robustness check add to the credibility of the conclusions.
- The diagnostic experiments are well designed: bootstrap confidence intervals, random-order baselines, and randomized, near-miss, frequency-removed, and dual-shortcut adversarial tests together make a systematic case for shortcut dependence. Candidate-anchored retrieval and EvRank give the community concrete directions to follow.
Critical weaknesses
- The headline conclusion does not generalize far beyond the synthetic setup. Every event is generated from a pattern, entities and relations are numeric IDs, and there is no background noise; calibration is only moderate, so the claim that “multi-hop reasoning is mainly a retrieval problem (82% to 98%)” holds only under the synthetic distribution. The OICL+ex upper bound also includes a solved example of the same pattern, which is almost unavailable in a real system, and the title’s either/or framing is easy to misread as retrieval being solvable.
- The comparison has fairness and statistical-power problems. There are only 603 queries at 3 hops, and the LLM runs use 200 to 500 seed queries, so the point estimates have wide intervals. GNNs score over all entities while LLMs score over an oracle context with only 1.9 to 3.5 entities on average, so the candidate sets are not symmetric and “3.7 times” is not an apples-to-apples comparison. Treating every FullEv=1 case as a reasoning failure also conflates “no pattern hint” with “cannot reason.”
Questions for authors
- Could you build evidence chains on a real TKG, even at small scale, using manual annotation or weak supervision, to test whether the ranked finding that 82% to 98% of errors come from retrieval still holds?
- Could you re-compare GNNs and LLMs under a unified candidate set? Does the random baseline being as high as 0.55 on 1-hop suggest the candidate set is too small and the task too easy?
- Can error attribution be split into “evidence present but no pattern hint” and “evidence present with a hint but composition fails”? What fractions do these two categories occupy in the Q1 intervention?
- 8B models remain near random under OICL+ex: is this a context-length issue, instruction following, or pattern-composition ability? Please provide a controlled analysis.
3 comments
Response (part 2)
Re: Q1. Evidence chains on a real TKG.
- We did this with weak supervision on ICEWS14. For each test query, the weak evidence is the body of the most confident grounding, from the query head, of a TLogic rule learned on the training split that predicts the true answer within 29 days. 286 of the 500 evaluated queries have such a chain (202 single facts, 84 multi-fact chains). For the multi-fact chains, the paper's eight label-free retrievers include the full chain in only 16 - 17% of contexts. The chain is missing for 96% (GPT-4.1) and 93% (Qwen3-32B) of the errors on these queries. When it is present, the models miss the top-3 only 10% and 19% of the time.
- We also ran the counterfactual. Injecting the chain into the retriever's own 25-fact budget repairs only 8 - 13% of GPT-4.1's errors beyond chance. The chain alone in a clean context repairs a further 21 - 22%. This mirrors our synthetic result that missing evidence is necessary but, inside a realistic context, far from sufficient. The chains are rule-derived, so they are a proxy for causes.
Re: Q2. A unified candidate set and random baseline performance of .55 on 1-hop.
- The .55 random Hits@1 is for OICL, the evidence alone, which names 1.9 entities on average at 1-hop. With the solved example (OICL+ex) it is .28. With entity history, contexts have 14 - 18 candidates and random Hits@1 is .06 - .10.
- Re-ranking the same OICL+ex candidates, RE-Net reaches .47, .40, .21 at 1-, 2-, 3-hops, against GPT-5.6-terra's .96. .76, .77. The task is not easy on these candidate sets. At 3-hops RE-Net gets .21 and the 8B models stay at chance.
Re: Q3. Evidence without a hit vs failed composition.
-
In the 2×2 of Table 3, OICL is evidence without a hint and OICL+ex is evidence with a hint. The hint adds +.29 to +.53 Hits@1 at 2 - 3 hops for GPT-4.1, gpt-oss and terra, and nothing for the 8B models. Under realistic retrieval, errors with the evidence but no hint are 2 - 17% of all errors at 2- 3 hops, and errors with both are at most 1.2%.
-
We also measured the attribution counterfactually. We injected the evidence and a solved example into each realistic retriever's own 25-fact context, and compared with the clean OICL+ex context. For chance-corrected shares of 3-hop errors (Hits@3), we get the following: | model | repaired inside the budget | repaired only when clean | persists | |---|---|---|---| | GPT-5.6-terra | .27 | .47 | .25 | | GPT-4.1 | .09 | .48 | .43 | | gpt-oss-20b | .17 | .38 | .45 | | Qwen3-32B | .10 | .20 | .71 | | Llama-3.1-8B, Qwen3-8B | ≤ .04 | ≤ .02 | ≥ .96 |
-
The last column is explicitly the category you asked about. There, composition fails even with the evidence and a hint in a clean context. It is 25 - 45% of the errors for the strongest models and nearly all errors of the 8B models.
Re: Q4. Why the 8B models stay near chance on OICL+ex.
- Context length is not likely to be the cause. OICL+ex contexts hold 3–7 facts.
- Instruction following is not the cause either. Open models are scored exactly by the log-probability of each candidate's label, so there is no output parsing.
- Re: what they pick. The 8B models mostly copy the solved example's own answer entity instead of mapping it onto the query. Qwen3-8B does so in 52, 29, 21% of queries at 1-, 2-, 3-hops, and Llama-3.1-8B in 38, 24, 24%. GPT-4.1 does so in 1 - 4% and terra in 0 - 12%. Without the example, no model picks that entity (<=2%).
- More examples (2 and 4) do not help them. Qwen3-8B goes .22, .14, .13 at 3-hops while chance falls from .16 to .07, and Llama-3.1-8B stays at .18. They do help Qwen3-32B (.34, .44, .51).
- A relation-renamed example leaves them at chance as well.
We believe the 8B failure is therefore most attributable to analogical mapping, the step that applies a solved example to the query. Our OICL vs. OICL+ex contrast isolates this step, and it is why their errors persist in the table above.
Please engage in the discussion with authors
Please engage in the discussion with authors, thank you.
AC
Response (part 1)
Thank you for your review and criticisms. We tried to address the weaknesses (W) you pointed out and your questions (Q) below:
Re: W1. Generalization beyond our synthetic setup.
Re: W2. Fairness and statistical power.
FullEv = 1 conflates "no hint" with "cannot reason". The 2×2 of Table 3 already separates the two directly. OICL is evidence without a hint, and OICL+ex is evidence with a hint (one solved example). Under realistic retrieval, almost no FullEv = 1 error has a hint in its context. At 2 - 3 hops, errors with both the evidence and a hint are at most 1.2% of all errors. The revision labels FullEv = 1 errors without a hint separately.