- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FaiTH, a training-free framework for temporal knowledge graph question answering (TKGQA). It follows a four-stage compile-ground-execute-reason pipeline and guarantees at most two LLM calls per question. The authors evaluate FaiTH on MultiTQ and two TimelineKGQA benchmarks across 11 LLM backbone configurations.
Critical strengths
S1. The paper clearly identifies and formalizes two key failure modes in TKGQA, namely unfaithful evidence and hallucination accumulation caused by multiple LLM calls, and the proposed pipeline provides targeted design responses to both. S2. The two-call LLM budget and the deterministic shortcut balance faithfulness and inference cost, making the method practically useful. S3. The experiments cover multiple benchmarks, diverse backbone configurations, and ablation settings, providing relatively strong empirical support for the effectiveness of the method.
Critical weaknesses
W1. The related work is largely enumerative, especially the Embedding and GNN methods section, and fails to clearly delineate the essential differences between this work and prior methods. W2. The explanations of key equations are insufficient. In particular, Eq. (6) is obscure in its notation, matching rules, and confidence cascade, which hurts readability. W3. The proofs of Theorem 1 and Theorem 2 mainly restate the set-membership conditions in Eqs. (8)–(12). The theoretical contribution is limited, and calling them theorems seems somewhat overstated. W4. Although the method significantly reduces the number of LLM calls, it does not report inference time, token consumption, or end-to-end latency, making it difficult to judge whether its efficiency truly improves over existing methods.
Questions for authors
D1. Provide a more detailed description of the related work, emphasizing the differences between the existing work and the work presented in this paper.
D2. Please explain why the boundary tie-preserving cap is the “only” permitted source of information loss under the completeness clause of Definition 3. Also explain how ties among boundary candidates are broken, whether the tie-breaking is random, and whether it affects reproducibility.
D3. The description of the formula needs to be improved for better comprehension.
D4. It is necessary to add experimental descriptions of the consumption (or inference time) of the entire model, and clarify the efficiency difference between the method proposed in this paper and existing methods.
D5. The paper emphasizes that multiple LLM calls lead to hallucination accumulation. It would be helpful to provide concrete failure cases alongside corresponding FaiTH examples to show how this problem is actually mitigated by the pipeline.
D6. If Theorem 1 and Theorem 2 are retained, please clarify the source of their non-triviality and discuss the limitations imposed by Assumption 1. Otherwise, they could be downgraded to propositions or consistency claims.
2 comments
Dear Reviewer 1M3N,
Thank you again for your constructive comments. We would like to gently mention that our response is posted above, and that the revised PDF, with changes marked in blue, implements your six requests D1 to D6. These include the reworked related work and Eq. (6), the report on cost and latency, and the concrete failure cases of multi-call pipelines. If any point needs more explanation, we would be glad to provide it while the discussion is open.
Thank you very much for your time.
Thank you for the constructive comments and for recognizing the two failure modes we formalize and the practical value of the two-call budget. The revised PDF (changes in blue) implements all six requests D1 to D6.
W1/D1. Related work. Sec. 2 now closes each family with its difference from FaiTH. Embedding and GNN methods rank the answer vocabulary with a trained scorer. Retrieve-then-reason pipelines and agents let the LLM apply temporal constraints over top-k retrieval. FaiTH executes these constraints as exact set operations before any LLM reads the evidence, and it needs no training. A new paragraph covers semantic parsing (Prog-TQA, TECQA), and App. A adds a side-by-side table (Table 5).
W2/D3. Eq. (6). We rewrote Eq. (6) and its explanation. Each rule returns the KG symbols a mention matches, with a match score that scales the rule weight. Rules are tried from the most to the least restrictive, and the first rule that fires decides. For example, "Governor of Thailand" equals no entity, and the Role (Country) rule maps it to "Governor (Thailand)". Table 7 lists the seven rules: on MultiTQ the three exact rules decide 97.58% of anchor mentions.
W3/D6. Theorems. We agree and renamed them Propositions, and App. B.6 now states their scope. They are short by design, because the operators are written to coincide with Definitions 1-2. Their value is that faithfulness becomes a set property that can be checked fact by fact and holds for every backbone. It is also testable: an exhaustive check over 38,182 questions with one executed relation finds no consistent fact missing (App. C.17). Their limits are their premises: a wrongly grounded mention or a wrongly compiled constraint makes the engine execute a wrong specification faithfully. Assumption 2 therefore makes correct compilation explicit. App. C.9 attributes 4.05% of errors to failed groundings, and 97.11% of specifications pass all checks of new App. C.15.
W4/D4. Time, tokens and latency. App. C.3 now reports them. We priced and timed FaiTH and RTQA on the same API model (deepseek-flash), replaying RTQA's own prompts. RTQA's calls and tokens come from our DS-V3.2 re-run, where its released code makes one decomposition call plus up to three per node (its paper reports 3.96, still 2.9x FaiTH's 1.35).
Table R1. Cost per MultiTQ question (off-peak list price, FaiTH latency on 300 stratified questions)
FaiTH's compile-prompt prefix is shared by all questions, so 95% of its prompt tokens hit the prefix cache. Stages 2-3 take 5.56 ms on average. For reference, TECQA issues 4 LLM calls per question.
D2. The cap, ties and reproducibility. Your question led us to remove this exception. Definition 3 now states soundness and completeness on the complete evidence set, and Eq. (12) has no cap. The bound is an evidence budget for Stage 4 (per benchmark: App. B.3). On MultiTQ it only limits how many facts the optional reason call reads: candidates, tie groups and every shortcut answer use the complete set. This corrects the description, and no reported number changes. On ties: most boundaries hold a single candidate (83.17% of ordering-directed MultiTQ entity questions). The reported runs ordered a tie group by the full timestamp and broke equal timestamps arbitrarily. Two arbitrary orders change the number of correct shortcut answers by 24 (0.044 points), a small effect on score reproducibility. The revision breaks equal timestamps by subject, relation and object, so tie-breaking now involves no randomness.
D5. Concrete multi-call failures. New App. C.19 pairs FaiTH (DS-V4-Flash) with our RTQA re-run (DS-V3.2) on MultiTQ under the same scoring code. On the 20,119 questions RTQA decomposes, FaiTH scores 70.86 vs 42.01, with 1.24 vs 9.50 LLM calls. When RTQA's first sub-answer comes from its closed-book branch (no retrieved facts), its final Hits@1 drops from 44.25 to 12.55. One of two appendix cases from a random sample of ten questions RTQA misses and FaiTH answers:
Table 4 agrees: forcing a reason call on every question lowers MultiTQ from 0.826 to 0.789 and TL-ICEWS from 0.696 to 0.456.
We welcome further questions.