- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
G-Speculator targets two GraphRAG failure modes that the authors attribute to low-quality graphs: redundant retrieval results (global traversal over a noisy graph returns many irrelevant passages) and biased retrieval paths (a static graph cannot adapt when relations are missing). It indexes the corpus with spaCy only. Each sentence is split at named entities into an entity–relation sequence, and the non-entity spans become hyperedges that connect all entities of that sentence (HypoKG). At query time, an LLM parses the question into triples with placeholders. Clues (entity, relation span) are scored by entity-linking confidence × IDF plus relation similarity, and the bridge entity is "speculated" as the nearest entity on the opposite side of the relation span in the sentence sequence. A CoT step then judges whether the evidence is sufficient and either answers or issues a new sub-query, with dense retrieval as a fallback when retrieval stalls. The method reports the best average F1 on HotpotQA, 2Wiki, MuSiQue, and ScaleQA, with fewer retrieved passages and zero LLM tokens for indexing.
Replacing exhaustive traversal with cheap local guessing plus LLM correction is a practical idea, and the results are strong. However, the efficiency claim covers only the retriever, while the method makes more LLM calls than the strongest baseline. The speculation rule is a positional heuristic that relies on undocumented components (notably prepending passage-title entities to every sentence) and has no handling of pronouns or ambiguous relations. HypoKG is never shown to fix the reachability problem used as motivation. The delta over KG-Translator is modest, a closely related method (T2RAG) is missing, and the evaluation is limited to Wikipedia-based multi-hop QA.
Critical strengths
- The preliminary study is a useful diagnostic. It quantifies evidence reachability on OpenIE-built KGs (second-evidence reachability drops to 55.0–66.8%) and illustrates the two failure cases concretely.
- Indexing is cheap and simple: spaCy only, zero LLM tokens, and about 226 s per 5k passages, vs. 4.30M/1.28M input/output tokens for OpenIE-based HippoRAG. Graph retrieval itself is fast (0.56 s retriever time, +0.13 s when the corpus grows 3.5×).
- Results are strong and the retrieved context is concise: best average F1 on all four datasets (overall 74.22 vs. 68.45 for KG-Translator), with about 2.9 retrieved passages and a Precision of 82.54 vs. 63.17 on 2Wiki.
- The appendix is thorough (hyperparameters, LLM backbones, entity linkers, case and error studies, component-wise latency), and code is released.
Critical weaknesses
- Efficiency is overclaimed.
- Table 3 reports indexing cost and retriever runtime, but online cost is dominated by LLM calls. G-Speculator makes about 3.91 LLM calls per query (R 1.72 + I 1.98 + A 0.21). That is about 60% more than KG-Translator (2.45) and lower only than HippoRAG+IRCoT (5.34).
- Table 5 shows that each round is dominated by query-graph construction (≈1.46 s) and CoT analysis (≈2.72 s), while clue exploration and speculation take ≈0.11 s. End-to-end per-query latency and online input/output tokens are not reported for any method.
- According to Algorithm 2, every round parses the query graph and then runs CoT, so R (1.72) and I (1.98) should be equal. Please explain the difference.
- The generative entity linker (Llama-3.2-1B trained on KG-Translator's data and run on a GPU) is not included in the cost comparison.
- The speculation rule is a positional heuristic, and key components are undocumented.
- The stated assumption is that subject and object are directly adjacent to the predicate, but neither Eq. 4 nor the code enforces adjacency. A clue pairs any entity with any relation span in the same sentence (ϕ(r) = E_s). Two examples from the paper:
- Case 1: "Aerodynamik" is separated from "from their tenth studio album" by two other elements.
- Case 2: "1173" lies between "Richardis of Bavaria" and "in".
- Pronouns are not handled at indexing time. They are not named entities, so they are absorbed into relation spans ("It is endemic to", "He married"). The only coreference step is an instruction in the CoT prompt, applied after retrieval.
- Instead, the released code (
modules/hypokg.py) prepends the passage-title entities to every sentence sequence (sid2seq[sid] = title_ents + sent_sequence) and attaches them to all hyperedges of every sentence. This is why Cases 2 and 4 start with "Otto I" and "Emu" before "He married" and "It is endemic to".- In effect, it is an implicit rule that a pronoun refers to the title entity, and it is not described in Sec. 3.1, Algorithm 1, or Appendix F.1.
- It attaches the wrong subject whenever a pronoun refers to a non-title entity or a sentence has a different explicit subject. The wrong bridge "Jülich" in Case 2 is itself a prepended title entity: p8 is "William IV, Count of Jülich", and "Jülich" does not occur in that sentence.
- It assumes titled, entity-centric passages.
- When a title contains several entities, their order comes from a set, so positions are not deterministic across runs.
- An ablation of this component is needed.
- Relation spans are untyped, and function words are not filtered.
- A span such as "in" gets the same similarity score whether it is temporal or locative.
- Identical spans within a sentence share one relation id, so speculation always uses the first occurrence. For example, in "(15 April 1452 in Vinci – 2 May 1519 in Amboise)", the second "in" can never yield Amboise.
- Entity types are only used as a binary named vs. value-like filter on non-final hops. The type implied by the query predicate (e.g., "born location is") is not used.
- The stated assumption is that subject and object are directly adjacent to the predicate, but neither Eq. 4 nor the code enforces adjacency. A clue pairs any entity with any relation span in the same sentence (ϕ(r) = E_s). Two examples from the paper:
- HypoKG is not shown to address the motivating problem.
- The reachability analysis (Fig. 2a) is computed on OpenIE KGs, never on HypoKG.
- HypoKG's hyperedges are intra-sentence, so links across sentences still depend on exact entity-string matches (e.g., "Richardis" vs. "Richardis of Bavaria" in Case 2).
- As it stands, unreachable evidence appears to be recovered by LLM query rewriting and the dense fallback rather than by the graph. Please report reachability on HypoKG under the same protocol.
- Novelty is limited relative to prior work, and closely related methods are missing.
- Much of the pipeline follows KG-Translator: spaCy-based hypergraph construction, the same generative entity linker and its training data, the same test splits, and ScaleQA. The main additions are keeping word order, the positional speculation rule.
- Missing comparison with T2RAG (Table 1, multi-step methods, p. 7). T2RAG also decomposes the question into triplets with placeholders and resolves them iteratively. It explicitly targets the efficiency of multi-round RAG and reports better results than IRCoT. This is very close to the query-graph construction and sequential placeholder filling used here, so it should be discussed and compared in both accuracy and cost.
- Missing comparison with HyperGraphRAG. HypoKG is a hypergraph, yet the paper neither discusses nor compares with HyperGraphRAG, which also uses hyperedges over multiple entities (n-ary relational facts) for RAG. The paper should explain how HypoKG differs, since HyperGraphRAG's hyperedges are n-ary facts extracted by an LLM while HypoKG's are spaCy sentence-level spans, and how their retrieval strategies differ. It should also compare the two in accuracy and cost.
- The evaluation scope is narrow.
- All four datasets are Wikipedia-based multi-hop QA (e.g., p. 24), and ScaleQA merges the other three. Their passages are short, titled, and entity-centric, which favors spaCy NER and the title-entity heuristic.
- HotpotQA, 2Wiki, and MuSiQue alone are no longer sufficient for current RAG research. Please evaluate on GraphRAG-Bench (Xiao et al., 2025), which tests domain-specific reasoning over 20 textbooks across 16 disciplines.
- The GraphRAG-Bench used in Appendix D.2 is a different benchmark (Xiang et al., 2026). Even there, the paper reports only contextual summarization and creative generation and compares only three baselines. Fact retrieval and complex reasoning, the two tasks closest to the paper's claims, are omitted.
- Precision favors methods that return fewer passages, while all baselines are fixed at top-5. Recall is reported only for MuSiQue in the multi-step setting (Fig. 7).
- The conciseness claim does not hold everywhere. On MuSiQue in the multi-step setting, G-Speculator retrieves 5.59 passages on average (more than top-5), and Precision drops from 45.80 to 37.81 as the number of steps grows.
Minor Weaknesses
- Obs. 16 (p. 23) claims G-Speculator is "insensitive to the choice of sentence embedding model", but there is no reference point. Adding Vanilla RAG with the same three embedding models would show whether the stability comes from speculative retrieval or from embeddings simply mattering little on these datasets.
- Sec. 4.1 says all methods use the same LLM (GPT-4o-mini), but Search-R1 and R1-Searcher are RL-trained policy models. Please clarify how they were run.
- Obs. 5 says triple extraction costs "up to 4.37×10⁶" input tokens, but Table 3 lists at most 4.30 for HippoRAG and GFM-RAG.
- The formal definition in Sec. 3.1 (R = ∪_s R_s with ϕ(r) = E_s) is ambiguous when the same span appears in several sentences. The code uses per-sentence relation ids.
- The case studies are inconsistent. In Case 2, the clue entity is "Otto I, Count of Guelders", but the sequence contains "Otto I". In Case 3, clue c2 uses "United States of America", which does not appear in s532 (it contains "American").
- The AI-use statement says generative AI was used to "generate synthetic datasets for the preliminary study", while Appendix E describes LLM-extracted evidence chains from PER-QA. Please clarify what was synthesized and how it was validated.
- The "guess first, correct the rest" framing invites comparison with speculative decoding, but the analogy is loose. The correction here is an unverified LLM judgment (Case 4 accepts "Australia" as sufficient), and it costs extra sequential LLM calls rather than a cheap parallel verification.
References
- Gong et al., "Beyond Chunks and Graphs: Retrieval-Augmented Generation through Triplet-Driven Thinking" (T2RAG), Findings of ACL 2026 / arXiv:2508.02435: placeholder-triplet decomposition with iterative resolution, efficiency-oriented, compared against IRCoT.
- Luo et al., "HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation," NeurIPS 2025: hypergraph-based RAG with n-ary relational facts as hyperedges.
- Xiao et al., "GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation," arXiv:2506.02404, 2025: a domain-specific benchmark built from 20 textbooks across 16 disciplines.
- Otmazgin et al., "F-coref: Fast, Accurate and Easy to Use Coreference Resolution," AACL-IJCNLP 2022 (System Demonstrations): a lightweight coreference resolver that could replace the title-entity heuristic.
Questions for authors
Questions
- What are the end-to-end per-query latency and online LLM token counts of G-Speculator, KG-Translator, HippoRAG+IRCoT, T2RAG, and HyperGraphRAG? Why do R and I differ in Table 3?
- What is evidence reachability on HypoKG under the protocol of Sec. 2?
- How much does prepending title entities contribute (ablation)? How does the method behave on corpora without passage titles, or when a pronoun refers to a non-title entity?
- How are ambiguous or repeated relation spans (e.g., two "in"s in one sentence) handled? Why not constrain the speculated entity using the predicate's expected type and spaCy labels?
9 comments
W3 & Q2: Regarding the evidence reachability of HypoKG
Thanks for your insightful comments. Following your suggestion, we analyze evidence reachability on HypoKG and compare it against the OpenIE-based KG.
Table 3: Evidence reachability on MuSiQue.
| Graph | E1 (1-hop) | E1 (2-hop) | E1 (3-hop) | E1 (unreachable) | E2 (1-hop) | E2 (2-hop) | E2 (3-hop) | E2 (unreachable) | Overall |
|---|---|---|---|---|---|---|---|---|---|
| OpenIE-based KG | 64.0 | 7.6 | 4.6 | 23.8 | 31.2 | 19.0 | 4.8 | 45.0 | 55.0 |
| HypoKG | 69.0 | 6.4 | 2.0 | 22.6 | 47.6 | 9.8 | 5.2 | 37.4 | 62.6 (+7.6%) |
As shown in Table 3, HypoKG effectively improves evidence reachability, which is reflected in two aspects: (1) complete evidence chains are more reachable (+7.6%), giving G-Speculator access to broader evidence; (2) evidence is reached at earlier hops (16.4% higher 1-hop reachability for E2), simplifying the retrieval process.
W4: Regarding the comparison with additional baselines
Thanks for your expert feedback. Following your suggestion, we add T2RAG and HyperGraphRAG as additional baselines.
Table 4: Supplementary baseline experiments.
| Method | HotpotQA | 2Wiki | MuSiQue |
|---|---|---|---|
| HyperGraphRAG | 75.09 | 56.07 | 36.92 |
| T2RAG | 77.11 | 77.61 | 37.43 |
| G-Speculator | 82.13 | 87.54 | 48.92 |
As shown in Table 4, G-Speculator maintains the best performance among baselines. Cost analysis is reported in W1 & Q1 (1).
Furthermore, these methods still suffer from redundant retrieval results (Section 2.2, Case 1) and biased retrieval paths (Section 2.2, Case 2), due to:
- Undifferentiated traversal wastes computation on irrelevant nodes and introduces redundant evidence;
- Missing relations or noise can divert traversal from correct reasoning paths.
Motivated by this, we propose a novel speculation paradigm. Its core idea draws from human reasoning in complex tasks: rather than exhaustively reviewing all materials, humans first form a tentative judgment and then seek targeted evidence to verify it.
- We construct HypoKG, which serializes passages into ordered entity-relation sequences preserving contextual word order. Unlike graphs that serve merely as passive knowledge structures, HypoKG actively provides linguistic priors to avoid blind traversal.
- On top of HypoKG, we propose G-Speculator, which first tentatively speculates evidence chains based on a linguistic assumption, then verifies and corrects them through CoT-guided analysis, precisely identifying and fully comprehending core evidence.
In addition, we also provide detailed explanations in the FAQs section (Appendix A.1 and A.3).
W5: Regarding the comparison across a broader evaluation scope
- Evaluation on additional benchmarks
Thanks for your helpful suggestions. We discuss the selection of multi-hop datasets in FAQs (Appendix A.4) and present the cross-domain performance in Appendix D.2. Following your suggestion, we further evaluate on additional cross-domain benchmarks and tasks.
Table 5: Accuracy (ACC) on GraphRAG-Bench (CS).
| Method | ACC |
|---|---|
| HippoRAG | 72.6 |
| LinearRAG | 71.5 |
| KG-Translator | 71.9 |
| G-Speculator | 74.0 |
Table 6: Accuracy (ACC) on GraphRAG-Bench (Medical & Novel)'s Fact Retrieval (FR) and Complex Reasoning (CR) tasks.
| Method | Medical (FR) | Medical (CR) | Novel (FR) | Novel (CR) |
|---|---|---|---|---|
| HippoRAG+IRCoT | 58.8 | 55.2 | 48.2 | 37.1 |
| KG-Translator | 42.7 | 42.1 | 38.9 | 30.9 |
| G-Speculator | 58.9 | 58.2 | 47.6 | 38.1 |
As shown in Tables 5 and 6, G-Speculator generalizes well across different domains and tasks.
- Detailed retrieval performance
Thank you for the constructive feedback. Following your suggestion, we report more comprehensive retrieval performance across question types.
Table 7: Retrieval quality report. We report Precision (left) and Recall (right).
| Method | 2Wiki Br. | 2Wiki Cp. | 2Wiki In. | 2Wiki Cp. | 2Wiki Bc. | MuSiQue Br. | MuSiQue Br. | MuSiQue Br. |
|---|---|---|---|---|---|---|---|---|
| IRCoT | 29.6 | 74.0 | 32.7 | 81.8 | 28.5 | 71.3 | 23.8 | 59.6 | 47.0 | 58.8 | 27.1 | 67.7 | 31.5 | 52.5 | 27.8 | 34.7 |
| Search-R1 | 31.1 | 77.7 | 33.8 | 84.5 | 28.2 | 70.4 | 29.4 | 73.6 | 51.4 | 64.2 | 29.0 | 72.6 | 25.4 | 42.3 | 20.6 | 25.8 |
| R1-Searcher | 33.2 | 83.0 | 36.8 | 92.0 | 31.9 | 79.8 | 33.8 | 84.4 | 40.8 | 51.0 | 29.4 | 73.6 | 26.8 | 44.7 | 24.9 | 31.2 |
| HippoRAG+IRCoT | 37.7 | 94.3 | 39.2 | 98.1 | 33.3 | 83.3 | 39.8 | 99.6 | 75.4 | 94.3 | 27.5 | 68.7 | 31.6 | 52.7 | 31.3 | 39.1 |
| GFM-RAG+IRCoT | 35.4 | 88.6 | 36.0 | 90.0 | 32.1 | 80.3 | 36.6 | 91.6 | 62.4 | 78.1 | 21.6 | 53.9 | 23.5 | 39.2 | 25.7 | 32.1 |
| KG-Translator | 58.6 | 95.9 | 68.1 | 95.3 | 50.5 | 90.5 | 72.2 | 96.8 | 73.2 | 87.1 | 31.6 | 68.7 | 29.8 | 47.7 | 26.5 | 31.6 |
| G-Speculator | 81.5 | 92.5 | 93.7 | 95.3 | 74.9 | 84.9 | 90.9 | 96.8 | 78.0 | 90.1 | 46.3 | 67.1 | 34.8 | 54.9 | 30.3 | 39.9 |
As shown in Table 7, G-Speculator effectively retrieves the required evidence while filtering out redundant passages. This reduced noise allows the QA model to focus on the key evidence during path correction, leading to improved QA results.
Dear Reviewer 9dMC,
We sincerely appreciate the time and efforts you have devoted to identifying potential issues in our paper. We are also grateful for the opportunity to further clarify the design motivations and experimental details of our framework, which helps us refine the work.
W1 & Q1: Regarding the efficiency
Thanks for your thoughtful suggestions. Following your suggestion, we supplement online LLM token counts and per-query latency to further evaluate the efficiency and cost of each method.
Table 1: Online LLM token counts, per-query latency (s), and F1 on 2Wiki.
As shown in Table 1, G-Speculator delivers notable performance improvements with competitive token cost and latency. Compared to KG-Translator, although our method makes more API calls, it uses 8.25% more total tokens due to differences in prompt design, while reducing latency by 13.21% and improving F1 by 9.02%.
Thanks for the focused comments. The R-stage API calls are activated only by query graph construction, while the I-stage calls are activated by graph speculative retrieval or evidence expansion. G-Speculator reuses CoT to assess evidence sufficiency after expansion, accounting for the difference between I-stage (1.98) and R-stage (1.72) calls. We have supplemented this step in Algorithm 2 in the revised paper.
Thank you for the critical feedback. The entity linking model has modest training overhead, requiring only 197.33 seconds on a single RTX 5090 GPU.
W2 & Q3 & Q4: Regarding further explanation of speculation rule
Thank you for the detailed review. Our linguistic assumption only applies to the speculation phase: given a clue (e′,r′), we take the nearest entity on the side of r′ opposite to e′ as the speculated entity e∗, which is adjacent to r′. Notably, the explored clue (e′,r′) does not require adjacency, it merely serves as an anchor for the retrieval direction, and can be any entity-relation pair in the same sentence.
Thank you for the constructive suggestions. We demonstrate G-Speculator's effectiveness on G-Bench (Novel/Medical) in Appendix D.2, which adopts an untitled setting. In addition, title prepending is a common configuration for sentence-level retrieval (e.g., LinearRAG, KG-Translator). This is because sentences after splitting often start with pronouns, requiring title entities for pronoun disambiguation. Following your suggestion, we provide additional ablation on title entities.
Table 2: Ablation study of title prepending on three multi-hop datasets.
As shown in Table 2, the ablation validates the effectiveness of title prepending. Notably, G-Speculator remains competitive even without title prepending, indicating that the core gains stem from the speculative retrieval mechanism.
Thanks for your expertise. Each relation span in a sentence is assigned a distinct id based on its position during serialization. Furthermore, during graph speculative retrieval, even when relation predicates are ambiguous or repeated, we ensure precise retrieval through the following mechanisms:
Multi-candidate exploration for comprehensive clues. Hyperparameter α (set to 3) controls clue exploration width, such that even the same relation spans yield multiple candidate paths rather than stopping at the first occurrence.
Clue trajectory scoring for implicit correction. Multiple clues form a trajectory π, where ω(π) is jointly determined by every clue's quality. Any weak clue drags down the overall ranking, allowing high-quality trajectories to naturally emerge.
Relation adjacency position implies expected entity types. The subject-predicate-object adjacency assumption reflects predicate semantics, where the entity position implies the expected type (e.g., a person after "married").