- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SPARK, which addresses knowledge graph question answering by combining structure-aware, path-prefix-conditioned tree search with graph-aware KGL answer generation. It selects relations and then entities while conditioning each expansion on the full reasoning prefix, and uses a shared graph encoder in a two-stage training strategy to coordinate retrieval and generation. Experiments on WebQSP and CWQ report improvements in retrieval and answer quality.
Critical strengths
-
The hierarchical relation-entity policy evaluates candidate expansions against the complete reasoning trajectory. It offers a principled way to search for coherent multi-hop evidence.
-
The typed symbols and ordered paths provide a compact representation of retrieved evidence, while projected entity and relation embeddings supply fine-grained graph information to the generator. The two-stage strategy first learns a retrieval policy, then uses a shared graph encoder to coordinate the retrieval and generation objectives.
-
Experiments on WebQSP and CWQ report improvements in retrieval and answer quality.
Critical weaknesses
-
The paper presents prefix-consistent supervision as a key contribution, but does not clearly report a direct comparison with edge-level positive labeling based on answer-path membership. The "without full-prefix conditioning" ablation in Table 5 tests a different design choice and does not isolate the effect of the labeling rule.
-
Table 3 compares full graph-aware KGL with plain text using the same retrieved paths, which supports the effectiveness of the overall interface but does not isolate the contribution of graph-embedding injection. A focused ablation that retains the KGL dictionary and ordered paths while removing the injected entity and relation embeddings would clarify this contribution.
-
The introduction (lines 51–53) motivates SPARK partly by the latency and computational cost of repeated LLM calls. Table 1 compares retrieval time across retrievers, and Table 3 compares generation time and token counts between gKGL and a plain-text variant. If end-to-end efficiency is a central claim, the paper should additionally compare total latency and LLM-call counts with competing KGQA systems under matched conditions.
-
The presentation and organization of this paper could be clearer.
(1) Notations in method
- denotes both the natural-language question and its encoded vector;
- the state repeats , which is already the final entity in .
(2) Result presentation
- Table 2 provides broad method categories, but still mixes closed models, open models of different sizes, and different retrieval paradigms. Grouping results more clearly by generator, scale, and retrieval method would make SPARK’s contribution easier to assess.
Questions for authors
Please refer to Critical Weaknesses.
10 comments
Q3: The introduction (lines 51–53) motivates SPARK partly by the latency and computational cost of repeated LLM calls. Table 1 compares retrieval time across retrievers, and Table 3 compares generation time and token counts between gKGL and a plain-text variant. If end-to-end efficiency is a central claim, the paper should additionally compare total latency and LLM-call counts with competing KGQA systems under matched conditions.
R3: Thank you for this comment. We would like to clarify what we claim about efficiency and how the existing tables support it.
SPARK's efficiency stems from an architectural property rather than implementation tuning: the tree-search retriever makes no LLM calls, and generation requires a single LLM forward pass per question, so SPARK performs exactly one LLM call per question end-to-end; the manuscript states that SPARK avoids the repeated LLM-controlled retrieval calls required by iterative agent-based methods (Section 4.2). This contrast is also visible in Table 1, where G-Retriever's retrieval alone takes 672s/1530s on WebQSP/CWQ versus 5s/9s for SPTS. Summing the retrieval and generation stages (Tables 1 and 3) yields an end-to-end latency of roughly 13s (5s retrieval + 8s generation) per question on WebQSP and 17s (9s retrieval + 8s generation) on CWQ on our hardware.
We agree that a fully matched end-to-end comparison across competing systems would strengthen the efficiency evidence. We would like to note that the LLM-call count is the more fundamental measure here: wall-clock latency depends heavily on hardware and implementation, whereas the number of LLM calls is implementation-independent and directly quantifies the overhead that the introduction motivates against. On this measure, SPARK's single call per question contrasts sharply with the repeated calls of iterative agent-based methods. A fully matched wall-clock comparison would require re-implementing and re-running the LLM-agent baselines on our hardware, which exceeds the scope of this revision; we consider it important future work.
Q4: The presentation and organization of this paper could be clearer.
R4: Thank you for this feedback. Although the comment is general, we have taken it as an opportunity to re-examine the manuscript and have made a series of concrete improvements to its presentation and organization:
- Terminology. We shortened the framework name and title to a compact form and unified the terminology throughout the paper; the full-name definition in the Introduction now matches the revised title ("Prefix-Conditioned"), and the manuscript consistently describes the retrieval–generation integration as "coordination" rather than "unifying", reflecting the actual design.
- Clarity of ablations. We gave every ablation variant a self-descriptive name that states exactly which design is removed (e.g., "w/o prefix-consistent labeling", "KGL") and aligned the accompanying text with the tables (five core SPTS designs now match the table rows).
- Organization. The paper now provides a roadmap at the start of Section 3 that maps each subsection to the pipeline stages in Fig. 1, and an interpretability analysis in the appendix (a case study of path selection and an attention-based analysis of graph-embedding influence), with expanded implementation details in the appendix.
We would be happy to address any particular aspect the reviewer finds unclear, and we welcome specific pointers for further polishing.
Q5: Notations in method
- 1) denotes both the natural-language question and its encoded vector;
- 2)the state repeats , which is already the final entity in .
R5: Thank you for the careful notation check. We address the two points as follows.
(1) We use plain for the natural-language question and bold for its encoded vector, following the standard convention that plain symbols denote inputs and boldface symbols denote their learned representations (as with and ). To make this distinction explicit and remove any ambiguity, we have added a sentence in Section 2 stating this convention at first use.
(2) Writing the search state as is deliberate: although already appears as the last element of , keeping it explicit emphasizes that all expansions are taken from the current entity and that ends with an entity rather than a relation. We have added a clarifying remark in the paper noting that is the final entity of .
Both revisions are highlighted in red in the revised manuscript.
Q6: Result presentation
- Table 2 provides broad method categories, but still mixes closed models, open models of different sizes, and different retrieval paradigms. Grouping results more clearly by generator, scale, and retrieval method would make SPARK’s contribution easier to assess.
R6: Thank you for this suggestion. We have reorganized Table 2 accordingly: within each category, methods are now grouped by generator family and scale with dashed separators, and the SPARK variants are collected in a separate final block for direct same-scale comparison. The revised table is included in the revised manuscript.
We sincerely thank you for your time and constructive comments. We are excited that you found our paper worthy of publication. Below, we provide detailed replies to your comments to resolve your concerns.
Q1: The paper presents prefix-consistent supervision as a key contribution, but does not clearly report a direct comparison with edge-level positive labeling based on answer-path membership. The "without full-prefix conditioning" ablation in Table 5 tests a different design choice and does not isolate the effect of the labeling rule.
R1: Thank you for this precise comment. We agree that the "w/o full-prefix conditioning" ablation removes the prefix conditioning from the scoring mechanism rather than from the supervision rule, and therefore does not isolate the contribution of prefix-consistent labeling itself.
To address this, we have added a controlled ablation to the revised manuscript (Table 5): a variant that keeps the full prefix-conditioned encoder and the same KL + pairwise-ranking objective, but replaces the prefix-consistent labels with edge-level labels, under which an edge is labeled positive whenever it appears in the supervision subgraph Gq⋆ , regardless of whether the extended trajectory remains a prefix of an answer path. This isolates the labeling rule while holding everything else fixed. The results confirm the argument in Section 3.1: this variant incurs the largest degradation among the retrieval ablations, dropping TR/AR from 0.912/0.954 to 0.795/0.824 on WebQSP and from 0.926/0.972 to 0.806/0.837 on CWQ, showing that edge-level labeling distributes positive probability mass over branches that diverge from all answer paths, which is exactly the erroneous supervision that the prefix-consistent rule is designed to remove. The new ablation and its discussion are highlighted in red in the revised manuscript.
Q2: Table 3 compares full graph-aware KGL with plain text using the same retrieved paths, which supports the effectiveness of the overall interface but does not isolate the contribution of graph-embedding injection. A focused ablation that retains the KGL dictionary and ordered paths while removing the injected entity and relation embeddings would clarify this contribution.
R2: We thank the reviewer for this comment. Since the gKGL vs. Plain-text comparison changes both the KGL interface and the graph-embedding injection, we have added a focused ablation to Table 3 that isolates the injection alone: KGL w/o graph injection keeps the symbol dictionary and ordered paths unchanged and only replaces the projected graph representation at each KGL symbol position with the mean of the subword token embeddings of that symbol's surface form, since a KGL symbol spans multiple subword tokens under the LLM tokenizer while occupying a single prompt position. With the generator, retrieved paths, and prompt structure held fixed, this variant drops Hits@1/F1 from 91.22/81.87 to 85.67/74.84 on WebQSP and from 76.50/71.01 to 68.29/66.12 on CWQ, which is even below Plain-text, while it retains the efficiency benefits of the KGL interface (325.75 vs. 401.48 average tokens; 9 vs. 13 seconds). The ablation thus yields a clean decomposition: the KGL interface contributes the efficiency gains, whereas the graph-embedding injection contributes the accuracy gains (5.55/7.03 Hits@1/F1 on WebQSP and 8.21/4.89 on CWQ). The new ablation and its discussion are highlighted in red in the revised manuscript.