- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents ONTOS for knowledge graph question answering. An LLM generates an abstract reasoning sketch, which guides graph search alongside retrieval of annotated training query templates. The resulting SPARQL candidates are executed and presented to the LLM for selection, with uncertain selections routed to program repair. Experiments evaluate the framework on CWQ, WebQSP, and GrailQA using several model sizes.
Critical strengths
-
The paper addresses an important problem in translating natural language questions into executable knowledge graph queries.
-
The experiments cover three established benchmarks and multiple model sizes, with strong reported results relative to several existing baselines.
Critical weaknesses
-
The paper criticizes hop-by-hop exploration, yet its experiments suggest that graph search remains a major source of performance. Removing both micro-track search and repair reduces GrailQA Hits@1 from 87.1% to 32.1%, while even Qwen3.5-0.8B achieves 76.4% overall Hits@1. These results raise the possibility that much of the performance comes from semantic matching over available graph relations and template reuse, rather than the proposed candidate-first organization. The experiments do not adequately distinguish these contributions.
-
The paper lacks comparisons with relevant Text-to-SPARQL query generation and repair methods, such as Interactive-KBQA, SPINACH, and GRASP, which are discussed in Related Work. This leaves the advantage of candidate synthesis and selection over interactive query generation and refinement insufficiently established.
Questions for authors
-
How much do LLM planning and selection contribute beyond semantic matching and template reuse, particularly for Qwen3.5-0.8B?
-
How does ONTOS compare with existing Text-to-SPARQL frameworks, such as Interactive-KBQA, SPINACH, and GRASP?
4 comments
Regarding W2 / Q2: Comparison with Interactive-KBQA, SPINACH, and GRASP
To address your concern about comparisons with Interactive-KBQA, SPINACH, and GRASP, the revised manuscript adds a protocol comparison and a re-evaluation of GRASP's public queries (Appendix F.8, pp. 31–32, Tables 27–29). Below, we summarize how ONTOS differs from these execution-feedback agents and report a new head-to-head comparison against Interactive-KBQA's updated GPT-4.1 public predictions.
1. Design and protocol comparison (Table 27). Interactive-KBQA, SPINACH, and GRASP rely on interactive query generation and refinement driven by execution feedback. By contrast, ONTOS separates one global conceptual plan from symbolic candidate synthesis, then lets the LLM compare complete executed programs to reduce error accumulation from premature local decisions.
- Interactive-KBQA (ACL 2024): The original paper evaluates on type-balanced Freebase subsets (600 CWQ / 300 WebQSP), reporting F1 of 49.07/71.20% with entity linking and 56.74/78.64% with golden entity names. Its updated GPT-4.1 run is evaluated directly below.
- SPINACH (Findings EMNLP 2024): Evaluates on Wikidata (WikiWebQuestions is distinct from Freebase WebQSP). We compare its exploration/backtracking design and tools; a Freebase numerical comparison would require porting its Wikidata-specific tools.
- GRASP / GRASP-F (ISWC 2025): Evaluates on 200-question Freebase samples and discovers entity identifiers from the question and endpoint, whereas ONTOS receives released topic MIDs and uses annotated training templates.
2. New comparison with Interactive-KBQA (GPT-4.1 public run). We obtained the authors' updated public run (GPT-4.1-2025-04-14) and matched exact question text against our evaluation set, yielding 500 CWQ, 1,621 WebQSP, and 72 GrailQA shared questions. When we score both the published answer lists and our frozen predictions with the same references and a common evaluator (retaining empty predictions), ONTOS outperforms Interactive-KBQA even before repair and with a 7B backbone (both ONTOS rows exclude repair; F1 is Macro-F1):
| Shared subset | n | Method and backbone | H@1 (%) | F1 (%) |
|---|---|---|---|---|
| CWQ | 500 | Interactive-KBQA, GPT-4.1 | 57.20 | 46.73 |
| CWQ | 500 | ONTOS, Qwen2.5-7B-Instruct | 86.60 | 76.34 |
| CWQ | 500 | ONTOS, Qwen3.5-27B | 90.40 | 80.91 |
| WebQSP | 1,621 | Interactive-KBQA, GPT-4.1 | 86.43 | 70.58 |
| WebQSP | 1,621 | ONTOS, Qwen2.5-7B-Instruct | 87.91 | 75.72 |
| WebQSP | 1,621 | ONTOS, Qwen3.5-27B | 88.77 | 78.08 |
| GrailQA | 72 | Interactive-KBQA, GPT-4.1 | 68.06 | 64.73 |
| GrailQA | 72 | ONTOS, Qwen2.5-7B-Instruct | 79.17 | 76.93 |
| GrailQA | 72 | ONTOS, Qwen3.5-27B | 81.94 | 80.66 |
The same advantage holds under the authors' released exact-set F1 rule and archived references: Interactive-KBQA achieves 46.02 / 70.02 / 61.86% F1 on CWQ / WebQSP / GrailQA, versus 75.39 / 71.91 / 75.15% for ONTOS (Qwen2.5-7B-Instruct, before repair).
3. GRASP historical-query reassessment (Tables 28–29). We re-executed both methods' recorded queries on the same local Freebase endpoint (20 s budget, original LIMIT preserved) under our common evaluator, following the authors' empty-gold exclusions while retaining missing/invalid predictions. Exact question matching yields 199 WebQSP questions with Qwen2.5-7B-Instruct and 191 shared CWQ questions across all 11 public GRASP configurations (no Qwen2.5-7B CWQ output is published):
| Shared subset | n | Method and backbone | H@1 (%) | F1 (%) |
|---|---|---|---|---|
| WebQSP | 199 | GRASP, Qwen2.5-7B-Instruct | 20.10 | 16.05 |
| WebQSP | 199 | ONTOS, Qwen2.5-7B-Instruct (selection / final) | 86.93 / 84.42 | 72.97 / 73.59 |
| CWQ | 191 | GRASP-F, GPT-4.1 mini + demos + feedback (best of 11) | 76.96 | 63.80 |
| CWQ | 191 | ONTOS, Qwen2.5-7B-Instruct (selection / final) | 86.39 / 85.86 | 72.27 / 73.18 |
Even against GRASP-F's strongest CWQ configuration (GPT-4.1 mini; all 11 configurations are in Table 29), ONTOS with Qwen2.5-7B-Instruct leads by more than 8 points in both H@1 and F1. Under the authors' row-major/ASK F1 metric, GRASP scores 24.14% on WebQSP (Qwen2.5-7B) and 65.64% on CWQ (best configuration). First-column and endpoint-sensitivity analyses in Appendix F.8 preserve the same advantage.
Together with the planning and selection controls in our W1/Q1 response, these comparisons directly benchmark ONTOS against the recent execution-feedback agents. We hope this addresses your concerns and would appreciate your reconsidering your assessment in light of the new evidence.
Thanks to the authors for their response and the additional experiments. These have addressed most of my concerns. Therefore, I will raise my score.
Thank you very much for taking the time to review our response and additional experiments, and for updating your assessment. We really appreciate your constructive feedback.
Regarding W1 / Q1: Contributions of Planning and Selection Beyond Matching and Template Reuse
ONTOS uses 2.00 LLM calls per question before repair—one global conceptual plan and one comparative selection over complete executed programs—while Micro symbolic search grounds that plan on the graph. The motivation is to avoid error accumulation from premature local decisions in step-by-step LLM exploration, which invokes the LLM repeatedly per hop or refinement step (for example, 9.78 calls/q for reproduced CoG on Qwen2.5-7B across all 6,159 questions at 60.38% H@1 / 48.88% F1, vs. 2.00 calls/q and 83.63% / 74.10% for ONTOS without repair; Appendix F.10, Table 32). Below, we quantify how much planning, selection, and template reuse each contribute, focusing on Qwen3.5-0.8B.
1. Planning and selection controls with Qwen3.5-0.8B. Across all 6,159 questions, we compare sketch-guided construction against a question-only control that keeps the exact same topic entities, ontology, relation embeddings, training-template memory, and eight-card interface. Without a sketch, Micro searches depths 1–4 using the raw question embedding and question-text operator/filter rules, while Macro uses question retrieval and KDE without sketch alignment. Both pools are evaluated with deterministic rank-1 selection and with the same Qwen3.5-0.8B selector (repair disabled; graph calls sum Micro and Macro; Appendix F.2, Tables 13–14).
Planning and selection show a clear complementary effect. On the sketch-guided pool, Qwen3.5-0.8B selection improves F1 by +5.64 points over deterministic rank 1 (65.75% vs. 60.11%); on the question-only pool, the same selector reduces F1 by 1.41 points (61.74% vs. 63.15%). With the selector fixed, sketch guidance adds +4.01 F1 and cuts graph calls by 79.2% (86.35 vs. 415.32). Because the selector reads only the eight candidate cards and never sees the sketch directly, this gain reflects how sketch guidance shapes the candidate pool into cleaner, more distinguishable programs—not a model recalling its own sketch or a same-model-only effect.
Oracle candidate quality alone does not explain this difference. Only for Qwen3.5-0.8B sketches is question-only oracle F1@8 slightly higher (86.05% vs. 85.05%), yet final selection F1 (measured with the Qwen3.5-0.8B selector and repair disabled) is worse on the question-only pool. Even on the 4,584 questions where both 8-card windows contain an F1-perfect candidate, the selector picks one in 3,409 sketch-guided cases versus 3,124 question-only cases. At every larger tested planner scale (Qwen3.5-2B/4B/9B/27B), sketch-guided oracle F1@8 also exceeds the question-only baseline (86.13% / 87.76% / 87.85% / 88.61% vs. 86.05%).
2. Separating planner and selector capacity. Crossing planner and selector scale across all 6,159 questions (repair disabled; Table 15) separates their individual contributions:
Upgrading only the planner from Qwen3.5-0.8B to Qwen3.5-27B raises F1 by +3.20 points under the 0.8B selector and +3.39 points under the 27B selector; upgrading only the selector on the 0.8B pool adds +10.05 F1 points. Both LLM roles contribute substantial, distinct gains.
3. Zero-shot generalization beyond template reuse. On GrailQA (where 562/1,000 questions are zero-shot with unseen schemas/relations), removing the Micro track (
Macro only, leaving only training templates + selection, with repair disabled) collapses Qwen3.5-27B from 86.8% / 83.4% to 32.1% H@1 / 26.3% F1 (Section 4.3, Table 2):Templates alone cannot ground unseen structures. Conversely, removing all training templates (
Micro only, repair disabled) retains 85.2% H@1 / 82.0% F1—within 1.4 F1 points of the full two-track pool—showing that performance is not driven by template reuse. Furthermore, on the 562 zero-shot questions, the full Qwen3.5-27B pipeline reaches 90.4% H@1 / 88.8% F1 (89.9% H@1 / 88.2% F1 before repair; Appendix D).