acceptodds
Under review as a conference paper at ICLR 2027

ONTOS: Candidate-First Knowledge Graph Question Answering with Ontology-Constrained Program Synthesis

Abstract

Knowledge graph question answering requires translating a question's intent into executable programs over a large graph schema. Direct query generation must resolve schema identifiers and query structure, while LLM-guided exploration repeatedly interleaves planning with graph traversal. We present Ontos, an ontology-driven, candidate-first framework that separates conceptual planning from initial program construction. An LLM produces an abstract derivation sketch without graph identifiers. A dual-track synthesizer then combines sketch-guided, schema-filtered search with type-checked instantiation of program structures from annotated training questions. The resulting candidates are executed and presented to the LLM for selection, with low-confidence selections routed to a bounded, ontology-grounded program-repair stage. Across 6,159 evaluation questions from CWQ, WebQSP and GrailQA, Ontos achieves pooled Hits@1 of 83.3%, 87.6% and 89.1% with Qwen2.5-7B, Qwen3.5-9B and Qwen3.5-27B, respectively, without task-specific LLM fine-tuning; on CWQ, all three exceed the results reported for the baselines we compare with, including GPT-4-based agents. With the 27B backbone, repair increases pooled Macro-F1 from 79.2% to 81.6% while processing 26.7% of questions. Stage-level ablations show complementary contributions from the two synthesis tracks. These results support ontology-constrained candidate construction and LLM-based selection as an effective division of work for knowledge graph question answering.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Knowledge Graph Question Answering, LLM Agents, Ontology-Constrained Reasoning, Neuro-Symbolic Reasoning, Program Synthesis and Repair, Text-to-SPARQL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4b9hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ONTOS for knowledge graph question answering. An LLM generates an abstract reasoning sketch, which guides graph search alongside retrieval of annotated training query templates. The resulting SPARQL candidates are executed and presented to the LLM for selection, with uncertain selections routed to program repair. Experiments evaluate the framework on CWQ, WebQSP, and GrailQA using several model sizes.

Critical strengths
  • The paper addresses an important problem in translating natural language questions into executable knowledge graph queries.

  • The experiments cover three established benchmarks and multiple model sizes, with strong reported results relative to several existing baselines.

Critical weaknesses
  • The paper criticizes hop-by-hop exploration, yet its experiments suggest that graph search remains a major source of performance. Removing both micro-track search and repair reduces GrailQA Hits@1 from 87.1% to 32.1%, while even Qwen3.5-0.8B achieves 76.4% overall Hits@1. These results raise the possibility that much of the performance comes from semantic matching over available graph relations and template reuse, rather than the proposed candidate-first organization. The experiments do not adequately distinguish these contributions.

  • The paper lacks comparisons with relevant Text-to-SPARQL query generation and repair methods, such as Interactive-KBQA, SPINACH, and GRASP, which are discussed in Related Work. This leaves the advantage of candidate synthesis and selection over interactive query generation and refinement insufficiently established.

Questions for authors
  • How much do LLM planning and selection contribute beyond semantic matching and template reuse, particularly for Qwen3.5-0.8B?

  • How does ONTOS compare with existing Text-to-SPARQL frameworks, such as Interactive-KBQA, SPINACH, and GRASP?

4 comments
AuthorsOpenReview ↗

Regarding W1 / Q1: Contributions of Planning and Selection Beyond Matching and Template Reuse

ONTOS uses 2.00 LLM calls per question before repair—one global conceptual plan and one comparative selection over complete executed programs—while Micro symbolic search grounds that plan on the graph. The motivation is to avoid error accumulation from premature local decisions in step-by-step LLM exploration, which invokes the LLM repeatedly per hop or refinement step (for example, 9.78 calls/q for reproduced CoG on Qwen2.5-7B across all 6,159 questions at 60.38% H@1 / 48.88% F1, vs. 2.00 calls/q and 83.63% / 74.10% for ONTOS without repair; Appendix F.10, Table 32). Below, we quantify how much planning, selection, and template reuse each contribute, focusing on Qwen3.5-0.8B.

1. Planning and selection controls with Qwen3.5-0.8B. Across all 6,159 questions, we compare sketch-guided construction against a question-only control that keeps the exact same topic entities, ontology, relation embeddings, training-template memory, and eight-card interface. Without a sketch, Micro searches depths 1–4 using the raw question embedding and question-text operator/filter rules, while Macro uses question retrieval and KDE without sketch alignment. Both pools are evaluated with deterministic rank-1 selection and with the same Qwen3.5-0.8B selector (repair disabled; graph calls sum Micro and Macro; Appendix F.2, Tables 13–14).

Candidate constructionDecision ruleH@1 (%)F1 (%)Graph calls/q
Question-onlyDeterministic rank 174.9363.15415.32
Question-onlyQwen3.5-0.8B selection72.9061.74415.32
Sketch-guided, Qwen3.5-0.8BDeterministic rank 172.2260.1186.35
Sketch-guided, Qwen3.5-0.8BQwen3.5-0.8B selection76.5265.7586.35

Planning and selection show a clear complementary effect. On the sketch-guided pool, Qwen3.5-0.8B selection improves F1 by +5.64 points over deterministic rank 1 (65.75% vs. 60.11%); on the question-only pool, the same selector reduces F1 by 1.41 points (61.74% vs. 63.15%). With the selector fixed, sketch guidance adds +4.01 F1 and cuts graph calls by 79.2% (86.35 vs. 415.32). Because the selector reads only the eight candidate cards and never sees the sketch directly, this gain reflects how sketch guidance shapes the candidate pool into cleaner, more distinguishable programs—not a model recalling its own sketch or a same-model-only effect.

Oracle candidate quality alone does not explain this difference. Only for Qwen3.5-0.8B sketches is question-only oracle F1@8 slightly higher (86.05% vs. 85.05%), yet final selection F1 (measured with the Qwen3.5-0.8B selector and repair disabled) is worse on the question-only pool. Even on the 4,584 questions where both 8-card windows contain an F1-perfect candidate, the selector picks one in 3,409 sketch-guided cases versus 3,124 question-only cases. At every larger tested planner scale (Qwen3.5-2B/4B/9B/27B), sketch-guided oracle F1@8 also exceeds the question-only baseline (86.13% / 87.76% / 87.85% / 88.61% vs. 86.05%).

2. Separating planner and selector capacity. Crossing planner and selector scale across all 6,159 questions (repair disabled; Table 15) separates their individual contributions:

Sketch sourceSelectorH@1 (%)F1 (%)
Qwen3.5-0.8BQwen3.5-0.8B76.5265.75
Qwen3.5-0.8BQwen3.5-27B85.5775.80
Qwen3.5-27BQwen3.5-0.8B80.0968.95
Qwen3.5-27BQwen3.5-27B88.4979.19

Upgrading only the planner from Qwen3.5-0.8B to Qwen3.5-27B raises F1 by +3.20 points under the 0.8B selector and +3.39 points under the 27B selector; upgrading only the selector on the 0.8B pool adds +10.05 F1 points. Both LLM roles contribute substantial, distinct gains.

3. Zero-shot generalization beyond template reuse. On GrailQA (where 562/1,000 questions are zero-shot with unseen schemas/relations), removing the Micro track (Macro only, leaving only training templates + selection, with repair disabled) collapses Qwen3.5-27B from 86.8% / 83.4% to 32.1% H@1 / 26.3% F1 (Section 4.3, Table 2):

Candidate constructionH@1 (%)F1 (%)
Micro + Macro86.883.4
Micro only: no training-template Macro track85.282.0
Macro only: training-template track32.126.3

Templates alone cannot ground unseen structures. Conversely, removing all training templates (Micro only, repair disabled) retains 85.2% H@1 / 82.0% F1—within 1.4 F1 points of the full two-track pool—showing that performance is not driven by template reuse. Furthermore, on the 562 zero-shot questions, the full Qwen3.5-27B pipeline reaches 90.4% H@1 / 88.8% F1 (89.9% H@1 / 88.2% F1 before repair; Appendix D).

AuthorsOpenReview ↗

Regarding W2 / Q2: Comparison with Interactive-KBQA, SPINACH, and GRASP

To address your concern about comparisons with Interactive-KBQA, SPINACH, and GRASP, the revised manuscript adds a protocol comparison and a re-evaluation of GRASP's public queries (Appendix F.8, pp. 31–32, Tables 27–29). Below, we summarize how ONTOS differs from these execution-feedback agents and report a new head-to-head comparison against Interactive-KBQA's updated GPT-4.1 public predictions.

1. Design and protocol comparison (Table 27). Interactive-KBQA, SPINACH, and GRASP rely on interactive query generation and refinement driven by execution feedback. By contrast, ONTOS separates one global conceptual plan from symbolic candidate synthesis, then lets the LLM compare complete executed programs to reduce error accumulation from premature local decisions.

  • Interactive-KBQA (ACL 2024): The original paper evaluates on type-balanced Freebase subsets (600 CWQ / 300 WebQSP), reporting F1 of 49.07/71.20% with entity linking and 56.74/78.64% with golden entity names. Its updated GPT-4.1 run is evaluated directly below.
  • SPINACH (Findings EMNLP 2024): Evaluates on Wikidata (WikiWebQuestions is distinct from Freebase WebQSP). We compare its exploration/backtracking design and tools; a Freebase numerical comparison would require porting its Wikidata-specific tools.
  • GRASP / GRASP-F (ISWC 2025): Evaluates on 200-question Freebase samples and discovers entity identifiers from the question and endpoint, whereas ONTOS receives released topic MIDs and uses annotated training templates.

2. New comparison with Interactive-KBQA (GPT-4.1 public run). We obtained the authors' updated public run (GPT-4.1-2025-04-14) and matched exact question text against our evaluation set, yielding 500 CWQ, 1,621 WebQSP, and 72 GrailQA shared questions. When we score both the published answer lists and our frozen predictions with the same references and a common evaluator (retaining empty predictions), ONTOS outperforms Interactive-KBQA even before repair and with a 7B backbone (both ONTOS rows exclude repair; F1 is Macro-F1):

Shared subsetnMethod and backboneH@1 (%)F1 (%)
CWQ500Interactive-KBQA, GPT-4.157.2046.73
CWQ500ONTOS, Qwen2.5-7B-Instruct86.6076.34
CWQ500ONTOS, Qwen3.5-27B90.4080.91
WebQSP1,621Interactive-KBQA, GPT-4.186.4370.58
WebQSP1,621ONTOS, Qwen2.5-7B-Instruct87.9175.72
WebQSP1,621ONTOS, Qwen3.5-27B88.7778.08
GrailQA72Interactive-KBQA, GPT-4.168.0664.73
GrailQA72ONTOS, Qwen2.5-7B-Instruct79.1776.93
GrailQA72ONTOS, Qwen3.5-27B81.9480.66

The same advantage holds under the authors' released exact-set F1 rule and archived references: Interactive-KBQA achieves 46.02 / 70.02 / 61.86% F1 on CWQ / WebQSP / GrailQA, versus 75.39 / 71.91 / 75.15% for ONTOS (Qwen2.5-7B-Instruct, before repair).

3. GRASP historical-query reassessment (Tables 28–29). We re-executed both methods' recorded queries on the same local Freebase endpoint (20 s budget, original LIMIT preserved) under our common evaluator, following the authors' empty-gold exclusions while retaining missing/invalid predictions. Exact question matching yields 199 WebQSP questions with Qwen2.5-7B-Instruct and 191 shared CWQ questions across all 11 public GRASP configurations (no Qwen2.5-7B CWQ output is published):

Shared subsetnMethod and backboneH@1 (%)F1 (%)
WebQSP199GRASP, Qwen2.5-7B-Instruct20.1016.05
WebQSP199ONTOS, Qwen2.5-7B-Instruct (selection / final)86.93 / 84.4272.97 / 73.59
CWQ191GRASP-F, GPT-4.1 mini + demos + feedback (best of 11)76.9663.80
CWQ191ONTOS, Qwen2.5-7B-Instruct (selection / final)86.39 / 85.8672.27 / 73.18

Even against GRASP-F's strongest CWQ configuration (GPT-4.1 mini; all 11 configurations are in Table 29), ONTOS with Qwen2.5-7B-Instruct leads by more than 8 points in both H@1 and F1. Under the authors' row-major/ASK F1 metric, GRASP scores 24.14% on WebQSP (Qwen2.5-7B) and 65.64% on CWQ (best configuration). First-column and endpoint-sensitivity analyses in Appendix F.8 preserve the same advantage.

Together with the planning and selection controls in our W1/Q1 response, these comparisons directly benchmark ONTOS against the recent execution-feedback agents. We hope this addresses your concerns and would appreciate your reconsidering your assessment in light of the new evidence.

Thanks to the authors for their response and the additional experiments. These have addressed most of my concerns. Therefore, I will raise my score.

Thank you very much for taking the time to review our response and additional experiments, and for updating your assessment. We really appreciate your constructive feedback.

Reviewer_LRpgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ONTOS, an ontology-constrained, candidate-first framework for knowledge graph question answering (KGQA). The approach first leverages an LLM to generate an abstract derivation sketch without concrete KG identifiers. A dual-track program synthesis mechanism (comprising a micro-track and a macro-track) is then employed to construct executable query candidates. Finally, the LLM selects from the candidates, with low-confidence predictions routed to a bounded repair stage. Experiments on CWQ, WebQSP, and GrailQA demonstrate that the proposed framework achieves strong performance without task-specific LLM fine-tuning.

Critical strengths
  1. The proposed framework separates abstract semantic planning from executable program construction via dual-track synthesis (micro-track search and macro-track template reuse). This lets the LLM focus on conceptual reasoning without hallucinating KG identifiers, while avoiding costly sequential exploration steps.

  2. Evaluations across three benchmarks demonstrate strong performance. Without task-specific LLM fine-tuning, ONTOS with 7B to 27B backbones exceeds the reported results of GPT-4-based agents on CWQ. Stage-level ablations support the complementary contributions of the two synthesis tracks, while repair provides additional improvements with stronger backbones.

Critical weaknesses
  1. Ontology constraints are central to ONTOS, but Table 2 mainly evaluates entire pipeline components rather than individual ontology operations. It does not separately assess relation filtering, type checking, inverse folding, or CVT handling. As a result, it remains unclear which ontology components contribute most to the reported improvements.

  2. Candidate selection and repair remain important bottlenecks. With Qwen3.5-27B, 96.6% of questions have a Hits@1-correct candidate, but selection achieves only 88.5%. Repair improves Hits@1 to 89.1%, yet slightly reduces accuracy for three other backbones (Table 3), raising concerns about robustness and generalizability.

Questions for authors
  1. In Section 3.2, the abstract derivation sketch is generated by a single LLM call. Could the authors analyze how sketch quality affects final performance, particularly whether subsequent program synthesis can effectively correct missing relations or constraints in the initial sketch?
  2. In Section 3.3, the micro-track uses embedding similarity for path scoring with a fixed beam width (K=128). Could the authors investigate how different beam widths and path scoring strategies affect candidate quality and search efficiency?
  3. In Section 3.3, candidates from the two synthesis tracks are interleaved, with only the first eight presented to the LLM. What motivates this fixed candidate window and fusion strategy? Could dynamically allocating candidates between the two tracks improve performance?
  4. In Section 3.4, the Pairwise Replacement Gate requires the edited program to win comparisons in both presentation orders. Could the authors further justify this design and clarify its advantages over a single comparison or alternative decision strategies?
9 comments
AuthorsOpenReview ↗

Regarding W1: Individual Ontology Operations

Thank you for the detailed feedback on ontology operations, candidate quality, and decision stages. To isolate each ontology operation, we separately ablate system-relation filtering, inverse folding, slot-type prechecks, and CVT handling across all 6,159 questions using frozen Qwen3.5-27B sketches (Appendix F.4, pp. 26–28, Tables 17–20).

Key findings. Inverse folding and slot-type prechecks drive the largest gains:

  • Inverse folding protects the finite 16-card window from redundant inverse-equivalent representations, preserving 32 F1-perfect questions@16 (4,205 vs. 4,173; +0.57 pp oracle F1@16) and cutting optimized Micro graph calls from 99.85 to 71.19/question.
  • Slot-type prechecks cut original Macro candidate-program executions by 69.95% (24.89 vs. 82.85/q) and optimized complete Macro graph calls by 31.83% (21.64 vs. 31.75/q), producing identical ordered candidates on 6,151/6,159 questions (99.87%). The 8 exceptions change template membership at the type check; the inspected Tokyo International Film Festival case traces to a festival-series type missing from the finite empirical range sample, trading a tiny recall loss (+1 F1-perfect question when disabled) for substantial pruning.
  • System-relation filtering lowers Micro graph calls (79.50 → 71.19/q) with a small oracle F1@16 gain (+0.0592 pp).
  • CVT handling: Comparing atomic entry–role expansion against a staged control (score/prune entries, materialize surviving CVTs, then expand roles, with both preserving CVT identity and role-only scores) yields near-identical quality (79.68% vs. 79.66% oracle F1@16; 4,205 vs. 4,204 F1-perfect), while optimized graph calls favor atomic expansion (71.19 vs. 80.74/q).

Quality metrics below use the original independently executed candidates (K=128, mean scoring, reaggregated over the first 16 nonempty deduplicated cards; MRR-perfect@16 is the reciprocal rank of the first candidate with answer F1=1):

Micro condition (n=6,159)Top1 F1 (%)MRR-perfect@16Oracle F1@8 (%)Oracle F1@16 (%)F1-perfect@16
All operations56.79.540377.0579.684,205
No system filter56.78.540077.0279.624,191
No inverse folding56.66.538876.7179.114,173
Staged CVT56.80.540377.0079.664,204
Macro condition (n=6,159)Top1 F1 (%)MRR-perfect@8Oracle F1@8 (%)F1-perfect@8Orig. exec calls/q
Slot-type prechecks on54.80.553778.134,38324.89
Slot-type prechecks off54.82.553978.154,38482.85

Optimized complete graph workloads (Micro and Macro) and dataset breakdowns are reported separately in Appendix F.4 (Tables 19–20, p. 28). These are candidate-stage controls without rerunning selector or repair.

AuthorsOpenReview ↗

Regarding Q1: Sketch Quality and Recovery of Missing Relations or Constraints

Initial synthesis is designed to ground the supplied sketch into executable SPARQL queries, not to act as a general semantic corrector for sketch omissions (Section 3.2–3.4, pp. 4–6; Appendix F.2–F.3, pp. 25–27, Tables 15–16). Micro search follows the planned conceptual hops and operators, while Macro synthesis retrieves and instantiates templates using the question and sketch alignment. When the initial sketch omits or misinterprets a constraint, that error can propagate into the candidate pool; semantic correction is delegated to the subsequent repair stage (which is only invoked on low-confidence selections).

1. Impact of sketch-conditioned source vs. selector capacity. To isolate how sketch quality affects downstream accuracy without repair masking the difference, we cross the sketch-conditioned candidate source (0.8B vs. 27B) with selector capacity (0.8B vs. 27B) on all 6,159 questions with repair disabled:

Sketch source0.8B selector H@1 / F1 (%)27B selector H@1 / F1 (%)
Qwen3.5-0.8B76.52 / 65.7585.57 / 75.80
Qwen3.5-27B80.09 / 68.9588.49 / 79.19

With the selector fixed, upgrading the sketch source from 0.8B to 27B improves F1 by +3.20 pp (0.8B selector) and +3.39 pp (27B selector), while top-8 oracle F1 rises from 85.05% to 88.61% (F1-perfect coverage: 77.76% → 81.77%). A stronger selector mitigates part of the gap but does not eliminate the effect of the sketch-conditioned pool.

2. Grounding vs. omission recovery (23-case audit). We audited 18 outcome-stratified random cases (two per dataset across three outcome strata) plus five targeted sketch-error cases, reading complete sketches including free-text intent (Appendix F.3, Table 16). In all six sampled cases where the bounded initial pool lacked an F1-perfect candidate but the final prediction was F1-perfect, the required constraint was already present in a sketch branch or intent (e.g., PDF software genres already planned both PDF Converter and Portable Document Format; deterministic completion attached the second genre, raising F1 from 0.80 to 1.00). Conversely, genuine sketch errors—such as encoding "lower casualty estimates of 1000" as < 1000—persisted uncorrected. Thus, initial synthesis grounds existing plan semantics, while recovery of true omissions is delegated to the subsequent repair stage.

AuthorsOpenReview ↗

Regarding Q2 / Q3: Beam Search, Path Scoring, and Candidate Windows

Regarding Q2: Beam Width and Path Scoring

We evaluated six beam widths K∈{8, 16, 32, 64, 128, 256} (with mean scoring) and three hop-score aggregations (mean, minimum, maximum at K=128) across all 6,159 questions using frozen 27B sketches (Appendix F.5, pp. 28–29, Tables 21–22). Changes directly govern pruning during Micro search, while the final Micro cap (16), Macro track (8), and fusion window (8) remain fixed. No selector or repair is rerun.

K / scoringMicro MRR-perfect@16Micro oracle F1@16 (%)Fused Top1 F1 (%)Fused oracle F1@8 (%)Micro+Macro graph calls/q
8 / mean.533575.2965.2788.4043.14
16 / mean.537477.6965.1288.4956.49
32 / mean.539679.1564.9888.5874.40
64 / mean.540379.6064.9588.6286.40
128 / mean.540379.6864.9988.6192.83
256 / mean.540579.6765.0088.6396.46
128 / min.476879.0363.7688.0392.04
128 / max.527177.5665.2088.3496.14

Beam width. Coverage saturates around K=64–128: increasing K=128 → 256 adds only +0.0215 pp fused oracle F1@8 for +5.10% Micro graph calls (71.19 → 74.82/q), while K=64 uses 9.03% fewer Micro calls with nearly identical coverage. Narrower beams (K=8, 16) cut Micro graph calls by 69.80% and 51.05%, trading off 4.39 and 1.98 pp of Micro oracle F1@16 (yet achieving slightly higher deterministic fused Top1 F1). Note that the auxiliary-branch cap min(32, K) and parent-path quota vary with K, so K=128 represents a clear coverage–cost tradeoff.

Path scoring. Mean aggregation outperforms minimum and maximum on Micro Top1 F1 (56.79% vs. 47.10% / 55.73%), Micro MRR-perfect@16 (.5403 vs. .4768 / .5271), and fused oracle F1@8 (88.61% vs. 88.03% / 88.34%), though maximum yields slightly higher deterministic fused Top1 F1 (65.20% vs. 64.99%). Dataset breakdowns are provided in Table 22 (p. 29).

Regarding Q3: Eight-Card Window, Fusion, and Dynamic Allocation

The 8-card window balances candidate coverage against selector context length. We analyze the exact bottlenecks, alternative window sizes, score-threshold windows, and dynamic track allocations across all 6,159 questions (Appendix F.6, pp. 28–30, Tables 23–24).

1. Where candidates are lost. For the 27B source, oracle F1 / F1-perfect coverage drops from 90.92% / 84.51% in the bounded pre-fusion union (first 16 raw candidates per track) to 90.73% / 84.27% in the 16-card fused pool (−15 F1-perfect questions), and to 88.61% / 81.77% at the 8-card cutoff (−154 F1-perfect questions; −17 and −179 for 0.8B).

2. Fixed window sizes (4/8/12/16) vs. score-threshold variable windows. Evaluating prefixes of the frozen fused order shows diminishing coverage gains past 8 cards:

Window policy0.8B mean cards0.8B oracle F1 (%)27B mean cards27B oracle F1 (%)
Fixed 43.9980.093.9983.55
Fixed 87.8385.057.8888.61
Fixed 1211.1786.8111.4890.17
Fixed 1613.9687.5114.8090.73
Threshold τ=0.606.9083.917.0988.27
Threshold τ=0.758.6485.399.0189.22

Moving 4 → 8 adds ~5.0 pp oracle F1, whereas 8 → 12 adds +1.76/+1.56 pp (0.8B/27B) at +43%/+46% more cards, and 12 → 16 adds +0.69/+0.56 pp. For variable-length windows, retaining all consensus cards and filtering others by within-track normalized score si≥smax⁡−τ(smax⁡−smin⁡)s_i \ge s_{\max} - \tau(s_{\max}-s_{\min}) (Table 23) trades mean card count for coverage without matching fixed-8's worst-case cap (P95 is 13–14 cards at τ∈0.60,0.75\tau \in \\{0.60, 0.75\\} vs. 8). Dataset effects also diverge (e.g., at τ=0.60\tau=0.60 on 27B, GrailQA saves cards with 88.53% vs. 88.40% oracle F1, whereas CWQ drops from 88.57% to 87.95%).

3. Dynamic allocation within 8 cards. Track strengths differ sharply by dataset: 27B Micro vs. Macro F1-perfect coverage is 62.93% vs. 77.06% on CWQ, but 82.60% vs. 25.20% on GrailQA (and 89.68% vs. 2.49% on GrailQA zero-shot). Under a fixed 8-card cap, neither fixed 0/8, 2/6, 4/4, 6/2, 8/0 core quotas (all ≤88.58% vs. 88.61% for 27B) nor two question-dependent heuristics allocating 4 remaining slots by distinct answer-set counts or score support (27B: 88.29%/88.33% oracle F1) improve overall oracle F1 over the original policy. Given these coverage–context tradeoffs, we retain the original 8-card policy.

Reviewer_Y6aCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ONTOS is a candidate-first approach to knowledge graph question answering. An LLM sketches the question without using graph identifiers; two ontology-constrained synthesis tracks produce executable candidate programs, which the LLM then selects from. Low-confidence selections can be repaired. Across 6,159 questions from CWQ, WebQSP, and GrailQA, the authors report strong Hits@1 and Macro-F1 results. Ablations suggest that the two synthesis tracks are complementary and that repair mainly improves Macro-F1 with larger backbones.

Critical strengths

S1. The components have a clear division of labor. Separating conceptual planning, ontology-constrained program construction, and candidate selection lets the two synthesis tracks cover both familiar structures and less common paths.

S2.The evaluation is thoughtfully analyzed. The paper tests multiple datasets and model sizes, and its ablations and error analysis help show where each stage succeeds and fails.

Critical weaknesses

W1. The reported gains over baselines are not yet confirmed in a controlled comparison. Differences in models, prompts, or implementation may affect the results, particularly the large margins highlighted in the paper.

W2. Reproducibility and cost claims need stronger support. The full prompts, examples, and code are not included in the submission. The token accounting also omits repeated history during repair, and runtime is not reported, so the practical efficiency advantage is difficult to assess.

Questions for authors
  1. Could you rerun the key baselines with the same models and settings, or provide per-question results that let readers verify the main performance gaps?

  2. Do the efficiency comparisons still hold when repeated conversation history is included in token counts and actual runtime is reported? This could affect the practical assessment of the method.

  3. Could you provide the complete prompts, demonstrations, code, and per-question predictions to support reproduction of the main results?

5 comments
AuthorsOpenReview ↗

Regarding W1 / Q1: Same-Backbone Baseline Comparison

Thank you for highlighting the need for a same-backbone comparison and clearer cost accounting. We retain the literature-reported baseline scores in Table 1, including the ToG/PoG/CoG entries taken from the CoG paper under their original evaluation protocols, and now add a same-backbone (Qwen2.5-7B), same-6,159-question comparison against our reproduced CoG (7B), evaluated with identical public question text, verified topic MIDs/names, and one common answer-hit and set-level Macro-F1 evaluator (Section 4.1–4.2, pp. 6–7; Appendix F.9, pp. 32–33, Tables 30–31).

1. Same-backbone CoG comparison across all 6,159 questions (Table 30).

DatasetnCoG (reproduced) H@1 / F1 (%)ONTOS selection H@1 / F1 (%)ONTOS final H@1 / F1 (%)
CWQ3,53149.48 / 41.6982.67 / 72.7082.41 / 73.31
WebQSP1,62876.66 / 55.5387.78 / 75.5386.98 / 75.94
GrailQA1,00072.40 / 63.4580.30 / 76.7180.50 / 77.28
All6,15960.38 / 48.8883.63 / 74.1083.31 / 74.65

With identical 7B backbone, questions, topic MIDs, and evaluator, ONTOS outperforms the reproduced CoG by +22.93 Hits@1 and +25.77 F1 overall (+32.93/+31.62 on CWQ, +10.32/+20.41 on WebQSP, +8.10/+13.83 on GrailQA). Specifically, we ran CoG's ACL 2026 public codebase (https://github.com/zjukg/CoG) with minimal fixes (search depth 4, 50-call budget, fixing a missing variable assignment in the normal-answer refinement branch).

2. Evaluator sensitivity vs. reproduction gap (Table 31). To separate evaluator choice from prediction quality, we also scored the exact same reproduced CoG predictions with CoG's released evaluation loop:

Dataset (n)CoG paper H@1 / F1Reproduced CoG, CoG native evalSame local predictions, common eval
CWQ (3,531)54.00 / 48.5049.50 / 44.4049.48 / 41.69
WebQSP (1,628)74.50 / 63.9076.66 / 65.9476.66 / 55.53
GrailQA (1,000)72.00 / --72.60 / 71.6572.40 / 63.45

Switching evaluators changes H@1 on only 1 CWQ and 2 GrailQA questions (0 on WebQSP), whereas F1 differs substantially because CoG's native F1 uses different splitting, alias accounting, and recall behavior. Even under CoG's own evaluator, local CWQ remains 4.50 H@1 points below the paper while WebQSP and GrailQA are +2.16 and +0.60 points above. These modest dataset-level shifts reflect multi-turn LLM decoding variance across sequential calls under concurrent serving; even against CoG's paper-reported H@1 (54.00 / 74.50 / 72.00), ONTOS still leads by +28.41 / +12.48 / +8.50 points. We also add same-question reassessments of GRASP's public outputs (Appendix F.8, Table 28: 16.05% vs. 73.59% F1 on 199 WebQSP questions with Qwen2.5-7B), noting that GRASP discovers entities itself while ONTOS and CoG receive topic MIDs and training templates.

AuthorsOpenReview ↗

Regarding W2 / Q2: Repeated History, Token Accounting, and Graph Workload

We agree that the original history-deduplicated token count in Table 7 is a content-accounting statistic and cannot support a transmitted-token efficiency claim. We have relabeled Table 7 and Section 4.4 (p. 9) accordingly, and added a primary successful-request token analysis (including every repeated system prompt, demonstration, and conversation prefix, without subtracting cached tokens; Appendix F.10, pp. 33–34, Tables 32–33) alongside a separate Freebase graph/name workload breakdown (Appendix F.11, p. 34, Table 34).

For the original full repair pipelines, original 9B/27B logs record 20,442.3 / 19,496.5 transmitted tokens/question (vs. 15,238.1 / 15,223.8 deduplicated), confirming that repeated repair history adds substantial input volume. To examine transmitted-token efficiency directly on Qwen2.5-7B across all 6,159 questions, we also evaluate compressed prompts using train-only demonstrations (reducing sketch demonstrations 10 → 4 from CWQ/GrailQA train and selector demonstrations 4 → 2 of 3 cards each from CWQ train, while keeping all real evaluation cards unchanged):

Qwen2.5-7B configuration (n=6,159)Successful calls/qInput/qOutput/qTotal/qH@1 / F1 (%)
Original sketch + original selector (no repair)2.0012,612.99255.6812,868.6783.63 / 74.10
Original sketch + compact selector (no repair)2.006,037.13189.566,226.6983.28 / 74.18
Compressed sketch + compact selector (no repair)2.003,117.37128.073,245.4482.79 / 73.06
CoG (reproduced, includes search/refinement)9.785,695.61258.615,954.2160.38 / 48.88

Note on scope: All ONTOS rows above cover sketch + selection excluding repair, whereas CoG (reproduced) includes its own full search and refinement. Compression reduces two-stage tokens by 74.78% vs. the original two-stage prompts and by 45.49% vs. reproduced CoG, at a loss of −1.04 F1 points overall (GrailQA −3.69 pp; CWQ −0.73 pp; WebQSP −0.06 pp; Table 33).

Freebase graph and name-lookup workload (Table 34, p. 34). Fewer LLM calls do not mean fewer database queries: ONTOS shifts search work from the LLM to the graph engine. On the compressed 7B pool across all 6,159 questions, optimized Micro and Macro use 69.11 and 26.40 logical graph queries/q (95.51 total, already including candidate execution and constraint/type probes), vs. 9.20 for reproduced CoG (CWQ: 121.19 vs. 11.79; WebQSP: 74.16 vs. 5.49; GrailQA: 39.63 vs. 6.08). Name lookups are tracked separately in their native units (ONTOS: 15.31 Micro batches/q + 8.05 Macro API calls/q; reproduced CoG: 6.90 name queries/q).

Per-question latency breakdown (TSPARQL+TLLMT_{\text{SPARQL}} + T_{\text{LLM}}). On our local Virtuoso endpoint, individual SPARQL queries take 3.34 ms (Micro-track) and 6.31 ms (Macro-track) on average (2.69–8.36 ms for name lookups); even executed strictly serially, all graph and name queries in ONTOS sum to only 0.50 s of Freebase query time per question across the 6,159 questions (0.49 s on CWQ, 0.68 s on WebQSP, and 0.23 s on GrailQA, vs. ~0.06 s for CoG). Because local SPARQL lookups operate at millisecond latency whereas iterative agents are bottlenecked by multi-round LLM prefill and decoding (e.g., 10.5–96.5 s/question reported for RJE, PoG, and ToG), trading +0.44 s of local database queries to reduce LLM invocation from 9.78 serial calls / 5,695.6 input / 258.6 output tokens in reproduced CoG (Tables 32–33) down to 2.00 calls / 3,117.4 input / 128.1 output tokens (4.9$\times$ fewer LLM rounds and ~2$\times$ fewer input and output tokens) substantially lowers LLM latency: under vLLM batching across all 6,159 questions, the two LLM stages in compressed ONTOS require only 305 s of total LLM inference time, compared with 985 s (3.2$\times$ longer) for reproduced CoG.

AuthorsOpenReview ↗

Regarding W2 / Q3: Prompts, Demonstrations, Code, and Per-Question Predictions

We have prepared a supplementary package containing the complete compact sketch and compact selector instructions and all their train-only demonstrations, full JSON output schemas, and actual sketch/selector request examples with their outputs. The instructions and demonstrations are also reproduced in Appendix F.12 (pp. 34–37), alongside the original prompt excerpts in Appendix B and pseudocode in Appendix A.

Per-question predictions and verification. The package covers the six original backbones, the two 27B track ablations, both new compact-selector configurations, and our same-7B CoG rerun: 11 configurations, each with the same 6,159 questions. Original-system records include selected/final answers and SPARQL programs, confidence and repair-routing metadata; the CoG export includes raw final text and parsed answers. We also provide individual scores, the frozen evaluation references, and separate CWQ, WebQSP and GrailQA summaries so that readers can check the comparisons directly.

How to use the material. A four-page PDF guide explains the file layout, evaluation rule, and model/database workload accounting. The package provides prompts and experimental outputs without source code; we will publicly release the source code upon acceptance.

We hope these additional experiments, cost breakdowns, and clarifications address your remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion for the Area Chair and Reviewers

Thank you to the Area Chair and reviewers for your constructive feedback. We updated the manuscript with controlled experiments in Appendix F (pp. 25–37, in blue) and a redesigned Figure 2, provided full prompts and per-question predictions in the supplementary material, and addressed every point in the individual threads.

Discussion status (3 / 3 / 2). 4b9h raised their score from 2 to 3. Y6aC confirmed the new experiments addressed most concerns and maintained 3. LRpg noted the new results substantially address their concerns and indicated willingness to raise their rating if cross-dataset generalization and Figure 2 readability were improved; both are now directly addressed in our latest reply and revised PDF.

Why ONTOS merits acceptance. Step-by-step LLM exploration on large KGs suffers from premature local decisions that prune viable paths and compound errors. ONTOS defers LLM commitment to 2.00 LLM calls/question before repair: one LLM call produces a global conceptual sketch without graph IDs, dual-track ontology-constrained symbolic synthesis grounds it into executable programs, and one LLM call selects among executed candidates. Across 6,159 questions from three distinct benchmarks without fine-tuning, ONTOS with Qwen3.5-27B achieves 89.2% H@1 / 81.2% F1 on CWQ, 90.1% / 80.4% on WebQSP, and 87.1% / 84.6% on GrailQA, including 90.4% / 88.8% on 562 unseen-schema zero-shot questions. On identical Qwen2.5-7B-Instruct inputs, ONTOS beats reproduced CoG by +22.93 H@1 / +25.77 F1 with 4.9× fewer LLM calls, 45.5% fewer transmitted tokens under compact prompts, and 3.2× less vLLM time.

Key concerns resolved during rebuttal:

  1. Controlled baselines, efficiency, and reproducibility (4b9h W2/Q2, Y6aC W1–W2): On identical Qwen2.5-7B-Instruct, 6,159 questions, topic MIDs, and evaluator, ONTOS beats reproduced CoG at 83.31% vs. 60.38% H@1 / 74.65% vs. 48.88% F1 (+32.93/+31.62 CWQ, +10.32/+20.41 WebQSP, +8.10/+13.83 GrailQA), plus native-evaluator checks (App. F.9, Tables 30–31). Under the same evaluator, ONTOS 7B leads Interactive-KBQA's GPT-4.1 public run across 2,193 shared questions (+29.40/+29.61 on 500 CWQ, +1.48/+5.14 on 1,621 WebQSP, +11.11/+12.20 on 72 GrailQA) and GRASP's public outputs (+64.32/+57.54 on 199 WebQSP; +8.90/+9.38 on 191 CWQ vs. GRASP-F GPT-4.1 mini; App. F.8, Tables 27–29). We report full transmitted tokens, prompt compression cutting tokens by 74.78% vs. original and 45.49% vs. CoG at 82.79% / 73.06% (App. F.10, Tables 32–33), 0.50 s/q Freebase latency, 305 s vs. 985 s vLLM time (App. F.11, Table 34), verbatim prompts (App. F.12), and 11-config predictions.

  2. Cross-dataset generalization and redesigned Figure 2 (LRpg follow-up): One identical training-free pipeline generalizes across three distinct distributions: WebQSP with 1–2 hop search queries, CWQ with up to 4-hop logical and comparative reasoning, and independently crowdsourced GrailQA across 86 domains and 3,720 relations with 562 unseen-schema zero-shot questions (App. C–D). Removing both training templates and repair retains 85.2% / 82.0% on GrailQA versus 32.1% / 26.3% and 8.4% H@1 on zero-shot for Macro-only (Sec. 4.3, Table 2), confirming ontology-grounded synthesis drives generalization. We also redesigned Figure 2 in the updated PDF, adding concise LLM reasoning rationales across planner, selector, and repair stages and streamlining the dual-track diagram.

  3. Planning and selection beyond matching and templates (4b9h W1/Q1): A 6,159-question question-only control (App. F.2, Tables 13–14) shows removing the LLM sketch increases graph calls 4.81× (415.32 vs. 86.35/q) and drops 0.8B selector F1 from 65.75% to 61.74%. Crossing planner and selector from 0.8B to 27B (Table 15) isolates planner (+3.2–3.4 F1) and selector (+10.1–10.2 F1) gains.

  4. Sketch quality, repair scope, and pairwise gate (LRpg W2, Q1, Q4): A 23-case audit confirms initial synthesis grounds sketch semantics while repair handles omitted constraints (App. F.3, Table 16). Core construction and selection reach 88.49% / 79.19% at 27B and 83.63% / 74.10% at 7B before repair (App. F.7, Tables 25–26). Optional training-free repair turns net positive from Qwen3.5-4B onward, and the both-order pairwise gate beats no-gate on all six backbones.

  5. Individual ontology operations and search/window controls (LRpg W1, Q2–Q3): Full 6,159-question ablations (App. F.4, Tables 17–20) isolate each constraint: inverse folding preserves 32 F1-perfect@16 questions and cuts Micro calls from 99.85 to 71.19/q; slot-type prechecks cut original Macro executions by 69.95%; system filtering and atomic CVT expansion further reduce calls. Beam-width, hop-scoring, and 8-card window controls (App. F.5–F.6, Tables 21–24) confirm saturation at K=64–128 and diminishing gains past eight cards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.