- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper adapts the KBQA‑R1 agent‑reinforcement‑learning paradigm to incomplete knowledge‑base question answering (IKBQA).
The authors formulate incomplete‑KBQA as a multi‑turn Markov decision‑making process. Building upon the existing KBQA‑R1 pipeline (RRS trajectory sampling, multi‑turn SFT cold‑start, GRPO reinforcement learning, S‑Expression executable reasoning), they add two new atomic actions: Search Web (mapped to WEB‑JOIN) for retrieving external evidence and Combine (mapped to UNION) to merge knowledge‑graph outputs with external retrieved results into a unified executable logical form that supports downstream reasoning operations. Two new benchmarks WebQSP‑IKG and GraphQ‑IKG are constructed by controlled hiding of KG triples to simulate knowledge incompleteness for multi‑turn agent training and evaluation. Empirical results show clear F1 improvements over prior incomplete‑KBQA baselines (GoG, DoM) on their curated benchmarks.
Critical strengths
- Solid executable integration of external evidence
- Action‑aware credit assignment is a useful incremental RL improvement
- New task‑specific benchmarks for trainable IKBQA agents
- Strong empirical performance on the proposed benchmarks
Critical weaknesses
- Nearly the full training pipeline is inherited from KBQA‑R1: multi‑turn MDP formulation, RRS referenced‑rejection sampling, hint‑free multi‑turn SFT, base GRPO setup and S‑Expression execution engine. Major modifications are confined to two new atomic actions plus the GRPO credit‑assignment tweak.
- There is no ablation where the pre‑trained KBQA‑R1 model is directly prompted with descriptions of
Search‑Web/Combinewithout re‑training via SFT+RL. This experiment would help quantify how much performance gain comes purely from prompt engineering versus the full re‑training pipeline. - All external evidence is pre‑cached offline, rather than retrieved from open‑web corpora at runtime. The setup avoids real‑world challenges: open‑domain retrieval noise, entity‑relation alignment failures, cache misses.
Questions for authors
Please see above weaknesses. The limited novelty of the whole framework and the missing comparison with zero‑shot prompting of original KBQA‑R1 are major concerns.
6 comments
**[Response to Weakness 1(b)] Clarifying Contributions beyond the Inherited Training Framework**
Table 6: Trajectory acceptance and SFT initialization F1 (%).
\text{Dataset} & \text{Pipeline} & \text{Accepted/Total} & \text{Acceptance (\\%)} & \text{SFT Init F1} \\\\ \hline \text{WebQSP} & \text{KBQA-R1} & 1505/2929 & 51.4 & 72.1 \\\\ \text{WebQSP-IKG} & \text{IKBQA-R1} & 2213/2929 & 75.6 & 69.2 \\\\ \hline \text{GraphQ} & \text{KBQA-R1} & 1562/2332 & 67.0 & 47.8 \\\\ \text{GraphQ-IKG} & \text{IKBQA-R1} & 1626/2220 & 73.2 & 41.7 \\\\ \end{array}$$ **Table 8: Textual versus executable evidence integration on 256 WebQSP-IKG test questions.** $$\begin{array}{lllrrr} \text{Evidence integration} & \text{Setting} & \text{Model} & \text{F1 (\\%)} & \text{LLM calls} & \text{Tokens In / Out} \\\\ \hline \text{Executable program} & \text{IKBQA-R1 harness} & \text{GPT-5.6-Sol} & 72.39 & 4.82 & 13{,}502 / 622 \\\\ \text{Textual context} & \text{Adapted harness} & \text{GPT-5.6-Sol} & 65.07 & 4.12 & 7{,}834 / 638 \\\\ \Delta\text{ (textual - executable)} & & & -7.32 & -0.70 & -5{,}668.1 / {+}15.5 \\\\ \hline \text{Executable program} & \text{IKBQA-R1 harness} & \text{GPT-6-Sol} & 72.42 & 5.03 & 14{,}352 / 513 \\\\ \text{Textual context} & \text{Adapted harness} & \text{GPT-6-Sol} & 62.02 & 4.62 & 9{,}350 / 659 \\\\ \Delta\text{ (textual - executable)} & & & -10.39 & -0.41 & -5{,}001.6 / {+}146.0 \\\\ \hline \text{Executable program} & \text{IKBQA-R1 (RL)} & \text{Llama-3.1-8B} & 73.79 & 5.21 & 14{,}482 / 592 \\\\ \text{Textual context} & \text{IKBQA-R1 (RL)} & \text{Llama-3.1-8B} & 49.91 & 6.23 & 11{,}716 / 701 \\\\ \Delta\text{ (RL textual - executable)} & & & -23.88 & +1.02 & -2{,}765.9 / {+}109.2 \\\\ \end{array}$$ **Table 15: Effect of local credit allocation on answer F1 (%) for WebQSP-IKG and GraphQ-IKG.** $$\begin{array}{lrr} \text{Credit assignment} & \text{WebQSP-IKG} & \text{GraphQ-IKG} \\\\ \hline \text{Trajectory-level GRPO} & 71.4 & 42.4 \\\\ \text{Uniform local allocation} & 73.5 & 46.2 \\\\ \text{Aligned action-aware allocation} & 75.1 & 49.9 \\\\ \end{array}$$ **Table 5: Average LLM calls per sample during inference. Lower values indicate fewer policy decisions.** $$\begin{array}{lrr} \text{Method} & \text{WebQSP} & \text{GraphQ} \\\\ \hline \text{DoM} & 12.33 & 15.06 \\\\ \text{KBQA-o1 (Complete KG)} & 28.8 & 78.0 \\\\ \text{KBQA-R1 (Complete KG)} & 2.65 & 3.16 \\\\ \text{IKBQA-R1 (Incomplete KG)} & 4.77 & 5.35 \\\\ \end{array}$$ **Table 7: F1 and inference costs of generalist models and learned policies in IKBQA-R1. “Harness” uses the IKBQA-R1 action space and feedback format with an off-the-shelf model, while IKBQA-R1 uses the learned RL policy.** $$\begin{array}{llrrrrrrrr} \text{Harness} & \text{Model} & \text{F1 (\\%)} & & & \text{Turns} & & & \text{Tokens In / Out} & \\\\ & & \text{WebQSP} & \text{GraphQ} & \text{Avg.} & \text{WebQSP} & \text{GraphQ} & \text{Avg.} & \text{WebQSP} & \text{GraphQ} \\\\ \hline \text{IKBQA-R1 Harness} & \text{GPT-5.6-Sol} & 68.22 & 54.59 & 61.41 & 4.92 & 5.67 & 5.36 & 14{,}338 / 659 & 16{,}587 / 988 \\\\ \text{IKBQA-R1 Harness} & \text{GPT-6-Sol} & 66.16 & 51.14 & 58.65 & 5.18 & 5.90 & 5.60 & 15{,}185 / 535 & 17{,}483 / 763 \\\\ \hline \text{IKBQA-R1 Harness} & \text{DeepSeek-V4.1-Flash} & 54.04 & 49.54 & 51.79 & 5.32 & 5.76 & 5.57 & 17{,}097 / 476 & 18{,}674 / 683 \\\\ \text{SFT policy} & \text{Llama-3.1-8B} & 69.17 & 41.73 & 55.45 & 5.21 & 4.73 & 4.93 & \text{—} & \\\\ \text{IKBQA-R1} & \text{Llama-3.1-8B} & 75.05 & 49.93 & 62.49 & 4.77 & 5.35 & 5.11 & 12{,}550 / 347 & 13{,}763 / 773 \\\\ \end{array}$$[Response to Weakness 2(a)] Tool Descriptions and the Contribution of Policy Training
Thank you for suggesting a comparison between tool prompting and policy training.
Descriptions of Search_Web and Combine explain how the tools are used, but attributing performance gains to prompting requires comparisons with policy training. We provide evidence through the same-environment comparison of generalist models, SFT, and RL in Table 7, together with training-stage ablations in Table 4. Same-backbone SFT-to-RL gains are 5.88/8.20 F1 points on WebQSP-IKG/GraphQ-IKG, showing that further environment-interaction training improves performance after an initial tool-use policy has been established.
The KBQA-R1 checkpoint has been specialized through SFT and RL for interaction over complete KGs, with a training action space that excludes Search_Web and Combine. New tool descriptions provide operational rules, while our training further optimizes the policy for these actions and subsequent execution. To examine these roles, we report generalist models directly using the tool interface and same-backbone training comparisons within our task.
In main-text Table 7, DeepSeek-V4.1-Flash, GPT-5.6-Sol, and GPT-6-Sol act directly as agents without additional task-specific training. All models share the tools, execution feedback, KG, evidence cache, and eight-turn interaction limit. Each generates one trajectory per question without gold reference-action hints. This setting allows generalist models to use their existing capabilities and tool descriptions, while the same 8B backbone's SFT and RL results characterize gains from further policy optimization.
The performance results show that task training provides further gains after tool descriptions are supplied. On WebQSP-IKG, DeepSeek-V4.1-Flash, GPT-5.6-Sol, and GPT-6-Sol obtain 54.04, 68.22, and 66.16 F1, respectively. Llama-3.1-8B reaches 69.17 after SFT and 75.05 after RL, exceeding the two GPT models by 6.83 and 8.89 percentage points. Same-backbone SFT-to-RL improvements are from 69.17 to 75.05 on WebQSP-IKG and from 41.73 to 49.93 on GraphQ-IKG, gains of 5.88 and 8.20 points. On GraphQ-IKG, DeepSeek, GPT-5.6-Sol, and GPT-6-Sol obtain 49.54, 54.59, and 51.14, respectively; the trained 8B obtains 49.93, slightly above DeepSeek but below both GPT models. Its two-dataset average F1 is 62.49, compared with 61.41 for GPT-5.6-Sol, 58.65 for GPT-6-Sol, and 51.79 for DeepSeek. In this two-dataset comparison, the trained 8B policy therefore has higher average F1 than the three direct generalist agents, while retaining the performance differences across datasets.
Table 4 further examines training stages within the task. SFT alone obtains 69.2/41.7 F1 on WebQSP-IKG/GraphQ-IKG; RL without SFT initialization obtains 70.5/43.5; full training obtains 75.1/49.9. Combining SFT initialization and subsequent RL outperforms either stage alone on both datasets, providing same-backbone evidence for the training procedure.
The performance gains are not obtained through more interaction turns. The RL policy uses 4.77 and 5.35 turns on WebQSP-IKG and GraphQ-IKG, respectively. GPT-5.6-Sol uses 4.92 and 5.67, GPT-6-Sol uses 5.18 and 5.90, and DeepSeek uses 5.32 and 5.76. The trained policy averages fewer interaction turns than all three generalist models on both datasets.
For token usage in Table 7, the RL policy's WebQSP-IKG input/output counts are 12,550/347, compared with 14,338/659 for GPT-5.6-Sol, 15,185/535 for GPT-6-Sol, and 17,097/476 for DeepSeek. On GraphQ-IKG, the RL policy uses 13,763/773, compared with 16,587/988, 17,483/763, and 18,674/683 for the three generalist models in the same order. The trained policy uses fewer input tokens on both datasets and fewer output tokens on WebQSP-IKG. Its GraphQ-IKG output count is below GPT-5.6-Sol, close to GPT-6-Sol, and above DeepSeek. The trained policy thus maintains fewer interaction turns and input tokens, while output-length differences depend on the dataset and comparison model.
Figure 14 (Appendix D.3) and Figure 15 (Appendix D.5) further characterize behavioral changes after training. Figure 14 shows improved outcome categories for 203 WebQSP-IKG questions versus declines for 95, and improvements for 399 GraphQ-IKG questions versus declines for 211. In Figure 15, nonempty-evidence coverage on GraphQ-IKG increases from 36.12% to 69.13%, and successful-integration coverage from 31.24% to 65.32%. These results connect same-backbone SFT-to-RL F1 gains with improvements in evidence acquisition and integration, supporting further task-policy training beyond tool descriptions.
We appreciate this suggestion, which helped organize the evidence on tool interfaces and task-specific training.
[Response to Weakness 1(a)] Clarifying Contributions beyond the Inherited Training Framework
Thank you for identifying the training components inherited from KBQA-R1.
Initializing a policy with supervised trajectories and then applying reinforcement learning through environment interaction is a general approach to training multi-turn tool-use policies. We adopt KBQA-R1's specific training framework, retaining its multi-turn MDP formulation, RRS, hint-free multi-turn SFT, base GRPO, and S-Expression execution foundations. Our method turns external evidence into program state that supports continued computation and uses action structure to refine GRPO's token-level advantage allocation. The interface comparison in Table 8 and allocation-rule comparison in Table 15 provide performance evidence for these two designs, respectively.
First, we construct data and an environment for policy learning. We replay gold programs on the complete KG, collect direct triples and CVT evidence for relation steps, hide supporting facts at one selected step, and produce a reference program with external retrieval and integration. The same rules apply to training and test splits: Table 10 reports 2,929/1,566 examples for WebQSP-IKG and 2,220/2,151 for GraphQ-IKG. RRS uses reference-action hints to prepare warm-start trajectories, with acceptance rates of 75.6%/73.2% (Table 6). Hints are removed from SFT dialogues and are absent from RL policy inputs and test inputs; gold annotations remain available internally for training rewards and action alignment. The sampling–SFT–RL procedure is inherited, while the data, environment, and execution supervision support learning under KG incompleteness.
Second, external evidence becomes part of a program state that supports continued execution.
Search_Webcreates aWEB_JOINbranch, andCombineintegrates it with the KG branch throughUNION, enabling further queries, intersections, filters, ordering, and counting. Figure 9 (Appendix C.1) shows continued city querying after retrieving an intermediate entity, yielding Fairfax; Figure 20 (Appendix G.1) shows subsequent type filtering and MAX ordering. In Table 8, executable integration improves the two GPT models by 7.32/10.39 F1 over textual context with shared questions and evidence, alongside the corresponding call and token costs. These results support using external evidence as input to subsequent program operations.To maintain this continuity, Lineage-Preserving Lazy Execution in Appendix E.2 retains the KG branch's source expression, relation path, direction, hidden-edge exclusions, and subsequent constraints alongside external results. The model receives a bounded preview, while later execution and trajectory rewards use the complete semantics, avoiding treating a truncated preview as the full entity set. This also supports filtering, intersection, and counting after high-fan-out queries. Precomputation and on-demand loading of large JOINs further reduce repeated-query costs; lazy state maintenance and query acceleration respectively support execution correctness and efficiency. Figure 19 (Appendix E.1) describes external-evidence construction and runtime integration.
Third, action-aware credit assignment refines base GRPO. Algorithm 3 (Appendix B.5) monotonically aligns generated actions with training gold-action anchors, allowing skips on both sides. Local quality based on tools, relations, and arguments is normalized by span length, projected to action and paired reasoning tokens, and incorporated through bounded advantage refinement. Trajectory rewards and group normalization are preserved; unmatched actions retain the trajectory-level advantage. Table 15 reports 71.4/42.4 F1 for base GRPO, 73.5/46.2 for uniform local allocation, and 75.1/49.9 for aligned allocation. Local supervision first adds 2.1/3.8 points, and action-specific allocation adds another 1.6/3.7. Figure 10 (Appendix C.2) illustrates allocation differences with fixed total credit, while the table provides performance evidence.
Together, these designs support reasoning quality and interaction efficiency. Table 5 reports 4.77/5.35 calls for the trained 8B policy on WebQSP/GraphQ, compared with 12.33/15.06 for DoM. In the same-environment comparison of Table 7, the 8B policy reaches 75.05 F1 on WebQSP versus 68.22 and 66.16 for the two GPT models, while using fewer input tokens. Building on the inherited training framework, our additions provide traceable data for policy learning, executable integration and lazy state maintenance for continued computation, and action-level credit for optimizing multi-step decisions.
We appreciate this observation, which helped distinguish the inherited framework from our execution and credit-assignment designs.