- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STIR-SQL trains a tool-using text-to-SQL model with reinforcement learning so that schema inspection, SQL generation, execution, and revision occur within one interactive trajectory. It combines database descriptions, four tools, and a composite reward. The system reports 69.7% execution accuracy on BIRD-dev, 88.7% on Spider-test, and 79.1% on Spider-DK. The supplement includes an untrained, same-backbone comparison of interaction designs and reports 23.7 seconds average inference latency for STIR-SQL.
Critical strengths
- The paper targets a consequential limitation of fixed schema-linking pipelines: an early grounding error can constrain subsequent SQL generation. Giving the policy access to database observations throughout inference is a sensible way to address it.
- The system achieves a strong reported BIRD-dev result with a 3B-active-parameter model. The ablations, self-correction analysis, and same-backbone supplementary comparison go beyond a leaderboard claim and give some evidence that the tool and reward choices matter.
Critical weaknesses
- The 3.5-point drop when interleaved reasoning is removed also removes multi-turn tool access, so it is not an estimate of the value of learning a unified trajectory. The supplement offers a more relevant same-backbone comparison, but it uses an untrained DeepSeek-V3.2 model rather than the trained STIR-SQL policy and changes the available tools or schema-handling procedure across conditions. Its 0.8–1.2-point differences therefore support the complete inference design, but do not establish that trajectory organization, rather than the changed affordances, explains the gain. This distinction matters because the paper's contribution is framed as trajectory learning, not simply a well-performing combination of tools and rewards.
- The supplement reports 23.7 seconds for STIR-SQL and 94.1 seconds for CHESS, but does not establish a common measurement protocol for model serving, hardware, token generation, and database/tool latency. The manuscript's “one LLM call” description is also difficult to reconcile with the supplement's 4.37 average generation calls; one trajectory need not be one model invocation. The latency figures are useful descriptive results, but they do not yet demonstrate that unifying the trajectory itself causes the claimed reduction in end-to-end cost.
Questions for authors
- Is there a comparison using the trained STIR-SQL backbone and the same training data, reward, schema descriptions, tools, and inference budget, with only the staged versus interleaved organization changed? Paired per-question results would make clear whether the claimed trajectory-learning contribution survives that control.
- Were the latency results measured on a common hardware and model-serving setup, with model generation and tool/database time included for every method? Please clarify what counts as the claimed single LLM call relative to the 4.37 average generation calls.
- Was the released reward implementation used to train the evaluated checkpoint? The manuscript describes valid tool calls rewarded up to 1.0, whereas the supplied code counts tool-call tags and caps this term at 0.5; resolving this would clarify which training objective the results evaluate.