- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies premature clarification termination in agentic Text-to-SQL: an agent may identify relevant ambiguities yet submit SQL without resolving them, sometimes obtaining the correct execution result through unverified assumptions. It proposes PlanPool, a persistent, dynamically editable clarification pool whose pending questions must be asked or explicitly dropped before submission, and evaluates it on three benchmarks derived from BIRD-Interact and Spider, reporting improved ambiguity coverage and grounded success with competitive execution accuracy.
Critical strengths
PlanPool requires no additional model training and integrates external state management with an explicit submission constraint. Within the shared evaluation harness, it improves ambiguity recall and grounded success over Plan-Prompt on all three datasets.
Critical weaknesses
- The groundedness metrics measure question coverage rather than resolution.
An ambiguity is counted as addressed when the simulator’s encoder maps a clarification question to its annotation (Section 2, Appendix B); whether the answer was sufficient, or was reflected in the final SQL, is not checked. The paper’s own numbers suggest that this gap is substantial. Since the four outcome categories partition the instances, Table 2 implies the unreported fourth category: "all annotated ambiguities addressed but incorrect SQL"(100 - EA - SF).
On BIRD-Full, this is 24.5% (100 - 67.1 - 8.4) for PlanPool versus 19.1%(100 - 69.8 -11.1) for Plan-Prompt: roughly one third of PlanPool’s fully covered trajectories fail, even though instances were filtered to be solvable once all annotated clarifications are provided.
GS and US are not independent: since EA = GS + US, higher GS automatically means lower US when EA stays the same. However, a correct answer without a matched clarification question is not necessarily a lucky guess. The agent may have obtained the needed information from the datasource, or the encoder may have missed a valid clarification. Interpreting US as ungrounded guessing therefore requires checking both whether user clarification was necessary and whether the question-to-annotation mapping was accurate.
- The experiments do not isolate the sources of improvement.
Table 3 reports unmodified pools in 65.5% of BIRD-Lite and 61.8% of BIRD-Full instances, but does not quantify the benefit of adding or dropping questions. Table 2 does not isolate the effects of explicit external state, requiring an empty pool before submission, and pool modifications. Although Section 3.1 studies fixed question budgets, PLANPOOL is not compared with baselines at matched budgets.
On BIRD-Full, PlanPool has lower observed EA than Plan-Prompt (67.1% versus 69.8%) despite asking more questions. Although Section 5.2 acknowledges that better coverage need not improve EA, a failure analysis and uncertainty estimates are needed to assess this trade-off.
Questions for authors
-
Grounding evaluation: Table 2 implies that approximately 24.5% of all BIRD-Full instances under PlanPool have full annotation coverage but incorrect SQL. Can the authors break down these failures into incomplete clarification? Please address this question for all three datasets.
-
Clarification necessity: Can the authors further analyze which questions require clarification to produce correct SQL and which do not affect the answer? Further discussion is possible regarding case studies and quantitative analysis.
-
Source of performance gains: How much of PLANPOOL’s improvement comes from its pool mechanism rather than asking more questions? Can the authors compare it with Plan-Prompt and fixed-budget baselines using the same question budget?