acceptodds
Withdrawn from ICLR 2027

Asking What Matters: Disagreement-Guided Ambiguity Detection for Interactive Text-to-SQL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive text-to-SQL requires an LLM agent not only to generate executable SQL, but also to decide which uncertainty to clarify within a limited interaction budget. However, current clarification policies have three limitations: (i) they target surface ambiguity without testing its SQL impact; (ii) they miss ambiguities implicit in schema routes or domain knowledge; and (iii) they spend limited turns on unfocused or redundant questions. These limitations expose a critical challenge: identifying SQL-impacting ambiguity before the intended SQL is known. In this paper, we propose DisGuAD, a Disagreement-Guided Ambiguity Detection framework that specializes clarification selection while leaving the downstream SQL generator unchanged. DisGuAD treats disagreement among plausible SQL interpretations as direct evidence of which uncertainty matters to the final query. Specifically, it constructs a query-term inventory, elicits contrastive SQL interpretations through complementary probe families, compares their canonical SQL components, and attributes recurring differences to user-facing clarification targets. A lightweight prioritization and dialogue-state mechanism then selects one high-impact unresolved target for clarification. Experiments on BIRD-Interact Lite and Full with two LLM backbones demonstrate consistent improvements in task success and clarification quality. Compared with the original interaction policy, DisGuAD improves normalized Reward by 4.40–8.97 points and raises overall critical-ambiguity coverage from 48.62% to 82.75% and from 45.76% to 85.14%. Its clarification signals further improve two independently designed static text-to-SQL frameworks by up to 13.97 Reward points while keeping their SQL generators fixed. These results demonstrate that disagreement among plausible SQL queries provides actionable evidence for selecting clarification targets that matter to downstream generation.

Primary area:
generative models
Keywords:
Interactive, Text-to-SQL, Code Generation, Large Language Model, Ambiguity Detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_89KAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DISGUAD, a clarification-selection framework for interactive text-to-SQL. Rather than directly asking an LLM to identify ambiguous expressions, the method generates contrastive SQL interpretations, compares their canonicalized components, and maps the resulting disagreements back to user-facing clarification targets. The framework combines a query-term inventory, four probe families, component-level disagreement voting, LLM-based attribution, and a heuristic prioritization mechanism with dialogue-state tracking. The downstream SQL generator remains unchanged.

Critical strengths
  • Addresses an important problem: selecting clarifications that materially affect SQL semantics under limited interaction budgets.
  • Presents a coherent, modular framework that maps SQL disagreement to user-facing questions while keeping the downstream generator unchanged.
  • Reports consistent improvements across two backbones and both benchmark variants, with additional benefits for two downstream SQL frameworks.
Critical weaknesses

Insufficient baselines: Missing comparisons with specialized clarification methods and compute-matched alternatives make the methodological novelty difficult to assess.
Incomplete ablations: The experiments do not isolate the contribution of explicit SQL differences and voting beyond candidate metadata and additional prompting.
Limited mechanism validation: Structural disagreement does not necessarily imply genuine user ambiguity, and lexical-overlap coverage does not establish successful ambiguity resolution.
Insufficient evaluation breadth: Results remain confined to one benchmark family with simulated users, without reported uncertainty estimates or end-to-end interaction and computational costs.

Questions for authors

What are the actual interaction and inference costs? Please report clarification-turn distributions, success at fixed turn limits, token usage, and latency. Is the candidate count held constant in each probe-family ablation?

How reliable are the intermediate signals and reported gains? Please provide a semantic audit of target matching, repeated-run or paired uncertainty estimates, and an explanation of the Table 2/Figure 3 coverage discrepancy.

Does explicit SQL disagreement outperform simpler alternatives under matched budgets? In particular, how does DISGUAD compare with direct SQL-impact ranking from the same query-term inventory, and with ordinary SQL sampling followed by the same mapper?

Reviewer_guVgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a prompting strategy for improving multi-turn ambiguity resolution in text-to-SQL tasks. The authors propose DISGUAD, is a plug-in clarification policy that extracts candidate ambiguous spans from the request, generates K=8 SQL interpretations using four probe prompt families, diffs their parsed SQL components and counts recurring disagreement types, uses an LLM to map those disagreements back to the extracted spans, and asks about the highest-priority unresolved span using a question template.

The authors run evals on BIRD-INTERACT Lite and Full with two backbones (DeepSeek-V4-Flash and GPT-5.6-Luna). They report Reward gains of 4.40–8.97 over the benchmark’s default clarification policy, though no comparisons are made to baselines from prior work, and most related work is omitted entirely from the paper.

Critical strengths

None

Critical weaknesses
  1. There are no comparisons to relevant baselines from prior work. The only comparison is the benchmark’s own policy, and reasoning settings aren’t reported or varied.

  2. Lack of novelty - generating SQL candidates and using disagreement to produce clarifying questions is present in numerous prior works that are not cited here.

  3. Experiments very limited. Limited ablation scope, only one hyperparameter sweep, no confidence intervals, error bars, or repeated seeds.

  4. User simulator is not explained. No description of what the simulator conditions on, no validation of its behavior, and no analysis of oracle leakage from the reference SQL.

Questions for authors

None

Reviewer_9HQHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks which ambiguity an interactive text-to-SQL agent should clarify under a limited question budget, noting that the default BIRD-Interact policy covers under half of the annotated critical ambiguities. The presented method, DisGuAD, is a plug-in clarification policy that generates contrastive SQL interpretations, compares their components, maps recurring disagreements back to user-facing terms, and asks about the highest-priority one, leaving the final SQL generator unchanged. On BIRD-Interact Lite and Full with two backbones, it improves Reward and raises critical-ambiguity coverage; it also significantly improves two static text-to-SQL frameworks.

Critical strengths

S1 The problem addressed in the paper is timely. Deciding what to ask, rather than when to ask, hasn’t been studied sufficiently yet but is an important question for interactive text-to-SQL agents.

S2 The introduced mechanism yields measurable gains across both benchmark variants and both backbones, and the coverage analysis (Table 2) shows the policy finds substantially more of the annotated critical ambiguities with fewer repeated questions.

S3 The mechanism is well composed, as the ablation shows, and is easily plugged into other text-to-SQL methods. Clarification signals improve two independently designed static frameworks with their generators held fixed.

Critical weaknesses

W1 The complexity of the method is not justified through the experiments. Removing any single probe family barely changes results, and no baseline simply samples several SQL queries and asks about the most-disagreed term. A compute-matched simple baseline is needed to show that the elaborate probe design matters.

W2 The paper is quite hard to follow due to unnecessary formalism and scattered details.

  • W2.1 The method is presented with heavier formalism. Section 3.1 introduces a task mapping, an environment tuple, agent modes and a history operator (Eqs. 1–2), and Eq. 3 then only states that DisGuAD is a function of the question, schema, knowledge and history. Little of this notation is used afterwards, so it adds reading cost without adding much precision. In general this paper would profit significantly from making more conscious decisions on when formalizations are needed and when they are only making the reading process harder.
  • W2.2 Key details are scattered across the paper or missing. Understanding the pipeline requires jumping between the method section, the dense overview figure and the appendix, and several details, such as how the heuristic candidates are produced, the priority weights and some evaluation metrics, are not given. A working example throughout the method section, with all hyperparameters stated where they are introduced, could fix this.

W3 The candidate budget and vote statistics are underspecified. The number of probes is introduced without a value and only appears later under a different symbol. Furthermore, the disagreement vote is a raw count over all probe pairs, and probes from different families are prompted to differ by design, so the vote may partly reflect the probe plan rather than genuine ambiguity in the request. Consistent notation, normalized votes and a breakdown of where disagreement comes from would show whether it is measured or manufactured.

W4 The paper doesn’t report the compute, latency or cost. Each clarification turn involves an extractor call, eight SQL generations, a mapper call and SQL parsing, yet Table 1 compares against the single-call original policy without reporting LLM calls, tokens or wall-clock time. Section 4.4 acknowledges qualitatively that larger budgets need more calls but doesn’t elaborate on this.

Questions for authors

Q1 What is DisGuAD's per-task cost versus the original policy, and do the gains hold under an equal compute budget?

Q2 Do the votes still pick the same targets when normalized, and how much of them comes from cross-family pairs?

Q3 Could SQL disagreement serve as a RL training signal for question selection, avoiding the inference-time cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.