acceptodds
Under review as a conference paper at ICLR 2027

FINCH 2.0: Knowing When Not to Answer in Conversational Financial Text-to-SQL

Abstract

Large language models perform well on single-turn text-to-SQL but degrade on multi-turn, ambiguous, or analytically complex questions. Existing benchmarks test these capabilities separately, and none combines a per-turn decision to answer, ask for clarification, or refuse with compound queries and adversarial variants of the same turns. We address this gap in finance. A seeded planner fixes each conversation's structure over 43 protocols in six series before an LLM writes any text, yielding FINCH 2.0: 16 databases in 7 financial verticals, with 469 tables, 3,674 columns, 13,024 conversations, and 176,860 turns, of which 158,118 are answerable question-SQL pairs. Automated checks validate every turn, and seven domain experts rate 86.0% of 1,287 sampled turns fully correct (95.7% in the test split). On a 2,400-conversation test split, three open-weight models refuse unanswerable questions reliably (recall 89.5–95.3%) but ask on only 34.9–57.7% of turns that need clarification; all fall below the 94.0% Action Accuracy of always answering; on turns a text-only classifier misjudges, Ask recall drops to 20.1–48.8%. SQL quality falls from 0.53–0.57 on easy queries to 0.28–0.33 on hard ones.

Primary area:
datasets and benchmarks
Keywords:
Text-to-SQL, Conversational Agents, Financial NLP, Synthetic Dataset

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r6a2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

FINCH 2.0 is a synthetic benchmark for financial text-to-SQL conversations in which each turn requires an ANSWER, ASK, or REFUSE decision. A seeded planner fixes the action and protocol before an LLM writes the utterance and, where appropriate, SQL. The corpus spans 16 databases, 13,024 conversations, and 176,860 turns; generation-time checks, post-hoc scoring, and an expert audit assess quality. On the selected test split, three open-weight models refuse most schema-unanswerable requests but ask for clarification on only 34.9%–57.7% of turns that require it.

Critical strengths
  1. The resource brings action choice, cross-turn reference, financial ambiguity, and compositional query structure into one protocol-labelled corpus at a scale that makes failure-mode analysis possible. The validation records make it more useful than a benchmark with only a gold SQL string and an aggregate score.
  2. The evaluation includes controls that limit over-interpretation. Always answering reaches 94.0% action accuracy, and an utterance-only classifier reaches 0.809 macro-F1, well above the tested language models. The authors acknowledge these shortcuts and show that low ASK recall persists on turns the classifier gets wrong; this is a more credible finding than action accuracy alone.
Critical weaknesses
  1. The action task does not yet separate linguistic shortcuts from grounded judgment. ANSWER accounts for 94.0% of test turns, and an utterance-only classifier outperforms the evaluated models even on macro-F1. The authors' analysis of classifier-error turns is helpful: under-asking persists there, so the finding is not merely a majority-class artifact. Still, that subset is not a test of whether the same question elicits different justified actions under different schemas or dialogue histories. Without such counterfactual cases, FINCH 2.0 measures per-turn action prediction more clearly than it measures when an agent should rely on database and conversational context before answering.
  2. The scored task is narrower than the conversational and compound-query coverage suggests. Each turn is replayed with earlier user utterances but without the model's own prior responses, and an ASK never elicits a follow-up conditioned on the model's question. Separately, the disjoint and hybrid compound requests have only one gold SQL for several requested outputs; 925 such test turns are excluded from SQL scoring. The authors acknowledge these limits, but they mean the reported results test offline action selection and only a subset of compound-query completion, not successful clarification or complete multi-output dialogue behavior.
Questions for authors
  1. Can you evaluate matched questions whose gold action changes with the available schema or preceding dialogue, while keeping the surface wording fixed or nearly fixed? The result, compared with the utterance-only classifier, would show how much of the action task requires grounded context rather than lexical cues.
  2. Is there additional evidence of successful clarification after a model-generated ASK, or of complete output scoring for the disjoint and hybrid requests? If not, would you frame the contribution explicitly as offline per-turn action selection and restrict the compound-SQL conclusions to the structures that receive complete scoring?
4 comments
AuthorsOpenReview ↗

Response Part (1/2): Grounded Judgment and the Matched Pair Contrast Set (W1, Q1)

We thank the reviewer for the thoughtful review and for recognising both the joint coverage of the resource and the controls in our evaluation. The reviewer's suggestion led directly to the matched pair experiment below, which we consider the most informative addition of this rebuttal.

W1 and Q1. Separating lexical cues from grounded judgment. We agree that the classifier error subset in the paper does not test whether the same question should receive different actions in different contexts. Following Q1, we built a contrast set of 2,000 matched pairs from scored test turns, 500 of each of four types and balanced over the 16 databases. The two members of a pair have identical user wording, and only the schema or the dialogue history shown to the model changes, which changes the gold action.

Pair typeGold action, original to changedWhat changes
Schema dropANSWER to REFUSEA column the gold query needs is removed from the schema
Near missANSWER to REFUSEThe same, while a similarly named column remains
History dropANSWER to ASKThe turn is posed without the earlier turn its reference needs
History resolveASK to ANSWERAn earlier statement that settles the ambiguity is added

A pair counts as solved only when both members receive their gold action. The text only classifier, which reads only the words of the question, sees identical text in both members, so it gives them the same action and solves no pair of any type, 0% by construction. Any accuracy above zero must therefore come from using the schema or the history. A judge given the removed column, the original turns and the gold query found 1,693 of the 2,000 pairs valid, and we report both sets, with the four types on the valid pairs.

ModelAll pairsValid pairs (95% CI)Schema dropNear missHistory dropHistory resolve
Gemma-4-31B44.049.0 (46.7 to 51.2)57.639.072.827.5
Nemotron-3-Super35.439.2 (36.9 to 41.5)41.130.953.032.9
DeepSeek-V4.1-Flash39.242.6 (40.3 to 44.9)44.632.464.031.2

This answers Q1 in two ways. First, a substantial part of the action task does require grounded context. The text only classifier, which outperforms every model on the original split, solves no pair, while the models solve 39% to 49% and change their action between the two members of a pair 56% to 61% of the time. Second, grounding is exactly where the models are weakest. They handle a missing antecedent well, asking on 73% to 94% of those changed turns, but when an earlier turn has already settled the ambiguity they still ask on 24% to 45% of turns. When the needed column is gone they refuse only 52% to 66% of the time, and only 46% to 54% when a similar column remains. The contrast set therefore confirms the reviewer's concern about the original split and gives FINCH 2.0 the test of contextual grounding it was missing. We will add it to the paper with Qwen included, and release it with the benchmark.

The low asking rate is also not an artefact of our labels. Two additional finance experts relabelled 300 turns blind, seeing neither our labels nor any model output, and agreed with each other at Cohen's κ = 0.975. They confirmed all 78 clarification labels and all 102 refusal labels in the sample. On these confirmed turns the four models ask on only 44.9% to 71.8% of clarification turns but refuse 88.2% to 95.1% of unanswerable ones, a gap that is significant for every model (Fisher exact p ≤ 1.6 × 10⁻⁴).

AuthorsOpenReview ↗

Response Part (2/2): Session Replay, Clarification Follow Through and Compound Turns (W2, Q2)

W2 and Q2. Offline replay, clarification follow through and compound turns. We agree with the reviewer's reading of the scored task, and we ran two experiments to measure how much it matters.

The model's own responses. We reran Gemma-4-31B on 300 test conversations as single sessions that keep its own earlier replies, scored on the same 3,658 turns as the paper's offline run.

Gemma-4-31B on the same 300 conversationsOffline (paper)Session with own replies
ASK recall32.935.7
REFUSE recall92.794.0
False caution6.910.6
Mean FINCH Score, all answerable turns0.4400.493
Mean FINCH Score, turns that refer to an earlier turn0.3490.421
Pass Rate33.938.0

The action decision barely changes, so the action findings are not an artefact of the offline replay. SQL quality rises, most on turns that refer back, so the offline protocol is a conservative lower bound for multi turn SQL.

Clarification after a model's ASK. The released conversations do not contain the user's answer to a clarifying question, so we added a follow through experiment. For each of the 470 gold clarification test turns on which a model asked, the generator model played the user, chose the intended reading and wrote the user's reply together with an executable gold query for it. Each model then continued from its own clarifying question, giving 1,262 model and turn pairs.

GemmaQwenNemotronDeepSeek
Question targets the annotated ambiguity (%)68635266
Answers after the user's reply (%)88.588.381.079.3
Mean FINCH Score after the reply0.4660.4760.4520.467
Mean FINCH Score on ordinary answerable turns0.4410.4500.4270.383

Clarification works when the question is right. After the reply, models answer with SQL quality equal to or above their quality on ordinary turns, and when a clarifying question misses the ambiguity, the final score drops by 0.115 (95% CI 0.052 to 0.178). Because the replies and their gold queries are model written and checked by execution only, we will present this as supporting evidence and say so in the paper.

Compound turns and framing. We could not produce complete gold for the 925 disjoint and hybrid turns within the rebuttal period, so we gladly accept both of the reviewer's suggestions. We will frame the contribution explicitly as offline per turn action selection, with the session and follow through results as complementary evidence, and restrict compound SQL conclusions to sequential and comparative queries (Q-2 and Q-4), which receive complete scoring.

Summary. On pairs with identical wording, the models solve 39% to 49% where the text only classifier solves none, which shows that the action task requires grounded context and that grounding is where models fail. The offline protocol does not drive the action findings, and clarification succeeds when the question targets the real ambiguity. We thank the reviewer again for questions that made the paper more precise, and we hope these results strengthen their confidence in the work.

Thank you for the detailed response. I appreciate the matched-pair experiment and clarification, as well as your clear explanation of the remaining limitations. I will maintain my score.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank you for taking the time to read our response and for your kind words about the matched pair experiment. Your questions shaped this part of the work directly, and we believe the paper is clearer and more precise about its scope as a result.

If any further analysis or clarification would strengthen the paper in your view, we would be glad to provide it during the discussion period.

Thank you again for your thoughtful and constructive review.

Reviewer_zpnmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents FINCH 2.0, a synthetic conversational text-to-SQL benchmark for finance in which each turn is labelled as answerable, needing clarification, or unanswerable. A planner fixes the gold action and phenomenon tags before an LLM writes the conversations over synthetic financial databases, and a multi-stage validation pipeline ends with an expert audit. Evaluating several open-weight models against simple baselines, the authors conclude that models refuse unanswerable questions reliably but rarely ask for clarification when it is needed, and that knowing when to ask is the main missing capability.

Critical strengths
  1. Honest evaluation. Trivial and text-only baselines beat the LLMs on the headline metric, which shows how misleading accuracy alone is for ask/answer/refuse evaluation; this is a reusable methodological lesson.
  2. Controllable, auditable construction. Labels and phenomenon tags are fixed before text generation and checked by an expert audit, enabling per-phenomenon diagnostics that existing finance text-to-SQL resources lack.
Critical weaknesses
  1. The "models under-ask" conclusion rests on a very small, thinly validated set of clarification labels, and a simple text classifier beats every model, so model failure is not separated from label noise. Targeted expert re-annotation could partly fix this.
  2. The contribution is largely a synthesis of known dimensions with weakly evidenced finance-specific realism, and the link to the original FINCH is little more than the name and a redefined metric. The heavy formal apparatus relative to the insight also makes the core evaluation logic hard to follow.
Questions for authors
  1. Where model and gold disagree about asking for clarification, what do independent experts say: model error or label error?
  2. In the ambiguity-cascade variants the gold stays "answer"; how many would an expert consider genuinely ambiguous, and do the conclusions hold without them?
  3. What does FINCH 2.0 inherit from FINCH beyond the metric name, and which of the many definitions and protocol codes are essential to the main claim?
5 comments
AuthorsOpenReview ↗

Response Part (1/3): Label Quality and Model Error (W1, Q1)

We thank the reviewer for the time and care spent on our paper. The questions led to the most useful experiments of this rebuttal and have clearly improved the work.

Annotation quality is a known weak point even in human labelled text to SQL data. A recent audit finds errors in 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow examples (Jin et al., 2026). An execution audit of the financial subsets of Spider, BIRD, BULL and BookSQL, made for FINCH (Singh et al., 2025), removed 18,298 of 85,641 gold queries, 21.4%, because they failed to execute or relied on values the data never stores. This includes 28.9% of BIRD's financial queries and 37.1% of BULL's, and a further 12.5% of Spider's needed their letter case corrected. In finance, where real data is confidential and a wrong number is costly, a synthetic benchmark has to earn trust rather than assume it. This is why every stage of FINCH 2.0 is controlled. A seeded planner fixes each turn's label before any text is written, every gold query must execute and pass result checks, and a three stage validation funnel audits the output. Seven finance experts, one per vertical, then rated 1,287 turns from 102 conversations across all 16 databases. They rated 86.0% fully correct overall and 95.7% in the test split, with 99.0% of refusal labels and 85.9% of clarification labels correct.

W1. Separating model failure from label noise. To separate the two directly, we ran the targeted reannotation the reviewer suggested, designed to go beyond the audit above, which used one rater per turn, showed raters our labels and covered 78 clarification turns. Two additional finance experts labelled 300 turns independently as ANSWER, ASK or REFUSE, seeing only the schema, the earlier questions and the current question, never our labels or any model output. The turns are all 78 clarification and all 102 unanswerable turns of the audit sample plus 120 random answerable ones, so the sample was not chosen by model behaviour.

The experiment tests one hypothesis. If the low asking rate were label noise, independent experts would reject a substantial share of our ASK labels, and on the labels they keep, ASK recall would rise to the level of REFUSE recall. The data rejects both parts.

First, the experts agree with each other at Cohen's κ = 0.975 (95% CI 0.950 to 0.995) and confirm 78 of 78 clarification labels, 102 of 102 refusal labels and 109 of 115 answerable labels. At 95% confidence, the Wilson upper bound on the error rate e is therefore 4.7% for clarification labels and 3.6% for refusal labels. Even if every such error sat on a turn a model answered, a measured ASK recall r could rise only to r / (1 − e). For e = 0.047, Gemma moves from 34.9% to 36.6%, Qwen from 49.0% to 51.4% and Nemotron from 57.7% to 60.5%, far below their REFUSE recall of 89.5% to 95.3%.

Second, we ran four models on the same 300 turns with the protocol of the paper's Section 5 and scored them on the confirmed labels.

ModelASK recall (78 turns)REFUSE recall (102 turns)Gap, 95% CIFisher p
Gemma-4-31B44.995.150.2 (38.3 to 61.8)1.3 × 10⁻¹⁴
Qwen3.8-27B47.493.145.7 (33.3 to 56.9)3.5 × 10⁻¹²
Nemotron-3-Super55.188.233.1 (20.8 to 45.7)8.7 × 10⁻⁷
DeepSeek-V4.1-Flash71.893.121.3 (10.4 to 32.6)1.6 × 10⁻⁴

Every gap is significant, close to the paper's Table 5 in size, and in the same model order. As for the text classifier, it is trained on turns from the same distribution and learns the generator's wording for each action. On a contrast set of matched pairs where the wording stays fixed and only the schema or the history changes the correct action, it solves no pair by construction, while the models solve 39% to 49%.

Q1. What do independent experts say when model and gold disagree? Each relabelled turn now carries three expert opinions, the original audit rating and the two blind relabels, and we take the majority of the three (two out of three) as the expert verdict, so no turn is dropped when two experts differ. Across the four models, the 300 turns contain 273 disagreements about asking. The expert majority sides with the gold label on 262 of them, 96.0% (Wilson 95% CI 92.9 to 97.7, binomial p < 10⁻⁶⁰ against an even split). When a model answered a gold ASK turn, the majority sided with the gold in all 140 cases, and when a model asked on a gold ANSWER turn, it sided with the gold in 122 of 133 cases. Where model and gold disagree, the experts therefore support the gold label, and the misses are model errors. We will add this reannotation and its statistics to the paper's Section 5 and appendix and release the expert labels.

AuthorsOpenReview ↗

Response Part (2/3): Novelty, Finance Realism and the Link to FINCH (W2)

We thank the reviewer for this point and answer each of its four parts in turn.

"The contribution is largely a synthesis of known dimensions." To the best of our knowledge, FINCH 2.0 is the first dataset that labels multi turn reference, schema scale, SQL complexity, ambiguity, adversarial variants and compound queries on the same turns of the same conversations, with a gold decision to answer, ask or refuse at every turn. The paper's Table 1 draws this comparison against sixteen benchmarks, including CoSQL, MMSQL, BIRD-Interact, Spider 2.0 and FINCH. None of them fully covers more than two of the seven dimensions, while FINCH 2.0 fully covers six together with the action decision. Labelling them jointly makes new failures measurable. On labels the experts confirm, models ask on 45% to 72% of clarification turns but refuse 88% to 95% of unanswerable ones. Refusal recall is 99% to 100% for requests outside the database's domain but only 68% to 82% when a related column exists. Models still ask on 24% to 45% of turns whose ambiguity an earlier turn settled, and solve only 39% to 49% of contrast pairs with identical wording. Each result needs a schema, a conversation and an action label in the same item, which no benchmark of a single capability provides.

Finance specific realism. The finance focus is part of the design and shows clearly in the data. Each of the 16 schemas covers one subdomain of a taxonomy of 7 verticals with FIBO and GICS boundaries, is modelled on the public documentation of production systems in that subdomain, and follows the standards and regulations that govern it.

VerticalModelled onStandards and regulations
Capital MarketsBloomberg TOMS, Murex, 360TISDA, ACTUS, BIS FX
Private MarketsiLEVEL, eFront, YardiILPA, NAREIT
Investment ManagementSimCorp, Aladdin, AvaloqGIPS, CFA, MSCI Barra
BankingTemenos T24, FinastraBasel III, SWIFT
Risk and ComplianceActimize, MSCI RiskManagerFATF 40, Basel III, BCBS 239
Reporting and TreasurySAP S/4HANA, KyribaIFRS, GAAP, XBRL
InsuranceGuidewire, Duck CreekIFRS 17, Solvency II, NAIC

The conversations come from 75 financial job roles, led by controllers, portfolio managers and market risk managers, and half of the conversations are posed by expert personas. Of the 158,118 answerable turns, 28,656 (18.1%) require a financial formula that a deterministic check verifies, among them spreads (1,044 turns), exposures (749), execution slippage and VWAP (703), net operating income (485), cap rates (392), reserves (310), IRR (273) and Sharpe ratios (241). Another 3,776 turns carry financial ambiguity (A-4), such as which duration measure a question means, the hardest ambiguity type for every model (Action Accuracy 49.8% to 51.4%). Experts from each vertical rated 98.1% of sampled test turns fully relevant.

"The link to the original FINCH is little more than the name and a redefined metric." FINCH 2.0 is built on FINCH (Singh et al., 2025) and uses its evaluation without modification. The FINCH Score is an F2 with a materiality tolerance over the cells of the returned rows, and the FINCH Diagnostic attributes each error to entity, constraint, computation or structural grounding (G1 to G4). FINCH's earliest description multiplied component similarity by an execution term, and FINCH later moved to this result level score, which suits conversation, where several queries can answer a follow up. The shared diagnostic shows what conversation changes. On single turn FINCH, entity grounding stays below 68% for every model. In FINCH 2.0, join structure is the weakest component (0.40 to 0.43), constraints follow (0.47 to 0.56) and entities hold up (0.72 to 0.76), and turns that refer back to an earlier turn score 0.35 to 0.37 on structure against 0.42 to 0.45 for the rest. So moving to conversation changes where financial SQL fails. Q3 in Part 3 lists what else FINCH 2.0 inherits.

"The heavy formal apparatus relative to the insight also makes the core evaluation logic hard to follow." We will streamline it so that the main text keeps only what the main claim needs, as detailed under Q3 in Part 3.

AuthorsOpenReview ↗

Response Part (3/3): What FINCH 2.0 Inherits and the Ambiguity Cascades (Q3, Q2)

Q3. What FINCH 2.0 inherits, and which definitions the main claim needs. Beyond the score and diagnostic in Part 2, FINCH 2.0 inherits four things. The first is the problem, financial text to SQL in which a wrong query still runs and returns a plausible number. The second is the domain taxonomy, whose banking, capital markets, corporate accounting and insurance domains we keep and refine into 7 verticals and 16 subdomains, dropping commerce and property. The third is the principle that a wrong gold query manufactures a failure no model committed, which is why every FINCH 2.0 gold query must execute and pass its checks. The fourth is the Easy, Medium and Hard difficulty rule, applied unchanged. The main claim needs only four objects, the turn label of the paper's Definition 1, the gold action ρ(t) derived from it, the action metrics of the paper's Eq. 7 with their two baselines, and the FINCH Score. In the final version we will keep these in the main text, move the paper's generator equation (Eq. 1), full protocol definition (Definition 2) and allocation and acceptance equations (Eqs. 4 and 5) to the appendix with one sentence each, and organise the paper's Section 5 around whether the model takes the gold action, whether its SQL returns the right result, and where it fails.

Q2. Are the ambiguity cascade variants genuinely ambiguous, and do the conclusions hold without them? We thank the reviewer for this question and answer both halves with new measurements over every scored variant.

How many are ambiguous. We checked all 3,574 scored R-3 variants (3,478 of them enter the robustness cost in the paper's appendix) with a judge that sees the intended query, which the evaluated model never sees. It decides whether a careful analyst reading only the conversation, the schema and the vaguer wording could reasonably write a materially different query, with a different metric, entity, filter, time window, grouping or aggregation. It judges 1,439 variants (40.3%) ambiguous, rising from 32.8% at the first rewrite to 47.7% at the second. Two financial domain experts then reviewed the judge's verdicts on 144 variants and were free to change any of them. They agree with each other on 140 (Cohen's κ = 0.944) and, on those, accept 98.6% of the judge's verdicts (κ = 0.971). They call 47.1% of the sample ambiguous (95% CI 39.1% to 55.4%), 37.7% at the first rewrite and 56.3% at the second, and every change they made went from one reading to ambiguous. Our description of R-3 as underspecified but not ambiguous therefore holds for the majority of variants, and we will refine it to cover the rest.

Whether the conclusions hold. They do. Action Accuracy, the recalls per action and macro F1 are computed on base turns only, so no cascade variant enters the main result. We also rescored every model with ASK accepted on the variants the judge calls ambiguous, and separately on the variants with one reading alone.

ModelAsks, ambiguousAsks, one readingPass Rate changeR-3 cost, allR-3 cost, one reading
Gemma-4-31B20.611.8+0.20.1110.072
Qwen3.8-27B26.812.7+0.20.0960.038
Nemotron-3-Super40.724.1+0.30.1060.050

Accepting ASK moves Pass Rate by at most 0.3 points, dropping R-3 entirely moves it by 1.3 points for every model, and the model order never changes. The one quantity that changes is the robustness cost of R-3, which falls to 0.038 to 0.072 on variants with one reading. That is still above the cost of paraphrase (at most 0.019) and below that of schema perturbation (0.093 to 0.121), so R-3 remains a real robustness test once the ambiguous variants are set aside. The table also shows that every model asks about twice as often on ambiguous variants as on those with one reading, so the variants do separate genuine ambiguity from terse wording. Yet even where asking is a reasonable action, models ask on only 20.6% to 40.7% of these variants, which agrees with the low asking rate measured on the labels the experts confirmed in Part 1.

In the final version we will correct the description of R-3 in the paper's Section 2, report the R-3 cost on both sets, and add this analysis to the paper's appendix. The release will mark the 1,439 ambiguous variants as accepting either ANSWER or ASK, with the judge's verdict and the competing reading for each.

Summary. On expert confirmed labels the gap between asking and refusing stays large and significant for every model, FINCH 2.0 uses FINCH’s evaluation and extends it to conversation, and the cascade variants do not drive any conclusion. We thank the reviewer once more for the careful and constructive review. We would warmly welcome their further guidance during the discussion period, and if any concern remains, we would be glad to run additional analyses or clarify any point. We hope these new results help the reviewer in reassessing the paper.

Reviewer_aH1BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FINCH 2.0, a synthetic conversational text-to-SQL benchmark for the financial sector. Each round of dialogue has a standard action (answer, follow-up question, or rejection), and for answerable rounds, a standard SQL query is also provided. The authors constructed 16 databases covering 7 financial verticals. A planner with a random seed first determines the label for each round and the phenomenon it examines, and then use an LLM to generates the questions and SQL. The data undergoes triple verification: execution testing, LLM review, and expert auditing.

Critical strengths

S1. The questions are important. In the financial sector, wrong SQL often executes silently and returns seemingly reasonable numbers, while the real data cannot be publicly disclosed. Therefore, there is a strong motivation to build a synthetic benchmark that examines "when not to answer."

S2. The dataset is large and well engineered, and it comes with a datasheet, per-turn validation records, and a repair manifest.

S3. The evaluation explicitly examined the label leakage issue.

Critical weaknesses

W1. Insufficient experimentation. Only three open-weight models were evaluated; the study excludes closed-source models as well as models fine-tuned on existing training sets. Consequently, it remains unclear whether these conclusions are generalizable or if the dataset is effective for model training.

W2. The conclusion that "models can reliably refuse to answer" applies only to simple scenarios. Many of the unanswerable dialogue turns involve data that falls well outside the scope of any given schema (e.g., ESG or GDP data). The dataset lacks "near-hit" scenarios—instances where column names mentioned in a query resemble those present in the schema. This conclusion should be qualified to apply only to requests that clearly fall outside the scope of the schema.

W3. The dialogue-level evaluation metrics fail to differentiate between the models. No model successfully completed an entire dialogue, and all three models performed nearly identically in terms of "single-turn pass rate" and "topic-module success rate." More granular evaluation metrics are required, such as the "number of correctly answered turns per dialogue."

W4. Lack of independence in evaluation. Gemma is used both as the judge and as one of the models being evaluated.

Questions for authors

Refer to weakness.

4 comments
AuthorsOpenReview ↗

Response Part (1/2): More Models and Near Miss Refusal

We thank the reviewer for the careful review and for recognising the importance of the problem, the engineering of the dataset and our check for label leakage. We address each weakness below with new experiments.

W1. More models, closed and fine tuned models, and training. We agree that broader coverage strengthens the paper, and we priced closed models carefully. The test split requires about 41,000 model calls (29,388 turns and 11,498 adversarial variants), each carrying the full database schema, which comes to about 720 million input tokens per model. From a pilot run and current list prices, one frontier closed model costs about USD 2,000 to 3,000 on the full split, so covering the GPT, Claude and Gemini families would exceed USD 5,000, which was beyond our budget within the rebuttal period. We therefore widened the evaluation with open weight models. Gemma already represents Google's model family as the open counterpart of Gemini. We did not evaluate OpenAI's open gpt-oss models, because the larger gpt-oss-120b generated our data and a model from the same family could be favoured. We add DeepSeek-V4.1-Flash, a fourth model family, run on the full test split with the paper's protocol. We plan to add a closed model on a stratified subset in the final version, together with a text to SQL model fine tuned on existing training sets.

ModelAction AccuracyASK recallREFUSE recallFalse cautionMean FINCH Score
Gemma-4-31B (paper)92.334.993.86.50.441
Qwen3.8-27B (paper)84.349.095.315.40.450
Nemotron-3-Super (paper)76.057.789.524.20.427
DeepSeek-V4.1-Flash63.670.494.337.80.383

The conclusions generalise. Every model refuses far more reliably than it asks, and the models that ask more also decline more answerable turns, so they differ mainly in how cautious they are. DeepSeek asks on 70.4% of clarification turns but declines 37.8% of answerable ones. On whether the data supports training, the text only classifier, trained on our training conversations, already reaches a macro F1 of 0.81, and 0.74 on databases it never saw, so the labels carry a learnable signal. We release the 10,624 training conversations with the benchmark and will add a fine tuned baseline in the final version.

W2. Refusal on near miss requests. We agree, and we thank the reviewer for pointing this out. We have now measured it. We split all 1,167 unanswerable test turns by how close the schema comes to the request, using a judge that names the missing concept and its closest column, and an embedding similarity that gives the same ordering.

RequestTurnsGemmaQwenNemotronDeepSeek
Outside the database's domain16399.4100.099.4100.0
Related topic, nothing to substitute90094.496.090.295.0
Near miss, a similar column exists10479.881.768.379.8

Refusal recall is 99% to 100% for requests outside the database's domain but only 68% to 82% for near misses, such as a credit rating requested from a schema that holds only a credit limit, and 12% to 24% of near misses are answered instead of refused. To test near misses at scale, we also built 444 matched pairs that remove a column the gold query needs while a similarly named column remains, and there refusal drops to 46% to 54%. The test split therefore does contain near misses, and the reviewer is right that refusal is reliable only far from the schema. We will qualify the claim exactly as suggested and add both analyses to the paper.

AuthorsOpenReview ↗

Response Part (2/2): Dialogue Level Metrics and Judge Independence

W3. Finer dialogue level metrics. We thank the reviewer for this suggestion and computed the requested measure along with several finer ones. Counting a turn as correct only when both the action and the SQL are right, the models complete 3.9 to 4.3 of about 12.2 turns per conversation. They are genuinely close here, because exact SQL accuracy, 25% to 31%, dominates every turn. Separating the action decision, which is the focus of the paper, separates the models clearly.

SystemCorrect turns, action and SQLCorrect actionsTurns before first wrong actionAll ASK and REFUSE right (%)Unnecessary declines
Gemma-4-31B4.1911.307.7848.30.75
Qwen3.8-27B4.3410.325.2257.01.77
Nemotron-3-Super4.289.303.6055.92.79
DeepSeek-V4.1-Flash3.917.792.2568.64.35
Always answern/a11.519.470.00.00
Text only classifiern/a11.9410.6739.80.06

All values are per conversation except the share of conversations with every ASK and REFUSE turn right, which is over the 691 conversations that contain such turns. Every pairwise difference in correct actions and in turns before the first wrong action is significant under a paired bootstrap (p < 0.001). Because always answering scores well on any accuracy measure, we also report the two sides of the decision. Models that get more ASK and REFUSE turns right broadly also decline more answerable turns, from Gemma at 48.3% and 0.75 declines to DeepSeek at 68.6% and 4.35, which shows the caution tradeoff that a single success rate hides. We will add these measures to the paper.

W4. Independence of the judge. We clarify first that no LLM judge scores the evaluated models. Actions are compared with gold labels and SQL is scored deterministically, so Gemma's only role was the Stage 2 score that selected the test split. To test that selection, we rescored all 3,035 candidate conversations with a second judge, gpt-oss-120b, which is not an evaluated model. As the generator of every conversation, it has no reason to favour one evaluated model over another, which is why we kept it out of W1's evaluation but can use it here.

CheckResult
Rank correlation of the two judges' scoresSpearman 0.41 (p < 10⁻¹⁰⁰)
Second judge separates selected from excluded conversationsAUC 0.72
Excluded conversations the second judge would also exclude42%, twice the chance rate
Largest change in the main metrics of the paper's Table 5 on the 2,030 conversations both judges select0.9 points, same model order
Correlation of each model's pass rate with the Gemma judge's scoreGemma 0.005, Qwen 0.008, Nemotron −0.002

The judges largely agree, the results are unchanged on the conversations both select, and Gemma's performance does not track its own judge's score any more than the other models' does, so there is no sign of self preference. We will report this and state the judge's role explicitly in the paper.

Summary. The findings hold across four model families, the refusal claim is now qualified and measured on near misses, the dialogue measures separate the models along the caution tradeoff, and an independent judge confirms the test split selection. We thank the reviewer again for the constructive review, and we hope these additions strengthen their confidence in the work.

Thanks for your response and most of my concerns have been resolved. I will keep my original rate.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank you for your response and are glad that most of your concerns have been resolved.

We remain happy to address any further question, and we thank you again for your careful and constructive review, which has clearly improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.