- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
FINCH 2.0 is a synthetic benchmark for financial text-to-SQL conversations in which each turn requires an ANSWER, ASK, or REFUSE decision. A seeded planner fixes the action and protocol before an LLM writes the utterance and, where appropriate, SQL. The corpus spans 16 databases, 13,024 conversations, and 176,860 turns; generation-time checks, post-hoc scoring, and an expert audit assess quality. On the selected test split, three open-weight models refuse most schema-unanswerable requests but ask for clarification on only 34.9%–57.7% of turns that require it.
Critical strengths
- The resource brings action choice, cross-turn reference, financial ambiguity, and compositional query structure into one protocol-labelled corpus at a scale that makes failure-mode analysis possible. The validation records make it more useful than a benchmark with only a gold SQL string and an aggregate score.
- The evaluation includes controls that limit over-interpretation. Always answering reaches 94.0% action accuracy, and an utterance-only classifier reaches 0.809 macro-F1, well above the tested language models. The authors acknowledge these shortcuts and show that low ASK recall persists on turns the classifier gets wrong; this is a more credible finding than action accuracy alone.
Critical weaknesses
- The action task does not yet separate linguistic shortcuts from grounded judgment. ANSWER accounts for 94.0% of test turns, and an utterance-only classifier outperforms the evaluated models even on macro-F1. The authors' analysis of classifier-error turns is helpful: under-asking persists there, so the finding is not merely a majority-class artifact. Still, that subset is not a test of whether the same question elicits different justified actions under different schemas or dialogue histories. Without such counterfactual cases, FINCH 2.0 measures per-turn action prediction more clearly than it measures when an agent should rely on database and conversational context before answering.
- The scored task is narrower than the conversational and compound-query coverage suggests. Each turn is replayed with earlier user utterances but without the model's own prior responses, and an ASK never elicits a follow-up conditioned on the model's question. Separately, the disjoint and hybrid compound requests have only one gold SQL for several requested outputs; 925 such test turns are excluded from SQL scoring. The authors acknowledge these limits, but they mean the reported results test offline action selection and only a subset of compound-query completion, not successful clarification or complete multi-output dialogue behavior.
Questions for authors
- Can you evaluate matched questions whose gold action changes with the available schema or preceding dialogue, while keeping the surface wording fixed or nearly fixed? The result, compared with the utterance-only classifier, would show how much of the action task requires grounded context rather than lexical cues.
- Is there additional evidence of successful clarification after a model-generated ASK, or of complete output scoring for the disjoint and hybrid requests? If not, would you frame the contribution explicitly as offline per-turn action selection and restrict the compound-SQL conclusions to the structures that receive complete scoring?
4 comments
Response Part (2/2): Session Replay, Clarification Follow Through and Compound Turns (W2, Q2)
W2 and Q2. Offline replay, clarification follow through and compound turns. We agree with the reviewer's reading of the scored task, and we ran two experiments to measure how much it matters.
The model's own responses. We reran Gemma-4-31B on 300 test conversations as single sessions that keep its own earlier replies, scored on the same 3,658 turns as the paper's offline run.
| Gemma-4-31B on the same 300 conversations | Offline (paper) | Session with own replies |
|---|---|---|
| ASK recall | 32.9 | 35.7 |
| REFUSE recall | 92.7 | 94.0 |
| False caution | 6.9 | 10.6 |
| Mean FINCH Score, all answerable turns | 0.440 | 0.493 |
| Mean FINCH Score, turns that refer to an earlier turn | 0.349 | 0.421 |
| Pass Rate | 33.9 | 38.0 |
The action decision barely changes, so the action findings are not an artefact of the offline replay. SQL quality rises, most on turns that refer back, so the offline protocol is a conservative lower bound for multi turn SQL.
Clarification after a model's ASK. The released conversations do not contain the user's answer to a clarifying question, so we added a follow through experiment. For each of the 470 gold clarification test turns on which a model asked, the generator model played the user, chose the intended reading and wrote the user's reply together with an executable gold query for it. Each model then continued from its own clarifying question, giving 1,262 model and turn pairs.
| Gemma | Qwen | Nemotron | DeepSeek | |
|---|---|---|---|---|
| Question targets the annotated ambiguity (%) | 68 | 63 | 52 | 66 |
| Answers after the user's reply (%) | 88.5 | 88.3 | 81.0 | 79.3 |
| Mean FINCH Score after the reply | 0.466 | 0.476 | 0.452 | 0.467 |
| Mean FINCH Score on ordinary answerable turns | 0.441 | 0.450 | 0.427 | 0.383 |
Clarification works when the question is right. After the reply, models answer with SQL quality equal to or above their quality on ordinary turns, and when a clarifying question misses the ambiguity, the final score drops by 0.115 (95% CI 0.052 to 0.178). Because the replies and their gold queries are model written and checked by execution only, we will present this as supporting evidence and say so in the paper.
Compound turns and framing. We could not produce complete gold for the 925 disjoint and hybrid turns within the rebuttal period, so we gladly accept both of the reviewer's suggestions. We will frame the contribution explicitly as offline per turn action selection, with the session and follow through results as complementary evidence, and restrict compound SQL conclusions to sequential and comparative queries (Q-2 and Q-4), which receive complete scoring.
Summary. On pairs with identical wording, the models solve 39% to 49% where the text only classifier solves none, which shows that the action task requires grounded context and that grounding is where models fail. The offline protocol does not drive the action findings, and clarification succeeds when the question targets the real ambiguity. We thank the reviewer again for questions that made the paper more precise, and we hope these results strengthen their confidence in the work.
Thank you for the detailed response. I appreciate the matched-pair experiment and clarification, as well as your clear explanation of the remaining limitations. I will maintain my score.
Thank You for Your Feedback
We sincerely thank you for taking the time to read our response and for your kind words about the matched pair experiment. Your questions shaped this part of the work directly, and we believe the paper is clearer and more precise about its scope as a result.
If any further analysis or clarification would strengthen the paper in your view, we would be glad to provide it during the discussion period.
Thank you again for your thoughtful and constructive review.
Response Part (1/2): Grounded Judgment and the Matched Pair Contrast Set (W1, Q1)
We thank the reviewer for the thoughtful review and for recognising both the joint coverage of the resource and the controls in our evaluation. The reviewer's suggestion led directly to the matched pair experiment below, which we consider the most informative addition of this rebuttal.
W1 and Q1. Separating lexical cues from grounded judgment. We agree that the classifier error subset in the paper does not test whether the same question should receive different actions in different contexts. Following Q1, we built a contrast set of 2,000 matched pairs from scored test turns, 500 of each of four types and balanced over the 16 databases. The two members of a pair have identical user wording, and only the schema or the dialogue history shown to the model changes, which changes the gold action.
A pair counts as solved only when both members receive their gold action. The text only classifier, which reads only the words of the question, sees identical text in both members, so it gives them the same action and solves no pair of any type, 0% by construction. Any accuracy above zero must therefore come from using the schema or the history. A judge given the removed column, the original turns and the gold query found 1,693 of the 2,000 pairs valid, and we report both sets, with the four types on the valid pairs.
This answers Q1 in two ways. First, a substantial part of the action task does require grounded context. The text only classifier, which outperforms every model on the original split, solves no pair, while the models solve 39% to 49% and change their action between the two members of a pair 56% to 61% of the time. Second, grounding is exactly where the models are weakest. They handle a missing antecedent well, asking on 73% to 94% of those changed turns, but when an earlier turn has already settled the ambiguity they still ask on 24% to 45% of turns. When the needed column is gone they refuse only 52% to 66% of the time, and only 46% to 54% when a similar column remains. The contrast set therefore confirms the reviewer's concern about the original split and gives FINCH 2.0 the test of contextual grounding it was missing. We will add it to the paper with Qwen included, and release it with the benchmark.
The low asking rate is also not an artefact of our labels. Two additional finance experts relabelled 300 turns blind, seeing neither our labels nor any model output, and agreed with each other at Cohen's κ = 0.975. They confirmed all 78 clarification labels and all 102 refusal labels in the sample. On these confirmed turns the four models ask on only 44.9% to 71.8% of clarification turns but refuse 88.2% to 95.1% of unanswerable ones, a gap that is significant for every model (Fisher exact p ≤ 1.6 × 10⁻⁴).