- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to decide whether to accept a generated SQL query. It trains a readout using a frozen judge’s score and hidden states, and separately examines ranking quality, threshold transfer, and the labels needed to validate error-risk claims. Experiments on WikiSQL and BIRD show gains for Qwen, but do not establish consistent transfer across judges or generators, or reliable acceptance at the 1% error target.
Critical strengths
The comparisons with scalar readouts and producer-aware threshold mixtures are useful. They provide evidence that the Qwen readout improves candidate ranking on the fixed evaluation bank beyond what threshold adjustment alone can achieve (Section 4.1)
Critical weaknesses
- The presentation is very poor.
- Many paper-specific terms are used before they are defined. For example, SOURCE and CAL already appear in the abstract and introduction, but are only explained in Section 3. Some basic terms, such as a WikiSQL “component,” are also not clearly defined. The paper needs a substantial rewrite in plain, readable language and the logic.
- There is not a single citation in the introduction to support the claims.
- There is a lot of defensive writing. For example, “Neither relative gain nor a favorable point estimate establishes deployment success” (p. 1) and “No intervention alone establishes one-percent certification” (Figure 1).
- The number of decimal places is inconsistent, e.g., 3.344% coverage versus 2.653370% risk in the same paragraph in Section 4.
- Overall, it is very difficult to follow the motivation, research question, and problem definition. The paper moves into detailed experimental results before making these basic points clear.
-
The research question is not meaningful and hard to understand. My understanding is that the paper trains a readout to improve the judgment of SQL generated by text-to-SQL systems. However, I am not convinced that the paper explains why its specific research question is meaningful beyond this narrow setting. The main experiments use fixed candidates from two SQL producer families and two judges. Although an additional Mistral producer is tested, the improvement does not clearly transfer to it (Section 7, Table 3). I do not see sufficient evidence that the findings generalize to broader settings or real-world scenarios with diverse requests.
-
The research scope is very narrow, only focusing on some small language models. More SOTA models are not evaluated, e.g. GPT series models. All of claims and experiments are established on 2~3 small language models and very outdated benchmarks, e.g. wikisql. These benchmarks already could be handled by current SOTA models. Thus, it is hard to get the value of this work.
-
Important technical details are unclear or missing. For example, the full judge prompts are not provided, and the construction of the “bounded packet of returned values” used in VALUES is not explained in enough detail. The generation setup for the original SQL candidates and the training data and procedure for the fixed Qwen VALUES adapter are also insufficiently documented. These details are needed to understand and reproduce the experiments, including whether the adapter’s training data were kept separate from the evaluation data.
Questions for authors
See the above section.