- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies cross-entry consistency in Text-to-SQL context banks and model generalization beyond the bank. The authors decompose SQL queries into semantic decisions, use two LLM judges to generate alignment, conflict, and novelty labels, and distill these labels into a pairwise scorer. They further propose a Bank Health Index, a two-track cleaning protocol, and CAT benchmarks organized by semantic distance. The experiments examine the effects of cleaning on fine-tuning, example conventions on inference, and increasing question distance on the benefit of context.
Critical strengths
- The focus on cross-entry convention consistency has practical value. The paper distinguishes whether each SQL query answers its own question from whether different examples follow the same conventions. It compares matched, opposite, mixed, and no-example conditions. Table 3 reports a 13.3-percentage-point accuracy gap between matched and opposite conventions. The experiment on naturally grouped questions additionally compares cases where the wording specifies the convention with cases where it does not, providing evidence that examples influence model behavior by resolving unspecified decisions.
- The cleaning experiments include meaningful controls. Table 2 compares the cleaned training set with an equally sized random subset stratified by database and SQL length, with improvements at every reported epoch. The cross-check against expert corrections in BIRD-Platinum also supports the presence of genuine annotation problems among the removed entries. The authors explicitly acknowledge that this experiment evaluates cleaning as a whole and does not isolate the contribution of cross-entry consistency.
Critical weaknesses
-
Key semantic judgment procedures lack complete reproducibility details.
The method relies on LLM judgments for semantic-unit grounding, ACN labeling, and cleaning decisions, but the complete prompts, output schemas, and post-processing implementation are not provided. The supplied PDF contains no accessible code link. Since the judge protocol was refined over 14 calibration rounds, these missing details are important for reproducing the labels and cleaning results.
-
The overall workflow is insufficiently clear.
The connections among annotation, scorer training, bank cleaning, and CAT construction are difficult to follow from the text alone. An overall framework diagram showing the data flow, separating training from application, and identifying the roles of the SQL parser, scorer, and LLM judges would make the method easier to understand.
Questions for authors
- Could you provide the implementation materials needed to reproduce the core pipeline? Could you share anonymous code and the final prompts, output schemas, and post-processing rules for the core pipeline? A worked example showing how semantic units are matched and converted into an ACN label would also help.
- What are the actual token usage and end-to-end computational costs? Please report per-pair teacher token usage, total student training tokens, and the end-to-end cost and runtime of bank cleaning and CAT construction. How much does distillation save compared with directly using LLM judges?
- Please consider expanding the ethics statement to describe human expert participation. Appendix C.1 mentions expert participation in 14 calibration rounds. Briefly describing the expert's background, involvement, and workload, and, where applicable, payment or compensation arrangements, would improve transparency about the human calibration process and make the ethics statement more complete.
2 comments
Thank you to the authors for their responses and additional experiments. I believe some of my experimental concerns have been addressed, but I remain largely unconvinced about the idea.
Moreover, the revisions to the PDF are extensive (and cover more than half of the original main draft), and I don’t have the bandwidth to fully re-evaluate the revised manuscript and all the additional experiments during the rebuttal period. Given the substantial presentation issues in the original submission, I believe the paper would benefit from a resubmission with a clearer focus on contributions and experimental evidence. Hence, I will maintain my current score.
Our Response to Reviewer zaXZ
We thank Reviewer zaXZ for the constructive feedback on reproducibility, workflow clarity, compute reporting, and ethics documentation; we have addressed all four points in the revised paper and supplementary archive.
Reproducibility materials, prompts, schemas, and worked example (Weakness 1 and Question 1)
We agree completely and have uploaded an anonymized supplementary archive containing the full pipeline code, verbatim prompts (for unit grounding, ACN labeling, Track A auditing, Track B adjudication, and CAT probe generation and verification), exact JSON output schemas for every LLM call, the deterministic post processing code that turns a judge's unit map into an ACN profile, the complete convention rulebook with conditions, exceptions, and SQL examples, and the scorer's input format and training code. In the revised paper, Appendix K also provides the verbatim prompts alongside a complete worked example taking one real pair through every stage: unit extraction, span assignment, role matching, per role scoring, and the final ACN profile.
End to end workflow diagram (Weakness 2)
Thank you for this suggestion. We added an overview diagram in Figure 2 (caption in red) tracing the data flow from end to end and separating one time training from application. Panel 1 shows scorer creation, which runs once: a SQL parser extracts decision units, LLM judges label pairs under human calibrated rules, and their consensus labels join exact programmatic variants to train the scorer. Panels 2 and 3 apply the frozen scorer, the only component that runs over all pairs: in Panel 2 its outputs compute the Bank Health Index directly and select candidate pairs for Track B rulebook adjudication alongside linear Track A auditing, while in Panel 3 it keeps CAT probes that fall in their target distance band with no conflict before LLM SQL verification. Every box is color coded by whether the work is done by deterministic code, the local scorer, an LLM call, or a human.
Token usage, runtime, and distillation savings (Question 2)
Appendix B now compares the full compute and token cost of the local scorer against direct LLM judging. Training the scorer is a one time job taking 30.8 hours on a single 80GB GPU. At inference it runs locally at 5 to 9 pairs per second, so 100,000 pairs take roughly 3 to 6 GPU hours. Because BIRD-dev alone has more than 100,000 same database pairs, auditing a bank without distillation would require over 200,000 frontier LLM calls on BIRD-dev at every audit. With the local scorer, only flagged pairs go to the LLM judges: just 2,127 pairs on BIRD-dev (about 2% of pairs) and 3,101 on BIRD-train (Table 10), cutting pairwise frontier LLM calls by roughly 98%.
Human expert participation in the Ethics Statement (Question 3)
Thank you for raising this. We expanded the Ethics Statement in the revised paper (in red) to describe human participation clearly: both the expert who adjudicated the 14 calibration rounds (Appendix C.1) and the three blind annotators in our new validation study (Appendix J) are authors of this paper working on Text-to-SQL. No external or crowd annotators were recruited, so no compensation was involved, and all annotated material comes strictly from public benchmark questions and SQL.