- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the robustness of small-scale Text-to-SQL models when database schemas and user questions provide insufficient semantic information. It proposes a two-stage framework. First, a large model uses database values, historical queries, and training-set gold SQL to synthesize schema descriptions, entity mappings, and sub-question hints, which are distilled into SynGen. Second, a downstream Noisy-SQL model is trained using noise injection and GRPO with an additional reward for identifying and correcting misleading hints. Experiments cover Spider, BIRD, robustness variants, and multiple model sizes. The appendix additionally provides cryptic-schema evaluation, human validation, and three-seed results.
Critical strengths
The paper focuses on the difficulties that small-scale Text-to-SQL models face when interpreting ambiguous database schemas and bridging the semantic gap between schemas and user questions. The authors use database example values, historical queries, and annotated training SQL to construct training data containing schema comments, entity-level hints, and sub-question-level hints through reverse semantic inference. These synthesized annotations serve as supervision for SynGen, which learns to generate auxiliary information intended to help the downstream model understand database schemas and their correspondence with user questions. A positive aspect is that the authors do not assume these enhancements are always reliable; they also consider how incorrect hints may interfere with SQL generation. To address this issue, they inject semantically similar but incorrect hints into the enhanced training data and incorporate a noise-awareness reward into GRPO training, encouraging the downstream model to identify and correct misleading information.
Critical weaknesses
W1. Missing controlled comparisons leave the independent contribution of reverse semantic inference insufficiently established. Appendix D.1 demonstrates improvements from adding the generated semantic information to other downstream systems, while Appendix D.3 compares different enhancers and teacher models. These results support the effectiveness of semantic enhancement and training SynGen. However, they do not sufficiently establish whether the gains arise specifically from the proposed reverse semantic inference, rather than general task-specific supervision and knowledge distillation. W2. The noise-aware reward assigns erroneous high scores to failed trajectories, weakening the interpretation of the training mechanism. The noise is deliberately constructed by the authors, and reward computation explicitly uses Noise_Labels. The model could therefore be required to output explicit noise predictions, enabling a verifiable reward computed directly against the known labels. The authors motivate introducing a Qwen2.5-72B judge by the diversity of free-form reasoning trajectories, but do not establish its advantage over a label-based reward. Furthermore, Figure 10 shows that 22 of the 24 trajectories judged by humans as failing to identify noise receive the judge’s maximum score, revealing substantial erroneous rewards on critical failure cases.
Questions for authors
1.What justifies the two-stage design? Could semantic enhancement and SQL generation be combined into a single model with one generation pass? How would accuracy, robustness, and end-to-end latency change relative to the current SynGen→Noisy-SQL pipeline? Do these results establish the need for a separate enhancer? 2.What practical deployment advantage does the small-model cascade offer over using the teacher directly? Could the authors compare the complete cascade against direct SQL generation by the teacher in terms of accuracy, end-to-end latency, total generated tokens, peak memory, and deployment cost, with hardware and inference settings clearly specified? Does the cost accounting include both SynGen and Noisy-SQL? Under what resource constraints and accuracy requirements does the cascade offer a more useful performance–cost trade-off? 3.Does the benefit of semantic enhancement depend on the downstream model’s capability? Could the authors compare the same strong SQL generator with and without SynGen-generated enhancements, and contrast its gains with those observed for small models? If such evidence is unavailable, should the conclusions about the enhancement strategy’s effectiveness be explicitly limited to the tested small-model settings? 4.Why is an LLM judge needed when noise labels are already known? Could the authors compare the current judge-based reward against a verifiable reward that directly scores explicit noise predictions against known labels, using identical noisy data, the same underlying rewards, and comparable training budgets? Furthermore, what causes the judge to assign maximum scores to 22 of the 24 human-rated failures in Figure 10, and how do these erroneous rewards affect training? While Table 6 supports the empirical utility of the noise reward, what additional evidence establishes that the benefits of an LLM judge justify its extra cost and reliability risks?