- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tackles the problem of translating natural language into SQL queries. It proposes TEG, a taxonomy-guided approach for correcting erroneous NL2SQL queries. It maps error types to SQL constructs to reconsider and corresponding edit instructions. The repairing method combines syntax-tree masking, sequential correction, and execution-guided candidate selection. The evaluation shows its effectiveness.
Critical strengths
-
Propose an automated repairing framework to errors in LLM-generated SQL queires. It utilizes an error taxonomy to guide the selection of repair solution
-
The evaluation shows that TEG outperforms existing works on different LLMs and settings. The ablation study also shows the effectiveness of the TEG modular design
Critical weaknesses
-
There have been some works published on software engineering venues (e.g., ICSE and FSE) to study the error in SQLs generated by LLMs from natural languages. Some of these works also propose automated fixing solutions. For example, MapleDoctor from FSE2026 conducts an empirical study on existing NL2SQL solutions to analyze the error types and frequency. It proposes solutions from compiler's perspective and uses rule-based syntax repair solutions, which is much faster than LLM-based solution of TEG. There are also attempts from database venues. The paper does not discuss how it is different from these works.
-
The evaluation is also insufficient. It studies and evaluates on the BUG subsect of the BIRD benchmark. However, there are other benchmarks tackling different types of tasks from BIRD. This benchmark setting raises the generalizability concerns. In addition, it is unfair to only evaluate on buggy code, as many repairing solutions (particularly those based on LLMs) are likely to turn a correct query into incorrect. Therefore, I would expect to see the mis-repair rate, and net win on the entire BIRD benchamrk. I am also a bit curious about the precision and recall of error detection.
Questions for authors
- How does TEG different from recent works in software engineering and database venues?
- Do the authors evaluate TEG on the entire BIRD benchmark?
3 comments
Response to reviewer Jifb (2/2)
[W2] Precision and Recall of Error Detection
TEG maintains repair accuracy close to that obtained with oracle error types, even though precision and recall reveal imperfect error-type detection. On NL2SQL-BUGs, Claude Opus 4.8 achieves multi-error sub-type precision, recall, and F1 of 55.1%, 32.7%, and 41.0%, respectively, with Exact Match of 7.1%. Thus, detection remains inaccurate even when partial matches receive credit. Nevertheless, with Qwen2.5-7B-Instruct as the LLM, TEG achieves 33.7% multi-error execution accuracy with predicted types versus 34.4% with oracle types, outperforming ErrorLLM's 23.5% using the same predictions. Overall accuracy is 36.1% versus 37.0% with oracle types. The predictions retain partial information, with 67.4% category-level precision. Error types also remain beneficial: the no-type TEG ablation in Table 4 achieves 31.9% overall accuracy with the same LLM, compared with 36.1% using predicted types. Thus, imperfect detection still provides useful guidance for TEG. We have added the detection metrics and extended predicted-type results to the revised paper.
| Error level | Metric (%) | Single-error | Multi-error | Overall |
|---|---|---|---|---|
| Sub-type | Precision | 65.4 | 55.1 | 56.6 |
| Recall | 61.8 | 32.7 | 35.4 | |
| F1 | 63.6 | 41.0 | 43.6 | |
| Exact Match | 52.7 | 7.1 | 16.2 | |
| Category | Precision | 70.2 | 66.9 | 67.4 |
| Recall | 66.4 | 39.6 | 42.2 | |
| F1 | 68.2 | 49.8 | 51.9 | |
| Exact Match | 55.5 | 11.2 | 20.0 |
Thanks the authors for the clarification. However, I still remain a negative attitude towards this paper.
Response to reviewer Jifb (1/2)
We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful feedback. We appreciate your positive assessment of our taxonomy-guided repair framework and the experimental evidence supporting its effectiveness.
[W1, Q1] Comparison with Prior SQL Repair Methods in SE and DB
MapleDoctor and DeepEye-SQL mainly fix syntax and execution errors, whereas TEG targets semantic errors in executable SQL. The revised paper provides detailed comparisons with SE research such as MapleDoctor and DB methods such as CatSQL, PURPLE, SQLDriller, and DeepEye-SQL. Among these, we focus on MapleDoctor (FSE 2026) and DeepEye-SQL (SIGMOD 2026) as key prior work from the two communities. MapleDoctor uses error symptoms and rule-based fixes, while DeepEye-SQL uses checker-triggered LLM revision. TEG's distinctive contribution is to map diagnosed semantic error types to AST construct classes (where) and edit operations (how), using construct-level masking to guide repairs even when executable SQL mismatches the question's intent.
TEG outperforms both baselines across all error-count groups on NL2SQL-BUGs. With Qwen2.5-7B-Instruct as the LLM, TEG achieves 37.0% overall accuracy, compared with 12.4% for MapleDoctor and 11.3% for DeepEye-SQL. Both baselines use repair procedures from their public implementations with the same model for LLM calls. We have added the NL2SQL-BUGs comparisons with MapleDoctor and DeepEye-SQL to the revised paper.
[W2, Q2] Generalizability Across Different Benchmarks
TEG improves model-generated SQL on the full BIRD development set. We fixed zero-shot Qwen2.5-7B-Instruct as the SQL generator and compared each method's refinement procedure on the same 1,534 generated queries, including both correct and incorrect SQL. No oracle correctness labels or error types were provided. All methods use this model; TEG and SQLFixAgent also use it for detection, while ErrorLLM uses its BIRD-trained detector.
TEG achieves the largest net gain, raising accuracy from 43.3% to 54.9%. It successfully repairs 195 queries and makes 17 mis-repairs (2.6% of the 664 initially correct queries), yielding a net gain of 178. Net Improvement is successful repairs minus mis-repairs. Although TEG makes more mis-repairs than MapleDoctor and DeepEye-SQL, its additional successful repairs yield net gains 22 and 42 queries higher, respectively. We have added the full BIRD development-set refiner comparison, including Fix, Mis-Repair, and Net Improvement, to the revised paper.
For the full BIRD development-set evaluation, TEG handles execution errors in addition to semantic errors. Execution results, including errors, empty results, and NULL values in the first five rows, are supplied to the detector as candidate error symptoms. Execution failures are also passed to the LLM. If the final query still fails or returns an empty result, an additional fix step uses relevant schema and database values.
TEG also improves the evaluated BIRD-CRITIC debugging tasks. Adding TEG to the official BIRD-CRITIC prompt raises Qwen2.5-7B-Instruct accuracy on its SQLite cases from 22.8% to 26.8%, with gains in all three task categories.