acceptodds
Under review as a conference paper at ICLR 2027

Why, Where, How: Taxonomy-guided Error Grounding for Code Repair in NL2SQL

Abstract

SQL queries that large language models write from natural language questions can execute successfully yet produce incorrect results, so execution alone does not reveal what to fix. An error taxonomy says *why* the query is wrong, but not *where* to look or *how* to change it. Existing methods can guide SQL correction through feedback, error reports, or generated plans alongside an unmasked query. We introduce **TEG**(**T**axonomy-guided **E**rror **G**rounding), which turns a supplied diagnosis into a structured correction input for natural language-to-SQL (NL2SQL) correction. Type-specific rules map each error type to construct classes to reconsider and an edit operation to request. TEG masks the selected constructs in the query when applicable and states that operation in an edit instruction. TEG generates candidate corrections from this input, uses execution feedback to guide candidate selection, and repeats the process one annotation at a time for queries with several errors. On NL2SQL-BUGs, TEG reaches 47.3 single-error execution accuracy and 37.0 overall with Qwen2.5-7B-Instruct. Across the model sizes and thinking modes evaluated in the main comparison, TEG outperforms all evaluated baselines on single-error queries, even when the baselines receive the same error-type annotations. With predicted types, TEG stays above direct LLM correction and ErrorLLM on single-error queries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
NL2SQL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JifbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the problem of translating natural language into SQL queries. It proposes TEG, a taxonomy-guided approach for correcting erroneous NL2SQL queries. It maps error types to SQL constructs to reconsider and corresponding edit instructions. The repairing method combines syntax-tree masking, sequential correction, and execution-guided candidate selection. The evaluation shows its effectiveness.

Critical strengths
  1. Propose an automated repairing framework to errors in LLM-generated SQL queires. It utilizes an error taxonomy to guide the selection of repair solution

  2. The evaluation shows that TEG outperforms existing works on different LLMs and settings. The ablation study also shows the effectiveness of the TEG modular design

Critical weaknesses
  1. There have been some works published on software engineering venues (e.g., ICSE and FSE) to study the error in SQLs generated by LLMs from natural languages. Some of these works also propose automated fixing solutions. For example, MapleDoctor from FSE2026 conducts an empirical study on existing NL2SQL solutions to analyze the error types and frequency. It proposes solutions from compiler's perspective and uses rule-based syntax repair solutions, which is much faster than LLM-based solution of TEG. There are also attempts from database venues. The paper does not discuss how it is different from these works.

  2. The evaluation is also insufficient. It studies and evaluates on the BUG subsect of the BIRD benchmark. However, there are other benchmarks tackling different types of tasks from BIRD. This benchmark setting raises the generalizability concerns. In addition, it is unfair to only evaluate on buggy code, as many repairing solutions (particularly those based on LLMs) are likely to turn a correct query into incorrect. Therefore, I would expect to see the mis-repair rate, and net win on the entire BIRD benchamrk. I am also a bit curious about the precision and recall of error detection.

Questions for authors
  1. How does TEG different from recent works in software engineering and database venues?
  2. Do the authors evaluate TEG on the entire BIRD benchmark?
3 comments
AuthorsOpenReview ↗

Response to reviewer Jifb (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful feedback. We appreciate your positive assessment of our taxonomy-guided repair framework and the experimental evidence supporting its effectiveness.

[W1, Q1] Comparison with Prior SQL Repair Methods in SE and DB

MapleDoctor and DeepEye-SQL mainly fix syntax and execution errors, whereas TEG targets semantic errors in executable SQL. The revised paper provides detailed comparisons with SE research such as MapleDoctor and DB methods such as CatSQL, PURPLE, SQLDriller, and DeepEye-SQL. Among these, we focus on MapleDoctor (FSE 2026) and DeepEye-SQL (SIGMOD 2026) as key prior work from the two communities. MapleDoctor uses error symptoms and rule-based fixes, while DeepEye-SQL uses checker-triggered LLM revision. TEG's distinctive contribution is to map diagnosed semantic error types to AST construct classes (where) and edit operations (how), using construct-level masking to guide repairs even when executable SQL mismatches the question's intent.

TEG outperforms both baselines across all error-count groups on NL2SQL-BUGs. With Qwen2.5-7B-Instruct as the LLM, TEG achieves 37.0% overall accuracy, compared with 12.4% for MapleDoctor and 11.3% for DeepEye-SQL. Both baselines use repair procedures from their public implementations with the same model for LLM calls. We have added the NL2SQL-BUGs comparisons with MapleDoctor and DeepEye-SQL to the revised paper.

Method1 error2 errors3+ errorsOverall
DeepEye-SQL17.310.87.711.3
MapleDoctor15.512.59.912.4
TEG (ours)47.336.729.637.0

[W2, Q2] Generalizability Across Different Benchmarks

TEG improves model-generated SQL on the full BIRD development set. We fixed zero-shot Qwen2.5-7B-Instruct as the SQL generator and compared each method's refinement procedure on the same 1,534 generated queries, including both correct and incorrect SQL. No oracle correctness labels or error types were provided. All methods use this model; TEG and SQLFixAgent also use it for detection, while ErrorLLM uses its BIRD-trained detector.

TEG achieves the largest net gain, raising accuracy from 43.3% to 54.9%. It successfully repairs 195 queries and makes 17 mis-repairs (2.6% of the 664 initially correct queries), yielding a net gain of 178. Net Improvement is successful repairs minus mis-repairs. Although TEG makes more mis-repairs than MapleDoctor and DeepEye-SQL, its additional successful repairs yield net gains 22 and 42 queries higher, respectively. We have added the full BIRD development-set refiner comparison, including Fix, Mis-Repair, and Net Improvement, to the revised paper.

MethodFixMis-RepairNet improvementExecution accuracy (%)
Generator only---43.3
Self-Debug (simple)1210243.4
Self-Debug (expl.)90504045.9
DIN-SQL77344346.1
MAC-SQL106310350.0
SQLFixAgent1691415553.4
ErrorLLM7987147.9
DeepEye-SQL141513652.2
MapleDoctor1671115653.5
TEG1951717854.9

For the full BIRD development-set evaluation, TEG handles execution errors in addition to semantic errors. Execution results, including errors, empty results, and NULL values in the first five rows, are supplied to the detector as candidate error symptoms. Execution failures are also passed to the LLM. If the final query still fails or returns an empty result, an additional fix step uses relevant schema and database values.

TEG also improves the evaluated BIRD-CRITIC debugging tasks. Adding TEG to the official BIRD-CRITIC prompt raises Qwen2.5-7B-Instruct accuracy on its SQLite cases from 22.8% to 26.8%, with gains in all three task categories.

AuthorsOpenReview ↗

Response to reviewer Jifb (2/2)

[W2] Precision and Recall of Error Detection

TEG maintains repair accuracy close to that obtained with oracle error types, even though precision and recall reveal imperfect error-type detection. On NL2SQL-BUGs, Claude Opus 4.8 achieves multi-error sub-type precision, recall, and F1 of 55.1%, 32.7%, and 41.0%, respectively, with Exact Match of 7.1%. Thus, detection remains inaccurate even when partial matches receive credit. Nevertheless, with Qwen2.5-7B-Instruct as the LLM, TEG achieves 33.7% multi-error execution accuracy with predicted types versus 34.4% with oracle types, outperforming ErrorLLM's 23.5% using the same predictions. Overall accuracy is 36.1% versus 37.0% with oracle types. The predictions retain partial information, with 67.4% category-level precision. Error types also remain beneficial: the no-type TEG ablation in Table 4 achieves 31.9% overall accuracy with the same LLM, compared with 36.1% using predicted types. Thus, imperfect detection still provides useful guidance for TEG. We have added the detection metrics and extended predicted-type results to the revised paper.

Error levelMetric (%)Single-errorMulti-errorOverall
Sub-typePrecision65.455.156.6
Recall61.832.735.4
F163.641.043.6
Exact Match52.77.116.2
CategoryPrecision70.266.967.4
Recall66.439.642.2
F168.249.851.9
Exact Match55.511.220.0

Thanks the authors for the clarification. However, I still remain a negative attitude towards this paper.

Reviewer_Hf3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TEG, a taxonomy-guided framework for repairing semantic errors in NL2SQL queries. TEG maps error types to SQL AST constructs and corresponding edit operations, applies type-specific masking to guide LLM-based correction, and uses execution feedback for candidate selection. For queries containing multiple errors, it performs sequential correction. Experiments on NL2SQL-BUGs demonstrate improvements over several existing baselines.

Critical strengths
  1. Clear Motivation: The paper addresses an important problem in NL2SQL: repairing SQL queries that execute successfully but produce semantically incorrect results.

  2. Intuitive Method: TEG provides a straightforward way to translate error types into targeted SQL modifications through AST-based masking and edit instructions.

  3. Comprehensive Comparisons: The paper evaluates multiple LLMs and baselines, including experiments with predicted error types and component ablations, demonstrating consistent improvements in the main comparisons.

Critical weaknesses
  1. Dependence on Error Annotations: TEG heavily relies on error-type annotations, which are generally unavailable in real-world NL2SQL scenarios. The paper reports only 52.7% exact-match accuracy for single-error type prediction, dropping to 8.8% for two-error queries and 3.5% for queries with three or more errors. Moreover, end-to-end evaluation with predicted types is limited to single-error queries, raising concerns about practical applicability.

  2. Missing Direct SQL Generation Baseline: The paper should evaluate whether the same LLMs can directly generate correct SQL from the original natural language questions. Since NL2SQL-BUGs contains constructed erroneous queries, the models might already solve many of these questions correctly without repair. Such a comparison would help establish the practical value of the proposed correction framework.

  3. Limited Dataset Evaluation: All experiments are conducted on NL2SQL-BUGs. Additional evaluations on other datasets would help establish the generalizability of TEG across different SQL errors and database scenarios.

  4. Insufficient Isolation of the Masking Contribution: The masking ablation also changes certain edit instructions and disables the retry mechanism. Therefore, the observed performance drop cannot be attributed solely to masking. A more controlled ablation is needed to establish the effectiveness of this core component.

Questions for authors

Handling Dependencies Between Error Types: SQL errors are often interdependent rather than independent. For example, an error of type A may cause another error of type B. Can TEG effectively handle such dependencies, given that it processes error types sequentially in a fixed order?

3 comments
AuthorsOpenReview ↗

Response to reviewer Hf3y (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful feedback. We appreciate your positive assessment of our motivation, intuitive design, and evaluation across models and settings.

[W1] Dependence on Error Annotations

TEG remains effective with imperfect predictions for multiple errors. With Qwen2.5-7B-Instruct as the LLM, overall accuracy is 36.1% with detector predictions and 37.0% with oracle types. The extended evaluation uses the same detector as Table 3 and identical predictions for TEG and ErrorLLM. The table includes the original single-error results and the new multi-error results. Although only 7.1% of multi-error predictions exactly match the annotations, TEG achieves 33.7% accuracy versus 34.4% with oracle types and 23.5% for ErrorLLM using the same predictions. Error-type information remains beneficial: in Table 4, removing types yields 38.2%, 34.7%, and 21.1% accuracy for one-, two-, and three-or-more-error queries, respectively, and 31.9% overall. The latter is 4.2 percentage points below TEG with predicted types, showing that useful guidance does not require an exactly correct error-type set. We have added the multi-error and overall results to the revised paper.

MethodSingle-errorMulti-errorOverall
Base30.022.123.7
ErrorLLM (oracle)33.621.624.0
ErrorLLM (predicted)32.723.525.3
TEG (oracle)47.334.437.0
TEG (predicted)45.533.736.1

[W2] Missing Direct SQL Generation Baseline

TEG outperforms direct generation, including with predicted error types. With oracle error types, the gains on single-error queries are 16.4, 20.9, and 12.8 percentage points for the 7B, 14B, and 32B models, respectively. Direct generation receives the natural-language question and database schema without the erroneous SQL, using zero-shot greedy decoding. For 7B over the full dataset, oracle-type TEG achieves 37.0% accuracy versus 22.4% for direct generation and 23.7% for Base repair. Using predicted types, 7B TEG also achieves 45.5% on single-error queries and 36.1% overall, compared with direct generation's 30.9% and 22.4%, respectively. Thus, direct generation with the same model does not reproduce TEG's repair gains. We have added the settings and results to the revised paper.

ModelSplitDirect SQL generationBaseTEG
7Bsingle30.930.047.3
14Bsingle31.835.552.7
32Bsingle44.541.857.3
7Boverall22.423.737.0

[W3] Limited Dataset Evaluation

TEG improves model-generated SQL beyond NL2SQL-BUGs on the full BIRD development set. We fixed zero-shot Qwen2.5-7B-Instruct as the SQL generator and compared each method's refinement procedure on the same 1,534 generated queries, including both correct and incorrect SQL. No oracle correctness labels or error types were provided. The evaluated TEG configuration repairs both execution errors and semantic errors, as detailed below. All methods use Qwen2.5-7B-Instruct as the LLM; TEG and SQLFixAgent also use it for detection, while ErrorLLM uses its BIRD-trained detector. Other configurations and correction-target selection follow the respective methods. We have added the full BIRD development-set refiner comparison, including Fix, Mis-Repair, and Net Improvement, to the revised paper.

MethodExecution accuracy (%)
Qwen2.5-7B-Instruct43.3
Self-Debug (simple)43.4
Self-Debug (expl.)45.9
DIN-SQL46.1
MAC-SQL50.0
SQLFixAgent53.4
ErrorLLM47.9
DeepEye-SQL52.2
MapleDoctor53.5
TEG54.9

For the full BIRD development-set evaluation, TEG handles execution errors in addition to semantic errors. Execution results, including errors, empty results, and NULL values in the first five rows, are supplied to the detector as candidate error symptoms. Execution failures are also passed to the LLM. If the final query still fails or returns an empty result, an additional fix step uses relevant schema and database values.

TEG also improves the evaluated BIRD-CRITIC debugging tasks. Adding TEG to the official BIRD-CRITIC prompt raises Qwen2.5-7B-Instruct accuracy on its SQLite cases from 22.8% to 26.8%, with gains in all three task categories.

AuthorsOpenReview ↗

Response to reviewer Hf3y (2/2)

[W4] Clarifying the Independent Effect of Masking

Masking alone improves single-pass repair accuracy. Using Qwen2.5-7B-Instruct and greedy decoding, both variants receive the same prompt and error annotations; only the replacement of selected SQL constructs with <MASK> differs. For multi-error queries, both variants perform one correction, with all applicable masks applied together in the masked variant. Adding masking raises overall accuracy from 19.9% to 26.4%, with gains in every error-count group. This isolates masking in a single-pass setting; the original sequential-pipeline ablation also changes edit instructions and retry behavior. We have clarified this distinction and added the controlled comparison to the Appendix.

Method1 error2 errors3+ errorsOverall
Base + types31.818.214.119.9
Base + types + masking42.724.916.926.4

[Q1] Handling Dependencies Between Error Types

An earlier repair can resolve multiple related error types. Although the annotation order is fixed, each step re-grounds its targets on the SQL produced by the preceding step, allowing later corrections to reflect earlier changes. With Qwen2.5-7B-Instruct and oracle types, 32.3% reach the correct result before the final correction step, consistent with an earlier edit resolving related errors together. The table separates these cases from queries first corrected at the final step; percentages use all queries in the corresponding multi-error group as the denominator.

Repair outcome (% of queries)Overall
Correct before the final step32.3
Remained correct at the final step28.7
Became incorrect by the final step3.6
First correct at the final step5.7
Correct after the final step34.4

Sequential repair still outperforms correcting all error types at once. The w/o sequential correction ablation in Table 6 reduces accuracy from 36.7% to 32.3% on two-error queries and from 29.6% to 23.9% on queries with three or more errors. A further 5.7% of multi-error queries first become correct at the final step, showing that later corrections contribute additional repairs. We have added this analysis to the revised paper.

Thank you for the detailed response and additional experiments. While they address some of my questions, my main concerns regarding overall contribution and practical applicability remain, so I will maintain my original score.

Reviewer_6tXjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed Taxonomy-guided Error Grounding (TEG) for correcting erroneous SQL queries generated from natural language. It maps the diagnosis to SQL construct classes and an edit instruction. For applicable error types, it masks the selected constructs before prompting an LLM to generate repairs. TEG further uses execution feedback and LLM-based candidate selection, and processes multiple error annotations sequentially.

Critical strengths

S1. The error grounding problem is practical and this paper provides a structural workflow to solve this problem.

S2. The proposed method is easy to understand, and the empirical evaluation is relatively comprehensive.

Critical weaknesses

W1. The contribution of the masking mechanism is not clearly isolated. The paper frames TEG’s contribution as mapping error types to the where and how of SQL repair. However, as described in Section 2, ErrorLLM also uses error types to identify relevant AST nodes and schema elements for refinement. A key distinction of TEG is that it masks selected SQL constructs before repair. Although the paper includes a no-masking ablation, this variant may also change the edit instructions and, in the multi-error setting, disables the mask-failure retry branch. The results therefore do not isolate the effect of masking from the other components of the correction pipeline.

W2. The evaluation of predicted error types is limited to single-error queries. The main experiments use oracle annotations from NL2SQL-BUGs, the only evaluated dataset. Although TEG is designed to process multiple errors sequentially, the predicted-type experiment covers only queries annotated with a single error. Consequently, it remains unclear how the method performs when predicted error types for a multi-error query are incomplete or incorrect. Moreover, TEG retains the supplied annotations throughout sequential repair, even though an earlier edit may change the query or introduce a new error; later edits may also undo earlier repairs.

Questions for authors

Q1. The masking rules select classes of SQL constructs rather than exact faulty locations, so correct constructs may also be masked. How often does this occur, and how does the amount or proportion of masked SQL affect repair accuracy?

Q2. For Missing and Rewrite errors, TEG leaves the SQL unmasked and relies on an edit instruction. How much of the overall improvement comes from these error groups versus the groups where masking is applied?

Q3. The ErrorLLM baseline replaces its learned question–schema encoder with lexical matching. How much does this adaptation affect its performance, and can the authors compare TEG with ErrorLLM using the latter’s original schema-context extraction procedure?

Q4. Will other debugging benchmarks, such as Bird-Critic, be used to evaluate TEG's effectiveness?

3 comments
AuthorsOpenReview ↗

Response to reviewer 6tXj (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful feedback. We appreciate your positive assessment of the practical relevance of error grounding, the clarity of our approach, and the evaluation.

[W1] Clarifying the Independent Effect of Masking

Masking alone improves single-pass repair accuracy. Like ErrorLLM, TEG uses error types to guide structural refinement; TEG additionally masks the selected SQL constructs before repair. Using Qwen2.5-7B-Instruct and greedy decoding, both variants receive the same prompt and error annotations; only the replacement of selected SQL constructs with <MASK> differs. For multi-error queries, both variants perform one correction, with all applicable masks applied together in the masked variant. Adding masking raises overall accuracy from 19.9% to 26.4%, with gains in every error-count group. This isolates masking in a single-pass setting; the original sequential-pipeline ablation also changes edit instructions and retry behavior. We have clarified this distinction and added the controlled comparison to the Appendix.

Method1 error2 errors3+ errorsOverall
Base + types31.818.214.119.9
Base + types + masking42.724.916.926.4

[W2] Robustness to Error Type Predictions in Multi-Error Queries

TEG remains effective with imperfect predictions for multiple errors. We extended the predicted-type evaluation using the same detector as in Table 3 and Qwen2.5-7B-Instruct as the LLM. TEG and ErrorLLM receive identical predictions; the table combines the original single-error results with the new multi-error results. Only 7.1% of multi-error predictions exactly match the annotations, yet TEG achieves 33.7% accuracy, compared with 23.5% for ErrorLLM using the same predictions and 22.1% for Base. Overall accuracy is 36.1% with predicted types versus 37.0% with oracle types. Error-type information still improves repair: the no-type TEG ablation in Table 4 achieves 31.9% overall accuracy with the same LLM, 4.2 percentage points below TEG with predicted types. Thus, the small oracle-to-predicted gap indicates robustness to imperfect predictions, not that the types are unnecessary. We have extended Table 3 with the multi-error and overall results.

MethodSingle-errorMulti-errorOverall
Base30.022.123.7
ErrorLLM (oracle)33.621.624.0
ErrorLLM (predicted)32.723.525.3
TEG (oracle)47.334.437.0
TEG (predicted)45.533.736.1

[W2] Reliability of Sequential Repair under Evolving Error Annotations

TEG re-grounds each repair on the updated SQL. The supplied error annotations remain fixed. If an earlier edit removes a later masking target, the retry template provides the current SQL and the previous and current error types so that the model can account for the earlier modification.

Query groupCorrect after the final stepCorrect before final stepFirst correct at final stepCorrect earlier but incorrect after the final step
Multi-error queries34.4%32.3%5.7%3.6%

Later repairs undo a correct intermediate result in 3.6% of all evaluated multi-error queries. Using Qwen2.5-7B-Instruct and oracle error types, we compared intermediate and final execution results with the reference result offline. We counted a query as losing correctness when it became correct at an intermediate step but was incorrect at the end. This measures loss of intermediate correctness, rather than all forms of error propagation. We have added this analysis to the Appendix.

AuthorsOpenReview ↗

Response to reviewer 6tXj (2/2)

[Q1] Masking Scope and Its Impact on Repair Accuracy

TEG restores most correct masked constructs and changes more erroneous constructs. On single-error queries evaluated with Qwen2.5-7B-Instruct, correct constructs are also masked in 71.7% of queries where masking is applied. Recovery is measured at the AST-node level on the originally correct constructs selected by TEG's masking rules, using the same construct set for all methods. TEG preserves or regenerates 87.0% of these constructs, below TEG w/o masking (96.1%) and Base (94.7%). However, it changes 70.7% of erroneous constructs, compared with 46.7% and 41.3%, respectively, and achieves higher repair accuracy on these queries.

MethodMasked correct constructs recovered (%)Erroneous construct changed (%)Execution accuracy on over-masked queries (%)
TEG87.070.745.5
TEG w/o masking96.146.739.4
Base94.741.330.3

Repair accuracy shows no consistent decline across increasing masking ratios. Across the nonzero masking-ratio groups, accuracy is 50.0%, 45.2%, and 51.6%, and exceeds both comparison methods in every group. These results compare different query groups rather than varying the mask ratio for a fixed query. We have added both analyses to the Appendix.

Masked-token ratioQueries (% of total)BaseTEG w/o maskingTEG
0% (no mask)16.4%22.238.938.9
3–17%27.3%40.043.350.0
17–43%28.2%25.832.345.2
44–66%28.2%29.048.451.6
Overall100%30.040.947.3

[Q2] Contribution of Non-Masking Error Categories

TEG improves both groups, with masking-applicable errors contributing most of the overall gain. In the single-error evaluation with Qwen2.5-7B-Instruct, Mismatch/Redundancy errors (masking group) account for approximately 87% of queries, while Missing/Rewrite errors (non-masking group) account for approximately 13%. TEG improves over Base by 16.7 and 21.4 percentage points, respectively; despite the larger within-group gain for non-masking errors, their smaller share limits their contribution to the overall improvement. The gain over CoT is also larger in the masking group: 12.5 versus 7.1 points. We have included this groupwise comparison in the Appendix.

MethodMasking groupNon-masking groupOverall
Base30.228.630.0
CoT34.442.935.5
TEG46.950.047.3

[Q3] Fairness of the ErrorLLM Baseline Adaptation

Our lexical adaptation did not disadvantage ErrorLLM: it scores 24.0% overall, above the 21.5% of a reproduction following the official repository, and TEG outperforms both. Lexical matching was a fallback for ErrorLLM's unavailable trained question–schema encoder. The official repository included the encoder implementation but not its training code or pretrained weights; running it untrained linked question parts to most schema elements, producing contexts exceeding 300K tokens for some inputs. For the additional comparison, we followed the official repository as closely as possible, retaining its QSS and AST components, LocLLM/RefLLM refinement, and prompts. We substituted RAT-SQL-style question–schema linking because ErrorLLM Sec. 2.3 explicitly identifies RAT-SQL as an option.

TEG's advantage persists in the reproduction based on ErrorLLM's official repository. In the table, “Official repo” denotes this reproduction with RAT-SQL-style linking substituted for the unavailable trained question–schema encoder. Both methods use Qwen2.5-7B-Instruct and oracle error types. TEG achieves 37.0% overall accuracy, compared with 24.0% for lexical matching and 21.5% for the official-repository reproduction. We have clarified these reproduction settings in the Appendix.

MethodSingle-errorMulti-errorOverall
ErrorLLM (Lexical)33.621.624.0
ErrorLLM (Official repo)34.518.221.5
TEG47.334.437.0

[Q4] Generalizability to Other Debugging Benchmarks

TEG generalizes as a plug-in to the evaluated BIRD-CRITIC debugging tasks. We evaluated all 500 SQLite cases, comparing the official prompt alone with the same prompt augmented with TEG. Qwen2.5-7B-Instruct was used for both error-type detection and SQL repair. TEG improves performance in all three categories, raising overall performance from 22.8% to 26.8%. These results support TEG's generalizability beyond NL2SQL-BUGs to the evaluated BIRD-CRITIC debugging tasks.

QueryPersonalizationManagementOverall
Official prompt25.416.325.322.8
TEG28.922.726.726.8

Thank you for the response. The response addressed some of my concerns, and I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.