acceptodds
Under review as a conference paper at ICLR 2027

DriftBench: An Execution-Based Benchmark for LLM SQL Query Repair Under Schema Evolution

Abstract

Production analytics break during schema evolution: a benign table or column rename can invalidate thousands of downstream queries. Large language models (LLMs) are an attractive repair tool. However, the field lacks execution-based benchmarks that (i) guarantee the drift actually affects the query, (ii) preserve semantics through migrated data, and (iii) score correctness beyond syntactic validity. We present , a procedural generator and evaluation harness for SQL repair under schema evolution. It builds paired original and drifted databases in isolated namespaces, migrates data through canonical migration plans, and derives oracle repairs by AST rewriting, so a repair counts as correct only if it reproduces the pre-drift result set. Across 18,624 system evaluations spanning seven LLMs, a deterministic schema-diff oracle, and a fuzzy heuristic, the best model reaches 92.0% semantic equivalence on the 2,000-instance rename benchmark against 13.7% for the heuristic. Under the default prompt, execution success overstates repair quality for every LLM on every rename suite; on the 292-instance rename suite, the gap is 5.9–19.9 points for LLMs and 53.1 for the heuristic; given both schema versions, a deterministic diff solves renames perfectly, yet on structural drift it collapses to 0–6.7% while LLMs reach 100%; and on column semantic transforms, six of nine systems execute 98–100% of the time while only two exceed 20% equivalence. To test transfer, we mine the histories of 40 open-source applications, identify 855 schema-verified renames across 34 of them, and build a 300-instance real-migration slice. There, seven current models execute 94–100% of the time but are correct on only 75–90%, and 93% of their failures are queries that run and return wrong rows; the real migration's rename list lifts them to 93–100%. Model rankings on the synthetic held-out-vocabulary and cross-domain suites predict real-slice rankings (Spearman and ), and the semantic-transform gap persists in current models (46–87% equivalence at 100% execution).

Primary area:
datasets and benchmarks
Keywords:
benchmark, schema evolution, SQL query repair, large language models, execution-based evaluation, semantic equivalence, database migration, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qTDNOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced DriftBench, a benchmark + eval harness for repairing SQL queries following schema drift. The task input consists of an original (failing) query and its error message, and correctness is defined by equivalent records returned by the fixed query (candidate) on the migrated DB vs. the original query on the original DB. Several experiments are run in full history, current schema only, and metadata assisted settings. The migrated databases are constructed with rename, table split, and unit change operations, with held out real dataset mined from genuine project histories.

Critical strengths

The benchmark seems to address a real world concern with convincing execution-based construction. The benchmark enforces drift impact by ensuring that queries succeed originally, fail post-migration, and the oracle query result matches as expected. Moreover the "true" distribution of migrations/schema drift is somewhat represented from mined histories.

Critical weaknesses

Underspecification in R2. Some inputs (query + error message) may not be resolvable with this information alone. Therefore it's unclear whether equivalence failure in this setting is a model shortcoming or a task specification issue. Can this ambiguity be automatically audited.

Some of the current models seem to leave limited headroom on this benchmark with glm-5.3-flash or deepseek-v4.1-flash approaching scores in the 90-95 range. This potentially remains useful as a diagnostic, the discriminative ability of this benchmark for future systems is less clear.

Questions for authors
  1. are there cases where case and whitespace normalization affect SQL results?
  2. are there preliminary results of SOTA models on a small subset of hardest tasks?
4 comments

We thank the reviewer for the positive assessment and the precise questions.


W1: is R2 underspecified, and can this be audited? R2 also includes the drifted schema (columns and types). We audited S8 automatically, using its 12 models as a panel together with the paired R3 runs (App. J).

  • Most failures are capability failures. 89.1% of R2 failures are on instances that another model solves from the identical input.
  • Only 2.3% of instances are underdetermined. Only 7 of 300 instances are solved by no model under R2. All 7 are solved once the rename list is given (R3). They are renames with no lexical link, e.g. invoices.closed → auto_advance and users.channel → pubsub_token.
  • Excluding them shifts every model by +1.2 to +2.1 pp and leaves the ranking unchanged.
  • Lexical ambiguity explains much of the rest. When the true target is the most string-similar candidate (174/300 instances), mean R2 Equiv is 89.8%; otherwise it is 62.8%.
  • In the release, we will flag the 7 instances and include the audit.

 

W2 / Q2: headroom, and SOTA models on the hardest tasks (§7.7, App. I.4).

  • Hard subset (139 instances solved by at most 4 of 7 current models): Claude-Opus-5.5 91.4%, GPT-6-Sol 60.4%, Gemini-3.1-Pro (preview) 38.8%.

  • Complete suites (to avoid selection effects):

    • real migrations: 92.7%, 85.7%, 91.3%;
    • column transform: 100%, 91.3%, 31.7%;
    • timestamp → epoch: 41%, 47%, 33%.

    Headroom remains even for frontier models, and the suites separate them.

 

Q1: does case or whitespace normalization affect results? We re-scored all 43,841 stored repairs with a comparator that keeps case and whitespace. No verdict changes (0 of 31,662; App. G).

We have uploaded an updated version of the revised manuscript with changes highlighted in blue to make it easier to identify the modifications discussed in our responses. The scientific content and results remain unchanged from the previously uploaded revision.

Thanks for the responses, the new analysis and harder subset seem to adequately address my main identified weaknesses, so increasing score by 1.

Dear Reviewer,

Thank you for taking the time to reconsider our work and for your positive reassessment. We sincerely appreciate your constructive feedback, which helped us strengthen the paper through additional analyses and experiments. We are pleased that these additions addressed your main concerns.

 

Kind regards,

The Authors

Reviewer_EALKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DriftBench, a benchmark for evaluating the ability of large language models (LLMs) to fix SQL queries. When table names, column names, or database structures change, the original SQL may no longer execute correctly. The authors test whether LLMs can correctly fix SQL queries by automatically generating database changes and the corresponding SQL queries requiring correction. Unlike methods that merely check whether the SQL executes successfully, DriftBench also compares whether the query results before and after correction are consistent. The authors tested multiple LLMs and traditional methods, supplementing their experiments with database migration logs from real open-source projects. The experiments revealed that while many SQL queries executed successfully, the results they returned were incorrect.

Critical strengths
  1. Database structure changes causing SQL statements to fail is a real problem. This paper focuses not only on whether the SQL can be re-executed, but also on whether the corrected version yields the correct results. Furthermore, the authors automatically generate database changes, the SQL statements requiring correction, and the corresponding correct answers and they then perform checks to ensure that the original SQL indeed fails and that the corrected SQL restores the original query results. This makes the entire evaluation process clear and facilitates reproducibility.

  2. The authors compared multiple LLMs and traditional repair methods, analyzed the impact of different types of database changes and additional prompts, and conducted supplementary tests using migration logs from real open-source projects. The experiments demonstrated that the successful execution of an SQL statement does not necessarily mean the repair was correct and this finding holds some practical value for real-world applications.

Critical weaknesses
  1. The benchmark is not particularly challenging and does not adequately evaluate newer models. I believe that some tasks are already relatively straightforward for stronger models. For example, in Table 1 of the main text, Phi-4-14B and Gemma3-27B achieved accuracy rates exceeding 90% on the main Rename task. In Table 3, GPT-4o-mini, Gemini-2.5-Flash-Lite, and Phi-4-14B all achieved 100% accuracy on the Table Split task. Although the authors tested newer models in follow-up experiments, they did not evaluate them on the full 2,000-row main test set. Furthermore, the truly challenging Column Semantic Transform task consists of only 104 samples. Therefore, I question whether the current benchmark is sufficiently challenging and capable of effectively distinguishing the capabilities of SOTA models. I recommend that the authors include newer SOTA models in the main experiments and add more complex SQL repair tasks.

  2. The scope of the Benchmark tasks is relatively limited, and the automatically generated data and reference answers may also affect the reliability of the evaluation. This paper primarily tests three types of database changes: Rename, Table Split, and Column Semantic Transform. However, in real-world scenarios, more complex changes may occur, such as field deletion, table merging, or multiple changes happening simultaneously. Furthermore, although the authors used database migration records from real projects, the SQL queries and data were still automatically generated. I am also concerned about whether the automatically generated reference answers are sufficiently reliable. The authors use sqlglot to modify the SQL and then verify the results by checking for consistency; however, the fact that the results match on a limited dataset does not necessarily mean the SQL statements are logically equivalent. Furthermore, samples that fail verification are discarded outright, which may cause the final benchmark to favor simpler tasks. I recommend that the authors provide further details on the number of discarded samples and the reasons for their exclusion, and include more complex SQL repair tasks that more closely resemble real-world scenarios.

Questions for authors
  1. Why were the newer models evaluated in Section 7.6 and the appendix not included in the main 2,000-instance S1 benchmark? Have the authors evaluated these models on S1? I am concerned that the main benchmark may already be relatively easy for stronger models.

  2. How many generated SQL instances were discarded during the validation process, and what were the main reasons? In particular, were more complex queries more likely to fail validation and be excluded from the final benchmark?

  3. Were the automatically generated oracle repairs manually checked by human annotators or SQL experts? If not, how do the authors ensure that these repairs are semantically correct, given that matching execution results on a limited dataset does not necessarily guarantee logical equivalence?

5 comments

We thank the reviewer for the thoughtful review and concrete suggestions.


W1 / Q1: newer models on S1, and difficulty.

  • S1 (§7.1, Table 2). We have now run the current models on S1: the same 2,000 instances on PostgreSQL, through the §7.6 endpoint. They reach 88.8–94.9% at 98.8–100% execution. We agree with the reviewer that the default-vocabulary renames of S1 are close to saturation for current models. Mixed drift remains the hardest category (87.3% on average for the current models).

  • Harder settings, and the frontier models on them (Claude-Opus-5.5, GPT-6-Sol, Gemini-3.1-Pro (preview); §7.7, Table 6, App. I):

    1. Six new drift families (600 instances):
      • 11 cost-efficient models: 70.8% equivalent at 98.3% execution.
      • Frontier models: 90.0%, 87.5%, and 74.7%.
      • Timestamp → epoch: 41%, 47%, and 33% for the frontier models.
    2. Real migrations (S8):
      • Frontier models: 92.7%, 85.7%, and 91.3%.
      • With the projects' full schemas: all 11 cost-efficient models drop (−5.4 pp on average).
    3. Column transform (S4): the frontier models reach 100%, 91.3%, and 31.7%.
    4. Hard subset (139 instances that at most 4 of 7 current models solve): 91.4%, 60.4%, and 38.8%.

    The benchmark therefore still separates frontier models, by up to 68 pp on a single suite.

  • Raising difficulty later. The generator is parameterized by drift family, vocabulary, domain, and schema size, so it can be made harder as models improve.

 

W2: scope of the drift types.

  • Added (S9, App. I.3):
    • table merge: customers and orders are denormalized into one table, so customer-level queries need de-duplication, and a naive COUNT(*) counts orders;
    • lookup normalization;
    • column split;
    • timestamp → epoch;
    • two changes applied at once: a table rename plus a unit change, and a split plus a table and key rename.
  • Simultaneous renames were already covered (1–4 operations per S1 instance, and mixed drift).
  • Field deletion is different in kind. When a query needs a dropped column, no result-preserving repair exists, so it needs abstention scoring. We list it as future work (App. E).
  • The real slice. It uses real schemas and real renames with templated queries and synthetic rows. Projects contain almost no self-contained queries over the renamed identifiers (10 candidates, none usable; App. B.1), and real application data is not public.

W2 / Q3: are the oracles reliable?

  • No human check before submission. We did not have the oracles checked by human annotators before submission.
  • Automatic verification (App. G). Every oracle reproduces the pre-drift result exactly on the instance's database and on 10 more databases with independently seeded rows. This holds for all 1,434 oracles of S2–S8, all 700 new instances, and all 2,000 S1 oracles.
  • Why rename oracles are reliable. A rename oracle substitutes identifiers, and the drifted database holds the same rows under the new names. Such a substitution reproduces the original result whenever it is scope-correct. When it is not, validation rejects it (Q2).
  • Stricter re-scoring of repairs. Under this criterion, 0.18% of the stored "equivalent" verdicts change (reply to Pd1o).

 

Q2: discarded instances and bias (§5.1, App. H).

  • S1 (PostgreSQL). 2,091 candidates → 2,000 accepted, so 91 were rejected (4.4%). Replaying the deterministic generator recovers all 91:
    • 83 rename a column whose name also exists in another table of the query (customer_id or unit_price);
    • the other 8 use templates whose result is not uniquely defined (ties under ORDER BY … LIMIT, or DISTINCT … LIMIT).
  • SQLite suites. Almost all rejections occur because SQLite reads unknown double-quoted identifiers as strings, so column-only renames in unqualified single-table queries still execute. The instances lost this way are the simplest ones.
  • One bias does exist. Our oracle renamed columns by name, so renaming one side of a shared join key could not be generated. This affected 13.5–28.4% of join candidates on SQLite.
    • Fix: we added these cases as a new 100-instance suite (S10) with a table-aware oracle.
    • Results: LLMs solve 93–100% of them, and the frontier models 100%. Name-based deterministic rewriting solves 0%. The exclusion therefore removed instances on which LLMs do well.
  • In the revision: rejection reasons per suite, and acceptance rates by template and category (Tables 33–34).

We have uploaded an updated version of the revised manuscript with changes highlighted in blue to make it easier to identify the modifications discussed in our responses. The scientific content and results remain unchanged from the previously uploaded revision.

Reviewer_Pd1oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DriftBench, a benchmark for repairing SQL queries that break after a schema change and a canonical migration plan is used for driving the drifted version of the database. A query is kept only when it runs before the drift but fails after it, and a repair counts as correct only if it reproduces the original pre-drift super set result. The benchmark covers three kinds of drift: renames, table splits, and column semantic transforms. It also has three levels of repair models that vary by how much information is provided to the repair model: both schemas (R1), only the drifted schema and the error message (R2, which is the default version), and finally R2 plus a description of the changes (R3). The main finding is that a repaired query that executes without error is often not correct, especially when the drift is semantic.

Critical strengths
  1. The problem of queries that silently return wrong output after a schema change is a common and costly failure in practice. A well-designed benchmark that targets this specific issue is genuinely useful, and according to the paper none currently exists that is designed for this exact issue.

  2. The experiments assess the effect of both the information available to the model and the difficulty and type of the drift itself. The benchmark and experiments convincingly show that query executability is not an adequate metric for this setting as in many cases the model produces a query that executes but returns output that differ from the pre-drift output.

Critical weaknesses
  1. The main correctness criterion (returning the same rows as the original query) can accept repairs that are actually wrong. For example this is a case found in the paper's own results: on the column-transform suite, every repair of the mapping based rewriter that is scored as correct is in the form of WHERE price cents > 50.0, which compares cents to a dollar threshold. It returns the same (non empty) set of products as the original one, only because in that test database every product happens to cost more than $50.

  2. The full schema of the benchmark dataset is never shown in the paper (only in the supplementary code), although it is small enough to fit in a short table and adding it could help readers better understand the results and examples.

  3. Previous schemas and migration history are usually recoverable from version control I believe, and with that history, renames are easy to resolve. Yet renames are the main type of drifts examined in the paper. Structural and semantic drifts are where LLMs can add real value, as the paper’s own results show, but they account for only a small portion of the benchmark.

  4. The results are hard to follow. They are reported mainly as dense numbers and model names, with little discussion of the underlying patterns, and they are also scattered.

  5. The scale of the benchmark does not reflect real world schemas. The synthetic schema has four tables, and the real-slice schemas have a median of eight. Production schemas are typically much larger, and repair could become harder at that scale. Consistent with this, the paper finds that its main difficulty point (the number of drift operations) barely affects strong models which suggests the current setup does not capture what makes the task hard in practice yet.

  6. The reported numbers are not yet stable enough. When the open models are re-run through a different serving endpoint in Appendix B.4, Phi-4 drops by 34.5 and 44.8 points on two of the suites. The benchmark contains only these open models, and is served locally with no description of the setup, and a well-performing model may not perform well on the real slice in the re-served run.

  7. A few relevant benchmarks are not discussed, notably BIRD-CRITIC (NeurIPS 2025) and Dr.Spider (ICLR 2023).

Questions for authors
  1. In which realistic deployments is the previous schema unavailable to the repair system, and how common are they compared with settings where migration history is recoverable? How do LLMs perform on the split and transform suites when given both schemas or the full change description?

  2. Which setup was used to serve the open models locally? Since Appendix B.4 shows that these models give very different results through another endpoint, could you rerun S1, the main suite, through the same endpoint and scoring protocol as in Section 7.6?

5 comments

We thank the reviewer for a detailed and constructive review. The S4 example is correct, and it led us to re-score every result.


W1: The criterion can accept wrong repairs.

You are right.

  • The S4 case. All seven "equivalent" verdicts of the mapping rewriter on S4 are wrong (price_cents > 50.0 or > 100.0).

    • Three match only because no product in that database is priced at or below the threshold.
    • Four differ from the original by one product. A 0.95 multiset-Jaccard tolerance in our comparator accepted them, and the paper should have stated that tolerance.
    • The rewriter's correct score is 0% (Table 3, Figure 2, App. G).
  • Bounding the problem everywhere. We re-scored all 43,841 stored repairs of S2–S8 under exact equality and under test-suite equivalence: the original result must be reproduced on the instance's database and on 10 more databases with independently seeded rows (App. G, Table 30).

    • Verdicts: 56 of 31,662 "equivalent" verdicts flip (0.18%).
    • Result cells: 240 of 268 are unchanged. Outside the scaling table, only the rewriter (6.7 → 0) and GPT-6-Luna on S3 (100 → 98; 100 → 97 in the feedback run) move by more than 1 pp.
    • Rankings: only two pairs change order: the two systems near zero on S4 (the rewriter and DeepSeek-R1-7B), and two models of the scaling series.
    • One statistic moves noticeably: the rank correlation between S3 and S8 falls from 0.62 to 0.52 (p = 0.089). Many models are tied near 100% on S3, so GPT-6-Luna's 2-point drop reorders them.
    • Oracles: all 1,434 oracles of S2–S8 pass on all 11 databases.
    • S1: on the paper's original runs (10,000 repairs), test-suite equivalence rejects none of the 6,361 "equivalent" verdicts. The local LLM results move by +0.2 to +1.3 pp (Table 32). Re-execution recovers 36 correct repairs that the original run had rejected because of PostgreSQL's tie-breaking (next point).
    • S1 has ties. Three S1 templates order by a count with ties (or select DISTINCT … LIMIT). PostgreSQL's order among tied rows changes when planner statistics appear mid-run. We therefore score S1 with automatic statistics disabled, as during suite validation. Under these conditions, every S1 oracle reproduces the original result on all 11 databases (App. G, Table 31).
  • In the revision:

    1. test-suite equivalence is the primary metric;
    2. the comparator is described exactly (§5.3);
    3. the S4 rewriter is reported as 0%.

W6 / Q2: serving setup, stability, and S1 through the common endpoint.

  • The local setup (App. K). The open models were served locally with Ollama (v0.30.11) using its 4-bit library builds: phi4:latest (14.7B, Q4_K_M), gemma3:27b (Q4_K_M), mistral:latest (Q4_0), and deepseek-r1:7b (Q4_K_M). Each request contained only the user prompt, with temperature 0, top-p 1, and at most 2,048 tokens.
  • The S1 re-run. As requested, we re-ran S1, the paper's 2,000 instances on PostgreSQL, through the same endpoint and protocol as §7.6. The prompts are byte-identical to the original run (same SHA-1 for 2,000/2,000).
  • Results (Equiv, all 2,000 instances; Table 2):
    • Current models: 88.8–94.9%. Claude-Haiku-4.5 94.9, GPT-6-Luna 94.7, GLM-5.3-Flash 94.5, DeepSeek-V4.1-Flash 94.0, Ministral-8B 93.3, Gemini-3.1-FL 90.8, gpt-oss-120b 88.8.
    • Paper's API models: GPT-4o-mini 90.5, Gemini-2.5-FL 90.0.
    • Execution: every LLM executes 98.8–100% of repairs, so the exec–equiv gap persists (5.2–11.2 pp for current models).
    • Re-served open models, paired with their local runs:
      • Gemma3-27B: 91.8 → 93.5 (per-instance agreement 96.7%);
      • Phi-4: 92.2 → 90.7 (agreement 95.9%).
  • What this shows.
    • Stability: on the main suite, re-serving moves the local results by less than 2 pp. The differences are small, but n = 2,000 makes them statistically detectable.
    • Where the shift was: the large Phi-4 shift of App. B.4 is confined to the held-out-vocabulary and hospital suites.
    • Its likely cause: the prompts there were also byte-identical. The shift is therefore consistent with a serving-stack effect (a 4-bit local build vs. the provider's weights) that matters most on unconventional names.
  • Unavailable models. Claude-3-Haiku, Mistral-7B, and DeepSeek-R1-7B are no longer offered by the provider, so they could not be re-served.
  • In the revision, Table 2 reports S1 for all eleven LLMs through the common endpoint, next to the local runs, which are labeled as such.

W5: schema scale (§7.7, App. I.1).

  • The experiment. We re-ran S8 showing each model the project's complete schema at that commit (median 31.5 tables, up to 183; median 273.5 columns) instead of the 8-table working schema. The queries, renames, and data stay identical. An alphabetical 8-table variant controls for table order.
  • Results.
    • All 11 models lose accuracy: mean Equiv 80.2% → 74.8% (−5.4 pp); execution 96.0% → 93.0%.
    • Where the loss comes from: 3.0 pp more repairs no longer execute, and 2.4 pp more execute but return wrong rows.
    • Significant losses (McNemar p < 0.01) for four models: Gemma3-27B −17.7, Phi-4 −8.7, Gemini-3.1-FL −7.7, and Gemini-2.5-FL −6.3 pp.
    • The strongest models on the full schema (DeepSeek-V4.1-Flash, GLM-5.3-Flash, GPT-6-Luna, gpt-oss-120b) lose 1.7–3.0 pp (Table 35).
    • One limit case: one full schema exceeds Phi-4's 16k-token context.
  • Conclusion: realistic schema size makes repair harder.

W3 / Q1: when is the previous schema unavailable, and how do models do with more information?

  • Where we agree. With the migration history, renames are easy, and the paper says so: R1 is exact for renames, and on real migrations the rename list lifts current models to 93–100%.
  • One nuance. A name-based AST rewrite given the exact mapping solves 0 of 100 new join-key renames. Rewriting must be scope-aware (§3, §7.7).
  • Where R2 applies (§3). R2 is the consumer's view: saved queries, dashboards, and notebooks owned by other teams; warehouse consumers after an upstream change; shared or vendor-managed databases. In these settings the query owner sees the current schema and the error, not the producer's migrations. We have no prevalence data and do not claim any, which is why we report all three regimes.
  • New results with more information (S3/S4, 11 models; §7.7, App. I.2):
    • Table split (S3): already 97.2% under R2; 98.0% under R1 and 99.5% under R3.
    • Column transform (S4): the semantic gap is largely about information. Mean Equiv rises from 49.9% (R2) to 81.6% given both schemas (R1) and to 93.3% given the change description (R3); 9 of 11 models reach ≥ 92% under R3.
    • Both schemas are not always enough. Gemini-2.5-FL stays at 15.4% under R1. The schema shows the new name and type, but not the unit change.
  • More structural and semantic drift. Coverage grows from 204 to 804 instances through six new families: lookup normalization, column split, timestamp → epoch, table merge, and two compound changes (rename + unit change, split + rename).
    • Pooled over 11 models (R2): 98.3% execute, but only 70.8% are equivalent.
    • Hardest families: timestamp → epoch 34.2%, table merge 56.3%, rename + unit change 68.3%, column split 72.5%.
    • Easier families: lookup normalization 98.3%, split + rename 95.2%.
    • With the change description (R3): 88.0%.
    • Deterministic baselines: 0–10%.

 

W2 / W4 / W7: presentation and related work.

The revision adds:

  • the full schemas of both domains (Table 9);
  • a findings-at-a-glance table (Table 1), with each finding pointing to its evidence. Some dense per-model detail moved to the appendix;
  • a comparison with Dr.Spider and BIRD-CRITIC (§2, Table 7):
    • Dr.Spider perturbs databases, questions, and SQL to test text-to-SQL robustness.
    • BIRD-CRITIC asks models to fix user-reported SQL issues on a fixed database, with task-specific tests.

Neither has a pre-/post-change pair of databases or the criterion of preserving the original query's result.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of new analyses, experiments, and revisions.

We thank the reviewers for their careful and constructive reviews. Three concerns are shared:

  1. Execution-based equivalence may accept wrong repairs;
  2. The main suite should be re-evaluated with current models under one serving stack;
  3. The benchmark should be harder and broader.

We have addressed these concerns in the revised manuscript as follows.


1. A stricter criterion, applied everywhere (§5.3, App. G).

   - What: a repair is now test-suite equivalent only if it reproduces the original result on the instance's database and on 10 more databases with independently seeded rows.

   - Re-scoring: we re-scored all 43,841 stored repairs of S2–S8. 56 of 31,662 "equivalent" verdicts (0.18%) change, and 240 of 268 result cells do not move. The largest change is the mapping rewriter on S4 (6.7% → 0%), the case Reviewer Pd1o identified.

   - S1: on the paper's original S1 runs (10,000 repairs), the criterion rejects none of the 6,361 "equivalent" verdicts.

   - Oracles: every oracle passes on all 11 databases.

Test-suite equivalence is now the primary metric throughout. The paper reports the submitted criterion alongside it as the single-database criterion (App. G, Tables 30 and 32).

 

2. S1 through one common endpoint (§7.1, Table 2).

We ran all 2,000 S1 instances with byte-identical prompts. Eleven LLMs reach 88.8–94.9%. The re-served Phi-4 and Gemma3-27B land within 1.8 pp of their local runs.

3. Harder and broader evaluation (new §7.7, Table 6, App. I).

   - Six new structural and semantic drift families (600 instances): 98.3% of repairs execute, but only 70.8% are correct.

   - Join-key renames (100 instances).

   - S8 with each project's full schema (up to 183 tables): accuracy drops for all 11 models.

   - Frontier models on the hardest suites: headroom remains, e.g., 33–47% on timestamp → epoch changes.

4. Transparency.

   - Validation and rejection statistics for every suite (App. H). This includes a limitation of our original oracle (join-key renames could not be generated), which the new suite S10 now covers.

   - An automatic audit of R2 underspecification (App. J).

   - The local serving setup (App. K).

The code, suites, and per-evaluation records are in the supplementary material.

AuthorsOpenReview ↗

Revised Manuscript with Highlighted Changes

Dear Reviewers and Area Chair,

We have uploaded an updated version of our revised manuscript, with changes highlighted in blue to make the modifications and additional experiments discussed in our responses easier to locate.

This update is purely visual. The scientific content, results, and supplementary material remain unchanged from our previously uploaded revision.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.