- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduced DriftBench, a benchmark + eval harness for repairing SQL queries following schema drift. The task input consists of an original (failing) query and its error message, and correctness is defined by equivalent records returned by the fixed query (candidate) on the migrated DB vs. the original query on the original DB. Several experiments are run in full history, current schema only, and metadata assisted settings. The migrated databases are constructed with rename, table split, and unit change operations, with held out real dataset mined from genuine project histories.
Critical strengths
The benchmark seems to address a real world concern with convincing execution-based construction. The benchmark enforces drift impact by ensuring that queries succeed originally, fail post-migration, and the oracle query result matches as expected. Moreover the "true" distribution of migrations/schema drift is somewhat represented from mined histories.
Critical weaknesses
Underspecification in R2. Some inputs (query + error message) may not be resolvable with this information alone. Therefore it's unclear whether equivalence failure in this setting is a model shortcoming or a task specification issue. Can this ambiguity be automatically audited.
Some of the current models seem to leave limited headroom on this benchmark with glm-5.3-flash or deepseek-v4.1-flash approaching scores in the 90-95 range. This potentially remains useful as a diagnostic, the discriminative ability of this benchmark for future systems is less clear.
Questions for authors
- are there cases where case and whitespace normalization affect SQL results?
- are there preliminary results of SOTA models on a small subset of hardest tasks?
4 comments
We have uploaded an updated version of the revised manuscript with changes highlighted in blue to make it easier to identify the modifications discussed in our responses. The scientific content and results remain unchanged from the previously uploaded revision.
Thanks for the responses, the new analysis and harder subset seem to adequately address my main identified weaknesses, so increasing score by 1.
Dear Reviewer,
Thank you for taking the time to reconsider our work and for your positive reassessment. We sincerely appreciate your constructive feedback, which helped us strengthen the paper through additional analyses and experiments. We are pleased that these additions addressed your main concerns.
Kind regards,
The Authors
We thank the reviewer for the positive assessment and the precise questions.
W1: is R2 underspecified, and can this be audited? R2 also includes the drifted schema (columns and types). We audited S8 automatically, using its 12 models as a panel together with the paired R3 runs (App. J).
invoices.closed → auto_advanceandusers.channel → pubsub_token.W2 / Q2: headroom, and SOTA models on the hardest tasks (§7.7, App. I.4).
Hard subset (139 instances solved by at most 4 of 7 current models): Claude-Opus-5.5 91.4%, GPT-6-Sol 60.4%, Gemini-3.1-Pro (preview) 38.8%.
Complete suites (to avoid selection effects):
Headroom remains even for frontier models, and the suites separate them.
Q1: does case or whitespace normalization affect results? We re-scored all 43,841 stored repairs with a comparator that keeps case and whitespace. No verdict changes (0 of 31,662; App. G).