acceptodds
Desk rejected at ICLR 2027

SCHEMA2BENCH:EVIDENCE-GROUNDED CROSS-DOMAIN TEXT-TO-SQL BENCHMARK MIGRATION

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Schema2Bench formulates cross-domain Text-to-SQL benchmark migration as instrument construction under information, candidate-support and budget constraints. It combines evidence-grounded candidate synthesis with response-blind selection. A support certificate identifies structural deficits, and a sharp release-stability bound budgets changes to a fixed release. On a 2,806-candidate pool with a 2,000-item release budget, cost-aware bounded exchange reduces mean joint test-effect fidelity (TEF) loss by 5.67% versus quota random on a fixed seven-model panel across ten shared-pool seeds. Deficit-guided acquisition reaches the full-pool attainable difficulty discrepancy with 200 additions, versus 400 for source-proportion and 806 for uniform acquisition at the tested budgets. A stratified audit by two control-qualified model families jointly supports 270/300 fixed-release items (weighted estimate 90.07%). In a separate equal-budget construction study, the Schema2Bench deterministic refill variant yields 383 executable, jointly supported items versus 363 for a template baseline, including 109 versus 17 JOIN queries. In a later operator-matched, four-database BIRD comparison with nine evaluator models, the domain-mean pairwise-gap error is 0.2150 for the Schema2Bench variant versus 0.2912 for the template baseline: an improvement of 0.0762 (95% SQL-skeleton bootstrap interval [0.0300, 0.1148]). Three domains improve and one does not. These results connect support diagnosis and construction with measured evaluator differences for the tested deterministic variants; they do not establish general dominance.

Primary area:
datasets and benchmarks
Keywords:
Text-to-SQL, Benchmark Migration, Data Selection, Evaluation Fidelity, Response-Blind Design, Support Certificate, Test-Effect Fidelity, Data-Centric ML

Related papers

Discussion (0)

Sign in to comment.