acceptodds
Under review as a conference paper at ICLR 2027

SCHEMA2BENCH:EVIDENCE-GROUNDED CROSS-DOMAIN TEXT-TO-SQL BENCHMARK MIGRATION

Abstract

Schema2Bench formulates cross-domain Text-to-SQL benchmark migration as instrument construction under information, candidate-support and budget constraints. It combines evidence-grounded candidate synthesis with response-blind selection. A support certificate identifies structural deficits, and a sharp release-stability bound budgets changes to a fixed release. On a 2,806-candidate pool with a 2,000-item release budget, cost-aware bounded exchange reduces mean joint test-effect fidelity (TEF) loss by 5.67% versus quota random on a fixed seven-model panel across ten shared-pool seeds. Deficit-guided acquisition reaches the full-pool attainable difficulty discrepancy with 200 additions, versus 400 for source-proportion and 806 for uniform acquisition at the tested budgets. A stratified audit by two control-qualified model families jointly supports 270/300 fixed-release items (weighted estimate 90.07%). In a separate equal-budget construction study, the Schema2Bench deterministic refill variant yields 383 executable, jointly supported items versus 363 for a template baseline, including 109 versus 17 JOIN queries. In a later operator-matched, four-database BIRD comparison with nine evaluator models, the domain-mean pairwise-gap error is 0.2150 for the Schema2Bench variant versus 0.2912 for the template baseline: an improvement of 0.0762 (95% SQL-skeleton bootstrap interval [0.0300, 0.1148]). Three domains improve and one does not. These results connect support diagnosis and construction with measured evaluator differences for the tested deterministic variants; they do not establish general dominance.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.