- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Databox, a task-specific synthetic database environment for execution-guided SQL debugging. It combines schema extraction, diagnostic data generation, and consistency validation to replace interaction with the original database. Databox-Local trains small models for local environment construction. Experiments across multiple agent scaffolds and backbone models report substantially lower optimizer-estimated execution costs with broadly comparable repair success. Additional evaluations examine repository-level debugging, alternative data generators, and text-to-SQL transfer.
Critical strengths
The paper addresses a practical limitation of execution-guided SQL agents: repeated interaction with large or sensitive databases. The proposed environment is compatible with several existing agent scaffolds, and the evaluation spans multiple backbones and repository-level workflows. Importantly, Table 4 compares diagnostic generation with sampling and general-purpose generators, providing evidence beyond simply showing that smaller databases are cheaper. The analysis of unauthorized database modifications also highlights a useful operational benefit of isolating exploratory execution.
Critical weaknesses
-
The main efficiency results do not establish the end-to-end trade-off. The headline reduction concerns optimizer-estimated SQL execution cost. However, Databox requires schema extraction, data generation, validation, and potentially repeated reconstruction before debugging begins. The main text does not provide a consolidated comparison including these costs. Actual SQL runtime reductions are encouraging, but may not imply lower overall latency or monetary cost when LLM calls dominate. Table 4 also compares repair success without reporting the corresponding costs, leaving unclear whether Databox offers a better quality–cost trade-off than simpler compact environments.
-
The diagnostic fidelity and performance-preservation claims need stronger qualification. Reproducing the original issue is necessary, but does not ensure that the sandbox distinguishes a correct repair from other incorrect candidates. Section 3.3 does not explain sufficiently how semantic issue fidelity is determined without original records or a reference repair. Aggregate success rates provide useful indirect evidence, but do not characterize misleading feedback or omitted data conditions. Performance preservation is also uneven: all three Qwen repository-level settings lose approximately 2.6–3.6 percentage points, and Databox-Local loses 5.1 points with Spider2Agent. Without uncertainty estimates and paired failure analysis, the reliability and boundaries of substitution remain unclear.
Questions for authors
-
What are the total construction-plus-debugging latency, token usage, and monetary costs, including failed construction attempts? How does Databox compare with the simpler population strategies in Table 4 under these measurements?
-
How is semantic issue fidelity checked when the reference repair and original records are unavailable? Could you measure how often the sandbox distinguishes the reference repair from multiple plausible incorrect repairs, rather than only reproducing the original defect?
-
Are the success-rate changes robust across repeated runs? Could you provide paired analysis of tasks gained and lost, particularly for the Qwen repository-level results and the Databox-Local Spider2Agent decrease? Please also clarify how construction failures enter the main success-rate denominator.
-
How are guideline exploration, guideline ranking, local-constructor training, and final evaluation separated? Since reference SQL informs offline guideline construction, explicit task and database separation would help establish generalization.
3 comments
W2 / Q2: Issue Fidelity and Repair Discrimination
We thank the reviewer for pointing out the need to clarify issue-fidelity validation and evaluate whether Databox can distinguish correct repairs from incorrect candidates.
1. Issue-Fidelity Validation Process
At deployment, the constructor does not have access to the original database records or a reference repair. Our issue-fidelity check therefore assesses whether executing the reported SQL query on the constructed Databox reproduces the behavior in the user issue description.
The check comprises two components. First, when the issue description includes explicit error-message keywords, these keywords are extracted and compared with the execution result of the reported SQL query on the Databox. Second, for issues without explicit error messages, an LLM checker assesses whether the observed behavior is consistent with the reported issue. The checker receives only the Databox schema, Databox data, the user issue description, the reported SQL query, and its execution result and compares the execution results with the behavior described in the user issue description.
To further explain the constructed Databox quality, we apply a manual audit on constructed Databoxes.
2. Manual Audit of Constructed Databoxes
Two SQL engineers manually examined 227 constructed Databoxes to evaluate their reproduction fidelity.
| Reproduction outcome | Instances | Percentage |
|---|---|---|
| Reported issue fully reproduced | 188 | 82.8% |
| Reported issue not reproduced | 27 | 11.9% |
| Different Issue | 9 | 4.0% |
| Partially Reproduced Issue | 3 | 1.3% |
| Total | 227 | 100.0% |
Overall, Databox successfully reproduces the reported issue in 82.8% of the audited instances, supporting its reliability in issue reproduction.
We further analyzed the 39 instances in which the reported issue was not fully reproduced.
| Contributing factor | Instances | Percentage of the 39 instances |
|---|---|---|
| Missing necessary data conditions | 19 | 48.7% |
| Missing or mismatched schema elements | 16 | 41.0% |
| Unclear or incorrect explanations in the issue description | 11 | 28.2% |
| Additional blocking conditions involving objects or constraints introduced during construction | 4 | 10.3% |
The most frequently identified factors were missing data conditions and missing or mismatched schema elements. Ambiguous or incorrect issue descriptions also adversely affect the quality of the constructed Databoxes.
3. Distinguishing Correct and Incorrect Repairs
In response to the reviewer's question, we additionally evaluated Databox's ability to distinguish correct repairs from plausible incorrect candidates generated by debugging agents. We manually evaluated 1732 candidate repairs produced by three models and examined whether Databox exposed a problem in each candidate repair.
| Model | Correct repair: no problem exposed | Incorrect repair: problem missed | Correct repair: mistakenly flagged | Incorrect repair: problem exposed | Candidate repairs |
|---|---|---|---|---|---|
| Gemini | 14.13% | 14.98% | 1.46% | 69.43% | 821 |
| Qwen | 35.25% | 10.85% | 4.23% | 49.66% | 590 |
| Sonnet | 32.09% | 9.96% | 3.74% | 54.20% | 321 |
Across all three agents, Databox identified 82.6% of incorrect repairs and correctly distinguished correct from incorrect repairs in 84.5% of evaluated cases.
W2 / Q3: Robustness and Paired Analyze
Repeated Runs. We thank the reviewer for raising the question of robustness. The main BIRD-CRITIC-PG experiment was repeated three times, and the reported results are averages. The small success-rate improvement with databoxes persisted across these runs. We have added this protocol to the revised appendix. The repository-level and Databox-Local experiments were each conducted once.
Construction Failures. All construction failures remain in the denominator of the reported downstream success rate. If execution validation remains unsuccessful, we provide an empty PostgreSQL sandbox to the debugging stage. If execution validation succeeds but issue-fidelity validation remains unsuccessful, we initialize the databox using the schema and data from the final construction iteration.
Paired Analysis: Databox-Local with Spider2Agent
The paired comparison identifies 7 gained tasks and 34 lost tasks, yielding a net loss of 27 tasks. Here the gained tasks mean Databox-Local success while LLM constructor failed. Manual inspection found no clear databox-quality diffrence for 3 gains and 6 losses. The remaining cases are summarized below.
| Outcome | Tasks | Explanation |
|---|---|---|
| Gained | 3 | Local test data cover diagnostic conditions missed by the LLM constructor. |
| Gained | 1 | The LLM constructor confuses similarly named columns. |
| Lost | 15 | Foreign-key violations prevent data population. |
| Lost | 7 | Schema extraction omits required columns. |
| Lost | 6 | Schema-extraction failures leave empty environments. |
Paired Analysis: Qwen Repository-Level
| Tool | Both succeed | Gained | Lost | Both fail | Net change |
|---|---|---|---|---|---|
| OpenCode | 170 | 48 | 66 | 246 | −3.40 pp |
| QwenCode | 176 | 49 | 63 | 242 | −2.64 pp |
| KimiCode | 211 | 24 | 43 | 252 | −3.58 pp |
Manual inspection identified the following explanations for gains and losses. Cases without a clear databox-related explanation are omitted. The gained tasks refer to Databox passed while realDB not passed.
| Outcome | Explanation | Percentage |
|---|---|---|
| Gained | Test data cover informative boundary conditions. | 68.8% |
| Gained | Test data expose a tie-breaking counterexample. | 18.8% |
| Lost | Column-name casing or identifier mismatches. | 31.8% |
| Lost | Missing diagnostic data conditions. | 30.3% |
| Lost | Differences in output formatting. | 10.6% |
| Lost | Incorrect database connection attempts. | 6.1% |
Q4: Separation with Evaluation
Thank you for requesting clarification. We distinguish guideline exploration, guideline ranking, local-constructor training, and final evaluation as follows.
| Stage | Data source | Use of reference SQL |
|---|---|---|
| Guideline exploration | 300 SWE-GYM instances, the designated training set provided by BIRD-CRITIC | The LLM examines problematic and reference SQL to derive diagnostic guidelines. |
| Guideline ranking | Remaining SWE-GYM instances, excluding the exploration seeds | Reference SQL is executed and used to score and rank guidelines. |
| Databox-Local constructor training | BIRD-SQL development set | Correct SQL is used to synthesize issues through QEP-based injection to populates issue instances for training. |
| Final evaluation | BIRD-CRITIC-PG evaluation set | Evaluation reference SQL is not used for guideline construction, local-constructor training, or deployment-time environment construction. |
For guideline library building, both stages use instances from the formal training set provided by BIRD-CRITIC, which do not overlap with BIRD-CRITIC-PG evaluation. Separately, Databox-Local is trained on synthetic issues derived from BIRD-SQL development text–SQL pairs, using databases and text-to-SQL pairs distinct from those in BIRD-CRITIC-PG. More details can be found in Appendix A and C.
We thank the reviewer for highlighting the need to distinguish database execution efficiency from end-to-end workflow efficiency.
W1/Q1: Construction-plus-debgging latency
We report costs along two dimensions: database-side costs and LLM-side costs, and the reported construction token counts include failed construction attempts, with detailed token usage provided in Tables 8 and 9. Our goal is to reduce the computational burden on the database side without substantially increasing LLM-side costs. This involves a trade-off: Databox incurs additional LLM costs during construction to enable more efficient database execution during debugging. And we want to further claim that Databox has better end-to-end performance in tasks involving larger database size.
1. Construction and debugging latency on BIRD-CRITIC-PG.
We report the following LLM side workflow latency, averaged across all nine settings:
The end-to-end efficiency gains on BIRD-CRITIC-PG are largely diluted by LLM latency. Nevertheless, this does not imply that Databox cannot reduce end-to-end latency. Instead, it motivates a closer examination of the conditions under which Databox provides greater benefits, which we will clarify later.
2. Comparison with simpler population strategies.
We additionally report repair success rates and LLM costs for the population strategies evaluated in Table 4.
Although the simpler strategies incur lower LLM-side costs, their success rates are lower than that of Databox without preview and therefore we don't apply these population strategy in our final Databox construction.
3. Databox Performance with DB Size Scaling
We further clarify why the additional LLM overhead from Databox can be justified when database size growing. Our experiments on BIRD-CRITIC-PG use a simplified benchmark setting with relatively small databases, where LLM calls account for a substantial proportion of the overall cost. For real-world tasks involving larger databases, this additional LLM overhead may become less significant relative to the cost of database execution.
To show this trends, we further evaluate Databox and RealDB with the SQL ACT agent on Spider2-BigQuery subset (150 instances).
As database size increases, SQL execution time and the number of required agent actions may also increase, potentially offsetting the additional LLM overhead introduced by Databox. Additionally, the increasing LLM token usage are substantially smaller than those observed for RealDB in the larger-schema groups. The relative reduction in reported total latency consequently increases from 11.3% to 80.4%.
Databox’s efficiency benefits depend on the workload. On BIRD-CRITIC-PG, the involved tasks contain only small size databases, and therefore construction overhead largely offsets the debugging-time savings. On the evaluated Spider2-BigQuery workloads, which is more related to real-world large scale databases, Databox exhibits substantially lower reported latency. This scaling discussion is added in our revised paper.
Thank you very much for pointing out this insightful suggestions!