acceptodds
Under review as a conference paper at ICLR 2027

Databox: Elevating SQL Debugging Agents with Instance-specific and Cost-Efficient Sandboxes

Abstract

The explosion of enterprise data and database-intensive applications has increased the complexity of SQL-based analytical and operational logic, raising the demand for automated query diagnosis and repair. Recent progress on SQL-based tasks has been driven by LLM-based agentic methods, which rely on execution-guided interaction loops with real database environments to iteratively inspect, reflect on, and refine candidate solutions. However, these agentic approaches typically require direct interaction with real databases, incurring high execution cost and latency on enterprise-scale applications and becoming impractical when privacy or deployment constraints restrict access to the underlying real data. To address these limitations, we propose Databox, an automatically constructed, instance-specific, data-specialized sandbox environment that serves as a drop-in replacement for real-database interaction. Databox is constructed with task-specific schema, diagnostic value, and consistency validation, enabling private, low-latency, and compute-efficient execution-guided SQL debugging without accessing the large original database. To further reduce reliance on large models and strengthen privacy guarantees, we introduce Databox-Local, which trains small language models for Databox construction and enables fully local deployment without external LLM calls. Experiments on BIRD-CRITIC-PG, spanning five widely used agent scaffolds and three backbone LLMs, show that Databox reduces optimizer-estimated execution cost by 65%-92%, with an average reduction of 85.2% relative to real-database interaction, while maintaining comparable task performance and slightly increasing the average success rate by 1.4%. To assess its practicality and generalizability, we further evaluate Databox on the repository-level benchmark BIRD-CRITIC-REPO, demonstrating the potential and adaptability to broader software engineering settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, Large Language Models, SQL Debugging, Database Sandboxes, Code Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oz22OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Databox, a task-specific synthetic database environment for execution-guided SQL debugging. It combines schema extraction, diagnostic data generation, and consistency validation to replace interaction with the original database. Databox-Local trains small models for local environment construction. Experiments across multiple agent scaffolds and backbone models report substantially lower optimizer-estimated execution costs with broadly comparable repair success. Additional evaluations examine repository-level debugging, alternative data generators, and text-to-SQL transfer.

Critical strengths

The paper addresses a practical limitation of execution-guided SQL agents: repeated interaction with large or sensitive databases. The proposed environment is compatible with several existing agent scaffolds, and the evaluation spans multiple backbones and repository-level workflows. Importantly, Table 4 compares diagnostic generation with sampling and general-purpose generators, providing evidence beyond simply showing that smaller databases are cheaper. The analysis of unauthorized database modifications also highlights a useful operational benefit of isolating exploratory execution.

Critical weaknesses
  1. The main efficiency results do not establish the end-to-end trade-off. The headline reduction concerns optimizer-estimated SQL execution cost. However, Databox requires schema extraction, data generation, validation, and potentially repeated reconstruction before debugging begins. The main text does not provide a consolidated comparison including these costs. Actual SQL runtime reductions are encouraging, but may not imply lower overall latency or monetary cost when LLM calls dominate. Table 4 also compares repair success without reporting the corresponding costs, leaving unclear whether Databox offers a better quality–cost trade-off than simpler compact environments.

  2. The diagnostic fidelity and performance-preservation claims need stronger qualification. Reproducing the original issue is necessary, but does not ensure that the sandbox distinguishes a correct repair from other incorrect candidates. Section 3.3 does not explain sufficiently how semantic issue fidelity is determined without original records or a reference repair. Aggregate success rates provide useful indirect evidence, but do not characterize misleading feedback or omitted data conditions. Performance preservation is also uneven: all three Qwen repository-level settings lose approximately 2.6–3.6 percentage points, and Databox-Local loses 5.1 points with Spider2Agent. Without uncertainty estimates and paired failure analysis, the reliability and boundaries of substitution remain unclear.

Questions for authors
  1. What are the total construction-plus-debugging latency, token usage, and monetary costs, including failed construction attempts? How does Databox compare with the simpler population strategies in Table 4 under these measurements?

  2. How is semantic issue fidelity checked when the reference repair and original records are unavailable? Could you measure how often the sandbox distinguishes the reference repair from multiple plausible incorrect repairs, rather than only reproducing the original defect?

  3. Are the success-rate changes robust across repeated runs? Could you provide paired analysis of tasks gained and lost, particularly for the Qwen repository-level results and the Databox-Local Spider2Agent decrease? Please also clarify how construction failures enter the main success-rate denominator.

  4. How are guideline exploration, guideline ranking, local-constructor training, and final evaluation separated? Since reference SQL informs offline guideline construction, explicit task and database separation would help establish generalization.

3 comments

We thank the reviewer for highlighting the need to distinguish database execution efficiency from end-to-end workflow efficiency.

W1/Q1: Construction-plus-debgging latency

We report costs along two dimensions: database-side costs and LLM-side costs, and the reported construction token counts include failed construction attempts, with detailed token usage provided in Tables 8 and 9. Our goal is to reduce the computational burden on the database side without substantially increasing LLM-side costs. This involves a trade-off: Databox incurs additional LLM costs during construction to enable more efficient database execution during debugging. And we want to further claim that Databox has better end-to-end performance in tasks involving larger database size.

1. Construction and debugging latency on BIRD-CRITIC-PG.

We report the following LLM side workflow latency, averaged across all nine settings:

EnvironmentConstruction: LLM latency (s)Construction: SQL execution (s)Debugging: LLM latency (s)Debugging: SQL execution (s)Total (s)
Databox7.250.04611.050.23518.581
RealDB——17.181.43618.616

The end-to-end efficiency gains on BIRD-CRITIC-PG are largely diluted by LLM latency. Nevertheless, this does not imply that Databox cannot reduce end-to-end latency. Instead, it motivates a closer examination of the conditions under which Databox provides greater benefits, which we will clarify later.

2. Comparison with simpler population strategies.

We additionally report repair success rates and LLM costs for the population strategies evaluated in Table 4.

Data generation strategySR (%)Construction tokensConstruction LLM latency (s)Debugging tokensDebugging LLM latency (s)Total LLM latency (s)
Databox without preview37.623,3232.3315,5929.3011.63
Preview-row copy35.6400.0012,3977.447.44
Bernoulli + FK closure35.4500.0013,1298.318.31
Databox with preview34.068,3655.1814,5208.8614.04

Although the simpler strategies incur lower LLM-side costs, their success rates are lower than that of Databox without preview and therefore we don't apply these population strategy in our final Databox construction.

3. Databox Performance with DB Size Scaling

We further clarify why the additional LLM overhead from Databox can be justified when database size growing. Our experiments on BIRD-CRITIC-PG use a simplified benchmark setting with relatively small databases, where LLM calls account for a substantial proportion of the overall cost. For real-world tasks involving larger databases, this additional LLM overhead may become less significant relative to the cost of database execution.

To show this trends, we further evaluate Databox and RealDB with the SQL ACT agent on Spider2-BigQuery subset (150 instances).

TablesQuestionsQuery time (s), Databox / RealDBDebugging turns, Databox / RealDBLLM tokens, Databox / RealDB
1–9290.34 / 2.222.34 / 2.9022,778 / 19,595
10–99790.62 / 2.672.81 / 3.3235,619 / 56,788
100–785420.55 / 5.643.15 / 3.8588,878 / 497,089
TablesLLM latency (s), Databox / RealDBReported total latency (s), Databox / RealDBTotal latency reduction
1–913.80 / 10.0314.600 / 16.46811.3%
10–9921.69 / 32.7223.430 / 41.58043.7%
100–78550.40 / 243.8252.130 / 265.53480.4%

As database size increases, SQL execution time and the number of required agent actions may also increase, potentially offsetting the additional LLM overhead introduced by Databox. Additionally, the increasing LLM token usage are substantially smaller than those observed for RealDB in the larger-schema groups. The relative reduction in reported total latency consequently increases from 11.3% to 80.4%.

Databox’s efficiency benefits depend on the workload. On BIRD-CRITIC-PG, the involved tasks contain only small size databases, and therefore construction overhead largely offsets the debugging-time savings. On the evaluated Spider2-BigQuery workloads, which is more related to real-world large scale databases, Databox exhibits substantially lower reported latency. This scaling discussion is added in our revised paper.

Thank you very much for pointing out this insightful suggestions!

W2 / Q2: Issue Fidelity and Repair Discrimination

We thank the reviewer for pointing out the need to clarify issue-fidelity validation and evaluate whether Databox can distinguish correct repairs from incorrect candidates.

1. Issue-Fidelity Validation Process

At deployment, the constructor does not have access to the original database records or a reference repair. Our issue-fidelity check therefore assesses whether executing the reported SQL query on the constructed Databox reproduces the behavior in the user issue description.

The check comprises two components. First, when the issue description includes explicit error-message keywords, these keywords are extracted and compared with the execution result of the reported SQL query on the Databox. Second, for issues without explicit error messages, an LLM checker assesses whether the observed behavior is consistent with the reported issue. The checker receives only the Databox schema, Databox data, the user issue description, the reported SQL query, and its execution result and compares the execution results with the behavior described in the user issue description.

To further explain the constructed Databox quality, we apply a manual audit on constructed Databoxes.

2. Manual Audit of Constructed Databoxes

Two SQL engineers manually examined 227 constructed Databoxes to evaluate their reproduction fidelity.

Reproduction outcomeInstancesPercentage
Reported issue fully reproduced18882.8%
Reported issue not reproduced2711.9%
Different Issue94.0%
Partially Reproduced Issue31.3%
Total227100.0%

Overall, Databox successfully reproduces the reported issue in 82.8% of the audited instances, supporting its reliability in issue reproduction.

We further analyzed the 39 instances in which the reported issue was not fully reproduced.

Contributing factorInstancesPercentage of the 39 instances
Missing necessary data conditions1948.7%
Missing or mismatched schema elements1641.0%
Unclear or incorrect explanations in the issue description1128.2%
Additional blocking conditions involving objects or constraints introduced during construction410.3%

The most frequently identified factors were missing data conditions and missing or mismatched schema elements. Ambiguous or incorrect issue descriptions also adversely affect the quality of the constructed Databoxes.

3. Distinguishing Correct and Incorrect Repairs

In response to the reviewer's question, we additionally evaluated Databox's ability to distinguish correct repairs from plausible incorrect candidates generated by debugging agents. We manually evaluated 1732 candidate repairs produced by three models and examined whether Databox exposed a problem in each candidate repair.

ModelCorrect repair: no problem exposedIncorrect repair: problem missedCorrect repair: mistakenly flaggedIncorrect repair: problem exposedCandidate repairs
Gemini14.13%14.98%1.46%69.43%821
Qwen35.25%10.85%4.23%49.66%590
Sonnet32.09%9.96%3.74%54.20%321

Across all three agents, Databox identified 82.6% of incorrect repairs and correctly distinguished correct from incorrect repairs in 84.5% of evaluated cases.

W2 / Q3: Robustness and Paired Analyze

Repeated Runs. We thank the reviewer for raising the question of robustness. The main BIRD-CRITIC-PG experiment was repeated three times, and the reported results are averages. The small success-rate improvement with databoxes persisted across these runs. We have added this protocol to the revised appendix. The repository-level and Databox-Local experiments were each conducted once.

Construction Failures. All construction failures remain in the denominator of the reported downstream success rate. If execution validation remains unsuccessful, we provide an empty PostgreSQL sandbox to the debugging stage. If execution validation succeeds but issue-fidelity validation remains unsuccessful, we initialize the databox using the schema and data from the final construction iteration.

Paired Analysis: Databox-Local with Spider2Agent

The paired comparison identifies 7 gained tasks and 34 lost tasks, yielding a net loss of 27 tasks. Here the gained tasks mean Databox-Local success while LLM constructor failed. Manual inspection found no clear databox-quality diffrence for 3 gains and 6 losses. The remaining cases are summarized below.

OutcomeTasksExplanation
Gained3Local test data cover diagnostic conditions missed by the LLM constructor.
Gained1The LLM constructor confuses similarly named columns.
Lost15Foreign-key violations prevent data population.
Lost7Schema extraction omits required columns.
Lost6Schema-extraction failures leave empty environments.

Paired Analysis: Qwen Repository-Level

ToolBoth succeedGainedLostBoth failNet change
OpenCode1704866246−3.40 pp
QwenCode1764963242−2.64 pp
KimiCode2112443252−3.58 pp

Manual inspection identified the following explanations for gains and losses. Cases without a clear databox-related explanation are omitted. The gained tasks refer to Databox passed while realDB not passed.

OutcomeExplanationPercentage
GainedTest data cover informative boundary conditions.68.8%
GainedTest data expose a tie-breaking counterexample.18.8%
LostColumn-name casing or identifier mismatches.31.8%
LostMissing diagnostic data conditions.30.3%
LostDifferences in output formatting.10.6%
LostIncorrect database connection attempts.6.1%

Q4: Separation with Evaluation

Thank you for requesting clarification. We distinguish guideline exploration, guideline ranking, local-constructor training, and final evaluation as follows.

StageData sourceUse of reference SQL
Guideline exploration300 SWE-GYM instances, the designated training set provided by BIRD-CRITICThe LLM examines problematic and reference SQL to derive diagnostic guidelines.
Guideline rankingRemaining SWE-GYM instances, excluding the exploration seedsReference SQL is executed and used to score and rank guidelines.
Databox-Local constructor trainingBIRD-SQL development setCorrect SQL is used to synthesize issues through QEP-based injection to populates issue instances for training.
Final evaluationBIRD-CRITIC-PG evaluation setEvaluation reference SQL is not used for guideline construction, local-constructor training, or deployment-time environment construction.

For guideline library building, both stages use instances from the formal training set provided by BIRD-CRITIC, which do not overlap with BIRD-CRITIC-PG evaluation. Separately, Databox-Local is trained on synthetic issues derived from BIRD-SQL development text–SQL pairs, using databases and text-to-SQL pairs distinct from those in BIRD-CRITIC-PG. More details can be found in Appendix A and C.

Reviewer_xaJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes Databox a framework to build sandbox databases around specific SQL issues to tackle database privacy concern as well as to save real database access cost.

Critical strengths
  1. The problem of privacy and database access cost is well motivated and practical
  2. The implementation shows consistent savings in database execution cost with relatively close performance to the setting where agents have real database access.
Critical weaknesses
  1. The current comparison between Real-DB access and Databox is not entirely clear. Since the Real-DB setting requires the agent to deal with the whole unfiltered database schema, it is unclear whether the gain comes from better schema linking (section 3.1) or from test value generation and consistency validation (section 3.2 and 3.3)

  2. The efficiency shown is limited to database operations saving, while the cost incurred to LLM token usage is large (by 17.1%–198.8% relative to Real DB, line 1030). Hence, the authors need to be clearer in their discussion when saying that the method "avoid LLM API costs" (line 482 in conclusion) or generically saying "compute-efficient execution-guided SQL debugging" in line 23 and 24 of abstract.

Questions for authors
  1. Can the author address my 1st weakness concern via running a more principled ablation study?

  2. It is unclear how the method will generalize to larger databases in "enterprise-scale applications" since so far the method has only been tested and built (guidelines extraction, Databox-Local training) around BIRD style databases where the number of tables and columns as well as the size of databases themself are relatively small compared to recent benchmark such as Spider2 (Lei et al., 2024) on Snowflake. Can the authors add an out of domain generation test of Databox to a small subset (100-150 questions) of Spider2-snow text-to-sql setting, with a more recent open source state-of-the-art agent system, just similar to how the method were tested with the smaller version Spider2-Lite SQLite?

  3. Since the tested models are relatively old and several breakthroughs have been made recently in coding agents. Can the author demonstrate Databox generalization to the case of (a) more recent coding agent system(s) (both with (a) newer model(s) and (a) newer harness(es))? The point is to show that Databox can generalize to more frontier system with higher initial debugging capabilities given the original database

3 comments

W1 / Q1

We thank the reviewer to provide such valuable feedback to distinguish the contribution of schema linking from the Databox environment. Follow your suggestion, we add one more ablation study to exclude the schema linking information difference.

1. Ablation on Schema Linking Information

We conducted an additional experiment using Gemini-2.5-Flash on the first 300 BIRD-CRITIC-PG instances. Specifically, we augmented the Real-DB setting with the schema selected by Databox as a hint, while retaining access to the original database. This comparison tests whether providing Databox's meta data alone reproduces its observed gains.

Agent harnessReal-DBReal-DB + Databox schema hintDatabox
SQL-ACT41.0%40.3%42.3%
Spider2Agent38.6%38.6%39.3%

Providing the schema hint does not improve the observed success rate for either agent harness. Databox outperforms the schema-hinted Real-DB setting by 2.0 percentage points for SQL-ACT and 0.7 percentage points for Spider2Agent. These results suggest that providing schema-linking information alone is insufficient to reproduce Databox's minor gain.

2. Action Distribution Analysis for Gain Reason

The action distributions reported in Appendix J provide additional insight into how agents interact with the two environments:

Action typeDataboxReal-DB
Correction81.3%58.1%
Inspection0.9%16.2%
Partial correction3.4%11.1%

(Remaining action types with smaller percentage omitted.)

With Databox, a larger proportion of agent actions consists of direct corrections, while smaller proportions involve inspection and partial correction. This pattern suggests that Databox's compact debugging environment and targeted feedback facilitate more focused correction trajectories and avoid distraction or resource waste in inspection or probing.

W2

We thank the reviewer for pointing out this writing ambiguity.

The phrase “avoid LLM API costs” in conclusion section refers specifically to construction with local SLMs, rather than to the entire debugging workflow. We will correct these ambiguity words in the revised paper.

We report costs and efficiency along two dimensions: database-side costs and LLM-side costs. Our goal is to reduce the computational burden on the database side while not significantly increase the LLM side cost, especially on large DB size. This involves a trade-off: Databox incurs additional LLM costs during construction to enable more efficient database execution during debugging.

1. Sources of Databox Token Overhead and Savings

Databox introduces additional construction tokens and input tokens during debugging. However, it also reduces token usage in the following ways:

  • Shorter execution feedback and fewer debugging actions. Databox reduces mean execution-feedback length by 5.9%–69.3% across all nine settings (Table 16) and reduces action counts in seven of the nine settings (Table 15).

  • Fewer output tokens in debugging. We compare the different model's average output token in our main experiments:

ModelRealDBDatabox
Gemini782648
Qwen223193
Claude264202

These savings do not necessarily offset the construction overhead and additional input tokens introduced by including Databox construction metadata in the debugging prompt. We will make this distinction explicit in the revised cost discussion.

2. Databox Brings Efficiency Improvements with Larger DBs

To examine how this trade-off varies across workloads, we additionally evaluate Databox and RealDB using the SQL ACT agent on 150 instances from Spider2-BigQuery. We group the instances by the number of tables in their database schemas. All paired values below are reported as Databox / RealDB.

Tables in schemaInstancesQuery time (s)Debugging turnsLLM tokens
1–9290.34 / 2.222.34 / 2.9022,778 / 19,595
10–99790.62 / 2.672.81 / 3.3235,619 / 56,788
100–785420.55 / 5.643.15 / 3.8588,878 / 497,089
Tables in schemaLLM latency (s)Reported total latency (s)Total latency reduction
1–913.80 / 10.0314.600 / 16.46811.3%
10–9921.69 / 32.7223.430 / 41.58043.7%
100–78550.40 / 243.8252.130 / 265.53480.4%

In the smallest-database group, Databox uses more LLM tokens than RealDB but has lower reported total latency. In the two larger-database groups, Databox uses fewer LLM tokens and achieves greater latency reductions. The relative reduction in reported total latency increases from 11.3% to 80.4% across the three groups.

Databox's efficiency benefits are workload-dependent. On BIRD-CRITIC-PG, construction overhead largely offsets the debugging-time savings; on the evaluated Spider2-BigQuery workloads, Databox achieves lower reported latency, with larger reductions observed for larger DB size.

We will incorporate these results and qualifications into the revised paper.

Q2: Generalization to Larger Databases

We thank the reviewer for this constructive suggestion to evaluate Databox on larger databases. We have not yet been able to establish contact with the Spider2-Snow team and therefore have not obtained access to their Snowflake databases. As an alternative, we evaluated Databox on 150 instances from the Spider2 BigQuery subset, using SQL-Act and Spider2Agent, both with Gemini-3.5-Flash as the backbone model. This subset also covers large size databases with up to 785 tables, with the distribution shown below.

Tables per databaseNumber of instances
1–929
10–9979
100–78542
Total150

We will conduct the requested Snowflake experiments as soon as we establish contact with the Spider2-Snow team and obtain database access.

Overall results. The following table compares each agent using the original database (RealDB) and Databox.

AgentDatabase configurationPass rate (%) ↑Mean SQL execution time (s) ↓Speedup
SQL-ActRealDB36.03.123—
SQL-ActDatabox36.70.5945.26×
Spider2AgentRealDB35.33.710—
Spider2AgentDatabox38.60.7704.82×

On this subset, Databox reduces mean SQL execution time by factors of 5.26× and 4.82× for SQL-Act and Spider2Agent, respectively, while maintaining or improving the observed pass rate. These results provide additional evidence that the same execution-efficiency benefits of Databox can extend to the BigQuery setting with larger tables.

Results by schema size. We further group the evaluated instances by the number of tables in their associated database.

Tables per databaseNumber of instancesRealDB pass rate (%)Databox pass rate (%)Difference (pp)
1–92951.737.9−13.8
10–997932.938.0+5.1
100–7854229.336.9+7.6

The results show a performance decline for databases with relatively few tables, where questions over smaller schemas require a larger fraction of the original database, making the omission of relevant schema elements during Databox construction more consequential.

Databox achieves higher pass rates over larger table size, with +5.1% with 10-99 tables and +7.6% with more than 100 tables, which support the applicability of Databox to larger schemas.

Q3: Generalization with more models and agents

We thank the reviewer for highlighting the importance of evaluating Databox with more recent models and agent systems.

Newer backbone model. We evaluated Gemini 4 Argon (High Thinking) on 120 BIRD-CRITIC-PG SQL debugging instances using SQL-ACT and Spider2Agent. For each framework, we compared Databox with execution against the original database (RealDB).

Agent frameworkDatabase settingSuccess rate (%) ↑Average SQL execution time (ms) ↓
SQL-ACTRealDB63.312.31
SQL-ACTDatabox61.79.50
Spider2AgentRealDB54.213.38
Spider2AgentDatabox55.07.67

With SQL-ACT, Databox reduced average SQL execution time by 22.8%, with a 1.6 percentage-point decrease in success rate. With Spider2Agent, Databox reduced average SQL execution time by 42.7%, with a 0.8 percentage-point increase in success rate. These results provide additional evidence that Databox’s SQL execution efficiency benefits extend to a newer model across both evaluated frameworks, with small observed differences in task success rates.

Additional agent harness. We further evaluated ReFoRCE with Gemini 4 Argon on a subset of 103 instances from Spider2’s BigQuery benchmark.

VariantsSuccess rate (%) ↑Average SQL query time (s) ↓
RealDB58.25%4.76
Databox59.22%1.08

Under ReFoRCE, Databox reduced average SQL query time by 77.3%, while success rate increased by 0.97 percentage points.

We are also extending our evaluation to include more recent agent harnesses and additional backbone models to further assess Databox's generalization to frontier coding agent systems. We thank the reviewer for suggesting these additional evaluations.

Reviewer_T2B7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Databox, a task-specific sandbox for execution-guided SQL debugging. Given an issue description, problematic SQL, and database schema, it extracts relevant schema components, generates diagnostic data using a reusable guideline library, and validates the resulting environment. Databox-Local further uses small language models for local construction. Experiments across several agent frameworks and backbones show substantial reductions in database execution cost while broadly maintaining SQL repair success rates.

Critical strengths
  • The paper addresses practical concerns in SQL debugging, including database access restrictions, execution overhead, and isolation of exploratory modifications.
  • The diagnostic guideline mechanism is a promising component. Distilling reusable strategies for constructing data that exposes SQL errors offers targeted support for debugging.
  • The evaluation covers multiple agent frameworks, model backbones, data-generation alternatives, and execution interfaces. Reporting actual execution times alongside optimizer estimates and model-side costs provides useful detail.
Critical weaknesses
  1. The cost of constructing Databox deserves a fuller assessment. Typical SQL execution times in the experiments are only tens of milliseconds, while sandbox construction adds model calls, initialization, validation, and retries. The appendix reports increased total token usage in every evaluated configuration and higher API costs in most configurations. The authors should assess the complete workflow’s time and cost, including data generation and training for Databox-Local, and compare against simpler alternatives such as output truncation, targeted diagnostic queries, and isolated test databases.

  2. The contribution of diagnostic guidelines should be evaluated separately. MAGIC (Askari et al., 2025) generates self-correction guidelines from SQL errors. Databox’s extension to diagnostic data construction is valuable and merits a clearer comparison with this line of work. The evaluation should include construction with and without guidelines, and a baseline that provides the same guidelines directly to the RealDB debugging agent. Some Databox configurations also supply construction SQL, inserted data, reproduction outputs, and construction history. Controlling for this additional information would clarify the contributions of diagnostic guidance and the sandbox environment. Statistical tests would also help assess the modest success-rate improvements.

  3. The issue-fidelity check is insufficiently specified. At deployment, the constructor uses the issue description without original records or a reference repair. The paper explains executability and integrity-constraint checks, but gives limited detail on how successful reproduction is determined. A model could hypothesize a cause, generate data consistent with that hypothesis, and declare success. The authors should describe the semantic validation procedure and audit accepted sandboxes for faithful reproduction, missed defects, and spurious failures.

  4. Compound defects and partial repairs need explicit evaluation. SQL queries may contain multiple interacting or mutually masking errors. A compact sandbox could expose one defect while omitting the conditions needed to reveal another, allowing a partial repair to pass. Experiments with compound defects and deliberately incomplete repairs would help assess this risk. The paper should also report cases where sandbox validation succeeds but independent evaluation fails, together with the resulting rework.

  5. The evaluation should include more recent models and suitable operating budgets. Some main-experiment backbones are relatively old, and the SQL-level setup limits context to 4,096 tokens and interaction to five rounds. These settings may amplify the benefits of simplified schemas and shorter feedback. Adding a recent strong model, with appropriate context and interaction budgets, would enable a useful comparison between direct debugging, on-demand test-data generation, and Databox.

Reference

Askari, A., Poelitz, C., & Tang, X. (2025). MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL. Proceedings of the AAAI Conference on Artificial Intelligence, 39(22), 23433–23441.

Questions for authors

Please refer to the Weaknesses section for my suggestions and points for clarification.

7 comments

We thank the reviewer for highlighting the need for a fuller assessment of Databox construction costs and their impact on overall workflow efficiency.

W1: Construction costs and end-to-end efficiency

We report costs along two dimensions: database-side costs and LLM-side costs. The reported construction token counts include failed construction attempts, with detailed token usage provided in Tables 8 and 9.

The training details for Databox-Local are provided in Appendix C and D. We further clarify that training the Databox-Local constructor incurs a one-time, offline cost. Once trained, the same constructor can be reused to generate Databox instances for subsequent SQL debugging tasks without additional training. Consequently, this fixed training cost is amortized over the tasks served by the constructor, and its per-task contribution decreases as the number of tasks increases. We therefore exclude offline training costs from the reported per-task construction and debugging costs.

Our goal is to reduce the computational burden on the database side while not significantly increasing the additional LLM overhead. This involves a trade-off: Databox incurs additional LLM costs during construction to enable more efficient database execution during debugging. And we want to further claim that Databox has better end-to-end performance in tasks involving larger database size.

1. Construction and debugging latency on BIRD-CRITIC-PG.

To assess the impact of construction overhead, we report the following workflow latency breakdown, averaged across all nine settings:

EnvironmentConstruction: LLM latency (s)Construction: SQL execution (s)Debugging: LLM latency (s)Debugging: SQL execution (s)Total (s)
Databox7.250.04611.050.23518.581
RealDB——17.181.43618.616

As the reviewer notes, construction overhead largely offsets the debugging-time savings, resulting in approximately equal total latency. However, this does not imply that Databox cannot reduce end-to-end latency. Instead, it motivates a closer examination of the conditions under which Databox provides greater benefits, as discussed below.

2. Comparison with simpler alternative strategies.

We additionally report repair success rates and LLM costs for the population strategies evaluated in Table 4.

Data generation strategySR (%)Construction tokensConstruction LLM latency (s)Debugging tokensDebugging LLM latency (s)Total LLM latency (s)
Databox without preview37.623,3232.3315,5929.3011.63
Preview-row copy35.6400.0012,3977.447.44
Bernoulli + FK closure35.4500.0013,1298.318.31
Databox with preview34.068,3655.1814,5208.8614.04

Although the simpler strategies incur lower LLM-side costs like preview-row copy and Bernoulli sampling, their success rates are lower than that of Databox without preview. We therefore retain the latter strategy in our final Databox construction procedure. These results suggests the trade-off between repair success and LLM overhead.

3. Databox performance across database sizes.

We further clarify why the additional LLM overhead from Databox can be justified as database size increases. Our experiments on BIRD-CRITIC-PG use a simplified benchmark setting with relatively small databases, where LLM calls account for a substantial proportion of the overall cost. For real-world tasks involving larger databases, this additional overhead may become less significant relative to the cost of database execution.

To investigate this trend, we further evaluate Databox and RealDB with the SQL ACT agent on a Spider2-BigQuery subset containing 150 instances, grouped by the number of database tables.

TablesQuestionsQuery time (s), Databox / RealDBDebugging turns, Databox / RealDBLLM tokens, Databox / RealDB
1–9290.34 / 2.222.34 / 2.9022,778 / 19,595
10–99790.62 / 2.672.81 / 3.3235,619 / 56,788
100–785420.55 / 5.643.15 / 3.8588,878 / 497,089
TablesLLM latency (s), Databox / RealDBReported total latency (s), Databox / RealDBTotal latency reduction
1–913.80 / 10.0314.600 / 16.46811.3%
10–9921.69 / 32.7223.430 / 41.58043.7%
100–78550.40 / 243.8252.130 / 265.53480.4%

Across groups with increasing numbers of tables, RealDB exhibits higher SQL execution times and requires more agent actions, making the additional construction overhead of Databox easier to offset. Furthermore, the increases in Databox's LLM token usage are substantially smaller than those observed for RealDB in the larger-schema groups. The relative reduction in reported total latency consequently increases from 11.3% to 80.4%.

Databox's efficiency benefits depend on the workload. On BIRD-CRITIC-PG, the involved tasks contain only small size databases, and therefore construction overhead largely offsets the debugging-time savings. On the evaluated Spider2-BigQuery workloads, which is more related to real-world large scale databases, Databox exhibits substantially lower reported latency. This scaling discussion is added in our revised paper.

We thank the reviewer for suggesting a closer comparison with MAGIC and a separate evaluation of our diagnostic guidelines. We recognize that our description could have distinguished the construction and debugging stages more clearly.

W2: Contribution of Diagnostic Guidelines

1. Role of the guidelines.

MAGIC generates self-correction guidelines that directly guide SQL revision. In contrast, Databox uses guidelines exclusively during sandbox construction to generate diagnostic test data. These guidelines are not provided to the debugging agent in either the Databox or RealDB setting. Our guidelines are used for data generation, rather than through textual instructions for SQL correction. We will cite MAGIC and clarify this distinction in the appendix.

Here we provide a table to compare our guidelines based test data generation and MAGIC's guideline for SQL correction.

MethodUsed for diagnostic data generationGuide SQL correctionVisible to the debugging agent
MAGICNoYesYes
DataboxYesNoNo

2. Ablation Study of Guidelines

We conducted an additional experiment on 200 instances from BIRD-CRITIC-PG using Gemini-3.0-Flash, comparing final debugging success rates under Databox construction with and without guidelines.

Agent HarnessWithout guidelines (%)With guidelines (%)Improvement (percentage points)
SQL-ACT45.047.0+2.0
Spider2Agent40.041.5+1.5
IntercodeSQL37.039.0+2.0

Guideline-based construction yields modest improvements across all three debugging methods, increasing final success rates by 1.5-2.0 percentage points. These results support the utility of construction-stage guidelines for downstream debugging, without the need to provide any text information to guide the debugging procedures.

We want to further clarity that the Databox meta data provided to debugging stage only describe the Databox environment: its schema, inserted data, and applied constraints, such as foreign keys. They do not provide additional diagnostic guidance beyond this environment specification and not involve any construction details such as guidelines used for data generation. Only the finalized schema, data, constraints of Databox will be passed to the debugging stage.

We thank the reviewer for pointing out this unclear guideline explanation. We fixed the unclear part and provide more detailed information in the revised paper.

Comments on the paper

2 comments

Dear reviewers, The authors' rebuttal has been posted. If you get a chance before tomorrow's discussion deadline, can you get involved in the discussion and check whether your concerns are addressed or not?

AuthorsOpenReview ↗

Summary of the revisions

We thank all reviewers for their carefule reviews and constructive feedback. We have uploaded a revised PDF. The main changes are:

  1. Construction costs and end-to-end efficiency (Reviewer T2B7, xaJd and oz22). We added construction-plus-debugging latency (App. G.1), database-size scaling results (App. G.2), and population-strategy cost comparisons (App. I.1). We also clarified the abstract and conclusion's efficiency claims.
  2. Issue fidelity and Databox audits (Reviewers T2B7 and oz22). We clarified the rule-based and LLM-based issue-fidelity check details in App. D.1. We also added a new App. O reports manual audits of Databoxes and their repair candidate distinguish qualities.
  3. Newer model and agent harness (Reviewers T2B7 and xaJd). Appendix O adds Gemini 4 Argon evaluations with SQL-ACT, Spider2Agent, and ReFoRCE on BIRD-CRITIC-PG and Spider2 BigQuery.
  4. Additional ablation studies (Reviewers T2B7 and xaJd). Appendix I groups the existing data-population ablation with new diagnostic-guideline and RealDB schema-hint ablations (Tables 17–18). The results show modest gains from guideline-based construction, while schema hints alone do not reproduce Databox's gains.
  5. Generalization to larger databases (Reviewer xaJd). Appendix N adds evaluations on 150 Spider2-BigQuery instances, alongside pass-rate analysis by database size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.