acceptodds
Under review as a conference paper at ICLR 2027

HarnessSQL: HARNESS-NATIVE TRAINING FOR SQL AGENTS IN REALISTIC DATABASE ENVIRONMENTS

Abstract

Text-to-SQL models are commonly trained to map questions directly to static queries, whereas real-world database agents operate through stateful, multi-turn interaction with live databases—inspecting schemas, executing probe queries, diagnosing errors, and revising hypotheses. This creates a critical train–deploy mismatch, as the execution harness that mediates this interaction is introduced only at inference time. To bridge this gap, we propose **HarnessSQL**, a harness-native post-training framework that preserves the full interaction structure throughout both supervised fine-tuning and reinforcement learning. **HarnessSQL** builds isolated, executable database environments paired with hidden execution oracles, rolls out teachers directly inside the target SQL harness, and retains only verified trajectories for full-sequence SFT, followed by execution-reward RL. Across Spider 2.0-SQLite, **HarnessSQL** dramatically boosts the execution accuracy of compact models, raising Qwen3-8B from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8%, while transferring effectively to out-of-distribution interactive benchmarks such as BIRD-Interact and LiveSQLBench. Our findings demonstrate that training database agents directly within their execution harness is essential for mastering complex, long-horizon database workflows.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text2SQL, Harness, Data Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u9P3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HarnessSQL conducts full-trajectory SFT and execution-reward RL using verified trajectories in executable environments to mitigate the train–deploy mismatch in SQL Agents. And authors verified the performance in Spider 2.0-SQLite and 2 OOD benchmarks.

Critical strengths
  1. Under same backbone, the execution accuracy of 8B increases significantly to 45.2% and 14B to 54.8%, which is amazing.

  2. Sufficient evidence to show the effectiveness of proposed method, including fewer invalid calls and improved valid submission, which shows that the harness interaction capability is indeed improved.

Critical weaknesses
  1. The contribution of the proposed method is not very clear since MTSQL-R1 already used verified trajectory SFT and interactive RL. There is a lack of direct comparison to show the independent contribution of harness alignment.

  2. The explanation of key ablation is not very sufficient. Turn-split SFT still keeps history prefix. And under the same conditions, like token weighting, the objective may equal full-trajectory SFT. Figure 3 does not remove the influence of loss normalization, truncation, and training budget.

Questions for authors

Could authors tell me more about how much benefit HarnessSQL can bring compared with existing interactive SFT/RL under same conditions?

3 comments
AuthorsOpenReview ↗

(1/2) Response by authors

W1: Contribution relative to MTSQL-R1, and the independent effect of harness alignment.

Thank the reviewer for this point. We agree that MTSQL-R1 [1] also uses verified-trajectory SFT followed by interactive RL, and we do not claim this two-stage recipe as our contribution. The differences lie in what the agent is trained on and where:

  • Data. MTSQL-R1 trains on annotated conversational text-to-SQL data (CoSQL/SParC). HarnessSQL synthesizes verified tasks SQL-first from unlabeled databases.
  • Interaction loop. MTSQL-R1 refines SQL inside a prompt-defined loop with the schema given upfront. HarnessSQL trains inside the deployment harness, where the schema is discovered through tools and a runtime enforces submission and context limits.

We formalize both points in our response to Reviewer iWJV (W1, Q2).

Independent contribution of harness alignment. The ablations below change only the harness and keep everything else fixed (Qwen3-8B, S2-SQLite):

Controlled comparisonWhat changesResult
Table 3: SFT data format, inference fixed to DSH-SQLNon-harness → harness-native trajectories (same synthesized tasks)9.6% → 30.4%
Table 7: synthesis harness, same teacher and data budgetGeneric Full DSH → DSH-SQL15.5% → 23.0%

Non-harness SFT even falls below the untrained base model (15.5%), which shows that interactive training alone does not transfer when the training and deployment harnesses differ.

Direct comparison. The released MTSQL-R1 checkpoint scores 2.2% / 0.0% / 2.6% on S2-SQLite / BI-mini / LSB-SQLite (and 5.2% / 2.7% / 6.7% when evaluated directly under our DSH-SQL harness), versus 45.2% / 9.0% / 24.8% for HarnessSQL-8B (full table in our response to Reviewer Dsq9). This comparison also differs in data and model scale, so we rely on the controlled ablations above to isolate the effect of the harness.

[1] Guo, T., et al. MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training. ACL 2026.

W2: Explanation of the trajectory-organization ablation.

We agree that turn-split keeps the full history prefix, and that with identical token weighting and batching its expected objective matches full-trajectory SFT. We will correct the "preserving interaction structure" wording in Section 5.3. To isolate the three confounders you raised, we ran a controlled ablation with Qwen3-8B. All four rows use the same fixed subset of 997 trajectories, 32K context, and identical hyperparameters. Spider 2.0-SQLite is averaged over 4 runs (mean ± std). BI-mini and LSB are single runs.

#SettingS2-SQLiteBI-miniLSB-SQLite
1Turn-split, per-sample loss, 3 epochs21.48 ± 2.92.6714.81
2Turn-split, per-sample loss, steps matched to full-trajectory (189)20.56 ± 2.52.6711.48
3Turn-split, global token-mean loss, 3 epochs21.67 ± 2.83.6716.67
4Full-trajectory SFT24.81 ± 1.33.6717.04
  • Loss normalization. Your intuition holds out of domain. Token-level normalization (row 3) matches full-trajectory SFT on BI-mini and comes within 0.4 points on LSB. On S2-SQLite it changes little, and a gap of about 3 points remains.
  • Training budget. Turn-split runs 3,114 optimizer steps in 3 epochs, against 189 for full-trajectory SFT. Matching the step count (row 2) does not help.
  • Truncation. It affects both settings equally and almost never applies: 1/997 full trajectories and 3/16,608 turn-split samples exceed 32K. In both settings these are handled identically by dropping the earliest complete turns, and no sample is discarded.

With weighting and budget controlled, the residual in-domain gap comes from batch composition: full-trajectory SFT updates all turns of an episode jointly, while turn-split shuffles turns across episodes. Full-trajectory SFT is also much cheaper. With the same 6.19M supervised tokens per epoch, turn-split needs 9.0× more input tokens and 9.7× more GPU-hours (91.7 vs. 9.4 on 8×H800), because it re-encodes the history prefix for every turn.

We will add this ablation and the revised explanation to Section 5.3.

AuthorsOpenReview ↗

(2/2) Response by authors

Q1: How much does HarnessSQL bring over existing interactive SFT/RL under the same conditions?

Controlled comparison. We fixed the 1,000 synthesized tasks, the Qwen3.6-27B teacher, the Qwen3-8B student, and the SFT budget, and changed only the interaction environment used for trajectory synthesis, training, and evaluation. The baseline is a conventional interactive loop in the style of MTSQL-R1 and SQL-Trail: the schema is given in the prompt, and a single SQL-execution action is repeated until a final answer.

EnvironmentS2-SQLiteBI-miniLSB-SQLite
Conventional interactive loop20.7%1.7%6.9%
HarnessSQL (DSH-SQL harness)23.0% (+2.3)4.0% (+2.3)14.8% (+7.9)

This is an SFT-only comparison at 1k tasks. The gain is largest out of domain (more than 2× on LSB-SQLite), which is where the interaction environment matters most.

Why a harness differs from an interactive loop. An interactive loop is a prompting format. A harness is a layered runtime that the agent is trained and deployed in (Appendix A, Tables 5–6). We restrict only the model-facing tools to four SQL tools. The underlying runtime layers are all kept:

AspectConventional interactive loopDSH-SQL harness
ToolsOne action (execute SQL); schema given upfrontTyped tools list_tables / schema / exec / submit; the schema is explored on demand, and submission is an explicit, validated action
Context / memoryContext grows until truncatedSession state with durable checkpoints, pressure-triggered compaction, bounded observations, and repeated-call reminders (22 / 19 activations in one 135-task evaluation; Table 6)
Skills / extensibilityBehavior fixed in the promptSkills are mountable plugins. Our results use none, so the gains do not rely on hand-written knowledge. SQL skills (e.g., dialect handling or solving strategies) can be added without changing the agent loop or the tool interface
Environment & concurrencyShared process; failures can leak across tasksPer-task process, workspace, and read-only container; killable per-query deadlines (60 s); parallel workers (at most 8) for large-scale rollouts and RL
GeneralizationA new benchmark requires a new loopA new backend or benchmark is a new plugin. The same four-tool abstraction covers SQLite, PostgreSQL, and BigQuery, and the BIRD-Interact profile swaps in its nine operations (including user clarification) with no change to the agent

The last row explains the OOD results. The model learns a protocol (explore → verify → submit) rather than a benchmark-specific prompt format, and this protocol transfers when the interface changes. The ablations in our response to W1 isolate the same effect: harness-native SFT reaches 30.4% vs. 9.6% for non-harness SFT (Table 3), and the specialized harness improves on the generic one by +7.5 points (Table 7).

Thank you for your clarification. These largely resolved my concerns. I decide to maintain my original score.

Reviewer_iWJVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HarnessSQL, a framework for training interactive Text-to-SQL agents in realistic database environments. The authors aim to address the gap between traditional Text-to-SQL training, which focuses on generating final SQL queries, and real-world SQL agents that need to interact with databases through multiple tool calls. HarnessSQL combines executable database sandboxes, an SQL-specific tool interface, and a two-stage training process using supervised fine-tuning on verified interaction trajectories followed by reinforcement learning with execution-based rewards. The authors evaluate the framework using Qwen3-8B and Qwen3-14B on Spider 2.0-SQLite, BIRD-Interact Mini, and LiveSQLBench. The results show substantial improvements in execution accuracy over the base models and existing baselines. Additional ablation studies examine the effects of harness-native training, trajectory organization, context length, and RL algorithms.

Critical strengths
  1. The paper addresses the gap between traditional Text-to-SQL training and real-world SQL agents that need to interact with databases through multiple tool calls. HarnessSQL provides a complete training pipeline that combines executable database environments, verified interaction trajectories, SFT, and RL. I think this is a practical approach to improving interactive SQL agents.
  2. HarnessSQL significantly improves execution accuracy on Spider 2.0-SQLite and shows improvements on BIRD-Interact Mini and LiveSQLBench. The authors also provide ablation studies comparing harness-native and non-harness training, full-trajectory and turn-split SFT, and different RL algorithms. These results provide good evidence that training agents within their execution environment is effective.
Critical weaknesses
  1. The main components of HarnessSQL, including tool-based database interaction, execution-verified trajectory collection, full-trajectory SFT, and execution-reward RL, are largely based on existing agent training techniques. While combining these components into a complete SQL training pipeline is useful, the paper does not clearly demonstrate what fundamentally new training method or mechanism it introduces beyond applying established techniques to interactive Text-to-SQL tasks. This limits the methodological contribution of the work.
  2. In Table 1, most baselines are traditional Text-to-SQL models rather than methods specifically designed for interactive SQL agent training. Although the reported performance improvements are substantial, the current comparisons do not fully establish how competitive HarnessSQL is against existing approaches that also train agents through multi-turn database interactions. This makes it difficult to assess the significance of the proposed framework beyond the general benefits of interactive training.
Questions for authors
  1. In Section 3, the proposed HarnessSQL framework combines executable database environments, verified teacher trajectories, full-trajectory SFT, and execution-reward RL. However, these components are already commonly used in agent training. Could the authors clarify what the main methodological contribution is beyond combining existing techniques for interactive Text-to-SQL? In particular, what makes the SQL-specific harness and training procedure fundamentally different from existing approaches that also train agents through tool interactions and execution feedback?
  2. In Table 1, most of the baselines are traditional Text-to-SQL models rather than methods trained for multi-turn database interaction. Recent approaches such as SQL-Trail [1] and MTSQL-R1 [2] also train SQL agents to interact with databases, use execution feedback, and iteratively refine their queries. Could the authors provide a more detailed comparison with these methods, ideally under comparable evaluation settings?
  3. In Section 5.3 and Figure 3, the authors show that full-trajectory SFT outperforms turn-split SFT. However, the paper states that turn-split SFT still conditions each action on its preceding history. Since both approaches therefore have access to the same interaction history, what exactly explains the performance difference? For example, could the improvement come from differences in loss weighting, sequence packing, or optimization rather than the preservation of interaction information itself?

[1] Hua, Harper, et al. "SQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. [2] Guo, Taicheng, et al. "Mtsql-r1: Towards long-horizon multi-turn text-to-sql via agentic training." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

5 comments
AuthorsOpenReview ↗

(1/3) Response by authors

W1: Methodological contribution beyond combining existing techniques.

We agree that SFT and DAPO are standard, and we do not claim a new loss. Our contribution is to formulate and solve two problems that existing text-to-SQL training leaves implicit.

Problem 1: Harness-aware policy learning. A deployed agent is the composition of πθ\pi_\theta with a harness H=(AH,ΩH,MH)H = (\mathcal{A}_H, \Omega_H, M_H): tool actions, observation function, and runtime middleware. The deployment objective is

JH(θ)=Eq∼Q, τ∼pθH(⋅∣q)[vq(τ)],J_H(\theta) = \mathbb{E}_{q \sim \mathcal{Q},\ \tau \sim p_\theta^{H}(\cdot \mid q)} \left[ v_q(\tau) \right],

where pθHp_\theta^{H} is the trajectory distribution induced by running πθ\pi_\theta inside HH. Existing methods optimize JH′J_{H'} for a different loop H′H' and then deploy under HH. Because HH determines every state sts_t, this shifts the context distribution at every turn, and the shift compounds over the horizon (median 16 turns) [1]. HarnessSQL fixes the same HH for synthesis, SFT,

max⁡θ Eτ∼pteacherH[∑t=1Tlog⁡πθ(at∣st) | vq(τ)=1],\max_\theta\ \mathbb{E}_{\tau \sim p^{H}_{\mathrm{teacher}}} \left[ \sum_{t=1}^{T} \log \pi_\theta(a_t \mid s_t) \ \middle|\ v_q(\tau) = 1 \right],

RL (max⁡θJH\max_\theta J_H), and evaluation. This alignment, not interactive training per se, is decisive. Under DSH-SQL inference, non-harness SFT degrades the base model (15.5% → 9.6%), while harness-native SFT reaches 30.4% (Table 3). Changing only the synthesis harness raises accuracy from 15.5% to 23.0% (Table 7).

Problem 2: Verified tasks from unlabeled databases. Given only a database DD, we invert the question → SQL direction and build tasks whose verifiers are correct by construction:

y⋆∼Pbp(⋅∣C(D),κ),q=g(y⋆),vq(τ)=1[Exec(y^τ,D)≡Exec(y⋆,D)],y^\star \sim P_{\mathrm{bp}}(\cdot \mid \mathcal{C}(D), \kappa), \quad q = g(y^\star), \quad v_q(\tau) = \mathbb{1} \left[ \mathrm{Exec}(\hat{y}_\tau, D) \equiv \mathrm{Exec}(y^\star, D) \right],

where C(D)\mathcal{C}(D) is the live catalog, κ\kappa are the difficulty controls, and ≡\equiv denotes equality of normalized results. A task is admitted only if Exec(y⋆,D)\mathrm{Exec}(y^\star, D) is non-degenerate and Exec(m(y⋆),D)≢Exec(y⋆,D)\mathrm{Exec}(m(y^\star), D) \not\equiv \mathrm{Exec}(y^\star, D) for every mutation m∈Mm \in \mathcal{M}, so the oracle rejects near-miss queries. The same oracle drives both training stages:

  • SFT rejection-samples teacher trajectories with vq(τ)=1v_q(\tau) = 1 from pteacherHp^{H}_{\mathrm{teacher}}.
  • RL trains on Deff=q:0<∑ivq(τi)<G\mathcal{D}_{\mathrm{eff}} = \\{ q : 0 < \sum_{i} v_q(\tau_i) < G \\}, the frontier of the current policy.

This yields verified, harness-native training data for any database without annotation.

We will rewrite the contribution list to state explicitly which parts are reused (SFT, DAPO, the DSH substrate; Appendix A, Table 5) and which are new (the two formulations above).

[1] Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.

AuthorsOpenReview ↗

(2/3) Response by authors

W2, Q1: Competitiveness against methods that also train agents through multi-turn interaction.

Conceptual differences. These are covered in detail in our responses to W1 and Q2. In short, prior multi-turn methods train on annotated question–SQL pools inside their own generate–execute loops. HarnessSQL synthesizes verified tasks from unlabeled databases and trains inside the same harness it is deployed in.

Beyond the general benefits of interactive training. To separate the harness from interactive training per se, we ran a controlled comparison. We fixed the 1,000 synthesized tasks, the teacher, the Qwen3-8B student, and the SFT budget, and changed only the environment: a conventional interactive loop (schema in the prompt, a single SQL-execution action) vs. the DSH-SQL harness.

Environment (SFT, 1k tasks)S2-SQLiteBI-miniLSB-SQLite
Conventional interactive loop20.7%1.7%6.9%
DSH-SQL harness23.0%4.0%14.8%

Comparison with existing methods. We evaluated inference-time agentic frameworks and trained multi-turn models on all three benchmarks. Baselines use their official implementations, and MTSQL-R1 is additionally evaluated under DSH-SQL. SQL-Trail and MTIR-SQL have not released code or checkpoints. SWE-SQL targets SQL debugging, a different task. MTSQL-R1 is therefore the only released trained multi-turn model.

MethodBackboneInteraction ParadigmS2-SQLite (ID)BI-mini (OOD)LSB-SQLite (OOD)
Reference Points
Qwen3-8B (Base)8BDSH-SQL (zero-shot)15.5%3.3%12.2%
Qwen3-14B (Base)14BDSH-SQL (zero-shot)22.2%4.7%9.6%
Qwen3.6-27B (Teacher)27BDSH-SQL60.0%19.3%25.6%
Claude Opus 5.5FrontierDSH-SQL64.4%21.3%34.2%
Inference-Time Multi-Turn / Agentic Baselines
ReFoRCEQwen3-8BMulti-turn refinement17.0%0.3%1.1%
ReFoRCEClaude Opus 5.5Multi-turn refinement54.8%2.3%7.4%
FlexSQLQwen3-8BFlexible execution loop9.6%0.3%2.9%
FlexSQLClaude Opus 5.5Flexible execution loop58.5%4.3%8.2%
Trained Multi-Turn SQL Models
MTSQL-R1-CoSQL4BMulti-turn execution loop2.2%0.0%2.6%
MTSQL-R1-CoSQL4BDSH-SQL5.2%2.7%6.7%
Ours: HarnessSQL
HarnessSQL (SFT)8B4-tool harness-native30.4%5.3%16.6%
HarnessSQL (SFT + RL)8B4-tool harness-native45.2%9.0%24.8%
HarnessSQL (SFT + RL)14B4-tool harness-native54.8%11.3%28.5%
  • The harness matters even without training. With the same frontier backbone, DSH-SQL outperforms ReFoRCE and FlexSQL by 6–10 points on S2-SQLite and by 3–5× on both OOD benchmarks.
  • Interactive training alone is not enough. MTSQL-R1, trained for multi-turn interaction, improves when moved into DSH-SQL but stays far below HarnessSQL. In the controlled comparison, the harness beats the conventional loop on all three benchmarks, by more than 2× on LSB-SQLite.
  • Training in the harness closes much of the gap at small scale. HarnessSQL-8B outperforms the 8B agentic baselines by more than 28 points on S2-SQLite and surpasses the Opus-based ReFoRCE and FlexSQL on LSB-SQLite.
AuthorsOpenReview ↗

(3/3) Response by authors

Q2: Detailed comparison with SQL-Trail and MTSQL-R1.

Thank the reviewer for this point. We will cite and discuss both SQL-Trail [1] and MTSQL-R1 [2] in the related work. The main difference is that "interacting with the database" means something fundamentally different in these works than in a modern agent harness (e.g., DeepSeek Harness, Codex, Claude Code).

SQL-Trail [1] / MTSQL-R1 [2]HarnessSQL (DSH-SQL harness)
What "multi-turn" meansIterative refinement of one SQL query (SQL-Trail); multi-turn conversational text-to-SQL (MTSQL-R1, e.g., CoSQL/SParC)Long-horizon agent episodes (median 16 turns, ~11.8K tokens)
Schema accessProvided upfront in the initial prompt (static DDL)Discovered on demand via sql_list_tables / sql_schema
Action spaceGenerate SQL receive execution resultTyped tools: list, inspect, execute, submit
TerminationFinal SQL extracted via tag/text parsing upon generation stopExplicit sql_submit tool protocol, verified by the harness
Runtime managementNone; the loop is a prompting formatMiddleware: observation bounding, repeated-call guards, context compaction, read-only sandbox, timeouts
Role of the loopTraining scaffold, defined by each paperDeployment runtime, held fixed for synthesis, SFT, RL, and evaluation

In short, these methods train a model to refine queries inside a prompt-defined loop. A modern harness, by contrast, is the runtime the agent is deployed in: it defines what the agent can observe and do, so it must be part of training (see our response to W1). This difference matters most on enterprise-scale databases like Spider 2.0, where providing the full schema upfront is infeasible and the agent must explore iteratively.

For comparable evaluation, we evaluated the released MTSQL-R1-CoSQL-4B checkpoint under both its native protocol and our identical DSH-SQL harness across all three benchmarks (full table in our response to W2):

  • Native protocol: 2.2% / 0.0% / 2.6% on S2-SQLite / BI-mini / LSB-SQLite.
  • DSH-SQL harness: 5.2% / 2.7% / 6.7% (vs. 45.2% / 9.0% / 24.8% for HarnessSQL-8B). While our DSH-SQL environment lifts MTSQL-R1 over its native setup, the large remaining gap confirms that inference-time harness access alone is insufficient without harness-aligned trajectory post-training. SQL-Trail has not released code or checkpoints, so we cannot evaluate it.

[1] Hua, H., et al. SQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL. ACL 2026.
[2] Guo, T., et al. MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training. ACL 2026.

Q3: What explains the gap between full-trajectory and turn-split SFT?

We agree with the premise. Under causal attention, turn-split conditions each turn on the same history prefix as full-trajectory SFT, so the gap cannot come from extra information. Neither setting uses sequence packing: each sample is a separate sequence with per-GPU batch size 1. Two factors remain, loss weighting and optimization. We ran a controlled ablation with Qwen3-8B. All four rows use the same fixed subset of 997 trajectories, 32K context, and identical hyperparameters. Spider 2.0-SQLite is averaged over 4 runs (mean ± std). BI-mini and LSB are single runs.

#SettingS2-SQLiteBI-miniLSB-SQLite
1Turn-split, per-sample loss, 3 epochs21.48 ± 2.92.6714.81
2Turn-split, per-sample loss, steps matched to full-trajectory (189)20.56 ± 2.52.6711.48
3Turn-split, global token-mean loss, 3 epochs21.67 ± 2.83.6716.67
4Full-trajectory SFT24.81 ± 1.33.6717.04
  • Optimization budget is not the cause. Matching optimizer steps (row 2) does not close the gap.
  • Loss weighting explains the OOD gap. Per-sample averaging up-weights short tool-call turns. Token-level normalization (row 3) matches full-trajectory SFT on BI-mini and comes within 0.4 points on LSB.
  • A residual in-domain gap remains. On S2-SQLite the gap is about 3 points, with lower variance for full-trajectory SFT. Once weighting is equalized, the only remaining difference is batch composition. Full-trajectory SFT updates all interdependent turns of an episode in one gradient step, while turn-split shuffles turns across episodes.
  • Efficiency. Both settings supervise the same 6.19M tokens per epoch, but turn-split re-encodes the history prefix for every turn. On the same 8×H800 setup this costs 9.0× more input tokens and 9.7× more GPU-hours (91.7 vs. 9.4). We will add this ablation to Section 5.3. We will also revise the claim there: the advantage comes from token-level weighting, episode-level batching, and training efficiency, not from turn-split lacking history.
Reviewer_Dsq9OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HarnessSQL, a post-training framework for interactive SQL agents that keeps the deployment harness in the training loop. Tasks are synthesized SQL-first with hidden execution oracles inside read-only sandboxes. A 4-tool SQL harness (list_tables / schema / exec / submit) is derived from DSH. A teacher model rolls out inside this harness, and the verified trajectories are used for full-trajectory SFT, followed by DAPO with a binary execution reward. Qwen3-8B/14B improve from 15.5/22.2% to 45.2/54.8% on Spider 2.0-SQLite, and the authors report transfer to BIRD-Interact Mini and LiveSQLBench.

Critical strengths
  1. The train–deploy mismatch is clearly motivated. Table 3 is the most convincing experiment: under the same DSH-SQL inference, harness-native SFT (30.4%) far outperforms non-harness SFT (9.6%), which is even below the untrained base (15.5%).
  2. The ablations are fairly complete: SFT vs. direct RL, full-trajectory vs. turn-split, context length, RL algorithm, teacher strength, and harness specialization.
  3. The behavioral analysis usefully separates what SFT contributes from what RL contributes.
Critical weaknesses
  1. No agentic baselines. Every baseline in Table 1 is a single-pass generator evaluated under its native protocol. Multi-turn and agentic text-to-SQL methods that the paper itself cites (ReFoRCE, MTIR-SQL, MTSQL-R1, SWE-SQL, FlexSQL) are not compared.
  2. Missing reference points. The teacher’s (Qwen3.6-27B) own accuracy on the benchmarks is not reported, and neither are frontier models. The Qwen3 base + DSH-SQL row is also absent from Table 1. Without these, it is hard to judge how much headroom remains and whether the student is simply distilling the teacher.
  3. Shared databases between training and test. The 30 training SQLite databases are byte-identical to the test databases.
  4. Weak non-harness baseline. The base model gets only 2.2% under the non-harness setting in Table 3, far below Qwen2.5-Coder-32B (16.3%). This suggests a weak prompt setup, which would inflate the measured harness gain.
  5. The RL stage is small. It uses only about 600 prompts (150 iterations x 4) out of a pool of 2,700. Table 8 shows that RL barely reduces SQL errors (9.0% -> 8.1%) and mainly improves termination. This weakens the claim that RL “optimizes complex execution-guided reasoning.”
Questions for authors
  1. What do the teacher and a frontier model score on all three benchmarks under DSH-SQL?
  2. How do HarnessSQL models perform on Spider 2.0 databases that were never used for synthesis?
  3. Which reward was actually used in RL? And was the main RL algorithm DAPO or GRPO?
  4. How would an agentic baseline such as ReFoRCE or MTIR-SQL at a similar scale compare under the same harness?
5 comments
AuthorsOpenReview ↗

(1/3) Response by authors

W1, W2, Q1: Teacher, frontier, and base reference points.

All reference rows are evaluated under DSH-SQL, the same protocol as our models.

MethodBackboneInteraction ParadigmS2-SQLite (ID)BI-mini (OOD)LSB-SQLite (OOD)
Reference Points
Qwen3-8B (Base)8BDSH-SQL (zero-shot)15.5%3.3%12.2%
Qwen3-14B (Base)14BDSH-SQL (zero-shot)22.2%4.7%9.6%
Qwen3.6-27B (Teacher)27BDSH-SQL60.0%19.3%25.6%
Claude Opus 5.5FrontierDSH-SQL64.4%21.3%34.2%
Inference-Time Multi-Turn / Agentic Baselines
ReFoRCEQwen3-8BMulti-turn refinement17.0%0.3%1.1%
ReFoRCEClaude Opus 5.5Multi-turn refinement54.8%2.3%7.4%
FlexSQLQwen3-8BFlexible execution loop9.6%0.3%2.9%
FlexSQLClaude Opus 5.5Flexible execution loop58.5%4.3%8.2%
Trained Multi-Turn SQL Models
MTSQL-R1-CoSQL4BMulti-turn execution loop2.2%0.0%2.6%
MTSQL-R1-CoSQL4BDSH-SQL5.2%2.7%6.7%
Ours: HarnessSQL
HarnessSQL (SFT)8B4-tool harness-native30.4%5.3%16.6%
HarnessSQL (SFT + RL)8B4-tool harness-native45.2%9.0%24.8%
HarnessSQL (SFT + RL)14B4-tool harness-native54.8%11.3%28.5%
  • Agentic baselines. At 8B, HarnessSQL outperforms ReFoRCE and FlexSQL by more than 28 points on S2-SQLite. With the same frontier backbone, DSH-SQL itself outperforms both frameworks by 6–10 points on S2-SQLite and by 3–5× out of domain, so the harness contributes beyond the backbone.
  • Trained multi-turn models. MTSQL-R1 improves on all three benchmarks when moved from its own loop to DSH-SQL (e.g., 2.2% → 5.2% on S2-SQLite), but stays far below HarnessSQL, which is trained inside the harness.
  • Is the student only distilling the teacher? No:
    • SFT alone (pure imitation) recovers only half of the teacher's in-domain accuracy (30.4% vs. 60.0%). RL then adds 14.8 points without any teacher signal.
    • On LSB-SQLite, the 8B model nearly matches the teacher (24.8% vs. 25.6%), and the 14B model surpasses it (28.5%).
  • Headroom. Headroom remains, mostly out of domain. The 14B model reaches 91% of the teacher on S2-SQLite but 59% on BI-mini, and Claude Opus 5.5 sets a higher ceiling on all three benchmarks.

We will add these rows, including the base + DSH-SQL rows, to Table 1.

W3, Q2: Shared databases between training and test.

Thanks for the comment. We clarify that our gains come from mastering procedural interaction rather than memorizing database schemas by the following results:

  1. New Zero-Shot Cross-Database Experiment:

To directly rule out schema memorization, we trained Qwen3-8B exclusively on the 715 DBT trajectories (Appendix K, Table 16), ensuring zero exposure to the 30 Spider 2.0-Lite SQLite databases:

ModelTraining DatabasesS2-SQLite Test Accuracy
Qwen3-8B BaseNone (Zero-shot)15.5%
Qwen3-8B (DBT-only)Unseen only (49 DBT DBs)25.2%
Qwen3-8B (Full SFT)All (79 DBs, 2,512 trajs)30.4%

The DBT-only model significantly beats the base model (+9.7%) and retains >80% of the full SFT performance, proving that our agent masters generalizable schema exploration rather than memorizing database schemas.

  1. Consistent Strong Performance on OOD Benchmarks:* Both BIRD-Interact Mini and LiveSQLBench feature completely unseen databases and disparate schema topologies. As shown in Table 1, HarnessSQL achieves 9.0% / 11.3% on BI-mini and 24.8% / 28.5% on LSB-SQLite for 8B and 14B models, outperforming all evaluated 32B-scale baselines by a large margin (1.7%–3.0% and 7.0%–8.9%).

  2. Strict Zero-Leakage Audit on Shared Databases (Appendix E):

    • Evaluating on shared databases mirrors realistic enterprise settings where schema structures are static while incoming queries vary over time.
    • As detailed in our three-tier audit (Appendix E, Table 12–14), across all shared databases, our synthesized tasks exhibit zero lexical exact/substring matches, a maximum sequence ratio of only 0.169, zero instances with embedding cosine similarity 0.75, and strictly zero execution result fingerprint overlap against the evaluation queries.
AuthorsOpenReview ↗

(2/3) Response by authors

W4: Weak non-harness baseline.

Thank you for this point. The 2.2% in Table 3 comes from a minimal setting: the model receives the question and database schema and outputs the final SQL directly, without reasoning. To rule out an under-tuned baseline, we re-evaluated the Qwen3 base models with the reasoning-based inference protocol of SQL-R1, the same protocol used for Qwen2.5-Coder-32B in Table 1:

ModelNon-harness (Arctic-Text2SQL-R1 reasoning protocol)DSH-SQL (zero-shot)
Qwen3-8B9.6%15.5%
Qwen3-14B11.9%22.2%
Qwen2.5-Coder-32B16.3%3.0%

Reasoning raises the non-harness baseline substantially (2.2% → 9.6% at 8B). The harness still adds +5.9 points at 8B and +10.3 points at 14B, and Qwen3-14B under DSH-SQL surpasses the 32B coder model. The harness gain therefore does not rely on a weak prompt.

Qwen2.5-Coder-32B, which was not trained for multi-turn tool use, does not benefit from the harness (16.3% → 3.0%). The gain from a harness depends on the model's ability to operate within it, which is exactly what harness-native training provides: after training, the same Qwen3 backbones reach 45.2% / 54.8%. We will replace the non-harness rows of Table 3 with this stronger setting.

W5: The RL stage is small and mainly improves termination.

We agree that the RL stage uses relatively few unique prompts, and we will state this explicitly. Two clarifications follow.

Algorithm (Q3). All reported RL checkpoints use DAPO with the binary execution reward (Appendix C.1, Table 10). The GRPO description in Appendix C.2 is an oversight from an earlier draft, and we will remove it.

Scale. Each prompt is rolled out as G=8 full multi-turn episodes in the live sandbox. DAPO's dynamic sampling (Eq. 11) rolled out 899 prompt groups to retain 600 groups with non-zero reward variance. In compute, the RL stage is therefore not small:

StageUnique promptsTrajectoriesAvg. turnsTokens processed (generated)GPU-hours
Full-trajectory SFT (3 epochs)2,5127,53616.088.8M (46.2M)23.3
Single-turn RL (600 prompts, est.)6004,8001.017.4M (2.2M)~38
Agent RL (ours, DAPO)6004,80018.081.5M (29.9M)178.0

Where the gain comes from. S2-SQLite, 135 tasks × 4 runs = 540 trajectories, the same runs as Appendix B.2, Table 8:

OutcomeSFTSFT + DAPOChange
Correct156 (28.9%)238 (44.1%)+82
Valid submission, wrong result281 (52.0%)273 (50.6%)−8
No valid submission103 (19.1%)29 (5.4%)−74

The reviewer is right that termination is a large part of the gain. However, the 74 rescued episodes end in correct answers: wrong submissions do not increase. RL is therefore not simply forcing earlier submission, and the agent still has to solve the task. The flat SQL execution-error rate (9.0% → 8.1%) counts every rejected sql_exec during exploration. Many of these rejections are benign: over 80% of failed executions in the teacher data are read-only guard blocks (e.g., queries starting with --; Appendix D), and the agent recovers immediately in 72.7% of cases (Table 8). We will revise the claim to: SFT establishes protocol and syntax competence, while RL improves episode-level reliability (termination, protocol adherence) and final-answer correctness. This replaces "optimizes complex execution-guided reasoning".

Q4: RL algorithm

Thanks for the question. We clarify the RL algorithm and reward formulation as follows:

  1. Main RL Algorithm (DAPO):

    • The primary algorithm across all main experiments (Table 1, Table 2, and Table 8) is DAPO.
    • This is explicitly stated in Section 3.3 (Line 227–228) ("We optimize the policy using Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)")and detailed with launch hyperparameters in Appendix C.1 (Table 10).
    • GRPO (along with GSPO) was evaluated solely as an algorithm ablation under matched budgets in Section 5.3 (Table 4) and Appendix C.2.
  2. Reward Function (Binary Execution Correctness):

    • The reinforcement learning stage strictly uses a sparse, binary execution-based terminal reward:
      • R(τ)=1R(\tau) = 1 if the executed relation of the submitted SQL query matches the normalized ground-truth result.
      • R(τ)=0R(\tau) = 0 otherwise (including syntactic failures, wrong outputs, timeouts, or missing submissions).
    • This formulation is defined in Section 3.3 (Line 229–231) and formally specified in Appendix C.2 (Equation 13). No intermediate reward shaping, token penalties, or syntactic partial credits are used.
AuthorsOpenReview ↗

(3/3) Response by authors

Q4: How would ReFoRCE or MTIR-SQL at a similar scale compare under the same harness?

The two methods differ from HarnessSQL along different axes:

DimensionReFoRCEMTIR-SQLHarnessSQL (ours)
TypeInference-time pipeline (no training)Multi-turn RL trainingHarness-native post-training (SFT + RL)
Training data–Existing annotated text-to-SQL setsSynthesized SQL-first from raw databases, with hidden execution oracles
Schema accessCompressed schema in the promptSchema in the promptExplored on demand via list_tables / schema
Interaction interfaceIts own pipeline (iterative column exploration, self-refinement, voting)Its own generate–execute loopTyped tools (list_tables / schema / exec / submit) in a runtime with session memory, context compaction, bounded observations, and sandboxed execution
ExtensibilityFixed pipelineFixed loopBackends, benchmarks, and skills are plugins on one agent loop
Training env. = deployment env.N/A✗ (trained in its own loop)✓ (same harness for synthesis, SFT, RL, and evaluation)
Supervision–Execution rewardVerified full-trajectory SFT, then execution reward

On "under the same harness": ReFoRCE is an inference pipeline, so running its 8B backbone in DSH-SQL corresponds to our zero-shot base row (15.5% on S2-SQLite). In its own pipeline at 8B, ReFoRCE reaches 17.0%, versus 45.2% for HarnessSQL-8B.

MTIR-SQL has not released checkpoints, so we use the released multi-turn RL model MTSQL-R1-CoSQL-4B as a proxy (table in our response to W1/W2). Moving it from its own loop into DSH-SQL improves it on all three benchmarks: 2.2% → 5.2% on S2-SQLite, 0.0% → 2.7% on BI-mini, and 2.6% → 6.7% on LSB-SQLite. It nevertheless remains far below HarnessSQL-8B (45.2% / 9.0% / 24.8%). The harness therefore helps even a model trained elsewhere, but most of the benefit comes from training inside it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Friendly Reminder to Reviewers

Dear Reviewer Dsq9 & iWJV & u9P3,

Thanks for your attention to this email. During the rebuttal phase, we extensively answered the questions raised by the reviewers and believe we have addressed the reviewers' concerns and made this interesting work better. This is a friendly reminder that the author-reviewer discussion period is coming to an end. We understand you are busy, but we would appreciate any additional feedback or suggestions you might have on our work. Your insights are valuable to us and HarnessSQL.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.