- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
HarnessSQL conducts full-trajectory SFT and execution-reward RL using verified trajectories in executable environments to mitigate the train–deploy mismatch in SQL Agents. And authors verified the performance in Spider 2.0-SQLite and 2 OOD benchmarks.
Critical strengths
-
Under same backbone, the execution accuracy of 8B increases significantly to 45.2% and 14B to 54.8%, which is amazing.
-
Sufficient evidence to show the effectiveness of proposed method, including fewer invalid calls and improved valid submission, which shows that the harness interaction capability is indeed improved.
Critical weaknesses
-
The contribution of the proposed method is not very clear since MTSQL-R1 already used verified trajectory SFT and interactive RL. There is a lack of direct comparison to show the independent contribution of harness alignment.
-
The explanation of key ablation is not very sufficient. Turn-split SFT still keeps history prefix. And under the same conditions, like token weighting, the objective may equal full-trajectory SFT. Figure 3 does not remove the influence of loss normalization, truncation, and training budget.
Questions for authors
Could authors tell me more about how much benefit HarnessSQL can bring compared with existing interactive SFT/RL under same conditions?
3 comments
(2/2) Response by authors
Q1: How much does HarnessSQL bring over existing interactive SFT/RL under the same conditions?
Controlled comparison. We fixed the 1,000 synthesized tasks, the Qwen3.6-27B teacher, the Qwen3-8B student, and the SFT budget, and changed only the interaction environment used for trajectory synthesis, training, and evaluation. The baseline is a conventional interactive loop in the style of MTSQL-R1 and SQL-Trail: the schema is given in the prompt, and a single SQL-execution action is repeated until a final answer.
| Environment | S2-SQLite | BI-mini | LSB-SQLite |
|---|---|---|---|
| Conventional interactive loop | 20.7% | 1.7% | 6.9% |
| HarnessSQL (DSH-SQL harness) | 23.0% (+2.3) | 4.0% (+2.3) | 14.8% (+7.9) |
This is an SFT-only comparison at 1k tasks. The gain is largest out of domain (more than 2× on LSB-SQLite), which is where the interaction environment matters most.
Why a harness differs from an interactive loop. An interactive loop is a prompting format. A harness is a layered runtime that the agent is trained and deployed in (Appendix A, Tables 5–6). We restrict only the model-facing tools to four SQL tools. The underlying runtime layers are all kept:
| Aspect | Conventional interactive loop | DSH-SQL harness |
|---|---|---|
| Tools | One action (execute SQL); schema given upfront | Typed tools list_tables / schema / exec / submit; the schema is explored on demand, and submission is an explicit, validated action |
| Context / memory | Context grows until truncated | Session state with durable checkpoints, pressure-triggered compaction, bounded observations, and repeated-call reminders (22 / 19 activations in one 135-task evaluation; Table 6) |
| Skills / extensibility | Behavior fixed in the prompt | Skills are mountable plugins. Our results use none, so the gains do not rely on hand-written knowledge. SQL skills (e.g., dialect handling or solving strategies) can be added without changing the agent loop or the tool interface |
| Environment & concurrency | Shared process; failures can leak across tasks | Per-task process, workspace, and read-only container; killable per-query deadlines (60 s); parallel workers (at most 8) for large-scale rollouts and RL |
| Generalization | A new benchmark requires a new loop | A new backend or benchmark is a new plugin. The same four-tool abstraction covers SQLite, PostgreSQL, and BigQuery, and the BIRD-Interact profile swaps in its nine operations (including user clarification) with no change to the agent |
The last row explains the OOD results. The model learns a protocol (explore → verify → submit) rather than a benchmark-specific prompt format, and this protocol transfers when the interface changes. The ablations in our response to W1 isolate the same effect: harness-native SFT reaches 30.4% vs. 9.6% for non-harness SFT (Table 3), and the specialized harness improves on the generic one by +7.5 points (Table 7).
Thank you for your clarification. These largely resolved my concerns. I decide to maintain my original score.
(1/2) Response by authors
Thank the reviewer for this point. We agree that MTSQL-R1 [1] also uses verified-trajectory SFT followed by interactive RL, and we do not claim this two-stage recipe as our contribution. The differences lie in what the agent is trained on and where:
We formalize both points in our response to Reviewer iWJV (W1, Q2).
Independent contribution of harness alignment. The ablations below change only the harness and keep everything else fixed (Qwen3-8B, S2-SQLite):
Non-harness SFT even falls below the untrained base model (15.5%), which shows that interactive training alone does not transfer when the training and deployment harnesses differ.
Direct comparison. The released MTSQL-R1 checkpoint scores 2.2% / 0.0% / 2.6% on S2-SQLite / BI-mini / LSB-SQLite (and 5.2% / 2.7% / 6.7% when evaluated directly under our DSH-SQL harness), versus 45.2% / 9.0% / 24.8% for HarnessSQL-8B (full table in our response to Reviewer Dsq9). This comparison also differs in data and model scale, so we rely on the controlled ablations above to isolate the effect of the harness.
[1] Guo, T., et al. MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training. ACL 2026.
We agree that turn-split keeps the full history prefix, and that with identical token weighting and batching its expected objective matches full-trajectory SFT. We will correct the "preserving interaction structure" wording in Section 5.3. To isolate the three confounders you raised, we ran a controlled ablation with Qwen3-8B. All four rows use the same fixed subset of 997 trajectories, 32K context, and identical hyperparameters. Spider 2.0-SQLite is averaged over 4 runs (mean ± std). BI-mini and LSB are single runs.
With weighting and budget controlled, the residual in-domain gap comes from batch composition: full-trajectory SFT updates all turns of an episode jointly, while turn-split shuffles turns across episodes. Full-trajectory SFT is also much cheaper. With the same 6.19M supervised tokens per epoch, turn-split needs 9.0× more input tokens and 9.7× more GPU-hours (91.7 vs. 9.4 on 8×H800), because it re-encodes the history prefix for every turn.
We will add this ablation and the revised explanation to Section 5.3.