- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RStar-SQL, which is an iterative search-and-training framework for text-to-SQL tasks. It uses MCTS to generate reasoning trajectories and evaluates candidate SQL using execution consistency and checks many SQL specific signals. The selected trajectories provide SFT data, and preferred/dispreferred trajectory pairs train a separate PPM that guides subsequent searches. The paper evaluates 7B and 8B Qwen models on several text-to-sql benchmarks and compares against many out-of-box LLMs, and prior studies.
Critical strengths
Unlike many RL and harness papers in the text-to-sql field, the paper propose an method to integrate SQL verification into iterative search and training and shows empirical gains on text-to-SQL tasks compared to the SFT baseline.
Critical weaknesses
-
Novelty contribution of this paper is limited. The contribution appears to be a SQL-specific supervision and integration recipe built on established MCTS and preference-learning methods. The paper appear to follow established Text-to-SQL search design, such as alpha-sql. In addition, Reasoning-SQL, Arctic-Text2SQL-R1, DualSQL, and ReEx-SQL also have similar design of composite SQL reward shaping. Several of the SQL-specific correctness constraints underlying the proposed signals are precedented in previous studies, such as DIN-SQL and Tool-SQL.
-
The current experiments do not sufficiently establish an advantage over the straightforward SFT + RL recipe with only terminal reward (e.g. Arctic-Text2SQL-R1), especially when the proposed SQL-specific verification signals could also serve as richer multi-step RL rewards.
Questions for authors
-
Can you elaborate more on which specific mechanism goes beyond Alpha-SQL’s structured MCTS? The methodological distinction and experimental controls are not sufficiently clear in this paper.
-
How does RStar-SQL compare with an SFT + RL baseline initialized from the same checkpoint and using the same SQL-specific reward signals? Does its advantage persist under comparable training resources and identical inference/search budgets?
-
It is not clear in the paper, even in the appendix, what the training setups are for the baseline methods. It is hard for the audience to build a mental model how different methods should be fairly compared.
-
From figure 4, I find it difficult to attribute the accuracy improvement entirely to the algorithm because the preference model learns from synthetic trajectory pairs, and subsequent rounds accumulate more training examples and perform additional training. A better comparison might be training the SFT baseline on exactly the same synthetic trajectories, and evaluate it using Alpha-SQL under the same inference budget.
-
Experiments on bird-interact and spider-2 could be some stronger signals to measure its capabilities on complex sql generation tasks.
6 comments
Response to Revewer 4rjj(2)
Q2. Comparable training resources and inference budgets
Thank you for highlighting the importance of resource-controlled comparisons. The RL controls start from the same SFT checkpoint, and the SQL-signal baseline incorporates the same verification signals as action-level rewards, rather than only as a terminal composite reward. RStar-SQL achieves 24.6% EX, compared with 23.6% for this baseline.
We have not collected complete end-to-end resource statistics for this comparison. Our separate inference-cost diagnostics do not provide comprehensive accounting of policy updates, SQL-PPM training, and search-time scoring across these methods. We therefore limit our conclusion to the observed 1.0-percentage-point accuracy gain under the reported initialization and feedback conditions, without claiming superior accuracy or efficiency at equal total compute.
Q3. Baseline training settings and reproducibility
We agree that baseline-specific training details are needed to assess fairness and reproduce the comparisons. Where applicable, our local implementations use common policy initialization, training data, shared optimization settings, and evaluation protocols. Method-specific differences concern the supervision objective, reward design, and search or preference-guidance mechanism.
The main protocol settings recorded in our experiments are summarized below. The filtering controls in Table R2-2 use the separately stated one-epoch, 2,048-token configuration.
Table R3-3. Main policy, search, and preference-model settings.
| Module | Hyperparameter | Setting |
|---|---|---|
| Policy SFT | Optimizer | AdamW |
| Policy SFT | Learning rate | 2 × 10⁻⁵ |
| Policy SFT | Training epochs | 3 |
| Policy SFT | Maximum sequence length | 4,096 |
| Policy SFT | Precision | BF16 |
| Policy SFT | Per-device batch size | 1 |
| Policy SFT | Gradient accumulation | 2 |
| Policy SFT | Effective global batch size | 16 on 8 GPUs |
| MCTS | Maximum search depth | 6 |
| MCTS | Rollouts | 8 |
| MCTS | UCT coefficient | 1.4 |
| MCTS | Random seed | 42 |
| MCTS | Concurrent workers | 64 |
| Generation | Reasoning temperature | 0.2 |
| Generation | SQL generation temperature | 0.8 |
| Generation | Candidates per SQL generation/revision action | 5 |
| Generation | SQL execution timeout | 60 s |
| SQL-PPM | Objective | Bradley–Terry ranking loss |
| SQL-PPM | Learning rate | 2 × 10⁻⁵ |
| SQL-PPM | Training epochs | 3 |
| SQL-PPM | Maximum sequence length | 2,048 |
| SQL-PPM | Per-device batch size | 8 |
| SQL-PPM | Maximum preference pairs per question | 2 |
| Inference | Engine | vLLM |
| Inference | Maximum context length | 8,192 |
Response to Revewer 4rjj(3)
Table R3-4. Reported RL baseline settings.
| Category | Hyperparameter | Setting |
|---|---|---|
| Optimization | RL algorithm | GRPO |
| Optimization | Optimizer | AdamW |
| Optimization | Learning rate | 1 × 10⁻⁶ |
| Optimization | Weight decay | 0.01 |
| Optimization | Training steps | 100 |
| Optimization | Batch size | 8 or 16 |
| Optimization | Rollout group size | 8 |
| Optimization | Trajectories per update | 64 or 128 |
| Optimization | Policy update epochs | 1 |
| Optimization | PPO/GRPO clip range | 0.2 |
| Optimization | KL coefficient | 0.001 |
| Optimization | Gradient clipping | 1.0 |
| Optimization | Precision | BF16 |
| Optimization | Maximum sequence length | 4,096 |
| Generation | Sampling temperature | 0.8 |
| Generation | Top-p | 1.0 |
| Generation | Maximum response length | 2,048 |
| Reward | Final correctness | SQL execution correctness |
| Reward | Additional SQL signals | Weighted semantic verification as action-level rewards |
| Hardware | GPUs | 8 |
| Training | Framework | DeepSpeed ZeRO-3 + vLLM |
We will make the baseline-specific configurations, including the per-run batch-size and trajectory-count assignments, explicit in the revised appendix.
Q4. SFT trained on identical synthetic trajectories and evaluated with Alpha-SQL
We include the requested data control: the SFT baseline uses the same training trajectories and positive-filtering rules, followed by Alpha-SQL search.
Table R3-5. Same-trajectory training control on LogicCat. EX (%).
| Method | Inference | EX (%) |
|---|---|---|
| SFT trained on the same trajectories | Alpha-SQL search | 21.5 |
| RStar-SQL | Full RStar-SQL search | 25.3 |
The 3.8-percentage-point improvement indicates that access to the synthetic trajectories alone does not explain the observed difference. This comparison controls the data factor; it does not by itself control total optimization or inference cost. We will specify the evolution rounds included in the trajectory set and report the associated training and search workload.
Respone to Reviewer 4rjj(1)
Thank you for the comments on methodological distinction, alternative training objectives, and reproducibility. We address the weaknesses and specific questions below.
W1. Novelty relative to existing SQL search and reward designs
We acknowledge the precedents for structured MCTS, SQL constraints, composite rewards, and preference learning. Our contribution focuses on how SQL verification evidence is converted into trajectory preferences and reused to guide subsequent search.
Specifically, execution consistency is augmented with schema and value grounding, execution behavior, join validity, and type–operator compatibility. The resulting reward is propagated through MCTS, and the mean and minimum action values along a trajectory support trajectory selection and preference construction. SQL-PPM then learns to score a reasoning prefix with a candidate action before terminal verification is available.
The component comparison below separates the base-policy and SFT-policy settings and examines verification and preference guidance within those settings. “SQL verification” denotes execution consistency augmented with the four verification signals.
Table R3-1. Component comparison on LogicCat. EX (%).
These results support contributions from verification and learned search guidance. We will clarify the scope of novelty without treating established search operations or individual SQL checks as unique to this work.
W2. Comparison with SFT + RL using the same SQL signals
We include RL controls initialized from the same SFT checkpoint. One uses final SQL execution correctness; the other combines final correctness with the same four SQL verification signals.
Table R3-2. Training-method comparison from the same SFT initialization on LogicCat. EX (%).
Adding SQL signals improves the RL control by 1.1 percentage points. RStar-SQL obtains a further 1.0-point gain over this control and a 2.1-point gain over final-reward RL. This provides evidence for our use of the verification information in this experimental setting; we do not interpret it as a general advantage over all RL recipes.
The final-reward control is not a full reproduction of Arctic-Text2SQL-R1. In the SQL-signal control, the verification signals are incorporated as action-level rewards during RL training.
Q1. Mechanisms beyond Alpha-SQL’s structured MCTS
The distinction has three parts:
We will also sharpen the related-work discussion. Reasoning-SQL uses SQL-tailored partial rewards for GRPO; Arctic-Text2SQL-R1 emphasizes lightweight execution rewards; and ReEx-SQL combines execution-aware training with tree-based decoding. Our distinction concerns preference construction and learned search guidance, rather than a broad claim that existing work lacks execution feedback or tree search.