- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SIPR, a framework that trains small language models to reason using multi-round parallel branches. It first uses supervised fine-tuning to teach the model how to plan distinct strategies, execute them in isolation, and synthesize findings across rounds, then applies outcome-based reinforcement learning so the model autonomously improves its exploration process. Experiments on Qwen3-4B show SFT substantially outperforms the base model, while RL further improves over SFT across seven math benchmarks with an average accuracy gain of 2.31 percentage points.
Critical strengths
Unlike prior work that merely increases branch count, SIPR structures multi-round plan-isolate-synthesize cycles. This teaches the model not only to solve problems but to autonomously plan diverse branches, cross-verify findings, and self-correct—internalizing "how to explore" as a policy optimizable through the RL loop.
Critical weaknesses
-
Missing controlled cross-paradigm baselines. The evaluation relies solely on intra-paradigm comparisons (SIPR-SFT vs. SIPR-RL).Without standard SFT on the same data or RL (e.g., GRPO) on the same DAPO-Math-17k pool, it remains unclear whether the reported gains stem from SIPR’s specific exploration-organization design or simply from applying RL to a well-formatted parallel initialization.
-
Limited model scale. Experiments are restricted to 4B models (Qwen3/3.5-4B), leaving it unclear whether SIPR generalizes to larger models with stronger baseline reasoning.
-
Lack of quantitative analysis on dynamic branching behavior. While SIPR allows the model to autonomously determine the number of branches per round, the paper does not provide corpus-level statistics on this behavior (e.g., the distribution of branch counts across rounds, or how the average number of branches correlates with problem difficulty). Without this data, it remains unclear whether the model genuinely learns to dynamically scale its exploration budget or simply converges to a fixed heuristic (e.g., always opening 2 branches) after RL.
Questions for authors
-
Could the authors provide statistics on the dynamic branching behavior? It would be highly informative to see the distribution of branch counts across different rounds and benchmarks.
-
Please provide a detailed analysis of the inference overhead introduced by SIPR. Since multi-round parallel exploration inherently consumes more compute than standard serial reasoning, it would be valuable to quantify this cost.
2 comments
Response to Reviewer twpQ (2/2)
Q3 (Weakness 3, Question 1). Could the authors provide statistics on the dynamic branching behavior, such as the distribution of branch counts across rounds and benchmarks? Does the model scale its exploration dynamically or converge to a fixed heuristic?
A3. The model varies its branch count within a solution and across solutions, and RL shifts branch allocation toward wider openings and narrower later rounds.
| Share of planning rounds on AMC/AIME | SIPR-SFT | SIPR-RL |
|---|---|---|
| First rounds opening at least six branches | 24.6% | 47.6% |
| Rounds from the fourth onward using one branch | 68.6% | 77.4% |
Table R8. Mean planned branches per completed planning round, SFT → RL, from 3,600 saved 4B responses.
| Benchmark | Round 1 | Round 2 | Round 3 | Round 4+ |
|---|---|---|---|---|
| MATH-500 | 4.82 → 5.28 | 4.45 → 4.48 | 3.74 → 3.51 | 1.63 → 1.49 |
| AMC23 | 4.84 → 5.28 | 4.39 → 4.36 | 3.61 → 3.28 | 1.56 → 1.35 |
| AIME24 | 4.68 → 5.31 | 4.04 → 4.20 | 3.32 → 3.06 | 1.47 → 1.41 |
| AIME25 | 4.60 → 5.12 | 4.04 → 4.10 | 3.18 → 3.03 | 1.82 → 1.37 |
Each checkpoint contributes 1,800 saved responses. Later-round means are conditional on reaching that round, and Round 4+ pools subsequent rounds.
- Across rounds and benchmarks. RL increases mean first-round width and reduces mean width from the third round onward on all four benchmarks.
- The model's own choice. Both policies run under the same non-increasing-width rule, which prevents a round from opening more branches than the preceding one; within it, the opening width and how quickly later rounds narrow are the model's choices.
- Effect on accuracy. Adaptive width also improves correct completion: at 16k sampled tokens, 13.61% of attempts finish correctly when the model chooses the width, versus 10.69% with four fixed branches (CQ2, Table R2).
Q4 (Question 2). Please provide a detailed analysis of the inference overhead that SIPR introduces relative to standard serial reasoning.
A4. SIPR samples 8.3% more tokens than serial reasoning and returns answers sooner (CQ4, Tables R4 and R5).
- Four A100 GPUs. Serving one question per engine, SIPR finishes the 360-attempt workload 15.6% sooner (7,867 s vs. 9,326 s), generates at 563 tokens/s versus 438 (+28.5%), and returns a correct answer within 60 seconds on 10.00% of attempts, compared with 5.00% for serial generation.
- Two RTX 4090 GPUs. SIPR raises sampled throughput at 4, 8 and 16 concurrent slots per GPU, to 1.30×, 1.34× and 1.29× that of the same-data serial RL model.
- Branch scheduling in isolation. A 30-pair test isolates branch scheduling on the same SIPR checkpoint. With identical attention visibility and token positions, throughput rises from 109.05 tokens/s with branches executed one at a time to 160.11 tokens/s with concurrent branches (1.47×).
Response to Reviewer twpQ (1/2)
Thank you for your careful review. CQ1–CQ4 refer to the General response; Tables R7 and R8 appear in this reply.
A1. The gains stem from the exploration-organization design. We trained the requested baseline: a serial model with standard SFT on the same data and 60 GRPO updates on the same DAPO-Math problems, with the same batch size, learning rate and KL coefficient as SIPR-RL (CQ1, Table R1).
A2. Yes. The same 10,000 demonstrations generated by the 4B teacher also train Qwen3-8B-Base without regenerating any data: seven-benchmark accuracy rises from 14.79% for Base to 41.78% after SFT and 42.80% after RL (Table R7).
Table R7. Qwen3-8B accuracy (%). Parentheses give SIPR-RL minus SIPR-SFT in percentage points.
SIPR-SFT is the model after 625 SFT updates and SIPR-RL after 60 further RL updates. Bold marks the higher trained-stage score, including ties. Evaluation settings follow Appendix C.2 of the revised manuscript.