acceptodds
Under review as a conference paper at ICLR 2027

Learning to Explore for Self-Improving Parallel Reasoning

Abstract

A self-improving reasoner must generate not only answers, but also experience from which it can learn. Native parallel reasoning offers multiple solution routes; turning those routes into useful exploration requires learning what to investigate and how to reuse the findings. We introduce Self-Improving Parallel Reasoning (), a training framework that connects strategy-guided exploration to autonomous reasoning and outcome-based learning. Mathematical strategies first guide the construction of multi-round demonstrations. Branches develop concrete plans from completed shared history, while current-round findings become available only after synthesis. Successful demonstrations retain intermediate candidates, including incorrect ones, so later reasoning can compare, check and revise earlier work. Supervised fine-tuning learns the visible plans, branch steps and takeaways without the private strategy directives. Reinforcement learning then updates these actions jointly from reference-scored, self-generated trajectories, using their recorded execution contexts. The updated policy generates the next batch of experience. We instantiate in two 4B model families. In the reported Qwen3-4B evaluation, ordinary RL improves over SFT on all seven mathematical benchmarks, averaging 2.31 percentage points. On 800 paired AMC/AIME responses per policy, literal within-round plan repetition falls from 29.63% to 23.38%. These results connect learning from outcomes with learning to organize the exploration that produces them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Recursive Self-Improvement, Parallel Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_twpQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SIPR, a framework that trains small language models to reason using multi-round parallel branches. It first uses supervised fine-tuning to teach the model how to plan distinct strategies, execute them in isolation, and synthesize findings across rounds, then applies outcome-based reinforcement learning so the model autonomously improves its exploration process. Experiments on Qwen3-4B show SFT substantially outperforms the base model, while RL further improves over SFT across seven math benchmarks with an average accuracy gain of 2.31 percentage points.

Critical strengths

Unlike prior work that merely increases branch count, SIPR structures multi-round plan-isolate-synthesize cycles. This teaches the model not only to solve problems but to autonomously plan diverse branches, cross-verify findings, and self-correct—internalizing "how to explore" as a policy optimizable through the RL loop.

Critical weaknesses
  1. Missing controlled cross-paradigm baselines. The evaluation relies solely on intra-paradigm comparisons (SIPR-SFT vs. SIPR-RL).Without standard SFT on the same data or RL (e.g., GRPO) on the same DAPO-Math-17k pool, it remains unclear whether the reported gains stem from SIPR’s specific exploration-organization design or simply from applying RL to a well-formatted parallel initialization.

  2. Limited model scale. Experiments are restricted to 4B models (Qwen3/3.5-4B), leaving it unclear whether SIPR generalizes to larger models with stronger baseline reasoning.

  3. Lack of quantitative analysis on dynamic branching behavior. While SIPR allows the model to autonomously determine the number of branches per round, the paper does not provide corpus-level statistics on this behavior (e.g., the distribution of branch counts across rounds, or how the average number of branches correlates with problem difficulty). Without this data, it remains unclear whether the model genuinely learns to dynamically scale its exploration budget or simply converges to a fixed heuristic (e.g., always opening 2 branches) after RL.

Questions for authors
  1. Could the authors provide statistics on the dynamic branching behavior? It would be highly informative to see the distribution of branch counts across different rounds and benchmarks.

  2. Please provide a detailed analysis of the inference overhead introduced by SIPR. Since multi-round parallel exploration inherently consumes more compute than standard serial reasoning, it would be valuable to quantify this cost.

2 comments
AuthorsOpenReview ↗

Response to Reviewer twpQ (1/2)

Thank you for your careful review. CQ1–CQ4 refer to the General response; Tables R7 and R8 appear in this reply.

Q1 (Weakness 1). Without standard SFT on the same data or RL (e.g., GRPO) on the same DAPO-Math-17k pool, do the gains stem from SIPR's exploration-organization design or from applying RL to a well-formatted parallel initialization?

A1. The gains stem from the exploration-organization design. We trained the requested baseline: a serial model with standard SFT on the same data and 60 GRPO updates on the same DAPO-Math problems, with the same batch size, learning rate and KL coefficient as SIPR-RL (CQ1, Table R1).

  • After SFT on the same data, the serial model averages 36.31% on the seven math benchmarks and SIPR 38.78%.
  • After RL on the same pool, the serial model averages 36.79% and SIPR 41.09% (+4.30 points). SIPR-RL is higher on all seven benchmarks and on GPQA (+9.09).
  • RL adds more to SIPR: 2.31 points, against 0.48 for the serial model.

Q2 (Weakness 2). Experiments are restricted to 4B models. Does SIPR generalize to larger models with stronger baseline reasoning?

A2. Yes. The same 10,000 demonstrations generated by the 4B teacher also train Qwen3-8B-Base without regenerating any data: seven-benchmark accuracy rises from 14.79% for Base to 41.78% after SFT and 42.80% after RL (Table R7).

Table R7. Qwen3-8B accuracy (%). Parentheses give SIPR-RL minus SIPR-SFT in percentage points.

StageMATH-500 (Mean@1)AMC23 (Mean@8)AIME24 (Mean@8)AIME25 (Mean@8)HMMT (Mean@8)Minerva (Mean@1)Olympiad (Mean@1)Avg. Math
Base43.2021.252.500.830.0019.1216.6214.79
SIPR-SFT81.4058.4420.4218.757.5056.9948.9641.78
SIPR-RL83.60 (+2.20)59.69 (+1.25)21.67 (+1.25)18.75 (+0.00)7.08 (−0.42)58.09 (+1.10)50.74 (+1.78)42.80 (+1.02)

SIPR-SFT is the model after 625 SFT updates and SIPR-RL after 60 further RL updates. Bold marks the higher trained-stage score, including ties. Evaluation settings follow Appendix C.2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer twpQ (2/2)

Q3 (Weakness 3, Question 1). Could the authors provide statistics on the dynamic branching behavior, such as the distribution of branch counts across rounds and benchmarks? Does the model scale its exploration dynamically or converge to a fixed heuristic?

A3. The model varies its branch count within a solution and across solutions, and RL shifts branch allocation toward wider openings and narrower later rounds.

Share of planning rounds on AMC/AIMESIPR-SFTSIPR-RL
First rounds opening at least six branches24.6%47.6%
Rounds from the fourth onward using one branch68.6%77.4%

Table R8. Mean planned branches per completed planning round, SFT → RL, from 3,600 saved 4B responses.

BenchmarkRound 1Round 2Round 3Round 4+
MATH-5004.82 → 5.284.45 → 4.483.74 → 3.511.63 → 1.49
AMC234.84 → 5.284.39 → 4.363.61 → 3.281.56 → 1.35
AIME244.68 → 5.314.04 → 4.203.32 → 3.061.47 → 1.41
AIME254.60 → 5.124.04 → 4.103.18 → 3.031.82 → 1.37

Each checkpoint contributes 1,800 saved responses. Later-round means are conditional on reaching that round, and Round 4+ pools subsequent rounds.

  • Across rounds and benchmarks. RL increases mean first-round width and reduces mean width from the third round onward on all four benchmarks.
  • The model's own choice. Both policies run under the same non-increasing-width rule, which prevents a round from opening more branches than the preceding one; within it, the opening width and how quickly later rounds narrow are the model's choices.
  • Effect on accuracy. Adaptive width also improves correct completion: at 16k sampled tokens, 13.61% of attempts finish correctly when the model chooses the width, versus 10.69% with four fixed branches (CQ2, Table R2).

Q4 (Question 2). Please provide a detailed analysis of the inference overhead that SIPR introduces relative to standard serial reasoning.

A4. SIPR samples 8.3% more tokens than serial reasoning and returns answers sooner (CQ4, Tables R4 and R5).

  • Four A100 GPUs. Serving one question per engine, SIPR finishes the 360-attempt workload 15.6% sooner (7,867 s vs. 9,326 s), generates at 563 tokens/s versus 438 (+28.5%), and returns a correct answer within 60 seconds on 10.00% of attempts, compared with 5.00% for serial generation.
  • Two RTX 4090 GPUs. SIPR raises sampled throughput at 4, 8 and 16 concurrent slots per GPU, to 1.30×, 1.34× and 1.29× that of the same-data serial RL model.
  • Branch scheduling in isolation. A 30-pair test isolates branch scheduling on the same SIPR checkpoint. With identical attention visibility and token positions, throughput rises from 109.05 tokens/s with branches executed one at a time to 160.11 tokens/s with concurrent branches (1.47×).
Reviewer_nCuDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method for parallel reasoning of models coordinated in multiple rounds. In each round the model proposes several plans, works on them, comes up with a takeaway and goes to the next round. The paper in particular focuses on math domain and creates an initial dataset of training where plans are inspired by a predefined set of 49 math strategies for solving problems. Authors compare SFT training vs. SFT + RL training and report that the SFT+RL outperforms the SFT training by 2.3%.

Critical strengths

The problem that the work studies is indeed important. It's important how we can scale compute of the models in parallel while still keeping enough diversity to allow us boost the performance of the models. The authors try to address this by a system where the model comes up with different plans and continues them in parallel, and repeats this process several rounds to get to the final answer.

Critical weaknesses

My main concern regarding the paper is the lack of any meaningful baseline. See below.

  • The most important weakness is that the paper does not compare the performance of the model with any other parallel or sequential compute scaling method. E.g., how much the method is doing better than majority voting and best-of-N methods. Or how the efficiency, compute, and token count compares.
  • Importantly, the paper does not compare the performance of their trained models with the performance of a one-shot (standard) model trained on the same datasets. I.e., it's not clear how much boost this method is giving.
  • Further the paper does not provide ablation experiments. There are many design choices in the framework (e.g., keeping wrong intermediate answers), but the paper does not provide ablation results for any of the design choices.
  • In my opinion the paper is not written well, and is very hard to read. It also uses some custom terms from recent work without defining or explaining them properly.
Questions for authors
  • Q1. Can you please explain again how the SFT dataset is constructed?
  • Q2. Can you confirm my understanding that you're using base Qwen models for training and evaluation and the Qwen-instruct model for creating the SFT data. Can we deduct that your SFT training is hence partly distilling the instruct model into the base model?
  • Q3. What is completion-trained RL?
  • Q4. Any additional experiments info on the baselines and ablations above would be appreciated.
1 comment

Thank you for your careful review. CQ1–CQ4 refer to the General response, which holds the shared tables.

Q1 (Weaknesses 1 and 2). How does the method compare with other parallel or sequential compute-scaling methods such as majority voting and best-of-N, including efficiency, compute and token count, and with a standard model trained on the same datasets?

A1. SIPR is ahead in all three comparisons.

  • A standard model trained on the same datasets. A serial model with the same demonstrations, RL problems and number of updates averages 36.79% on the seven math benchmarks, against 41.09% for SIPR-RL (+4.30 points), and is lower on every benchmark and on GPQA (+9.09; CQ1, Table R1).
  • Majority voting and best-of-N. At an equal budget of 16k sampled tokens, SIPR finishes 13.06% of attempts correctly, 3.4 times the 3.89% of majority voting, shortest-answer selection or a fixed verifier over four serial answers (CQ3, Table R3).
  • Efficiency, compute and token count. SIPR samples 8.3% more tokens than the serial model, finishes the workload 15.6% sooner at 28.5% higher throughput, and returns twice as many correct answers within 60 seconds (10.00% vs. 5.00%; CQ4, Tables R4 and R5).

Q2 (Weakness 3, Question 4). Please provide ablation results for the design choices, e.g., keeping wrong intermediate answers.

A2. Each design choice improves correct completion when tested alone on the same RL checkpoint.

  • Three inference-time ablations. Within 16k sampled tokens, letting later rounds read the branch calculations in addition to their summaries adds 3.47 points, letting the model choose its branch count adds 2.92 over a fixed width of four, and letting it branch again adds 1.94 over serial continuation after the first merge (CQ2, Table R2).
  • Keeping wrong intermediate answers. In a matched continuation run that differs only in whether incorrect intermediate candidates are kept, the effect of keeping them on accuracy is 1.61 percentage points higher on prompts containing both a correct and an incorrect candidate than on prompts containing the correct candidate alone (31 held-out problems, 2,976 responses per model, revised Appendix D.4).

Q3 (Questions 1 and 2). How is the SFT dataset constructed? Since the Qwen-instruct model creates the SFT data for the base model, is SFT training partly distilling the instruct model into the base model?

A3. Yes. The SFT stage is a distillation: the teacher is Qwen3-4B-Instruct-2507 and the student is Qwen3-4B-Base.

Distillation alone does not account for SIPR's accuracy: given the same teacher calculations and RL problems, the serial baseline averages 4.30 points lower after RL (Table R1); the two models differ in how the work is organized and executed. RL on the student's own solutions then adds 2.31 points to SIPR and 0.48 to the serial model.

The dataset is constructed in four steps (Sections 2–3 and Appendix B):

  1. Select strategies. A selector draws on 49 mathematical strategies and merges proposals that use the same transformation.
  2. Develop routes. The teacher develops these routes into plans, calculations and summaries.
  3. Filter. We retain 10,000 Polaris traces with correct final answers, including intermediate candidates for later checking.
  4. Train the student. The student sees the problem, plans, calculations and summaries; the strategy directives are used only to generate demonstrations.

Q4 (Weakness 4). The paper uses custom terms from recent work without defining or explaining them.

A4. The revised Section 2.1 opens with these definitions.

TermMeaning
PlanStates a branch's next task
StepPerforms that task. Steps read completed earlier rounds and their own plans, with current sibling step bodies hidden
TakeawaySummarizes findings and unresolved work. Synthesis reads the completed branches
Execution-context scoringRecomputes each action's probability using the visible tokens and positions it had during generation

Q5 (Question 3). What is completion-trained RL?

A5. Completion-trained RL replaces invalid attempts until each prompt has eight valid trajectories. Valid incorrect answers receive negative feedback. The settings are in Appendix A.2.

SettingSIPR-RLCompletion-trained RL
Trajectories per promptThe first four attempts, including invalid onesEight valid trajectories
Prompts per update1632
Updates6030

The completion-trained checkpoint solves 95% of AMC23 problems within eight attempts, up from 80% after SFT, and its coverage rises on all four contest benchmarks (submitted Table 2).

Reviewer_C8N2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Self-Improving Parallel Reasoning (SIPR), a framework for training models to plan, execute, and synthesize multiple reasoning branches across rounds. Strategy-guided demonstrations provide the initialization data, retaining incorrect intermediate candidates within trajectories that ultimately reach correct answers. Supervised fine-tuning learns the visible reasoning process without the external strategy directives. Subsequent reinforcement learning uses reference-answer rewards to jointly update planning, branch execution, and synthesis while preserving each action’s execution context.

Critical strengths
  • The explicit treatment of branch visibility and execution-context scoring addresses an important consistency issue when training on serialized parallel reasoning trajectories.
  • Paired uncertainty estimates and inspectable reasoning examples provide useful context for assessing the reported improvements.
Critical weaknesses

1. Limited methodological novelty relative to existing RL-based parallel reasoning frameworks.

The proposed direction substantially overlaps with several recent works that have already established reinforcement learning as an effective mechanism for parallel reasoning and exploration.

  • Adaptive Parallel Reasoning (APR, COLM 2025) already introduces end-to-end RL for adaptive parallel reasoning through spawn() and join() operations. It achieves 83.4% vs. 60.0% accuracy under a 4K context window and 75.2% vs. 57.3% accuracy at approximately 5 seconds of inference latency.
  • Parallel-R1 (2025) explicitly trains parallel exploration behaviors through a progressive SFT-to-RL curriculum. On Qwen3-4B, it improves the average benchmark score from 45.1 to 48.9, corresponding to an 8.4% relative improvement over standard RL.
  • Native Parallel Reasoner (NPR, ICML 2026) already combines teacher-free self-distillation with Parallel-Aware Policy Optimization, reporting performance improvements of up to 24.5% and inference speedups of up to 4.6× across eight reasoning benchmarks.
  • OPE (EMNLP 2026 Findings) explicitly addresses information redundancy among parallel reasoning paths through outline-guided exploration and iterative RL.

These studies have already explored several central ingredients of learning parallel reasoning, including adaptive branching, RL-based exploration, self-improvement, and exploration diversity. Therefore, the general idea of improving reasoning through learned parallel exploration is not sufficient to establish methodological novelty. The contribution needs to be distinguished at the level of the learning objective, exploration mechanism, or empirical capabilities that cannot be obtained using existing approaches.

2. A fundamental attribution problem: learned exploration versus increased inference-time computation.

A central concern in evaluating learned parallel reasoning is whether performance improvements reflect genuinely better exploration policies or merely increased inference-time computation.

This distinction is empirically important. In APR (COLM 2025), RL improves accuracy from 75.5% to 83.4% (+7.9 percentage points), while increasing total generated tokens by 52.5%. More importantly, when the number of child threads is constrained to the maximum of 10 both before and after RL, accuracy changes only from 83.2% to 83.3% (+0.1 percentage points). The authors explicitly conclude that the main RL benefit in this setting comes from increased test-time computation rather than better decision-making under a comparable computation budget.

This result demonstrates that seemingly substantial gains from learning parallel reasoning can be largely attributable to computation allocation. Consequently, accuracy improvements alone cannot establish that a proposed method learns a more effective exploration strategy.

A convincing evaluation must disentangle these factors through comparisons at matched total generated tokens, branch counts, training budgets, and inference latency. Without such evidence, the independent contribution of the learned exploration mechanism remains difficult to assess.

3. Encouraging parallel exploration does not necessarily improve reasoning quality.

Another fundamental concern is whether the learned exploration behavior is actually useful for solving problems, rather than merely increasing the frequency or diversity of parallel reasoning trajectories.

Parallel-R1 (2025, Table 3) provides direct empirical evidence of this distinction. Changing the reward from accuracy-oriented to parallelism-oriented increases the parallel reasoning ratio from 13.6% to 80.3%, but simultaneously reduces performance on multiple benchmarks:

  • AMC23: 69.7% → 59.4% (−10.3 percentage points).
  • AIME24: 18.3% → 15.2% (−3.1 percentage points).
  • MATH: 82.6% → 81.7% (−0.9 percentage points).

Importantly, these results do not imply that parallel exploration is inherently harmful. Rather, they demonstrate that optimizing parallel behavior and improving task accuracy are distinct objectives that can conflict.

Therefore, measurements such as the number of branches, exploration frequency, or trajectory diversity do not, by themselves, establish improvements in effective exploration. Stronger evidence would characterize the number of distinct successful solutions discovered, the redundancy across branches, and the resulting accuracy under fixed computation budgets. This distinction is particularly important for substantiating claims about learning to explore rather than simply learning to produce parallel reasoning structures.

4. Insufficient controlled comparisons with simpler parallel strategies.

The reported experiments do not compare SIPR against parallel sampling with shortest-completion selection, first-finished selection, or majority voting under matched budgets. The central comparison is SIPR-SFT versus SIPR-RL, yielding average mathematics gains of 2.31 percentage points on Qwen3-4B and 1.05 points on Qwen3.5-4B. These results demonstrate improvements from additional RL within the proposed framework, but do not establish whether its multi-round planning and synthesis offer advantages over simpler parallel sampling and selection. Such baselines are needed to justify the additional methodological complexity.

Questions for authors
  • Is the improvement caused by learned exploration or simply increased search compute?
  • What is fundamentally new beyond APR, Parallel-R1, NPR, and OPE?
  • Does self-improvement genuinely expand the model's reasoning capability?
3 comments
AuthorsOpenReview ↗

Response to Reviewer C8N2 (1/3)

Thank you for your careful review. CQ1–CQ4 refer to the General response; Tables R6 and R9 appear in this reply.

Q1 (Weakness 1, Question 2). What is fundamentally new beyond APR, Parallel-R1, NPR, and OPE, at the level of the learning objective, exploration mechanism, or empirical capabilities?

A1. SIPR trains a Base model to organize the exploration it will learn from. It differs from the four systems at each of the three levels named in the review.

  1. Learning objective. SIPR's contribution is the training target: a Base model learns incremental branch work whose calculations and candidate answers remain available to later rounds, within the established native execution format. After SFT the student writes its own plans. Feedback on the final answers in its own trajectories jointly updates planning, branch steps and synthesis for the next batch: a trajectory receives +1 for a valid, normally terminated response with a correct final answer and −1 otherwise, and each action's probability is recomputed using the visible tokens and positions it had during generation.
  2. Exploration mechanism. Parallelism is native and adaptive from demonstration to inference. A selector draws on 49 mathematical strategies and merges proposals that use the same transformation. Each of the 10,000 strategy-guided demonstrations advances every branch one step at a time and keeps the calculations and candidate answers, so a later round can check a candidate or continue a derivation. At inference the model chooses how many branches each round opens and when to synthesize and stop.
  3. Empirical capabilities. This reuse has a measurable effect: on the same RL checkpoint, letting later rounds read those calculations alongside their summaries raises correct completion within 16k sampled tokens from 10.00% to 13.47% (Table R2). On seven math benchmarks, SIPR-RL averages 11.97 points above the released Parallel-R1 S2 checkpoint (Table R6) and 4.30 points above a serial model trained on the same data, which it also leads on GPQA Diamond (Table R1).
AspectPrior systemsSIPR
Starting pointParallel-R1: parallel demonstrations from about 7,000 GSM8K problems. NPR: post-trained models initialized by self-distillationA Base model and 10,000 demonstrations guided by 49 mathematical strategies
Branch workAPR: threads coordinated by message passing. Parallel-R1: repeated exploration and summarization. OPE: outlines and complete pathsOne step per branch per round; calculations and candidate answers stay readable in later rounds
Training signalOPE: separate objectives for outlines and pathsOne outcome reward jointly updates planning, branch steps and synthesis
TasksAPR: CountdownSeven math benchmarks and GPQA Diamond

Comparison with Parallel-R1. We run the released Parallel-R1 S2 checkpoint with its authors' code. SIPR-RL averages 41.09% across the seven benchmarks, 11.97 points above the 29.12% of Parallel-R1 S2 at its extended output limit, with margins of 24.40 points on MATH-500, 41.37 on Minerva and 16.62 on OlympiadBench (Table R6).

Table R6. Qwen3-4B accuracy (%). Parentheses give score differences from Parallel-R1 S2 (32k), in percentage points.

MethodMATH-500 (Mean@1)AMC23 (Mean@8)AIME24 (Mean@8)AIME25 (Mean@8)HMMT (Mean@8)Minerva (Mean@1)Olympiad (Mean@1)Avg. Math
Parallel-R1 S2 (published)84.5067.5016.3019.00————
Parallel-R1 S2 (3k)57.4065.6315.4213.755.0011.7630.8628.55
Parallel-R1 S2 (32k)57.8066.2516.2515.005.0011.7631.7529.12
SIPR-SFT79.80 (+22.00)55.94 (−10.31)15.83 (−0.42)13.75 (−1.25)7.92 (+2.92)51.47 (+39.71)46.74 (+14.99)38.78 (+9.66)
SIPR-RL82.20 (+24.40)58.13 (−8.12)18.75 (+2.50)18.33 (+3.33)8.75 (+3.75)53.13 (+41.37)48.37 (+16.62)41.09 (+11.97)

The 3k row uses the original 3,000-token output limit. The 32k row retries the 119 capped answers once with a 32,768-token context and 30,768-token output cap, retaining completed answers. The reproduced and SIPR rows share one setting: eight answers per contest question and all 500 MATH-500 questions. The published row uses 16 answers per contest question and a filtered MATH subset. Avg. Math weights the seven benchmarks equally. HMMT denotes February 2025. Bold compares SIPR stages.

AuthorsOpenReview ↗

Response to Reviewer C8N2 (2/3)

Q2 (Weakness 2, Question 1). Is the improvement caused by learned exploration or simply increased search compute? The comparison should match total generated tokens and branch counts.

A2. SIPR's advantage holds when compute is matched, both against other methods and between its own SFT and RL stages.

(a) Against other methods: equal tokens, equal time.

Matched quantityOther methodSIPR-RL
16k sampled tokens: four sampled answers with vote, shortest or verifier selection3.89%13.06%
16k sampled tokens: one long serial answer12.64%13.06%
24k sampled tokens: one long serial answer12.78%14.17%
Gain from 8,000 further tokens (16k → 24k): one long serial answer+0.14+1.11
60-second deadline: serial generation5.00%10.00%
Correct answers per 1,000 GPU-seconds at 4 / 8 / 16 slots: serial RL10.09 / 17.51 / 25.1515.90 / 25.77 / 30.86
  • Equal tokens. SIPR is ahead at the same sampled-token budget, with every branch counted toward it (CQ3, Table R3).
  • Tokens and time actually used. On the seven math benchmarks SIPR-RL samples 13.5k tokens per response, all branches included, against 12.1k for the same-data serial model (11% more), and scores 41.09% against 36.79% (Table R1). On the same 360 contest attempts it samples 8.3% more tokens and finishes 15.6% sooner; it returns more correct answers per GPU-second at every load (CQ4, Tables R4 and R5).
  • More tokens do not close the gap. One long serial answer gains 0.14 points when its budget grows by 50%, from 16k to 24k tokens, where SIPR gains 1.11. Raising Parallel-R1's output limit from 3k to 32k tokens moves its average from 28.55% to 29.12%, against 41.09% for SIPR-RL (Table R6).

(b) Between SFT and RL: equal branches, fewer tokens. The APR result cited in the review shows an RL gain of 0.1 points once the number of child threads is constrained to a maximum of ten. We apply an analogous control: the SFT and RL policies each receive the same allowance of at most ten branch starts per attempt, counted across all rounds (90 contest problems, 720 attempts per policy).

Table R9. SIPR-SFT and SIPR-RL under the same allowance of ten branch starts per attempt. The first three rows give the compute each policy uses; the last three give correct completion (%) within each sampled-token budget.

MeasureSIPR-SFTSIPR-RLRL − SFT
Branch starts per attempt (mean)9.369.34−0.02
Rounds per attempt (mean)2.622.43−0.19
Sampled tokens per attempt (mean)16,03215,525−3.2%
Correct completion within 8k tokens6.538.61+2.08
Correct completion within 16k tokens7.7810.83+3.05
Correct completion within 24k tokens8.0610.97+2.91
  • Compute is matched. Both policies use the allowance almost fully (76.9% and 78.2% of attempts reach ten branch starts), and the RL policy samples 3.2% fewer tokens in fewer rounds. In the APR result cited in the review, the RL gain came with 52.5% more generated tokens.
  • RL is ahead along the whole token curve. Tripling the token budget does not close the gap: SFT at 24k (8.06%) remains below RL at 8k (8.61%).
  • Same question, same branch count. Among attempts that use at most 16k tokens, pairing SFT and RL attempts on the same question with the same number of branch starts (326 attempts per policy, 87 questions) gives RL a correct-completion rate 4.93 points higher.
  • Branches at a fixed token budget. With tokens held at 16k, letting the RL policy choose its width raises correct completion from 10.69% with four fixed branches per round to 13.61% (Table R2).
  • Matched training budget. With the same data and the same number of updates, RL adds 2.31 points to SIPR and 0.48 to the serial model (CQ1).
AuthorsOpenReview ↗

Response to Reviewer C8N2 (3/3)

Q3 (Weakness 4). How does SIPR compare with simpler parallel strategies under matched budgets: parallel sampling with shortest-completion, first-finished or majority-vote selection?

A3. SIPR is ahead of each of these strategies at matched tokens and at matched latency (first table of A2; CQ3, Table R3).

  • Matched tokens. At 16k sampled tokens, four short serial answers reach 3.89% whether selected by vote, shortest completion or a fixed verifier, with all four candidates counted toward the budget. SIPR finishes 13.06% of attempts correctly.
  • Matched latency. Under a 60-second deadline on four A100s, voting, shortest-answer or first-finished selection reaches 3.89%, serial generation 5.00% and SIPR 10.00%.

Q4 (Weakness 3, Question 3). Does self-improvement genuinely expand the model's reasoning capability? Evidence should cover distinct successful solutions, redundancy across branches, and accuracy under fixed computation budgets.

A4. Yes. After RL the model finds more distinct correct routes, repeats itself less and is more accurate at a fixed budget. The table follows the three kinds of evidence requested.

EvidenceMeasureSIPR-SFTSIPR-RL
Distinct successful solutionsSampled MATH-500 solutions with at least two distinct correct routes12%20%
Redundancy across branchesBranches that exactly copy earlier branch text8.54%3.26%
Redundancy across branchesResponses that repeat a plan within a round29.63%23.38%
Accuracy under a fixed budgetCorrect completion within 16k tokens and ten branch starts7.78%10.83%

The first three rows are from the revised Appendix D.2 and the last from Table R9.

  • Coverage. With completion-trained RL, the share of problems solved within eight attempts rises on all four contest benchmarks, from 80% to 95% on AMC23, 30% to 36.67% on AIME24, 30% to 33.33% on AIME25 and 20% to 26.67% on HMMT (submitted Table 2).
  • An inspectable case. In an AIME24 walking-time problem, the RL model rejects a candidate that implies a negative walking time, carries the valid calculation into synthesis and returns the correct 204 minutes (Appendix D).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): contributions and main results

We thank all reviewers for their careful reading. This comment summarizes what SIPR contributes and what the results show. The next two comments answer four questions raised by more than one reviewer (CQ1–CQ4), each with its table. The individual replies give the direct answer to each question and add what is specific to each review.

Research question. We study a question central to recursive self-improvement. How should a model organize its exploration so that its own experience becomes more useful for learning? Within a solution, later rounds use earlier calculations to guide the next step. Across training batches, feedback on the model's own trajectories updates the policy that generates the next batch.

What is new.

  1. Demonstrations that teach how to explore. Each of the 10,000 strategy-guided demonstrations advances every branch one step at a time and keeps the calculations and candidate answers, so a later round can check a candidate or continue a derivation.
  2. Native, adaptive parallelism from demonstration to inference. A Base model learns to write its own plans, to choose how many branches each round opens, and to decide when to synthesize and stop. In SFT, RL and inference alike, a branch step reads the completed rounds and its own plan, with sibling steps hidden.
  3. Self-improvement on the model's own exploration. After SFT the student writes its own plans. Feedback on the final answers in its own trajectories jointly updates planning, branch steps and synthesis for the next batch.

Main results (Qwen3-4B unless stated).

FindingResultEvidence
RL on the model's own exploration improves all seven math benchmarks38.78% → 41.09% on average (+2.31 points)Table R1
SIPR outperforms a serial model trained on the same data41.09% vs. 36.79% (+4.30); GPQA +9.09Table R1
The RL gain holds at matched branch counts and tokens7.78% → 10.83% within 16k tokens under a ten-branch allowance; RL samples 3.2% fewer tokens per attemptTable R9 (reply to Reviewer C8N2)
Later rounds reuse earlier calculations10.00% → 13.47% correct completion within 16k tokensTable R2
SIPR outperforms sampling and selection at equal token budgets13.06% vs. 3.89% at 16k tokensTable R3
SIPR answers sooner than serial generation28.5% higher throughput; 10.00% vs. 5.00% correct within 60 secondsTables R4 and R5
SIPR-RL exceeds the released Parallel-R1 checkpoint41.09% vs. 29.12% (+11.97)Table R6 (reply to Reviewer C8N2)
The 4B-generated demonstrations transfer to Qwen3-8B14.79% (Base) → 41.78% (SFT) → 42.80% (RL)Table R7 (reply to Reviewer twpQ)
AuthorsOpenReview ↗

General response (2/3): common questions CQ1–CQ2

CQ1. Do the gains come from how SIPR organizes exploration, or from the training data and RL alone?

Raised by Reviewer nCuD (Weakness 2), Reviewer twpQ (Weakness 1) and Reviewer C8N2 (Weakness 2, matched training budgets).

Answer. They come from how the exploration is organized. A serial model trained on the same demonstrations and the same RL problems averages 4.30 points below SIPR-RL and is lower on all seven math benchmarks and on GPQA. RL adds 2.31 points to SIPR and 0.48 to the serial model (Table R1).

Control. Both serial checkpoints start from Qwen3-4B-Base. Serial SFT is the model after 625 supervised updates on the same 10,000 demonstrations with each demonstration's calculations in linear order. Serial RL adds 60 GRPO updates on the same 17,917 RL problems, with the same batch size, learning rate and KL coefficient as SIPR-RL. The two models receive the same number of full-model SFT and RL updates, and their supervised token totals differ by less than 1%.

Table R1. 4B accuracy (%) on seven math benchmarks and GPQA, before and after RL. The RL gap is SIPR-RL minus Serial RL in percentage points.

BenchmarkMetricSerial SFTSerial RLSIPR-SFTSIPR-RLRL gap
MATH-500Mean@176.6077.0079.8082.20+5.20
AMC23Mean@855.0054.0655.9458.13+4.07
AIME24Mean@817.5016.2515.8318.75+2.50
AIME25Mean@815.0015.8313.7518.33+2.50
HMMTMean@83.336.257.928.75+2.50
MinervaMean@143.3843.0151.4753.13+10.12
OlympiadMean@143.3245.1046.7448.37+3.27
GPQA DiamondMean@126.2626.2630.8135.35+9.09
Math averageSeven-benchmark mean36.3136.7938.7841.09+4.30

On GPQA Diamond, each setting uses one answer per question: Serial SFT and Serial RL each answer 52/198 correctly, while SIPR-SFT and SIPR-RL answer 61/198 and 70/198, respectively.

CQ2. Which design choices produce the gains?

Raised by Reviewer nCuD (Weakness 3, Question 4), Reviewer twpQ (Weakness 3) and Reviewer C8N2 (Weakness 3, accuracy under fixed budgets).

Answer. Three choices each raise correct completion within 16k sampled tokens on the same RL checkpoint: letting later rounds read earlier calculations (+3.47 points), letting the model choose its branch count (+2.92) and letting it branch again after a merge (+1.94, Table R2).

Table R2. Correct natural completions (%) within 16k actual sampled tokens, with 720 attempts per setting. Parentheses give SIPR minus control in percentage points.

SIPR componentControlSIPR
Full branch history10.0013.47 (+3.47)
Adaptive width10.6913.61 (+2.92)
Repeated branching11.5313.47 (+1.94)

Control. Each row tests one change on the same 4B RL checkpoint in a separate generation run. In both history conditions, the model reads all completed branches to write its summary. Full history also exposes those calculations to later rounds. The other controls fix width at four or switch to serial decoding after the first merge. The threshold applies offline to each saved attempt.

AuthorsOpenReview ↗

General response (3/3): common questions CQ3–CQ4

CQ3. How does SIPR compare with simpler parallel or sequential strategies at the same budget?

Raised by Reviewer C8N2 (Weakness 4) and Reviewer nCuD (Weakness 1).

Answer. At an equal budget of 16k sampled tokens, SIPR finishes 13.06% of attempts correctly, 3.4 times the 3.89% reached by voting, shortest-answer or verifier selection over four serial answers. At 24k tokens it stays ahead of the best four-answer selector and of one long serial answer (Table R3).

Table R3. Correct natural completions (%) at each actual sampled-token threshold. Each setting has 720 attempts on 90 AIME24, AIME25 and HMMT questions.

Method16k tokens24k tokens
Serial long12.6412.78
Four short + vote3.899.17
Four short + shortest3.898.61
Four short + verifier3.899.44
SIPR-RL13.0614.17

Control. Thresholds are applied offline to saved generations. Each short candidate receives one quarter of the budget and must finish within that allocation to be eligible. All four sampled prefixes count toward cost, including unfinished and unselected candidates. Saved generation caps are 6k per short answer, 24k for long serial generation, and 30k for SIPR. The fixed verifier is RLHFlow/Llama3.1-8B-PRM-Deepseek-Data; its compute is not charged to the four-answer baseline. Serial models use the same demonstration content and RL problem pool as SIPR.

CQ4. What does parallel execution cost relative to serial reasoning?

Raised by Reviewer nCuD (Weakness 1, efficiency and token count), Reviewer twpQ (Question 2) and Reviewer C8N2 (Weakness 2, inference latency).

Answer. SIPR samples 8.3% more tokens than the same-data serial model and returns answers sooner. On four A100s it finishes the workload 15.6% sooner at 28.5% higher throughput and returns twice as many correct answers within 60 seconds (10.00% vs. 5.00%, Table R4). On two RTX 4090 GPUs its throughput is 1.29–1.34× that of the serial model, with more correct answers within 120 seconds and per GPU-second at all three loads (Table R5).

Table R4. Generated work and throughput on four A100 GPUs with one active question per engine, 360 attempts per method.

MethodTotal sampled tokens (millions)Sampled tokens/s
Same-data serial RL4.08438
SIPR-RL4.42563
Relative change+8.3%+28.5%

Control. Each GPU runs one engine on the same 90 questions, with four attempts per question. Throughput counts tokens sampled in the main sequence and all branches, including unfinished attempts, per second of generation time.

Table R5. Sampled throughput and correct answers on two RTX 4090 GPUs at 4, 8 and 16 concurrent slots per GPU. Token rates are summed over both GPUs. Paired cells give SIPR / serial RL.

Slots per GPUSIPR tokens/sSerial RL tokens/sThroughput ratioCorrect within 120 sCorrect, finalCorrect per 1,000 GPU-seconds
4642.6495.21.30×31 / 2031 / 2515.90 / 10.09
81,006.2750.41.34×26 / 1332 / 2725.77 / 17.51
161,336.11,035.01.29×15 / 930 / 2730.86 / 25.15

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.