acceptodds
Under review as a conference paper at ICLR 2027

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

Abstract

Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self-generated rollouts. However, we identify a critical support-limited bottleneck in this paradigm: on challenging reasoning tasks, the target model’s samples often exhibit semantic redundancy and pro- vide negligible reward contrast for policy updates. In this paper, we propose to overcome this limitation through a weak-to-strong learning paradigm, where a policy’s exploration is informed by a weaker but computationally efficient aux- iliary model. We introduce W2SPO, an off-policy RL method that injects short auxiliary segments—often as brief as 8 tokens—into intermediate target-model trajectories and the target model then completes the reasoning path from these diverted states. Policy updates are restricted to these short inserted segments based on final verifiable rewards. Empirically, W2SPO achieves superior perfor- mance among evaluated 4B-scale models on mathematical reasoning benchmarks, outperforming evaluated post-trained baselines. Compared with vanilla GRPO under the same sampling budget, W2SPO improves Pass@1 from 62.3% to 64.2% while achieving a 3.55× training speedup. Additional experiments show gains with a non-Qwen 7B target and on a multi-turn tool-use benchmark. These re- sults suggest that weak auxiliary branches can induce stronger target reasoning policies by expanding local exploration support. Code and data are available at https://anonymous.4open.science/r/W2SPO-1B86/

Primary area:
generative models
Keywords:
Large language models, Reasoning, Weak-to-strong learning, Model collaboration, Local exploration, Efficient post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e2rnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces W2SPO (weak-to-strong off-policy RL), which addresses the semantic redundancy and all-wrong group issues in reasoning RL. Analyses show that leveraging a weaker model to generate a short segment (as short as 8 tokens) can guide the target actor to explore more diverse trajectories and improve Pass@k results. To address the off-policy issue, advantage reweighting is proposed. Experiments on mathematical reasoning benchmarks using Qwen3-4B-Instruct show that W2SPO effectively improves performance while significantly decreasing training time by only optimizing the short branch segments. The conclusion is also extended to other models and a tool-use benchmark.

Critical strengths
  1. The proposed method is novel and appealing. The motivation is clear and the idea is intuitive.
  2. The method addresses the off-policy issue through advantage reweighting.
Critical weaknesses
  1. The experimental baseline is questionable. In Table 1, Qwen3-4B-Instruct is reported at 40% on AIME 2025. However, official results (available on Hugging Face) reach 47.4%, which already outperforms the proposed method (46.7%). This issue must be clarified to validate the true effectiveness of the method.
  2. When clipping the policy update, the probability from the old policy πθold\pi_{\theta_{\text{old}}} is used for the importance ratio. This is problematic because the branch is actually proposed off-policy by another model. Although the appendix explains this as a heuristic anchor, there is still potential theoretical risk in this implementation.
  3. Analyses on insertion position and branch length are still worth exploring during RL training, as the inference-only pilot study may not reflect actual optimization dynamics.
Questions for authors
  1. Why not optimize the other tokens instead of the branch only? Could joint optimization with continuation tokens further improve performance?
Reviewer_TmKVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes W2SPO, a weak-to-strong off-policy reinforcement learning framework designed to address the limited exploration diversity and insufficient reward contrast in conventional self-rollout RL methods such as GRPO. W2SPO introduces short reasoning branches generated by a weaker auxiliary model into intermediate trajectories of the target model, which then completes the reasoning process. The final verifier rewards are used to compute importance-weighted advantages, while policy updates are restricted to the inserted branch tokens. Experiments on mathematical reasoning benchmarks demonstrate improvements over GRPO and target-only branching, achieving higher Pass@1 accuracy and a 3.55× training speedup. Additional experiments examine sampling diversity, ablations, and generalization to another model architecture and multi-turn tool-use tasks.

Critical strengths
  • The paper addresses an important limitation of self-rollout RL, where semantically redundant trajectories and all-wrong groups provide insufficient learning signals. Leveraging a weaker auxiliary model to introduce diverse local reasoning branches is an interesting approach that improves exploration without requiring stronger teachers or complete expert trajectories.
  • W2SPO consistently outperforms matched-sampling GRPO across the evaluated mathematical reasoning benchmarks while achieving a 3.55× training speedup. The controlled A-only branching ablation and additional generalization experiments further support the effectiveness of the proposed framework.
Critical weaknesses
  • The effectiveness of W2SPO appears to depend on the auxiliary model's ability to provide diverse and useful reasoning branches. However, the paper only evaluates two auxiliary models without systematically investigating how their reasoning capabilities, exploration diversity, or behavioral differences from the target model affect downstream performance. This limits our understanding of when and why weak-to-strong exploration is effective.
  • The proposed importance-weighted advantage estimation relies on a heuristic coefficient derived from likelihoods computed under potentially different tokenizers, rather than an exact importance-sampling ratio. Although the paper provides some empirical validation, the effects of distribution mismatch and the sensitivity to the weighting mechanism are not sufficiently investigated, raising concerns about the robustness and general applicability of the optimization method.
Questions for authors
  • How does the choice of auxiliary model B affect the final performance of W2SPO? For example, could a more exploratory model B enhance exploration and lead to greater performance improvements, while a more conservative model B, whose reasoning patterns largely overlap with those of the target model A, provide limited benefits? Are there experimental results investigating the relationship between auxiliary-model diversity, distributional differences, and downstream performance?
  • Since the proposed importance coefficient is a heuristic approximation rather than an exact importance-sampling ratio, how sensitive is W2SPO to the weighting hyperparameters and the distribution mismatch between models A and B? Can the authors provide additional evidence supporting the stability of this mechanism across different auxiliary models?
  • Figure 3 shows that 16-token branches can achieve better Pass@8 performance than 8-token branches, yet the subsequent RL experiments primarily use 8-token branches inserted after 50 tokens. How sensitive is the final RL training performance to these choices? Does the 8-token configuration provide the best trade-off between training efficiency and downstream accuracy?
Reviewer_dw9iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes W2SPO, a weak-to-strong off-policy optimization method for reinforcement learning of large language model reasoning. The method uses a frozen auxiliary model to generate short reasoning branches, which are then extended by the target model. Policy updates are applied to the local branch tokens based on the final verifiable rewards, with the goal of improving exploration diversity while reducing training costs.

On mathematical reasoning tasks, the authors report a Pass@1 of 64.2% for W2SPO, outperforming the GRPO baseline under a matched sampling budget (62.3%), along with a 3.55× training speedup. The ablation studies provide preliminary evidence for the effectiveness of cross-model branching. However, the statistical reliability of the reported performance gains has not been sufficiently established. Furthermore, the current experiments do not adequately demonstrate the unique cost–performance advantages of using a weaker auxiliary model over a stronger one. Overall, the method is promising, but its performance claims and the motivation for the auxiliary model selection require stronger empirical support.

Critical strengths
  1. Local Branch Exploration Is a Meaningful Design Choice.

W2SPO does not require the auxiliary model to generate complete reasoning trajectories. Instead, it injects short branches at intermediate states of the target model's reasoning process, allowing the target model to complete the subsequent reasoning. This design uses the auxiliary model specifically to propose alternative local continuations, whose effectiveness is evaluated through final verifiable rewards. The ablation study shows that removing cross-model branches reduces Overall Pass@1 from 64.2% to 61.9%, providing preliminary evidence for the benefit of cross-model exploration.

  1. The Training Efficiency Gains Are Promising.

Under a matched sampling budget, the authors report a 3.55× speedup in per-step training time while achieving a higher Pass@1 score. If these efficiency gains and performance improvements are further validated, W2SPO could offer practical benefits for reasoning-oriented reinforcement learning under limited computational resources.

Critical weaknesses
  1. The Necessity of a Weak Auxiliary Model and Its Cost–Performance Trade-off Are Insufficiently Validated.

The ablation study using target-model-generated branches provides evidence for the effectiveness of cross-model branching, but it does not establish that a weaker auxiliary model offers unique advantages over a stronger one. The default auxiliary model, Gemma-3-4B-IT, is comparable in parameter scale to the target model, Qwen3-4B, while the evaluation using Mistral-7B reports only a single performance result. The paper does not systematically characterize the relationship between auxiliary branch quality, inference cost, and downstream training performance across models of different scales. In particular, given that only six 8-token auxiliary branches are generated per problem, it remains unclear whether the additional cost of using a larger model would outweigh the potential benefits of improved exploration. Furthermore, the reported 3.55× speedup is primarily attributable to the reductions in the forward and training stages, and the contribution of using a smaller auxiliary model to the overall efficiency gains has not been isolated. I encourage the authors to compare auxiliary models of different scales under comparable total computational budgets and report both the standalone inference overhead of the auxiliary models and the end-to-end training cost.

  1. The Statistical Reliability of the Reported Performance Gains Remains Unclear.

The paper reports a Pass@1 score of 64.2% for W2SPO, compared with 62.3% for GRPO under a matched sampling budget, corresponding to an absolute improvement of 1.9 percentage points. Although the appendix reports a W2SPO result of 63.9% using another random seed, the corresponding multi-seed GRPO results are not provided, nor are confidence intervals or significance tests for the main performance comparison. Consequently, the current evidence is insufficient to determine whether the reported improvement is robust to training randomness. I encourage the authors to conduct multi-seed comparisons between W2SPO and GRPO and report paired statistical tests or bootstrap confidence intervals on the same evaluation instances to substantiate the paper's primary performance claim.

Questions for authors
  1. Regarding Auxiliary Model Selection and Computational Cost: Could the authors report the actual computational cost of generating auxiliary branches, separately from the end-to-end training cost? Furthermore, why was Gemma-3-4B-IT chosen over a potentially stronger auxiliary model? Is the single replacement experiment with Mistral-7B sufficient to establish that larger auxiliary models do not offer a better cost–performance trade-off?

  2. Regarding the Statistical Reliability of the Performance Gains: W2SPO achieves a Pass@1 of 64.2%, compared with 62.3% for GRPO under a matched sampling budget. Could the authors provide multi-seed comparisons between the two methods and report confidence intervals or paired statistical tests to establish whether the 1.9-percentage-point improvement is robust and reproducible?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.