- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces W2SPO (weak-to-strong off-policy RL), which addresses the semantic redundancy and all-wrong group issues in reasoning RL. Analyses show that leveraging a weaker model to generate a short segment (as short as 8 tokens) can guide the target actor to explore more diverse trajectories and improve Pass@k results. To address the off-policy issue, advantage reweighting is proposed. Experiments on mathematical reasoning benchmarks using Qwen3-4B-Instruct show that W2SPO effectively improves performance while significantly decreasing training time by only optimizing the short branch segments. The conclusion is also extended to other models and a tool-use benchmark.
Critical strengths
- The proposed method is novel and appealing. The motivation is clear and the idea is intuitive.
- The method addresses the off-policy issue through advantage reweighting.
Critical weaknesses
- The experimental baseline is questionable. In Table 1, Qwen3-4B-Instruct is reported at 40% on AIME 2025. However, official results (available on Hugging Face) reach 47.4%, which already outperforms the proposed method (46.7%). This issue must be clarified to validate the true effectiveness of the method.
- When clipping the policy update, the probability from the old policy is used for the importance ratio. This is problematic because the branch is actually proposed off-policy by another model. Although the appendix explains this as a heuristic anchor, there is still potential theoretical risk in this implementation.
- Analyses on insertion position and branch length are still worth exploring during RL training, as the inference-only pilot study may not reflect actual optimization dynamics.
Questions for authors
- Why not optimize the other tokens instead of the branch only? Could joint optimization with continuation tokens further improve performance?