- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a way to improve RLOO’s performance by changing only the sampling strategy, rather than modifying the algorithm itself. The experimental results demonstrate the effectiveness of this approach.
Critical strengths
- The paper is easy to follow, and the proposed method is practical, especially in settings where modifying a well-established algorithm is difficult.
- The paper takes a novel perspective by shifting the focus from changing the optimization objective to improving the sampling procedure.
Critical weaknesses
- The paper could be strengthened by evaluating the proposed sampling strategy on a wider range of baseline models, as its effectiveness may depend on the specific model used.
- Figure 1 could be clearer, particularly panel (a), as I found it difficult to grasp the key idea of the proposed method on first reading.
Questions for authors
- I wonder whether this method would also be effective for other RL algorithms, such as GRPO, in sparse-reward settings.
3 comments
The authors' response has addressed my concerns, and I will maintain my original score.
Thank you for your time and constructive feedback. We are glad that our response has addressed your concerns.
Thank you for the suggestions on model coverage, Figure 1, and extending the sampler to GRPO. We expanded the model comparison, revised the illustration to make the rollout-level construction easier to follow, and tested the sampler in GRPO-style training.
Highlights
Additional model families. To test the sampler on other model families, we added 12 raw-RLOO runs on MiniCPM5-2B and LFM2.5-2.6B: HO and iid with three matched seeds per model, following the main training protocol. Table 2 in Section 6.1 presents the new results, and Appendix F gives every run. With iid test decoding for both methods, mean pass@1 improves in all four comparisons:
These results extend the training gains to two further model families. The larger mean gain is on code for MiniCPM5-2B and on AIME25 for LFM2.5-2.6B; Appendix F discusses this model–task variation in relation to reward geometry.
Figure 1. We revised the illustration to show how stratum assignments operate within autoregressive rollout generation. A new top strip identifies four rollouts with numbered, color-coded circles and shows their token sequences in a 2×2 layout. The instruction “At each token, assign strata using (a)” directly connects rollout generation to the distinct/shared mixture. The caption explains how the calibrated mixture yields uniform pairwise cell probabilities, and panel (b) connects fresh tokenwise sampling to complete-trajectory pair independence.
GRPO-style training. To test whether the sampling benefit carries over to a group-normalized learning rule, we added six Qwen3-4B runs using group-standardized advantages and one on-policy update per rollout batch, with HO and iid sharing the training stream, optimizer, and token cap. Both methods use iid endpoint evaluation. The results are:
All three run differences are positive on each benchmark. Section 6.3, Table 4, and Appendix E report this group-normalized experiment. Equation (11) shows that normalization rescales mixed-group reward contrasts, while groups with identical rewards retain zero advantages; higher-order coordination therefore remains relevant to the normalized update. At the frozen iid-RLOO halfway checkpoint, the same-group diagnostic shows 7.1% lower normalized-gradient variance, complementing the positive training gains. The added model-family experiments and this tested GRPO-style configuration jointly support the sampler's practical use across the evaluated models and training settings.