- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper improves upon GRPO by diversifying the first token of rollouts through adjusted sampling frequency, substantially expanding the exploration space and enhancing mathematical and coding capabilities.
Critical strengths
- This method introduces no additional time overhead and serves as a plug-and-play complement to the GRPO algorithm.
- Experiments demonstrate that this method consistently achieves solid improvements across multiple models on both mathematical reasoning and code generation tasks.
Critical weaknesses
-
Novelty is the biggest issue of this paper; the core contribution resembles an engineering finding and improvement, currently lacking sufficient research novelty.
-
The paper mentions that the post-training of instruct models leads to the concentration phenomenon of the first token, and the experiments only focus on instruct models. Does this mean the method only holds due to the particularity of instruct models, and can the authors provide additional training results on base models?
Questions for authors
- Could the authors provide a more detailed analysis to demonstrate the advantages of the method? A few points currently confuse me: in Table 4, it seems that simply modifying the temperature already yields a notable improvement. What I am most concerned about is whether there are additional guarantees beyond the intuitive argument of expanding the exploration space. If this core concern can be addressed, I will raise my score.
- Is the improvement of this method over DAPO driven by an increased proportion of informative/valid groups, or by the internal quality of these valid groups? Since DAPO already ensures that gradient updates are computed only on valid groups.
4 comments
Rebuttal: Part 2 of 2
W2. Applicability across a broader range of models
Following the reviewer's suggestion, we added experiments with four base models spanning different families and included the result in Section 5.5 and Appendix B.9 of the revised manuscript.
- Qwen2.5-3B and Gemma3-4B-pt are trained on GSM8K:
| Base model | Method | Pass@1 | Pass@8 | Pass@64 |
|---|---|---|---|---|
| Qwen2.5-3B | Pre-RLVR | 67.85 | 93.71 | 96.59 |
| Qwen2.5-3B | GRPO | 83.24 | 94.92 | 97.88 |
| Qwen2.5-3B | GRPO + REFT | 84.23 | 95.30 | 98.86 |
| Gemma3-4B-pt | Pre-RLVR | 5.99 | 19.86 | 65.05 |
| Gemma3-4B-pt | GRPO | 56.18 | 83.55 | 93.63 |
| Gemma3-4B-pt | GRPO + REFT | 60.65 | 84.53 | 94.92 |
- Olmo-3-1025-7B is trained on BigMath-Easy, and Qwen3.5-9B-Base on BigMath:
Math Avg. is the unweighted average over MATH-500, AIME24, AIME25, and AMC23.
| Base model | Method | Math Avg. Pass@1 | Math Avg. Pass@8 | Math Avg. Pass@64 |
|---|---|---|---|---|
| Olmo-3-1025-7B | Pre-RLVR | 29.66 | 47.59 | 67.35 |
| Olmo-3-1025-7B | GRPO | 33.84 | 56.08 | 72.36 |
| Olmo-3-1025-7B | GRPO + REFT | 40.79 | 64.00 | 77.17 |
| Qwen3.5-9B-Base | Pre-RLVR | 40.18 | 60.44 | 71.70 |
| Qwen3.5-9B-Base | GRPO | 44.78 | 66.25 | 75.43 |
| Qwen3.5-9B-Base | GRPO + REFT | 59.65 | 74.99 | 83.50 |
Additionally, we evaluated Qwen3.5-9B in thinking mode, trained on BigMath with a maximum completion length of 32,768 tokens:
| Method | Math Avg. Pass@1 | Math Avg. Pass@8 | Math Avg. Pass@64 |
|---|---|---|---|
| Qwen3.5-9B | 56.82 | 76.34 | 90.48 |
| GRPO | 61.66 | 81.34 | 92.67 |
| GRPO + REFT | 70.52 | 84.57 | 92.87 |
REFT consistently improves every reported sampling budget across these base models and the thinking-enabled setting. These results show that prior instruction tuning is not essential for REFT to help, and that its benefits extend to longer reasoning traces.
Q2. Valid-group rates and continuation learning under DAPO
DAPO already trains on valid groups, but passing the validity filter does not determine which reasoning trajectories those groups contain. Our additional results support improved learning from the explored trajectories, beyond differences in the frequency of valid groups.
1. Group validity and successful exploration.
We decomposed the uninformative groups into all-correct and all-wrong cases. Each entry below reports the run-averaged percentage for DAPO → DAPO + REFT. The valid-group rate is computed as 100% minus the total uninformative-group rate.
| Training setting | All correct (%) | All wrong (%) | Valid (%) |
|---|---|---|---|
| Qwen2.5-3B-Instruct, GSM8K | 66.13 → 67.69 | 1.69 → 1.52 | 32.18 → 30.79 |
| Qwen2.5-3B-Instruct, BigMath-Easy | 26.15 → 27.78 | 8.00 → 5.85 | 65.85 → 66.37 |
| Qwen2.5-7B-Instruct, BigMath | 8.27 → 9.06 | 17.66 → 13.62 | 74.07 → 77.32 |
REFT does not consistently increase the valid-group rate, as it decreases on GSM8K. Yet all-wrong groups become less frequent across all three settings, while all-correct groups become more common. This indicates that REFT explores successful trajectories more effectively, even without increasing the valid-group rate.
2. Better continuations with first-token selection controlled.
For Qwen2.5-3B-Instruct trained on GSM8K, we repeated the continuation controls in W1 using DAPO and DAPO + REFT. At evaluation, we varied the first-token source and continuation policy, and separately fixed both policies to the initial model's Top-1 first token.
| First-token source or condition | Continuation policy | Avg@8 | Pass@8 |
|---|---|---|---|
| DAPO | DAPO | 85.27 | 95.15 |
| DAPO | DAPO + REFT | 85.65 | 95.83 |
| DAPO + REFT | DAPO | 84.89 | 94.84 |
| DAPO + REFT | DAPO + REFT | 86.02 | 96.06 |
| Initial model Top-1, fixed | DAPO | 85.33 | 94.69 |
| Initial model Top-1, fixed | DAPO + REFT | 85.54 | 95.83 |
Under either first-token distribution, the REFT-trained continuation policy improves both metrics. The improvement also holds when both policies start from the same fixed first token, whereas switching only the first-token source does not help.
Thus, on GSM8K, REFT learns better continuations despite a lower valid-group rate. Along with the reduction in all-wrong groups, this supports our explanation that REFT discovers useful trajectories and improves learning within valid groups, beyond altering their frequency. The figure of training reward curve is included in Appendix B.7 of the revised manuscript.
Thank you for your detailed response and additional experiments. They have addressed my concerns, so I have raised my score accordingly.
We are delighted that our additional experiments addressed your concerns. Thank you for your careful review and thoughtful suggestions, which helped us improve our paper.
Rebuttal: Part 1 of 2
We appreciate your careful engagement with our work and welcome the opportunity to clarify our contribution. Below, we address your questions with additional experiments, analyses, and clarifications.
We respectfully disagree that REFT is only an engineering improvement. Much prior work targets high-entropy positions where the model already considers several alternatives (Section 2). We identify a different opportunity at the first token, whose distribution is sharply concentrated despite lower-probability candidates supporting accurate, distinct continuations. REFT explores these underrepresented starting points to discover useful trajectories and learn better continuations, with gains that persist even when first-token selection is held fixed, as shown below.
The simplicity of REFT is a strength, and a minimal intervention isolates the value of this overlooked exploration site. In Table 4, the compared diversification methods improve Pass@1 but reduce Pass@64 relative to GRPO, whereas REFT improves both. This Pass@64 advantage also holds in the harder BigMath-Easy setting, as shown in our additional experiments (Appendix B.5 of the revised manuscript). Together with the semantic-diversity analysis in Section 3, these results support our interpretation that REFT discovers useful, previously underexplored trajectories and that the added rollout diversity translates into broader successful coverage.
To examine continuation learning directly, we varied which trained policy supplies the first token and which generates the continuation at evaluation, using Qwen2.5-3B-Instruct trained on GSM8K. We crossed the GRPO and GRPO + REFT policies and also fixed both policies to the initial model's Top-1 first token for each prompt. Avg@8 measures average correctness across eight responses, while Pass@8 measures whether at least one is correct.
REFT continuations improve both metrics under either first-token distribution and when both policies start from exactly the same fixed token. Changing only the first-token distribution does not improve GRPO continuations. These controls show that the benefit extends to learning better continuations, supporting our interpretation that first-token exploration helps discover useful reasoning trajectories and learn to generate them reliably. We report the full continuation-control results under both GRPO and DAPO in Appendix F (Table 29) of the revised manuscript.
The First-token T=2 result is our milder control at the exploration site identified in this work. Its improvement therefore supports our main contribution, which is targeting the first token for diversification.
REFT assigns uniform weight to the top-20 first-token candidates, corresponding to the T→∞ within that restricted set. REFT additionally selects distinct candidates without replacement and allocates continuations evenly among them. This ensures balanced coverage of the selected first tokens within each rollout group, providing greater opportunities to explore diverse trajectories, whereas T=2 can leave the distribution sharply concentrated.
Consistent with a milder intervention, First-token T=2 gives intermediate Pass@1 and Pass@8 results, but it does not improve Pass@64 over GRPO, suggesting that T=2 is insufficient to drive broader solution discovery. REFT's stronger, T→∞ -level exploration improves all three. Beyond the exploration intuition, the continuation controls in W1 provide direct evidence of improved continuation quality after training, including when the first token is fixed.