acceptodds
Under review as a conference paper at ICLR 2027

Diversifying RLVR Rollouts via First-Token Exploration

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains reasoning models without labeled trajectories, using groups of verifier-scored rollouts to explore alternative reasoning paths. Limited rollout diversity is a central bottleneck, typically addressed through adjustments to temperature, prefixes, or rollout selection. We identify the *first token* of the response as a structurally distinct target for diversification, largely overlooked in prior work. We find that the first-token distribution is sharply concentrated and only weakly related to downstream correctness, as lower-probability candidates can yield similarly accurate responses. Diversifying the first token can therefore broaden the reasoning paths explored within each rollout group with little loss in response quality. Motivated by this observation, we introduce REFT (Rollout Exploration with First-Token Diversification), a lightweight modification to RLVR. REFT samples first tokens uniformly from the policy's top- candidates and allocates rollouts evenly across the sampled tokens, leaving the rest of the pipeline unchanged. We evaluate REFT on eight models spanning multiple architectures and sizes (0.5B–14B), with mathematical reasoning and code-generation tasks under GRPO and DAPO. Across these settings, REFT consistently improves Pass@1, Pass@8, and Pass@64. It also outperforms competing diversification methods at every evaluated budget, incurring the lowest rollout cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, reinforcement learning with verifiable rewards, rollout diversity, first-token exploration

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_aMNpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper improves upon GRPO by diversifying the first token of rollouts through adjusted sampling frequency, substantially expanding the exploration space and enhancing mathematical and coding capabilities.

Critical strengths
  • This method introduces no additional time overhead and serves as a plug-and-play complement to the GRPO algorithm.
  • Experiments demonstrate that this method consistently achieves solid improvements across multiple models on both mathematical reasoning and code generation tasks.
Critical weaknesses
  • Novelty is the biggest issue of this paper; the core contribution resembles an engineering finding and improvement, currently lacking sufficient research novelty.

  • The paper mentions that the post-training of instruct models leads to the concentration phenomenon of the first token, and the experiments only focus on instruct models. Does this mean the method only holds due to the particularity of instruct models, and can the authors provide additional training results on base models?

Questions for authors
  • Could the authors provide a more detailed analysis to demonstrate the advantages of the method? A few points currently confuse me: in Table 4, it seems that simply modifying the temperature already yields a notable improvement. What I am most concerned about is whether there are additional guarantees beyond the intuitive argument of expanding the exploration space. If this core concern can be addressed, I will raise my score.
  • Is the improvement of this method over DAPO driven by an increased proportion of informative/valid groups, or by the internal quality of these valid groups? Since DAPO already ensures that gradient updates are computed only on valid groups.
4 comments
AuthorsOpenReview ↗

Rebuttal: Part 1 of 2

We appreciate your careful engagement with our work and welcome the opportunity to clarify our contribution. Below, we address your questions with additional experiments, analyses, and clarifications.

W1. Research novelty

We respectfully disagree that REFT is only an engineering improvement. Much prior work targets high-entropy positions where the model already considers several alternatives (Section 2). We identify a different opportunity at the first token, whose distribution is sharply concentrated despite lower-probability candidates supporting accurate, distinct continuations. REFT explores these underrepresented starting points to discover useful trajectories and learn better continuations, with gains that persist even when first-token selection is held fixed, as shown below.

The simplicity of REFT is a strength, and a minimal intervention isolates the value of this overlooked exploration site. In Table 4, the compared diversification methods improve Pass@1 but reduce Pass@64 relative to GRPO, whereas REFT improves both. This Pass@64 advantage also holds in the harder BigMath-Easy setting, as shown in our additional experiments (Appendix B.5 of the revised manuscript). Together with the semantic-diversity analysis in Section 3, these results support our interpretation that REFT discovers useful, previously underexplored trajectories and that the added rollout diversity translates into broader successful coverage.

To examine continuation learning directly, we varied which trained policy supplies the first token and which generates the continuation at evaluation, using Qwen2.5-3B-Instruct trained on GSM8K. We crossed the GRPO and GRPO + REFT policies and also fixed both policies to the initial model's Top-1 first token for each prompt. Avg@8 measures average correctness across eight responses, while Pass@8 measures whether at least one is correct.

First-token source or conditionContinuation policyAvg@8Pass@8
GRPOGRPO85.0295.38
GRPOGRPO + REFT86.0396.06
GRPO + REFTGRPO84.6795.38
GRPO + REFTGRPO + REFT86.6796.13
Initial model Top-1, fixedGRPO85.3795.68
Initial model Top-1, fixedGRPO + REFT86.0395.91

REFT continuations improve both metrics under either first-token distribution and when both policies start from exactly the same fixed token. Changing only the first-token distribution does not improve GRPO continuations. These controls show that the benefit extends to learning better continuations, supporting our interpretation that first-token exploration helps discover useful reasoning trajectories and learn to generate them reliably. We report the full continuation-control results under both GRPO and DAPO in Appendix F (Table 29) of the revised manuscript.

Q1. First-token temperature and REFT's additional benefit

The First-token T=2T=2 result is our milder control at the exploration site identified in this work. Its improvement therefore supports our main contribution, which is targeting the first token for diversification.

REFT assigns uniform weight to the top-20 first-token candidates, corresponding to the T→∞T \to \infty within that restricted set. REFT additionally selects distinct candidates without replacement and allocates continuations evenly among them. This ensures balanced coverage of the selected first tokens within each rollout group, providing greater opportunities to explore diverse trajectories, whereas T=2T=2 can leave the distribution sharply concentrated.

  • For Qwen2.5-3B-Instruct trained on GSM8K (Table 4):
MethodPass@1Pass@8Pass@64
GRPO84.5395.3898.64
First-token T=285.5995.8398.56
GRPO + REFT86.2896.1399.39

Consistent with a milder intervention, First-token T=2T=2 gives intermediate Pass@1 and Pass@8 results, but it does not improve Pass@64 over GRPO, suggesting that T=2T=2 is insufficient to drive broader solution discovery. REFT's stronger, T→∞T \to \infty -level exploration improves all three. Beyond the exploration intuition, the continuation controls in W1 provide direct evidence of improved continuation quality after training, including when the first token is fixed.

AuthorsOpenReview ↗

Rebuttal: Part 2 of 2

W2. Applicability across a broader range of models

Following the reviewer's suggestion, we added experiments with four base models spanning different families and included the result in Section 5.5 and Appendix B.9 of the revised manuscript.

  • Qwen2.5-3B and Gemma3-4B-pt are trained on GSM8K:
Base modelMethodPass@1Pass@8Pass@64
Qwen2.5-3BPre-RLVR67.8593.7196.59
Qwen2.5-3BGRPO83.2494.9297.88
Qwen2.5-3BGRPO + REFT84.2395.3098.86
Gemma3-4B-ptPre-RLVR5.9919.8665.05
Gemma3-4B-ptGRPO56.1883.5593.63
Gemma3-4B-ptGRPO + REFT60.6584.5394.92
  • Olmo-3-1025-7B is trained on BigMath-Easy, and Qwen3.5-9B-Base on BigMath:

Math Avg. is the unweighted average over MATH-500, AIME24, AIME25, and AMC23.

Base modelMethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Olmo-3-1025-7BPre-RLVR29.6647.5967.35
Olmo-3-1025-7BGRPO33.8456.0872.36
Olmo-3-1025-7BGRPO + REFT40.7964.0077.17
Qwen3.5-9B-BasePre-RLVR40.1860.4471.70
Qwen3.5-9B-BaseGRPO44.7866.2575.43
Qwen3.5-9B-BaseGRPO + REFT59.6574.9983.50

Additionally, we evaluated Qwen3.5-9B in thinking mode, trained on BigMath with a maximum completion length of 32,768 tokens:

MethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Qwen3.5-9B56.8276.3490.48
GRPO61.6681.3492.67
GRPO + REFT70.5284.5792.87

REFT consistently improves every reported sampling budget across these base models and the thinking-enabled setting. These results show that prior instruction tuning is not essential for REFT to help, and that its benefits extend to longer reasoning traces.

Q2. Valid-group rates and continuation learning under DAPO

DAPO already trains on valid groups, but passing the validity filter does not determine which reasoning trajectories those groups contain. Our additional results support improved learning from the explored trajectories, beyond differences in the frequency of valid groups.

1. Group validity and successful exploration.

We decomposed the uninformative groups into all-correct and all-wrong cases. Each entry below reports the run-averaged percentage for DAPO → DAPO + REFT. The valid-group rate is computed as 100% minus the total uninformative-group rate.

Training settingAll correct (%)All wrong (%)Valid (%)
Qwen2.5-3B-Instruct, GSM8K66.13 → 67.691.69 → 1.5232.18 → 30.79
Qwen2.5-3B-Instruct, BigMath-Easy26.15 → 27.788.00 → 5.8565.85 → 66.37
Qwen2.5-7B-Instruct, BigMath8.27 → 9.0617.66 → 13.6274.07 → 77.32

REFT does not consistently increase the valid-group rate, as it decreases on GSM8K. Yet all-wrong groups become less frequent across all three settings, while all-correct groups become more common. This indicates that REFT explores successful trajectories more effectively, even without increasing the valid-group rate.

2. Better continuations with first-token selection controlled.

For Qwen2.5-3B-Instruct trained on GSM8K, we repeated the continuation controls in W1 using DAPO and DAPO + REFT. At evaluation, we varied the first-token source and continuation policy, and separately fixed both policies to the initial model's Top-1 first token.

First-token source or conditionContinuation policyAvg@8Pass@8
DAPODAPO85.2795.15
DAPODAPO + REFT85.6595.83
DAPO + REFTDAPO84.8994.84
DAPO + REFTDAPO + REFT86.0296.06
Initial model Top-1, fixedDAPO85.3394.69
Initial model Top-1, fixedDAPO + REFT85.5495.83

Under either first-token distribution, the REFT-trained continuation policy improves both metrics. The improvement also holds when both policies start from the same fixed first token, whereas switching only the first-token source does not help.

Thus, on GSM8K, REFT learns better continuations despite a lower valid-group rate. Along with the reduction in all-wrong groups, this supports our explanation that REFT discovers useful trajectories and improves learning within valid groups, beyond altering their frequency. The figure of training reward curve is included in Appendix B.7 of the revised manuscript.

Thank you for your detailed response and additional experiments. They have addressed my concerns, so I have raised my score accordingly.

We are delighted that our additional experiments addressed your concerns. Thank you for your careful review and thoughtful suggestions, which helped us improve our paper.

Reviewer_5cj4OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies the first token as an overlooked source of rollout diversity in RLVR and proposes REFT. REFT samples some top first-token candidates evenly. The method is simple and validated.

Critical strengths

The core observation is interesting and leads to a very simple method: lower-probability first tokens can still produce good but different continuations, so explicitly covering them is a cheap way to improve rollout diversity.

The experiments are quite comprehensive.

Critical weaknesses

The comparison with existing diversification methods is limited. It would be more convincing to verify this on at least one harder math setting or with a larger model.

Questions for authors

Do the authors have evidence that REFT still outperforms diversification methods on a harder math setting or a larger model? This would help determine how general the claimed advantage over prior diversification methods is.

Can the authors clarify how much of the gain comes from increased rollout diversity or the implicit upweighting of trajectories starting from low-probability first tokens?

3 comments

We are delighted that you found the core observation interesting and sincerely thank you for your thoughtful review. Below, we report the additional comparisons and analyses prompted by your suggestions.

W1 & Q1. Additional comparisons with existing diversification methods

We have extended the comparison to a harder mathematical setting, training Qwen2.5-3B-Instruct on BigMath-Easy across three seeds. Math Avg. is the unweighted average over MATH-500, AIME24, AIME25, and AMC23.

MethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Qwen2.5-3B-Instruct23.2745.7561.95
GRPO29.45 ± 0.7647.42 ± 0.8464.27 ± 0.89
Rollout-MT26.68 ± 0.3346.63 ± 0.7462.46 ± 0.68
Token-MT29.13 ± 0.8446.05 ± 0.6563.27 ± 0.92
LATR30.10 ± 0.3248.36 ± 0.6663.42 ± 0.99
GRPO + REFT33.34 ± 1.4853.47 ± 1.1869.02 ± 1.20

The paper's main finding holds in this harder setting. REFT achieves the highest scores across all three sampling budgets. In particular, REFT improves Pass@64 over GRPO, while the compared diversification methods fall below GRPO. Together with the Pass@1 gain, this shows that REFT improves both single-sample accuracy and success with a larger sampling budget. We include the full per-benchmark results in Appendix B.5 of the revised manuscript.

Q2. The roles of diversity and implicit reweighting

REFT helps in two connected ways. (1) It gives useful but underexplored first-token candidates more training exposure, allowing reward feedback to adjust their probabilities. (2) It learns better continuations through this broader exploration. Our new controls show that the continuation-quality benefit persists even when first-token selection is held fixed.

1. Discovering useful first-token candidates.

As shown in Section 3, low-probability first tokens can still lead to accurate and distinct continuations. REFT gives these candidates greater representation in training by allocating rollouts more evenly across them. Reward feedback can then reinforce successful trajectories and the first tokens that lead to them.

2. Learning better continuations from those starting points.

To test whether the gains extend beyond first-token selection, we crossed the first-token distribution and continuation policy of the GRPO- and REFT-trained models at evaluation, using Qwen2.5-3B-Instruct on GSM8K. For each first-token distribution, we compared continuations generated by the two policies. Avg@8 measures average correctness across eight responses, while Pass@8 measures whether at least one is correct.

First-token distributionContinuation policyAvg@8Pass@8
GRPOGRPO85.0295.38
GRPOREFT86.0396.06
REFTGRPO84.6795.38
REFTREFT86.6796.13

The comparison shows two clear patterns:

  • Under either fixed first-token distribution, REFT continuations achieve higher Avg@8 and Pass@8, including when GRPO supplies the first token.
  • Changing only the first-token distribution, while retaining GRPO continuations, does not improve either metric.

The revised Appendix F (Table 29) also reports the same pattern when the first token is fixed to the initial model's Top-1 candidate for each prompt. This further shows that REFT learns better continuations even when both policies start from exactly the same token.

Broader exploration and implicit upweighting are coupled during training, so these controls do not assign a separate share of the gain to each. They establish that REFT's gains include better learned continuations, supporting our interpretation that exploring from underrepresented first tokens helps discover useful trajectories and learn to generate them more reliably.

The rebuttal resolves my main empirical concern and substantially addresses the second one, although the exact contribution of exploration versus implicit reweighting remains disentangled only partially.

AuthorsOpenReview ↗

Response to Reviewer 5cj4's Follow-up

Thank you for your thoughtful follow-up.

Regarding exploration versus implicit reweighting, diversity lays the groundwork for discovering useful alternatives that reward feedback can then reinforce. Exploring underrepresented first tokens also increases the training exposure of their continuations, so the two effects arise together. Separating their individual contributions remains challenging, and we consider this an interesting direction for future work.

Reviewer_DHZBOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes REFT, which diversifies RLVR rollouts by sampling plausible first response tokens more evenly, and reports improved mathematical reasoning and code-generation performance across multiple models and RL objectives.

Critical strengths
  • The diagnostic analysis is strong and insightful. The paper shows that first-token probabilities are highly concentrated even though lower-ranked candidates can lead to correct and distinct continuations. This identifies a concrete, underexplored source of useful RLVR diversity.

  • The method is simple yet effective. REFT changes only how the first token is sampled and allocated across a fixed rollout budget. Its simplicity makes the reported gains especially compelling and the method easy to adopt.

  • The experimental evaluation is comprehensive and strong. The authors test multiple model families and sizes, GRPO and DAPO, mathematical reasoning and code generation, and several sampling budgets. The consistent gains and targeted controls provide strong support for the proposed explanation.

  • The deep analysis in Section 7 is strong and further supports the proposed claims.

Critical weaknesses
  • Direct comparison to diversity-related RLVR methods is missing, such as DIVER [1]. The authors are suggested to compare with these closely related baselines under a matched rollout or compute budget.

  • The boundary of applicability is unclear. REFT is motivated by a sharply concentrated first-token distribution whose lower-ranked tokens remain useful. The paper itself shows a base-model case where correctness falls substantially with token rank, but does not test training under such a condition. It is therefore unclear when uniform top-N exploration stops helping.

  • The altered rollout distribution warrants further analysis. REFT samples first tokens differently from the policy while applying the unchanged GRPO/DAPO loss without importance sampling correction. The authors discuss this choice and show stable training in a profiled setting, but its effects across other first-token distributions remain uncertain.

[1] Diversity-incentivized exploration for versatile reasoning.

Questions for authors
  • How does REFT compare with DIVER under a matched rollout or compute budget? Are their gains complementary when the two methods are combined?

  • Does REFT still improve training when lower-ranked first tokens are substantially less likely to produce correct responses, as in the base-model example? What determines the range of settings in which uniform top-N sampling helps?

  • How large is the mismatch between REFT’s first-token sampling distribution and the policy across the evaluated settings, and how does it relate to training stability and performance without importance sampling correction?

4 comments
AuthorsOpenReview ↗

Rebuttal: Part 1 of 2

We are grateful for your encouraging assessment and the care you put into reviewing our work. We have added experiments and analyses to address your suggestions, as detailed below.

W1&Q1. Comparison and combination with DIVER

DIVER makes an important contribution by explicitly inducing the learning of various correct trajectories by reflecting diversity among responses as a reward. In contrast, REFT operates during the rollout generation phase and encourages broad exploration across the trajectory through dispersing and securing the starting points of the exploration. We evaluated DIVER-MIX with standard sampling, First-token T=2, and REFT under a matched rollout budget, using GRPO as the underlying RL algorithm. Math Avg. is the unweighted average over MATH-500, AIME24, AIME25, and AMC23.

  • Qwen2.5-7B-Instruct trained on BigMath
MethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Qwen2.5-7B-Instruct25.2349.5866.26
GRPO37.5855.3568.95
GRPO + DIVER-MIX38.3355.9871.24
First-token T=239.9856.1370.46
First-token T=2 + DIVER-MIX40.0355.8869.53
REFT43.8259.7271.54
REFT + DIVER-MIX43.8859.4673.68
  • Qwen2.5-3B-Instruct trained on GSM8K
MethodPass@1Pass@8Pass@64
Qwen2.5-3B-Instruct80.1494.1698.18
GRPO84.5395.3898.64
GRPO + DIVER-MIX85.1495.5398.10
First-token T=285.5995.8398.56
First-token T=2 + DIVER-MIX84.4695.6098.41
REFT86.2896.1399.39
REFT + DIVER-MIX86.0596.0699.32
  • Qwen2.5-3B-Instruct trained on BigMath-Easy
MethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Qwen2.5-3B-Instruct23.2745.7561.95
GRPO28.9148.3964.34
GRPO + DIVER-MIX28.8148.9263.51
First-token T=228.9148.2865.80
First-token T=2 + DIVER-MIX28.6648.2465.13
REFT32.1352.3867.82
REFT + DIVER-MIX31.4552.2367.52

REFT and REFT + DIVER-MIX outperform the other compared methods across all three settings. Overall, the performance trends remain broadly consistent when DIVER is added. To examine this result, we measured DIVER's raw diversity scores during training. Each cell reports ED / TD score.

SettingGRPO + DIVER-MIXFirst-token T=2 + DIVER-MIXREFT + DIVER-MIX
Qwen2.5-7B-Instruct, BigMath0.3563 / 0.87310.3706 / 0.89650.3761 / 0.9071
Qwen2.5-3B-Instruct, GSM8K0.6038 / 0.84820.5706 / 0.86070.6152 / 0.8667
Qwen2.5-3B-Instruct, BigMath-Easy0.3744 / 0.86070.3954 / 0.87550.3980 / 0.9014

REFT+DIVER has the highest raw ED and TD, showing that the REFT-based rollouts remain more diverse. However, the performance when combining DIVER was largely similar. One possible explanation is that the exploration effects of the two methods overlap to some extent. If REFT includes diverse trajectories during the rollout process, some of the exploration encouraged by DIVER’s reward may already be induced during generation, making the benefits of introducing diversity rewards relatively small.

Additionally, reward clipping may have had some influence. We also measured the fraction of reward-eligible correct responses whose ED or TD reward component exceeded its configured upper bound. Each cell reports ED clipping / TD clipping.

SettingGRPO + DIVER-MIXFirst-token T=2 + DIVER-MIXREFT + DIVER-MIX
Qwen2.5-7B-Instruct, BigMath34.89% / 27.68%38.20% / 38.40%41.27% / 47.64%
Qwen2.5-3B-Instruct, GSM8K66.72% / 16.49%62.32% / 20.11%67.84% / 24.70%
Qwen2.5-3B-Instruct, BigMath-Easy42.81% / 20.94%43.32% / 27.34%43.67% / 39.20%

REFT + DIVER-MIX has the largest clipped fractions. Higher clipped fractions indicate that some of the additional diversity produced by REFT cannot translate into a larger diversity bonus, and this may have also partially limited the additional gains of the combination. Details about DIVER are provided in Appendix C.2 and full benchmark results are in Appendix B.6 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal: Part 2 of 2

W2 & Q2. Applicability across a broader range of models

We extended our experiments to four base models spanning different families and additionally evaluated REFT with thinking enabled.

  • Qwen2.5-3B and Gemma3-4B-pt are trained on GSM8K:
Base modelMethodPass@1Pass@8Pass@64
Qwen2.5-3BPre-RLVR67.8593.7196.59
Qwen2.5-3BGRPO83.2494.9297.88
Qwen2.5-3BGRPO + REFT84.2395.3098.86
Gemma3-4B-ptPre-RLVR5.9919.8665.05
Gemma3-4B-ptGRPO56.1883.5593.63
Gemma3-4B-ptGRPO + REFT60.6584.5394.92
  • Olmo-3-1025-7B is trained on BigMath-Easy, and Qwen3.5-9B-Base on BigMath:
Base modelMethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Olmo-3-1025-7BPre-RLVR29.6647.5967.35
Olmo-3-1025-7BGRPO33.8456.0872.36
Olmo-3-1025-7BGRPO + REFT40.7964.0077.17
Qwen3.5-9B-BasePre-RLVR40.1860.4471.70
Qwen3.5-9B-BaseGRPO44.7866.2575.43
Qwen3.5-9B-BaseGRPO + REFT59.6574.9983.50

Additionally, we evaluated Qwen3.5-9B in thinking mode, trained on BigMath with a maximum completion length of 32,768 tokens:

MethodMath Avg. Pass@1Math Avg. Pass@8Math Avg. Pass@64
Qwen3.5-9B56.8276.3490.48
GRPO61.6681.3492.67
GRPO + REFT70.5284.5792.87

REFT consistently improves every reported sampling budget across these base models and the thinking-enabled setting with longer reasoning traces. These results show that prior instruction tuning is not essential for REFT to help, and that its benefits extend to longer reasoning traces, supporting REFT as a general rollout-exploration strategy. We added these results to Section 5.5, with per-benchmark details in Table 21 (Appendix B.9) of the revised manuscript.

W3 & Q3. First-token sampling mismatch and training stability

Thank you for raising this point. REFT samples lower-probability first tokens more frequently without an additional first-token importance correction. Across six settings, we measured the size of this shift and examined its relation to training stability and performance.

We measured KL and TV at the first token. For a fixed context c(x)c(x), each of the M=∣FN(x)∣M = |\mathcal F_N(x)| candidates has marginal probability 1/M1/M under REFT’s proposal qq. Using the full-vocabulary old-policy probability pf=πθold(f∣c(x))p_f = \pi_{\theta_{\mathrm{old}}}(f \mid c(x)), we estimate

DKL(q∥p)=Ef∼q[−log⁡(Mpf)]andTV(q,p)=1−Ef∼q[min⁡(1,Mpf)].D_{\mathrm{KL}}(q \Vert p) = \mathbb{E}_{f \sim q}[-\log(Mp_f)] \quad \text{and} \quad \mathrm{TV}(q,p) = 1 - \mathbb{E}_{f \sim q}[\min(1,Mp_f)].

Training settingFirst token KL (nats)First token TVBaseline gradient norm+REFT gradient normBaseline reward, start → end+REFT reward, start → end
Qwen2.5-3B-Instruct, GSM8K, GRPO6.7570.84770.036 (0.12)0.049 (0.14)1.678 → 1.9941.573 → 2.052
Qwen2.5-0.5B-Instruct, GSM8K, DAPO5.3940.86640.094 (0.17)0.153 (0.45)0.496 → 1.2780.388 → 1.399
Qwen2.5-14B-Instruct, BigMath, GRPO6.5220.85290.030 (0.08)0.038 (0.13)0.530 → 0.8190.515 → 0.848
Gemma3-4B-PT, GSM8K, GRPO1.7390.65090.140 (0.30)0.156 (0.49)0.298 → 1.3280.314 → 1.395
Qwen2.5-7B-Instruct, GSM8K, GRPO5.5720.82130.022 (0.20)0.034 (0.15)1.883 → 2.1071.946 → 2.131
Qwen3.5-9B thinking, BigMath, GRPO2.4330.67390.010 (0.05)0.022 (0.45)0.264 → 0.4580.286 → 0.553

Gradient norms are reported as mean (maximum) before gradient clipping. Since REFT samples first-token candidates uniformly, the first-token mismatch was substantial. REFT had higher mean gradient norms, while the fraction of tokens with policy ratios outside the clipping range was less than 6.37×10−36.37\times10^{-3} (0.637%). The reward and gradient-norm curves in Appendix E of the revised manuscript show no sustained instability or reward collapse. REFT also improves evaluation performance in the setting with the largest KL, Qwen2.5-3B-Instruct on GSM8K (Tables 1 and 3).

Thank the authors for the detailed reply and the comprehensive additional experiments. I will raise my score. Besides, an importance sampling correction can make the overall method more solid and sound.

We are happy that our response helped address your concerns. Thank you for your valuable suggestions, which have helped strengthen our work and will inform our revisions.

Comments on the paper

1 comment

General Response

We sincerely appreciate the reviewers’ careful assessments and constructive suggestions. The revised manuscript has been uploaded, and the updates are marked in blue. We completed the following additional experiments and analyses in response to the reviewers’ requests:

  • Broader model applicability (Reviewers DHZB and aMNp). We evaluated four base models spanning Gemma, OLMo, and Qwen and additionally tested Qwen3.5-9B with thinking enabled.

  • Additional diversification comparisons (Reviewer 5cj4). We extended the Table 4 comparisons with Rollout-MT, Token-MT, and LATR to the harder BigMath-Easy setting across three seeds.

  • Comparison and combination with DIVER (Reviewer DHZB). We added DIVER-MIX rewards to GRPO, First-token T=2T=2, and REFT across three settings and analyzed trajectory diversity and reward clipping.

  • Continuation learning (Reviewers 5cj4 and aMNp). We added crossed-policy and fixed-first-token continuation controls under GRPO, and clarified the comparison with first-token temperature adjustment.

  • Contribution beyond DAPO filtering (Reviewer aMNp). We compared uninformative-group fractions across three settings, distinguishing all-correct from all-wrong groups, and repeated the continuation controls under DAPO to assess learning benefits.

  • Sampling mismatch and training stability (Reviewer DHZB). We measured first-token KL and TV across six settings and examined training reward curves and gradient norms.

We hope these additional analyses and clarifications provide a clearer understanding of REFT. Thank you again for your insightful feedback, which has helped strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.