acceptodds
Under review as a conference paper at ICLR 2027

Learned DRIFT on the Logit Manifold for LLM Reasoning Exploration

Abstract

Reinforcement learning from verifiable rewards (RLVR) can only reinforce what it samples, which makes the rollout group the bottleneck: the policy can progress no further than the spread of rewards in the group allows, and only along the trajectories it has already sampled. Existing exploration methods push on both axes blindly, temperature scaling, entropy regularization, and looser clipping widening the sampling distribution in every direction at once while group filtering simply samples harder. We introduce **DRIFT** (**D**irected **R**ollout with **I**nferred **F**eedback at **T**riggers), which applies a Langevin step to steer generation itself, to produce groups with diverse rewards and trajectories. The optimal drift is the gradient of expected reward in logit space, an expectation over completions that no rollout can evaluate before it terminates. With a first-order expansion, we show that a step along any partially aligned direction earns the optimal drift's gain scaled by the cosine between the two. DRIFT therefore estimates such a direction online, from prior steering feedback, using the entropy remaining at branch points as its signal. Training on Qwen3-4B-Base and Qwen3-8B-Base, DRIFT improves reasoning performance under both GRPO and DAPO, raising the benchmark average by 1.5 to 3.1 points in every pairing and AIME accuracy by up to 4.6 points.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Reinforcement Learning, Post-training, Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Qi44OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Group relative RLVR depends on informative rollout groups. Uniform rewards provide no correctness contrast, while repetitive successful responses may limit exploration. The paper proposes DRIFT to steer autoregressive generation through logit perturbations at branch points identified by the effective number of candidate tokens. Each perturbation combines random noise with a drift vector updated online using entropy at subsequent branch points. A per token importance weight partially corrects for the modified sampling distribution. Experiments on Qwen3 models with 4B and 8B parameters using GRPO and DAPO report average improvements of 1.52 to 3.08 percentage points across five training seeds. Additional experiments examine rollout group quality, compare exploration methods, and evaluate code generation.

Critical strengths

DRIFT addresses rollout quality directly and composes with GRPO and DAPO without an additional value model or external teacher. Results across five seeds and two model sizes support its empirical usefulness. Comparisons with random steering at matched perturbation magnitude and temperature sampling at matched average KL strengthen the evaluation. Analysis using a fixed checkpoint also shows improvements in correctness, coverage, and textual diversity among correct responses.

Critical weaknesses
  1. The motivation targets correct and diverse rollouts, but the drift update uses only entropy at subsequent branch points. Preserving entropy does not establish higher continuation value. The theoretical benefit remains conditional on positive value alignment, and the experiments do not directly validate this condition for the proposed estimator.

  2. Transferring the drift across coordinates ordered by probability rank within the top KK tokens requires further justification. Token identities and continuation values change between branch points, yet no ablation establishes whether the carried direction remains useful.

  3. The practical effect of the per token importance correction is unclear. Appendix F.8 acknowledges its bias relative to the on policy gradient, but the experiments neither isolate its contribution nor report the behavior of the correction weights.

  4. The additional computational and time overhead is not quantified, leaving the benefit under comparable computational resources unclear.

Questions for authors
  1. Does entropy based drift improve continuation success rates and achieve positive value alignment at branch points?

  2. How does performance change when the carried direction’s coordinates are shuffled or momentum is removed?

  3. How does performance change without importance correction, and how are the correction weights distributed during training?

  4. What additional computation and time does DRIFT require? Do its gains persist under comparable computational resources?

  5. Do correct responses show different solution strategies, or mainly differences in wording and length?

2 comments

Thank you for the additional experiments. Although the reliability of the entropy based estimator remains uncertain, the potential to generate more diverse correct solutions is encouraging.

The new analysis measures diversity through gradient similarity. Can you provide several comparable examples from DRIFT and GRPO on the same questions, illustrating whether the differences reflect distinct reasoning strategies? If these examples further substantiate meaningful solution diversity, I will raise my score to 3.

Thank you for providing the concrete examples. I will raise my score.

Reviewer_itQWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DRIFT aims to improve rollout quality in RLVR by steering generation at selected branch points. It perturbs the top-KK logits using a learned direction mixed with random noise. The direction is updated within each rollout using entropy feedback from later branches. A per-token importance correction accounts for the modified sampler during training. Experiments on Qwen3 models show improvements under both GRPO and DAPO.

Critical strengths
  1. The paper targets a real bottleneck: poor rollout quality limits what group-relative training can learn.
  2. The method needs no extra value model or search tree. It uses existing logits and a small rollout-local state.
Critical weaknesses
  1. The link between entropy feedback and correctness remains unclear. Higher entropy may reflect confusion rather than useful exploration. The theory requires reward alignment but does not establish it.
  2. DRIFT carries the learned direction across branches by token probability rank. For example, a preference for the second-ranked token is reused at the next branch, even if that token has a different meaning. It is unclear why this rank-based transfer should remain useful.
  3. Token edit distance measures textual variation. It does not establish diversity in solution strategies.
Questions for authors
  1. On a manageable subset of fixed prefixes, could you estimate candidate continuation values through repeated completions and report the empirical alignment between the learned direction and the estimated branch-value gradient, including how alignment changes over a rollout?

  2. Could you compare the current direction-transfer mechanism with coordinate shuffling between branches, resetting the accumulated direction, and a fixed rank-based perturbation, while keeping the sampling budget matched?

  3. Could you supplement the edit-distance results with an analysis of distinct solution strategies—for example, a blinded manual classification of correct solutions on a representative subset?

Reviewer_yR7nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a directed exploration method DRIFT for group relative RLVR settings. The authors propose that a Langevin-style steering step on the logits at critical branching points can help increase rollout diversity and correctness in a group. The drift term steers towards possibly high reward continuations and the diffusion term preserves coverage. DRIFT estimates this steering direction using an entropy based estimate in an online setting. Theoretical analysis for the steering mechanism as well as empirical results are provided. DRIFT shows improvement in group rollout quality and shows consistent gains across most evaluation metrics.

Critical strengths
  • [S1] Novel method to perform directed exploration by applying Langevin step based steering to logits at critical branching points. The model weights need not be modified during rollout generation to promote exploration. Simple advantage correction for off-policy action at critical branching points.
  • [S2] DRIFT shows consistent performance across most evaluation metrics. Interestingly DRIFT generally shows improvements to GPQA-Diamond despite only being trained on a math subset of the Guru reasoning corpus.
  • [S3] Theoretical analysis is provided.
  • [S4] DRIFT is also potentially applicable to other domains as demonstrated by performance in coding tasks.
Critical weaknesses
  • [W1] Prior work on critical branching tokens is not addressed.
  • [W2] Wall-clock time showing DRIFT's overhead compared to baseline not shown.
  • [W3] Only two models are considered, both belonging to the Qwen3 family.
  • [W4] Since the mixing weight α\alpha plays an important role, beyond the α=0\alpha=0 ablation, sensitivity analysis of it experimentally would have been illuminating. Similarly the role of response length in the choice of entropy target could also have been explored experimentally, since that would have a role in deciding how long to keep branching depending on the domain of application.
Questions for authors
  • [Q1] Can the authors provide a sensitivity analysis for α\alpha and discuss how its design affects the performance as hinted in future work section?
  • [Q2] Authors are encouraged to discuss prior work about critical branching tokens. They are also encouraged to discuss papers which explicitly try to improve exploration focusing on such critical branching tokens.

[1] Wang, Shenzhi, et al. "Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning." NeurIPS 2025.

[2] Lin, Zicheng, et al. "Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability" ICML 2025.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.