- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Group relative RLVR depends on informative rollout groups. Uniform rewards provide no correctness contrast, while repetitive successful responses may limit exploration. The paper proposes DRIFT to steer autoregressive generation through logit perturbations at branch points identified by the effective number of candidate tokens. Each perturbation combines random noise with a drift vector updated online using entropy at subsequent branch points. A per token importance weight partially corrects for the modified sampling distribution. Experiments on Qwen3 models with 4B and 8B parameters using GRPO and DAPO report average improvements of 1.52 to 3.08 percentage points across five training seeds. Additional experiments examine rollout group quality, compare exploration methods, and evaluate code generation.
Critical strengths
DRIFT addresses rollout quality directly and composes with GRPO and DAPO without an additional value model or external teacher. Results across five seeds and two model sizes support its empirical usefulness. Comparisons with random steering at matched perturbation magnitude and temperature sampling at matched average KL strengthen the evaluation. Analysis using a fixed checkpoint also shows improvements in correctness, coverage, and textual diversity among correct responses.
Critical weaknesses
-
The motivation targets correct and diverse rollouts, but the drift update uses only entropy at subsequent branch points. Preserving entropy does not establish higher continuation value. The theoretical benefit remains conditional on positive value alignment, and the experiments do not directly validate this condition for the proposed estimator.
-
Transferring the drift across coordinates ordered by probability rank within the top tokens requires further justification. Token identities and continuation values change between branch points, yet no ablation establishes whether the carried direction remains useful.
-
The practical effect of the per token importance correction is unclear. Appendix F.8 acknowledges its bias relative to the on policy gradient, but the experiments neither isolate its contribution nor report the behavior of the correction weights.
-
The additional computational and time overhead is not quantified, leaving the benefit under comparable computational resources unclear.
Questions for authors
-
Does entropy based drift improve continuation success rates and achieve positive value alignment at branch points?
-
How does performance change when the carried direction’s coordinates are shuffled or momentum is removed?
-
How does performance change without importance correction, and how are the correction weights distributed during training?
-
What additional computation and time does DRIFT require? Do its gains persist under comparable computational resources?
-
Do correct responses show different solution strategies, or mainly differences in wording and length?
2 comments
Thank you for providing the concrete examples. I will raise my score.
Thank you for the additional experiments. Although the reliability of the entropy based estimator remains uncertain, the potential to generate more diverse correct solutions is encouraging.
The new analysis measures diversity through gradient similarity. Can you provide several comparable examples from DRIFT and GRPO on the same questions, illustrating whether the differences reflect distinct reasoning strategies? If these examples further substantiate meaningful solution diversity, I will raise my score to 3.