- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose attention dropout as a mechanism for exploration in Coconut-style continuous latent reasoning. They introduce Dropout-GRPO, where dropout is applied during prompt processing, latent recurrence, and answer generation, with Bernoulli masks shared across latent steps with overlapping attention positions and replayed during policy updates. The experimental validation is performed with Qwen2.5-1.5B, achieving gains in pass@1 across mathematical reasoning benchmarks compared to the base model and other Coconut-bootstrapped training (SFT, RL) recipes. They also introduce theoretical contributions, in the form of a policy-gradient analysis of dropout-induced exploration, analyzing the training gradients and explaining the role of mask replay.
Critical strengths
-
Dropout-GRPO is a pretty interesting mechanism to introduce latent computation and exploration by perturbing the computation without requiring a distribution over tokens for each latent state. This gives us a useful and practical instantiation of RL for latent reasoning that seems to be somewhat effective.
-
Appendix B builds up a series of theoretically relevant properties on the core premise (dropout and mask replay). Also, the ablation in Appendix F.2 (inference-time enabling vs. disabling dropout) is interesting.
Critical weaknesses
-
Experimental validation is only performed with Qwen2.5-1.5B; it is important to demonstrate empirical efficacy both across sizes and across model families. In particular, the Qwen2.5 models have been shown to achieve strong performance with random or even flipped rewards [1], and as such, despite achieving gains, it is unclear whether this is truly an outcome of this mechanism.
-
Appendix C.2 points out that there is a mismatch between the theoretical implementation and the experimental results, which use an opposite log-ratio for the regularization without replacing it to the corrected one. It is thus unclear whether correcting this to accurately reflect the algorithm described in this paper would have an impact on its empirical performance and our takeaways on its efficacy.
[1] Spurious Rewards: Rethinking Training Signals in RLVR, Shao et al. 2025
Questions for authors
-
Could you rerun the main experiments and regularizer analyses with the corrected log-ratio, and separate out the results using the old regularization?
-
What would happen if you were to disable dropout but keep the same stochastic answer sampling, number of rollouts, filtering, etc. (aside from the analysis in Table 5)? And could you try latent-only vs. prompt- and answer-only dropout to see what is giving the biggest performance lift?