acceptodds
Withdrawn from ICLR 2027

Dropout-GRPO: Variational Attention Dropout for Continuous Latent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models(LLMs) have achieved considerable success in various reasoning tasks by utilizing Chain-of-Thought(CoT) reasoning with intermediate language tokens. An emerging family of models utilize the latent continuous space of the LLMs in pursuit of robustness and efficiency, bypassing the restrictions of linguistic fluency and verbosity. However, often such models are deterministic by design and require complex architectural modifications to achieve variation in the reasoning trajectory necessary for critic-free post-training methodologies such as GRPO. We investigate dropout as a lightweight mechanism for exploring alternative latent trajectories without redesigning the recurrence. By applying dropout to attention probabilities, we perturb the continuous latent trajectories. We use rollout-specific attention-mask patterns shared across latent steps and replay the corresponding dropout realization during policy updates. This supplies rollout diversity without changing the latent recurrence architecture. We evaluate this approach on arithmetic benchmarks and support it with a mathematical analysis of replay and the scope of the resulting policy update. The results validate that latent reasoning models can benefit from post-training through reinforcement learning with verifiable rewards by using dropout. The corresponding code is shared via supplementary materials, and the repository will be made public upon acceptance of the paper.

Primary area:
reinforcement learning
Keywords:
latent reasoning, reinforcement learning, dropout

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vi8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose attention dropout as a mechanism for exploration in Coconut-style continuous latent reasoning. They introduce Dropout-GRPO, where dropout is applied during prompt processing, latent recurrence, and answer generation, with Bernoulli masks shared across latent steps with overlapping attention positions and replayed during policy updates. The experimental validation is performed with Qwen2.5-1.5B, achieving gains in pass@1 across mathematical reasoning benchmarks compared to the base model and other Coconut-bootstrapped training (SFT, RL) recipes. They also introduce theoretical contributions, in the form of a policy-gradient analysis of dropout-induced exploration, analyzing the training gradients and explaining the role of mask replay.

Critical strengths
  1. Dropout-GRPO is a pretty interesting mechanism to introduce latent computation and exploration by perturbing the computation without requiring a distribution over tokens for each latent state. This gives us a useful and practical instantiation of RL for latent reasoning that seems to be somewhat effective.

  2. Appendix B builds up a series of theoretically relevant properties on the core premise (dropout and mask replay). Also, the ablation in Appendix F.2 (inference-time enabling vs. disabling dropout) is interesting.

Critical weaknesses
  1. Experimental validation is only performed with Qwen2.5-1.5B; it is important to demonstrate empirical efficacy both across sizes and across model families. In particular, the Qwen2.5 models have been shown to achieve strong performance with random or even flipped rewards [1], and as such, despite achieving gains, it is unclear whether this is truly an outcome of this mechanism.

  2. Appendix C.2 points out that there is a mismatch between the theoretical implementation and the experimental results, which use an opposite log-ratio for the regularization without replacing it to the corrected one. It is thus unclear whether correcting this to accurately reflect the algorithm described in this paper would have an impact on its empirical performance and our takeaways on its efficacy.

[1] Spurious Rewards: Rethinking Training Signals in RLVR, Shao et al. 2025

Questions for authors
  1. Could you rerun the main experiments and regularizer analyses with the corrected log-ratio, and separate out the results using the old regularization?

  2. What would happen if you were to disable dropout but keep the same stochastic answer sampling, number of rollouts, filtering, etc. (aside from the analysis in Table 5)? And could you try latent-only vs. prompt- and answer-only dropout to see what is giving the biggest performance lift?

Reviewer_REnzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of applying GRPO to COCONUT-style continuous latent-reasoning models, in which deterministic latent recurrence yields identical trajectories across rollouts and thus lacks exploration. The authors propose Dropout-GRPO, which activates attention dropout to diversify latent trajectories without architectural changes, and replays stored RNG states to preserve rollout likelihoods, supported by a mathematical analysis of mask replay and policy-gradient properties. The authors experimented on Qwen2.5-1.5B and show pass@1 gains over the COCONUT SFT baseline.

Critical strengths
  1. The proposed method utilizes native attention dropout to provide exploration for COCONUT-style latent recurrence without modifying the model architecture. Unlike vocabulary-based latent approaches that reimpose a discrete bottleneck, this mechanism perturbs continuous states without additional computate.

  2. The authors provide theoretical analysis of conditional likelihood consistency under mask replay and a scaled policy-gradient identity for an idealized sequence-sum objective. Moreover, the paper also reports practical implementation choices, such as sequence-mean aggregation etc. apart from idealized unbiased-gradient guarantees.

Critical weaknesses
  1. The absolute accuracy of the trained model remains modest and still lags far behind the zero-shot Qwen2.5-1.5B. Moreover, although computational efficiency is a primary motivation for continuous latent reasoning, the paper provides no empirical analysis of inference latency/token savings.

  2. Exact mask replay relies on matching tensor shapes, batch schedules, and specific attention backends, meaning that saving RNG states alone cannot prevent training–inference mismatch when the rollout engine and training engine differ.

  3. Alternative RL approaches for latent reasoning, such as HRPO and LEPO, are only briefly mentioned without any empirical comparison. Including empirical baselines or a deeper comparative discussion against hybrid/relaxed latent RL methods can further improve the paper presentation.

Questions for authors

n/a

Reviewer_jz1DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

if you try to run GRPO on a COCONUT-style model, the latent trajectory is a deterministic function of the prompt, so with greedy decoding every rollout in a group comes out identical, the advantages are all zero, and nothing trains. The fix is to leave attention dropout switched on during rollouts (prompt, latent steps, and answer decoding), remember the state, and replay the exact same masks when scoring the answer for the policy update

Critical strengths

The idea is simple and genuinely cheap. Reusing the dropout that already lives in the attention layers as a source of replayable noise means no extra parameters, no projection back onto the vocabulary, no change to the recurrence. If the gain is real, that’s a nice property compared to something like LEPO, and it’s the kind of trick I’d expect people to actually adopt.

Critical weaknesses

The “deterministic GRPO” row in Table 3 uses greedy decoding that’s why it matches the SFT model which is arithmetic I think. What I suggest is to see GRPO with sampled answer tokens and dropout off.

The baseline is one seed, the method is three. Could you add more seeds, I don’t think we can say anything about the robustness at this point

Questions for authors

What does GRPO get with sampled answer tokens and dropout off, same budget? And can you split dropout by phase (latent only vs. decoding only)? Which split was used for checkpoint selection? Was any test set involved? Why isn’t LEPO compared empirically, given it’s the closest method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.