acceptodds
Under review as a conference paper at ICLR 2027

LaDi-RL: Latent Diffusion Reasoning Prevents Diversity Collapse in Reinforcement Learning

Abstract

Reinforcement learning has become a central paradigm for improving LLM reasoning, but most existing methods optimize policies over discrete token sequences. This creates a mismatch between the optimization space and the structure of reasoning: many important decisions are semantic, global, and trajectory-level rather than local token choices. Continuous latent-space RL offers a promising alternative, but the latent policy must model a complex, multi-modal distribution over valid reasoning trajectories. We therefore propose Latent Diffusion Reasoning with Reinforcement Learning (LaDi-RL), where a diffusion model generates latent reasoning trajectories through iterative denoising, enabling structured exploration and expressive distribution modeling. This comes with a credit-assignment challenge: the policy acts in latent space, but rewards are observed only after the latent is decoded into text, so a naive rollout cannot tell whether an incorrect answer comes from a poor latent trajectory or from an imperfect textual realization. We address this with hierarchical latent-text rollouts, which decode several answers per latent and score the latent by their mean reward, separating the quality of the reasoning from the luck of its decoding. Empirically, LaDi-RL outperforms token-level RL by 10.5% on code generation and 6.5% on math reasoning in pass@1, and raises pass@k above its pre-RL starting point, which token-level RL fails to do.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, latent diffusion, reasoning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_RfRxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between the optimization space and the structure of reasoning in reinforcement learning (RL) post-training for large language models (LLMs) by proposing Latent Diffusion Reasoning with Reinforcement Learning (LaDi-RL). The method employs a diffusion model to generate reasoning trajectories in a continuous latent space, enabling structured exploration and enhancing reasoning diversity. To address the credit assignment challenge between latent reasoning and text decoding, the authors introduce a hierarchical latent-text rollout mechanism, which generates multiple answers for each latent trajectory and uses their average reward to reduce the impact of decoding randomness on latent policy optimization. Additionally, a repulsion-based diversity guidance mechanism is incorporated to further encourage diverse exploration. Extensive experiments on code generation and mathematical reasoning benchmarks demonstrate the effectiveness of the proposed method in improving reasoning accuracy and mitigating diversity collapse.

Critical strengths

The paper is well-structured and presents a clear and logically coherent methodology. The experimental evaluation is comprehensive, covering multiple representative benchmarks and extensive comparisons with state-of-the-art methods. The results are convincing and provide strong empirical support for the effectiveness of the proposed approach.

Critical weaknesses
  1. This paper proposes a repulsion-based diversity guidance mechanism to encourage diverse exploration by adjusting the geometric distances between latent reasoning trajectories. Although Table 3 and Table 10 demonstrate the advantages of the proposed method in semantic diversity, and Table 4 and Figure 5 show that diversity guidance improves reasoning accuracy, the experiments still lack direct validation of the relationship between geometric changes in the latent space and semantic diversity. For example, how does semantic diversity change when diversity guidance is removed? What is the relationship between the geometric distances of latent trajectories and the differences in their resulting reasoning strategies? The paper does not provide targeted correlation analyses to address these questions.
  2. The paper proposes a hierarchical latent-text rollout mechanism that averages the rewards of multiple decoded outputs from the same latent reasoning trajectory to reduce the impact of decoding randomness on credit assignment for the latent policy. Although Table 5 and Appendix E.3 demonstrate the advantages of reward averaging over using a single randomly selected reward, the reliability of reward estimation under limited sampling remains insufficiently analyzed. In particular, the experiments use only four decoded outputs per latent trajectory by default. Whether such a small sample size is statistically sufficient to reliably estimate the expected reward and relative advantage of a latent trajectory remains unclear. Furthermore, since the averaged reward is influenced by both the quality of latent reasoning and the capability of the text decoder, it is unclear whether the proposed mechanism can effectively disentangle their respective contributions. Additional analyses of reward estimation variance, relative advantage stability, and credit assignment effectiveness under different sampling budgets would further strengthen the justification for this mechanism.
Questions for authors

see Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer RfRx (Part 1)

Thank you for the constructive review, and for recognizing our clear methodology, comprehensive evaluation, and convincing results.

W1: Latent geometry vs. semantic diversity

(a) Semantic diversity with and without guidance. Thank you for this question. For each problem, we sample 16 latent reasoning trajectories from the same model with guidance on and off, decode each into its reasoning text, and measure their semantic diversity (GPT-5 judge as in Table 3; code, five-benchmark avg.):

ModelGuidance offGuidance on
LaDiR.41.49
LaDi-RL.55.61

These results show that guidance increases the semantic diversity of the reasoning trajectories for both models.

(b) Latent distance vs. reasoning strategy. Latents that are close in Euclidean distance decode to similar reasoning, for three reasons:

  • The VAE is trained to ignore small Euclidean shifts. LaDiR's latents come from a VAE (Kingma & Welling, 2014). During training, Gaussian noise is added to each latent, and the decoder still has to reconstruct the same reasoning, so it learns to map nearby latents to the same reasoning. Text VAEs behave the same way: moving along a straight line in latent space changes the decoded text gradually (Bowman et al., 2016; Li et al., 2020).
  • The diffusion policy also works in Euclidean space. Flow matching adds Gaussian noise and is trained with an L2 loss, so its exploration is measured in Euclidean distance too.
  • Euclidean repulsion is a common choice for diversity. SVGD (Liu & Wang, 2016) and particle guidance for diffusion models (Corso et al., 2024) both diversify samples with Euclidean (RBF) kernels; SVGD also sets the bandwidth by the same median heuristic we use.

Experiment (new). For 500 code problems, we sample 16 latent reasoning trajectories per problem from LaDi-RL (guidance off, as at evaluation) and decode each into its reasoning text. For every pair of latents, GPT-5 rates how different the two reasoning strategies are (1 = same, 5 = completely different; Appendix E.5), and we record the Euclidean distance between the two latents. Grouping pairs by rating:

GPT-5 strategy difference1 (same)2345 (completely different)
Euclidean latent distance (mean ± SD)7.2±2.414.3±3.020.5±3.225.7±3.431.6±3.8

The Spearman correlation between latent distance and strategy difference is ρ = 0.76 (p < 0.001).

These results show that the more different two strategies are, the farther apart their latents are. We will add both analyses to Section 5.2 and the references to Section 3.2.

References

  • Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. ICLR.
  • Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating sentences from a continuous space. CoNLL.
  • Li, C., Gao, X., Li, Y., Peng, B., Li, X., Zhang, Y., & Gao, J. (2020). Optimus: Organizing sentences via pre-trained modeling of a latent space. EMNLP.
  • Liu, Q., & Wang, D. (2016). Stein variational gradient descent: A general purpose Bayesian inference algorithm. NeurIPS.
  • Corso, G., Xu, Y., De Bortoli, V., Barzilay, R., & Jaakkola, T. (2024). Particle guidance: Non-I.I.D. diverse sampling with diffusion models. ICLR.
AuthorsOpenReview ↗

Response to Reviewer RfRx (Part 2)

W2: Reliability of M = 4 and credit assignment

(a) Is M = 4 enough? Thank you for this careful question. We fix N = 4 latents and vary M (M = 1 and 4 are from Table 5; the rest are new). To check how reliable the M-answer estimate is, we take 500 RL prompts, decode 64 answers per latent, and treat their mean reward as the latent's true value:

M (answers per latent)124 (ours)816
Standard deviation of a latent's mean reward.41.29.21.15.10
Advantage has the same sign as with the true value71%80%88%93%96%
Final pass@1 (code avg.)41.770.877.077.978.3
Time per RL step0.35×0.6×1.0×1.8×3.4×

At M = 1, the text advantage is also zero, which further lowers its pass@1.

These results show that larger M is more stable, but beyond M = 4 pass@1 barely improves (+0.9 at 1.8× cost), so M = 4 is a good trade-off.

(b) Can the latent and decoder contributions be disentangled? Setup: for a prompt x, the latent policy samples a reasoning trajectory z, the decoder writes an answer y from x and z, and only y gets a reward R(y).

  • Decoder: yes. Its advantage compares answers written from the same z, so z's effect cancels out.
  • Latent: it can be disentangled from decoding noise, but not from the decoder. z never gets a reward of its own; it only gets feedback through the answers written from it. Its credit is therefore its expected reward E[R(y) | x, z] under the current decoder. Averaging makes this estimate less noisy, but it cannot remove the decoder. This is the standard situation for latent-variable models trained with REINFORCE (Mnih & Gregor, 2014) and for high-level policies in hierarchical RL (Sutton et al., 1999; Nachum et al., 2018).

We'd like to note that our paper already tests this separation (Table 4). Since both policies share one backbone, we keep two model copies: one is trained, the other stays frozen (code pass@1, avg. of 3):

SettingTrained copyFrozen copypass@1
LaDiR (start)––63.9
Text onlyDecoderLatent policy67.3
Latent onlyLatent policyDecoder70.3
BothBoth–75.6

These results show that each part improves on its own, the latent policy contributes more (70.3 vs. 67.3), and the two gains add up.

References

  • Mnih, A., & Gregor, K. (2014). Neural variational inference and learning in belief networks. ICML.
  • Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2), 181–211.
  • Nachum, O., Gu, S., Lee, H., & Levine, S. (2018). Data-efficient hierarchical reinforcement learning. NeurIPS.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Reviewer_98xHOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers RL for latent reasoning models, using diffusion with flow matching to generate latent reasoning trajectories. It uses a hierarchical rollout: first sampling multiple latents ZZ, then decoding several outputs YY from each latent. There are two possible sources of error: a bad latent or a bad decoding. To assign credit to each, the latent policy uses the average reward of its decoded outputs, while the decoder uses relative rewards among outputs from the same latent. Both are jointly fine-tuned using GRPO.

The method encourages diversity by checking the distances between latents and applying a repulsive force when they are too close. Distant latents are pulled slightly closer, and the guidance becomes weaker as denoising proceeds.

The results on code generation and mathematical reasoning, including comparisons with code-specific models and pass@k results, support the paper’s claim of improving performance while maintaining diversity.

Critical strengths
  1. The problem is very well motivated, and the solution is simple and elegant. Working in latent space can encourage more meaningful diversity beyond just lexical changes, while the hierarchical rollout helps separate the credit for the latent and the decoder. The paper is also well written and clearly structured.

  2. The comparisons, especially with code-specific models, strengthen the paper’s claims. The pass@k results also support the claim that the method improves performance while maintaining diversity.

Critical weaknesses
  1. There is no direct ablation/theoretical analysis showing that larger Euclidean distances between latents correspond to more semantically different reasoning strategies.

  2. The effects of A^\hat{A} and Aˉ\bar{A} are not completely decoupled, since the latent’s average reward still depends on the decoder’s output. Averaging reduces noise but does not separate latent policy from decoding policy.

Questions for authors

Regarding the weaknesses above, could you provide ablation studies/theoretical guarantees addressing these concerns?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 98xH

Thank you for the positive review, and for recognizing that the problem is well motivated, the solution simple and elegant, and the comparisons and pass@k results supportive of our claims.

W1: Do larger Euclidean distances mean different reasoning strategies?

Why Euclidean distance is meaningful in our latent space. Thank you for this question. Latents that are close in Euclidean distance decode to similar reasoning, for three reasons:

  • The VAE is trained to ignore small Euclidean shifts. LaDiR's latents come from a VAE (Kingma & Welling, 2014). During training, Gaussian noise is added to each latent, and the decoder still has to reconstruct the same reasoning, so it learns to map nearby latents to the same reasoning. Text VAEs behave the same way: moving along a straight line in latent space changes the decoded text gradually (Bowman et al., 2016; Li et al., 2020).
  • The diffusion policy also works in Euclidean space. Flow matching adds Gaussian noise and is trained with an L2 loss, so its exploration is measured in Euclidean distance too.
  • Euclidean repulsion is a common choice for diversity. SVGD (Liu & Wang, 2016) and particle guidance for diffusion models (Corso et al., 2024) both diversify samples with Euclidean (RBF) kernels; SVGD also sets the bandwidth by the same median heuristic we use.

Experiment (new). For 500 code problems, we sample 16 latent reasoning trajectories per problem from LaDi-RL (guidance off, as at evaluation) and decode each into its reasoning text. For every pair of latents, GPT-5 rates how different the two reasoning strategies are (1 = same, 5 = completely different; Appendix E.5), and we record the Euclidean distance between the two latents. Grouping pairs by rating:

GPT-5 strategy difference1 (same)2345 (completely different)
Euclidean latent distance (mean ± SD)7.2±2.414.3±3.020.5±3.225.7±3.431.6±3.8

The Spearman correlation between latent distance and strategy difference is ρ = 0.76 (p < 0.001).

These results show that the more different two strategies are, the farther apart their latents are. We will add this analysis to Section 5.2 and the references to Section 3.2.

W2: Are the latent and text advantages decoupled?

Thank you for this careful observation. Ā compares latents by the mean reward of their M answers; Â compares answers to the same latent.

  • Ā reduces decoding noise. Averaging over M answers removes the bad luck of any single answer from text decoder, so a good latent is not penalized for one poorly decoded answer. Ā still reflects how well the current decoder can realize the latent, since a latent is only useful through the answers decoded from it, just as a high-level policy in hierarchical RL is credited under the current low-level policy.
  • Â is fully decoupled from the latent. All M answers share the same latent, so the latent's contribution is the same for all of them and cancels when their mean is subtracted. Â only measures whether an answer is better or worse than its siblings: the decoder is rewarded for writing better answers, not for receiving a good latent.

Ablation. We'd like to note that the paper already tests this separation (Table 4, "uncoupled" rows). Since the latent and text policies share one backbone, we keep two copies of the model: one is trained with only one advantage, and the other is kept frozen to run the other policy. Each policy can thus be trained while the other stays exactly fixed (code pass@1, avg. of 3):

SettingTrained copyFrozen copypass@1
LaDiR (start)––63.9
Text onlyText decoder, with ÂLatent policy67.3
Latent onlyLatent policy, with ĀText decoder70.3
Latent + TextBoth–75.6

These results show that each advantage improves its own policy while the other is fixed, and the two gains add up. We will revise Section 3.1 to state precisely what is decoupled: Â is fully independent of the latent, while Ā still depends on the decoder's ability, and averaging removes only the decoding noise.

References

  • Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. ICLR.
  • Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating sentences from a continuous space. CoNLL.
  • Li, C., Gao, X., Li, Y., Peng, B., Li, X., Zhang, Y., & Gao, J. (2020). Optimus: Organizing sentences via pre-trained modeling of a latent space. EMNLP.
  • Liu, Q., & Wang, D. (2016). Stein variational gradient descent: A general purpose Bayesian inference algorithm. NeurIPS.
  • Corso, G., Xu, Y., De Bortoli, V., Barzilay, R., & Jaakkola, T. (2024). Particle guidance: Non-I.I.D. diverse sampling with diffusion models. ICLR.

Thank you for the detailed response. The new experiments address my concerns. I will keep my score.

Thank you for your positive feedback! We are glad that our additional experiments have addressed your concerns. We sincerely appreciate your time and support!

Reviewer_FAEhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Reinforcement learning (RL) is observed to often "sharpen" an LLM's distribution, instead of improving it. In this lens, RL methods take an LLM's existing pass@K rate and condenses it to a pass@1 rate. One hypothesis for why this behavior occurs is because rollouts contain insufficient diversity. Ladi-RL attempts to tackle this important problem. However, Ladi-RL identifies a key bottleneck in prior approaches: "semantic diversity is a global, trajectory-level property, but the LLM policy explores through local token-level perturbations." To remedy this, they propose generating trajectories in continuous latent space.

Their base model LaDiR is trained as an encoder-decoder model with a latent space learned with flow matching. To perform RL, they use a Flow-GRPO inspired method. In this setting, credit assignment is tricky because it is difficult to identify if low reward originates from a bad latent or bad decoding. To resolve this issue, they propose hierarchical rollouts that isolates the reward under each latent. Diversity guidance, through repulsion, is also added.

Finally, the authors demonstrate strong performance on a variety of models, benchmarks, baseline methods, and ablations.

Critical strengths
  • The paper identifies an incredibly important problem in RLVR -- lack of diversity -- and proposes a sound method to fix it
  • The paper has numerous strong experiments
Critical weaknesses

The overall method is quite involved, with numerous different parts and motivations. I am not reducing my score because of this, but I wanted to highlight this to the authors.

For example, the LaDiR model is trained with a stochastic decoding process, which introduces the credit assignment problem they later solve with hierarchical rollouts. It is not clear to me that a Monte Carlo estimate of multiple rewards from a single latent is enough to fix this credit assignment problem. Furthermore, additional continuous latent models can be trained that have deterministic, not stochastic, decoding procedures (e.g. learned embeddings with an arg max), which avoid this problem entirely.

A second example is that the method introduces repulsion to increase diversity of the samples. This makes it confusing if Ladi-RL on its own is sufficient for generating diverse samples. The authors provide ablations demonstrating that performance declines by pass@1 by 4.0, 2.0, and 3.2 points in Table 4. However, this makes the performance of Ladi-RL much closer to baselines reported in Table 2.

Questions for authors
  1. Can you (further) clarify in the manuscript why more diverse rollouts are intended to help prevent the sharpening problem RL often encounters?
6 comments
AuthorsOpenReview ↗

Response to Reviewer FAEh (Part 1)

Thank you for the thoughtful review, and for recognizing that diversity collapse is an important problem in RLVR, that our method addresses it soundly, and that the experiments are strong.

W1: Is Monte Carlo averaging enough, and why not deterministic decoding?

(a) Why Monte Carlo averaging fixes credit assignment. Thank you for this question. The problem is that a single reward mixes two things: how good the latent is, and how lucky that particular decoding was. For latent znz_n and its answers m=1,…,Mm = 1, \dots, M, we write

Rn,m=V(zn)+ϵn,m,E[ϵn,m]=0,R_{n,m} = V(z_n) + \epsilon_{n,m}, \qquad \mathbb{E}[\epsilon_{n,m}] = 0,

where V(zn)V(z_n) is the latent's expected reward and ϵn,m\epsilon_{n,m} is the decoding luck. Hierarchical rollouts give each policy only its own part:

  • Latent advantage: average over answers. The luck averages out, with variance reduced by MM, so latents are compared by their quality: Rˉn=1M∑m=1MRn,m=V(zn)+ϵˉn,Var(ϵˉn)=Var(ϵ)M.\bar R_n = \frac{1}{M}\sum_{m=1}^{M} R_{n,m} = V(z_n) + \bar\epsilon_n, \qquad \mathrm{Var}(\bar\epsilon_n) = \frac{\mathrm{Var}(\epsilon)}{M}.
  • Text advantage: subtract that average. V(zn)V(z_n) cancels exactly, so the decoder is judged only against other answers to the same latent: Rn,m−Rˉn=ϵn,m−ϵˉn.R_{n,m} - \bar R_n = \epsilon_{n,m} - \bar\epsilon_n.

Intuitively, averaging tells us how good a latent is, and comparing answers within a latent tells us how well the decoder wrote the answers.

(b) Why stochastic decoding. We'd like to clarify that this choice comes from the shared backbone, not from a preference between deterministic and stochastic decoding:

  • The problem. The latent policy and the decoder share one backbone, so every latent update also changes the decoder. Without a text-level gradient to correct this, the decoder drifts and training collapses (46.2, Table 4).
  • Our solution. Stochastic decoding gives several different answers per latent, so the decoder can also be trained by RL.
  • The alternative. One can instead load a second, frozen copy of the model as the decoder. This avoids the drift, and deterministic decoding then works well, as the reviewer suggests. But it needs two models in memory, and the decoder can no longer improve.

Code pass@1 (avg. of MBPP, HumanEval, LCB V6; Table 4 + one new row):

VariantModel copiesDecoder trainedDecodingpass@1
LaDiR (start)1––63.9
Latent only1No (drifts)Sampled46.2
Latent only + frozen decoder2NoSampled70.3
Latent only + frozen decoder (new)2NoGreedy70.6
LaDi-RL1YesSampled75.6

These results show that a shared backbone collapses without a text gradient; a frozen copy avoids this but costs an extra model, while LaDi-RL trains both parts in one model and performs best (+5.0).

AuthorsOpenReview ↗

Response to Reviewer FAEh (Part 2)

W2: Is LaDi-RL sufficient without diversity guidance?

Thank you for this careful observation. We'd like to note that without guidance, LaDi-RL still outperforms the strongest baselines (p@1 / p@128, mean ± SD over 3 seeds):

MethodAIME25OlympiadBenchMBPP+LCB V6
Multiplex Thinking19.7† / 56.7±1.841.7† / 67.9±0.766.8±0.6 / 84.7±0.742.9±0.9 / 62.8±1.0
ReLaX19.2±0.6 / 54.4±1.940.9±0.4 / 65.8±0.869.6±0.5 / 85.5±0.745.6±0.8 / 63.4±0.9
PKPO18.6±0.7 / 57.3±1.939.8±0.5 / 68.3±0.767.5±0.6 / 86.4±0.643.8±0.9 / 64.6±0.9
LaDi-RL w/o guidance20.8±0.5 / 61.1±1.543.6±0.3 / 71.2±0.672.9±0.4 / 88.6±0.549.4±0.7 / 67.3±0.8
LaDi-RL21.0±0.5 / 63.3±1.444.3±0.3 / 72.4±0.575.1±0.4 / 89.6±0.452.6±0.6 / 68.4±0.7

†Reported by Tang et al. (2026), as in Table 1.

These results show that latent diffusion alone beats the strongest baselines on every benchmark in both pass@1 and pass@128; guidance adds a further gain.

Q1: Why do diverse rollouts help prevent sharpening?

Thank you for this question; we will clarify it in Section 1.

  • What sharpening is. As RL training proceeds, the policy concentrates on a few modes: entropy and diversity drop, and rollouts for the same prompt become nearly identical. When all rollouts in a group get the same reward, the group-relative advantage is zero, so GRPO has no signal left to improve. Pass@1 may rise, but other correct solutions are lost, and pass@k at large k may be below the starting model.
  • Why text RL sharpens. Text RL acts in a discrete space: probabilities over the vocabulary sum to one, so raising the probability of one token necessarily lowers all others. Repeated over many tokens, the policy collapses onto the few sequences it already samples well.
  • Why LaDi-RL is less prone to it. A diffusion model naturally represents a multi-mode distribution: different initial noises are denoised into different reasoning modes. RL updates a Gaussian transition at a denoising step, which shifts each sample locally in continuous space toward better reasoning. RL can therefore raise the probability of several high-reward modes at the same time, rather than raising one mode by lowering the others.

Evidence (code, five-benchmark avg.; start = SFT for GRPO and LaDiR for LaDi-RL; last column is new):

MethodSemantic diversity (start → end)pass@128 (start → end)Groups with zero reward variance (end)
Text RL (GRPO).27 → .1883.3 → 82.746%
LaDi-RL.39 → .5287.1 → 90.414%

These results show that GRPO sharpens: diversity and coverage drop, and nearly half of its groups give no learning signal. LaDi-RL raises diversity and coverage together, and keeps most groups informative. It also keeps more distinct correct strategies (semantic diversity .36 vs. .14 on correct solutions, Table 10).

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Comments on the paper

3 comments
Area_Chair_qzAgOpenReview ↗

ICLR2027 Discussiion Reminder

Dear Reviewers,

Thank you very much for your time and efforts in reviewing for ICLR 2027.

The authors have posted their responses to your comments and questions. As the author–reviewer discussion period is approaching its end, I would appreciate it if you could review their responses and engage in further discussion as needed, particularly if any of your concerns remain unresolved.

Thank you again for your valuable contributions!

Best regards, AC

AuthorsOpenReview ↗

General Response (Part 1/2)

We thank all reviewers for their constructive feedback.

What reviewers found strong

  • Problem. Lack of diversity in RLVR is an important problem (FAEh), and the paper motivates it well (98xH).
  • Method. The solution is simple and elegant (98xH) and sound (FAEh), and the methodology is clear and logically coherent (RfRx). Exploring in latent space gives more meaningful diversity than lexical changes, and the hierarchical rollout separates the credit of the latent from that of the decoder (98xH).
  • Experiments. The evaluation is comprehensive and convincing, with many benchmarks and comparisons to state-of-the-art methods (RfRx). The experiments are numerous and strong (FAEh). The comparisons with code-specific models and the pass@k results support the claims (98xH).
  • Presentation. The paper is well written and clearly structured (98xH, RfRx).

Headline results: LaDi-RL improves pass@1 over vanilla GRPO by 10.5 points on code and 6.5 on math, and its pass@k stays above its LaDiR starting point at every k on all five pass@k benchmarks, while GRPO's pass@128 falls below its SFT starting point on the three code benchmarks.

Main concerns and how we addressed them

1. Does latent distance reflect different reasoning strategies, and does guidance raise semantic diversity? (RfRx, 98xH) (98xH: resolved)

  • The more different two strategies are, the farther apart their latents are (new experiment). On 500 code problems, GPT-5 rates how different the strategies of two latents are. The mean Euclidean distance rises from 7.2 for "same strategy" to 31.6 for "completely different" (Spearman ρ = 0.76, p < 0.001; new Figure 5a).
  • Guidance raises semantic diversity (new experiment). With guidance on, the diversity of the decoded reasoning text rises from .41 to .49 for LaDiR and from .55 to .61 for LaDi-RL (new Figure 5c). Table 3 scores final code instead, so its values differ slightly.
  • We also explain why Euclidean distance is meaningful in this space, with references. Nearby VAE latents decode to similar reasoning, and RBF repulsion with a median bandwidth is a standard way to diversify samples (SVGD, particle guidance).

2. Is M = 4 answers per latent enough, and are the latent and decoder credits separated? (RfRx, 98xH) (98xH: resolved)

  • Larger M is more reliable, but M = 4 is a good trade-off (new experiment). The latent advantage has the same sign as a 64-answer reference in 71%, 88%, and 96% of cases for M = 1, 4, and 16. Beyond M = 4, pass@1 gains only +0.9 at 1.8× the time per RL step (new Figure 5b).
  • The text advantage is fully independent of the latent. It compares answers decoded from the same latent, so the latent's value cancels exactly.
  • The latent advantage removes decoding noise but still depends on the decoder. Averaging M rewards divides the noise variance by M. As for a high-level policy in hierarchical RL, a latent is credited by how well the current decoder realizes it.
  • Each part improves on its own, and the gains add up (Table 4, original paper). With a frozen copy running the other policy, training only the decoder reaches 67.3, only the latent policy 70.3, and both 75.6, starting from 63.9.
AuthorsOpenReview ↗

General Response (Part 2/2)

3. Why stochastic decoding instead of a deterministic decoder? (FAEh) (FAEh: resolved)

Method differences.

  • In LaDi-RL, the latent policy and the decoder share one backbone, so every latent update also changes the decoder.
  • A deterministic decoder needs a second, frozen model copy. It then avoids decoder drift, but it doubles the models in memory and the decoder can no longer improve.

Results (code pass@1, Table 4 and one new row).

  • With a shared backbone and no text gradient, training collapses (46.2, Table 4).
  • With a frozen decoder copy, greedy decoding gives 70.6 (new experiment, App. E.10), close to sampling (70.3, Table 4).
  • LaDi-RL trains both parts in one model and performs best (75.6, Table 4).

4. Is LaDi-RL sufficient without diversity guidance? (FAEh) (FAEh: resolved)

  • Without guidance, LaDi-RL still beats the strongest baselines (new experiment). It beats Multiplex Thinking, ReLaX, and PKPO on AIME25, OlympiadBench, MBPP+, and LCB V6, in both pass@1 and pass@128, over 3 seeds (new Table 6 in the main text).
  • For example, on LCB V6 it reaches 49.4 pass@1, compared with 45.6 for ReLaX. Guidance adds a further gain, to 52.6.

5. Why do diverse rollouts help prevent sharpening? (FAEh) (FAEh: resolved)

  • The mechanism. When all rollouts in a group get the same reward, GRPO's group advantage is zero and learning stalls. A diffusion policy can raise several high-reward modes at once instead of raising one mode at the cost of others.
  • GRPO sharpens; LaDi-RL does not (Table 3, original paper). GRPO lowers semantic diversity (.27 to .18) and pass@128 (83.3 to 82.7) from its start. LaDi-RL raises both (.39 to .52 and 87.1 to 90.4).
  • LaDi-RL keeps most groups informative (new experiment). At the end of training, 46% of GRPO's groups have zero reward variance, compared with 14% for LaDi-RL (App. E.11).

Revised PDF

Below, each reviewer's request is followed by where we added it. Changes are coloured by reviewer (98xH blue, FAEh green, RfRx orange, shared purple).

FAEh

  • Is Monte Carlo averaging enough for credit assignment? → App. E.3, paragraph "What the two advantages separate"; Sec. 5.2, new Figure 5b.
  • Why stochastic rather than deterministic decoding? → Sec. 5.2, new paragraph "Why stochastic decoding with a shared backbone"; App. E.10 "Frozen Decoder with Greedy Decoding" (Table 13).
  • Is LaDi-RL sufficient without diversity guidance? → Sec. 5.2, new Table 6 and the "Diversity guidance" paragraph.
  • Why do diverse rollouts prevent sharpening? → Sec. 1, paragraph beginning "Effective latent-space exploration"; Sec. 5.2, new paragraph "Why diversity helps avoid sharpening"; App. E.11 "Diverse Rollouts and Sharpening" (Table 14).

98xH

  • Do larger Euclidean distances mean different reasoning strategies? → Sec. 3.2, "Diversity Guidance" paragraph (why Euclidean distance is meaningful); Sec. 5.2, new Figure 5a and the paragraph "Latent distance and reasoning strategy".
  • Are the latent and text advantages decoupled? → Sec. 3.1 "Hierarchical Rollout", end of the first paragraph; App. E.3, paragraph "What the two advantages separate".

RfRx

  • Semantic diversity with vs. without guidance? → Sec. 5.2, new Figure 5c and the "Diversity guidance" paragraph; protocol in App. E.8.
  • Latent distance vs. reasoning strategy? → Sec. 5.2, new Figure 5a and the paragraph "Latent distance and reasoning strategy"; protocol in App. E.8.
  • Is M = 4 answers per latent enough? → Sec. 5.2, new Figure 5b and the "Hierarchical Rollout" paragraph; App. E.9 "Number of Answers per Latent" (Table 12).
  • Can latent and decoder credit be separated? → Sec. 3.1 "Hierarchical Rollout"; App. E.3, paragraph "What the two advantages separate".

The main text stays within 10 pages. Because Figure 5 and Table 6 are new, later figure and table numbers shift by one.

Discussion status

  • Reviewer FAEh: replied that the new results "address my main concerns" and maintains a positive score of 3. All the three suggested clarifications are now in the revised PDF.
  • Reviewer 98xH: replied that "the new experiments address my concerns" and keeps the score of clear acceptance of 4.
  • Reviewer RfRx: no reply yet.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.