- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the mismatch between the optimization space and the structure of reasoning in reinforcement learning (RL) post-training for large language models (LLMs) by proposing Latent Diffusion Reasoning with Reinforcement Learning (LaDi-RL). The method employs a diffusion model to generate reasoning trajectories in a continuous latent space, enabling structured exploration and enhancing reasoning diversity. To address the credit assignment challenge between latent reasoning and text decoding, the authors introduce a hierarchical latent-text rollout mechanism, which generates multiple answers for each latent trajectory and uses their average reward to reduce the impact of decoding randomness on latent policy optimization. Additionally, a repulsion-based diversity guidance mechanism is incorporated to further encourage diverse exploration. Extensive experiments on code generation and mathematical reasoning benchmarks demonstrate the effectiveness of the proposed method in improving reasoning accuracy and mitigating diversity collapse.
Critical strengths
The paper is well-structured and presents a clear and logically coherent methodology. The experimental evaluation is comprehensive, covering multiple representative benchmarks and extensive comparisons with state-of-the-art methods. The results are convincing and provide strong empirical support for the effectiveness of the proposed approach.
Critical weaknesses
- This paper proposes a repulsion-based diversity guidance mechanism to encourage diverse exploration by adjusting the geometric distances between latent reasoning trajectories. Although Table 3 and Table 10 demonstrate the advantages of the proposed method in semantic diversity, and Table 4 and Figure 5 show that diversity guidance improves reasoning accuracy, the experiments still lack direct validation of the relationship between geometric changes in the latent space and semantic diversity. For example, how does semantic diversity change when diversity guidance is removed? What is the relationship between the geometric distances of latent trajectories and the differences in their resulting reasoning strategies? The paper does not provide targeted correlation analyses to address these questions.
- The paper proposes a hierarchical latent-text rollout mechanism that averages the rewards of multiple decoded outputs from the same latent reasoning trajectory to reduce the impact of decoding randomness on credit assignment for the latent policy. Although Table 5 and Appendix E.3 demonstrate the advantages of reward averaging over using a single randomly selected reward, the reliability of reward estimation under limited sampling remains insufficiently analyzed. In particular, the experiments use only four decoded outputs per latent trajectory by default. Whether such a small sample size is statistically sufficient to reliably estimate the expected reward and relative advantage of a latent trajectory remains unclear. Furthermore, since the averaged reward is influenced by both the quality of latent reasoning and the capability of the text decoder, it is unclear whether the proposed mechanism can effectively disentangle their respective contributions. Additional analyses of reward estimation variance, relative advantage stability, and credit assignment effectiveness under different sampling budgets would further strengthen the justification for this mechanism.
Questions for authors
see Critical Weaknesses
5 comments
Response to Reviewer RfRx (Part 2)
W2: Reliability of M = 4 and credit assignment
(a) Is M = 4 enough? Thank you for this careful question. We fix N = 4 latents and vary M (M = 1 and 4 are from Table 5; the rest are new). To check how reliable the M-answer estimate is, we take 500 RL prompts, decode 64 answers per latent, and treat their mean reward as the latent's true value:
| M (answers per latent) | 1 | 2 | 4 (ours) | 8 | 16 |
|---|---|---|---|---|---|
| Standard deviation of a latent's mean reward | .41 | .29 | .21 | .15 | .10 |
| Advantage has the same sign as with the true value | 71% | 80% | 88% | 93% | 96% |
| Final pass@1 (code avg.) | 41.7 | 70.8 | 77.0 | 77.9 | 78.3 |
| Time per RL step | 0.35× | 0.6× | 1.0× | 1.8× | 3.4× |
At M = 1, the text advantage is also zero, which further lowers its pass@1.
These results show that larger M is more stable, but beyond M = 4 pass@1 barely improves (+0.9 at 1.8× cost), so M = 4 is a good trade-off.
(b) Can the latent and decoder contributions be disentangled? Setup: for a prompt x, the latent policy samples a reasoning trajectory z, the decoder writes an answer y from x and z, and only y gets a reward R(y).
- Decoder: yes. Its advantage compares answers written from the same z, so z's effect cancels out.
- Latent: it can be disentangled from decoding noise, but not from the decoder. z never gets a reward of its own; it only gets feedback through the answers written from it. Its credit is therefore its expected reward E[R(y) | x, z] under the current decoder. Averaging makes this estimate less noisy, but it cannot remove the decoder. This is the standard situation for latent-variable models trained with REINFORCE (Mnih & Gregor, 2014) and for high-level policies in hierarchical RL (Sutton et al., 1999; Nachum et al., 2018).
We'd like to note that our paper already tests this separation (Table 4). Since both policies share one backbone, we keep two model copies: one is trained, the other stays frozen (code pass@1, avg. of 3):
| Setting | Trained copy | Frozen copy | pass@1 |
|---|---|---|---|
| LaDiR (start) | – | – | 63.9 |
| Text only | Decoder | Latent policy | 67.3 |
| Latent only | Latent policy | Decoder | 70.3 |
| Both | Both | – | 75.6 |
These results show that each part improves on its own, the latent policy contributes more (70.3 vs. 67.3), and the two gains add up.
References
- Mnih, A., & Gregor, K. (2014). Neural variational inference and learning in belief networks. ICML.
- Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2), 181–211.
- Nachum, O., Gu, S., Lee, H., & Levine, S. (2018). Data-efficient hierarchical reinforcement learning. NeurIPS.
Dear Reviewers,
We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.
Thank you for your time and consideration!
Response to Reviewer RfRx (Part 1)
Thank you for the constructive review, and for recognizing our clear methodology, comprehensive evaluation, and convincing results.
W1: Latent geometry vs. semantic diversity
(a) Semantic diversity with and without guidance. Thank you for this question. For each problem, we sample 16 latent reasoning trajectories from the same model with guidance on and off, decode each into its reasoning text, and measure their semantic diversity (GPT-5 judge as in Table 3; code, five-benchmark avg.):
These results show that guidance increases the semantic diversity of the reasoning trajectories for both models.
(b) Latent distance vs. reasoning strategy. Latents that are close in Euclidean distance decode to similar reasoning, for three reasons:
Experiment (new). For 500 code problems, we sample 16 latent reasoning trajectories per problem from LaDi-RL (guidance off, as at evaluation) and decode each into its reasoning text. For every pair of latents, GPT-5 rates how different the two reasoning strategies are (1 = same, 5 = completely different; Appendix E.5), and we record the Euclidean distance between the two latents. Grouping pairs by rating:
The Spearman correlation between latent distance and strategy difference is ρ = 0.76 (p < 0.001).
These results show that the more different two strategies are, the farther apart their latents are. We will add both analyses to Section 5.2 and the references to Section 3.2.
References