- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies RL for diffusion LLMs, and proposes relative score policy optimization (RSPO). The key observation is that the advantage can be written as centered log probability ratio from optimal and reference policies, which can be further approximated by ELBO difference estimated under the same mask assignment (for variance reduction). The authors then propose a policy-gradient style loss in which the advantage is subtracted by times centered ELBO difference. Experiments on LLaDA show that RSPO achieves nice performance boost on reasoning and planning tasks
Critical strengths
- The paper is mostly easy to follow, though in some places the writing can be improved
- The proposed method is simple and concise, and has fixed point guarantee under the optimal policy (assuming that the ELBO is exactly the same as the likelihood)
- Experimental results show promising performance
Critical weaknesses
- The experiments are limited to LLaDA and four standard benchmarks. As a DLLM RL paper in late 2026, this is far from a strong enough evaluation. Running RL on different models such as Dream and other benchmarks like HumanEval and MBPP would be more convincing.
- The authors did not show the performance with generation budget 128 as in the d1 paper. Besides, ESPO (https://arxiv.org/abs/2512.03759) and d2 (https://arxiv.org/abs/2509.21474) can also reach around 92% zero-shot accuracy on Sudoku, and d2 gets comparable performance on gsm8k and math500 as well. The authors are encouraged to involve more existing benchmarks for comparison
- The implementation details in the appendix are too brief. For example, the experimental setup for figure 3(b) is not clear, are these variance measured at the same step with the same set of prompts and mask assignments? The mean offset and std in table 2 are extremely small, which is also very weird (are they measured on the initial checkpoint?)
Questions for authors
- I don't understand we need to introduce in line 175 (it should appear in section 3.2). Essentially, if you don't consider the extreme case , then and can be just one hyperparameter because the final loss equation (3) only depends on their ratio, and the constant scalar for the loss doesn't matter for modern optimizers like Adam.
- The writing in many places can be improved. and are never used in the paper after their definitions. The sentence "If an implementation stores cross-entropy or negative ELBO rather than ELBO, the current-reference difference must be reversed" is nonsensical for the paper, more like AI agent's warning to the reader. "Eq. equation 2" should be "Eq. 2" in Remark 1. In line 193 why does the authors say ""? We haven't introduced the target (3) yet. I suggest the authors to carefully polish the paper manually
- As (6) is a quadratic loss, why not writing it as a matching loss ? This should be more intuitive and easier to the reader
- The RSPO loss is essentially matching ELBO difference (from current to reference) with the advantage, instead of naively doing policy gradient with ELBO approximation. A similar matching loss with ELBO approximation has been proposed for continuous diffusion models in https://arxiv.org/abs/2602.04663. Interestingly, their paper shows that the time weight that exactly produces the ELBO through Girsanov theorem (for DLLM it's ) doesn't work but the uniform time weight works. I'm curious if the authors have considered the choice of time weight here?