acceptodds
Under review as a conference paper at ICLR 2027

Relative Score Policy Optimization for Diffusion Language Models

Abstract

Diffusion large language models (dLLMs) offer a promising route to parallel and efficient text generation, but improving their reasoning ability requires effective post-training. Reinforcement learning with verifiable rewards (RLVR) is a natural choice for this purpose, yet its application to dLLMs is hindered by the absence of tractable sequence-level log-ratios, which are central to standard policy optimization. The lack of tractable sequence-level log-ratios forces existing methods to rely on high-variance ELBO-based approximations, where high verifier rewards can amplify inaccurate score estimates and destabilize RL training. To overcome this issue, we propose Relative Score Policy Optimization (RSPO), a simple RLVR method that uses verifiable rewards to calibrate noisy likelihood estimates in dLLMs. The core of our algorithm relies on a key observation: a reward advantage can be interpreted not only as an update direction, but also as a target for the relative log-ratio between the current and reference policies. Accordingly, RSPO calibrates this noisy relative log-ratio estimate by comparing its reward advantage with the reward-implied target relative log-ratio, updating the policy according to the gap between the current estimate and the target rather than the raw advantage alone. Experiments on mathematical reasoning and planning benchmarks show that RSPO yields especially strong gains on planning tasks and competitive mathematical-reasoning performance.

Primary area:
generative models
Keywords:
Diffusion Language Models, Post training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PTHQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RL for diffusion LLMs, and proposes relative score policy optimization (RSPO). The key observation is that the advantage can be written as centered log probability ratio from optimal and reference policies, which can be further approximated by ELBO difference estimated under the same mask assignment (for variance reduction). The authors then propose a policy-gradient style loss in which the advantage is subtracted by λ\lambda times centered ELBO difference. Experiments on LLaDA show that RSPO achieves nice performance boost on reasoning and planning tasks

Critical strengths
  • The paper is mostly easy to follow, though in some places the writing can be improved
  • The proposed method is simple and concise, and has fixed point guarantee under the optimal policy (assuming that the ELBO is exactly the same as the likelihood)
  • Experimental results show promising performance
Critical weaknesses
  • The experiments are limited to LLaDA and four standard benchmarks. As a DLLM RL paper in late 2026, this is far from a strong enough evaluation. Running RL on different models such as Dream and other benchmarks like HumanEval and MBPP would be more convincing.
  • The authors did not show the performance with generation budget 128 as in the d1 paper. Besides, ESPO (https://arxiv.org/abs/2512.03759) and d2 (https://arxiv.org/abs/2509.21474) can also reach around 92% zero-shot accuracy on Sudoku, and d2 gets comparable performance on gsm8k and math500 as well. The authors are encouraged to involve more existing benchmarks for comparison
  • The implementation details in the appendix are too brief. For example, the experimental setup for figure 3(b) is not clear, are these variance measured at the same step with the same set of prompts and mask assignments? The mean offset and std in table 2 are extremely small, which is also very weird (are they measured on the initial checkpoint?)
Questions for authors
  • I don't understand we need to introduce λ\lambda in line 175 (it should appear in section 3.2). Essentially, if you don't consider the extreme case λ=0\lambda=0, then λ\lambda and β\beta can be just one hyperparameter because the final loss equation (3) only depends on their ratio, and the constant scalar for the loss doesn't matter for modern optimizers like Adam.
  • The writing in many places can be improved. τi\tau_i and ei(θ)e_i(\theta) are never used in the paper after their definitions. The sentence "If an implementation stores cross-entropy or negative ELBO rather than ELBO, the current-reference difference must be reversed" is nonsensical for the paper, more like AI agent's warning to the reader. "Eq. equation 2" should be "Eq. 2" in Remark 1. In line 193 why does the authors say "δi(θ)→Ai/λ\delta_i(\theta)\to A_i/\lambda"? We haven't introduced the target (3) yet. I suggest the authors to carefully polish the paper manually
  • As (6) is a quadratic loss, why not writing it as a matching loss ∣∣δ^−?∣∣2||\hat\delta - ?||^2? This should be more intuitive and easier to the reader
  • The RSPO loss is essentially matching ELBO difference (from current to reference) with the advantage, instead of naively doing policy gradient with ELBO approximation. A similar matching loss with ELBO approximation has been proposed for continuous diffusion models in https://arxiv.org/abs/2602.04663. Interestingly, their paper shows that the time weight that exactly produces the ELBO through Girsanov theorem (for DLLM it's 1/t1/t) doesn't work but the uniform time weight works. I'm curious if the authors have considered the choice of time weight here?
Reviewer_RNSvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Relative Score Policy Optimization (RSPO), where the reward advantages are used as a target for log likelihood-ratio (LLR) of current and reference policies (approximated by ELBO of the generated sequences). They show that this reduces to multiplying the LLR with the gap between the advantages and the corresponding (stop gradient) LLRs, rather than only the advantages as in standard ELBO-based methods. Across representative benchmarks (?), the algorithm demonstrates improved performance and stable training.

Critical strengths
  • Showing the equivalence of quadratic optimization and RSPO in terms of gradients showing first order agreement between the two objectives (Theorem 1).
Critical weaknesses
  • [Soundness] Missing critical baselines and lot of recent prior work SPG [1], d2 [2], EGSPO(-SA) [3].

  • Lack of the immediate baseline RSPO (λ=0\lambda=0) makes it difficult to see where the true gains are coming from. If 0 is already good enough, the claims of the paper would weaken.

  • Using λ=0\lambda=0 seems like it is a simpler version of SPG with ELBO multiplying both positive and negative advantages. In that case, the paper is offering limited novelty (only in terms of connecting or offering new perspective rather than a genuinely useful practical algorithm).

[1] Chenyu Wang and Paria Rashidinejad and DiJia Su and Song Jiang and Sid Wang and Siyan Zhao and Cai Zhou and Shannon Zejiang Shen and Feiyu Chen and Tommi Jaakkola and Yuandong Tian and Bo Liu. SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models. The Fourteenth International Conference on Learning Representations (ICLR). 2026. https://openreview.net/forum?id=18j5Q49GwN.

[2] Guanghan Wang and Gilad Turok and Yair Schiff and Marianne Arriola and Volodymyr Kuleshov. d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation. Forty-third International Conference on Machine Learning (ICML). 2026. https://openreview.net/forum?id=ldCiNVFt8O.

[3] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

Questions for authors
  • Can you include λ=0\lambda=0 results?
  • Can you clarify the contribution of the paper if we set it to 0?
  • Can you compare with the case with 0 and SPG [1]?
  • Can you include EGSPO(-SA) [2] in the related work and compare with it if time permits?

[1] Chenyu Wang and Paria Rashidinejad and DiJia Su and Song Jiang and Sid Wang and Siyan Zhao and Cai Zhou and Shannon Zejiang Shen and Feiyu Chen and Tommi Jaakkola and Yuandong Tian and Bo Liu. SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models. The Fourteenth International Conference on Learning Representations (ICLR). 2026. https://openreview.net/forum?id=18j5Q49GwN.

[3] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

Reviewer_JYrEOpenReview ↗
Rating
1(−2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning method for diffusion language models (dLLMs) to address the inaccurate estimation of policy log-ratios using ELBO-based approximations. From the perspective of KL-regularized policy optimization, this paper derives a reward induced target for the current-reference log-ratio and approximates it using ELBO-based relative scores in dLLMs. Based on this formulation, the paper introduces a simple residual-feedback objective to optimize the relative scores toward their targets. Experiments on public benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper presents a well-motivated and insightful RLVR framework for dLLMs. Although the method relies on certain approximations, its design is simple, easy to implement, and supported by a clear theoretical foundation derived from KL-regularized policy improvement. The proposed relative-score feedback mechanism also provides useful insights for future research on policy optimization for dLLMs.

  • The paper evaluates RSPO on both mathematical reasoning and planning benchmarks, demonstrating its effectiveness across different tasks. In particular, RSPO achieves substantial improvements over existing baselines on challenging planning tasks such as Sudoku and Countdown. Furthermore, the training reward curves and ablation studies provide useful insights into the effectiveness of the proposed relative-score feedback mechanism and its key components.

Critical weaknesses
  • The paper identifies noisy and inaccurate ELBO-based relative scores as a key limitation of existing RL methods for dLLMs. However, Equation (2) still estimates the current-reference log-ratio using the difference between two ELBO scores, which remains an inaccurate and noisy approximation. Moreover, Equation (1) derives the reward-implied target by eliminating the prompt-dependent normalization factor through centering. While this derivation holds for exact log-ratios, it is unclear whether the same target relationship remains valid when ELBO-based approximations are used, since their approximation errors may not cancel after centering. The current theoretical analysis does not sufficiently justify this approximation or explain why the proposed residual feedback can effectively calibrate these inaccurate relative scores.

  • The feedback coefficient (\lambda) plays an important role in RSPO, but the paper only evaluates three values ((\lambda=0), (0.01), and (1)). The authors should provide a more comprehensive hyperparameter study with intermediate values (e.g., (0.05), (0.1), and (0.2)) to better understand its impact on performance and evaluate the robustness of the proposed method.

Questions for authors
  • Could the authors further explain how the proposed method calibrates inaccurate ELBO-based relative scores, and whether the ELBO approximation preserves the elimination of the normalization factor through centering?

  • Could the author analyze the influence of \lambda under more values?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.