- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a unified perspective of the two methods of policy optimization for reinforcement learning in diffusion language models (DLMs): ELBO-based lower bound on sequence likelihood and an exact trajectory likelihood of that specific roll-out. The work shows that the exact likelihood gradient of the sequence is the expectation of per-order gradients where the expectation is over the posterior on the reveal orders. The ELBO uses a proxy distribution of uniform over the reveal orders and trajectory-based methods use a Dirac distribution on a specific trajectory, forming the two points on the spectrum. The paper introduces a flexible single-parameter family of distributions on "reveal orders" based on confidence that interpolates between these two points showing improved performance on held-out checkpoint evaluation, with demonstrable gains in logical reasoning benchmarks (Countdown and some generation lengths for Sudoku).
Critical strengths
-
Providing an interesting mathematical unification of ELBO-based and trajectory-based methods as two approximations of the exact sequence policy gradient.
-
Mathematical theory on gradient error bound and direct motivation for parametrization.
Critical weaknesses
- [Soundness] Weak results on mathematical reasoning benchmarks.
- [Novelty] Missing relevant prior work [1], which proposes the "right" partial states/masks and also an interesting trajectory-based method.
[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.
- [Presentation] It is unclear how the intermediate states are constructed. Do we need extra forward passes to get the partial states (unseen in rollouts) in addition to while computing the policy gradients?
Questions for authors
-
Can you include coding benchmarks (MBPP, HumanEval) to strengthen the paper?
-
Can you compare and contrast with the trajectory-based prior work [1]?
[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.
- Can you please clarify how are unseen partial states sampled and whether they need additional forward passes?
3 comments
Reply
Thanks for the response and apologies for the late reply. I want to understand the distinction of your algorithm in ESPO and d2 as baselines clearly:
- In -ESPO, the rollout procedure is the same as the baseline, but the relative weights of the gradient depends on the (and the family of disturbutions).
- In -d2-StepMerge, the roll procedure changes from baseline to based on the family of reveal order distribution, but the there is no additional reweighting of the gradient terms like in the last ESPO case.
Can you confirm that these statements are correct?
Sorry for the late reply. We thank the reviewer for the follow-up. Both statements are correct if the reviewer is not referencing the rollout during the generation of the responses but the rollout during their scoring. In more detail:
-
In -ESPO, the rollouts and the mask sampling are the same as in ESPO and only the weights of each position are changed following our dependent approach.
-
In -StepMerge there is indeed no additional reweighting: Eq. (16) is used unchanged, with every token weighted as in d2-StepMerge. What changes is the reveal order used to score the generated response: given a generated response , d2-StepMerge uses the order realized during generation phase, whereas we sample a new order of the same response from the confidence measure, segment by segment.
Concerning the generation phase however, we want to stress that responses are generated exactly as in the corresponding baseline. Our method only acts on how the log-likelihood gradient of an already generated is estimated. The table below summarizes the differences.
| ESPO | τ-ESPO | d2-StepMerge | τ-d2-StepMerge | |
|---|---|---|---|---|
| Generation of response | standard | same as ESPO | standard | same as d2-StepMerge |
| Partial states scored | masks sampled by ESPO | same masks as ESPO | states along the generation rollout order | states along an order sampled from |
| Weight of each scored token | , stop-gradient | 1 | 1 | |
| Order measure | uniform, via weighting | confidence, via weighting | Dirac on the generation rollout order, via partial states | confidence, via partial states |
| Forward passes | one per mask | one per mask | per order | per order |
We thank the reviewer for their comments, which help us better position the paper within the literature and improve its clarity.
1. Relation to [1] (Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages, https://arxiv.org/abs/2603.12554)
We thank the reviewer for pointing out this relevant recent paper, which we will include and discuss in the revised manuscript. The paper is indeed relevant but has a contribution which is orthogonal to ours. While both works exploit the structure of the denoising process to improve RL, they address distinct questions. Kunde et al. [1] formulate denoising as an MDP and propose selecting high-entropy denoising steps for policy updates, together with stepwise advantage estimation. Their entropy criterion determines which intermediate states along a sampled trajectory should contribute to optimization. The sampled reveal order itself is determined during the rollout phase, as is common in trajectory-based approaches.
In contrast, our work investigates the distribution over reveal orders underlying the sequence-likelihood gradient. We show that the exact gradient averages per-order gradients under the posterior over reveal orders, whereas existing ELBO-based and single-trajectory approaches correspond to different choices of order measure. Our confidence-based distribution provides a temperature-controlled family of such measures, and we study its approximation quality through chi-squared divergence, effective sample size, and gradient error.
Thus, although both methods depart from uniform treatment of denoising decisions, they operate on different dimensions: [1] allocates computation across steps of a trajectory, whereas our method changes the measure over possible trajectories. The two approaches are complementary and could potentially be combined.
2. Partial states and cost
Our method is built on top of the baseline estimator: it only changes which reveal orders are scored, and it uses as confidences the token probabilities that the baseline already computes. It therefore requires no additional forward passes.
τ-ESPO. No new partial states are introduced: masks m are sampled exactly as in ESPO. ESPO runs one forward pass on ym, which gives pθ(yi∣ym,x) for every masked position i, and averages their log-probabilities uniformly. We reuse these same probabilities as weights:
∑iℓmilogpθ(yi∣ym,x)⟶∑isg[qc,iτ(m)]milogpθ(yi∣ym,x),qc,iτ(m)∝pθ(yi∣ym,x)1/τ,
where ℓ is the number of masked positions and sg denotes the stop-gradient. The masks and the forward passes are the same as in ESPO.
τ-d2-StepMerge. d2-StepMerge splits an order into N segments B1,…,BN and scores each segment with one forward pass on the state y(n) at its start (Eq. 16):
logπ^θ(y∣x)=∑n=1N∑i∈Bnlogpθ(yi∣y(n),x).
d2-StepMerge takes the segments from the rollout order. We instead sample them sequentially: the pass on y(n) gives the probabilities of all masked positions, from which we draw Bn with weights ∝pθ(yi∣y(n),x)1/τ; the same pass scores Bn, and unmasking it gives y(n+1). The partial states y(n) therefore differ from those of the rollout, but there is still one forward pass per segment: N per order, as in d2-StepMerge.
We acknowledge that the implementation details were not clear enough in the paper. We will fix this by adding an algorithm box for both procedures.
3. Coding benchmarks
We agree with the reviewer that providing results for coding benchmarks would strengthen the paper, but we could not run them during the rebuttal. We will try to include MBPP and HumanEval in the final version and mention their absence as a limitation.