acceptodds
Under review as a conference paper at ICLR 2027

Reinforcing Diffusion Language Models with Clean State Objectives along Denoising Trajectories

Abstract

Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce **Clean State Policy Optimization (CSPO)**, a novel training paradigm that selectively samples denoising timesteps and directly targets the final clean state for policy optimization. To reduce computational cost and stabilize training dynamics, CSPO optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that CSPO achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work provides a practical approach to efficient and stable reinforcement learning in diffusion language models.

Primary area:
reinforcement learning
Keywords:
diffusion language models, reinforcement learning, policy optimization, clean-state prediction, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DDAXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use clean state likelihoods generated from the model from partial states (instead of next state) in the policy gradient method for reinforcement learning (known as CSPO) for diffusion language models (DLMs), since there is an inherent lack of exact likelihood of the model generating the particular sequence in DLMs. Furthermore to reduce expensive gradient computations for all steps, the work gives a simple way to pick only k steps from the roll-outs where the model assigned least confidence. Across math and code benchmarks, model trained with CSPO demonstrates good cross-setting generalization.

Critical strengths
  • Good results on within-domain cross-dataset generalization.
  • "Novel" idea of picking steps based on weight-probability to reduce the expensive computation of gradient on all steps.
Critical weaknesses
  • Not using the 2 tokens/step decoding strategy which more faithfully represents diffusion language modeling. At least, provide the avg. decoded tokens/step in the dynamic decoding strategy.

  • Lack of accuracy results with full clean state loss without token clipping, so as to assess the argument for clean states quantitatively.

  • Picking the steps based on a metric computed from unmasking distributions (confidence, entropy) is not entirely novel. See [1].

[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

Questions for authors
  • Can you include a discussion with [1] in the related work, especially about their entropy vs. your weighted probability-based step-selection?
  • Can you compare with [1] by evaluating your checkpoints with 2 tokens/step (w/o training)?
  • Can you provide the results (at least on any one dataset) for your algorithm CSPO trained with 2 tokens/step w/o token clipping?

[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

Reviewer_28WLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Clean State Policy Optimization (CSPO) for reinforcement learning with diffusion language models. The method follows the rollout denoising trajectory, samples a subset of timesteps using uncertainty-based weights, and estimates the policy loss by predicting the final clean state x0 from each selected noisy state. It clips low-probability clean-state tokens to stabilize the loss and adds a k3 KL-divergence regularizer against a reference model.

Critical strengths
  1. The combination of sequential rollout contexts, selective timestep sampling, and clean-state policy loss is a reasonable way to reduce the cost of evaluating all denoising steps. The weighted sampling rule is a concrete design that can be implemented within the existing diffusion-RL pipeline.

  2. The paper is easy to follow and evaluates two DLM backbones across math, code, and planning tasks, multiple generation lengths, dynamic/static unmasking, and several design ablations.

Critical weaknesses
  • The main design choices are not isolated by matched baselines. The paper includes a temperature sweep in which τ→∞\tau \to \infty approaches uniform timestep sampling, and it includes an x0x_0-versus-xt−1x_{t-1} ablation. However, it does not provide a clean, explicitly labeled random-timestep baseline with matched sampled-timestep count, compute, and training budget. Likewise, the x0x_0 target is already standard in diffusion/ELBO objectives, so the paper needs to show the incremental contribution of CSPO’s trajectory conditioning, weighted selection, clipping, and normalization relative to an ELBO/random-masking baseline. Without this matrix of controls, the source of the improvement is difficult to identify.

  • The KL choice and baseline comparison are under-justified. CSPO uses the k3k_3 KL estimator, but there is no ablation against k1k_1, k2k_2, or no KL regularization under matched β\beta and training settings. This matters because the paper attributes some Dream instability to rapidly increasing KL divergence and uses different KL coefficients for LLaDA and Dream. In addition, the ESPO numbers appear lower than the corresponding results reported in the original ESPO work, while the manuscript only says that ESPO is modified from ELBO estimation. It is unclear whether ESPO and the other baselines were fully retrained, reimplemented with the authors’ settings, or evaluated from published results. This makes the main comparison difficult to audit.

Questions for authors
  1. Does the τ→∞\tau \to \infty condition in Figure 3 constitute the intended random-timestep baseline? Please report a dedicated uniform/random selection run with the same number of sampled timesteps, compute, optimizer steps, and random seeds as CSPO.
  2. Why is k3k_3 used for the KL estimator? Please compare k1k_1, k2k_2, k3k_3, and no KL regularization with matched β\beta tuning, especially for Dream where KL spikes are central to the analysis.
  3. Relative to the existing ELBO/random-masking objective, what is the incremental contribution of clean-state x0x_0 prediction? A controlled factorial ablation over random versus sequential masking, uniform versus weighted timestep selection, x0x_0 versus xt−1x_{t-1} targets, and clipping would clarify this.
  4. Were ESPO, d1, TraceRL, and Coupled-GRPO retrained from the same backbones and data under the same compute budget? Please explain the discrepancy between the ESPO results here and those in the original ESPO paper, and provide the exact implementation and hyperparameter settings.
  5. How sensitive are the results to the clipping threshold, the number of selected timesteps kk, and the model-specific KL coefficients β\beta?
Reviewer_tGMxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework called Clean State Policy Optimization (CSPO), which focuses on improving reinforcement learning for diffusion language models. The main motivation is that estimating policy losses along denoising trajectories is computationally expensive, while existing approximations based on random masking introduce a mismatch between training and inference.

The core method introduced by the paper is to optimize predictions of the final clean state x0x_0, conditioned on intermediate states of the actual denoising trajectory. To reduce computational costs, the authors propose a weighted sampling strategy that selects timesteps with high uncertainty, together with a token clipping mechanism that discards low-confidence predictions.

On the experimental side, CSPO is evaluated on LLaDA and Dream, across mathematical reasoning, code generation and planning benchmarks, and compared against several existing RL methods for diffusion language models.

Critical strengths
  • The paper treats an important problem in the area of post-training diffusion language models. In particular, the approach of using intermediate rollout states while supervising the final clean state is interesting, and provides a practical alternative to existing likelihood estimation techniques. To me that is an interesting point of view.

  • Empirical results are interesting. In particular, CSPO consistently improves over the base models, with substantial improvements on Dream. The experiments cover two different diffusion language model architectures, several benchmarks, and different unmasking strategies, which is a positive aspect of the paper.

Critical weaknesses
  • My main concern is about the mathematical formulation of the policy optimization objective. In particular, Equations (3) and (4) are presented as PPO-style and REINFORCE objectives, respectively, but it is unclear to me how the proposed clean-state likelihood corresponds to the probability of the actions actually taken during denoising. This seems to introduce a bias in the policy-gradient estimation, and the manuscript does not clearly characterize or justify this approximation. I discuss this point in more detail below.

  • Despite interesting empirical results, the experiments do not sufficiently disentangle the different ingredients of CSPO and their computational costs. In particular, the method uses 16 selected timesteps per training example, while some existing baselines rely on substantially cheaper loss estimation procedures. It is therefore difficult to assess how much of the improvement comes from the clean-state objective, the timestep sampling strategy, or simply the additional optimization compute.

Questions for authors

I have a few main questions, especially concerning the mathematical formulation of the method and the interpretation of the experimental results.

  1. My main question concerns Equations (3) and (4). Correct me if I am wrong, but in a standard policy-gradient formulation for a diffusion process, the gradient involves the log-probabilities of the transitions actually sampled along the denoising trajectory, i.e. terms of the form ∇θlog⁡pθ(xt−1∣xt)\nabla_\theta \log p_\theta(x_{t-1}\mid x_t). However, CSPO instead uses log⁡pθ(x0∣xt)\log p_\theta(x_0\mid x_t), including predictions of tokens that may only be generated several denoising steps later. Therefore, I do not see why Equation (4) can be interpreted as a REINFORCE estimator of the actual rollout policy gradient. The same question arises for the importance ratios in Equation (3). Is the proposed loss meant to be a biased surrogate objective rather than an estimator of the actual policy gradient? If so, I think that explicitly discussing this distinction and providing some theoretical or empirical justification would considerably strengthen the paper. Again, correct me if I am wrong.

  2. Related to the theoretical motivation, I am not entirely convinced by the argument in Section 5.2, following Equation (8). The authors state that predicting xt−1x_{t-1} implicitly commits to a particular path through x0x_0-space, while predicting x0x_0 retains the full distribution over clean sequences. However, Equation (8) precisely defines pθ(xt−1∣xt)p_\theta(x_{t-1}\mid x_t) by marginalizing over x0x_0, so I do not see why the former should necessarily commit to a particular path. Moreover, Table 4 reports that clean-state estimation uses approximately 13% of the response tokens, compared to only 2% for next-state estimation. Could the observed improvement simply come from supervising more tokens? I think that a comparison controlling for the number of supervised tokens would help distinguish these explanations.

  3. I also have a question about the token clipping procedure. In Equation (4) and Algorithm 3, the objective is computed over the retained tokens of the clean state. Does this include positions that have already been unmasked at timestep tt, or only positions that remain masked? In the former case, the model would receive a training signal for predicting tokens already visible in its conditioning context, which could significantly affect the interpretation of the objective. Also, could the authors clarify why removing low-probability tokens leads to such a large reduction in gradient norm in Table 4? For a standard softmax cross-entropy, a small target probability does not by itself imply an unbounded gradient with respect to the logits.

  4. My next question concerns computational efficiency. The paper emphasizes a 16-fold reduction in the number of loss-estimation passes compared to exhaustive trajectory optimization, which is correct, but the relevant comparison against existing RL methods seems less clear. In particular, Figure 3 compares different numbers of sampled timesteps against training iterations, although these settings have different computational costs. Could the authors provide a comparison against the main baselines using matched training FLOPs or wall-clock time? I would also be interested in seeing whether the gains from weighted timestep sampling remain significant when the number of sampled steps and total optimization compute are fixed.

  5. Finally, I have a question about the experimental protocol. Appendix E.1 mentions that, for LLaDA on mathematical tasks, the training data is filtered to retain examples for which at least one correct response is obtained across 16 generations. Appendix E.3 also mentions filtering rollout groups with identical rewards. Are these preprocessing and filtering strategies shared by all the baselines? I think that clarifying this point, together with reporting variability across runs, would make the empirical comparison more convincing.


Despite these questions, I think that the paper contains some interesting ideas and promising empirical results, especially on Dream. However, I currently have concerns about the mathematical interpretation of the proposed objective, as well as the extent to which the experiments isolate its main contributions. I am therefore slightly in favor of rejection at this stage, but I am clearly open to increasing my score if the authors can clarify these points, particularly the first two questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.