- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends continuous-time RL to discrete diffusion models using controlled continuous-time Markov chains (CTMCs). It derives policy gradients and PPO/GRPO algorithms. It also incorporates intermediate rewards and introduces trajectory subsampling for efficient training. Experiments on synthetic tasks and diffusion language models demonstrate improvements over existing baselines, particularly on Sudoku and coding benchmarks.
Critical strengths
- The paper provides a unified continuous-time RL framework for discrete diffusion models with theoretical derivations of policy gradients and PPO/GRPO algorithms.
- The incorporation of intermediate rewards enables more fine-grained credit assignment throughout the denoising process.
- The method demonstrates promising empirical improvements on reasoning and coding benchmarks, particularly Sudoku.
Critical weaknesses
- The theoretical novelty over existing continuous-time RL and CTMDP literature, particularly Meng et al. (2026), remains unclear.
- The comparison with DAM may be unfair, as CTRL-PPO uses an exact critic while DAM relies on Monte Carlo estimation.
- The experiments do not adequately isolate the contributions of the proposed components. In particular, the substantial improvement on Sudoku appears largely attributable to the additional intermediate reward, yet no comparison is provided where the baselines use the same reward. Furthermore, the authors do not ablate the state-conditioned probability ratios against existing approximations (e.g., d1/d2) or systematically compare the proposed exploration strategy with standard temperature sampling under otherwise identical settings. These missing controls make it difficult to determine which components drive the reported performance gains.
Questions for authors
-
The large Sudoku improvement appears mainly driven by intermediate rewards. Could the authors compare against baselines using identical rewards and isolate the contributions of state-conditioned probability ratios and exploration strategies?
-
CTRL-PPO uses an exact critic, while DAM relies on Monte Carlo estimation. Could the authors provide comparisons under matched computational budgets or comparable estimation settings?
3.Could the authors explain which theoretical results are fundamentally new compared to existing continuous-time RL frameworks for continuous-state MDPs, rather than straightforward extensions to discrete state spaces?
5 comments
Part 2
Q4. The large Sudoku improvement appears mainly driven by intermediate rewards. Could the authors compare against baselines using identical rewards and isolate the contributions of state-conditioned probability ratios and exploration strategies?
A4. We thank the reviewer for this insightful question. To disentangle the effects of intermediate rewards from those of the policy optimization algorithm, we have added ablation results in Tables 1 and 5 of the revised paper, including a variant of CTRL without intermediate rewards (CTRL w.o. IR) because incorporating intermediate rewards into other baselines is not straightforward following the argument in A3.
The results show that the Sudoku improvements are primarily driven by the policy optimization algorithm rather than reward shaping. Specifically, CTRL w.o. IR achieves an accuracy of , outperforming the strongest baseline, d2 (), by percentage points. Notably, d2 also evaluates multiple intermediate states and uses the same number of forward passes of per update. This comparison highlights the effectiveness of CTRL's state-conditioned probability ratios and temperature-based exploration, even without intermediate rewards. Adding intermediate rewards further improves accuracy by percentage points, reaching .
To further isolate the contribution of temperature-based exploration, we conduct an additional ablation on Sudoku using the standard sampling temperature of . CTRL without intermediate rewards achieves under this setting, compared with with temperature-based exploration. This suggests that the exploration strategy provides an additional -point improvement. For completeness, the Sudoku accuracies of LLaDA, d1, d2, CTRL w.o. IR with temperature-based exploration, CTRL w.o. IR with the standard temperature, and full CTRL are , , , , , and , respectively.
Q5. CTRL-PPO uses an exact critic, while DAM relies on Monte Carlo estimation. Could the authors provide comparisons under matched computational budgets or comparable estimation settings.
A5. We thank the reviewer for this request. We ran the comparison under matched estimation settings and matched optimization budgets, over 5 seeds. The revised Section 5.1 and Appendix H.1 describe the experimental settings.
Matched estimation settings and optimization budgets. CTRL-PPO is run with an exact critic and with a Monte Carlo critic using completions per context, matching DAM’s adjoint sample budget. DAM is run with its Monte Carlo adjoint () and with an exact adjoint. All methods use the same initialization and 4 gradient steps per iteration.
Results. The results are summarized as follows (Section 5.1 and Table 4 in the revised paper). Under matched settings, PPO converges at the same rate as DAM, with both reaching within 30 iterations. With a Monte Carlo critic, PPO converges equally fast but levels off at .
| Method | Value / adjoint | Iter. to | |
|---|---|---|---|
| CTRL-PPO | Exact | 29 | |
| CTRL-PPO | MC () | 29 | |
| DAM | MC () | 29 | |
| DAM | Exact | 29 |
Q6. Could the authors explain which theoretical results are fundamentally new compared to existing continuous-time RL frameworks for continuous-state MDPs, rather than straightforward extensions to discrete state spaces?
A6. We thank the reviewer for the question. Our core continuous-time RL results indeed follow the strategy of the continuous-state literature, as our proofs acknowledge. These are the policy gradient theorem (Theorem 1) and the -function characterization (Proposition 2), following Jia & Zhou (2022a; 2023). In the discrete setting, Ito calculus is replaced by the CTMC generator and Dynkin’s formula, but the logical structure is the same, because both SDE and CTMC are the more general Feller processes, so we would not claim that our theoretical results are fundamentally new. Nevertheless, we regard these results (policy gradient and -function) as necessary foundations for a model-free policy-optimization theory of controlled CTMCs in model-free RL, which to our knowledge had not been written down before.
We Are Looking Forward to Your Reply
Dear Reviewer KAzG,
Thank you for your detailed review. We have addressed your concerns point by point in our rebuttal and made corresponding revisions. Given the approaching rebuttal deadline, we would greatly appreciate your feedback, particularly if our clarification of the novelty of the model-free CTRL theoretical framework and our ablation study on intermediate rewards address your concerns. We would be happy to address any additional questions you may have.
Best,
Authors
Part 1
We appreciate the reviewer's insightful comment. Our responses are provided below, and all revisions have been highlighted in blue in the revised manuscript.
Q1. The theoretical novelty over existing continuous-time RL and CTMDP literature, particularly Meng et al. (2026), remains unclear.
A1. We thank the review for this question. We address each point below.
Relation to classical CTMDP and continuous-space CTRL. Classical CTMDP theory (Miller, 1968; Guo & Hernández-Lerma, 2009; Piunovskiy & Zhang, 2020) primarily focuses on model-based stochastic control, including optimality equations and the existence of optimal policies when transition rates and reward functions are specified. In contrast, our work focuses on model-free policy optimization, where the transition dynamics are not assumed to be known. We acknowledge that related mathematical objects and formulations have appeared in the classical CTMDP literature. However, this literature has primarily focused on model-based stochastic control, particularly optimal control and the associated HJB equations, rather than model-free reinforcement learning and policy optimization.
We acknowledge that our proof techniques follow a similar overall strategy of continuous-sapce CTRL. This similarity is natural because both SDEs and CTMCs belong to the broader class of Feller processes, and their infinitesimal generators provide a common mathematical foundation for formulating continuous-time policy evaluation and optimization. However, extending this framework from continuous state spaces to discrete, high-dimensional state spaces requires developing the corresponding policy optimization formulation for controlled CTMCs.
Relation to Meng et al. (2026). Our work was developed independently and predates Meng et al. (2026). To the best of our knowledge, we are the first to formulate a model-free continuous-time policy optimization framework for discrete diffusion models using a q-function and its corresponding policy gradient representation.
The two works have different primary objectives and algorithmic settings. Meng et al. (2026) focus on q-learning with relatively low-dimensional action spaces, such as prices and server counts, motivated by network pricing and queueing applications. In contrast, our work develops PPO- and GRPO-based policy optimization algorithms for fine-tuning discrete diffusion models and discrete diffusion language models (dLLMs). In this setting, the action space consists of products of high-dimensional probability simplices, ΔV−1, where the vocabulary size is approximately V≈105. This setting introduces substantially different computational and algorithmic considerations for policy optimization, which motivate our proposed methods.
Q2. The comparison with DAM may be unfair, as CTRL-PPO uses an exact critic while DAM relies on Monte Carlo estimation.
A2. We thank the reviewer for pointing this out. Please refer to our response to Q5, where we provide additional experiments comparing CTRL-PPO with Monte Carlo estimation and DAM with an exact adjoint, ensuring a fairer comparison between the two methods.
Q3. The experiments do not adequately isolate the contributions of the proposed components. In particular, the substantial improvement on Sudoku appears largely attributable to the additional intermediate reward, yet no comparison is provided where the baselines use the same reward. The authors do not ablate the state-conditioned probability ratios against existing approximations (e.g., d1/d2) or systematically compare the proposed exploration strategy with standard temperature sampling under identical settings.
A3. We thank the reviewer for suggesting these ablation studies. Please refer to our response to Q4 for the corresponding results. However, we note that incorporating intermediate rewards into other baselines, particularly d1 and SPG, is not straightforward, because all three baseline methods optimize sequence-level objectives and have no mechanism for assigning a reward to an individual intermediate state. In d1 and SPG, the intermediate states are not part of the training objective at all. d1 approximates the sequence log-likelihood with a one-step estimate on a randomly masked copy of the final completion. SPG instead optimizes upper and lower bounds on the sequence log-likelihood, again estimated by randomly re-masking the final completion. In both cases, the partially denoised states that the model actually visited during generation are discarded. Since an intermediate reward is defined on exactly those states, these objectives have no term to which it could be attached. Therefore, we focus on isolating the contribution of intermediate rewards within CTRL, while keeping the outcome-level reward and all other training settings fixed. Therefore, we conduct ablation studies on removing intermediate rewards from CTRL across all four reasoning tasks.