acceptodds
Under review as a conference paper at ICLR 2027

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models

Abstract

We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynamics are modeled as a controlled continuous-time Markov chain (CTMC). We consider policy optimization problems and derive the corresponding policy gradient methods, leading to continuous-time variants of proximal policy optimization (PPO) and group relative policy optimization (GRPO). As a primary application, we develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models. The proposed framework enables reward-driven optimization without requiring differentiability on the reward signals. In contrast to the existing GRPO-based approaches that only rely on terminal rewards, our formulation allows intermediate reward or advantage signals to be incorporated throughout the denoising trajectory. Importantly, when specialized to masked diffusion models (MDMs), our framework encompasses a rich class of policy parameterizations over the vocabulary simplex with analytically tractable probability ratios, providing a unified perspective on exploration and policy optimization in MDMs. For masked diffusion large language models (dLLMs), we further propose trajectory subsampling techniques to efficiently estimate computationally prohibitive trajectory likelihoods, reducing the computational cost of computing per-position probability ratios. We showcase the effectiveness of our methods on both low-dimensional entropy-regularized optimization problems and RL post-training of dLLMs on mathematical reasoning and coding tasks.

Primary area:
reinforcement learning
Keywords:
Continuous-Time Discrete-Space Reinforcement Learning, Discrete Diffusion Models Fine-Tuning, Controlled Continuous-Time Markov Chains

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KAzGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends continuous-time RL to discrete diffusion models using controlled continuous-time Markov chains (CTMCs). It derives policy gradients and PPO/GRPO algorithms. It also incorporates intermediate rewards and introduces trajectory subsampling for efficient training. Experiments on synthetic tasks and diffusion language models demonstrate improvements over existing baselines, particularly on Sudoku and coding benchmarks.

Critical strengths
  1. The paper provides a unified continuous-time RL framework for discrete diffusion models with theoretical derivations of policy gradients and PPO/GRPO algorithms.
  2. The incorporation of intermediate rewards enables more fine-grained credit assignment throughout the denoising process.
  3. The method demonstrates promising empirical improvements on reasoning and coding benchmarks, particularly Sudoku.
Critical weaknesses
  • The theoretical novelty over existing continuous-time RL and CTMDP literature, particularly Meng et al. (2026), remains unclear.
  • The comparison with DAM may be unfair, as CTRL-PPO uses an exact critic while DAM relies on Monte Carlo estimation.
  • The experiments do not adequately isolate the contributions of the proposed components. In particular, the substantial improvement on Sudoku appears largely attributable to the additional intermediate reward, yet no comparison is provided where the baselines use the same reward. Furthermore, the authors do not ablate the state-conditioned probability ratios against existing approximations (e.g., d1/d2) or systematically compare the proposed exploration strategy with standard temperature sampling under otherwise identical settings. These missing controls make it difficult to determine which components drive the reported performance gains.
Questions for authors
  1. The large Sudoku improvement appears mainly driven by intermediate rewards. Could the authors compare against baselines using identical rewards and isolate the contributions of state-conditioned probability ratios and exploration strategies?

  2. CTRL-PPO uses an exact critic, while DAM relies on Monte Carlo estimation. Could the authors provide comparisons under matched computational budgets or comparable estimation settings?

3.Could the authors explain which theoretical results are fundamentally new compared to existing continuous-time RL frameworks for continuous-state MDPs, rather than straightforward extensions to discrete state spaces?

5 comments

We appreciate the reviewer's insightful comment. Our responses are provided below, and all revisions have been highlighted in blue in the revised manuscript.


Q1. The theoretical novelty over existing continuous-time RL and CTMDP literature, particularly Meng et al. (2026), remains unclear.

A1. We thank the review for this question. We address each point below.

Relation to classical CTMDP and continuous-space CTRL. Classical CTMDP theory (Miller, 1968; Guo & Hernández-Lerma, 2009; Piunovskiy & Zhang, 2020) primarily focuses on model-based stochastic control, including optimality equations and the existence of optimal policies when transition rates and reward functions are specified. In contrast, our work focuses on model-free policy optimization, where the transition dynamics are not assumed to be known. We acknowledge that related mathematical objects and formulations have appeared in the classical CTMDP literature. However, this literature has primarily focused on model-based stochastic control, particularly optimal control and the associated HJB equations, rather than model-free reinforcement learning and policy optimization.

We acknowledge that our proof techniques follow a similar overall strategy of continuous-sapce CTRL. This similarity is natural because both SDEs and CTMCs belong to the broader class of Feller processes, and their infinitesimal generators provide a common mathematical foundation for formulating continuous-time policy evaluation and optimization. However, extending this framework from continuous state spaces to discrete, high-dimensional state spaces requires developing the corresponding policy optimization formulation for controlled CTMCs.

Relation to Meng et al. (2026). Our work was developed independently and predates Meng et al. (2026). To the best of our knowledge, we are the first to formulate a model-free continuous-time policy optimization framework for discrete diffusion models using a qq-function and its corresponding policy gradient representation.

The two works have different primary objectives and algorithmic settings. Meng et al. (2026) focus on qq-learning with relatively low-dimensional action spaces, such as prices and server counts, motivated by network pricing and queueing applications. In contrast, our work develops PPO- and GRPO-based policy optimization algorithms for fine-tuning discrete diffusion models and discrete diffusion language models (dLLMs). In this setting, the action space consists of products of high-dimensional probability simplices, ΔV−1\Delta^{V-1}, where the vocabulary size is approximately V≈105V\approx 10^5. This setting introduces substantially different computational and algorithmic considerations for policy optimization, which motivate our proposed methods.


Q2. The comparison with DAM may be unfair, as CTRL-PPO uses an exact critic while DAM relies on Monte Carlo estimation.

A2. We thank the reviewer for pointing this out. Please refer to our response to Q5, where we provide additional experiments comparing CTRL-PPO with Monte Carlo estimation and DAM with an exact adjoint, ensuring a fairer comparison between the two methods.


Q3. The experiments do not adequately isolate the contributions of the proposed components. In particular, the substantial improvement on Sudoku appears largely attributable to the additional intermediate reward, yet no comparison is provided where the baselines use the same reward. The authors do not ablate the state-conditioned probability ratios against existing approximations (e.g., d1/d2) or systematically compare the proposed exploration strategy with standard temperature sampling under identical settings.

A3. We thank the reviewer for suggesting these ablation studies. Please refer to our response to Q4 for the corresponding results. However, we note that incorporating intermediate rewards into other baselines, particularly d1 and SPG, is not straightforward, because all three baseline methods optimize sequence-level objectives and have no mechanism for assigning a reward to an individual intermediate state. In d1 and SPG, the intermediate states are not part of the training objective at all. d1 approximates the sequence log-likelihood with a one-step estimate on a randomly masked copy of the final completion. SPG instead optimizes upper and lower bounds on the sequence log-likelihood, again estimated by randomly re-masking the final completion. In both cases, the partially denoised states that the model actually visited during generation are discarded. Since an intermediate reward is defined on exactly those states, these objectives have no term to which it could be attached. Therefore, we focus on isolating the contribution of intermediate rewards within CTRL, while keeping the outcome-level reward and all other training settings fixed. Therefore, we conduct ablation studies on removing intermediate rewards from CTRL across all four reasoning tasks.

Q4. The large Sudoku improvement appears mainly driven by intermediate rewards. Could the authors compare against baselines using identical rewards and isolate the contributions of state-conditioned probability ratios and exploration strategies?

A4. We thank the reviewer for this insightful question. To disentangle the effects of intermediate rewards from those of the policy optimization algorithm, we have added ablation results in Tables 1 and 5 of the revised paper, including a variant of CTRL without intermediate rewards (CTRL w.o. IR) because incorporating intermediate rewards into other baselines is not straightforward following the argument in A3.

The results show that the Sudoku improvements are primarily driven by the policy optimization algorithm rather than reward shaping. Specifically, CTRL w.o. IR achieves an accuracy of 82.982.9\\%, outperforming the strongest baseline, d2 (68.068.0\\%), by 14.914.9 percentage points. Notably, d2 also evaluates multiple intermediate states and uses the same number of forward passes of 1616 per update. This comparison highlights the effectiveness of CTRL's state-conditioned probability ratios and temperature-based exploration, even without intermediate rewards. Adding intermediate rewards further improves accuracy by 5.35.3 percentage points, reaching 88.288.2\\%.

To further isolate the contribution of temperature-based exploration, we conduct an additional ablation on Sudoku using the standard sampling temperature of 0.90.9. CTRL without intermediate rewards achieves 79.579.5\\% under this setting, compared with 82.982.9\\% with temperature-based exploration. This suggests that the exploration strategy provides an additional 3.43.4-point improvement. For completeness, the Sudoku accuracies of LLaDA, d1, d2, CTRL w.o. IR with temperature-based exploration, CTRL w.o. IR with the standard temperature, and full CTRL are 11.511.5\\%, 22.922.9\\%, 68.068.0\\%, 82.982.9\\%, 79.579.5\\%, and 88.288.2\\%, respectively.


Q5. CTRL-PPO uses an exact critic, while DAM relies on Monte Carlo estimation. Could the authors provide comparisons under matched computational budgets or comparable estimation settings.

A5. We thank the reviewer for this request. We ran the comparison under matched estimation settings and matched optimization budgets, over 5 seeds. The revised Section 5.1 and Appendix H.1 describe the experimental settings.

Matched estimation settings and optimization budgets. CTRL-PPO is run with an exact critic and with a Monte Carlo critic using K=12K=12 completions per context, matching DAM’s adjoint sample budget. DAM is run with its Monte Carlo adjoint (K=12K=12) and with an exact adjoint. All methods use the same initialization and 4 gradient steps per iteration.

Results. The results are summarized as follows (Section 5.1 and Table 4 in the revised paper). Under matched settings, PPO converges at the same rate as DAM, with both reaching DKL<10−3D_{\mathrm{KL}}<10^{-3} within 30 iterations. With a Monte Carlo critic, PPO converges equally fast but levels off at DKL≈4×10−5D_{\mathrm{KL}}\approx 4\times10^{-5}.

MethodValue / adjointDKL(p∗∥pθ)D_{\mathrm{KL}}(p^*\\\|p_\theta)Iter. to DKL<10−3D_{\mathrm{KL}}<10^{-3}
CTRL-PPOExact(1.4±0.1)×10−15(1.4\pm0.1)\times10^{-15}29
CTRL-PPOMC (K=12K=12)(3.8±0.4)×10−5(3.8\pm0.4)\times10^{-5}29
DAMMC (K=12K=12)(2.1±0.1)×10−15(2.1\pm0.1)\times10^{-15}29
DAMExact(2.1±0.1)×10−15(2.1\pm0.1)\times10^{-15}29

Q6. Could the authors explain which theoretical results are fundamentally new compared to existing continuous-time RL frameworks for continuous-state MDPs, rather than straightforward extensions to discrete state spaces?

A6. We thank the reviewer for the question. Our core continuous-time RL results indeed follow the strategy of the continuous-state literature, as our proofs acknowledge. These are the policy gradient theorem (Theorem 1) and the qq-function characterization (Proposition 2), following Jia & Zhou (2022a; 2023). In the discrete setting, Ito calculus is replaced by the CTMC generator and Dynkin’s formula, but the logical structure is the same, because both SDE and CTMC are the more general Feller processes, so we would not claim that our theoretical results are fundamentally new. Nevertheless, we regard these results (policy gradient and qq-function) as necessary foundations for a model-free policy-optimization theory of controlled CTMCs in model-free RL, which to our knowledge had not been written down before.

AuthorsOpenReview ↗

We Are Looking Forward to Your Reply

Dear Reviewer KAzG,

Thank you for your detailed review. We have addressed your concerns point by point in our rebuttal and made corresponding revisions. Given the approaching rebuttal deadline, we would greatly appreciate your feedback, particularly if our clarification of the novelty of the model-free CTRL theoretical framework and our ablation study on intermediate rewards address your concerns. We would be happy to address any additional questions you may have.

Best,

Authors

Reviewer_6bx1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates continuous-time RL for controlled continuous-time Markov chains (CTMCs) with discrete states. It derives a q-function, a policy gradient theorem, and PPO/GRPO-style algorithms. It then applies this to fine-tuning discrete diffusion models by treating the denoiser output (or score) as the action. For masked diffusion LLMs, the authors consider several exploration policies over the simplex. They find these unstable, so in practice they use a vertex-supported policy whose probability ratio is the product of token probabilities conditioned on the intermediate state x_t. The method allows hand-designed intermediate rewards along the denoising trajectory and uses trajectory subsampling (N of T steps) to reduce forward passes. Experiments cover a 2D checkerboard toy task with PPO, and GRPO fine-tuning of LLaDA-8B-Instruct on GSM8K, MATH500, Countdown, Sudoku, HumanEval, and MBPP. Results are compared against d1, d2, and SPG, with large gains on Sudoku (88.2%) and coding.

Critical strengths

Principled view of dLLM RL. The paper connects GRPO for masked diffusion to a well-defined continuous-time control problem. It also points out a real inconsistency in d1: rollouts unmask two tokens per step, but d1’s loss is AR-style and conditions only on the prompt. Fixing this by conditioning the ratio on the intermediate state x_t is a reasonable and useful observation.

Strong headline numbers. If the results hold up under a fair protocol, the gains on Sudoku, Countdown, and HumanEval are large and practically relevant.

Critical weaknesses
  1. The theory and the actual algorithm are largely disconnected. The continuous-time framework motivates actions on the full simplex. However, all three full-simplex policies (Dirichlet, temperature softmax, logistic-normal) are reported to be unstable (Remark 2). The method actually used puts its mass on simplex vertices, so it reduces to standard token-level GRPO with per-step ratios conditioned on x_t. That is essentially a step-wise trajectory likelihood, similar in spirit to d2 and TraceRL. The q-function, the policy gradient theorem, and PPO are not used in the dLLM experiments. Appendix G says PPO was unstable. So the main contribution claimed, continuous-time RL, does little work in the results.

  2. Limited theoretical novelty. The theory closely follows Wang et al. (2020) and Jia & Zhou (2022a; 2023), transferred from SDEs to CTMCs. The authors acknowledge that concurrent work (Meng et al., 2026) also derives the HJB equation, the exploratory rate, and the q-function for this setting. Theorem 2 is taken from Lemma 1 of Zhang et al. (2025). Proposition 4, the unbiasedness of uniform subsampling, is trivial.

  3. Rollout/training policy mismatch. Rollouts are generated with exponential-temperature softmax exploration. The training ratio in Eq. 11, however, uses the plain denoiser probabilities p_θ. This makes the update off-policy without any correction, which contradicts the paper’s main argument about consistency between sampling and optimization.

  4. The comparison with baselines is unfair. The authors attribute the Sudoku gain to hand-designed intermediate rewards, for example penalizing characters outside {1, 2, 3, 4}. These are task-specific reward shaping, and the baselines don’t receive them. An ablation is missing: CTRL without intermediate rewards, or baselines given the same shaped reward. Without it, the gains can’t be credited to the algorithm. The intermediate rewards for the coding tasks are not described at all, even though HumanEval improves by about 17 points over the best baseline.

  5. The evaluation protocol favours the proposed method. Table 1 reports the best checkpoint over training, which amounts to selection on the test set. The sequence length shown per task (128 for Sudoku, 256 otherwise) was also chosen after seeing results. Each result comes from a single run, with no seeds or error bars, and evaluation uses a batch size of 8. Baselines were re-run under 0-shot prompting with CTRL’s hyperparameters (e.g., clip 0.5) rather than their own tuned settings. SPG’s own paper used 3-shot prompting.

  6. The synthetic experiment is not informative. It uses L = 2 and T = 2, and PPO gets an exact tabular critic while DAM and d1 rely on sampled estimates. The authors note that d1 is not designed for this task.

  7. Compute cost is not reported. CTRL needs N forward passes per update, while d1 needs one. There is no wall-clock or FLOP comparison, so the efficiency claims can’t be judged.

  8. Missing baselines. The paper discusses TraceRL, DiffPO, wd1, and ESPO, all closely related and some using process rewards, but does not compare against them.

Questions for authors

see weakness

6 comments

We appreciate the reviewer's detailed comments. Our responses are provided below, and all revisions have been highlighted in blue in the revised paper.


Q1. The theory and the actual algorithm are largely disconnected.

A1. We thank the reviewer for raising these comments. We respond to each point below.

On the vertex policy reducing to token-level GRPO. We agree that the vertex-supported policy yields per-step token-level ratios conditioned on xt\mathbf x_t for dLLMs. We see this as a special case of the framework, not a departure from it. The framework treats the denoiser output as an action and asks which policy class makes the ratio tractable and stable. For large vocabularies, the answer is the degenerate (vertex) member of that class. The framework implies that the ratio must be conditioned on the actual intermediate state xtx_t rather than the prompt. As shown in Appendix F.2, d1’s loss implicitly corresponds to a policy (Equation (23)) that differs from the one it actually samples from (Equation (22)). Our loss (13) removes this mismatch.

On similarity to d2 and TraceRL. We agree the methods are related, in that all three use states from the realized trajectory. The differences are as follows. d2 approximates the trajectory likelihood with block composite factors, whereas we compute the exact per-step ratio and use subsampling, which gives an unbiased estimate (Proposition 4). TraceRL supports process rewards through a learned value network with GAE using a Qwen3-4B process reward model, while our framework instead derives per-step advantages directly from group-normalized intermediate rewards (Ag,tA_{g,t} in Algorithm 2), eliminating the expensive value network. More fundamentally, our objective (8)–(9) is derived from a controlled-CTMC formulation. That formulation gives intermediate rewards a principled role as the running reward rr, and Theorem 2 gives the KL regularizer as an exact path-measure divergence rather than a heuristic penalty. We have added a paragraph clarifying the differences on Lines 396-402 in the revised paper.

On the q-function, the policy gradient theorem, and PPO being unused. We respectfully note that the synthetic experiment (Section 5.1) uses exactly these components: PPO with the q-function as the advantage and an exact or Monte-Carlo-based critic. It outperforms d1 and matches DAM under the same computational budgets. For dLLMs, we agree that PPO is currently impractical. A learned value network over 8B-parameter states is costly, and the rollout-based estimator in Appendix G has high variance. However, the dLLM GRPO algorithm (Algorithm 2) is still the framework’s algorithm. Its advantage Ag,tA_{g,t} combines running rewards along the trajectory with the terminal reward.

Overall, our main contribution is the development of a CTRL framework for discrete spaces, together with its adaptation to dLLM post-training and a systematic empirical evaluation.


Q2. Limited theoretical novelty.

A2. We thank the reviewer and agree that our theoretical development follows the strategy of Wang et al. (2020) and Jia & Zhou (2022a; 2023), as we acknowledge in the paper. This similarity is expected: SDEs and CTMCs are both Feller processes, and the generator-based arguments (Feynman–Kac, Dynkin’s formula, and q-function) carry over. We do not claim new proof techniques. Our theoretical contribution is the formulation and its application in fine-tuning discrete diffusion models, which we clarify below.

On novelty of the formulation. Classical CTMDP theory (Miller, 1968; Guo & Hernández-Lerma, 2009; Piunovskiy & Zhang, 2020) assumes known rates and rewards. We develop the model-free, exploratory counterpart for finite-horizon, time-inhomogeneous controlled CTMCs with terminal rewards, and derive a policy gradient theorem (Theorem 1) for parameterized stochastic policies over general, possibly continuous and high-dimensional action spaces. The key modeling step is recognizing that the reverse process of a discrete diffusion model is a controlled CTMC with linear action structure (Example 1, Proposition 3), with the score or denoiser as the action.

On Meng et al. (2026). Our work was developed independently and predates Meng et al. (2026). Both papers extend Wang et al. (2020) and Jia & Zhou (2023) to discrete state spaces and define an exploratory rate, an HJB equation, and a qq-function, which we acknowledge in Appendix A. The papers differ in setting and applications. Meng et al. (2026) study q-learning with jump rewards and low-dimensional action spaces (e.g., prices, server counts) for network pricing. We study finite-horizon, time-inhomogeneous problems with terminal rewards, derive a policy gradient theorem, and develop PPO and GRPO algorithms for high-dimensional space in dLLMs.

Q3. Rollout/training policy mismatch.

A3. We thank the reviewer for identifying this important distinction. We acknowledge that the ratio is not an exact importance-sampling correction for the actual rollout distribution induced by stochastic temperature perturbations. Instead, it serves as a practical surrogate probability ratio, motivated by implementation simplicity and empirical training stability, particularly given the large vocabulary of dLLMs (see Remark 3 in the revised paper). We have revised the derivation of this surrogate objective on Lines 343-361.

We would also like to clarify the scope of our claim regarding "consistency between sampling and optimization." This consistency refers to the conditioning of the policy and its probability ratios on the actual intermediate state xt\mathbf x_t, rather than on the prompt alone as in d1. Specifically, as shown in Appendix F.2, the d1 loss implicitly optimizes a policy (Equation (23)) that differs from the policy used to generate rollouts (Equation (22)), resulting in a mismatch between sampling and optimization. In contrast, our loss in Equation (13) explicitly conditions the token-level policy ratio on the actual intermediate state xt\mathbf x_t, thereby eliminating this particular mismatch.


Q4. The comparison with baselines is unfair.

A4. We thank the reviewer for raising these important concerns. To address this concern, we conducted an ablation study by training CTRL with only the final outcome-level reward, removing all intermediate rewards (denoted as ``CTRL w.o. IR''). All other training settings remain unchanged, and all baselines are evaluated using the same outcome-level reward signal, ensuring a fair comparison. We have added the ablation study on Lines 492-511 in the revised paper.

Regarding the coding tasks, no intermediate reward functions are used in our experiments. CTRL is trained solely with the final outcome-level reward on these tasks. We have added this clarification to the revised manuscript (Line 511).

Our ablation results on CTRL with and without intermediate rewards reveal three findings:

  • Sudoku: The gains primarily come from CTRL's policy optimization algorithm. Even without intermediate rewards, CTRL achieves an accuracy of 82.9%, outperforming the strongest baseline, d2, by 14.9 percentage points. Adding intermediate rewards yields a further improvement of 5.3 percentage points. This demonstrates that although intermediate rewards provide additional benefits, they are not the primary source of CTRL's substantial improvement over existing methods.

  • GSM8K and MATH500: CTRL remains competitive without intermediate rewards. At sequence length 256, CTRL w/o IR achieves the best or near-best performance. Intermediate rewards lead to relatively modest additional improvements, suggesting that CTRL's optimization algorithm is effective even with sparse outcome-level supervision on these tasks.

  • Countdown: Intermediate rewards provide substantial benefits. Without intermediate rewards, CTRL achieves an accuracy of only 47.7%, falling behind d2, which also uses multiple forward passes. This suggests that intermediate rewards are particularly important for Countdown, where the outcome-level reward is highly sparse: a solution receives a positive reward only when the final expression is exactly correct. Intermediate rewards provide denser feedback and facilitate credit assignment, which is especially beneficial for CTRL's state-conditioned policy updates.

Overall, these results demonstrate that CTRL's optimization algorithm accounts for most of the Sudoku improvements, while intermediate rewards provide additional benefits, especially on tasks with sparse outcome-level rewards.


Q5. The evaluation protocol favours the proposed method.

A5. We thank the reviewer for raising these important points regarding the evaluation protocol. We clarify each point below.

Checkpoint selection. We applied the same checkpoint selection protocol to all methods, evaluating every 200 training steps and reporting the best-performing checkpoint. Thus, all methods, including the baselines, are subject to the same selection procedure.

Sequence length. Table 4 provides results for all methods at both sequence lengths (128 and 256). As explained in Lines 1628–1633, we report Sudoku results with a completion length of 128 and results on all other tasks with a completion length of 256. This choice is based on the observation that models consistently perform better with the shorter completion length across all methods, including the base LLaDA model, as shown in Table 4. We therefore apply the same sequence-length setting to all methods for each task.

Seeds and variance. We acknowledge that reporting results across multiple random seeds would provide a more comprehensive assessment of performance variability. Due to computational constraints, we report single-run results, following the evaluation practice adopted in d1 and d2, which also report their best checkpoints from single runs. We agree that multi-seed evaluations would be a valuable direction for further investigation.

Baseline hyperparameters. For a fair comparison, we reran each baseline (d1, d2, and SPG) using the default hyperparameters provided in their respective official implementations, rather than applying CTRL's hyperparameters to all methods.

SPG and 3-shot prompting. We used 0-shot prompting consistently across all methods, during both training and evaluation, to ensure a controlled comparison under the same prompting setup. We note that the results reported in the original SPG paper use 3-shot prompting and are therefore obtained under a different evaluation protocol. Consequently, these results are not directly comparable to those in Table 1.


Q6. The synthetic experiment is not informative. It uses L = 2 and T = 2, and PPO gets an exact tabular critic while DAM and d1 rely on sampled estimates. The authors note that d1 is not designed for this task.

A6. We thank the reviewer for pointing this out.

Exact critic vs. sampled estimates. We reran the experiment with matched estimation settings and matched optimization budgets in Section 5.1 of the revised paper. We use matched estimators: CTRL-PPO is evaluated with both an exact critic and a Monte Carlo critic using K=12K=12 rollouts, the same sample budget as DAM’s adjoint. DAM is evaluated with both its Monte Carlo adjoint and an exact adjoint. In addition, all methods share the same initialization, step-size schedule and four gradient steps per iteration, with DAM’s step size scaled to match the scale of the PPO gradient.

Under these settings, CTRL-PPO and DAM both recover p∗p^* to numerical precision (DKL≈10−15D_{\mathrm{KL}}\approx 10^{-15}), and both reach DKL<10−3D_{\mathrm{KL}}<10^{-3} within 30 iterations. The choice of estimator does not affect DAM at any budget, and it leaves CTRL-PPO’s convergence speed unchanged, adding only a small variance floor (DKL≈4×10−5D_{\mathrm{KL}}\approx 4\times 10^{-5}). We have accordingly removed our earlier claim that PPO converges faster than DAM.

d1 “not designed for this task.” The setting is in fact favorable to d1 in one respect: with L=2L=2, d1’s one-step log-probability estimator conditions on the actual state at which each token is unmasked, so its likelihood is exact. d1’s remaining gap to p∗p^* (DKL≈0.074D_{\mathrm{KL}}\approx0.074) therefore reflects its objective (terminal reward only, with group-standardized advantages) rather than likelihood approximation error. The experiment thus isolates the effect of the objective, which is the comparison we intend. We have revised the text accordingly.


Q7. Compute cost is not reported. CTRL needs N forward passes per update, while d1 needs one. There is no wall-clock or FLOP comparison, so the efficiency claims can’t be judge.

A7. We thank the reviewer for pointing this out. We have added a discussion on the computational cost to the revised paper on Lines 513-528. The table below reports the average wall-clock time for 1,000 training steps on the four mathematical reasoning tasks with L=128L=128 and identical batch and group sizes for all methods. Notably, despite requiring 16×16\times more forward passes, CTRL incurs only a 6.3×6.3\times increase in wall-clock time while achieving substantially higher accuracy, making this additional computational cost worthwhile. CTRL is also faster than d2 under the same number of forward passes of 16, which likewise evaluates multiple intermediate states.

MethodForward passes per updateWall-clock (1k steps)Relative to d1
d112h 30m1.0×
CTRLN=16N=169h 30m6.3×
d2N=16N=1611h 36m7.7×

In addition, We do not claim CTRL is more efficient than d1. Our efficiency claim is narrower: using trajectory subsampling in CTRL reduces cost from TT to NN forward passes per rollout compared with exact evaluation of the loss (13), while remaining unbiased (Proposition 4), as discussed in Lines 389–395.

Reviewer_apP3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a continuous-time reinforcement learning framework, called CTRL, for fine-tuning discrete diffusion models. The main idea is to formulate the reverse diffusion process as a controlled Continuous-Time Markov Chain (CTMC), where the score function is interpreted as an action. This formulation allows the authors to derive policy-gradient methods, including continuous-time variants of PPO and GRPO.

For masked diffusion models, the framework is specialized by considering policies over the probability simplex of the denoiser. The authors also propose incorporating intermediate rewards throughout the denoising trajectory, as well as a trajectory subsampling strategy to reduce the computational cost of evaluating policy probability ratios.

On the experimental side, the method is evaluated on a synthetic distribution-matching problem and on reasoning and coding tasks using LLaDA-8B-Instruct. The method generally improves upon the considered baselines, with particularly interesting results on Sudoku and coding benchmarks.

Critical strengths
  • The paper treats an important problem in the area of discrete generative modeling, about the fine-tuning of discrete diffusion models with reinforcement learning (RL).

  • The paper is well written overall, and it is quite clear to understand the background, the mathematical formulation, and the problem it aims to tackle.

  • The approach of bringing continuous-time stochastic control techniques to discrete state spaces is interesting, and provides a different perspective on existing policy-gradient methods (to my knowledge). In particular, I find the mathematical setup really well written, and better from what we usually see in this literature.

  • Empirical results are interesting. In particular, the method consistently improves upon the considered baselines, with substantial improvements on Sudoku and coding tasks. The trajectory subsampling strategy also seems to provide a useful computational trade-off when evaluating the GRPO objective.

Critical weaknesses
  • The main weakness I see is the limited novelty of the practical algorithm compared to existing methods. In particular, approaches such as TraceRL [1] and d2 [2] have already proposed trajectory-aware policy optimization for diffusion language models, by conditioning on intermediate denoising states and estimating trajectory likelihoods. TraceRL also considers process rewards and techniques to reduce the computational cost of training. Although the continuous-time perspective is interesting, it is currently unclear to me what fundamentally new algorithmic ingredient is introduced by the proposed GRPO objective in Equation (12), beyond a different theoretical formulation and the use of trajectory subsampling.

  • I have some concerns about the consistency between the theoretical framework and the actual sampling procedure. The CTMC formulation in Equation (15) defines transition rates that do not depend on the confidence of the selected position, whereas the practical algorithm uses top-confidence unmasking. Moreover, the actual rollouts use temperature-perturbed distributions, while the probability ratios in Equation (11) appear to be computed using the standard denoiser probabilities. As a result, I am not convinced that the proposed objective is exactly consistent with the sampling policy, which is nevertheless one of the main motivations of the paper (see questions below).


[1] Wang et al., Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models (TraceRL), ICLR 2026.

[2] Wang et al., d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation, ICML 2026.

Questions for authors

I have five main questions, mostly concerning the novelty of the method and the consistency between the theoretical formulation and the practical algorithm.

1. Novelty relative to existing trajectory-based RL methods.

I would have appreciated a clearer discussion of the methodological differences with existing approaches, in particular TraceRL [1] and d2 [2]. Both methods already leverage intermediate denoising states to construct policy-gradient objectives, and TraceRL additionally considers process rewards and a strategy for reducing the number of forward passes during training.

In particular, Equation (12) seems quite close to existing trajectory-conditioned GRPO objectives, except for the precise construction of the probability ratios and advantages. Can the authors clarify what is fundamentally new in the proposed algorithm compared to these methods? Is the contribution mainly the continuous-time theoretical justification, or does the resulting algorithm provide something that previous methods could not achieve?

I think this distinction is important to better understand the significance and originality of the paper.

2. Consistency between the CTMC formulation and confidence-based sampling.

I spent some time going through the mathematical formulation, and I am not sure that the CTMC described in Section 4 corresponds to the actual sampling algorithm.

Indeed, from Equation (15), for a masked position ii, the reverse transition rate is

=\frac{1}{1-t}p_\theta^{(i)}(j\mid x).$$ Summing over all possible tokens $j$, we obtain $$\sum_j R_t^\theta(x,x^{i\leftarrow j}) =\frac{1}{1-t},$$ which is independent of the position $i$ and of the confidence of its predicted tokens. Therefore, if I understand correctly, the underlying CTMC does not favor high-confidence positions when deciding where to jump. However, the actual sampling algorithm in Equation (10) deterministically selects the $k$ highest-confidence positions at each step. I know these confidence-style samplers are heavily used, in particular for masked diffusion models. But this seems to define a different transition process, whose position-selection mechanism depends on the model parameters. How is this discrepancy accounted for in the theoretical framework? In particular, Equation (11) only considers the probability ratios of the sampled tokens, but seems to ignore the probability of selecting their positions. Since this selection is parameter-dependent, I am not convinced that Equation (11) represents the exact likelihood ratio of the actual sampling policy. Could the authors clarify whether Equation (11) is intended as an exact likelihood ratio or as a surrogate conditioned on the selected positions? This matters for the claimed consistency between the theoretical objective and the practical algorithm. **3. Probability ratios under temperature-based exploration.** Related to the previous question, Appendix H.2 explains that the training rollouts use a dimension-aware temperature-softmax exploration strategy, where the temperature variables are independently sampled from an exponential distribution. However, Equation (11) seems to construct the policy probability ratios directly from $p_\theta$ and $p_{\theta^{\mathrm{old}}}$. Unless I misunderstood the implementation, the distributions used to generate the rollouts are therefore different from those used to compute the probability ratios. Can the authors clarify whether the sampled temperature variables are stored and reused when evaluating the new and old policies, or whether the ratios are evaluated using the original, unperturbed denoiser probabilities? If the latter is the case, is there any theoretical justification for using these ratios as importance-sampling corrections for the actual rollout distribution? **4. Understanding the empirical improvements and the role of intermediate rewards.** One of the strongest empirical results is the improvement on Sudoku, which the authors mainly attribute to the use of intermediate rewards. However, the compared methods do not appear to benefit from the same reward-shaping mechanism. This makes it difficult to disentangle the contribution of the proposed policy optimization algorithm from that of the additional reward information. I would have appreciated an ablation study comparing CTRL with and without intermediate rewards, ideally alongside existing methods equipped with similar reward signals. Similarly, it would be interesting to isolate the impact of conditioning on intermediate states from the effect of trajectory subsampling, while accounting for the corresponding computational costs. Of course this represents a non trivial addition, but if the authors has any insights about this disentangling, i think this will greatly benefits the paper. **5. Connection between the KL-regularized objective and the policy-gradient theorem.** I also had a theoretical question regarding the transition from Equation (8) to Equation (9). Theorem 1 derives a policy-gradient expression for a controlled CTMC with a prescribed running reward $r(t,x,a)$. However, after rewriting the path-space KL regularization using Theorem 2, the resulting running reward in Equation (9) depends explicitly on the policy through $\mathbb E_{\pi_\theta}[a]$. Consequently, differentiating the objective would appear to introduce additional terms due to the explicit dependence of the reward on $\theta$. (Correct me if I am wrong, maybe I am missing something important...). Is there a cancellation argument, or a specific action-level reward construction, that ensures the policy-gradient expression of Theorem 1 remains valid for this KL-regularized objective? I think a more explicit derivation would strengthen the theoretical contribution. Related to this, it is not clear to me from Appendix H.2 whether the path-KL regularization is actually used in the LLM experiments, since the corresponding coefficient is not specified. --- Despite these questions, I think the paper contains some interesting contributions, particularly in its continuous-time perspective and empirical results. However, I am currently slightly in favor of rejection, mainly because the practical algorithm seems quite close to existing trajectory-aware methods, while the claimed consistency between the theoretical framework and the sampling procedure remains unclear to me. I am open to increasing my score if the authors can clarify these points, especially the precise novelty relative to previous methods and the correctness of the probability ratios used in practice. --- [1] Wang et al., *Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models* (TraceRL), ICLR 2026. [2] Wang et al., *d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation*, ICML 2026.
6 comments

We appreciate your detailed review. Our responses are provided below, and all revisions have been highlighted in blue in the revised manuscript.


Q1. The main weakness I see is the limited novelty of the practical algorithm compared to existing methods. It is currently unclear to me what fundamentally new algorithmic ingredient is introduced by the proposed GRPO objective in Equation (12), beyond a different theoretical formulation and the use of trajectory subsampling.

A1. We thank the reviewer for this insightful comment. We would like to clarify the primary scope and contributions of our work. For the new algorithmic ingredient, please refer to our response to Q3.

The main objective of this work is not merely to introduce another practical loss function for fine-tuning diffusion language models, but rather to establish a general continuous-time RL framework for discrete state spaces, grounded in classical stochastic control theory, and demonstrate its applicability to discrete diffusion models, with a particular focus on dLLMs.

We acknowledge that existing methods, including TraceRL and d2, have explored trajectory-aware policy optimization for diffusion language models. Our work differs primarily in its starting point and theoretical foundation: rather than designing an optimization objective specifically for dLLMs, we first develop a general model-free continuous-time policy optimization framework for discrete spaces, and then instantiate it for any score-based discrete diffusion models.

However, adapting this general framework to dLLMs presents substantial computational challenges due to their large vocabulary sizes and long sequence lengths. A naive implementation of CTRL with general probability-distribution-valued policies would be prohibitively expensive. We therefore introduce several practical design choices to make the framework computationally tractable for dLLM fine-tuning. In particular, we parameterize policies over the vertices of the probability simplex and employ trajectory subsampling to reduce the computational cost of policy optimization.

Overall, we view the main contribution as establishing a unified theoretical and algorithmic framework for model-free continuous-time RL in discrete spaces, rather than claiming that the GRPO objective in isolation constitutes an entirely new optimization paradigm. We believe this provides a principled foundation for RL fine-tuning of discrete diffusion models and complements existing trajectory-based approaches.


Q2. I have some concerns about the consistency between the theoretical framework and the actual sampling procedure. I am not convinced that the proposed objective is exactly consistent with the sampling policy.

A2. Thank you for pointing this out. The concerns are addressed in our responses to Q4 and Q5.


Q3. Novelty relative to existing trajectory-based RL methods. I would have appreciated a clearer discussion of the methodological differences with existing approaches, in particular TraceRL [1] and d2 [2]. Can the authors clarify what is fundamentally new in the proposed algorithm compared to these methods? Is the contribution mainly the continuous-time theoretical justification, or does the resulting algorithm provide something that previous methods could not achieve?

A3. We appreciate the reviewer’s insights. We have added a paragraph clarifying the differences between our method and TraceRL and d2 in the revised paper (Lines 396–403). Our new algorithmic ingredient and main differences with TraceRL and d2 are summarized as follows:

  • We use per-step advantages with intermediate rewards without a value network. d2 has no intermediate rewards at all. TraceRL also supports process rewards, through a learned value network with GAE. Our framework instead derives per-step advantages directly from group-normalized intermediate rewards (Ag,tA_{g,t} in Algorithm 2), so no value network is trained or evaluated.

  • Our KL term enters the running reward. The KL to the pretrained model comes from the path-measure KL (Theorem 2), not from a heuristic per-token penalty. Also, our CTRL framework implies that the KL divergence term is incorporated into the running reward, which is the intermediate reward function (IRF) minus a KL-to-pretrained penalty (Equation (9)). This is a concrete consequence of our continuous-time RL objective. However, d2 and TraceRL use per-token KL estimators added to the GRPO loss.

  • Our trajectory subsampling is unbiased while step merging in biased. d2-StepMerge is a deterministic approximation with an analytically bounded but nonzero error. TraceRL’s shrinkage is also deterministic merging, and the d2 authors themselves call it heuristic. Our random subsampling keeps every evaluated term conditioned on its exact xt\mathbf x_t, so the estimate of Equation (13) is unbiased.

Q4. Consistency between the CTMC formulation and confidence-based sampling. How is this discrepancy accounted for in the theoretical framework? Could the authors clarify whether Equation (11) is intended as an exact likelihood ratio or as a surrogate conditioned on the selected positions?

A4. We thank the reviewer for this careful observation. It is indeed a surrogate ratio conditioned on the selected positions, not the exact likelihood ratio of the top-k confidence sampler. We have clarified the derivation of the surrogate probability ratio on Lines 338-361 in the revised version.

Exact form. During the RL rollout, the action at\mathbf a_t is sampled from the old policy πθold(⋅∣t,xt)\pi^{\theta_\mathrm{old}}(\cdot\mid t,\mathbf x_t). Since the positions with top-kk confidence are now parameter-dependent, we denote this set by Mt,θTopkConf\mathcal M_{t,\theta}^\mathrm{TopkConf}. The exact probability ratio under the current policy πθ\pi^\theta is therefore

{\pi^{\theta_\mathrm{old}}(\mathbf a_t\mid t,\mathbf x_t)}=\mathbf 1\\{\mathcal M_{t,\theta}^\mathrm{TopkConf} =\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}\\} \frac{ \prod_{i\in\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}} \pi^{\theta,(i)}(\mathbf a_t^{(i)}\mid t,\mathbf x_t) }{ \prod_{i\in\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}} \pi^{\theta_\mathrm{old},(i)}(\mathbf a_t^{(i)}\mid t,\mathbf x_t) }=\mathbf 1\\{\mathcal M_{t,\theta}^\mathrm{TopkConf} =\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}\\} \prod_{i\in\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}} \frac{\tilde p_\theta^{(i)}(x_T^i\mid \mathbf x_t)} {\tilde p_{\theta_\mathrm{old}}^{(i)}(x_T^i\mid \mathbf x_t)},$$ where $\tilde{\mathbf p}_\theta^{(i)}(\mathbf x_t):=\mathrm{softmax}((\mathbf f_{\theta}^{(i)}(\mathbf x_t)_j/\tau_j)_{j\in\mathcal V})$ is the temperature-perturbed denoiser. The indicator appears because of the action construction in Equation (11) in the revised paper. For each $i\in\mathcal M_{t,\theta_{\mathrm{old}}}^\mathrm{TopkConf}$, the action component $\mathbf a_t^{(i)}$ is sampled from the $V$ vertices of the simplex $\Delta^{V-1}$ according to the categorical distribution $\tilde{\mathbf{p}}_{\theta_\mathrm{old}}^{(i)}(\mathbf x_t)$. In contrast, for $i\notin\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}$, the action component is deterministic, given by $\mathbf a_t^{(i)}=\mathbf e_{x_t^i}$. Consequently, the sampled action $\mathbf a_t$ is attainable under the current policy $\pi^\theta$ if and only if the two policies induce the same top-$k$ confidence set, i.e., $\mathcal M_{t,\theta}^\mathrm{TopkConf}=\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}$. This support constraint gives rise to the indicator in the exact probability ratio. **Tractable surrogate.** Directly evaluating the exact ratio requires tracking the change in the top-$k$ confidence set, which introduces the potentially discontinuous indicator. In our implementation, we therefore use the tractable surrogate probability ratio $$\rho_t^{\theta}\approx \prod_{i\in\mathcal M_{t,\theta_\mathrm{old}}^\mathrm{TopkConf}} \frac{\tilde p_\theta^{(i)}(x_T^i\mid \mathbf x_t)} {\tilde p_{\theta_\mathrm{old}}^{(i)}(x_T^i\mid \mathbf x_t)},$$ obtained by dropping the indicator. Importantly, this surrogate is locally exact under a mild non-degeneracy condition: if the $k$-th and $(k+1)$-th largest confidence scores are strictly separated at every visited state, then the top-$k$ confidence set remains unchanged in a sufficiently small neighborhood of $\theta_\mathrm{old}$. Throughout this neighborhood, the indicator is identically equal to $1$, and hence the surrogate ratio coincides with the exact likelihood ratio. Thus, although dropping the indicator yields an approximation in general, it introduces no discrepancy locally around $\theta_\mathrm{old}$ under this condition. ___ **Q5.** Can the authors clarify whether the sampled temperature variables are stored and reused when evaluating the new and old policies, or whether the ratios are evaluated using the original, unperturbed denoiser probabilities? If the latter is the case, is there any theoretical justification for using these ratios as importance-sampling corrections for the actual rollout distribution? **A5.** We thank the reviewer for this question. The sampled temperatures are not stored. The ratios are evaluated with the unperturbed denoiser probabilities $\mathbf p_\theta$, so this is a second surrogate approximation. We have clarified this in the revised paper on Lines 354-362 and let us explain this below.

We use the temperature-perturbed rollouts during sampling to encourage exploration, so that at\mathbf a_t is sampled from the perturbed denoiser p~θ(xt)=softmax((fθ(xt)j/τj)j),\tilde{\mathbf p}_{\theta}(\mathbf x_t)=\mathrm{softmax}((f_{\theta}(\mathbf x_t)_j/\tau_j)_j),

which is a perturbation of the standard pθ(xt)=softmax(fθ(xt))\mathbf p_\theta(\mathbf x_t)=\mathrm{softmax}(\mathbf f_\theta(\mathbf x_t)) and can be viewed as an exploration of the standard pθ(xt)\mathbf p_\theta(\mathbf x_t). However, perturbing the denoiser probability vector yields a simplex-level exploration (Appendix F.1.2), which would lead to unstable training due to the large vocabulary size (Equation (18) and Remark 3 in the revised paper) when computing probability ratio which would include a summation over the vocabulary. Also, the exact probability ratio of simplex-level temperature-softmax cannot be exactly calculated because the softmax function is not injective, so the approximation still appears if we adopt the simplex-level calculation (Equation (18)). Thus, during loss calculation in optimization, we simplify the computation of the probability ratio using the standard denoiser pθ(xt)\mathbf p_\theta(\mathbf x_t) as a surrogate without considering the stochasticity of the temperature, leading to a vertex-level expression where the summation over the vocabulary is eliminated; this simplifies the practical implementation and would significantly improves the training stability as we empirically observed. Our final surrogate probability ratio is therefore

\prod_{i\in\mathcal M_{t,\theta_{\mathrm{old}}}^{\mathrm{TopkConf}}} \frac{p_\theta^{(i)}(x_T^i\mid \mathbf x_t)} {p_{\theta_\mathrm{old}}^{(i)}(x_T^i\mid \mathbf x_t)}.$$ We acknowledge that this ratio is not an exact importance-sampling correction for the actual rollout distribution induced by the stochastic temperature perturbations. Rather, it serves as a surrogate probability ratio motivated by implementation simplicity and empirical training stability. ___ **Q6.** I would have appreciated an ablation study comparing CTRL with and without intermediate rewards, ideally alongside existing methods equipped with similar reward signals. Similarly, it would be interesting to isolate the impact of conditioning on intermediate states from the effect of trajectory subsampling, while accounting for the corresponding computational costs. **A6.** We thank the reviewer for this insightful comment. We address the three concerns separately and have added the corresponding discussions to the revised paper (Lines 492–511). **1. CTRL with and without intermediate rewards.** We conducted the suggested ablation by training CTRL using only the final outcome-level reward, without intermediate rewards (denoted as ``CTRL w.o. IR''). The results are reported in Table 1 and Table 5 in the revised paper. We make three observations: - **Sudoku improvements primarily come from the policy optimization algorithm, rather than reward shaping alone.** Even without intermediate rewards, CTRL achieves 82.9, outperforming the strongest baseline d2 by 14.9 percentage points. Intermediate rewards provide further gains of 5.3 points. - **On GSM8K and MATH500, CTRL remains competitive without intermediate rewards.** It achieves the best or near-best performance at sequence length 256. Intermediate rewards provide relatively modest gains. - **On Countdown, intermediate rewards play a crucial role.** Without them, CTRL achieves only 47.7, falling behind d2 that also uses multiple forward passes. Intermediate rewards provide denser credit assignment, which is particularly beneficial for CTRL's state-conditioned policy updates. Overall, these results demonstrate that CTRL's optimization algorithm accounts for most of the Sudoku improvements, while intermediate rewards provide additional benefits, especially on tasks with sparse outcome-level rewards. **2. Baselines equipped with similar reward signals.** Incorporating intermediate rewards into d1, d2, and SPG is not straightforward. These methods optimize sequence-level objectives and do not naturally support per-state reward signals. Therefore, we focus on isolating the contribution of intermediate rewards within CTRL, while keeping the outcome-level reward and all other training settings fixed. This ablation directly demonstrates that the improvements are not solely attributable to additional reward information. **3. Conditioning on intermediate states vs. trajectory subsampling.** We clarify that trajectory subsampling serves as an efficient, unbiased estimator of the full GRPO objective that would require computationally expensive $T$ forward passes. Instead, CTRL samples $N$ intermediate times to estimate this objective using only $N$ forward passes. Thus, trajectory subsampling is an efficiency mechanism for evaluating the state-conditioned objective, rather than a separate source of algorithmic improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.