acceptodds
Desk rejected at ICLR 2027

dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of a mixture path and a masked source distribution. While several recent works have explored reinforcement learning for dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate the denoising process as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using Flow-GRPO, while also demonstrating strong capabilities on understanding tasks.

Primary area:
generative models
Keywords:
Discrete Flow Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2EupOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to fine-tune discrete flow models, which generate all tokens in parallel and refine them over several denoising steps, with GRPO-style reinforcement learning. Policy-gradient training needs the ratio between the new and old policy's probability of each denoising step, and existing RL methods for diffusion language models only approximate this ratio and mostly cover the masked special case. The authors derive the exact per-step ratio for general discrete flow models, it combines the model's prediction of the final token with a term that depends on the transition rate, and it can be estimated from samples already drawn during generation without extra forward passes. Applied to the multimodal model FUDOKI withoutCFG, dFlowGRPO raises PickScore from 20.87 to 23.00 and GenEval from 61% to 93%, and ScienceQA accuracy from 75.1% to 81.1%, while most auxiliary understanding benchmarks change little. On the same base model, the mean-field baselines diffu-GRPO and diffu-GSPO collapse during training, whereas dFlowGRPO keeps improving.

Critical strengths

The exact per-step ratio is the right object, and the empirical consequence is visible. Replacing mean-field likelihood surrogates with the token-wise expected posterior ratio under a general probability path is principled, reduces to the known masked-model ratio as a special case, and costs no extra forward passes. It also appears to buy training stability that the approximations do not: in Fig. 5 both mean-field baselines collapse during training while dFlowGRPO improves steadily . Non-collapse against a same-base approximate-ratio method is the strongest evidence in the paper that the exactness matters.

The gains are not confined to the training reward. The clearest test is the held-out object split: the model is trained on 60 GenEval object categories and evaluated on prompts built only from the other 20, and overall GenEval still rises from 0.5602 to 0.7720, with every category improving. The GenEval-trained model also improves all seven T2I-CompBench++ categories, the PickScore-trained model improves the other DrawBench preference metrics it was not trained on , and the ScienceQA-trained model slightly improves MMMU and MM-Vet. DPG-Bench stays roughly flat. The main risk in reward-driven fine-tuning of a generator is overfitting the reward model.

Critical weaknesses

The controlled comparison behind the main claim is narrow, and the baselines may be disadvantaged by the setup. The abstract claims that dFlowGRPO outperforms existing GRPO-type methods for dLLMs, and is competitive with continuous flow-based models trained using Flow-GRPO. The only experiment that compares RL methods on the same base model uses a single reward (PickScore), without KL regularization. The main GenEval comparison in Tab 1 lists results from different base models, training data and sampling settings, while its FUDOKI reference is evaluated with CFG while dFlowGRPO is not. It does not isolate the effect of the training objective.

Questions for authors
  1. Were the clipping parameters and learning rate tuned separately for diffu-GRPO and diffu-GSPO in Fig. 5? If not, do the baselines still collapse with their own hyperparameter search, and does the gap persist when plotted against wall-clock time (Table 8)?

  2. Which checkpoint do the no-LLM-extraction ScienceQA numbers "54.1% to 59.1%" (p8) refer to? In Fig. 2 (left), the dashed curve is at roughly 59% around step 900 and roughly 63% at step 1500, which is where the 81.1% with LLM extraction is marked. Please report both settings for the same checkpoint.

  3. Please report POPE and MME-P numerically in the main text alongside the gains, so that the 86.1 to 84.4 and 1485.4 to 1479.8 changes (printed in the Fig. 2 radar chart) are visible outside the radar chart, and state explicitly which auxiliary metrics regress.

2 comments

We thank the reviewer for the valuable comments and suggestions. We appreciate the time you spent on the paper. We address your concerns below.

W1: The controlled comparison behind the main claim is narrow, and the baselines may be disadvantaged by the setup. The abstract claims that dFlowGRPO outperforms existing GRPO-type methods for dLLMs, and is competitive with continuous flow-based models trained using Flow-GRPO. The only experiment that compares RL methods on the same base model uses a single reward (PickScore), without KL regularization. The main GenEval comparison in Tab 1 lists results from different base models, training data and sampling settings, while its FUDOKI reference is evaluated with CFG while dFlowGRPO is not. It does not isolate the effect of the training objective.

A: Thanks for your comment. First, Flow-GRPO uses a single reward in each setting, and we follow their setup and also train the model with one reward in each setting. For FUDOKI, we consider three settings; we use PickScore and GenEval for image generation, and ScienceQA for multimodal understanding. The experimental results show that our method improves the reward in all three settings. We also conduct experiments on FS-DFM for sentiment-controlled text completion, where the reward improves from 1.52 to 1.99. Therefore, these results show the effectiveness of our method across different tasks and base models. Second, for image generation, we only train the model with KL regularization in the GenEval setting to show that KL regularization can mitigate reward hacking, and keep the other experiments without KL regularization. Third, for Table 1, we just want to use the GenEval score reported in the FUDOKI paper directly to show that our method can improve the performance of the base model and achieve a higher score even when we start from a model without CFG. (Of course, from Figure 3, we can see that FUDOKI's score without CFG is 0.61, which is lower than the reported CFG version's score of 0.77) Finally, to post-train FUDOKI with CFG, we can just use CFG during both rollout and policy updates.

Q1: Were the clipping parameters and learning rate tuned separately for diffu-GRPO and diffu-GSPO in Fig. 5? If not, do the baselines still collapse with their own hyperparameter search, and does the gap persist when plotted against wall-clock time (Table 8)?

A: This is a good observation. We did not tune the baselines separately, and used the same learning rate and clipping parameters as in our main experiments to keep the settings consistent. We observed that both mean-field methods' rewards improved overall during the first 2000 steps, suggesting that these parameters were within a reasonable range. Additionally, dFlowGRPO exceeds the best observed rewards of both baselines with less or comparable estimated training time and continues to improve after achieving them (see both Figure 5 and Table 8, and see the following table). Smaller learning rates or tighter clipping parameters may slow reward improvement under the same time budget and may delay collapse.

MethodPickScoreStepsHours
diffu-GRPO (peak)21.21350030.45
dFlowGRPO (first above)21.345006.40
diffu-GSPO (peak)22.20270022.41
dFlowGRPO (first above)22.24170021.76

Q2: Which checkpoint do the no-LLM-extraction ScienceQA numbers "54.1% to 59.1%" (p8) refer to? In Fig. 2 (left), the dashed curve is at roughly 59% around step 900 and roughly 63% at step 1500, which is where the 81.1% with LLM extraction is marked. Please report both settings for the same checkpoint.

A: Thanks for pointing this out. This is a reporting error, and we have corrected the paper to report both settings at step 1500 (from 75.1% to 81.1% with LLM extraction, and 54.1% to 62.8% without LLM extraction).

Q3: Please report POPE and MME-P numerically in the main text alongside the gains, so that the 86.1 to 84.4 and 1485.4 to 1479.8 changes (printed in the Fig. 2 radar chart) are visible outside the radar chart, and state explicitly which auxiliary metrics regress.

A: Thanks for your suggestion. We have added these numbers in Section 5.3 of the revised version. Please see our updated PDF.

We thank the reviewer for the comments and suggestions, which have helped us improve the quality of our paper. We would be glad to clarify anything that remains unclear.

Thank you for the clarifications. My questions are resolved, and I keep my score.

Reviewer_PVvkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes dFlowGRPO, a GRPO-style RL method for discrete flow models (DFMs) with general probability paths, conditional rates and non-masked source distributions. Following DDPO/Flow-GRPO, it casts the denoising process as an MDP. Theorem 1 derives the per-step transition probability of the always-valid Euler sampler by marginalizing over the posterior sample X1X_1. The resulting policy ratio is a product of token-level expectations of posterior ratios weighted by a rate-dependent term, and it reduces to the known masked-diffusion ratio as a special case. The ratio is estimated with nn self-normalized Monte Carlo samples from the old posterior (Eqs. 6–7), without extra forward passes, and plugged into a clipped objective with a geometric mean over tokens (Eq. 2). Training rollouts are sampled with rates averaged over the same nn samples (Alg. 2).

The method is applied to FUDOKI, a multimodal DFM with a metric-induced path, trained with 8 steps and no CFG:

  • GenEval improves from 0.61 to 0.93 (Table 1);
  • PickScore improves from 20.87 to 23.00;
  • ScienceQA improves from 75.1 to 81.1 (Fig. 2).

Further evaluations cover DrawBench auxiliary rewards, DPG-Bench, T2I-CompBench++, held-out object categories (Table 5) and a second backbone (FS-DFM, Table 10). Mean-field baselines (diffu-GRPO/GSPO) collapse on FUDOKI (Fig. 5), and ablations vary nn and the rate weight (Fig. 4, Table 2).

Critical strengths

1. It addresses a real gap: policy-gradient RL for discrete flow models beyond masked diffusion. RL for discrete diffusion has relied on likelihood surrogates built for masked paths, such as mean-field and ELBO approximations. This paper gives the exact per-step transition probability of the single-sample Euler sampler for any factorized path and conditional rate (Theorem 1, App. B). The result recovers the masked-diffusion ratio of Zhang et al. (2026) as a special case, and it can be estimated from posterior samples that the sampler already draws. This makes MDP-based GRPO applicable to non-masked DFMs such as FUDOKI, where the mean-field surrogates tested collapse (Fig. 5). The direction is practically relevant as DFMs with uniform or metric-induced paths gain traction.

2. Large, consistent gains on a non-masked multimodal DFM, with sensible checks against reward hacking. Without CFG and with 8 training steps, FUDOKI improves on GenEval from 0.61 to 0.93 (Table 1), and on ScienceQA from 75.1 to 81.1 (Fig. 2). On held-out object categories, GenEval rises from 0.56 to 0.77 (Table 5). Auxiliary rewards and benchmarks mostly improve and do not degrade materially (Figs. 1, 3; Tables 3–4). Results are stable across evaluation seeds (Table 6), and the method also works on a second, text-only DFM (Table 10). These gains are the main reason I see the paper as close to the bar despite the concerns below.

Critical weaknesses

1. The relationship between the stated sampler and the transition-ratio derivation would benefit from clarification.

  • Sampler vs. ratio. Theorem 1 and Eqs. 6–7 concern the Euler kernel in which each token's rate comes from a single posterior sample. Although Algorithm 1 mentions rate averaging as a variance-reduction option, the proof in Appendix B is written for the single-sample kernel. Algorithm 2 (lines 9–11) instead samples rollouts with the rate averaged over the nn samples. These kernels generally differ. With step size hh, exit rate λ\lambda, and its sample average λˉ\bar\lambda, their stay probabilities are E[exp⁡(−hλˉ)]\mathbb{E}[\exp(-h\bar\lambda)] (averaged) and E[exp⁡(−hλ)]\mathbb{E}[\exp(-h\lambda)] (single-sample). By convexity, the first is no larger than the second, and they generally differ for non-degenerate rates. It would be helpful to quantify the practical size of this discrepancy at the 8-step training setting. As written, for n≥2n \ge 2, Eqs. 6–7 therefore appear not to estimate, in general, the transition ratio of the rate-averaged behavior kernel stated in Algorithm 2.
  • The n=1n=1 claim. The paper states that the n=1n=1 estimator "has a non-negligible bias" (p. 6). If X1X_1 is the same posterior sample that generated xtk+1x_{t_{k+1}}, the corresponding posterior-ratio estimator satisfies E[ρ(X1)∣xtk,xtk+1]=rk\mathbb{E}[\rho(X_1)\mid x_{t_k},x_{t_{k+1}}]=r_k under the single-sample kernel. It is then conditionally unbiased for this transition ratio, although its variance may be substantial. This statement concerns the ratio estimator, rather than the nonlinear clipped training objective. I would appreciate clarification of which notion of bias is intended.
  • Possible ways to address this concern. My main request is to clarify the implemented sampler and explain how the ratio is justified for that kernel. One possible consistent alternative is to sample each token's transition from one uniformly chosen XjX_j; Eqs. 6–7 would then give the Rao–Blackwellization of the n=1n=1 estimator. The estimator's error can also be measured on FS-DFM, where App. B gives the exact ratio.

2. Further evidence would help assess the incremental benefit of rate-awareness and interpret the baseline comparisons.

  • Ablations. Under a 13 h budget (Table 2; single runs, not trained to convergence), every n≥4n \ge 4 lies within 21.78–21.86 PickScore, against 21.66 for n=1n=1 and 21.55 for w=1w=1. In Fig. 4, n=24n=24 leads n=1n=1 by 22.83 vs. 22.47 after 4,100 updates, at about 1.4× the cost per update (Table 8). These results provide preliminary evidence for rate-awareness. However, no multi-seed training runs are available (App. D.6), so the robustness of these gaps to run-to-run variance remains unclear.
  • Baselines. The controlled comparisons to existing GRPO-style baselines focus on mean-field methods. They are run with dFlowGRPO's clipping range (Sec. 5.4; selected in App. E.1), although diffu-GRPO clips token-level rather than geometric-mean ratios. Appendix D.2 also compares dFlowDPO under the same denoising-step and KL settings, which is a useful additional comparison. An ELBO-type baseline could provide a useful additional point of comparison, if feasible.
  • Claims. The abstract says that dFlowGRPO "outperforms existing GRPO-type methods for dLLMs" and is "competitive with continuous flow-based models trained using Flow-GRPO". Both statements appear to rest largely on Table 1, where each method is applied to a different base model. Some qualification would help distinguish the reported model-level performance from conclusions about the training algorithms themselves.

The results show substantial gains from GRPO-style training on a DFM. The current ablations provide preliminary evidence for the rate-aware ratio, but do not yet establish how robust or substantial its additional benefit is.

Questions for authors
  1. [Clarification] Algorithm 2 (lines 9–11) samples transitions with the rate averaged over nn posterior samples, whereas Theorem 1 and the proof underlying Eqs. 6–7 use the single-sample Euler kernel. Algorithm 1 also mentions averaging as a variance-reduction option. Could you clarify which sampler your implementation uses, in training and at evaluation, and how the transition-ratio derivation applies to that sampler? This is my main question about soundness. If a sampler change is needed, one possible alternative is to sample each token's transition from one uniformly chosen XjX_j; it would be helpful to understand whether the conclusions remain valid under a consistent formulation.
  2. [Clarification] For n=1n=1, is the X1X_1 in the ratio the same sample that produced xtk+1x_{t_{k+1}} (Alg. 2, lines 8 and 14)? If so, the posterior-ratio estimator satisfies E[ρ(X1)∣xtk,xtk+1]=rk\mathbb{E}[\rho(X_1)\mid x_{t_k},x_{t_{k+1}}]=r_k for the corresponding single-sample transition ratio. Could you clarify whether the "non-negligible bias" on p. 6 refers to this ratio estimator or to a quantity involving the nonlinear training objective?
  3. [Suggestion] FS-DFM uses a mixture path, for which App. B gives the exact ratio in closed form, while App. F uses the n=24n=24 estimator. Would it be possible to report the discrepancy between r^k\hat r_k and the exact rkr_k on the same rollouts? If computationally feasible, training with the exact ratio would also be an informative follow-up.
  4. [Suggestion] If computationally feasible, a comparison of the full method, n=1n=1 and w=1w=1 for the same number of updates on PickScore, ideally with at least three training seeds, would help assess the reliability of the rate-dependent weight's additional benefit. A more focused estimate of training variability would also be useful if full repeated runs are too costly.
  5. [Clarification] Could you clarify whether the clipping range in Fig. 5 was tuned for the mean-field baselines, in particular for diffu-GRPO's token-level ratios? An ELBO-based baseline on FUDOKI would be helpful if feasible. For Table 1, it would also be useful to identify which numbers are taken from the original papers and the corresponding NFE and CFG settings.
4 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the valuable comments and suggestions. We appreciate the time you spent on the paper. We address your concerns below.

Q1 & W1 sampler vs. ratio: Algorithm 2 (lines 9–11) samples transitions with the rate averaged over nn posterior samples, whereas Theorem 1 and the proof underlying Eqs. 6–7 use the single-sample Euler kernel. Algorithm 1 also mentions averaging as a variance-reduction option. Could you clarify which sampler your implementation uses, in training and at evaluation, and how the transition-ratio derivation applies to that sampler? This is my main question about soundness. If a sampler change is needed, one possible alternative is to sample each token's transition from one uniformly chosen XjX_j; it would be helpful to understand whether the conclusions remain valid under a consistent formulation.

A: Thanks for your question and suggestion. First, in our training experiments, we use nn MC samples for both rate estimation and transition probability ratio estimation, as shown in Algorithm 2 for empirical implementation. Our main theory in Theorem 1 is based on the original Euler sampler with a single MC sample to estimate the transition rate in FUDOKI and [1]. As discussed in Section 3 of [1], computing the marginal transition rate exactly can be intractable, since it has a computational complexity of ∣S∣2\lvert \mathcal{S}\rvert^2, so we use the sample mean as a practical approximation, since we want to make full use of the MC samples from the posterior; that is, we not only use Equation 6 as an approximation to the transition probability ratio, but also we use the sample mean of conditional rates to approximate the marginal transition rate. Second, for evaluation, we use the original sampler (with one MC sample) in all experiments for a fair comparison. Finally, following your suggestion, we uniformly select one of the 24 posterior samples to estimate the transition rate and use all 24 samples to estimate the transition probability ratio. After 6300 training steps, this setting achieves a PickScore of 23.69, compared with 23.00 when using the sample mean to estimate the transition rate under the same evaluation pipeline. For FS-DFM, this setting also achieves a slightly higher reward than averaging the rates (see our response to Q3 and our uploaded revised version). These results show that our method remains effective with one-sample rate estimation and that the performance gains do not depend on averaging the conditional rates. (see Appendix D.9 in our uploaded revised version.)

[1] Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective. ICLR 2025.

Q2 & W1 the n=1n=1 claim: For n=1n=1, is the X1X_1 in the ratio the same sample that produced xtk+1x_{t_{k+1}} (Alg. 2, lines 8 and 14)? If so, the posterior-ratio estimator satisfies E[ρ(X1)∣xtk,xtk+1]=rk\mathbb{E}[\rho(X_1)\mid x_{t_k},x_{t_{k+1}}]=r_k for the corresponding single-sample transition ratio. Could you clarify whether the "non-negligible bias" on p. 6 refers to this ratio estimator or to a quantity involving the nonlinear training objective?

A: Thanks for pointing this out. In our implementation for n=1n=1, the generating MC sample is reused, so the ratio estimator is conditionally unbiased given the current–next state pair. Our bias discussion fixes the current–next state pair and samples X1d∼p1∣tkθold,d(⋅∣xtk)X_1^d\sim p_{1|t_k}^{\theta_{old},d}(\cdot\mid\mathbf{x}_{t_k}) independently of the observed next state. Given a fixed trajectory, the estimator is generally biased when we use a small number of posterior samples drawn independently of the observed next state, while the estimator in (6) is asymptotically unbiased as nn goes to infinity. We use uppercase X1dX_1^d for these auxiliary MC samples and lowercase xtk+1d\mathbf{x}_{t_{k+1}}^d for the fixed next state. We can see that

&\mathop{\mathbb{E}}\limits_{X_1^d\sim p_{1|t_k}^{\theta_{old},d}(\cdot\mid\mathbf{x}_{t_k})}\\!\left[ \frac{p_{1|t_k}^{\theta,d}(X_1^d\mid\mathbf{x}_{t_k})} {p_{1|t_k}^{\theta_{old},d}(X_1^d\mid\mathbf{x}_{t_k})} \right]=1,\\\\ &\mathop{\mathbb{E}}\limits_{X_1^d\sim p_{1|t_k}^{\theta_{old},d}(\cdot\mid\mathbf{x}_{t_k})}\\!\left[ w_{Q,k}^{\theta_{old},d}(\mathbf{x}_{t_k},\mathbf{x}_{t_{k+1}}^d,X_1^d) \frac{p_{1|t_k}^{\theta,d}(X_1^d\mid\mathbf{x}_{t_k})} {p_{1|t_k}^{\theta_{old},d}(X_1^d\mid\mathbf{x}_{t_k})} \right]=\frac{p_\theta^d(\mathbf{x}_{t_{k+1}}^d\mid\mathbf{x}_{t_k})}{p_{\theta_{old}}^d(\mathbf{x}_{t_{k+1}}^d\mid\mathbf{x}_{t_k})}, \end{aligned}$$ where the unweighted expectation is $1$, while the expectation with the exact weight equals the desired transition probability ratio. Our experiments also show the benefit of weighting and multiple MC samples (see Table 2). We have clarified this in Section 4.2 of our uploaded revised version.
AuthorsOpenReview ↗

Rebuttal 2

Q3: FS-DFM uses a mixture path, for which App. B gives the exact ratio in closed form, while App. F uses the n=24n=24 estimator. Would it be possible to report the discrepancy between r^k\hat r_k and the exact rkr_k on the same rollouts? If computationally feasible, training with the exact ratio would also be an informative follow-up.

A: Thanks for your question and suggestion. We compare ratios estimated following Algorithm 2 with the exact ratios in Appendix B, and vary the rate estimator and the number of MC samples used for ratio estimation. For computing the exact ratios, we consider both transitions generated using one MC sample and those using multiple MC samples (see Algorithm 1). We compare the ratios on the same FS-DFM rollouts. We use two (θold,θ)(\theta_{old},\theta) checkpoint pairs, at updates (0,48)(0,48) and (48,96)(48,96), with 64 IMDB test prompts and 8 rollouts per prompt. Each rollout uses 8 Euler steps; we exclude the final direct posterior sampling. We present the absolute log-ratio error ∣log⁡r^kd−log⁡rkd∣|\log\hat r_k^d-\log r_k^d| (×10−2\times10^{-2}), where we take the mean over kk and dd (see Table 11). Here, rkdr_k^d and r^kd\hat r_k^d are the token-level ratios. For one-sample transition rate estimation, we uniformly choose one of the nn posterior samples for estimating the transition rate and use all nn samples to estimate the ratio. We can see that with n=24n=24 and weighting, the log-ratio errors are similar for the two ways of estimating the rate. Additionally, the error decreases as we use more MC samples.

Rate estimationRatio samples nnWeight(0,48)(0,48)(48,96)(48,96)
One MC sample1w^\hat w5.065.10
One MC sample4w^\hat w2.632.70
One MC sample8w^\hat w1.851.89
One MC sample24w^\hat w1.141.10
Mean of nn MC samples24w^\hat w1.141.12
Mean of nn MC samples24112.072.03

We compare training using the exact ratio formula in Appendix B with training with Algorithm 2, where the rate is estimated using either one MC sample or the mean of 24 MC samples. We present the evaluation results on the IMDB test set after 240 training updates. We can see that the three training settings achieve similar performance, which shows that our method works similarly with both ways of estimating the rate.

Rate estimationRatio computationSuccess rateSuccess (pos. label)Success (neg. label)distinct⁡1\operatorname{distinct}_1Reward
Mean of 24 MC samplesAlgorithm 2 (n=24n=24)0.67380.56210.78560.92131.5931
One MC sampleAlgorithm 2 (n=24n=24)0.68840.57920.79760.91991.6072
One MC sampleExact (Appendix B)0.67030.54310.79760.92581.5923

Q4 & W2 ablation: If computationally feasible, a comparison of the full method, n=1n=1 and w=1w=1 for the same number of updates on PickScore, ideally with at least three training seeds, would help assess the reliability of the rate-dependent weight's additional benefit. A more focused estimate of training variability would also be useful if full repeated runs are too costly.

A: Thanks for your suggestion. First, we cannot afford the computational resources required to repeat all three settings with at least three training seeds as we mentioned in Appendix D.7; Flow-GRPO also reports that repeated experiments are too costly. Second, we evaluate the base model and our trained models with three evaluation seeds on the full test sets for PickScore, GenEval and ScienceQA (Table 6). The improvements are consistent across all three seeds and much larger than the standard deviations across evaluation seeds (e.g., PickScore increases from 20.84–20.87 to 22.98–23.00). Third, the reward continues to improve over 6300 training steps for PickScore and 3300 training steps for GenEval (see Figure 1 and Figure 3). Finally, the full method achieves higher rewards than both n=1n=1 and w=1w=1 under the same 13-hour budget (Table 2), which provides evidence for the benefit of our method.

AuthorsOpenReview ↗

Rebuttal 3

Q5 & W2 baselines: Could you clarify whether the clipping range in Fig. 5 was tuned for the mean-field baselines, in particular for diffu-GRPO's token-level ratios? An ELBO-based baseline on FUDOKI would be helpful if feasible. For Table 1, it would also be useful to identify which numbers are taken from the original papers and the corresponding NFE and CFG settings.

A: Thanks for your question. First, we use the same learning rate and clipping parameters as in our main experiments and do not tune the mean-field baselines separately. Both baselines improve overall during the first 2000 steps, and dFlowGRPO exceeds their best observed rewards with less estimated training time (see our response to Reviewer 2Eup Q1). Second, ELBO-based methods for masked dLLMs [2] are not directly applicable to general conditional probability paths and source distributions. Therefore, we develop an ELBO-based policy optimization framework for general probability paths (see Appendix D.3 in our uploaded revised version). We evaluate the ELBO-based method on FUDOKI with the PickScore reward. The results show that dFlowGRPO reaches the peak reward of the ELBO-based method in comparable estimated training time and continues to improve afterwards. Third, the results of other methods in Table 1 are taken from existing works. We use the reported FUDOKI's score to show the improvement over the base model (see our response to Reviewer 2Eup W1). Our model uses 20 sampling steps without CFG (20 NFEs). The FUDOKI paper uses 32 steps by default, corresponding to 64 NFEs with CFG.

[2] Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective. ICLR 2026.

W2 claims: The abstract says that dFlowGRPO "outperforms existing GRPO-type methods for dLLMs" and is "competitive with continuous flow-based models trained using Flow-GRPO". Both statements appear to rest largely on Table 1, where each method is applied to a different base model. Some qualification would help distinguish the reported model-level performance from conclusions about the training algorithms themselves.

A: Thanks for your comment. First, in Table 1, we compare our model's score with the scores reported in existing papers, including those for continuous flow-based models. We want to show that our trained model achieves competitive performance. Since these results use different base models, we do not claim that Table 1 directly compares the training methods. Second, to compare the training methods, we conduct experiments with dFlowGRPO and mean-field methods on the same base model (Figure 5).

We thank the reviewer for the comments and suggestions, which have helped us improve the quality of our paper. We would be glad to clarify anything that remains unclear.

AuthorsOpenReview ↗

Follow-up on the rollout sampler

Dear Reviewer PVvk,

Thank you again for raising the question about the rollout sampler. Appendix D.9 now reports the experiment you suggested. Using one randomly selected posterior sample to set the rollout rate and all 24 samples to estimate the transition ratio yielded a PickScore of 23.69 at 6,300 steps, compared with 23.00 for the rate-averaged rollout under the same evaluation protocol.

As the discussion period draws to a close, we would appreciate your view on whether this experiment and our clarification of Theorem 1 address your concern. We would be happy to clarify any remaining points.

Best, Authors

Reviewer_UxtyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes dFlowGRPO, a reinforcement learning framework for general discrete flow models that supports different probability paths and non-masked source distributions. The authors formulate denoising as a decision process and construct GRPO policy updates using posterior probability ratios weighted by conditional transition rates, estimated through Monte Carlo sampling. Experiments demonstrate the framework's effectiveness on image generation and multimodal understanding with FUDOKI, as well as text completion with FS-DFM.

Critical strengths
  1. A reinforcement learning framework with broader applicability. The method incorporates both conditional transition rates and the posterior model into policy updates, extending the scope from specific masked settings to more general discrete flow paths. This offers a meaningful approach to unified reward-based post-training across different discrete generative processes.
  2. Empirical support for practical effectiveness. Training with different rewards improves image generation, question answering, and text completion, while fixed-budget ablations support the value of explicit rate information. These results provide evidence of practical optimization benefits beyond a purely formal generalization.
Critical weaknesses
  1. Empirical support for the generality claim remains limited. Although the paper evaluates both FUDOKI and FS-DFM, the latter is tested mainly on sentiment-controlled text completion, and performance across different source distributions and conditional rate settings is not systematically established. This limits my confidence in the method's reliable applicability across discrete generation paradigms.
Questions for authors
  1. Are there experiments with masked source distributions or alternative conditional rate settings beyond FUDOKI's metric-induced path and FS-DFM's mixture path with a uniform source? Although LLaDA and Dream are cited as representative masked diffusion models, they are not directly evaluated, leaving the conclusions potentially dependent on the selected backbones. Are comparisons with existing RL methods on these models available to substantiate the method's effectiveness?
  2. The paper includes fixed-time ablations of the weighting scheme, but multi-sample estimation still increases the cost per training step. Compared with other trajectory-based or likelihood-approximation methods, can the authors demonstrate advantages in final performance and convergence efficiency under the same computational budget?
2 comments

We thank the reviewer for the valuable comments and suggestions. We appreciate the time you spent on the paper. We address your concerns below.

W1: Empirical support for the generality claim remains limited. Although the paper evaluates both FUDOKI and FS-DFM, the latter is tested mainly on sentiment-controlled text completion, and performance across different source distributions and conditional rate settings is not systematically established. This limits my confidence in the method's reliable applicability across discrete generation paradigms.

A: Thanks for your comment. Our formulation already covers the standard masked setting (see the discussion under Theorem 1); that is, it recovers the transition probability ratio derived in dTRPO [1], whose experiments provide evidence for using this ratio in the models with masked source and mixture path. Our experiments focus on the extension beyond this setting, using models with a metric-induced path (FUDOKI) or a mixture path (FS-DFM) with a uniform source. We will make the connection to the masked setting clearer (see the discussion under Theorem 1 in our uploaded revised version). Also see our response to Q1 for details.

[1] dTRPO : Trajectory Reduction in Policy Optimization of Diffusion Large Language Models. ICML 2026.

Q1: Are there experiments with masked source distributions or alternative conditional rate settings beyond FUDOKI's metric-induced path and FS-DFM's mixture path with a uniform source? Although LLaDA and Dream are cited as representative masked diffusion models, they are not directly evaluated, leaving the conclusions potentially dependent on the selected backbones. Are comparisons with existing RL methods on these models available to substantiate the method's effectiveness?

A: Thanks for your question. First, note that for the mixture path with masked source distribution, the transition probability ratio reduces to

\frac{p^{\theta,d}_{1\mid t_k}(\mathbf{x}_{t_{k+1}}^d\mid\mathbf{x}_{t_k})} {p^{\theta_{old},d}_{1\mid t_k}(\mathbf{x}_{t_{k+1}}^d\mid\mathbf{x}_{t_k})}.$$ which recovers the ratio in Theorem 3.2 of dTRPO (equation 10 in their ICML paper), whose experiments already study policy optimization in models with masked source distributions (LLaDA and Fast-dLLM-v2). This establishes the masked setting as a theoretical special case of our framework. Our experiments focus on the extension beyond this setting, using FUDOKI's metric-induced path and FS-DFM's mixture path with a uniform source. We have clarified this connection to prior work in the discussion following our Theorem 1 in the main paper. **Q2:** The paper includes fixed-time ablations of the weighting scheme, but multi-sample estimation still increases the cost per training step. Compared with other trajectory-based or likelihood-approximation methods, can the authors demonstrate advantages in final performance and convergence efficiency under the same computational budget? **A:** Thanks for your question. First, multi-sample estimation indeed increases the computational cost per training step, so we conduct the MC ablation under the same time budget (see Table 2 in Appendix). Due to limited computational resources, we set the budget to 13 hours on 8 H100s, since we need to test different values of $n$. At this budget, $n=8$ achieves a PickScore of 21.86 compared with 21.66 for $n=1$, which shows that multiple samples can improve performance under the same time budget. Second, our policy optimization framework reaches the best rewards of the mean-field methods in less estimated training time and continues to improve after achieving them (see Figure 5 and Table 8, and the Table in the response to reviewer 2Eup). Third, for comparison with likelihood-approximation methods, we have added an ELBO-based policy optimization method on FUDOKI. The results show that dFlowGRPO reaches the peak reward of the ELBO-based method in comparable estimated training time and continues to improve afterwards (see Appendix D.3). We thank the reviewer for the comments and suggestions, which have helped us improve the quality of our paper. We would be glad to clarify anything that remains unclear.
AuthorsOpenReview ↗

Follow-up on scope and training cost

Thank you for raising the questions about generality and training cost. The revised discussion after Theorem 1 clarifies how the ratio reduces to the known masked-source case. Our own experiments cover FUDOKI’s metric-induced path and FS-DFM’s uniform-source mixture path. At the same training-time budget, the eight-sample estimator reached a PickScore of 21.86 versus 21.66 for one sample. Appendix D.3 also reports the FUDOKI comparison with an ELBO-based method. We would appreciate your assessment of whether these results address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewers,

Since the discussion period will end soon, we wanted to follow up on our rebuttal. We would appreciate it if you could take a look and let us know whether it addresses your concerns. If you have any remaining questions, we’d be happy to answer them.

Thank you very much for your time.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.