- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to fine-tune discrete flow models, which generate all tokens in parallel and refine them over several denoising steps, with GRPO-style reinforcement learning. Policy-gradient training needs the ratio between the new and old policy's probability of each denoising step, and existing RL methods for diffusion language models only approximate this ratio and mostly cover the masked special case. The authors derive the exact per-step ratio for general discrete flow models, it combines the model's prediction of the final token with a term that depends on the transition rate, and it can be estimated from samples already drawn during generation without extra forward passes. Applied to the multimodal model FUDOKI withoutCFG, dFlowGRPO raises PickScore from 20.87 to 23.00 and GenEval from 61% to 93%, and ScienceQA accuracy from 75.1% to 81.1%, while most auxiliary understanding benchmarks change little. On the same base model, the mean-field baselines diffu-GRPO and diffu-GSPO collapse during training, whereas dFlowGRPO keeps improving.
Critical strengths
The exact per-step ratio is the right object, and the empirical consequence is visible. Replacing mean-field likelihood surrogates with the token-wise expected posterior ratio under a general probability path is principled, reduces to the known masked-model ratio as a special case, and costs no extra forward passes. It also appears to buy training stability that the approximations do not: in Fig. 5 both mean-field baselines collapse during training while dFlowGRPO improves steadily . Non-collapse against a same-base approximate-ratio method is the strongest evidence in the paper that the exactness matters.
The gains are not confined to the training reward. The clearest test is the held-out object split: the model is trained on 60 GenEval object categories and evaluated on prompts built only from the other 20, and overall GenEval still rises from 0.5602 to 0.7720, with every category improving. The GenEval-trained model also improves all seven T2I-CompBench++ categories, the PickScore-trained model improves the other DrawBench preference metrics it was not trained on , and the ScienceQA-trained model slightly improves MMMU and MM-Vet. DPG-Bench stays roughly flat. The main risk in reward-driven fine-tuning of a generator is overfitting the reward model.
Critical weaknesses
The controlled comparison behind the main claim is narrow, and the baselines may be disadvantaged by the setup. The abstract claims that dFlowGRPO outperforms existing GRPO-type methods for dLLMs, and is competitive with continuous flow-based models trained using Flow-GRPO. The only experiment that compares RL methods on the same base model uses a single reward (PickScore), without KL regularization. The main GenEval comparison in Tab 1 lists results from different base models, training data and sampling settings, while its FUDOKI reference is evaluated with CFG while dFlowGRPO is not. It does not isolate the effect of the training objective.
Questions for authors
-
Were the clipping parameters and learning rate tuned separately for diffu-GRPO and diffu-GSPO in Fig. 5? If not, do the baselines still collapse with their own hyperparameter search, and does the gap persist when plotted against wall-clock time (Table 8)?
-
Which checkpoint do the no-LLM-extraction ScienceQA numbers "54.1% to 59.1%" (p8) refer to? In Fig. 2 (left), the dashed curve is at roughly 59% around step 900 and roughly 63% at step 1500, which is where the 81.1% with LLM extraction is marked. Please report both settings for the same checkpoint.
-
Please report POPE and MME-P numerically in the main text alongside the gains, so that the 86.1 to 84.4 and 1485.4 to 1479.8 changes (printed in the Fig. 2 radar chart) are visible outside the radar chart, and state explicitly which auxiliary metrics regress.
2 comments
Thank you for the clarifications. My questions are resolved, and I keep my score.
We thank the reviewer for the valuable comments and suggestions. We appreciate the time you spent on the paper. We address your concerns below.
W1: The controlled comparison behind the main claim is narrow, and the baselines may be disadvantaged by the setup. The abstract claims that dFlowGRPO outperforms existing GRPO-type methods for dLLMs, and is competitive with continuous flow-based models trained using Flow-GRPO. The only experiment that compares RL methods on the same base model uses a single reward (PickScore), without KL regularization. The main GenEval comparison in Tab 1 lists results from different base models, training data and sampling settings, while its FUDOKI reference is evaluated with CFG while dFlowGRPO is not. It does not isolate the effect of the training objective.
A: Thanks for your comment. First, Flow-GRPO uses a single reward in each setting, and we follow their setup and also train the model with one reward in each setting. For FUDOKI, we consider three settings; we use PickScore and GenEval for image generation, and ScienceQA for multimodal understanding. The experimental results show that our method improves the reward in all three settings. We also conduct experiments on FS-DFM for sentiment-controlled text completion, where the reward improves from 1.52 to 1.99. Therefore, these results show the effectiveness of our method across different tasks and base models. Second, for image generation, we only train the model with KL regularization in the GenEval setting to show that KL regularization can mitigate reward hacking, and keep the other experiments without KL regularization. Third, for Table 1, we just want to use the GenEval score reported in the FUDOKI paper directly to show that our method can improve the performance of the base model and achieve a higher score even when we start from a model without CFG. (Of course, from Figure 3, we can see that FUDOKI's score without CFG is 0.61, which is lower than the reported CFG version's score of 0.77) Finally, to post-train FUDOKI with CFG, we can just use CFG during both rollout and policy updates.
Q1: Were the clipping parameters and learning rate tuned separately for diffu-GRPO and diffu-GSPO in Fig. 5? If not, do the baselines still collapse with their own hyperparameter search, and does the gap persist when plotted against wall-clock time (Table 8)?
A: This is a good observation. We did not tune the baselines separately, and used the same learning rate and clipping parameters as in our main experiments to keep the settings consistent. We observed that both mean-field methods' rewards improved overall during the first 2000 steps, suggesting that these parameters were within a reasonable range. Additionally, dFlowGRPO exceeds the best observed rewards of both baselines with less or comparable estimated training time and continues to improve after achieving them (see both Figure 5 and Table 8, and see the following table). Smaller learning rates or tighter clipping parameters may slow reward improvement under the same time budget and may delay collapse.
Q2: Which checkpoint do the no-LLM-extraction ScienceQA numbers "54.1% to 59.1%" (p8) refer to? In Fig. 2 (left), the dashed curve is at roughly 59% around step 900 and roughly 63% at step 1500, which is where the 81.1% with LLM extraction is marked. Please report both settings for the same checkpoint.
A: Thanks for pointing this out. This is a reporting error, and we have corrected the paper to report both settings at step 1500 (from 75.1% to 81.1% with LLM extraction, and 54.1% to 62.8% without LLM extraction).
Q3: Please report POPE and MME-P numerically in the main text alongside the gains, so that the 86.1 to 84.4 and 1485.4 to 1479.8 changes (printed in the Fig. 2 radar chart) are visible outside the radar chart, and state explicitly which auxiliary metrics regress.
A: Thanks for your suggestion. We have added these numbers in Section 5.3 of the revised version. Please see our updated PDF.
We thank the reviewer for the comments and suggestions, which have helped us improve the quality of our paper. We would be glad to clarify anything that remains unclear.