- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DFS-GRPO, a reward-guided rollout sampler for reinforcement-learning post-training of diffusion and rectified-flow models. Under a fixed rollout budget, rewards from completed trajectories guide the selection, reuse and expansion of promising denoising prefixes. Initial-noise truncation strengthens exploitation during training, while subtree-budget allocation and Top-2 prefix retention preserve alternative continuations. The method retains the Flow-GRPO loss formulation, and the trained models use standard inference without tree search.
The paper provides an idealized local directional-alignment result under exact continuation-value ranking, a Gaussian-mixture interpretation of reward confidence, and a standard stationarity bound for exact gradient ascent. Experiments on SD3-Medium, FLUX.1-dev, several image-generation benchmarks and a video-generation extension report improvements in reward optimization, downstream performance and empirical training stability.
Critical strengths
-
A clear and practically useful sampling contribution. The method uses completed-rollout rewards to determine which denoising prefixes receive further sampling, allowing rewards to influence the construction of training groups directly. This differs from sharing prefixes under a predefined tree and from selecting noise using one-step previews. The trained model uses standard Gaussian initialization and ordinary inference with a single LoRA adapter, so the reported gains do not require search at deployment time.
-
Positive controlled results and informative design ablations. Table 1 reports improvements across two image-generation backbones and four individually optimized rewards. Under the two-reward recipe, Tables 2-3 also show improvements over the strong Flow-GRPO-S3 baseline with tree sampling and truncation. The Top-1/Top-2, subtree-allocation and BoN ablations examine relevant design choices, while repeated training curves and GPU-hour comparisons provide useful evidence about stability and optimization efficiency.
Critical weaknesses
W1. The headline gains are not supported by a clearly documented matched comparison. The abstract reports relative improvements of 2.4%, 5.1% and 5.6% under matched training rewards. These figures come from DFS with the extended recipe, which adds OCR and GroundingDINO spatial rewards and structured prompts. However, Tables 2-3 and the experimental description do not clearly establish that the strongest Flow baseline uses this same recipe. Under the stated two-reward setting, the corresponding improvements are approximately 0.91%, 2.33% and 2.35%. These positive results support the sampler, but do not isolate its contribution to the larger headline gains. The authors should clarify the baseline recipe and, if it differs, provide the matched control or restrict the attribution accordingly. Uncertainty estimates for the main benchmark comparisons would also help assess robustness beyond the repeated training curves.
W2. The theoretical scope is narrower than the strongest improvement and stability claims. The paper explicitly presents its local analysis as idealized, and Theorem 4.1 provides a useful mechanism-level explanation. However, it ranks candidates by exact continuation values, whereas the implemented prefix selection uses a single stochastic terminal reward. The effect of ranking noise is not covered. Equation (6) is also a bound for exact gradient ascent, rather than for the group-normalized, clipped and KL-regularized GRPO update under adaptive prefix reuse. The connection between that update and the gradient of the stated policy objective remains unestablished. These results therefore do not provide an algorithm-level improvement or training-stability guarantee. The framing in the title and abstract, particularly "guarantee training stability," should distinguish the proved local properties from the empirical training results, unless a stronger connection is established.
Questions for authors
Q1. Matched experimental recipe and headline attribution. Does the strongest Flow baseline in Tables 2-3 use the same four rewards, reward weights, structured prompts, training budget and result-selection rule as the extended DFS variant? If so, please make this explicit. If not, could the authors provide this control or restrict the abstract's improvement figures to the existing comparable settings?
Q2. Theory, sampling distribution and the implemented update. What can be established about prefix ranking from a single sampled terminal reward, beyond the exact-value setting of Theorem 4.1? Please clarify the sampling distribution and objective associated with the actual GRPO update, including how shared prefixes are counted or weighted and how BoN selection is applied. Are any algorithm-level improvement or stability guarantees intended, or should the claims be limited to the local mechanism and empirical observations?
Q3. Main-benchmark uncertainty and result selection. The repeated training curves provide useful stability evidence. Could the authors additionally report repeated-run results or appropriate uncertainty estimates for the two-reward DFS variant and the strongest matched Flow baseline on the main benchmarks? Please clarify the common rule for selecting runs and checkpoints, including how the reported best Flow run was selected and how DFS results were selected.