acceptodds
Under review as a conference paper at ICLR 2027

DFS-GRPO: Reward Guided Tree Search Leads to Provable Improvement in Diffusion Models

Abstract

We propose DFS-GRPO, a reward-guided tree search method that introduces reward-guided multi-tree depth-first search (DFS) sampling to improve diffusion post-training by directing a fixed rollout budget toward high-reward regions while preserving exploration. DFS sampling uses rewards from completed rollouts to select and reuse promising denoising prefixes, progressively directing the sampling groups toward high-reward regions. We combine this sampler with initial noise truncation to strengthen exploitation, while subtree budget allocation and retention of multiple promising branches preserve exploration. An idealized local analysis shows that reward-based selection biases the expected state update toward the gradient of expected reward. Experiments on SD3-Medium and FLUX.1-dev demonstrate that DFS-GRPO improves reward optimization and maintains exploration to guarantee training stability. With matched training rewards, DFS-GRPO outperforms the strongest Flow-GRPO-S3 baseline (ODE–SDE sampling, tree sampling, and noise truncation) across three benchmarks on FLUX.1-dev, achieving relative improvements of on UniGenBench, on GenEval, and on LongText-Bench (English) over this baseline. Under standard inference, the resulting model reaches on OneIG-Bench Alignment, comparable to the newer-generation Z-Image-Turbo and to Imagen-3.

Primary area:
generative models
Keywords:
GRPO algorithm in Diffusion Model, Tree Search, Guidance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1kkTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DFS-GRPO, a reward-guided rollout sampler for reinforcement-learning post-training of diffusion and rectified-flow models. Under a fixed rollout budget, rewards from completed trajectories guide the selection, reuse and expansion of promising denoising prefixes. Initial-noise truncation strengthens exploitation during training, while subtree-budget allocation and Top-2 prefix retention preserve alternative continuations. The method retains the Flow-GRPO loss formulation, and the trained models use standard inference without tree search.

The paper provides an idealized local directional-alignment result under exact continuation-value ranking, a Gaussian-mixture interpretation of reward confidence, and a standard stationarity bound for exact gradient ascent. Experiments on SD3-Medium, FLUX.1-dev, several image-generation benchmarks and a video-generation extension report improvements in reward optimization, downstream performance and empirical training stability.

Critical strengths
  1. A clear and practically useful sampling contribution. The method uses completed-rollout rewards to determine which denoising prefixes receive further sampling, allowing rewards to influence the construction of training groups directly. This differs from sharing prefixes under a predefined tree and from selecting noise using one-step previews. The trained model uses standard Gaussian initialization and ordinary inference with a single LoRA adapter, so the reported gains do not require search at deployment time.

  2. Positive controlled results and informative design ablations. Table 1 reports improvements across two image-generation backbones and four individually optimized rewards. Under the two-reward recipe, Tables 2-3 also show improvements over the strong Flow-GRPO-S3 baseline with tree sampling and truncation. The Top-1/Top-2, subtree-allocation and BoN ablations examine relevant design choices, while repeated training curves and GPU-hour comparisons provide useful evidence about stability and optimization efficiency.

Critical weaknesses

W1. The headline gains are not supported by a clearly documented matched comparison. The abstract reports relative improvements of 2.4%, 5.1% and 5.6% under matched training rewards. These figures come from DFS with the extended recipe, which adds OCR and GroundingDINO spatial rewards and structured prompts. However, Tables 2-3 and the experimental description do not clearly establish that the strongest Flow baseline uses this same recipe. Under the stated two-reward setting, the corresponding improvements are approximately 0.91%, 2.33% and 2.35%. These positive results support the sampler, but do not isolate its contribution to the larger headline gains. The authors should clarify the baseline recipe and, if it differs, provide the matched control or restrict the attribution accordingly. Uncertainty estimates for the main benchmark comparisons would also help assess robustness beyond the repeated training curves.

W2. The theoretical scope is narrower than the strongest improvement and stability claims. The paper explicitly presents its local analysis as idealized, and Theorem 4.1 provides a useful mechanism-level explanation. However, it ranks candidates by exact continuation values, whereas the implemented prefix selection uses a single stochastic terminal reward. The effect of ranking noise is not covered. Equation (6) is also a bound for exact gradient ascent, rather than for the group-normalized, clipped and KL-regularized GRPO update under adaptive prefix reuse. The connection between that update and the gradient of the stated policy objective remains unestablished. These results therefore do not provide an algorithm-level improvement or training-stability guarantee. The framing in the title and abstract, particularly "guarantee training stability," should distinguish the proved local properties from the empirical training results, unless a stronger connection is established.

Questions for authors

Q1. Matched experimental recipe and headline attribution. Does the strongest Flow baseline in Tables 2-3 use the same four rewards, reward weights, structured prompts, training budget and result-selection rule as the extended DFS variant? If so, please make this explicit. If not, could the authors provide this control or restrict the abstract's improvement figures to the existing comparable settings?

Q2. Theory, sampling distribution and the implemented update. What can be established about prefix ranking from a single sampled terminal reward, beyond the exact-value setting of Theorem 4.1? Please clarify the sampling distribution and objective associated with the actual GRPO update, including how shared prefixes are counted or weighted and how BoN selection is applied. Are any algorithm-level improvement or stability guarantees intended, or should the claims be limited to the local mechanism and empirical observations?

Q3. Main-benchmark uncertainty and result selection. The repeated training curves provide useful stability evidence. Could the authors additionally report repeated-run results or appropriate uncertainty estimates for the two-reward DFS variant and the strongest matched Flow baseline on the main benchmarks? Please clarify the common rule for selecting runs and checkpoints, including how the reported best Flow run was selected and how DFS results were selected.

Reviewer_hbQnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DFS-GRPO, a reward-guided tree search method for reinforcement learning post-training of diffusion models. The method reuses promising denoising prefixes and combines initial noise truncation, sub-tree budget allocation, and Top-2 branch retention to balance exploration and exploitation under a fixed rollout budget. The authors provide a local theoretical analysis and demonstrate improvements in reward optimization and training stability on SD3-Medium and FLUX.1-dev.

Critical strengths
  1. The paper reuses high-reward denoising prefixes to improve sampling efficiency while maintaining sufficient exploration.
  2. The experiments cover multiple generative models, reward functions, and image generation benchmarks. The authors also provide ablations on training stability, rollout allocation, and branch retention, showing promising improvements across several settings.
Critical weaknesses
  1. The core idea of the proposed method is closely related to prior work on reward-guided trajectory reuse and tree-based RL, but these connections are not sufficiently discussed. For example, PrefixRL, PATR, and Tree-GRPO have explored reusing promising trajectory prefixes or tree-structured rollouts to improve exploration efficiency in LLM RL. The authors should better clarify the methodological novelty beyond these approaches and existing tree-based diffusion RL methods.
  2. The choice of retaining the Top-2 branches appears heuristic and lacks sufficient experimental justification. Figure 5(b) only compares Top-1 and Top-2, without exploring Top-3, Top-4, or adaptive retention strategies. Moreover, retaining two high-reward branches does not necessarily guarantee meaningful exploration diversity.
  3. The rollout distribution induced by reward-guided DFS may be inconsistent with the old-policy sampling assumption in standard GRPO. In Section 3, the GRPO objective assumes rollouts sampled from the old policy. However, Section 4 reuses reward-selected prefixes, and Eq. (4) shows that reward-based selection changes the expected state update under idealized conditions. It remains unclear whether directly applying the unchanged Flow-GRPO loss introduces bias into gradient estimation.
Questions for authors

See weaknesses.

Reviewer_8KhfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DFS-GRPO, a reward-guided sampling strategy designed to enhance the exploration-exploitation balance in Flow-GRPO. The method integrates reward-guided tree search, initial noise truncation, subtree budget allocation, and Top-2 branch retention to improve reward optimization and training stability.

Critical strengths
  1. Practical sampling strategy. The proposed method introduces a straightforward, reward-guided sampling strategy to enhance the balance between exploration and exploitation without modifying the original Flow-GRPO objective.

  2. Empirical improvements. The experiments demonstrate consistent reward improvements across Flow-GRPO variants for multiple reward functions and generative models.

Critical weaknesses
  1. Insufficient comparisons with existing methods. The experiments mainly compare with variants of Flow-GRPO, while comparisons with closely related tree-based sampling methods, such as [1, 2], are absent. Comparisons with other reward-alignment methods, such as DiffusionNFT [3], are limited to a single OCR experiment. More comprehensive comparisons are necessary to establish its empirical advantages over existing approaches.

  2. Limited analysis of diversity and reward overoptimization. Another important consideration in reward alignment is maintaining sample diversity and generation quality while improving target rewards. Although the paper reports benchmark results, further evaluations of diversity and held-out generation quality under individual reward optimization would help assess the potential for reward overoptimization or mode collapse.

[1] Li et al. "BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models." ICLR (2026).

[2] Fu et al. "Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling." arXiv:2509.23352 (2025).

[3] Zheng et al. "DiffusionNFT: Online Diffusion Reinforcement with Forward Process." ICLR (2026).

Questions for authors
  1. Could the authors provide comparisons with existing tree-based sampling methods, such as [1, 2], and representative reward-alignment methods, such as DiffusionNFT [3]?

  2. Could the authors assess sample diversity and the quality of held-out generations during reward optimization to investigate potential reward overoptimization and mode collapse?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.