acceptodds
Withdrawn from ICLR 2027

Transition-GRPO: Improving Flow Model Alignment with Direct Transition Targets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning has emerged as an effective approach for aligning diffusion and flow-based generative models with downstream preferences. Backward-process methods such as Flow-GRPO optimize stochastic reverse-SDE transitions, but often exhibit slower reward improvement than forward-process alternatives, raising the question of how these transitions can be optimized more effectively. We revisit the Flow-GRPO objective and show that its local update reveals a simple policy-improvement direction in transition-mean space. Building on this observation, we propose Transition-GRPO (T-GRPO), which uses a KL-proximal formulation to derive explicit, advantage-adjusted transition-mean targets and replaces likelihood-ratio optimization with direct regression toward these targets. We further extend this formulation to jointly perform CFG distillation and reward optimization during a short initialization stage, using CFG-generated trajectories to construct advantage-adjusted transition targets for a non-CFG model. Experiments demonstrate that T-GRPO consistently accelerates convergence and achieves higher rewards under a fixed training budget than Flow-GRPO. It also surpasses the forward-process DiffusionNFT baseline on preference-based rewards, while DiffusionNFT remains stronger on structured tasks. Moreover, T-GRPO yields additional gains when integrated with existing acceleration methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RL for diffusion, flow model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BqxwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Transition-GRPO (T-GRPO), a reinforcement-learning fine-tuning method for flow-matching models. Instead of optimizing the usual clipped likelihood-ratio objective, T-GRPO constructs a target for each reverse transition mean by shifting the rollout mean in the direction of the sampled transition, with the shift scaled by the group advantage. It then trains the model to regress toward this detached target. The paper evaluates the method primarily on SD3.5 Medium. According to the reported score, T-GRPO improves over Flow-GRPO on PickScore, HPSv2.1, GenEval, and OCR.

Critical strengths

1、 T-GRPO turns advantage-weighted transition updates into regression toward a constructed target. This is a straightforward formulation that could be useful for practitioners and may be easier to integrate with some flow-model training setups.

2、Positive results on several metrics, including a substantial OCR improvement. On SD3.5 Medium, T-GRPO improves over Flow-GRPO across different metrics.

Critical weaknesses

1、The theoretical justification is limited. The analysis shows that T-Loss matches the local direction of an unclipped Flow-GRPO update, up to a scale factor, but does not explain why fixed-target regression improves finite-step optimization or training stability. The role and sensitivity of the target scale (set as 10) are also unclear.

2、The gains are uneven and do not establish broad superiority. T-GRPO improves over Flow-GRPO on the reported metrics, but DiffusionNFT remains stronger on GenEval and OCR.

3、The qualitative evidence for improved visual quality is weak. Figure 3 provides only a small number of examples, and the visual differences do not clearly substantiate a substantial quality improvement.

4、The comparisons with stronger GRPO variants are narrow. The paper does include compatibility experiments with methods such as MixGRPO and TempFlow-GRPO, but these results are reported only for PickScore. It is therefore unclear whether the gains extend to structured metrics such as GenEval and OCR, or how much comes specifically from T-Loss rather than the full T-GRPO recipe.

Questions for authors

How sensitive are the results to the parameter (target scale)? Could the authors report a sweep or explain how to choose this value?

Could the authors evaluate the MixGRPO and TempFlow-GRPO comparisons on additional core metrics, and isolate the effect of T-Loss from T-Init?

Reviewer_36yJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Transition-GRPO (T-GRPO), a reinforcement learning method for aligning flow-based generative models with downstream rewards. Unlike Flow-GRPO, which relies on likelihood-ratio-based optimization, T-GRPO directly regresses toward advantage-adjusted transition-mean targets derived from a KL-proximal formulation using sampled reverse-SDE residuals. The authors also introduce a CFG-guided initialization stage to provide a stronger starting policy for subsequent non-CFG training. Experiments across four image-generation reward tasks show that T-GRPO consistently outperforms Flow-GRPO. Compared with DiffusionNFT, T-GRPO achieves better results on two preference-based metrics, although DiffusionNFT maintains an advantage on structured tasks such as GenEval and OCR.

Critical strengths
  1. The proposed objective provides a clear connection between policy improvement and transition-level regression, with a reasonable mathematical motivation.

  2. There exist consistent improvements over Flow-GRPO. The method achieves better results across four reward tasks, and the ablation studies indicate that transition regression contributes most of the improvements.

Critical weaknesses
  1. One main concern unclear practical advantages over DiffusionNFT. DiffusionNFT can efficiently reuse reward-evaluated samples by repeatedly applying forward noising, whereas T-GRPO relies on collected reverse-SDE transitions. It is unclear whether the additional trajectory information compensates for this limitation in data reuse. Moreover, DiffusionNFT remains stronger on GenEval and OCR. Without comparisons under matched sample, reward-query, and computational budgets, T-GRPO's practical advantages remain insufficiently justified.

  2. There lacks sufficient analysis of the optimization gains. Reward-guided regression is closely related to existing approaches such as AWR. Furthermore, T-GRPO shares the same local improvement direction as unclipped Flow-GRPO, up to scaling. Comparisons with unclipped Flow-GRPO and simple advantage-weighted regression baselines are needed to establish whether the proposed target construction provides benefits beyond existing optimization techniques.

Questions for authors
  1. Can the authors compare T-GRPO and DiffusionNFT under matched reward-evaluation, generation, and computational budgets? How does their relative performance change when increasing the number of training updates per reward-evaluated sample?

  2. How does T-GRPO compare against unclipped Flow-GRPO and standard advantage-weighted regression on the same reverse-SDE trajectories, with appropriately tuned optimization scales?

Reviewer_snrPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes T-GRPO, a modified version of the Flow-GRPO objective for RL fine-tuning of diffusion/flow models. The authors show that the Flow-GRPO update moves each transition mean along the advantage-weighted sampled noise. Based on this, they derive a closed-form target for each transition mean with a KL-proximal formulation, and replace the likelihood-ratio objective with a regression of the predicted transition mean toward this target. Additionally, they add T-Init, a 30-epoch warm start in which a CFG teacher generates trajectories whose advantage-weighted transitions serve as regression targets for the non-CFG student. This is to improve the non-CFG version with a "CFG warm start". T-GRPO is evaluated with SD3.5-M against Flow-GRPO and DiffusionNFT. The experiments include four different rewards, where the evaluation is mostly done on the training reward. T-GRPO outperforms Flow-GRPO on four rewards, and DiffusionNFT on 2/4.

Critical strengths
  • Proposes a simple change to the Flow-GRPO objective that’s easy to plug into existing variants.
  • Transition-GRPO achieves gains over Flow-GRPO on all four tasks. Additionally, the components (loss vs. init) are ablated.
  • The work is honest about DiffusionNFT still being better on GenEval and OCR. There’s also a cross-reward check at matched target reward (B.5).
Critical weaknesses

My main concerns are evaluation focused on the train reward and whether the gains come from the objective itself or from a larger effective update (and the corresponding missing hyperparameters). I would consider raising my score once these have been clarified.

  • Reward hacking not really considered. This is in general, a problem with RL Diffusion papers recently. The methods are only evaluated on the training reward, which means whichever method reward hacks the fastest is "best". While the paper does have a table in the Appendix that evaluates 3 ckpts for different rewards, it is insufficient to draw any conclusions apart from the training reward. E.g. the base model is not evaluated. The authors say: "Mitigating such reward trade-offs is outside the scope of this work, which focuses on improving the efficiency of reward optimization". I disagree with this, it is in my view an integral part of diffusion fine-tuning, and one of the biggest considerations is how to prevent reward hacking. Without proper disentangled evaluation from the training reward, it is not possible to evaluate the usefulness of a new method. One example of this is that the final Geneval-trained T-GRPO ckpt performs worse one PickScore, ImageReward and HPSv2 compared to the Flow-GRPO one.
  • Appendix A.4 shows that T-GRPO and Flow-GRPO follow the same local update direction. The difference is how far the policy moves per batch, set by η in T-GRPO, where the ablation in Figure 4 shows η is the key factor with larger η resulting in better results. This suggests that larger magnitude updates could lead to better performance. For Flow-GRPO the clip range and learning rate determine the magnitude of the update. To me it remains unclear whether the Flow-GRPO baseline was tuned for the clip-range/lr. This makes it hard to conclude why T-GRPO outperforms here and whether it might mainly be the magnitude of the update. The hyperparameter details (optimizer, learning rate, clip range, number of updates per batch) are also missing (Appendix B.1 refers to itself).
  • (minor) T-Init: The benefit of T-Init seems to be rather small. If added together with the T-Loss it adds ~0.01 at the endpoint of the runs. Additionally, it’s described as CFG distillation, which could be a bit confusing. The below-average teacher samples (due to advantage normalization) push the student away from the CFG direction, so it’s unclear how much CFG is actually distilled. A comparison to a plain CFG-distillation warm start might be interesting to better understand what exactly is happening here.
Questions for authors
  • Could you report the base model scores for Table 3? Additionally, eval curves over training could help understand the reward hacking of different methods better.
  • What are the clipping ratio and learning rate for the Flow-GRPO runs and were they ablated? (e.g. higher lr/clipping ratio)
  • How does T-init compare to plain CFG-distillation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.