acceptodds
Under review as a conference paper at ICLR 2027

RewardFlow: Learning to Align Flow Models with Reward Distributions

Abstract

Post-training alignment is important for flow-based text-to-image generation, but existing GRPO-style methods usually optimize scalar rewards and tend to concentrate the policy on a few reward-favored samples. This limits diversity, weakens generalization to unseen preference metrics, and becomes unstable under multi-reward optimization. In this paper, we propose RewardFlow, a reward distribution matching framework for post-training flow matching models. Instead of directly maximizing scalar rewards, RewardFlow constructs a reward-induced distribution over sampled denoising trajectories and aligns the flow policy with this distribution, allowing multiple high-quality generations to receive positive optimization signals. To stabilize image flow optimization, we use a warm-started learnable prompt-conditioned partition function for reward calibration and a step-wise KL constraint to preserve the pretrained flow prior. Experiments on SD3.5-M and FLUX.1-dev under both single-reward and multi-reward settings show that RewardFlow achieves stronger preference alignment, better generalization, and more diverse generations. Notably, under multi-reward training on SD3.5-M, RewardFlow improves HPS-v2.1 from 0.326 to 0.362 and the unseen Aesthetic Score from 6.226 to 6.470 over Flow-GRPO. Our code will be released on GitHub.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Flow Matching, GRPO, Post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VZLcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper defines distribution weights using the exponential of rewards and combines them with the reference model's distribution to construct a target distribution. The goal is to reduce the loss of diversity caused by scalar reward maximization and conflicts in multi-reward optimization. The paper uses KL divergence to this target distribution to motivate its training loss. Experiments on SD3.5-M and FLUX.1-dev show improvements in both single-reward and multi-reward settings, with larger gains in the multi-reward setting.

Critical strengths
  1. Experiments on SD3.5-M and FLUX.1-dev show improvements in both single-reward and multi-reward settings. The gains in the multi-reward setting are larger.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The target distribution, constructed by combining exponential reward weights with the reference model's distribution, can be derived as the optimal solution to expected reward maximization with KL regularization:

max⁡π[Eπ[R]−λDKL(π∥πref)].\max_{\pi}\left[\mathbb{E}_{\pi}[R]-\lambda D_{\mathrm{KL}}\left(\pi\Vert\pi_{\mathrm{ref}}\right)\right].

Its optimal distribution is:

π⋆(τ)∝πref(τ)exp⁡(R(τ)λ).\pi^\star(\tau)\propto\pi_{\mathrm{ref}}(\tau)\exp\left(\frac{R(\tau)}{\lambda}\right).

This has the same form as the target distribution defined in this paper, with β=1/λ\beta=1/\lambda. KL-regularized versions of PPO and GRPO use surrogate losses to optimize this type of regularized reward objective. Therefore, at the level of the idealized objective, there is no fundamental distinction between "Scalar Rewards" and "Reward-Induced Distributions" when KL regularization is included. I think this raises a serious concern about the paper's theoretical motivation and claimed contribution.

  1. The evaluated models are relatively old. The paper does not report results on newer open-source models such as Qwen-Image or Z-Image.
Questions for authors
  1. Can the authors explain more fully, from a theoretical perspective, how the proposed training loss differs from KL-regularized GRPO and PPO?
  2. Have the authors tried experiments on more recent open-source models?
Reviewer_BaLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that GRPO-style post-training of flow models maximizes a scalar reward. The authors say this collapses diversity, transfers poorly to held-out metrics, and becomes unstable when several rewards are combined. Then the authors replace it with reward-distribution matching. The target is the reference trajectory distribution tilted by exp⁡(βR)\exp(\beta R), and the policy is fit to it with a trajectory-balance (TB) squared residual, borrowed from GFlowNets and from FlowRL in LLMs.

Three practical changes make this work for images:

  1. The reference log-density is taken out of the TB residual and replaced by an external step-wise Gaussian KL.
  2. A prompt-conditioned MLP log-partition zϕ(c)z_\phi(c) is learned and warm-started.
  3. Group-normalized rewards are used together with a clipped importance weight.

Experiments on SD3.5-M and FLUX.1-dev cover single-reward training (HPS, GenEval, OCR) and two multi-reward mixtures. RewardFlow beats Flow-GRPO and MixGRPO on the trained rewards and on held-out rewards, and the paper adds diversity metrics, a user study, and ablations of β\beta, λKL\lambda_{\mathrm{KL}}, and z(c)z(c).

Critical strengths

The heterogeneous HPS+GenEval+OCR setting is the most convincing result. Flow-GRPO barely moves HPS there (0.293→0.294), while RewardFlow reaches 0.355. It also improves PickScore, ImageReward, and Aesthetic.

Critical weaknesses

1. Limited novelty, and the objective actually optimized departs substantially from the stated theory.

The core formulation (an exp⁡(βR)\exp(\beta R)-tilted reference, a trajectory-balance loss, a learned prompt-conditioned log⁡Z\log Z, and clipped importance weights) is essentially FlowRL / GFlowNet-TB carried over to Flow-GRPO's SDE MDP. The paper's distinct contribution therefore rests on the practical modifications, but these change the target distribution:

  • log⁡πref\log \pi_{\mathrm{ref}} is removed from the residual and replaced by an external step-wise KL.
  • sθs_\theta is the per-step average log-probability (divided by NN), not the trajectory log-probability.
  • Rewards are z-scored within each group.

App. E derives that the external KL corresponds to the target πrefαrefexp⁡(βλr^)\pi_{\mathrm{ref}}^{\alpha_{\mathrm{ref}}}\exp(\beta_\lambda \hat r) with αref=λKL/(1+λKL)\alpha_{\mathrm{ref}} = \lambda_{KL}/(1+\lambda_{KL}). With the default λKL=0.1\lambda_{KL}=0.1, the reference exponent is only about 0.090.09, so the target is nearly prior-free. This contradicts the motivation in §4.1 of reweighting the pretrained distribution. Moreover, the derivation holds only at the level of the KL expansion: because sθs_\theta is step-averaged, E[sθ−βr^]\mathbb{E}[s_\theta - \beta \hat r] is not a KL divergence; the loss actually minimized is a squared TB residual plus a KL penalty, whose stationary point need not coincide with that KL optimum; and the group-normalized r^\hat r makes the target depend on each sampled group. The "distribution matching" view therefore motivates the method but does not describe what is optimized, which may be closer to advantage-weighted regression with a weak anchor.

2. The diversity and generalization evidence is weakly specified, and there is no statistical reporting.

  • The diversity metrics in Table 4 are not defined. SSIM↑ and BFID↑ are treated as better, and FID↓ is grouped under "diversity," but the paper does not state whether these are computed pairwise among same-prompt samples or against base-model outputs or a reference set. If SSIM is computed pairwise within a prompt, higher SSIM means less diversity.
  • The single-reward gains over Flow-GRPO are small (e.g., HPS 0.360 vs. 0.368, GenEval 0.837 vs. 0.845), and no seeds, standard deviations or confidence intervals are reported for any table.
  • App. D states that both methods share the training configuration, but the KL coefficient used for Flow-GRPO is not reported, so it is unclear whether the baseline's regularization was tuned for each setting.
  • The user study (Table 5) gives no details on participants, number of prompts or statistical testing.
Questions for authors
  1. What is actually optimized. App. E derives that the external KL corresponds to the target. Three gaps remain: (i) sθs_ \theta is the per-step average log-probability, so E[sθ−βr^]\mathbb{E}[s _\theta - \beta \hat r] is not a KL divergence; (ii) the loss actually minimized (Eq. 13) is a squared TB residual plus a KL penalty, whose stationary point need not coincide with that KL optimum; (iii) r^\hat r is z-scored within each group, so the target depends on the sampled group. Can the authors characterize the stationary point of Eq. 13 itself, or show empirically that the trained policy approximates a reward-tilted distribution? Also, with the default λKL=0.1\lambda _{KL}=0.1, αref≈0.09\alpha _{\mathrm{ref}} \approx 0.09, i.e., the target is nearly prior-free. How is this reconciled with the motivation in §4.1 of reweighting the pretrained distribution?

  2. Relation to GRPO. How does the gradient of LTB\mathcal{L} _{\mathrm{TB}} with respect to θ\theta differ from a GRPO or advantage-weighted gradient once zϕ(c)z _\phi(c) absorbs the group mean and the rewards are z-scored? An analytical or empirical comparison (e.g., gradient cosine similarity, or an advantage-weighted baseline with the same KL anchor) would clarify where the gains come from.

  3. Diversity metrics. How exactly are DreamSim, FID, BFID and SSIM in Table 4 computed: pairwise among samples of the same prompt, against base-model outputs, or against a reference set? If SSIM is computed pairwise within a prompt, higher SSIM indicates less diversity; please clarify why SSIM↑ is treated as better.

  4. Baseline regularization and variance. App. D states that both methods share the training configuration, but the KL coefficient used for Flow-GRPO is not reported. What value was used, and was it tuned for each setting? Please also report mean ± standard deviation over at least three seeds for the main tables, including the multi-reward settings, since several single-reward gaps are small (e.g., HPS 0.360 vs. 0.368).

  5. User study. What was the protocol: number of participants, number and selection of prompts, comparison format, and statistical test used to assess significance?

Reviewer_BskyOpenReview ↗
Rating
4
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

RewardFlow introduces a trajectory-balance-inspired objective for aligning flow models with reward-induced distributions, using a warm-started prompt-conditioned calibration head and step-wise KL regularization. Experiments on SD3.5-M and FLUX.1-dev report improved alignment across single- and multi-reward settings, alongside positive diversity and human-preference results.

Critical strengths
  1. Strong empirical results across different alignment targets. RewardFlow improves multiple metrics across two backbones, with particularly substantial gains in multi-reward training. Improvements on held-out evaluation metrics further support its practical value beyond optimizing a single reward.
  2. A concrete optimization design supported by component ablations. The trajectory-level residual and prompt-conditioned calibration provide a practical alternative to advantage-based updates. Ablations support the usefulness of the calibration head, warm initialization, and reference constraint, making the contribution more convincing than an objective change alone.
  3. A well-developed mathematical formulation. The paper explicitly formulates the reward-induced trajectory distribution, derives the optimal prompt-only calibration term for a fixed policy, and analyzes the reference constraint. These derivations provide useful motivation for the method.
Critical weaknesses
  1. Diversity evaluation can be extended to more scenarios. The paper provides encouraging results, but the metric definitions and human study protocol require more detailed specification. It also does not directly establish whether the benefits extend to within-image diversity, such as generating distinct facial identities in a group photograph (which is a challenging problem in the field at present).
  2. The theoretical interpretation requires clarification. The reward-induced KL objective is equivalent to KL-regularized reward maximization, while the practical squared-residual objective introduces further modifications. A clearer distinction between the target distribution and the implemented optimization would strengthen the explanation of why the method works. This does not negate the reported empirical gains.
Questions for authors
  1. Do the diversity gains extend to within-image variation? Group photos with distinct facial identities would be a useful test.
  2. How well does the calibration head transfer to new prompt domains? Could integrating it into the backbone improve scalability and generalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.