acceptodds
Withdrawn from ICLR 2027

SGA-Flow-GRPO: Spatial Gradient-Guided Credit Assignment for Flow-GRPO

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement Learning (RL) has proven effective in aligning flow-based generative models with human preferences. Recently, Flow-GRPO has emerged as an efficient critic-free paradigm by calculating advantages over sampled candidate trajectories. However, standard Flow-GRPO applies a uniform scalar advantage across both temporal denoising steps and spatial latent dimensions, without explicitly accounting for the spatial structure of generated images, which may lead to sub-optimal policy updates. To address this, we propose a novel gradient-guided spatial credit assignment framework tailored for Diffusion Transformers (DiTs). We first reformulate the transition-level log-likelihood in Flow-GRPO into a token-wise representation natively aligned with DiT patch architectures, constructing spatially fine-grained importance sampling ratios. To allocate localized credit without rigid, boundary-sensitive segmentation heuristics, we introduce a continuous spatial credit map derived from reward gradients. Crucially, we employ an outlier-robust normalization scheme based on Median Absolute Deviation (MAD) coupled with temperature scaling, effectively eliminating gradient noise while highlighting functional prompt-aligned regions. Extensive evaluations on GenEval show that our approach delivers SOTA alignment quality, achieving a convergence rate comparable to top-tier methods like DiffusionNFT while substantially improving upon Flow-GRPO-based methods in alignment performance.

Primary area:
generative models
Keywords:
Generative Models, Reinforcement Learning, Post-training, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MnB7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper decomposes the transition log-likelihood in Flow-GRPO into token-level terms and uses a reward-gradient-based rule to weight the token-level advantages. On SD3.5-Medium and GenEval, the method reports faster RL training convergence than Flow-GRPO.

Critical strengths
  1. The reported experiments on SD3.5-Medium and GenEval show faster convergence than Flow-GRPO in terms of training steps.
Critical weaknesses
  1. I am skeptical of the paper's theoretical starting point, namely decomposing the transition likelihood of a generated sample at the token level. The image distribution is a joint distribution with dependencies across patches, and I do not think it can simply be decomposed without further justification. Eqs. (5) and (6) are confusing to me. The network πθ\pi_\theta takes the whole noisy image as input and predicts a velocity field for the whole image. The output of any patch therefore depends on the full input. I do not see how the decomposition in Eqs. (5) and (6) reflects the model used in practice or the underlying image-generation problem.
  2. The reward-gradient-based spatial weight is a manually designed heuristic rather than something derived from a stated loss or policy-gradient objective. This weakens the theoretical motivation for the weighting rule.
Questions for authors
  1. Can the authors provide a mathematical derivation of the token-level likelihood and importance ratio in Eqs. (5) and (6), making clear whether each patch is conditioned on the full xTx_T?
  2. In Eqs. (5) and (6), does TT refer to a single denoising timestep tt? Elsewhere in the paper, TT denotes the total number of denoising steps. Please clarify the notation.
Reviewer_T8TpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SGA-Flow-GRPO, which combines token-wise likelihood ratios aligned with DiT latent patches and spatial weighting of group-relative advantages. Spatial weights are derived from PickScore and ImageReward gradients using MAD-based normalization and temperature-scaled sigmoid mapping. Evaluations on SD3.5-Medium using GenEval report improved alignment and fewer optimization steps than Flow-GRPO. Ablations suggest that likelihood tokenization primarily improves convergence, while spatial weighting provides additional gains in evaluation scores.

Critical strengths
  1. Token-wise likelihood decomposition provides a mathematically well-defined finer-grained optimization signal for DiT patches, although its clipped objective is no longer equivalent to the original transition-level formulation.

  2. Component-wise ablations. The experiments separately examine likelihood tokenization and spatial weighting, suggesting that the former primarily improves convergence while the latter further improves alignment scores.

Critical weaknesses
  1. Practical efficiency remains unclear. Spatial weighting requires image gradients through PickScore and ImageReward, but the presented comparisons primarily measure optimization steps. GPU-hour comparisons against Flow-GRPO, the tokenization-only variant, and DiffusionNFT are needed to account for this additional cost.

  2. Auxiliary gradients are not necessarily task-specific credit signals. Under CFG = 4.5, advantages come from GenEval, whereas spatial weights come from separate preference models. The reported gains support the weighting scheme’s utility, but do not establish that auxiliary-score sensitivity faithfully reflects local contributions to the task reward.

  3. Experimental coverage and reward controls are limited. Evaluation is confined to GenEval, without standalone OCR or PickScore-only training results in the presented experiments. Under CFG = 1.0, the method uses multiple scalar rewards while baselines default to GenEval alone; despite reward-combination ablations, reward matching across comparisons requires clarification.

  4. The spatial module’s marginal benefit needs stronger validation. Ablations attribute most convergence gains to tokenization, with spatial weighting improving scores but not noticeably accelerating convergence.

Questions for authors

See Weakness.

Reviewer_hCnMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SGA-Flow-GRPO, extending Flow-GRPO with patch-level likelihood ratios and reward-gradient-based spatial weights. Experiments on SD3.5-Medium show faster convergence than Flow-GRPO and competitive results on GenEval and Pick-a-Pic. Ablations attribute most of the acceleration to likelihood tokenization, with spatial weighting providing further score improvements.

Critical strengths

The token-level likelihood function form is a simple, plug-and-play extension of Flow-GRPO. Figure 6 shows that it accelerates the optimization process compared to the original form; therefore, this component may have significant practical value.

Critical weaknesses
  1. Inverse-gradient weighting is not well supported. Equation (10) assigns higher weights to regions with smaller reward gradients, but a small gradient does not necessarily mean that a region is correct or deserves more credit. Figure 3 provides a qualitative example, while the ablation studies do not directly compare this rule with positive-gradient or shuffled weights. The reported training acceleration stems primarily from likelihood tokenization.
  2. The method requires differentiable reward models. GRPO’s advantage in industrial applications lies precisely in its ability to use black-box rewards, including rubric-based and closed-source VLM evaluators. However, SGA requires differentiable auxiliary models to compute spatial weights. The paper does not establish whether these weights reflect what the actual reward evaluator values.
  3. The performance gains are not yet convincing enough. On GenEval, SGA scores 0.95 at 1.8k iterations, compared with DiffusionNFT’s 0.94 at 1.7k iterations. Without uncertainty estimates, it is unclear whether this small difference is reliable. Furthermore, the paper does not clearly establish how much spatial weighting contributes beyond tokenization alone.
Questions for authors
  1. Evidence request regarding Weakness 1: What evidence supports the interpretation that smaller reward gradients deserve greater credit? Can you provide controlled comparisons with positive-gradient weighting and shuffled spatial weights, alongside the token-only baseline?
  2. Has the method achieved significant performance gains on additional benchmarks or in additional use cases, thereby demonstrating its generalizability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.