acceptodds
Under review as a conference paper at ICLR 2027

STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training

Abstract

Existing RL post-training methods for text-to-image generation usually convert the final-image reward into a single scalar advantage and apply it with the same strength to the entire generative trajectory. However, text-to-image generation naturally has temporal and spatial structure: different denoising steps are responsible for different generation stages, and the content that truly determines text alignment often appears only in part of the image. This granularity mismatch makes it difficult for policy updates to focus on the generative components that actually affect the reward. To address this issue, we propose SpatioTemporal Adaptive Reward (STAR) Allocation for RL post-training of text-to-image diffusion and flow models. STAR uses text-image attention inside the generative model and starts from the core content that the user truly cares about in the prompt. It constructs spatial allocation maps that dynamically vary across denoising steps and rollouts, and allocates the same group-relative advantage to more relevant latent regions with almost no additional computational overhead. STAR then applies stronger policy updates to these regions through a spatially resolved policy objective. We use Stable Diffusion 3.5 Medium as the base model and evaluate on three tasks: GenEval, OCR text rendering, and PickScore. Experimental results show that STAR improves compositional semantic alignment, text rendering, and preference optimization without changing the external reward source, achieving , , and on GenEval, OCR, and PickScore, respectively.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Reinforcement Learning Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ezFmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpatioTemporal Adaptive Reward (STAR) Allocation for RL post-training of text-to-image diffusion and flow models. STAR uses text-image attention inside the generative model to construct spatial allocation maps that vary across denoising steps and rollouts, allocates the same group-relative advantage to more relevant latent regions, and applies stronger policy updates to them through a spatially resolved policy objective.

Critical strengths

The paper targets a real problem with a cheap fix. In RL post-training for text-to-image models, one image-level score gets applied equally to every denoising step and image region, even though only part of the image (like the chair in “a red chair”) decides the score. STAR reuses the attention the model already computes to put more weight on prompt-relevant regions, with no extra reward model and only under 2% more training time and 1.56% more memory, so it’s easy to drop into existing pipelines.

The paper also worked out why the obvious approach fails: multiplying weights into the reward gets averaged away, since the probability ratio is shared across positions. Their stop-gradient trick leaves the loss and clipping unchanged but lets each position keep its own weight in the gradient. Gains appear across three different reward types (rule-based, OCR, learned preference), so it doesn’t look tied to one setup.

Critical weaknesses

The evidence doesn’t show where the gains come from, or that attention does what the paper claims. Improvements are small and near saturation (GenEval 0.95 → 0.9759, PickScore 23.31 → 23.60), with no multiple seeds or error bars. The ablations only vary STAR’s own design choices; key controls are missing (random or shuffled spatial weights, uniform up-weighting, spatial-only vs. temporal-only, sensitivity to λ). The paper also never shows real attention heatmaps or coefficient maps across denoising steps (Figure 1 is only a schematic), so it is unclear whether the maps localize prompt-relevant regions. Per-step heatmaps, a quantitative overlap check with detection or segmentation masks, failure cases, and same-prompt, same-seed comparisons of Flow-GRPO vs. Flow-GRPO+STAR would help.

The evaluation is narrow, and the theory is disconnected from the experiments. The only baseline is Flow-GRPO, on a single backbone (SD3.5-M), with no comparison to related credit assignment methods (e.g., B2-DiffuRL) and no image quality, diversity, or human evaluation. Theorems 3 and 4 rely on assumptions that are never tested, and Theorem 5 only gives a first-order gain for small step sizes, so the theory adds little real support.

Questions for authors
  1. Are the reported numbers (GenEval 0.9759, OCR 0.9757, PickScore 23.60) from a single run or averaged over seeds? Could you report mean ± std over several seeds for both Flow-GRPO and STAR under identical training budgets? If the gaps fall within seed noise, my assessment would change substantially.
  2. Could you compare STAR against (a) random or shuffled spatial coefficients, (b) constant uniform up-weighting (e.g., all positions multiplied by 1.25, the mean of C), (c) spatial-only and temporal-only variants, and (d) different values of λ? Which of these recovers most of the improvement?
  3. Were Flow-GRPO and STAR trained with identical hyperparameters, steps, and reward evaluations? Are the Flow-GRPO numbers (GenEval 0.95, OCR 0.92, PickScore 23.31) reproduced in your setup or taken from the original paper? The OCR baseline in particular seems low.
  4. Can you empirically check the assumptions behind Theorems 3-4, for instance whether the per-position gradient projections a_{b,u} correlate with H_{b,u}? Otherwise, what supports the claim that the gain comes from attention-gradient alignment?
  5. The paper says STAR applies to SDXL and Qwen-Image style backbones, but only SD3.5-M is tested. Is there any evidence on another backbone? How sensitive is the method to the text-unit extraction (LLM-based for OCR vs. the deterministic rule)?
Reviewer_86nBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes STAR, a SpatioTemporal reward allocation method for text-to-image RL post-training. STAR leverages text-image attention maps to redistribute the scalar reward across denosing timesteps and spatial tokens. The paper further provides a theoretical analysis of the gradient reweighting approach, establishing better local optimization than uniform weighting under gradient coherence and attention–gradient alignment assumptions. Built upon Flow-GRPO, the proposed method is evaluated on GenEval, OCR text rendering, and PickScore benchmarks.

Critical strengths
  • The proposed STAR introduces a lightweight attention-based reward allocation mechanism without requiring additional reward models. The reported computational overhead is minimal, making the approach practical for existing diffusion RL pipelines.

  • Beyond the heuristic attention-based design, the paper provides a theoretical analysis involving gradient coherence and attention–gradient alignment, establishing sufficient conditions under which the proposed reweighting achieves better local optimization than uniform weighting.

Critical weaknesses
  • I appreciate the extensive comparisons with different generative models, including autoregressive and diffusion models. However, the main focus of this paper is RL-based post-training, and the only RL method included in the comparison is Flow-GRPO, which is an essential baseline given that STAR builds upon it. While comparisons with other representative diffusion post-training methods, particularly recent advanced approaches such as DiffusionNFT [1], are missing. These comparisons are insufficient to establish STAR's competitiveness against existing post-training approaches.

  • The paper's core claim on SpatioTemporal credit assignment appears overstated. Despite the theoretical analysis of local optimization, text-image attention primarily captures semantic relevance rather than the actual contributions of spatial regions or denoising steps to the final reward. Such assumptions are not empirically validated. Moreover, critical ablations, such as random attention allocation and matched uniform weighting, are missing.

[1] Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

I have no further questions about this work. Please refer to the main weaknesses.

Reviewer_meK1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAR, an RL post-training method for text-to-image diffusion models by assigning stronger training signals to image regions that matter most for the prompt, rather than applying the same reward uniformly to every region and denoising step. It uses the diffusion model's existing text-image attention maps to decide where to focus optimization, and improves Flow-GRPO's performance in the GenEval benchmark.

Critical strengths
  • The paper proposes an interesting approach that leverages the model's internal text-image attention maps to spatially reweight the advantage during RL post-training. This provides a simple and computationally efficient way to incorporate fine-grained information into policy optimization without requiring additional reward models.
  • STAR achieves consistent, albeit relatively modest, improvements over Flow-GRPO across GenEval, OCR text rendering, and PickScore, demonstrating the potential effectiveness of the proposed reward allocation strategy with minimal computational overhead.
Critical weaknesses
  • The experimental evaluation is somewhat limited, particularly regarding compositional generation. The main result emphasizes GenEval, where the Flow-GRPO baseline already achieves a high overall score of 0.95, and several subcategories are nearly saturated. Although STAR further improves the overall score to 0.9759, the high baseline performance makes it difficult to assess the practical significance of the improvement and whether the method can handle more challenging compositional scenarios. Evaluating STAR on more demanding benchmarks, such as T2I-CompBench++, would provide stronger evidence of its effectiveness on complex attribute binding, spatial relationships, and compositional reasoning. Such evaluations would also help establish whether the proposed fine-grained reward allocation generalizes beyond relatively saturated benchmarks.
  • Theorem 1 establishes a local decomposition of the standard policy-gradient identity for Gaussian diffusion transitions. While mathematically valid, this result is not specific to the proposed method and does not establish any relationship between text-image attention weights and reward-sensitive gradient contributions. Therefore, its relevance to motivating STAR's attention-based reweighting is limited. Besides, Theorem 4 assumes favorable alignment between attention weights and useful local gradient contributions through semantic separation and covariance conditions. However, this relationship is central to the proposed method and is largely assumed rather than theoretically established or directly validated. Consequently, the optimization improvement in Theorem 5 holds only under these additional assumptions and does not provide a general guarantee for STAR.
Questions for authors

The paper does not directly investigate whether regions with higher text-image attention actually correspond to regions with larger or more useful reward-gradient contributions. Quantitative analysis of the correlation between attention weights and local reward sensitivities would substantially strengthen the proposed motivation. Furthermore, since STAR's allocation coefficients are greater than or equal to one, improvements over Flow-GRPO could partially arise from increased gradient magnitudes rather than better spatial credit assignment. A mean-matched uniform-weighting baseline would help disentangle these effects. Can the authors provide more experimental evidence to address these issues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.