acceptodds
Under review as a conference paper at ICLR 2027

CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models

Abstract

Online reinforcement learning has been successfully extended to flow matching for diffusion model (DM) image generation. However, this paradigm faces three limitations: (1) **Window selection.** Existing methods typically manually set the stochastic differential equation (SDE) sampling window, i.e., the denoising steps where exploration noise is injected. We instead determine it from each model's denoising trajectory. (2) **Reward saturation.** Current methods rely heavily on scoring models trained on human annotations; we find that such scores are extremely high and nearly indistinguishable on the latest SOTA open-source DMs, making advantage estimation largely ineffective. (3) **Sample inefficiency.** A single scalar reward collapses different failure modes into almost identical scores, leaving minimal gradient guidance for targeted improvement. To address these issues, we propose **CAST** (**C**ausal **A**dvantage-**S**tructured **T**raining), an RL fine-tuning method for pretrained DMs, which (1) identifies the denoising step at which each model fixes the objects and their spatial arrangement in the image and uses that timing to set the SDE window, (2) decomposes each prompt via Causal Scene Graphs (CSG) into *verifiable-atoms*, i.e., minimal semantic units such as an object, count, attribute, or spatial relation that can each be checked independently, and rewards each atom separately, and (3) projects the signed atom-level advantages into pixel space through teacher-forced attention and uses them to spatially weight the SDE policy objective. We fine-tune two of the strongest open-source DMs, FLUX.2-dev and Qwen-Image-2512, with CAST, and evaluate them on GenEval 2, a compositional benchmark on which these models still fail frequently, and on Qwen-Image-Bench for overall quality. Within almost the same training budget, CAST's improvement over the base model on the most challenging GenEval 2 prompts is up to that of Flow-GRPO, while overall generation quality also improves.

Primary area:
generative models
Keywords:
Diffusion Models, Flow Matching, Reinforcement Learning, Text-to-Image Generation, Compositional Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hHHqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses compositional failures in reinforcement learning on diffusion models. The proposed CAST method decomposes synthetic prompts generated from Causal Scene Graphs into independently verifiable atoms, scores each atom with a frozen VLM, and extracts teacher-forced attention maps to locate its evidence. The atom-level normalized advantages are then projected into a spatial advantage map that weights patch-level transition ratios in a windowed Flow-GRPO objective. Experiments on FLUX and Qwen-Image report gains on multiple benchmarks.

Critical strengths
  1. The paper tackles an important limitation of scalar rewards for compositional generation. Preserving atom-level correctness signals and tying them to spatial regions is a meaningful direction for improving credit assignment in diffusion RL.

  2. This paper is well-written and easy to follow.

Critical weaknesses
  1. The reward-saturation argument is overstated. CAST and GRPO normalize rewards within a group, so a purely compressed but still reliably ordered reward scale does not by itself eliminate the advantage signal. The reported disagreement between PickScore and GenEval 2 supports objective mismatch, but high mean scores and low coefficients of variation alone do not establish that preference rewards are ineffective.

  2. I am curious that whether Causal Scene Graph really works in image generation. In addition, I don't think the compositional image generation task itself, which is the main task in this paper, contain or present specific and unique causal relations. Can authors explain the motivation more clearly? Please see questions.

Questions for authors

CAST adopts a causal graph to help locate desired attributes. However, causal relation is just an interpretation that people describe relations between entities and it is hard to validate whether it is really useful for image understanding. In addition, the prompts used in the paper even do not formulate explicit causal relations, e.g., Pickscore is measured mainly from human preferences and its prompts typically do not contain reasonable causal graphs. I wonder whether this operation is necessary. Especially current prompts are more complex than simple compositional objects.

I respect the experiments and efforts authors have made. But at this time, it seems the contribution of this manuscript is mainly designing prompts for VLM queries and provide extra supervised signal during RL. The expression upon "causal" is very unconvincing to me. However, I'm willing to enhance my score if new insights are provided during rebuttal.

Reviewer_9DBjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CAST investigates how reinforcement learning can help already strong text-to-image models better adhere to complex prompts. The authors use Causal Scene Graphs to represent objects, attributes, counts, and relations, and employ a frozen Qwen3-VL-8B model to score each semantic atom. CAST computes group-relative advantages separately for each atom and uses attention heatmaps to assign these advantages to the corresponding image regions. It also sets a model-specific early stochastic exploration window based on changes in semantic structure during denoising. The paper presents extensive experiments demonstrating the effectiveness of CAST across multiple metrics on both FLUX.2-dev and Qwen-Image-2512.

Critical strengths
  1. CAST adopts an interesting design: it first constructs a scene graph and then jointly generates the prompt and its verification questions. The training data follows a “sample a graph, then generate a prompt” pipeline. As the graph’s nodes and relations are created, the corresponding verification questions, correct answers, and associated words in the prompt are determined simultaneously. This ensures that the semantic decomposition required for training is established before any image is generated.

  2. The experiments are extensive, and the results demonstrate the effectiveness of CAST.

  3. The proposed approach of mapping each atom’s advantage to its corresponding image region is particularly interesting. The method extracts attention from the relevant text tokens to visual tokens in the first decoder layer (L0) to construct a heatmap for each atom. These heatmaps spatially weight the atom-level advantages, which are then used to weight the local policy loss.

Critical weaknesses

1.Restricting stochastic exploration to a model-specific early denoising window appears reasonable and potentially effective. However, this design is not particularly novel, and its effectiveness needs to be demonstrated experimentally. 2.The paper lacks direct validation of whether the attention maps actually identify the regions that need to be corrected. It primarily uses ablations of the atom–heatmap pairings to demonstrate that these heatmaps are useful for training, but does not directly evaluate their localization accuracy or faithfulness under interventions.

Questions for authors

1.The paper lacks direct validation of whether the attention maps actually identify the regions that need to be corrected. It primarily uses ablations of the atom–heatmap pairings to demonstrate that these heatmaps are useful for training, but does not directly evaluate their localization accuracy or faithfulness under interventions. 2.The effectiveness of restricting stochastic exploration to a model-specific early denoising window needs to be demonstrated experimentally.

Reviewer_8XyDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CAST, an online RL fine-tuning method for flow-matching text-to-image models aimed at compositional prompt adherence. It samples SDE exploration in a model-specific early window located by the knee of a CLIP-similarity curve, scores each prompt atom with a frozen VLM (Soft-TIFA, Qwen3-VL-8B), normalizes atom scores within a group, and projects the signed atom advantages onto patches via teacher-forced VLM attention to weight per-patch likelihood ratios in a clipped GRPO objective. It is validated on FLUX.2-dev and Qwen-Image-2512 using GenEval 2 (full set and a 50-prompt hard subset) and Qwen-Image-Bench, against SFT and Flow-GRPO baselines at matched budget.

Critical strengths
  1. Atom-specific signed spatial credit, with a control that isolates it. Combining per-atom advantages with each atom's own attention map lets different atoms push different image regions in opposite directions, which a scalar or single-map scheme cannot express. The shuffled-map ablation (85.00 vs 85.95 G2 Overall, Table 2) indicates the gain depends on correct atom-to-map pairing rather than on spatial reweighting alone.

  2. Carefully matched comparison against a same-reward baseline. Flow-GRPO (G2) shares CAST's reward, prompt manifest, seed, window, schedule and budget (Sec. 5.1; Table 1b, 32.90 vs 32.43 H200-h), so the CAST vs Flow-GRPO (G2) gap isolates the advantage structure. CAST improves on this baseline on both backbones (G2 Overall +1.41 and +1.22; Hard-50 +5.27 and +2.73), with negligible extra compute.

  3. Results on strong current backbones without quality loss. Experiments use FLUX.2-dev and Qwen-Image-2512 rather than older SD-class models, and Qwen-Image-Bench Overall does not degrade (slightly higher on both, Table 1a). This matters because reward optimization often trades quality for alignment, and the qualitative figures suggest the scalar-reward baseline drifts toward simplified scenes while CAST stays close to the base style (Figs. 5–6).

Critical weaknesses
  1. The training reward and the evaluator appear to be the same model, on in-distribution prompts. As far as I can tell, training uses GenEval 2 Soft-TIFA with Qwen3-VL-8B, which is also the evaluator for all reported compositional numbers (Sec. 5.2), and the training prompts are synthesized to mirror GenEval 2's templates (Appendix D). Without an independent judge, an additional compositional benchmark, or a human evaluation, it is difficult to separate the claimed improvement in compositional adherence from optimization of this specific VLM's scores.

  2. Several components appear to have close precedent, which narrows the contribution. Per-atom group normalization before summation closely resembles the decoupled per-reward normalization of GDPO (arXiv 2601.05242); patch-level advantage maps with per-position likelihood ratios appear in ViPO (arXiv 2511.18719), and region-level credit for Flow-GRPO in SpatialFlow-GRPO (arXiv 2606.26872); early-timestep emphasis is also studied in TempFlow-GRPO (ICLR 2026) and MixGRPO. Apart from MixGRPO, these works do not seem to be cited or discussed. In light of this, the main distinctive contribution seems to be pairing each atom's signed advantage with its own VLM attention map, and the paper may read as incremental unless this component is shown to be effective and general.

  3. The evidence that the atom-specific map is effective, and effective beyond Flow-GRPO, could be strengthened. The most direct evidence is a single comparison of matched vs shuffled maps (85.95 vs 85.00 G2 Overall, Table 2), from one training run on FLUX.2-dev, without a Hard-50 result, and the localization accuracy of the layer-0 attention heatmaps is not reported. The map is also evaluated only within a Flow-GRPO-style per-transition SDE ratio; recent likelihood-free or ELBO-based methods such as DiffusionNFT (ICLR 2026) and EPG/PEPG (arXiv 2602.04663) train with a forward-process velocity-regression loss, where M_p could enter naturally as a per-patch weight on the regression error, and demonstrating gains on top of these optimizers would make a much stronger case that the map is a general credit-assignment component rather than a correction specific to one baseline. In addition, the headline 3.07× ratio divides +4.05 by +1.32, a denominator smaller than one replicate standard deviation, and training-seed variance is not reported.

Questions for authors
  1. Independent evaluation. Could the authors report Hard-50 and full GenEval 2 results scored by a VQA model other than Qwen3-VL-8B (or human judgments on a sample of Hard-50), together with one additional compositional benchmark that is not mirrored by the prompt generator? If CAST's margin over Flow-GRPO (G2) persists, this would largely address Weakness 1; if it shrinks, the gains may partly reflect reward-specific optimization.

  2. Variance and window. Would the authors be able to provide results over at least three training seeds for CAST and Flow-GRPO (G2), with paired per-prompt confidence intervals on Hard-50, as well as one window ablation (e.g., a full-trajectory or late window compared with the knee-based window)? This would clarify whether the reported gain ratios are robust and how much the model-specific window contributes.

  3. Effectiveness and generality of the atom-specific map. Could the authors (a) apply the per-atom map to at least one recent optimizer outside the Flow-GRPO family, such as DiffusionNFT or EPG/PEPG, for example by weighting the per-patch velocity-regression error with M_p, and compare it with the same optimizer using the aggregated reward; (b) report the matched-vs-shuffled and per-atom-without-map ablations on Hard-50 and on Qwen-Image-2512; and (c) provide a measure of heatmap localization (e.g., overlap with detector boxes, including missing-object and wrong-count atoms)? Given that the other components have close precedent (Weakness 2), these results would help establish whether the paper's distinctive component is a general improvement or specific to the Flow-GRPO setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.