acceptodds
Under review as a conference paper at ICLR 2027

Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

Abstract

Unified multimodal models can both understand and generate images, making it possible, in principle, for them to iteratively repair their own generations: diagnose what an image gets wrong, revise it, observe the rendered result, and reflect again. Because whether a revision is effective can only be determined after rendering, reflection and image generation must be optimized jointly over the entire interaction trajectory. Supervised fine-tuning (SFT) on reflection trajectories teaches the interleaved reasoning-and-generation format, but does not distinguish which reflections actually lead to better images; meanwhile, naive reinforcement learning (RL) that optimizes only the renderer or a single output head leaves much of the potential gain untapped. We introduce **UMM-Reflection**, a trajectory-level RL framework that optimizes complete reflection trajectories within a single unified multimodal model. Sibling trajectories share the same initial image, allowing group-relative advantages to directly compare alternative reflection strategies, while a single trajectory-level advantage jointly updates both the reflection tokens and the flow-based image revisions, avoiding the combinatorial blow-up of per-round credit assignment. Unlike single-round editing methods or agentic pipelines that rely on an external critic, UMM-Reflection propagates credit across multiple rounds and across both roles of the same model, while requiring no external verifier at inference time. Built on BAGEL, UMM-Reflection improves GenEval by 12.05 points over SFT, with strong zero-shot transfer to WISE (+10.97), OneIG-Bench (+3.48), and T2I-CompBench++ (+4.63), none of which is used during training.

Primary area:
generative models
Keywords:
Unified Model, Reinforcement Learning, Self-reflection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LQFTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UMM-Reflection trains a unified BAGEL model to iteratively inspect and revise its generated images. After SFT on externally constructed reflection trajectories, the method samples complete trajectories from a shared initial image and uses a trajectory-level advantage to update both reflection tokens and flow-based revisions. Experiments on GenEval and three benchmarks not used for RL training evaluate generation quality and the contributions of joint training and iterative refinement.

Critical strengths
  1. The improvements over reflection SFT are substantial, including 12.05 points on GenEval and 10.97 points on WISE, with additional gains on OneIG-Bench and T2I-CompBench++. Improvements on three benchmarks not used for RL training support the generalization of the learned refinement policy beyond the training setting.

  2. The experiments provide useful evidence for jointly optimizing reflection and image revision. Table 3 reports GenEval scores of 84 for joint RL, compared with 78 for text-only RL and 73 for flow-only RL. The instruction-swapping experiment further supports the benefit of improved textual guidance. Comparisons with sample selection and repeated SFT editing under a four-image budget also strengthen the evidence that the gains are not explained simply by generating more images.

Critical weaknesses
  1. I find the methodological advance over prior unified-model RL refinement limited. AlphaGRPO [1] already fixes a common initial image for refinement and uses a shared advantage to jointly optimize reflection text and flow-based image revision, while Vinci [2] already studies multi-round reflection with RL in a unified model. UMM-Reflection extends joint text–flow optimization to complete multi-round repair trajectories with a shaped trajectory reward. The extensive experiments demonstrate the effectiveness of this extension. However, the two highlighted design choices—sharing the initial image and jointly updating text and image policies with one advantage—already appear in AlphaGRPO, and their extension across repair rounds is not positioned against Vinci. This overlap narrows the methodological novelty relative to the paper’s framing.

References:

[1] Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, and Hengshuang Zhao. AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in Unified Multimodal Models via Decompositional Verifiable Reward. ICML, 2026.

[2] Wang Lin, Wentao Hu, Liyu Jia, Kaihang Pan, Majun Zhang, Zhou Zhao, Fei Wu, Jingyuan Chen, and Hanwang Zhang. Vinci: Deep Thinking in Text-to-Image Generation using Unified Model with Reinforcement Learning. NeurIPS, 2025.

Questions for authors
  1. Could you clarify the technical differences from AlphaGRPO’s joint text–flow optimization and Vinci’s multi-round RL, and explain how these differences address challenges specific to multi-round text–flow refinement? A controlled comparison with a single-round joint RL policy, using the same SFT initialization, prompt pool, verifier, and a comparable training budget, would help establish the value of this extension. Both policies should be evaluated under the same multi-round inference protocol.

  2. Under Equations 1–2 and the stated reward coefficients, the verifier-score sequences (0, 1, 1, 1) and (0, 1, 0, 1), each with three edits and no premature DONE, receive rewards of 1.30 and 1.40, respectively. Thus, losing and recovering previously achieved progress can receive a higher reward than preserving it. How often do sampled trajectories become verifier-correct, regress, and then recover? Do the gains persist when the progress terms are computed only from improvements beyond the best previous verifier score?

For future work, evaluation on another unified backbone would help establish how broadly the training approach applies beyond BAGEL.

Additional comment on related work:

I also suggest discussing several related approaches to learning reflection and refinement. GenAgent [3] trains multi-turn interactions with image-generation tools using both final-image rewards and pairwise improvement rewards. RL-RIG [4] uses Reflection-GRPO to train an editing-instruction actor and an image editor in separate stages. R³-Bench [5] evaluates visual diagnosis and rectification separately and trains a refiner using feedback from executed image edits. These works provide useful context for clarifying what UMM-Reflection gains from jointly optimizing reflection and rendering within one model, and how the quality of the learned reflection is assessed.

Additional references:

[3] Kaixun Jiang et al. GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning. ECCV, 2026.

[4] Tianyu Wang et al. RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection. arXiv:2602.19974, 2026.

[5] Junjie Wang et al. Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation. ICML, 2026.

8 comments
AuthorsOpenReview ↗

Reviewer LQFT — Part 1/2

W1 + Q1. Difference from AlphaGRPO and Vinci; single-round control

Our task is interleaved generation itself: in a unified model, rendering an image, diagnosing it, and re-rendering with the full text–image history in context forms a natural reflection chain, and UMM-Reflection trains this entire multi-round chain with one trajectory-level RL reward. AlphaGRPO trains only a single reflection step without image history ("we utilize single-turn conversations rather than multi-turn conversations (containing the history of the initially generated images)", AlphaGRPO, App.), so it cannot learn to act on its own earlier revisions. Vinci regenerates a new image each round on Emu3, a discrete-token model where standard GRPO applies unchanged; its cold-start chains are built from independently sampled candidates ordered by score (Vinci, Sec. 3.2), sibling groups start from different first images, and the text is rewarded by a separate MLLM judge. UMM-Reflection instead repairs the current image, preserving correct regions while fixing errors; all siblings share one initial image, so the advantage compares repair strategies rather than first draws; and reflection text is credited only through the image it produces, eliminating the need for a text judge. This positioning is now explicit in the revised paper, with AlphaGRPO and Vinci cited in Related Work.

Multi-round text–flow refinement differs because a revision is evaluated only after later rounds render it. Credit must therefore propagate to every reflection and every flow transition in the trajectory: shared-root sampling isolates the repair strategy from initial-draw luck, and a single trajectory-level advantage updates both heads without the KNK^N branching of per-round credit (Sec. 3.4).

The controlled single-round comparison uses the same SFT parent, prompt pool, verifier, and KK, with 500 updates each and the same 3-round inference at evaluation.

RL horizonGenEvalWISEOneIGT2I-Comp++Repair
SFT72.063.079.450.420.6
1-round joint RL76.666.480.751.935.1
3-round joint RL (ours)82.070.382.454.161.1

3-round training outperforms single-round training on all four benchmarks (GenEval +5.4, WISE +3.9, OneIG +1.7, T2I-CompBench++ +2.2) and raises the repair rate from 35.1% to 61.1%. We have added this experiment to the revised paper.


Q2. Path dependence of the reward

β\beta credits improvement accumulated across multiple edits, e.g., (0.1, 0.3, 0.45, 1), rather than a single lucky fix (Sec. 3.4); with β=0\beta = 0, GenEval drops from 84 to 79 (Table 3). The (0, 1, 0, 1) ordering is not exploitable: a failed repair, (0, 1, 0, 0), scores −0.20-0.20, so deliberately breaking a correct image yields an expected 1.6r−0.21.6r - 0.2 versus 1.30 for keeping it, which is lower for every per-edit repair probability r<15/16r < 15/16, while the policy's per-edit repair rate is at most 38% (Sec. 6.1). Trajectories that rise, drop, and recover make up 2.4% of training rollouts and 1.4% on GenEval.

When progress and multi-improvement terms are computed only from improvement beyond the best previous score, as the reviewer suggests ((0, 1, 0, 1) then scores 0.80), the gains persist:

Reward (500 updates)GenEvalWISEOneIGT2I-Comp++RepairDamageCycle rate (GenEval, %)
ours82.070.382.454.161.19.31.4
best-so-far progress82.370.282.254.459.79.00.7

Both rewards match within 0.3 points across all four benchmarks. We have added this ablation to the revised paper.

AuthorsOpenReview ↗

Reviewer LQFT — Part 2/2

Related work: GenAgent, RL-RIG, R³-Bench

We have made this distinction explicit and cite all three works in the revised Related Work. Each splits reflection and rendering across separate models: GenAgent's MLLM agent calls FLUX.1-dev as a fixed tool; RL-RIG trains its VLM Actor and image editor separately with Reflection-GRPO; and R³-Bench's R³-Refiner is an MLLM refiner attached to external T2I models. UMM-Reflection trains reflection and rendering in one unified model under a single advantage.

Direct evaluation isolates the learned reflection: with the image, renderer, and noise fixed, the RL policy's instruction repairs 48.4% of failing states vs. 21.4% for SFT (App. J), and masking the image given to reflection while still passing it to the renderer drops repair from 64.9% to 21.4%.


Other backbones

UMM-Reflection requires only a single model that both reads and renders images with an evaluable rendering policy likelihood (here, Flow-GRPO's SDE sampler); nothing in the reward, shared-root sampling, or shared advantage is specific to BAGEL. BAGEL is the most widely used open unified model that both understands and generates images (Sec. 4.1), and the method applies directly to other unified models of this kind. Extending UMM-Reflection to more recent unified models such as SenseNova-U1 is our future work.

Thank you again for your thoughtful feedback! We hope our responses have addressed your concerns. If any concerns remain, we'd be happy to discuss them further. If you feel your concerns have been resolved, we’d appreciate it if you would reconsider your overall score in light of our responses.

Reviewer_4HgXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UMM-Reflection, which trains a unified BAGEL model to inspect an initial image, write an edit instruction, and render up to three successive revisions. After supervised initialization on externally generated trajectories, the method samples sibling trajectories from the same detached initial image and uses one group-relative trajectory advantage to update both textual and flow-based actions. Experiments on GenEval and three transfer benchmarks report substantial improvements over reflection SFT and several matched generation and editing controls.

Critical strengths
  • Sharing one detached initial image among 16 sibling trajectories makes the group-relative comparison focus on repair behavior rather than luck in the first generation. One trajectory outcome then trains both the reflection text and the flow-based revisions of the same model.

  • The validation addresses several important alternative explanations. Table 3 compares joint RL with text-only and flow-only updates, forced three-round SFT, and Best-of-4 under the same four-image budget, while Appendix J holds the image, renderer, and noise fixed when comparing edit instructions.

  • The gains over reflection SFT are substantial and transfer beyond the training benchmark. UMM-Reflection improves GenEval by 12.05 points, WISE by 10.97, OneIG-Bench by 3.48, and T2I-CompBench++ by 4.63, while Table 8 shows the conditional repair rate increasing from 20.59% to 64.94%.

Critical weaknesses
  • Eqs. (1)–(2) can prefer a damage-and-repair cycle over a monotone trajectory with the same final score. For example, 0.4→0.6→0.4→0.60.4\rightarrow0.6\rightarrow0.4\rightarrow0.6 receives R=0.68R=0.68, while 0.4→0.6→0.6→0.60.4\rightarrow0.6\rightarrow0.6\rightarrow0.6 receives R=0.66R=0.66, because the progress and multi-improvement coefficients sum to more than the damage coefficient. The training curves do not show reward collapse, but the objective can still reinforce an unnecessary damaging edit.

  • Table 3 shows that SmultiS_{\mathrm{multi}} helps the final system, but not that it improves multi-round repair as intended. Removing it increases the reflection gain from 11 to 16 points and nearly preserves the repair rate, while most of the final-score difference comes from R0 dropping from 73 to 63. This suggests that the term mainly limits drift in the shared initial generator rather than providing the claimed multi-round credit.

Questions for authors
  • It would be helpful to report how often verifier scores decrease and later recover within one trajectory. An ablation with λ≥α+β\lambda \geq \alpha+ \beta or a path-invariant reward would show whether the cycle incentive affects training.

  • It would be helpful to mask or mismatch the current-image input to the reflection policy while still giving the correct image to the renderer. This would test whether the learned instruction depends on diagnosing the observed image.

  • A comparison with a terminal-only reward and a round-specific return computed from the existing verifier scores would clarify what the shared trajectory advantage contributes.

6 comments
AuthorsOpenReview ↗

Reviewer 4HgX — Part 1/2

W1 + Q1. The multi-improvement term and the (0, 1, 0, 1) example

β\beta credits improvement that accumulates across several edits, e.g., (0.1, 0.3, 0.45, 1), rather than a single lucky fix (Sec. 3.4). The multi-round loop requires this design: with β=0\beta = 0, GenEval drops from 84 to 79 and damage rises from 8.8% to 11.2% (Table 3).

The (0, 1, 0, 1) ordering is not exploitable. A deliberate break pays off only if the next edit repairs it: because a failed repair, (0, 1, 0, 0), scores −0.20-0.20, breaking a correct image yields an expected 1.6r−0.21.6r - 0.2 against 1.30 for keeping it, which is lower for every per-edit repair probability r<15/16r < 15/16. During training, the policy's per-edit repair rate is at most 38% (Sec. 6.1), and measured damage falls from 20% to 10% (Sec. 6.1). At evaluation, damage shrinks with every round (5.3% → 2.8% → 0.8%), while trajectories that rise, drop, and recover account for 2.4% of training rollouts and 1.4% on GenEval.

To confirm that the gain of β\beta does not come from this ordering, we train a best-so-far variant in which both the progress and multi-improvement terms count only improvement above the best score so far; (0, 1, 0, 1) then scores 0.80 and (0, 1, 1, 1) scores 1.30. Same SFT parent, prompt pool, and K=16K=16, 500 updates each:

RewardGenEvalWISEOneIGT2I-Comp++RepairDamageCycle rate (GenEval, %)
ours82.070.382.454.161.19.31.4
best-so-far progress82.370.282.254.459.79.00.7

The best-so-far variant matches ours on all four benchmarks (within 0.3 points), confirming that the benefit of β\beta comes from crediting multi-step improvement rather than the (0, 1, 0, 1) ordering. The cycle statistics and this ablation are added to the revised paper.


W2. Why removing β\beta lowers R0

R0 and repair cannot be separated here because the same flow head renders the initial image and every revision, and "the detached initial rendering is not assigned an RL loss, although it can change across updates because its generation parameters are also used for repair" (App. A). β\beta determines which edits train that head: with it, credit goes to trajectories that improve in targeted steps; without it, one large change earns the same, pushing the head toward re-rendering the whole image and causing the shared generator to drift. Per-edit change (DINO distance between consecutive images) is 0.31 without β\beta and 0.18 with it, and R0 falls from 73 to 63. This R0 drop is a direct consequence of what β\beta trains in the repair rounds.

The larger reflection gain without β\beta (+16 vs. +11) stems from a lower starting point that leaves more failing images to fix. Holding the starting images fixed isolates the repair rounds; on identical initial images (generated by the full model):

Reflection policy, same initial imagesFinalRepairDamage
without β\beta79.255.211.5
with β\beta (ours)83.864.98.8

From identical starting images, the full reward repairs 9.7 points more and damages 2.7 points less. Both analyses are added to the revised paper.

AuthorsOpenReview ↗

Reviewer 4HgX — Part 2/2

Q2. Does the reflection depend on the observed image?

Yes. Following the reviewer's protocol, the renderer always receives the correct current image, while only the reflection input is changed:

Reflection inputGenEvalRepairDamage
correct current image (ours)83.864.98.8
masked (blank) image66.521.416.0
image from another prompt68.724.714.5

Without the correct image, repair drops to the SFT level (21.4% and 24.7% vs. 20.6% for SFT) and damage nearly doubles, showing that the instruction is driven by diagnosing the observed image. This matches App. H, where 45–48% of the reflection's image attention goes to the current image. This test is added to the revised paper.


Q3. Terminal-only reward and round-specific return

Same SFT parent, prompt pool, and K=16K=16, 500 updates each:

VariantGenEvalWISEOneIGT2I-Comp++RepairDamage
terminal qTq_T only78.369.881.553.551.310.3
round-specific reward-to-go77.967.980.452.246.811.8
full reward, trajectory advantage (ours)82.070.382.454.161.19.3

Both alternatives are worse on all four benchmarks and damage more. Reward-to-go credits each round only for what follows it, so an edit whose effect is completed by the next edit receives little or negative credit; the trajectory advantage avoids this, while "the per-round progress terms in R(τ) still reward each round that improves the image" (Sec. 3.4). Both ablations are added to the revised paper.

Thank you again for your thoughtful feedback! We hope our responses have addressed your concerns. If any concerns remain, we'd be happy to discuss them further. If you feel your concerns have been resolved, we’d appreciate it if you would reconsider your overall score in light of our responses.

Reviewer_PB8eOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UMM-Reflection, which trains a BAGEL-based unified model to reflect on and repair its own images over multiple rounds. The authors first perform SFT on trajectories generated by external teachers. They then sample multiple reflection-and-editing trajectories from the same initial image and compute a trajectory-level group-relative advantage from final image quality and changes across rounds, jointly optimizing the reflection text and image generation. Relative to SFT, the method improves GenEval accuracy from approximately 72% to 84% and the repair rate for initially incorrect images from 20.59% to 64.94%. It also improves performance on three benchmarks not used for RL training.

Critical strengths
  1. The cross-benchmark results provide some evidence of transfer. WISE, OneIG-Bench, and T2I-CompBench++ are not used for RL training, yet UMM-Reflection outperforms SFT on all three under the paper's controlled local evaluation protocol. The gains therefore are not confined to the official GenEval test prompts.

  2. The analyses shed light on RL's effect. Early pass@16 shows that SFT can already produce successful repairs; later pass@1 gains, correctness probes, and fixed-renderer instruction swaps suggest that RL makes effective repairs more reliable, without fully isolating the mechanism.

Critical weaknesses

1. Limited Novelty

AlphaGRPO[1] already trains BAGEL for self-reflective refinement with GRPO, sharing one advantage across text and image-flow steps without an SFT cold start. The submission neither cites nor compares against it. The distinct contribution is multi-round trajectory training, not joint text/flow RL or shared-root sampling, which AlphaGRPO also uses. A controlled one- versus three-round comparison from the same SFT checkpoint, reward, and image budget would isolate that contribution; published GenEval/WISE scores are not protocol-matched. Earlier self-critique and iterative editing work [2–4] further calls for a precise novelty claim. The Best-of-4 and external-critic baselines do not resolve this comparison.

[1] AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward [2] Thinking with Generated Images [3] Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Models [4] GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing

2. Coupling Between the Reward and the Primary Evaluation

The RL reward uses a GenEval-style verifier, and the primary GenEval result uses a related scorer on the same six task types. Despite no exact prompt overlap and gains on other benchmarks, the in-domain improvement may partly reflect verifier or task-family adaptation. The authors should reports the performance on more benchmarks i.e., WISE, OneIG-Bench and T2I-CompBench++ also on ablation studies.

3. The Need for a Trajectory-Level Advantage Is Not Yet Established

One trajectory-level advantage updates every round, so an edit that damages the image may still be reinforced if a later edit repairs it. The K^N cost of exact per-round branching does not rule out cheaper controls using recorded scores. The key ablations are

  • Credit assignment: Compare the shared trajectory advantage with per-round score changes or approximate reward-to-go, updating only the corresponding round. These are inexpensive controls, not exact same-state advantages; reward-to-go can still credit an earlier mistake for a later repair.
  • Shared root: Compare shared and independent initial images with the same number of repair trajectories, reporting the extra cost of independent initial generations.
  • Reward design: Compare terminal-only with the full shaped reward, as discussed in Weakness 4.
  • Group size: Test K=4, 8, 16, and 32 while accounting for total generated images and training time.

The existing text-only/flow-only ablations do not address these questions.

4. Ablations on Reward Shaping

The reward contains several hand-set terms. Table 3 removes the multi-improvement term, and Appendix P tests another DONE reward, but neither provides a terminal-only baseline or isolates the remaining terms. Adding terms one at a time would clarify their effects on accuracy, repair, damage, and edit count.

5. Stopping Behavior and Damage Need Explanation

The main model averages the maximum three edits and damages 8.8% of initially correct images, versus 6.5% for SFT. The wrong-DONE penalty does not directly discourage stopping on a correct image, so what causes the extra edits and damage? Per-round DONE/EDIT decisions would clarify the stopping behavior.

Questions for authors
  1. What are the end-to-end inference latency and per-prompt GPU cost of UMM-Reflection, direct T2I-RL, and Best-of-4 on the same hardware, including controller calls?

  2. Could the authors report per-round DONE/EDIT rates and repair/damage transitions, stratified by initial correctness and verifier confidence? This would clarify why the model uses all three edits and has a higher damage rate.

  3. Do the gains persist at BAGEL's native 1024×1024 resolution, especially for counting, spatial relations, and text? Please use a matched evaluation protocol and report the added inference cost.

6 comments
AuthorsOpenReview ↗

Reviewer PB8e — Part 1/3

W1. Novelty vs. AlphaGRPO

Interleaved text–image generation in a unified model forms a natural reflection chain: the model renders an image, diagnoses it, and renders again with its full text–image history in context. UMM-Reflection trains this entire multi-round chain with one trajectory-level RL reward. AlphaGRPO, also on a unified model, trains only a single reflection step without image history ("we utilize single-turn conversations rather than multi-turn conversations (containing the history of the initially generated images)", AlphaGRPO, App.), so it cannot learn to act on its own earlier revisions. That is where SFT fails most, circling between edits (Fig. 1) and gaining only +2 points over three rounds (Fig. 4).

Controlled comparison: identical SFT parent, prompt pool, verifier, and KK, with 500 updates each under the same 3-round inference.

RL horizonGenEvalWISEOneIGT2I-Comp++Repair
SFT72.063.079.450.420.6
1-round RL76.666.480.751.935.1
3-round RL (ours)82.070.382.454.161.1

Under identical settings, 3-round training outperforms 1-round training across all four benchmarks (GenEval +5.4, WISE +3.9, OneIG +1.7, T2I-CompBench++ +2.2) and raises the repair rate from 35.1% to 61.1%. Training across rounds drives this gain. We have added this experiment to the revised paper.

Thinking with Generated Images and MINT rely on imitation learning, and GenArtist depends on an external critic at inference; none trains reflection with RL across rounds. We have made this positioning explicit in the revised paper and cite AlphaGRPO and these works in Related Work.


W2. Reward–evaluation coupling

The transfer results rule this out. (i) Direct Flow-GRPO with the same verifier and prompt pool reaches 76 on GenEval but only 54 on WISE, below Base (55), confirming that fitting the verifier does not transfer, whereas UMM-Reflection reaches 74 on WISE. (ii) Across the three unseen benchmarks, the largest gain occurs on WISE (+10.97 over SFT), evaluated by GPT-4o on world-knowledge prompts unrelated to the GenEval detector. (iii) Every new training ablation below is reported across all four benchmarks, and the rankings on WISE, OneIG, and T2I-CompBench++ match GenEval.


AuthorsOpenReview ↗

Reviewer PB8e — Part 2/3

W3 + W4. Trajectory-level advantage and reward terms

Identical SFT parent, prompt pool, and K=16K=16, with 500 updates each:

VariantGenEvalWISEOneIGT2I-Comp++RepairDamage
terminal qTq_T only78.369.881.553.551.310.3
round-specific reward-to-go77.967.980.452.246.811.8
full reward, trajectory advantage (ours)82.070.382.454.161.19.3

Both alternatives underperform across all four benchmarks and incur higher damage: terminal-only loses 3.7 GenEval points, and round-specific reward-to-go loses 4.1. Because reward-to-go credits each round solely for subsequent outcomes, an edit whose effect is completed by the following edit receives little or negative credit. We have added both ablations to the revised paper.

Our design already provides a per-round signal: "the trajectory-level advantage keeps the K-sample cost of GRPO while the per-round progress terms in R(τ) still reward each round that improves the image" (Sec. 3.4). Computing exact per-round group advantages requires KN=163=4,096K^N = 16^3 = 4{,}096 rollouts per root (Sec. 3.4).

Reward terms. Beyond the terminal-only ablation above, we isolate the multi-improvement term: removing β\beta drops GenEval from 84 to 79 and raises damage from 8.8% to 11.2% (Table 3).

Shared root. "All K sibling trajectories start from one shared initial image, so the group-relative advantage compares reflection strategies rather than lucky first draws" (Sec. 1). From one shared root, 16 siblings already contain a correct repair for 78% of failing roots while a single rollout succeeds 23% of the time (App. I), so every group carries a contrast between repair strategies on the same image. With independent roots, sibling rewards also differ by the quality of their initial images, which group normalization cannot separate from the repair strategy, and each group costs 15 extra initial generations.

Group size. K=16K=16 follows Flow-GRPO and DanceGRPO; the 78% figure above confirms that 16 siblings already provide the contrast GRPO requires.


AuthorsOpenReview ↗

Reviewer PB8e — Part 3/3

W5 + Q2. Damage across rounds

UMM-Reflection incurs lower damage per edit than SFT. SFT's 6.5% rate reflects an average of 1.65 edits. At the same three edits, SFT damages 9.7% (Table 3), whereas UMM-Reflection damages 8.8% and repairs 64.9% vs. 28.8%. During training, damage drops from 20% to 10% (Sec. 6.1). Paired against SFT, RL wins on 109 prompts and loses on 38 (p<10−8p<10^{-8}).

Per-round transitions on GenEval (correct→wrong: % of initially correct images; wrong→correct: % of initially incorrect images):

Roundcorrect→wrongwrong→correct
R15.346.1
R22.811.7
R30.87.1

Every additional round yields a net positive gain: damage declines from 5.3% to 0.8%, while each round repairs at least 7.1% of failing images. We have added the per-round transition table to the revised paper.

Q1. Latency and cost

Measured on 1× H100 at 512² with 50 steps, controller calls included:

MethodImagesLatency (s)GenEval
T2I-RL18.275.5
Best-of-4 + UND selection430.780.0
UMM-Reflection, 1 round217.681.1
UMM-Reflection, 3 rounds439.183.8

With one round, UMM-Reflection already outperforms Best-of-4 (81.1 vs. 80.0) at 57% of its latency without running any external model at inference. We have added this latency comparison to the revised paper.

Q3. 1024²

All arms, including Base, are trained and evaluated at 512² (App. A). We evaluate those same checkpoints directly at 1024² without retraining under the same instruction-voice protocol:

@1024²GenEval
Base76.2
SFT77.1
UMM-Reflection84.4

The gain over SFT holds at 1024² (+7.3 GenEval). We have added the 1024² results to the revised paper.

Comments on the paper

2 comments

Dear Reviewer, thank you again for your thoughtful feedback. We would appreciate hearing whether our rebuttal addresses your concerns. If any questions remain, we would be happy to discuss them before the discussion period ends. If our response has resolved your concerns, we hope you will consider raising your score.

Dear Reviewers and Area Chair,

Thank you for the constructive feedback. With the discussion period drawing to a close, we would like to briefly summarize the main additions from our rebuttal. All changes have been added to the revised version of the paper.

  • PB8e. We clarified the distinction from AlphaGRPO and added a controlled comparison of single-round and three-round RL. Three-round training improved all four benchmarks and raised the repair rate from 35.1% to 61.1%. We also added terminal-only and per-round reward-to-go ablations, per-round repair and damage statistics, latency comparisons, and evaluation at 1024².

  • 4HgX. We tested a best-so-far reward that gives no credit for recovering previously lost progress. It halved the cycle rate while matching the original reward within 0.3 points across all four benchmarks. We also added fixed-initial-image and masked or mismatched reflection-input controls to isolate the effects of the reward and the observed image.

  • LQFT. We clarified the distinction from Vinci and reported rendered-image counts and GPU-hours. With less compute, the three-round checkpoint outperformed single-round RL on three of four benchmarks and improved the repair rate. We also discussed iterative SFT and evaluated the existing checkpoint on DPG-Bench, T2I-CoReBench, GEdit-Bench, and ImgEdit-Bench, with gains over reflection SFT across all four.

Thank you to those who have already responded. If you have not yet had a chance to comment, or if any concerns remain, we would appreciate a brief follow-up before the discussion closes. If the new evidence changes your assessment, we would be grateful if you would consider updating your review and score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.