acceptodds
Under review as a conference paper at ICLR 2027

ProOPD: Process-Guided On-Policy Distillation for Iterative Text-to-Image Generation

Abstract

Unified multimodal models (UMMs) can improve text-to-image generation through an iterative process that interleaves textual reasoning, image synthesis, reflection, and refinement, but training such policies faces two challenges: heterogeneous task objectives and the limited guidance that final-image rewards provide for intermediate decisions. We introduce ProOPD, a process-guided on-policy distillation framework for iterative text-to-image generation. A shared student produces its own text and image trajectories, and frozen stage-specific teachers provide dense supervision at the resulting token prefixes and visual sampling states. The supervision covers initial reasoning and generation as well as subsequent reflection and refinement, following the states encountered by the student as its policy changes. We combine textual distribution matching and visual velocity matching with image-level outcome optimization to train the generation process toward improved image quality. Experiments on four challenging text-to-image benchmarks (OneIG-EN, LongText-EN, WISE, and GenEval++) assess generation performance and the effectiveness of iterative refinement, with ablations examining the contributions of process supervision and outcome optimization.

Primary area:
generative models
Keywords:
unified multimodal models, text-to-image, RL, OPD

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bcU2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProOPD introduces process-guided on-policy distillation for iterative text-to-image generation in a unified BAGEL model. Frozen stage-specific teachers supervise the student’s own trajectories through textual distribution matching and visual velocity matching, combined with image-reward GRPO. Experiments on four benchmarks compare the method with joint SFT, reward-only training, and distillation-only training.

Critical strengths
  1. ProOPD provides teacher supervision for both text and image generation on the student's own rollouts, while retaining a single model at inference.

  2. The comparisons in Table 3 use the same initialization and rollout procedure. Adding process supervision to GRPO improves LongText by 3.66 points and WISE by 4.00 points. Table 6 also controls for differences in the initial images and plans when comparing refinement.

Critical weaknesses
  1. I cannot tell whether the model has learned to diagnose errors and write better edit instructions, or mainly become a better image editor. Table 6 fixes the initial images and plans, but critique, replanning, and refinement still change together. This comparison evaluates the later stages as a whole, without separating gains from better reflection and edit instructions from gains in image editing.

  2. The gains from adding GRPO to Pure OPD are uneven across benchmarks. In Table 3, ProOPD gains 1.95 points on OneIG text rendering and 1.70 on WISE, but only 0.31 on LongText, while scoring 0.36 points lower on GenEval++. These results are reported at a single 160-update checkpoint. It is unclear whether the relative performance remains similar over training or reflects different convergence rates.

Questions for authors
  1. How much of the refinement gain comes from better reflection and edit instructions, and how much from improved image editing? A comparison that combines the SFT and ProOPD textual outputs with each model’s image editor, using the same initial images, would help separate these contributions.

  2. How was the 160-update checkpoint in Table 3 selected, and how does the relative performance of ProOPD, Pure OPD, and GRPO change over training? Please also compare their training costs.

  3. Which models or data sources produced the reference images and stage-specific textual supervision used for SFT? Please summarize the retained example counts for each specialist teacher and the generation/editing mixture for the refinement teacher. Beyond the reported image-quality and prompt-satisfaction filtering, how was the consistency of initial plans with reference images and edit instructions with accepted refined images validated?

  4. Which models and criteria were used to construct the semantic and quality checks for GRPO rewards, and how were verifier responses and OCR outputs converted into reward scores?

For future work, I would like to see performance over several refinement rounds and a comparison with an external VLM guiding the image generator at a comparable inference budget.

Additional comment on related work:

I also suggest discussing related work on learning and evaluating reflection-guided image refinement. R³-Bench [1] separately evaluates visual error diagnosis and the effectiveness of corrective instructions. RL-RIG [2] uses Reflection-GRPO to train an editing-instruction actor and an image editor in separate stages. Reflect-DiT [3] learns to refine generations using previous images and feedback from an external VLM. These works provide useful context for positioning ProOPD’s stage-specific distillation and for distinguishing improvements in reflection from improvements in image editing.

References:

[1] Junjie Wang et al. Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation. ICML, 2026.

[2] Tianyu Wang et al. RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection. arXiv:2602.19974, 2026.

[3] Shufan Li et al. Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflection. ICCV, 2025.

7 comments

Could you please respond to the remaining questions before the discussion closes? If the additional evaluations are still in progress, a brief update would also be helpful. Thank you.

Thank you very much for following up, and for the time and care you have put into reviewing our work. All additional evaluations are now complete, and we will post a full response to all of your questions as soon as possible.

AuthorsOpenReview ↗

Response to Reviewer bcU2 (1/4)

We sincerely thank the reviewer for the detailed review and constructive feedback. We address each point below.

W1/Q1. Contributions of textual instruction and image editing

We conducted a 2×2 crossover, combining SFT and ProOPD textual instruction (critique and edit instructions) with each model’s image editor. All four conditions use identical SFT initial plans and images. Each set of critiques and edit instructions is reused unchanged across both editors, with paired editing noise. Stopped samples retain their initial images.

Textual instructionImage editorWISELongTextOneIG AlignOneIG TextReasoningStyleDiversity
SFTSFT69.4058.3383.8773.4932.4844.4921.15
SFTProOPD70.2065.1285.0777.7832.4944.4319.36
ProOPDSFT66.1059.4683.3175.9332.7544.4221.80
ProOPDProOPD71.2066.7285.2479.3532.6744.4919.64

With textual instruction fixed, replacing the SFT editor with the ProOPD editor improves LongText by 6.79–7.26 points, OneIG Text by 3.42–4.29 points, and WISE by 0.80–5.10 points. With the editor fixed, replacing SFT textual instruction with ProOPD textual instruction improves LongText by 1.13–1.60 points and OneIG Text by 1.57–2.44 points. Its WISE effect depends on the editor: −3.30 points with the SFT editor and +1.00 point with the ProOPD editor.

These results show that improved image editing (refinement) provides the larger text-rendering gains, while learned textual instruction (critique and replaning/editing) contributes additional gains and interacts with the editor. Other OneIG dimensions show mixed changes. This control measures the downstream effectiveness of textual instruction, including stopping decisions.

Reviewer_CGAzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProOPD, a process-guided on-policy distillation framework for iterative text-to-image generation with unified multimodal models. The method aims to address the capability-integration gap between a pipeline of stage-specific specialists and a single jointly fine-tuned model. ProOPD employs five frozen teachers for planning, generation, critique, replanning, and refinement. A shared student generates its own trajectories and receives process-level supervision through reverse KL divergence for textual actions and velocity matching for visual actions. The framework further combines these distillation objectives with GRPO-based image-level outcome optimization.

Critical strengths

Promising experimental improvements. ProOPD improves over joint SFT on several benchmarks, particularly text rendering and compositional instruction following.

Critical weaknesses
  1. Insufficient ablations. The paper lacks sufficient ablation studies to clarify the contribution of each teacher and the necessity of using multiple specialized teachers. In particular, it would be valuable to investigate how removing individual teachers (e.g., planning, critique, or refinement) affects the overall performance. Additionally, comparing the proposed multi-teacher framework with distillation from a single jointly trained teacher would help establish whether the improvements primarily arise from stage-specific specialization or from the on-policy distillation procedure itself.
  2. Lack of more evidence for reliable iterative refinement. The paper emphasizes iterative generation and refinement, but the experiments consider at most one or two refinement steps. Consequently, it remains unclear whether the learned policy supports stable improvements over multiple iterations.
Questions for authors
  1. Can the authors provide more ablation studies?
  2. Does repeated refinement beyond two generation rounds lead to further improvements or accumulating errors?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CGAz

We sincerely thank the reviewer for the detailed review and constructive feedback. We address each point below.


W1/Q1. Additional ablations of stage-specific supervision

Given the cost of training a separate model for each teacher removal, we prioritized one single-stage ablation and two modality-level ablations. We removed OPD only from refinement to measure the contribution of direct editing supervision to the final image. We also tested text-only and image-only OPD to compare supervision of the three language stages with supervision of the two visual stages. All three variants retain the five-stage rollout and the same GRPO losses. The table reports Round 2 scores at 160 updates, alongside the manuscript's objective ablations. Bold marks the highest score for each metric.

Training settingStages receiving OPDWISELongTextOneIG TextGenEval++
GRPONone65.7062.6180.8075.00
Pure OPDAll five stages68.0065.9679.5775.36
Full ProOPDAll five stages69.7066.2781.5275.00
Text-only OPDPlanning, critique, replanning66.1054.1374.4073.21
Image-only OPDGeneration, refinement68.9065.0080.5073.93
w/o refinement OPDPlanning, generation, critique, replanning68.5063.5579.5471.07

The Round 1 scores below compare the text-only variant with the variant that adds generation OPD.

Training settingWISELongTextOneIG TextGenEval++
Text-only OPD66.1039.8547.6159.64
w/o refinement OPD68.6049.9552.6159.29

Adding generation OPD to the text-only variant improved text rendering before any editing (+10.10 on LongText and +5.00 on OneIG Text), and the gains persisted after refinement. The same change lowered Round 2 GenEval++ by 2.14 points. Adding refinement OPD to the four-stage variant improved both text rendering and compositional instruction following in the final image (+2.72 on LongText and +3.93 on GenEval++).

Image-only OPD outperformed text-only OPD on all four Round 2 metrics, with the largest gaps on LongText and OneIG Text. Adding text-stage supervision to image-only OPD improved all four metrics further, including LongText (+1.27) and OneIG Text (+1.02). The two forms of supervision therefore contribute complementary gains. Full ProOPD led the six settings on WISE and both text metrics, with GenEval++ close to Pure OPD (75.00 vs. 75.36).

Regarding the suggested single-teacher comparison, we would like to clarify that it requires first training a single model that performs all five stages well. That model could already serve directly as the policy we aim to learn.

W2/Q2. Repeated refinement

We evaluated the same ProOPD checkpoint over three rounds. Round 1 produced the initial image, and Rounds 2 and 3 performed the first and second refinements, respectively. Of the four context formats we tested for Round 3, retaining the original plan, both earlier images, and the previous critique and edit instruction gave the highest WISE and LongText scores. The table reports the three-round results for that format.

RoundWISELongTextOneIG TextOneIG AlignmentOneIG ReasoningOneIG StyleOneIG Diversity
1. Initial generation65.5049.7747.9285.9822.6442.4914.94
2. First refinement69.7066.2781.5285.1832.3843.7618.96
3. Second refinement71.5069.1080.9584.8031.9442.5719.45

The second refinement improved WISE by 1.80 points and LongText by 2.83 points. WISE judgments changed from incorrect to correct on 74 examples and from correct to incorrect on 56, so the net gain came with regressions on some examples. The reported OneIG Text, Alignment, Reasoning, and Style scores were lower by 0.57, 0.38, 0.44, and 1.19 points, respectively, and Diversity was higher by 0.49 points.


We thank the reviewer again for the constructive feedback. We hope the additional results and clarifications address the concerns raised, and we would appreciate a reassessment in light of this evidence.

Reviewer_Y1ugOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProOPD, whose central idea is to combine stage-specific expert supervision with RL. Instead of evaluating only the final image, ProOPD also teaches the model how to plan, generate, critique, replan, and refine with the corresponding process experts for each of the five stages. This method helps to improve the final model performance compared to only using SFT.

Critical strengths
  • The paper demonstrates strong performance improvements over several baselines across multiple benchmarks.
  • It also presents an interesting finding that joint SFT struggles to integrate stage-specific capabilities compared to separately trained specialists.
  • The proposed stage-specific on-policy distillation is a promising approach for integrating these capabilities into a unified model for iterative text-to-image generation. This is an interesting and somewhat novel point compared to the MOPD method this paper is based on.
Critical weaknesses

The authors compare GRPO, Pure OPD, and the combined ProOPD objective at a fixed number of optimization updates. However, these approaches incur different computational costs, particularly because OPD requires additional forward passes through multiple specialist teachers. Although the paper acknowledges this limitation, it does not report per-method GPU-hours or performance-versus-compute curves. Consequently, it is unclear whether ProOPD's improvements over GRPO-only reflect a more effective learning objective or simply greater computation. I recommend reporting detailed compute breakdowns and benchmark performance curves against cumulative training GPU-hours, including a compute-matched GRPO-only baseline. In addition, I recommend using more update steps when reporting the performance curves so that it is easier to see the trend with more compute.

Questions for authors
  • The paper demonstrates the benefit of combining OPD and GRPO but does not isolate the contributions of textual and visual process supervision. Could the authors provide ablations comparing text-only OPD, image-only OPD, and full ProOPD? Additionally, how much does each stage-specific teacher contribute?
  • MOPD trains its domain-specific teachers using RL before performing multi-teacher on-policy distillation. In contrast, ProOPD employs SFT-trained stage-specific teachers without explaining this design choice. Could the authors clarify why domain-specific RL was not considered for teacher training?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Y1ug

We sincerely thank the reviewer for the detailed assessment and constructive suggestions. We address each point below.


W1. Training costs and compute-matched performance

We report cumulative training costs for the three objectives. At 160 updates, ProOPD uses about twice the GPU-hours of GRPO. The timing breakdown shows similar student rollout times across methods, with most of the additional time spent constructing teacher targets and replaying student trajectories for the distillation updates. We also resume the GRPO experiment from the complete checkpoint and evaluated its 320-update checkpoint for comparison at a similar reported GPU-hour budget.

We included the benchmark curves against cumulative GPU-hours in the revised manuscript at Figure 3.

Timing breakdown for selected components

MethodStepsTraining hoursStudent rollout (min/update)Teacher forward (min/update)Training replay (min/update)
Pure OPD16034.082.271.636.80
ProOPD16039.122.371.727.34
GRPO32038.732.38-2.33

Training costs are reported on a common scale. Training hours include periodic evaluation and checkpointing. Component times are estimates and can overlap.

At approximately 1,200 GPU-hours, ProOPD s80 leads GRPO s160 by 4.80 points on WISE and 5.54 on LongText. GRPO leads by 0.73 points on OneIG Text and 3.21 on GenEval++.

At the larger reported budget of about 2,500 GPU-hours, ProOPD s160 improves over the longer-training GRPO s320 baseline on all four benchmarks. The largest gains are on GenEval++ (+3.93 points) and LongText (+2.12), with gains of 1.40 on OneIG Text and 0.40 on WISE. The LongText advantage is present at both comparable budgets.

Q1. Additional ablations

We compared text-only OPD, image-only OPD, and full ProOPD. Given the cost of retraining for each teacher removal, we also selected refinement for a single-stage ablation to measure the contribution of direct editing supervision. All variants retain the five-stage rollout and the same GRPO losses. The table reports Round 2 scores after 160 updates.

Training settingStages receiving OPDWISELongTextOneIG TextGenEval++
Text-only OPDPlanning, critique, replanning66.1054.1374.4073.21
Image-only OPDGeneration, refinement68.9065.0080.5073.93
w/o refinement OPDPlanning, generation, critique, replanning68.5063.5579.5471.07
Full ProOPDAll five stages69.7066.2781.5275.00

Image-only OPD improves over text-only OPD on all four metrics, with the largest gains on LongText (+10.87) and OneIG Text (+6.10). Adding textual supervision to image-only OPD improves all four scores further. Visual supervision thus accounts for the larger gains in text rendering in these comparisons, and textual supervision adds to those gains.

The Round 1 scores below compare the text-only variant with the variant that adds generation OPD.

Training settingWISELongTextOneIG TextGenEval++
Text-only OPD66.1039.8547.6159.64
w/o refinement OPD68.6049.9552.6159.29

Adding generation OPD to the text-only variant improves text rendering before editing (+10.10 on LongText and +5.00 on OneIG Text). Adding refinement OPD to that four-stage variant improves Round 2 LongText by 2.72 points and GenEval++ by 3.93 points, showing the value of direct editing supervision for text rendering and compositional instruction following. The textual ablations estimate the combined contribution of planning, critique, and replanning.

Q2. SFT for stage-specific teachers

Running RL separately for all five specialists would require additional rollouts and reward evaluations. Also, in existing RL approaches, reliable direct rewards for planning, critique, and replanning are hard to obtain. Image rewards reflect both these decisions and how later stages execute them, making credit assignment difficult.

We train the specialists with SFT on stage-specific demonstrations and apply image-level RL to the shared student over the full rollout. ProOPD can also distill from RL-trained specialists.


We thank the reviewer again for the constructive feedback. We hope the additional results and clarifications address the concerns raised, and we would appreciate a reassessment in light of this evidence. If any questions remain, please let us know so we can provide further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and the Area Chair

Dear Reviewers and Area Chair,

We sincerely thank the reviewers and the Area Chair for their time and careful consideration. We appreciate the reviewers' recognition of ProOPD's stage-specific supervision and its gains over Joint SFT. Their feedback helped us clarify where these gains come from and how they change with training and inference compute.

Core contributions

  • We identify a capability integration gap in iterative text-to-image generation. Under the same generation and refinement workflow, a pipeline of separately trained specialists renders text more accurately than a single model trained jointly with SFT.
  • We introduce ProOPD to integrate planning, generation, critique, replanning, and refinement in one policy. On the student's own rollouts, frozen specialists provide next-token distributions for text and velocity targets for images. ProOPD combines these stage-specific targets with image-level GRPO to train the same policy across all five stages.
  • A single ProOPD model at inference improves over Joint SFT on all four benchmarks after refinement, including gains of 8.08 points on OneIG Text and 12.50 on GenEval++. It also surpasses the specialist pipeline on OneIG Text and cuts the LongText gap by about half.

Additional evidence and revisions

  • Textual instruction and image editing (bcU2). A 2×2 crossover holds initial plans and images fixed and varies the sources of textual instruction and image editing. The ProOPD editor yields the larger gains on LongText and OneIG Text. ProOPD instructions improve both scores with either editor.
  • Stage and modality ablations (CGAz, Y1ug). We compare text-only OPD, image-only OPD, and OPD without refinement supervision. Generation supervision improves text rendering in the initial image, and refinement supervision further improves LongText and GenEval++. Adding textual supervision to image-only OPD improves all four final-image benchmarks.
  • Checkpoint selection and training compute (bcU2, Y1ug). We explain the choice of 160 updates using held-out reward trends from all three objectives and report benchmark results across checkpoints, a timing breakdown, and a longer GRPO comparison through 320 updates. At roughly 1,200 GPU-hours, ProOPD leads GRPO on WISE and LongText, while GRPO leads on OneIG Text and GenEval++. At the larger reported budget, ProOPD s160 scores higher than GRPO s320 on all four benchmarks, with the largest gains on GenEval++ (+3.93 points) and LongText (+2.12). Figure 3 of the revised manuscript plots benchmark scores against cumulative GPU-hours.
  • Repeated refinement and external VLM guidance (bcU2, CGAz). A second refinement raises WISE by 1.80 points and LongText by 2.83, with a small decline on OneIG Text. With the same initial images, plans, and image editor, ProOPD's instructions yield higher LongText and OneIG Text scores than both external VLMs, while Qwen3.8-Max achieves the highest WISE score.
  • Training details and related work (bcU2, Y1ug). We clarify the sources, composition, and filtering of the SFT data, how GRPO rewards are constructed and scored, and why we used SFT to train the specialists. The revised Related Work section discusses R³-Bench, RL-RIG, and Reflect-DiT.

The new compute analysis and related-work discussion are marked in blue in the revised manuscript. We provide the full results and clarifications in our individual replies. We hope these additions address the reviewers' concerns and would appreciate a reassessment in light of the new evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.