acceptodds
Under review as a conference paper at ICLR 2027

UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Abstract

Unified models capable of interleaved generation have emerged as a promising paradigm, with the community increasingly converging on autoregressive modeling for text and flow matching for image generation. To advance this direction, we propose a unified reinforcement learning framework tailored for interleaved generation. We validate our approach on its fundamental unit: a single round of reasoning-driven image generation, where the model first expands the user prompt through reasoning, followed by image synthesis. Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards, we introduce UniGRPO to jointly optimize text and image generation policies using GRPO. Adopting a minimalist methodology to avoid over-design, we leverage established training recipes for both modalities by seamlessly integrating standard GRPO for reasoning and FlowGRPO for visual synthesis. To ensure scalability to multi-round interleaved generation, we introduce two critical modifications to the original FlowGRPO: (1) eliminating classifier-free guidance to maintain linear, unbranched rollouts, which is essential for scaling to complex scenarios involving multi-turn interactions and multi-condition generation (e.g., editing); and (2) replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields, providing a more robust and direct regularization signal to mitigate reward hacking. Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning, providing a robust and scalable baseline for future post-training of fully interleaved models.

Primary area:
generative models
Keywords:
Reasoning-Driven Image Generation, Flow Matching, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_z4y6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniGRPO, a reinforcement learning framework that jointly optimizes textual reasoning and flow-based visual generation within unified multimodal models. Specifically, the authors formulate the sequential prompt–reasoning–image generation process as a unified Markov Decision Process, coupling TextGRPO and FlowGRPO via shared group-relative advantages computed from final visual rewards. To stabilize and streamline visual policy optimization, the training recipe incorporates two key modifications: removing classifier-free guidance during RL exploration to simplify rollouts, and replacing the latent KL divergence penalty with a mean squared error regularizer between the policy and reference velocity fields to prevent reward hacking. The proposed method is implemented on the BAGEL architecture following supervised fine-tuning on an internal dataset. Experimental evaluations on an internal text alignment benchmark and GenEval demonstrate performance gains over text-only, image-only, and hybrid baselines, achieving scores of 0.8381 and 0.90, respectively. Ablation studies are provided to validate the necessity of CFG removal and the velocity regularization.

Critical strengths
  1. The proposed UniGRPO integrates TextGRPO and FlowGRPO into a single reinforcement learning loop, enabling textual reasoning and image generation to be optimized using shared advantages derived from final-image rewards. This provides a coherent approach to adapting both stages toward the same downstream objective. The improvements over text-only and image-only optimization in Table 1 provide empirical support for the complete training recipe, making it a useful baseline for reasoning-driven visual generation.
  2. The CFG ablation in Figure 4 shows that training without CFG achieves comparable or better evaluation performance when CFG is applied at evaluation in both settings. Although this experiment uses the original BAGEL at 512 resolution, it provides useful evidence that the additional unconditional model evaluations can be omitted from training rollouts without an apparent performance penalty in the tested setting. This is a practical simplification with potential computational benefits, although end-to-end training speedups are not quantified.
Critical weaknesses
  1. The paper includes a two-stage baseline that first optimizes image generation with ReFL and then optimizes text generation with TextGRPO. However, neither the main experiments nor the appendix reports a matched sequential or alternating TextGRPO–FlowGRPO baseline. Consequently, the existing comparison changes both the image-side optimization objective and the training schedule, making it difficult to attribute the gains specifically to simultaneous joint optimization. A controlled comparison using the same initialization, reward, component objectives, rollout settings, and matched training budget would more convincingly establish the value of the proposed joint training strategy.
  2. Figure 5 provides encouraging evidence for the selected MSE configuration, but the KL run is terminated after grid-like artifacts appear at approximately 250 steps. Stopping a visibly degraded run is reasonable. However, the failure of this configuration does not establish that MSE is preferable to appropriately tuned KL regularization. In particular, the observation that KL substantially suppresses training reward does not demonstrate that its coefficient is well chosen, and no coefficient-sensitivity analysis is reported. Morever, the explanation that weaker KL constraints at high-noise timesteps create opportunities for reward hacking is not directly tested through timestep-resolved measurements or controlled changes to the regularization weights. The current results therefore leave unclear whether the improvement arises from removing timestep-dependent weighting, changing the effective regularization strength, or other optimization effects. This limits the evidence for both the claimed robustness and the proposed mechanism.
Questions for authors
  1. Could authors compare simultaneous joint optimization against sequential or alternating updates using the same TextGRPO and FlowGRPO objectives, initialization, reward, image-side modifications, and matched training budget? Does the advantage of UniGRPO persist when the component objectives and training resources are controlled?
  2. How were the KL and MSE coefficients selected, and does the advantage of velocity MSE persist after a reasonable coefficient sweep for both regularizers? Please also clarify the exact latent KL implementation and its timestep-dependent weighting. Is there any evidence, such as per-timestep velocity deviations or regularization-gradient measurements, linking the observed artifacts to weak constraints at high-noise timesteps?
Reviewer_hpVNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces UniGRPO, a unified reinforcement learning framework designed for reasoning-driven image generation on multimodal models that combine autoregressive text with flow-matching image generation. The authors formulate the "Prompt Thinking Image" sequence as a single MDP optimized via GRPO with sparse terminal rewards. To make the training scalable and stable, the paper introduces two practical adjustments to FlowGRPO: (1) training without Classifier-Free Guidance (CFG) to avoid computational branching during rollouts, and (2) replacing standard latent KL with an unweighted velocity-field MSE penalty to mitigate reward hacking. The method is evaluated on an internal 150-prompt Text Alignment (TA) benchmark and GenEval, outperforming disjoint text/image RL baselines.

Critical strengths

1. A timely unified RL recipe for multimodal generation. Integrating discrete AR reasoning tokens and continuous flow-matching latent steps into a single GRPO objective is a relevant and natural direction as the community explores native interleaved models.

2. Practical training stabilization tricks. Dropping CFG during RL rollouts effectively simplifies the computation graph, and the ablation demonstrates that velocity MSE prevents the grid artifacts and training collapse caused by standard latent KL divergence.

Critical weaknesses

1. Evaluation credibility and circular verifier bias. The main evaluation relies on an internal Text Alignment (TA) benchmark with only 150 prompts, scored by InternVL. Crucially, the reward model used during RL training is also fine-tuned from InternVL. This creates a risk of in-distribution reward hacking, where the model merely learns to exploit the specific quirks of the InternVL judge. The lack of independent verification (e.g., GPT-4o judge or human blind evaluation) makes the reported performance gains unconvincing.

2. Substantial overclaiming regarding interleaved/multi-round capability. The title, abstract, and introduction emphasize "interleaved generation" and multi-round scalability. The empirical validation is confined to a single-turn pipeline (Prompt Thinking Image), and actual multi-round interleaved interaction is deferred to future work.

3. Narrow experimental scope on a single base model. All experiments are conducted on Bagel. It remains unclear whether the proposed joint recipe and velocity regularizer generalize to other unified multimodal architectures (e.g., Show-o or Transfusion-based variants).

Questions for authors
  1. Given that both the reward model and the evaluation judge share the InternVL backbone, can you provide evaluation results from an independent, external judge (e.g., GPT-4o, Claude 3.5, or a blind human preference study) to confirm that the gains represent visual alignment rather than verifier overfitting?

  2. Since the paper is framed around interleaved generation, do you have any empirical results on multi-turn editing or visual dialogue to substantiate the claim that UniGRPO scales to multi-round settings?

  3. How sensitive is the framework to the choice of the base model? Have you tested UniGRPO on any other unified multimodal backbone besides Bagel?

Reviewer_uKoZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UniGRPO, a unified reinforcement learning framework for jointly optimizing textual reasoning and image generation in unified multimodal models (UMMs). The framework combines GRPO for autoregressive text generation with FlowGRPO for image synthesis, using a shared reward signal to optimize both policies. The authors further remove classifier-free guidance (CFG) during rollouts and replace the latent KL penalty with a velocity-field MSE penalty to improve training efficiency and stability. Experimental results show improvements in image generation quality over the evaluated baselines.

Critical strengths
  1. Practical significance. The paper addresses an important problem: jointly optimizing textual reasoning and image generation in unified multimodal models. Although similar ideas have been explored in prior work, this remains a meaningful research direction.

  2. Simple and practical framework. The proposed framework is relatively straightforward and builds upon established GRPO and FlowGRPO methods. The CFG-free rollout and velocity MSE regularization provide practical solutions for improving training efficiency and stability.

  3. Promising experimental results. The experiments show that joint optimization achieves better performance than optimizing the text or image policy alone. The ablation studies on CFG and regularization also provide useful insights into training stability.

Critical weaknesses
  1. Limited novelty of the core contribution. Prior work, particularly UniRL-Zero, has already explored the joint optimization of textual reasoning and image generation in unified multimodal models. Specifically, UniRL-Zero jointly optimizes the language model and diffusion model using a shared advantage signal. This significantly overlaps with the core idea of UniGRPO and weakens its novelty. The authors should clarify the key differences and additional contributions compared to this prior work.

  2. Limited justification for velocity MSE regularization. Replacing KL regularization with velocity MSE essentially changes the timestep-dependent weighting of the regularization term. It is unclear whether the improvement comes from the uniform weighting of velocity MSE or simply from the insufficient KL penalty at high-noise timesteps. It would be helpful to investigate whether increasing the KL penalty at high-noise timesteps could achieve similar results.

Questions for authors

Please refer to the two concerns raised in the Weaknesses section, particularly regarding the novelty compared to UniRL-Zero and the effectiveness of stronger KL regularization at high-noise timesteps.

I am open to raising my score if the authors can adequately address these concerns during the rebuttal.

Reviewer_a4mVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniGRPO, a unified reinforcement learning framework for reasoning-driven visual generation. Its main contribution is to formulate the Prompt => Thinking => Image generation process as a single Markov Decision Process and jointly optimize autoregressive text reasoning and flow-based image synthesis. Specifically, UniGRPO integrates standard GRPO for language generation with FlowGRPO for image generation, dusing share group-relative advantages computed from sparse terminal rewards on the final generated images. This enables both components to be optimized within a unified training loop rather than independently.

To improve training efficiency and stability, the authors introduce two practical modifications to FlowGRPO: (1) removing classifier-free guidance (CFG) during training to simplify rollouts and reduce computational overhead, and (2) replacing noise-weighted latent KL regularization with velocity-field MSE regularization to mitigate reward hacking. The framework is evaluated on the Bagel model, achieving 0.8381 on Text Alignment and 0.90 on GenEval, outperforming the evaluated ReFL, FPO, FlowGRPO, and TextGRPO baselines. Ablation studies further support the effectiveness of the two training modifications. Overall, the paper's primary contribution is a unified training recipe for jointly optimizing reasoning and image generation, rather than a fundamentally new reinforcement learning algorithm.

Critical strengths

A good problem to study. Jointly optimizing reasoning and visual synthesis is an important direction for unified multimodal models. Formulating discrete language generation and continuous image denoising within a single MDP provides a coherent framework for optimizing both modalities using shared terminal rewards.

Simple and practical training design. UniGRPO combines established GRPO and FlowGRPO methods without introducing excessive algorithmic complexity. The CFG-free training strategy simplifies rollouts, while velocity-field MSE regularization addresses instability and reward hacking in flow-based RL.

The authors show UniGRPO achieves good performance. Jointly optimizing the reasoning and image-generation components can outperform optimizing either component independently.

Critical weaknesses

The core framework combines existing TextGRPO and FlowGRPO objectives using shared advantages. While this integration is useful, the methodological contribution seems a bit incremental.

The paper emphasizes scalability to multi-round interleaved generation and multi-condition tasks, but evaluates only single-round reasoning-driven image generation.

Although joint optimization improves performance, it is unclear whether the gains arise from improved reasoning, better prompt expansion, or additional optimization of the image-generation component. Stronger prompt-rewriting baselines or reasoning-specific ablations would help clarify this distinction.

Questions for authors

What is the key methodological distinction between UniGRPO and directly combining TextGRPO and FlowGRPO with shared terminal rewards?

Could you compare against strong prompt-rewriting or prompt-expansion baselines to establish whether the improvements specifically require RL-optimized reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.