- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes UniGRPO, a reinforcement learning framework that jointly optimizes textual reasoning and flow-based visual generation within unified multimodal models. Specifically, the authors formulate the sequential prompt–reasoning–image generation process as a unified Markov Decision Process, coupling TextGRPO and FlowGRPO via shared group-relative advantages computed from final visual rewards. To stabilize and streamline visual policy optimization, the training recipe incorporates two key modifications: removing classifier-free guidance during RL exploration to simplify rollouts, and replacing the latent KL divergence penalty with a mean squared error regularizer between the policy and reference velocity fields to prevent reward hacking. The proposed method is implemented on the BAGEL architecture following supervised fine-tuning on an internal dataset. Experimental evaluations on an internal text alignment benchmark and GenEval demonstrate performance gains over text-only, image-only, and hybrid baselines, achieving scores of 0.8381 and 0.90, respectively. Ablation studies are provided to validate the necessity of CFG removal and the velocity regularization.
Critical strengths
- The proposed UniGRPO integrates TextGRPO and FlowGRPO into a single reinforcement learning loop, enabling textual reasoning and image generation to be optimized using shared advantages derived from final-image rewards. This provides a coherent approach to adapting both stages toward the same downstream objective. The improvements over text-only and image-only optimization in Table 1 provide empirical support for the complete training recipe, making it a useful baseline for reasoning-driven visual generation.
- The CFG ablation in Figure 4 shows that training without CFG achieves comparable or better evaluation performance when CFG is applied at evaluation in both settings. Although this experiment uses the original BAGEL at 512 resolution, it provides useful evidence that the additional unconditional model evaluations can be omitted from training rollouts without an apparent performance penalty in the tested setting. This is a practical simplification with potential computational benefits, although end-to-end training speedups are not quantified.
Critical weaknesses
- The paper includes a two-stage baseline that first optimizes image generation with ReFL and then optimizes text generation with TextGRPO. However, neither the main experiments nor the appendix reports a matched sequential or alternating TextGRPO–FlowGRPO baseline. Consequently, the existing comparison changes both the image-side optimization objective and the training schedule, making it difficult to attribute the gains specifically to simultaneous joint optimization. A controlled comparison using the same initialization, reward, component objectives, rollout settings, and matched training budget would more convincingly establish the value of the proposed joint training strategy.
- Figure 5 provides encouraging evidence for the selected MSE configuration, but the KL run is terminated after grid-like artifacts appear at approximately 250 steps. Stopping a visibly degraded run is reasonable. However, the failure of this configuration does not establish that MSE is preferable to appropriately tuned KL regularization. In particular, the observation that KL substantially suppresses training reward does not demonstrate that its coefficient is well chosen, and no coefficient-sensitivity analysis is reported. Morever, the explanation that weaker KL constraints at high-noise timesteps create opportunities for reward hacking is not directly tested through timestep-resolved measurements or controlled changes to the regularization weights. The current results therefore leave unclear whether the improvement arises from removing timestep-dependent weighting, changing the effective regularization strength, or other optimization effects. This limits the evidence for both the claimed robustness and the proposed mechanism.
Questions for authors
- Could authors compare simultaneous joint optimization against sequential or alternating updates using the same TextGRPO and FlowGRPO objectives, initialization, reward, image-side modifications, and matched training budget? Does the advantage of UniGRPO persist when the component objectives and training resources are controlled?
- How were the KL and MSE coefficients selected, and does the advantage of velocity MSE persist after a reasonable coefficient sweep for both regularizers? Please also clarify the exact latent KL implementation and its timestep-dependent weighting. Is there any evidence, such as per-timestep velocity deviations or regularization-gradient measurements, linking the observed artifacts to weak constraints at high-noise timesteps?