acceptodds
Withdrawn from ICLR 2027

When Co-Training Becomes Redundant: Update Geometry in Unified Multimodal Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal models (UMMs) combine multimodal understanding and generation in one backbone through separate experts. In image-generation reinforcement learning (RL), the reward signal backpropagates through both experts, so it can update the understanding expert as well as the generation expert, yet the role of the former's updates remains unclear as prior work typically keeps it frozen. In this work, we show that the understanding expert's contribution is front-loaded. Two velocity-space probes including compensability (measuring whether its updates carry content the generator cannot locally reproduce) and transport (measuring how strongly its updates align with directions the generator realizes efficiently), reveal that the early updates of the understanding expert are both distinct and efficient, while both signals decay as training proceeds. This finding is also mirrored at the reward level, where later understanding training steps add no measurable gain. These observations motivate a staged protocol: a geometry-based freeze rule that reads the two probes relative to their running peak, and freezes the understanding expert at a predicted step. Across two backbones, three reward models, and three RL algorithms, staged training consistently outperforms naive joint training, improving GenEval2 by up to while reaching higher rewards in substantially fewer training hours. Our results suggest the understanding expert's updates are best treated as a trajectory-dependent, front-loaded resource rather than a fixed co-training partner.

Primary area:
generative models
Keywords:
Image Generation, Diffusion Models, Unified Understanding and Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XPehOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when co-training the understanding and generation experts becomes redundant in unified multimodal RL. It shows that the understanding expert is mainly useful in the early training stage and proposes a staged strategy that freezes it once its contribution declines. The analysis is interesting and the method improves both efficiency and generation performance, but the evaluation is limited to MoT-style architectures and the freeze rule still depends on extra probes and calibrated thresholds. Overall, I find the main idea and observations interesting, and I would consider a higher score if the rebuttal can address these concerns well.

Critical strengths
  1. The paper reveals that the benefit of jointly updating the understanding expert is strongly stage-dependent, and identifies a clear “front-loaded” contribution pattern during RL training.

  2. The compensability and transport probes provide an interpretable way to characterize when co-training becomes redundant, and the staged strategy is validated across multiple backbones, reward models, and RL algorithms.

Critical weaknesses
  1. Both evaluated backbones use MoT-style architectures with explicitly separated understanding and generation experts, so it is unclear whether the same redundancy pattern and freeze strategy hold for more tightly shared unified multimodal models.

  2. Compensability and transport require additional velocity/Jacobian-based measurements, while the thresholds are calibrated on BAGEL before being transferred to SenseNova-U1. The practical overhead and robustness across substantially different models remain insufficiently explored.

Questions for authors
  1. How well does the proposed freeze criterion generalize to unified multimodal models without explicitly separated understanding and generation experts?

  2. What is the additional computational overhead of the compensability and transport probes during training, and is the resulting efficiency gain still significant after including this cost?

  3. How sensitive is the method to the manually calibrated freeze thresholds, and would the same thresholds remain effective across larger model scales, different tasks, or substantially different RL settings?

Reviewer_1jisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the contribution of the understanding expert during image-generation RL training. Through several motivating experiments, the authors find that the understanding expert should be frozen at some point during training. Building on this observation, the authors define two probes, compensability and transport, and use them to establish a criterion for freezing the UND expert. Finally, a series of experiments validates the effectiveness of the proposed approach.

Critical strengths

The problem studied in this paper is of practical relevance. Its approach of examining the independent contribution of the UND expert is also worth further exploring.

Critical weaknesses
  1. The two probes are significant intermediate quantities supporting the paper’s central claim. However, their computational details are not presented sufficiently, and the resulting values appear to exhibit noticeable variability, which weakens the credibility of the conclusions.

  2. The final RL training strategy is not fully consistent with the preceding analysis. Please see Question 3 for details.

Questions for authors
  1. The compensability probe lacks several important robustness evaluations, such as the choice of the timestep tt, the value of mm (I would like to know whether γ\gamma converges as mm increases.), the stability of γexcess\gamma_{\mathrm{excess}} across different Δvrand\Delta v_{rand}, and the scale of ε\varepsilon. These evaluations would clarify whether the observed temporal trend really reflects the underlying training dynamics.

  2. Figure 5 shows that jointly increasing both αG\alpha_G and αU\alpha_U often leads to a higher reward. Does this indicate that the selected checkpoint has not fully converged? The conclusion would be more convincing if the authors varied αU\alpha_U and αG\alpha_G while keeping the total update magnitude fixed, for example by constraining αU2∥ΔθU∥2+αG2∥ΔθG∥2\alpha_U^2\|\Delta\theta_U\|^2+\alpha_G^2\|\Delta\theta_G\|^2 to be constant. Results from later or demonstrably converged checkpoints would also be informative.

  3. According to the definitions in the paper, both γ\gamma and τ\tau are invariant to a constant rescaling of Δv\Delta v. Consider a case in which every optimization step produces the same velocity-change direction and JGJ_G remains approximately unchanged. The raw γexcess\gamma_{\mathrm{excess}} and τ/τrand\tau/\tau_{\mathrm{rand}} would then remain constant in expectation across probe windows, indicating that UND’s geometric contribution has not decayed. However, since Equation (3) additionally divides these quantities by LkL_k, the acquisition ratios decrease when the probe windows become longer. Consequently, UND may be frozen simply because the window changes from 20 to 40 or 100 steps. This appears inconsistent with the paper’s central claim that freezing is triggered by a genuine decline in UND’s contribution. It would strengthen the method if the authors could provide additional corresponding justification.

  4. The method probes the fixed absolute checkpoints 20,40,60,100, and 200. However, the point at which UND’s contribution starts to diminish is likely to depend on many factors such as the learning rate, RL algorithm, and the backbone model. The same number of optimizer steps may represent different amounts of optimization progress, which limits the scalability of the proposed schedule to broader training scenarios. A more adaptive probe schedule—for example, based on cumulative parameter change—could improve the general applicability of the method.

  5. Appendix A reports a maximum gradient norm of 1.0, but it is unclear whether gradient clipping is applied globally. If so, the joint setting is more likely to exceed the clipping threshold, which would mechanically scale down GEN’s gradient compared to the frozen-UND setting. This could partially explain the smaller GEN update norm reported in the middle panel of Figure 2. So it would be helpful to give more relative explanation.

  6. ΔvU\Delta v_U represents a change accumulated across multiple optimization steps, whereas JGJ_G describes the local sensitivity to GEN parameters at a single checkpoint. Given this mismatch in temporal scope, it is unclear whether combining these two quantities in the probe calculation provides a valid measure.

  7. The transport probe is defined using JG⊤ΔvJ_G^\top\Delta v. However, if we let A′=cAA'=cA and B′=B/cB'=B/c, the model’s forward output will remain unchanged but the Jacobian blocks may have different values. Is τ\tau invariant under this transformation? If not, it would be helpful for the authors to clarify whether any scale normalization needs to be applied to AA and BB.

Reviewer_oAFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to allocate updates between the understanding and generation experts in unified multimodal models (UMMs) during image-generation reinforcement learning. The authors observe that updates to the understanding expert are useful early in training but become increasingly redundant later. They quantify this behavior with two velocity-space probes, compensability and transport, and use their peak-relative decay to decide when to freeze the understanding expert. The proposed staged training first updates both experts and then updates only the generation expert after the probe-based freeze criterion is triggered. Experiments on BAGEL and SenseNova-U1, with GenEval, GenEval2, and HPSv3 rewards and DiffusionNFT, Flow-GRPO, and AWM, show improvements over generation-only and naive joint training, including a reported 11.1% relative gain on GenEval2GM.

Critical strengths
  1. Reasonable motivation
    Post-training UMMs for generation involves an important choice: freezing the understanding branch, jointly training both branches throughout training, or using different strategies at different stages. Existing work has explored freezing or joint optimization, but has less systematically studied how the marginal value of understanding-expert updates changes over training.

  2. Interpretable analysis perspective is provided
    Rather than reporting only an empirical freeze step, the paper attempts to explain why understanding-expert updates are useful early and redundant later through the local geometry of the generation velocity field. Compensability and transport correspond to two distinct questions: whether the generation expert can reproduce the understanding expert's change, and whether it can realize that change efficiently. This conceptual distinction is clear.

  3. Easy to integrate into existing RL pipelines
    The method does not change the reward, model architecture, or RL objective. It adds probes and a freeze decision, and can therefore in principle be combined with different flow-matching RL algorithms.

  4. Broader experimental coverage
    The paper covers two backbones, three reward models, and three RL algorithms, and includes generation-only, naive joint, and staged training.

Critical weaknesses

1. Potential evaluation leakage and reward over-optimization

The evaluation is tightly coupled to the reward used for training. For example, GenEval is used as the reward and GenEval is also reported as the evaluation metric; similarly, GenEval2 and HPSv3 are used as rewards and then evaluated with their corresponding metrics. This creates a potential reward-model over-optimization or benchmark-leakage risk, and makes the conclusions less solid. The paper should report the main and comparative experiments on additional independent benchmarks, such as TIIF-Bench, WISE, DPG-Bench, OneIG-Bench, and CVTG-2K, to provide stronger evidence that the method improves general image generation quality.

2. The source and distribution of the training data are not specified

3. The comparison experiments are not sufficiently complete

The current baselines are mainly generation-only, naive joint training, and one fixed learning-rate decay schedule for the understanding expert. The following comparisons are still missing:

  • Systematic comparisons with fixed freeze steps, such as 20, 40, 60, 100, and 200, across all settings;
  • Generation-only, naive joint, and staged results with different LoRA ranks, such as rank 128 and rank 256, as well as a full-parameter fine-tuning setting;
  • A more direct update-budget baseline, such as fixing the total LoRA update norm or the total number of trainable parameters.

4.The causal link between the geometric signals and redundancy is not sufficiently established

The paper interprets the decay of two local velocity-space signals as evidence that understanding-expert updates have become redundant, but the current evidence is mainly correlational. The local geometry of an update at a particular checkpoint, timestep, and set of probe inputs does not by itself imply that continuing to update the expert cannot improve future reward.

In particular, freezing the understanding expert changes the subsequent optimization trajectory. The paper uses checkpoint composition on the same joint-training trajectory to support the reward-level conclusion, but this does not directly verify that a probe-based freeze decision remains optimal on the intervened trajectory. A more convincing evaluation would report complete results for multiple candidate freeze steps in each setting and test whether the probe-predicted freeze step is close to an independently determined oracle freeze step.

5. The scope of the paper is narrow

Both backbones are based on MoT and use explicit understanding/generation experts. The conclusions are therefore more accurately scoped to MoT-like UMMs with explicit expert separation. The current evidence is insufficient for UMMs with different parameter-sharing schemes or visual representations, such as Janus/Janus-Pro, Show-o, Emu3, and Transfusion.

Questions for authors
  1. Regarding the compensability in Eq. (1), how do the authors justify the statement that “gamma_excess < 0 means that the understanding expert writes content that the generation expert cannot reproduce”? Since Q contains only 64 random directions and provides only a lower-bound approximation to the Jacobian range, what is the precise scope of this interpretation?

  2. Have the authors compared freezing triggered only by compensability with freezing triggered only by transport? How much does each condition contribute to the final gains?

  3. The paper only considers text-to-image generation. Does staged training lead to similar conclusions for image-editing tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.