acceptodds
Under review as a conference paper at ICLR 2027

Cross-Feature Guided Reinforcement Learning for Unified Multimodal Understanding and Generation

Abstract

Unified multimodal models (UMMs) integrate multimodal understanding and image generation, but joint reinforcement learning (RL) often suffers from optimization conflict because the two branches are optimized together with limited interaction. To address this issue, we propose **Cross-Feature Guided Reinforcement Learning (CGRL)**, which enables reciprocal guidance between the two branches during joint RL. Leveraging the semantic correspondence between paired images and captions, CGRL introduces cross-feature guidance by comparing each branch's source representation with the generated token hidden states of the opposite branch at the same backbone layer. The resulting guidance modulates the sequence-level advantage to obtain token-level advantages, allowing more precise policy updates. Experiments on Janus-Pro and Show-o, covering pure autoregressive and hybrid autoregressive–diffusion architectures, show consistent improvements over joint GRPO in multimodal understanding, image captioning, and text-to-image generation. Further gradient analysis reveals increased cross-branch gradient agreement. These results demonstrate that CGRL reduces the optimization conflict between understanding and generation and enables mutually beneficial joint RL in unified multimodal models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Multimodal Models, Unified Understanding and Generation, Reinforcement Learning, Cross-Feature Guidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wQFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cross-Feature Guided Reinforcement Learning (CGRL), a method for jointly improving multimodal understanding and generation in unified multimodal models. To address optimization conflicts and the lack of fine-grained supervision in joint GRPO, CGRL leverages paired image-text data to introduce cross-branch feature guidance. Specifically, it computes cosine similarities between generated-token hidden states and source representations from the opposite branch. These similarity scores are then used to reweight token-level policy updates during RL training. Experiments on Janus-Pro and Show-o demonstrate consistent improvements over joint GRPO across multimodal understanding, image captioning, and text-to-image generation benchmarks. Additional analyses suggest that CGRL improves gradient agreement between the two branches.

Critical strengths
  1. Simple and practical method. CGRL introduces a straightforward way to incorporate cross-branch feature guidance into token-level advantage weighting without requiring additional trainable modules. The method can be applied to both autoregressive and diffusion-based generation architectures.

  2. Consistent improvements across models and tasks. Experiments on Janus-Pro and Show-o show consistent improvements over joint GRPO across multimodal understanding, image captioning, and text-to-image generation. The ablation studies and gradient analysis further support the effectiveness of the proposed approach.

Critical weaknesses
  1. Limited performance gains. My main concern is that the improvements over joint GRPO are relatively small on most benchmarks. It would be helpful to report results from multiple evaluation runs to assess whether these gains are consistent and reliable.
  2. In Section 4.4 (Table 4), Cross outperforms Self on GenEval (76.9 vs. 74.6) and DPG-Bench (83.94 vs. 83.19). It is unclear whether the gains come from cross-branch interaction or simply more meaningful same-modality feature matching. Moreover, single-branch training with Cross guidance already achieves comparable results (e.g., 76.4 vs. 76.9 on GenEval), raising questions about the claimed mutual reinforcement between understanding and generation.
Questions for authors

Please see weakness.

2 comments
AuthorsOpenReview ↗

Response 1/2: Reliability and Consistency of Improvements

We sincerely thank Reviewer wQFh for recognizing CGRL's practical design and consistent improvements across models and tasks. To directly address your concern regarding performance consistency and the value of joint training over single-branch variants, we provide repeated-run statistics across three independent training seeds.

1. Standard deviations of the main results. The results in Tables 1–3 are averaged over three independent training runs. We provide the corresponding standard deviations below.

ModelMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑MMT-Bench ↑GenEval ↑DPG ↑
Janus-Pro-1.5B43.156.462.547.473.082.63
+ GRPO46.0±0.1956.7±0.2465.2±0.1047.9±0.0774.4±0.3283.03±0.22
+ CGRL46.9±0.1757.6±0.2166.1±0.1348.3±0.0576.9±0.2883.94±0.26
Janus-Pro-7B46.572.269.853.578.183.79
+ GRPO48.7±0.1471.9±0.1069.5±0.2454.1±0.0878.6±0.2784.41±0.11
+ CGRL49.3±0.2072.8±0.1270.7±0.1754.3±0.0780.3±0.3084.68±0.15
Show-o-1B29.341.547.537.967.378.40
+ GRPO30.2±0.1141.7±0.2048.2±0.2138.3±0.1268.6±0.2479.14±0.15
+ CGRL30.7±0.1542.5±0.1749.6±0.1438.8±0.1670.2±0.1879.65±0.13

Across all three backbones and all benchmarks, CGRL consistently outperforms joint GRPO with small run-to-run variance (typically ±0.05\pm 0.05 to ±0.30\pm 0.30), confirming that the improvements are statistically reliable even under a compact 22K training budget.

2. Joint CGRL vs. single-branch Cross (U+Cross and G+Cross). We also repeat the single-branch Cross baselines across three independent runs:

MethodMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑MMT-Bench ↑GenEval ↑DPG ↑
U+Cross46.9±0.1558.2±0.2566.0±0.0848.2±0.02––
G+Cross––––76.4±0.2283.67±0.32
Joint CGRL46.9±0.1757.6±0.2166.1±0.1348.3±0.0576.9±0.2883.94±0.26

While U+Cross and G+Cross act as single-task specialists (optimizing only one loss and leaving the other task unoptimized), Joint CGRL trains a single unified model that further improves generation over G+Cross (+0.5 GenEval, +0.27 DPG) while maintaining competitive understanding performance relative to U+Cross—and substantially outperforming standard Joint GRPO on both fronts.

3. Effectiveness under limited training data. Despite using only 22K training samples, CGRL achieves notable gains over GRPO, including +2.5 GenEval on Janus-Pro-1.5B, +1.7 GenEval on Janus-Pro-7B, and +1.4 CV-Bench-3D on Show-o-1B. Moreover, captioning-based training improves downstream visual understanding performance, demonstrating cross-task transferability.

Together, the repeated-run results and extensive ablations support the reliability and effectiveness of CGRL.

AuthorsOpenReview ↗

Response 2/2: Cross-Branch Interaction vs. Same-Modality Matching

To directly disentangle cross-branch internal interaction from same-modality feature matching, we isolate both factors through controlled ablations and examine their effects in joint optimization.

1. Joint optimization enables complementary benefits. Figure 1 shows that standard joint GRPO underperforms task-specific GRPO training, indicating optimization interference.

While U+Cross and G+Cross leverage representations from the opposite branch for token-level advantage modulation, they optimize only a single task loss—acting as task-specialized models that leave the other branch unoptimized. Their improvements demonstrate the effectiveness of Cross guidance even in single-task training.

Joint CGRL simultaneously optimizes both objectives within a single unified model and surpasses the corresponding task-specific GRPO baselines on both tasks. These results support the benefits of cross-branch guidance for joint understanding-generation optimization.

2. Additional ablations on same-modality matching. In Table 4, Self compares generated tokens against the conditioning input of the same branch, whereas Cross compares generated representations against paired reference representations from the opposite branch. To investigate whether CGRL's improvements arise from same-modality matching rather than cross-branch guidance, we introduce an alternative reward formulation and conduct controlled ablations.

Specifically, Cross-modal reward is our original SigLIP2 reward, SigLIP2(i,t)+SigLIP2(t,i)\mathrm{SigLIP2}(i,t)+\mathrm{SigLIP2}(t,i), measuring image-text alignment. Same-modality reward uses SigLIP2(i,i)+SigLIP2(t,t)\mathrm{SigLIP2}(i,i)+\mathrm{SigLIP2}(t,t), measuring generated-to-reference similarity within each modality.

RowMethodRewardGuidanceMMStar ↑GenEval ↑DPG ↑
1GRPOCross-modal–46.074.483.03
2GRPOSame-modality–45.774.682.73
3CGRLSame-modalityInternal LLM47.276.883.62
4CGRLSame-modalityExternal CLIP46.374.583.36
5CGRL (Ours)Cross-modalInternal LLM46.976.983.94
  • Same-modality supervision alone provides no consistent gains (row 2 vs. 1). Replacing the original cross-modal reward with a same-modality reward yields comparable performance.
  • Cross guidance provides additional benefits (row 3 vs. 2). Adding Cross guidance improves all three benchmarks (+1.5 MMStar, +2.2 GenEval, and +0.89 DPG), demonstrating its effectiveness beyond same-modality supervision.
  • Internal representations provide more effective guidance (row 3 vs. 4). Under the same reward formulation, internal LLM guidance consistently outperforms external CLIP guidance, highlighting the value of fine-grained interactions through internal representations.
  • Cross guidance is robust to reward formulations (row 5 vs. 3). The original CGRL achieves performance comparable to the same-modality reward variant, indicating that the benefits of Cross guidance do not strongly depend on the sequence-level reward formulation.

Together, these results demonstrate that CGRL's improvements cannot be explained solely by same-modality matching or the choice of sequence-level reward. Instead, they support the additional effectiveness of fine-grained cross-branch guidance through the model's internal representations.

Reviewer_TKQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Cross-Feature Guided Reinforcement Learning, CGRL, to address the optimization conflict in joint reinforcement learning of text-to-image generation and image caption tasks in unified multimodal models. Specifically, CGRL computes cosine similarities between the hidden states of generated tokens and the mean-pooled source representation of the opposite branch at the same backbone layer, so that caption tokens are compared with the input image and image tokens with the input caption. These similarities are normalized within each rollout and converted into bounded token-level guidance weights, which modulate the sequence-level GRPO advantage to obtain token-level advantages. Experiments on Janus-Pro and Show-o show consistent improvements over joint GRPO in multimodal understanding, image captioning, and text-to-image generation, together with increased cross-branch gradient agreement. These results suggest that CGRL reduces the optimization conflict between understanding and generation and enables mutually beneficial joint RL.

Critical strengths
  1. The paper is well motivated by the conflict between image understanding and generation during joint RL. Figure 1 shows that joint GRPO underperforms task-specific RL on both tasks.
  2. Using intermediate hidden-state similarities to reweight token-level advantages is an interesting design. The gradient analysis suggests that it helps align the optimization directions of the two tasks.
  3. Experiments on Janus-Pro and Show-o demonstrate consistent overall improvements over joint GRPO across understanding, captioning, and image generation benchmarks
Critical weaknesses
  1. The evaluation is limited to models with discrete image generation. Validation on stronger models with continuous flow matching-based generation is missing, such as BAGEL.
  2. The weighting scheme has a counterintuitive effect when the advantage is negative. The guidance weights are always positive, but the sequence-level advantage can be either positive or negative. Eq. 8 multiplies this advantage by the token-specific weight, which is larger for tokens more aligned with the reference. When the advantage is positive, these tokens receive stronger reinforcement. When it is negative, however, the same weighting makes their advantages more negative and can penalize them more strongly than less-aligned tokens, all else being equal and before clipping takes effect. The paper does not explain why this behavior is desirable or analyze its impact on training.
  3. Training is limited to captioning and text-to-image generation with paired image-caption data. More complex understanding tasks are not explored as training objectives, and why captioning-based RL improves downstream understanding needs further explanation.
Questions for authors
  1. Can CGRL be extended to continuous diffusion or flow-based models such as BAGEL?
  2. Does assigning larger weights to better-aligned tokens cause undesirable suppression when the advantage is negative? Have you analyzed this effect?
  3. Why does captioning-based RL improve downstream understanding? Can CGRL also support more complex training tasks such as visual question answering and reasoning?
3 comments
AuthorsOpenReview ↗

Response 1/3: Generalization to Continuous Image Generation

We sincerely thank Reviewer TKQa for recognizing CGRL's motivation, fine-grained guidance design, and consistent empirical improvements.

To validate CGRL on continuous generation models, we extend it to BAGEL-7B-MoT, referring to the continuous image generation RL implementation in R3 [1]. We use a learning rate of 5×10−65\times10^{-6}, 20 image sampling steps, and text/image CFG scales of 4/1.5.

ModelMMBench ↑MMVP ↑GenEval ↑DPG-Bench ↑
BAGEL-7B85.069.382.085.07
BAGEL-7B + GRPO85.669.183.985.53
BAGEL-7B + CGRL86.370.785.886.72

CGRL consistently outperforms GRPO on both understanding and generation benchmarks, including +1.9 on GenEval and +1.19 on DPG-Bench.

These results demonstrate that CGRL is applicable to continuous image generation models, extending its effectiveness beyond discrete-token generation architectures.

Reference

[1] Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models

AuthorsOpenReview ↗

Response 2/3: Effect of Negative-Advantage Reweighting

We further examine whether Cross guidance causes undesirable suppression of highly aligned tokens when the sequence-level advantage is negative. Specifically, we introduce a Positive-only Cross ablation that retains the original sequence-level advantages for negative-advantage rollouts without token-level reweighting, while applying Cross guidance only to positive-advantage rollouts.

MethodMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑GenEval ↑DPG ↑
CGRL46.957.666.176.983.94
Positive-only Cross46.757.965.777.183.83

The two variants achieve comparable performance across all benchmarks, with no consistent improvement from removing negative-advantage reweighting. Beyond empirical parity, two token-level factors explain why negative-advantage reweighting does not cause harmful suppression:

  1. Reduced weight contrast in low-reward rollouts: Negative-advantage rollouts generally exhibit uniformly lower token-to-reference similarities and smaller intra-sequence variance than positive rollouts. After rollout-level normalization, the resulting token weights remain closer to 1, yielding near-uniform sequence penalties rather than sharp token-level suppression.

  2. Preventing shortcut overfitting on flawed trajectories: When an entire rollout receives a negative reward, locally aligned tokens often reflect isolated keyword matching or partial visual attributes embedded within an overall incorrect or hallucinated structure. Down-weighting these trajectories slightly stronger at locally salient tokens prevents the policy from reinforcing shortcut associations on failed generations.

Together, both the mechanistic properties and the Positive-only Cross ablation confirm that negative-advantage reweighting does not adversely impact training dynamics or overall performance.

AuthorsOpenReview ↗

Response 3/3: Transfer from Captioning to VQA and Extension to Joint VQA–Generation Training

1. Why can captioning-based training benefit VQA?

Captioning and VQA are both vision-language understanding tasks that rely on shared visual-semantic capabilities, such as object recognition, attribute identification, and relationship understanding. Therefore, representations learned through captioning can transfer to downstream VQA tasks, consistent with observations in DIVA [1] and CapRL [2].

Importantly, CGRL leverages these shared representations through cross-branch guidance, enabling understanding and generation to benefit from jointly learned visual-semantic information.

2. Extension to joint VQA and text-to-image generation training.

To further validate CGRL beyond captioning, we conduct additional joint VQA and text-to-image generation training on Janus-Pro-1.5B using the same 22K dataset [3], which contains both VQA and captioning annotations. The VQA reward combines answer correctness and format rewards.

We adapt Cross guidance to VQA through three modifications:

  1. Question-conditioned representations: We concatenate the question and generated answer to capture answer semantics conditioned on the question.
  2. Multiple-choice answers: We replace predicted option letters (A/B/C/D) with their corresponding answer text to obtain more meaningful semantic representations.
  3. Caption-based guidance: We perform an additional forward pass on the paired caption to extract its hidden states, which guide the question-conditioned answer representations through similarity-based token weighting.

For the generation branch, generated image tokens are matched with the input image representations extracted from the understanding branch, preserving bidirectional Cross guidance.

3. Experimental results.

MethodMMMU ↑MathWe ↑MMVet ↑LogicVT ↑GenEval ↑DPG ↑
Janus-Pro-1.5B36.35.939.823.973.082.63
CoRL-S1 [4]40.314.042.530.275.983.30
GRPO39.414.543.229.874.882.75
CGRL44.718.746.635.876.883.86

CGRL outperforms both GRPO and CoRL-S1 across all evaluated benchmarks. Compared with GRPO, it achieves substantial improvements on general VQA tasks (MMMU: +5.3, MMVet: +3.4) and visual reasoning tasks (LogicVT: +6.0, MathWe: +4.2), alongside gains in image generation (GenEval: +2.0, DPG: +1.11).

These results provide direct evidence that CGRL is not limited to captioning-based training but also supports joint optimization of more complex visual understanding and image generation tasks, demonstrating its broader applicability across understanding objectives.

References

[1] DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement

[2] CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning

[3] https://huggingface.co/datasets/mm-vl/x2x_rft_22k

[4] Co-Reinforcement Learning for Unified Multimodal Understanding and Generation

Reviewer_S7aFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CGRL jointly post-trains image captioning and text-to-image generation in unified multimodal models. It sums SigLIP2 rewards, normalizes them within rollout groups, and scales the resulting advantages using positive weights derived from generated hidden states and paired source features from the opposite branch. Show-o uses transition-level weighting. Experiments on Janus-Pro 1.5B/7B and Show-o 1.3B show broad improvements over joint GRPO. The evidence supports a useful reference-guided weighting method; its distinctive reciprocal-learning and conflict-reduction explanation remains insufficiently isolated.

Critical strengths
  1. Broad and relevant empirical support. Tables 1–3 cover three checkpoints and two architectures, with improvements in 19/20 understanding comparisons. Janus-Pro 7B improves MMMU from 40.6 to 45.3; Janus-Pro 1.5B improves GenEval from 74.4% to 76.9%. Transfer to VQA and NoCaps gives evidence beyond the training reward.
  2. Simple intervention with substantive controls. The method adds no learned guidance network. Table 4 compares single-task and joint RL, separate and summed rewards, single-branch Cross, guidance sources, and layers. Appendix A.7 adds KL, reward, and strength ablations. These establish practical usefulness and should receive meaningful credit.
Critical weaknesses
  1. The distinctive contribution is not adequately separated from existing semantic token weighting. GCPO [1], publicly available in May 2026, already multiplies group-relative advantages by semantically informed token weights and evaluates Janus-Pro image generation. CGRL differs in its opposite-branch source features and joint training, but lacks a matched alternative weighting baseline. Moreover, Cross in Eq. (5) compares generated text with a paired caption and generated visual tokens with a paired image; Self instead uses each branch’s conditioning input. Thus, Table 4 changes both branch origin and same-modality versus cross-modality reference matching. It supports this source choice but does not isolate a benefit from evolving reciprocal representations. A frozen pretrained reference-feature control with the same paired information would directly test that interpretation. GvU [2] also merits discussion, while distinguishing its aggregated intrinsic reward from CGRL’s per-token modulation. Neither prior establishes that CGRL is a duplicate.

  2. The evidence for conflict reduction and mutual reinforcement is weaker than the main claims. The single-branch Cross controls are valuable, but full joint Cross adds only 0.5 percentage points on GenEval and 0.27 on DPG; MMStar is unchanged and CV-Bench 2D decreases from 58.2 to 57.6 (Table 4). No repeated-run uncertainty is reported. Figure 3 and Appendix A.4 count coordinates with matching gradient signs and comparable magnitudes. This statistic ignores gradient energy and depends on relative task scaling: more qualifying coordinates need not imply a more favorable gradient inner product. The gradient diagnostic is supporting correlation, not sufficient evidence that destructive interference has decreased. Together, these limitations leave the central mechanistic contribution less established than the overall improvement over vanilla joint GRPO.

Questions for authors
  1. Contribution test. Under the same data, SigLIP2 reward, joint update budget, and comparable weight scale, how does CGRL compare with a GCPO-style semantic weighting alternative and with frozen pretrained paired-reference features? These targeted comparisons would most directly affect the novelty assessment.
  2. Mutual-benefit evidence. Are the Table 4 gains over U+Cross and G+Cross stable across training seeds? Please state how task exposure and optimization budgets are matched. Paired evaluation intervals would help separate evaluation variation from training variation.
  3. Mechanism clarification. Which checkpoints, minibatches, and weighted or unweighted losses produced Figure 3? Do gradient cosine or conflict-frequency measurements on matched, unweighted objectives agree? Eqs. (7)–(8) also penalize highly aligned tokens more on negative-advantage rollouts; what validates the claimed finer credit assignment in that case?
5 comments
AuthorsOpenReview ↗

Response 1/3: Distinction from Existing Semantic Token Weighting

We sincerely thank Reviewer S7aF for recognizing CGRL's broad empirical improvements, practical design, and extensive ablation studies. We provide additional controlled comparisons to clarify its distinctive contribution.

1. CGRL vs. GCPO and GvU. GCPO [1] derives token importance from the divergence between predictions under different conditions, whereas CGRL leverages paired image and caption representations from the opposite task branch for fine-grained token-level advantage modulation.

GvU [2] constructs understanding-driven intrinsic semantic rewards to guide visual generation. In contrast, CGRL retains SigLIP2 sequence-level rewards and uses bidirectional internal representations to modulate token-level advantages while jointly optimizing understanding and generation.

Thus, CGRL differs in its guidance source, the role of internal semantic information, and its joint optimization mechanism.

2. Direct comparison with GCPO-style weighting. We integrate GCPO-style weighting into the same joint training framework, following its official implementation. Token importance is computed from the KL divergence between predictions under positive and negative conditions, followed by rank-based transformation and normalization to a sequence mean of 1.

The training data, SigLIP2 rewards, rollout settings, and optimization budget are identical. CGRL also uses weights centered around 1, providing comparable average weighting scales.

MethodMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑GenEval ↑DPG ↑
Joint GRPO46.056.765.274.483.03
Joint GRPO + GCPO Weighting45.757.265.575.383.37
CGRL46.957.666.176.983.94

CGRL outperforms GCPO-style weighting across all five benchmarks, including +1.2 on MMStar and +1.6 on GenEval. These results support the additional effectiveness of cross-branch internal guidance over condition-sensitive semantic token weighting in joint optimization.

3. Cross-branch guidance vs. same-modality matching. To examine whether CGRL's gains arise from same-modality matching, we introduce an alternative reward. Cross-modal reward is our original SigLIP2 reward, SigLIP2(i,t)+SigLIP2(t,i)\mathrm{SigLIP2}(i,t)+\mathrm{SigLIP2}(t,i), measuring image-text alignment. Same-modality reward uses SigLIP2(i,i)+SigLIP2(t,t)\mathrm{SigLIP2}(i,i)+\mathrm{SigLIP2}(t,t), measuring generated-to-reference similarity within each modality.

RowMethodRewardGuidanceMMStar ↑GenEval ↑DPG ↑
1GRPOCross-modal–46.074.483.03
2GRPOSame-modality–45.774.682.73
3CGRLSame-modalityInternal LLM47.276.883.62
4CGRLSame-modalityExternal CLIP46.374.583.36
5CGRL (Ours)Cross-modalInternal LLM46.976.983.94
  • Row 2 vs. 1: Same-modality rewards alone provide no consistent gains.
  • Row 3 vs. 2: Adding Cross guidance improves all three benchmarks (+1.5 MMStar, +2.2 GenEval, +0.89 DPG), demonstrating benefits beyond same-modality supervision.
  • Row 3 vs. 4: Internal LLM guidance consistently outperforms external CLIP guidance under the same reward.
  • Row 5 vs. 3: Both reward formulations achieve comparable performance with internal Cross guidance, indicating robustness to the sequence-level reward formulation.

Together, these results support that CGRL's improvements stem primarily from fine-grained internal cross-branch guidance rather than merely same-modality matching.

4. Dynamic vs. frozen reference representations. In the original CGRL, Cross guidance is computed using internal representations from the model being jointly optimized, allowing the guidance features to evolve during training. To investigate the benefits of this dynamic interaction, we introduce two additional controls: (1) Frozen Pretrained LLM, which uses a frozen Janus-Pro-1.5B to extract paired image and caption reference features, while generated token representations remain from the trainable model; and (2) External Frozen CLIP, which uses external CLIP features for similarity-based guidance.

Guidance SourceMMStar ↑GenEval ↑DPG ↑
Frozen Pretrained LLM45.875.383.47
External Frozen CLIP46.374.583.36
Dynamic LLM (CGRL)46.976.983.94

Dynamic CGRL outperforms both frozen LLM and external CLIP guidance, highlighting the benefits of leveraging dynamically evolving internal MLLM representations to facilitate mutual guidance and shared semantic learning between understanding and generation.

References

[1] Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

[2] Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models

AuthorsOpenReview ↗

Response 2/3: Reliability and Mutual Benefits of Joint Optimization

We provide additional repeated-run results and training-budget details to evaluate the reliability and mutual benefits of joint optimization.

1. Benefits of joint optimization. Figure 1 shows that standard joint GRPO underperforms task-specific GRPO training due to optimization interference between understanding and generation. While U+Cross and G+Cross leverage opposite-branch representations for token-level modulation, they only optimize a single task objective—effectively acting as task-specialized models that leave the opposite task unoptimized. In contrast, Joint CGRL simultaneously optimizes both objectives within a single unified model, overcoming the interference observed in joint GRPO while matching or surpassing task-specialized baselines across both domains.

U+Cross and G+Cross leverage opposite-branch guidance but optimize only one task loss. Joint CGRL simultaneously optimizes both objectives and achieves additional improvements on several understanding and generation benchmarks.

2. Standard deviations of the main results. Tables 1–3 report mean scores over three independent training runs. We provide the corresponding standard deviations below.

ModelMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑MMT-Bench ↑GenEval ↑DPG ↑
Janus-Pro-1.5B43.156.462.547.473.082.63
+ GRPO46.0±0.1956.7±0.2465.2±0.1047.9±0.0774.4±0.3283.03±0.22
+ CGRL46.9±0.1757.6±0.2166.1±0.1348.3±0.0576.9±0.2883.94±0.26
Janus-Pro-7B46.572.269.853.578.183.79
+ GRPO48.7±0.1471.9±0.1069.5±0.2454.1±0.0878.6±0.2784.41±0.11
+ CGRL49.3±0.2072.8±0.1270.7±0.1754.3±0.0780.3±0.3084.68±0.15
Show-o-1B29.341.547.537.967.378.40
+ GRPO30.2±0.1141.7±0.2048.2±0.2138.3±0.1268.6±0.2479.14±0.15
+ CGRL30.7±0.1542.5±0.1749.6±0.1438.8±0.1670.2±0.1879.65±0.13

CGRL achieves higher mean scores than joint GRPO across all reported benchmarks, with relatively small standard deviations.

3. Stability relative to single-branch Cross. We further repeat U+Cross, G+Cross, and Joint CGRL across three independent training runs.

MethodMMStar ↑CV-Bench-2D ↑CV-Bench-3D ↑MMT-Bench ↑GenEval ↑DPG ↑
U+Cross46.9±0.1558.2±0.2566.0±0.0848.2±0.02––
G+Cross––––76.4±0.2283.67±0.32
Joint CGRL46.9±0.1757.6±0.2166.1±0.1348.3±0.0576.9±0.2883.94±0.26

These repeated-run results clarify the distinct value of Joint CGRL:

(1)Overcoming multi-task interference: Compared to the true joint-training baseline (Joint GRPO: 56.7 on CV-Bench-2D, 74.4 on GenEval), Joint CGRL achieves substantial gains across both understanding (+0.9 on CV-Bench-2D, +0.9 on MMStar) and generation (+2.5 on GenEval, +0.91 on DPG).

(2)Unified Pareto efficiency vs. single-task specialists: Unlike U+Cross and G+Cross—which require maintaining separate task-specific checkpoints and risk capability degradation on the unoptimized branch—Joint CGRL achieves stronger generation performance than G+Cross (GenEval: +0.5; DPG: +0.27) while preserving near-identical understanding performance to U+Cross within a single unified model.

4. Matched training settings. All configurations use the same 22K training pairs, 1,400 optimizer updates, a global batch size of 16, and identical rollout settings, as detailed in Appendix A.2 (Table 5).

Single-branch configurations optimize the corresponding task loss, while joint configurations optimize both objectives. Task exposure and optimizer update counts are controlled, while total computation differs between single-branch and joint training.

Joint GRPO and CGRL use identical two-task optimization settings, ensuring a direct comparison of their optimization strategies.

AuthorsOpenReview ↗

Response 3/3: Gradient diagnostics and negative-advantage weighting

We further investigate gradient compatibility and token-level credit assignment through additional analyses and ablations.

1. Gradient diagnostic settings. Figure 3 analyzes task gradients during RL training, initialized from the official Janus-Pro and Show-o checkpoints. All experiments use the same 22K training dataset, a global batch size of 16, and the minibatch configuration specified in Appendix A.2 (Table 5). Understanding and generation losses are equally weighted:

Ljoint=LU+LG.L_{\mathrm{joint}}=L_U+L_G.

We collect gradient statistics at two stages:

  • During training: We compute task gradients during RL optimization using each method's actual loss: the original GRPO loss for GRPO and Cross-weighted advantages for CGRL.
  • After training: We load the Janus-Pro-1.5B checkpoints after 1,400 updates and recompute task gradients with Cross reweighting disabled for both methods.

Both stages compute task gradients over the same shared parameter set.

2. Gradient cosine similarity and conflict frequency. To complement the coordinate-based statistics in Figure 3, we introduce two metrics that directly characterize overall gradient alignment. Gradient cosine similarity measures the agreement between task-gradient directions:

\frac{g_U^\top g_G}{\|g_U\|_2\|g_G\|_2}.$$ Conflict frequency measures the proportion of batches with $g_U^\top g_G<0$. | Method | Setting | Cross Reweighting | Gradient Cosine ↑ | Conflict Frequency ↓ | |:---|:---|:---:|:---:|:---:| | GRPO | During training | No | 0.22 | 0.19 | | CGRL | During training | Yes | **0.25** | **0.17** | | GRPO | After training | No | 0.24 | 0.16 | | CGRL | After training | No | **0.26** | **0.12** | CGRL achieves higher gradient cosine similarity and lower conflict frequency at both stages. Importantly, these improvements persist after training even without Cross reweighting, indicating that CGRL learns more compatible task-gradient directions rather than merely altering gradient statistics through token-level weights. Combined with the performance improvements, these results provide further evidence that CGRL helps mitigate optimization interference between understanding and generation. **3. Effect of negative-advantage reweighting.** We further examine whether Cross guidance causes undesirable suppression of highly aligned tokens when the sequence-level advantage is negative. Specifically, we introduce a **Positive-only Cross** ablation that retains the original sequence-level advantages for negative-advantage rollouts without token-level reweighting, while applying Cross guidance only to positive-advantage rollouts. | Method | MMStar ↑ | CV-Bench-2D ↑ | CV-Bench-3D ↑ | GenEval ↑ | DPG ↑ | |:---|:---:|:---:|:---:|:---:|:---:| | CGRL | **46.9** | 57.6 | **66.1** | 76.9 | **83.94** | | Positive-only Cross | 46.7 | **57.9** | 65.7 | **77.1** | 83.83 | The two variants achieve comparable performance across all benchmarks, with no consistent improvement from removing negative-advantage reweighting. Beyond empirical parity, two token-level factors explain why negative-advantage reweighting does not cause harmful suppression: 1. Reduced weight contrast in low-reward rollouts: Negative-advantage rollouts generally exhibit uniformly lower token-to-reference similarities and smaller intra-sequence variance than positive rollouts. After rollout-level normalization, the resulting token weights remain closer to 1, yielding near-uniform sequence penalties rather than sharp token-level suppression. 2. Preventing shortcut overfitting on flawed trajectories: When an entire rollout receives a negative reward, locally aligned tokens often reflect isolated keyword matching or partial visual attributes embedded within an overall incorrect or hallucinated structure. Down-weighting these trajectories slightly stronger at locally salient tokens prevents the policy from reinforcing shortcut associations on failed generations. Together, both the mechanistic properties and the Positive-only Cross ablation confirm that negative-advantage reweighting does not adversely impact training dynamics or overall performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: thank you to all reviewers

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We appreciate their recognition of CGRL's contributions and have conducted extensive additional experiments to further strengthen our work.

1. Key strengths recognized by the reviewers

  • Clear motivation and practical design (TKQa, wQFh). Both reviewers highlight that CGRL addresses optimization interference between understanding and generation through a simple and effective token-level guidance mechanism without introducing additional trainable modules.
  • Broad and consistent empirical improvements (S7aF, TKQa, wQFh). All three reviewers recognize CGRL's improvements across Janus-Pro and Show-o, covering multimodal understanding, captioning, and image generation benchmarks.
  • Extensive ablations and optimization analysis (S7aF, TKQa, wQFh). The reviewers acknowledge the value of our single-branch and joint training comparisons, guidance-source ablations, and gradient analyses.

2. Additional experiments addressing the reviewers' concerns

  • Reliability and mutual reinforcement (wQFh, S7aF). We report standard deviations from three independent runs across three backbones and further repeat U+Cross, G+Cross, and Joint CGRL. These results support the consistency of CGRL's improvements and clarify the additional benefits of joint optimization.

  • Distinctive contribution of cross-branch guidance (wQFh, S7aF). We introduce matched GCPO-style weighting, same-modality reward, external CLIP, and frozen pretrained LLM comparisons. CGRL outperforms GCPO-style weighting across five benchmarks, including +1.2 on MMStar and +1.6 on GenEval. Frozen-reference ablations further support the value of dynamically updated internal representations.

  • Generalization to broader architectures and tasks (TKQa). We extend CGRL to the continuous-generation BAGEL-7B-MoT model and joint VQA–text-to-image training, demonstrating improvements on continuous image generation, general VQA, and visual reasoning benchmarks.

  • Optimization mechanisms (S7aF, TKQa). We introduce gradient cosine similarity and conflict-frequency diagnostics. CGRL achieves higher gradient cosine similarity and lower conflict frequency both during training and after training without Cross reweighting. A Positive-only Cross ablation further demonstrates that negative-advantage reweighting has a limited impact on overall performance.

Together, these additional results strengthen the empirical evidence for CGRL's effectiveness, clarify its distinction from existing semantic token weighting methods, and demonstrate its generality across architectures and training objectives.

We thank the reviewers again for their valuable suggestions, which have helped us strengthen the evaluation and clarify the contributions of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.