- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Cross-Feature Guided Reinforcement Learning (CGRL), a method for jointly improving multimodal understanding and generation in unified multimodal models. To address optimization conflicts and the lack of fine-grained supervision in joint GRPO, CGRL leverages paired image-text data to introduce cross-branch feature guidance. Specifically, it computes cosine similarities between generated-token hidden states and source representations from the opposite branch. These similarity scores are then used to reweight token-level policy updates during RL training. Experiments on Janus-Pro and Show-o demonstrate consistent improvements over joint GRPO across multimodal understanding, image captioning, and text-to-image generation benchmarks. Additional analyses suggest that CGRL improves gradient agreement between the two branches.
Critical strengths
-
Simple and practical method. CGRL introduces a straightforward way to incorporate cross-branch feature guidance into token-level advantage weighting without requiring additional trainable modules. The method can be applied to both autoregressive and diffusion-based generation architectures.
-
Consistent improvements across models and tasks. Experiments on Janus-Pro and Show-o show consistent improvements over joint GRPO across multimodal understanding, image captioning, and text-to-image generation. The ablation studies and gradient analysis further support the effectiveness of the proposed approach.
Critical weaknesses
- Limited performance gains. My main concern is that the improvements over joint GRPO are relatively small on most benchmarks. It would be helpful to report results from multiple evaluation runs to assess whether these gains are consistent and reliable.
- In Section 4.4 (Table 4), Cross outperforms Self on GenEval (76.9 vs. 74.6) and DPG-Bench (83.94 vs. 83.19). It is unclear whether the gains come from cross-branch interaction or simply more meaningful same-modality feature matching. Moreover, single-branch training with Cross guidance already achieves comparable results (e.g., 76.4 vs. 76.9 on GenEval), raising questions about the claimed mutual reinforcement between understanding and generation.
Questions for authors
Please see weakness.
2 comments
Response 2/2: Cross-Branch Interaction vs. Same-Modality Matching
To directly disentangle cross-branch internal interaction from same-modality feature matching, we isolate both factors through controlled ablations and examine their effects in joint optimization.
1. Joint optimization enables complementary benefits. Figure 1 shows that standard joint GRPO underperforms task-specific GRPO training, indicating optimization interference.
While U+Cross and G+Cross leverage representations from the opposite branch for token-level advantage modulation, they optimize only a single task loss—acting as task-specialized models that leave the other branch unoptimized. Their improvements demonstrate the effectiveness of Cross guidance even in single-task training.
Joint CGRL simultaneously optimizes both objectives within a single unified model and surpasses the corresponding task-specific GRPO baselines on both tasks. These results support the benefits of cross-branch guidance for joint understanding-generation optimization.
2. Additional ablations on same-modality matching. In Table 4, Self compares generated tokens against the conditioning input of the same branch, whereas Cross compares generated representations against paired reference representations from the opposite branch. To investigate whether CGRL's improvements arise from same-modality matching rather than cross-branch guidance, we introduce an alternative reward formulation and conduct controlled ablations.
Specifically, Cross-modal reward is our original SigLIP2 reward, , measuring image-text alignment. Same-modality reward uses , measuring generated-to-reference similarity within each modality.
| Row | Method | Reward | Guidance | MMStar ↑ | GenEval ↑ | DPG ↑ |
|---|---|---|---|---|---|---|
| 1 | GRPO | Cross-modal | – | 46.0 | 74.4 | 83.03 |
| 2 | GRPO | Same-modality | – | 45.7 | 74.6 | 82.73 |
| 3 | CGRL | Same-modality | Internal LLM | 47.2 | 76.8 | 83.62 |
| 4 | CGRL | Same-modality | External CLIP | 46.3 | 74.5 | 83.36 |
| 5 | CGRL (Ours) | Cross-modal | Internal LLM | 46.9 | 76.9 | 83.94 |
- Same-modality supervision alone provides no consistent gains (row 2 vs. 1). Replacing the original cross-modal reward with a same-modality reward yields comparable performance.
- Cross guidance provides additional benefits (row 3 vs. 2). Adding Cross guidance improves all three benchmarks (+1.5 MMStar, +2.2 GenEval, and +0.89 DPG), demonstrating its effectiveness beyond same-modality supervision.
- Internal representations provide more effective guidance (row 3 vs. 4). Under the same reward formulation, internal LLM guidance consistently outperforms external CLIP guidance, highlighting the value of fine-grained interactions through internal representations.
- Cross guidance is robust to reward formulations (row 5 vs. 3). The original CGRL achieves performance comparable to the same-modality reward variant, indicating that the benefits of Cross guidance do not strongly depend on the sequence-level reward formulation.
Together, these results demonstrate that CGRL's improvements cannot be explained solely by same-modality matching or the choice of sequence-level reward. Instead, they support the additional effectiveness of fine-grained cross-branch guidance through the model's internal representations.
Response 1/2: Reliability and Consistency of Improvements
We sincerely thank Reviewer wQFh for recognizing CGRL's practical design and consistent improvements across models and tasks. To directly address your concern regarding performance consistency and the value of joint training over single-branch variants, we provide repeated-run statistics across three independent training seeds.
1. Standard deviations of the main results. The results in Tables 1–3 are averaged over three independent training runs. We provide the corresponding standard deviations below.
Across all three backbones and all benchmarks, CGRL consistently outperforms joint GRPO with small run-to-run variance (typically ±0.05 to ±0.30), confirming that the improvements are statistically reliable even under a compact 22K training budget.
2. Joint CGRL vs. single-branch Cross (U+Cross and G+Cross). We also repeat the single-branch Cross baselines across three independent runs:
While U+Cross and G+Cross act as single-task specialists (optimizing only one loss and leaving the other task unoptimized), Joint CGRL trains a single unified model that further improves generation over G+Cross (+0.5 GenEval, +0.27 DPG) while maintaining competitive understanding performance relative to U+Cross—and substantially outperforming standard Joint GRPO on both fronts.
3. Effectiveness under limited training data. Despite using only 22K training samples, CGRL achieves notable gains over GRPO, including +2.5 GenEval on Janus-Pro-1.5B, +1.7 GenEval on Janus-Pro-7B, and +1.4 CV-Bench-3D on Show-o-1B. Moreover, captioning-based training improves downstream visual understanding performance, demonstrating cross-task transferability.
Together, the repeated-run results and extensive ablations support the reliability and effectiveness of CGRL.