acceptodds
Under review as a conference paper at ICLR 2027

Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) combine image generation and visual understanding in a shared backbone. Since generation and understanding are inverse tasks, recent studies self-train UMMs by letting the two branches cooperatively supervise each other. We introduce MATE (Mutually Adversarial self-Training with Evolving data), a reinforcement-learning-based post-training framework in which the two branches instead challenge each other, and the challenges evolve as the model trains. MATE lets generation and understanding take turns to be challenger and solver. Given an image, the understanding branch proposes several candidate descriptions that the generation branch must turn back into similar images, and vice versa. The candidates are screened for consistency with the image or prompt they were proposed from, and the solver is trained on the candidate it handles worst. The adversary thus comes from the model's own outputs, and no separate adversary is trained. Moreover, the candidates that defeat one branch become the sources of the next challenges to the other in the next epoch, which keeps the challenges evolving with the model and turns the training into self-play in data space. On Janus-Pro-1B, MATE improves GenEval by 2.4 points, DPG-Bench by 1.7 points, and the average over nine understanding benchmarks by 0.7 points, while strengthening consistency across repeated image-text cycles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
unified multimodal models, multimodal post-training, self-play

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6kQXOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MATE, a post-training method for unified multimodal models. The generation and understanding branches take turns producing inputs for each other. After filtering candidates for consistency with the source, the method selects the candidate that the other branch performs worst on and updates that branch with GRPO. The selected candidates then become the source inputs for the next epoch. On Janus-Pro-1B, MATE improves GenEval by 2.4 points, DPG-Bench by 1.7 points, and the average over nine understanding benchmarks by 0.7 points. The paper also evaluates consistency over repeated image-text cycles.

Critical strengths
  1. The method is straightforward and makes good use of having generation and understanding in the same model. Selecting valid examples that the other branch struggles with is a reasonable way to focus training, and passing these examples into the next epoch gives a concrete mechanism for adapting the training data. The procedure is clearly described and does not require training a separate adversary.

  2. The results show improvements in both generation and understanding, rather than improving one at the expense of the other. MATE outperforms the reimplemented UniRL and SUDER baselines on both generation benchmarks and the understanding average. I also appreciate that the paper explicitly acknowledges the small ablation differences and the overlap between the consistency reward and the round-trip evaluation.

Critical weaknesses

lack of computational comparison. MATE samples four candidates and up to eight rollouts per candidate, and training takes about 29 hours on six B200 GPUs. Sharing the optimizer, dataset, and batch size does not establish that the baselines have comparable sampling or training budgets. Without a compute-matched comparison, it is hard to judge whether MATE is a better use of resources than spending the same budget on a simpler post-training method. This matters particularly because the gains are relatively modest and the main results are limited to one base model.

Questions for authors

Could author report the rollout counts, reward-model calls, optimizer updates, and GPU hours for MATE and the main baselines? A comparison under a matched total compute budget would materially affect my assessment.

2 comments

Removal of the reward model. In the submitted version, we used a frozen external reward model to provide consistent scoring for stable training. To assess whether MATE’s gains depend on external supervision, we additionally trained it using its own cross-branch likelihoods as rewards, without an external reward model, and adopted this version as our main model in the revised paper. This changes the reward source while preserving MATE’s core mutual-challenge and data-evolution mechanisms.

W1&Q1

UniRL already uses more resources and SUDER does not gain with more compute. Our updated MATE uses rewards from the model itself without an external reward model. All three methods train Janus-Pro-1B for 1,040 steps on six B200 GPUs, with their computational costs reported below.

UniRL matches MATE’s sampled-sequence count and optimizer updates but consumes more GPU-hours because of its data construction recipe and performs worse on both generation and understanding averages.

SUDER uses four rollouts per prompt without candidate selection, resulting in lower computational cost. Matching MATE’s GPU-hours would require approximately 1.8× as much training; however, SUDER’s generation and understanding averages change by less than 0.1 points over the final third of training, suggesting limited benefit from extending the run.

We acknowledge that the methods’ different sampling procedures make it difficult to match training steps, rollout counts, and GPU-hours simultaneously.

MethodRollout counts / stepReward-model callsOptimizer updatesGPU-hours
MATE1,72801,040140.6
UniRL1,72801,040462.4
SUDER19201,04079.2

Thank you for your reply, that solved my concerns.

Reviewer_miRiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MATE, a self-training framework for unified multimodal models that enables image generation and visual understanding to challenge each other. The method selects difficult but valid samples generated by one branch to train the other using GRPO. These selected samples are further used to update the training data across epochs.

Critical strengths
  1. The idea of mutual hard example mining between image generation and visual understanding is intuitive and reasonable. It provides a simple way to construct challenging training samples without an additional adversarial model.
  2. The evolving data mechanism allows the training inputs to adapt as the model improves, rather than relying entirely on a fixed dataset. The overall framework is simple and can be incorporated into existing unified multimodal models.
Critical weaknesses
  1. Although the overall idea is intuitive, selecting the hardest valid samples does not necessarily lead to the most effective learning. The paper provides limited analysis of why this strategy is preferable to other difficulty-based sampling methods.
  2. MATE is closely related to hard example mining, adversarial training, and self-play. More targeted comparisons are needed to better establish the advantages of the proposed mutual challenge and data evolution mechanisms.
  3. Sharing a backbone and alternately training the two branches does not necessarily imply that they benefit each other. The paper lacks direct analysis of how updates to one branch affect the other.
Questions for authors
  1. Why does MATE always select the lowest-reward valid candidate? Could the authors compare this strategy with random sampling or moderately difficult sample selection to better justify the proposed hard example mining mechanism?
  2. Could the authors provide a more direct analysis of mutual improvement between generation and understanding, for example by comparing one-way training with alternating bidirectional training?
  3. Could MATE’s gains primarily come from exposure to more diverse training inputs? In the “w/o evolving pools” ablation, the initial input pools are kept fixed, but the selected challenges may still vary across epochs.
4 comments

Removal of the reward model. In the submitted version, we used a frozen external reward model to provide consistent scoring for stable training. To assess whether MATE’s gains depend on external supervision, we additionally trained it using its own cross-branch likelihoods as rewards, without an external reward model, and adopted this version as our main model in the revised paper. This changes the reward source while preserving MATE’s core mutual-challenge and data-evolution mechanisms.

W1

Difference from existing hardest sampling methods. We thank the reviewer for this comment. Existing methods like Kimi k1.5 increases the sampling probability of problems with lower success rates (Kimi Team, 2025), and ALBEF samples semantically similar but mismatched image–text pairs within a mini-batch (Li et al., 2021). These methods mine challenging examples from an existing training pool, but in MATE, the opposite branch generates four candidates for each source and then selects the admitted candidate with the lowest mean reward. Selection therefore targets difficulties among newly generated alternatives for each source. Furthermore, the selected challenges become sources for the next epoch, allowing the training inputs themselves to evolve. This process operates in both directions between image generation and visual understanding, with each branch taking turns as challenger and solver and both capabilities being trained.

W2

Mutual challenge and evolving data. While existing works like OHEM (Shrivastava et al., 2016) mines high-loss examples from existing images, MATE lets generation and understanding challenge each other without a separate adversary. Selected challenges become the next epoch’s sources, enabling self-play through evolving data. The ablations from Fig. 6(b) suggest that both challenge selection and data evolution contribute to generation gains, while understanding remains comparable.

MethodGen. Avg. ↑Und. Avg. ↑
Janus-Pro-1B78.356.9
MATE80.357.6
MATE w/o evolving pools79.957.7
MATE w/o adversarial training79.557.8

W3&Q2

Effects of branch-specific training. Our updated MATE uses rewards from the model itself without an external reward model. In the additional experiments below, training either branch alone performs worse than MATE on both GenEval and understanding. Single-branch training is also less efficient for jointly improving both capabilities, since it directly optimizes only one of the two tasks. The following results are from our additional experiments during rebuttal.

MethodGenEval ↑Und. Avg. ↑
Janus-Pro-1B74.456.9
MATE77.457.8
Generation branch only76.257.3
Understanding branch only71.556.4

Q1

Hardest-only versus all-candidate training. We evaluated training on all admitted candidates as a test of broader difficulty coverage, including the candidates that random or moderate selection could choose. This yields similar GenEval but substantially lower understanding performance than MATE.

MethodGenEval ↑Und. Avg. ↑
Janus-Pro-1B74.456.9
MATE77.457.8
All candidates trained77.045.7

Q3

Input diversity is inherent to our training mechanism. Mutual challenge and data-space self-play naturally introduce more diverse training inputs, so improvement from this diversity is part of MATE’s benefit. While challenges can vary with fixed pools, they remain derived from the original sources; MATE lets these sources themselves evolve through selected outputs. In the Fig. 6(b), fixing the pools lowers the generation average from 80.3 to 79.9, and additionally replacing hard selection with cooperative selection lowers it to 79.5, suggesting that both mechanisms contribute to the gains.

Dear Reviewer miRi,

Thank you again for suggesting the one-way training comparison. In our rebuttal, we found that alternating between generation and understanding performed better than training either branch alone on both GenEval and average understanding performance. We also compared selecting the hardest valid candidate with training on all valid candidates.

As the discussion draws to a close, could you let us know whether these experiments address your main concerns? We would be happy to clarify any points that remain open. If the results resolve your concerns, we would appreciate your considering them in your final evaluation.

Thank you for your time and feedback.

Dear Reviewer miRi,

Apologies for the additional follow-up. As only a few hours remains in the discussion period, we wanted to kindly ask whether our response has addressed your main concerns. We would be very happy to clarify any remaining questions.

If you feel that our response has resolved your concerns, we would sincerely appreciate your consideration of it in your final evaluation.

Thank you again for your time and consideration.

Thank you for the additional experiments. The one-way training results make a stronger case for the mutual training design. While I still think the motivation for hard-example selection and data evolution could be clearer, the new evidence has addressed some of my concerns. I will raise my rating from 2 to 3. But I would also encourage the AC to take into account the assessments of reviewers with more direct experience in this field.

Reviewer_vqWsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MATE, an RL post-training framework for unified multimodal models. The generation and understanding branches take turns as challenger and solver. The challenger proposes candidates that pass a source-consistency threshold, and the solver is trained with GRPO on the candidate it handles worst. The selected outputs become the next epoch's input pools for the other branch. Scores come from a frozen UnifiedReward model. On Janus-Pro-1B, MATE improves GenEval, DPG-Bench, the average of nine understanding benchmarks, and round-trip consistency.

Critical strengths
  1. Leveraging the inverse symmetry of image captioning and text-to-image generation for self-play requires no separate adversary network or additional human annotations.

  2. Filtering candidates via a source-consistency threshold prevents selecting degenerate prompts, and reusing the probe rollouts directly for GRPO updates avoids redundant sampling.

Critical weaknesses
  1. The source of the gains is unclear. Direct GRPO with the same reward model is not reported. The cooperative variant already recovers most of the gains, and training on all admitted candidates matches selecting the hardest one. Much of the improvement may come from RL with a strong external reward model.

  2. The baseline setup is unclear. UniRL and SUDER originally use different data and rewards, and the paper does not state what their re-implementations use.

  3. Scalability is untested. All results use 5,000 pairs and one 1B UMM, which does not support the motivation of going beyond annotated data. Each input needs up to 32 rollouts scored by a 7B reward model, so the cost of scaling is also a concern.

Questions for authors
  1. How does MATE compare with direct GRPO on the same corpus, using the same reward model and a matched sampling budget?
5 comments

W1&Q1

MATE now does not need external reward model now. In the submitted version, we used a frozen external reward model to provide consistent scoring for stable training. We agree that with the external reward model, our results do not separate MATE’s mechanism from Direct GRPO with the reward model. Therefore, we trained MATE without any reward model and proposed it as our main model and updated our pdf. Its rewards come from itself, which scores each output with the opposite branch. The generated image is scored by the per-token log-likelihood that the understanding branch assigns to the description it was generated from, and a generated description by the per-token log-likelihood that the generation branch assigns to the VQVAE tokens of the image it describes. MATE without external reward model reaches consistently better performance than SUDER, which likewise rewards each branch with the model’s own likelihoods and also gains over direct GRPO with reward model.

MethodSingle Obj.Two Obj.CountingColorsPositionColor Attr.GenEvalDPG-Bench
Janus-Pro-1B98.484.153.191.263.856.074.482.1
+ UniRL97.582.851.692.066.560.575.282.1
+ SUDER98.885.951.989.472.063.076.883.1
+ Direct GRPO w/ UnifiedReward99.486.648.492.370.862.876.783.8
+ MATE w/ UnifiedReward (submitted)99.187.953.890.765.564.076.883.8
+ MATE w/o external reward model (ours)99.186.954.789.170.064.877.483.8
MethodPOPEMME-CMMBSEEDMMStarV*BenchCV-2DCV-3DSEED2+Avg.
Janus-Pro-1B84.3263.674.566.943.540.859.858.950.956.9
+ UniRL83.8258.973.465.742.042.457.055.149.255.7
+ SUDER84.9261.874.666.943.941.461.859.550.957.4
+ Direct GRPO w/ UnifiedReward84.4286.474.267.543.141.460.859.351.457.5
+ MATE w/ UnifiedReward (submitted)84.2281.474.767.043.742.960.559.251.257.6
+ MATE w/o external reward model (ours)84.4276.873.667.242.744.562.060.551.257.8

W2

Since UniRL and SUDER don’t have released everything needed to reproduce them, we reimplemented them both on Janus-Pro-1B using exactly the same setup as MATE.

W3

We agree that we did not test MATE on different scales due to the limit of computation and will acknowledge it in the revision. We have removed the external reward model and proved the effectiveness of MATE, so the reward cost for scalability can be controlled now.

Due to limited computational resources, training of the 7B model is still ongoing, and we will add the evaluation results to this response as soon as they become available.

W3: 7B results

Image generation.

ModelSingle Obj.Two Obj.CountingColorsPositionColor Attr.GenEvalDPG-Bench
Janus-Pro-7B98.185.457.590.477.066.879.284.1
MATE-7B99.191.757.890.476.570.080.985.7

Visual understanding.

ModelPOPEMME-CMMBSEEDMMStarV*CV-2DCV-3DSEED2+Avg.
Janus-Pro-7B86.3301.178.672.048.248.772.171.856.563.5
MATE-7B85.4309.378.872.449.149.772.071.757.263.9

Together with the 1B results in our previous response, these additional experiments provide evidence that MATE’s benefits extend to the larger 7B backbone within the Janus-Pro family. We thank the reviewer for motivating this additional evaluation.

Dear Reviewer vqWs,

Thank you again for suggesting the direct GRPO comparison. We included it in our rebuttal and added results on Janus-Pro-7B in our follow-up. Could you let us know whether these results address your concerns about the source of MATE’s gains and whether they carry over to a larger model?

We would be happy to clarify any remaining questions before the discussion closes. If these results resolve your concerns, we would appreciate your taking them into account in your final evaluation.

Thank you again for your time.

Comments on the paper

1 comment

Dear Area Chair,

We would like to highlight two main updates in our rebuttal and revised manuscript:

  1. MATE now improves both generation and understanding without an external reward model. Training rewards come from the UMM itself, with each branch scoring the other branch’s outputs through cross-modal likelihoods. This directly addresses the concern that our improvements depend on a strong external evaluator. We have expanded Section 4.1 and Appendix C to explain this scoring procedure.

  2. We have expanded the experimental evidence with 7B results and additional ablations. The new experiments extend evaluation beyond the original 1B model and include ablations of bidirectional training and candidate selection. These additions examine performance at a larger model scale and help assess which components contribute to the improvements.

We have responded to the reviewers’ concerns but have not yet received follow-up. As the discussion period is nearing its end, we would appreciate, if appropriate, your help in encouraging the reviewers to consider these updates and identify any remaining concerns.

Thank you for your time and help.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.