- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when visual generation supervision improves visual understanding. The authors first compare training recipes using paired Jigsaw and Zoom-In tasks that share inputs but produce image or text answers. They then introduce OmniTaskonomy to measure transfer from 19 I2I tasks to 25 understanding capabilities. On BAGEL, I2I training followed by I2T fine-tuning supports positive transfer, but the gains are selective and some capabilities get worse. Gradient alignment is also positively correlated with transfer gains, suggesting a possible signal for task selection.
Critical strengths
-
The paired Jigsaw and Zoom-In tasks use the same inputs and require the same ordering operation, with only the answer format changing (Section 3). This makes the comparison easier to interpret and reduces differences in task content.
-
The authors compare six training recipes and vary both I2I and I2T data budgets (Figure 2). These experiments help show how training order and supervision amounts affect transfer, rather than only showing that an additional generation stage improves a score.
-
Figure 4 breaks down transfer by source task and target capability, reports both gains and drops, and averages results over three seeds. Readers can see which generation objectives help which capabilities, as well as where this supervision fails to help.
Critical weaknesses
-
All core training experiments use BAGEL-7B-MoT (Section 3 and Appendices A.1 and D.1). The preferred curriculum, transfer map, and gradient-alignment findings therefore come from one architecture. Since generation-to-understanding transfer depends on how the two pathways interact, I cannot tell whether these findings would hold in another omni model. This matters because the paper presents them as guidance for training omni models more broadly; the current evidence does not establish how far that guidance extends.
-
In the cross-task study, each source model receives roughly 50k I2I examples before the same 50k I2T stage, while the baseline receives only the I2T stage (Section 5.1 and Appendix D.1). The I2I tasks also use different datasets, including Taskonomy, VisGym, COCO, and ConceptEdit (Appendix C.4). Without controls for the additional training budget and source data, it is hard to separate the contribution of the generation objective from extra visual supervision or dataset differences. This directly affects how I interpret the transfer map: it shows which training pipelines help, but does not yet isolate which generation objectives are responsible. Together with the single-architecture evaluation, this leaves the paper’s main recommendations insufficiently supported for acceptance.
Questions for authors
-
Were any of the training recipes tested on an omni model other than BAGEL? If you have such results, even for the paired tasks, please share them. If not, how much of the advantage of I2I → I2T do you think depends on BAGEL’s architecture? I’m trying to understand how broadly the training recommendation applies.
-
For the cross-task study, did you run any controls that match the extra training budget or use the same source images with a different supervision objective? If so, what did they show? If these controls were not run, which existing results help separate the effect of the I2I objective from extra training and differences between source datasets?
3 comments
[W2, Q2a]. Additional training controls
First, we would like to clarify that our question is whether I2I supervision can compensate for limited task-specific I2T data. For example, colorization and inpainting pairs can be constructed at scale from existing images, without collecting additional annotations for the downstream understanding task. Also in many real settings there might be a large amount of I2I data (e.g. video) while I2T data is very scarce. We are not trying to say that I2I data is better or more efficient than I2T data, but are trying to leverage them for better I2T performance.
Second, we compare I2I and I2T pre-stage supervision using the same amount of counting data, followed by the same downstream LLaVA stage. OOD counting accuracy is reported as mean SEM over three training seeds:
| Training condition | OOD counting accuracy (%) |
|---|---|
| LLaVA only | |
| I2T VisGym LLaVA | |
| I2I VisGym LLaVA |
We show that I2I training can even achieve a slightly higher mean accuracy than using an I2T pre-stage using the same amount of data, supporting generation supervision as an alternative to additional I2T pretraining (Appendix A.2.1, Table 3).
[W2, Q2b]. Robustness to source-data choice
Thanks for the suggestion. To understand the effect of the data source domain, we tested whether each objective's transfer pattern persists when we replace its source images with images from another domain while keeping the downstream I2T setup fixed. We reconstructed Jigsaw on Taskonomy instead of VisGym, and colorization on COCO instead of Taskonomy.
The results show that the transfer patterns remain highly consistent across source domains. The original and replacement transfer profiles have Pearson correlations of 0.81 for Jigsaw and 0.80 for colorization, measured across 19 downstream capabilities. In particular, the strong positive transfer to 2D Ordering remains statistically significant across both source domains for both objectives:
| I2I objective | Source replacement | Profile Pearson | 2D Ordering: original (pp) | 2D Ordering: replacement (pp) |
|---|---|---|---|---|
| Jigsaw | VisGym Taskonomy | 0.81 | ||
| Colorization | Taskonomy COCO | 0.80 |
Transfer gains are changes in I2T accuracy relative to I2T-only, averaged over three training seeds. denotes under the exact paired permutation test.
These findings provide evidence that the observed transfer patterns are not simply artifacts of the original source datasets. While the magnitude of individual gains can vary with the source domain, the overall transfer patterns remain strongly correlated, and the strongest positive transfer directions persist. This supports our central finding that I2I objectives exhibit characteristic transfer patterns across I2T capabilities.
We have updated the full source-swap results in Appendix A.2, Fig. 7. We provide the complete numerical results below.
Jigsaw: Complete source-swap results.
| Target capability | VisGym source (pp) | Taskonomy source (pp) | |
|---|---|---|---|
| Category recognition | 962 | ||
| Appearance understanding | 687 | ||
| Visual similarity | 248 | ||
| State recognition | 340 | ||
| Activity understanding | 307 | ||
| Anomaly detection | 348 | ||
| Situation understanding | 216 | ||
| OCR and text recognition | 219 | ||
| Lighting understanding | 156 | ||
| Depth understanding | 820 | ||
| Metric 3D relation | 723 | ||
| Orientation understanding | 201 | ||
| 2D spatial relation | 927 | ||
| Multi-view Reasoning | 149 | ||
| Localization | 468 | ||
| Connectivity | 134 | ||
| Counting | 1542 | ||
| 2D Ordering | 190 | ||
| Visual Correspondence | 508 |
Colorization: Complete source-swap results.
| Target capability | Taskonomy source (pp) | COCO source (pp) | |
|---|---|---|---|
| Category recognition | 962 | ||
| Appearance understanding | 687 | ||
| Visual similarity | 248 | ||
| State recognition | 340 | ||
| Activity understanding | 307 | ||
| Anomaly detection | 348 | ||
| Situation understanding | 216 | ||
| OCR and text recognition | 219 | ||
| Lighting understanding | 156 | ||
| Depth understanding | 820 | ||
| Metric 3D relation | 723 | ||
| Orientation understanding | 201 | ||
| 2D spatial relation | 927 | ||
| Multi-view Reasoning | 149 | ||
| Localization | 468 | ||
| Connectivity | 134 | ||
| Counting | 1542 | ||
| 2D Ordering | 190 | ||
| Visual Correspondence | 508 |
denotes the target evaluation-set size. Each entry reports the change in I2T accuracy relative to the corresponding I2T-only baseline, in percentage points, averaged over three training seeds. denotes under the exact paired permutation test used in Fig. 7.
Author Response to Reviewer wxPy
Thank you for the questions about architecture generality, the additional training stage, and source-data choice. We added experiments for each concern: a controlled replication on Janus-Pro-1B which uses a substantially different architecture, an I2I-versus-I2T pre-stage comparison, and matched-objective source-domain swaps. We give the settings and numerical results below.
[W1, Q1]. Generalization across model architectures
Our original study focuses on BAGEL's Mixture-of-Transformers (MoT) architecture [1], an important design for unified multimodal models that has also been adopted in physical AI systems such as Cosmos 3 [2]. Beyond this original scope, we further demonstrate that our central findings on training recipes generalize to a substantially different model architecture. Specifically, we conducted additional experiments on Janus-Pro-1B [3], an autoregressive unified multimodal model. Unlike BAGEL, which adopts a MoT design with flow-based image generation, Janus-Pro uses separate visual encoders for understanding and generation, a shared autoregressive Transformer, and discrete visual tokens for image generation.
We reproduced the controlled Jigsaw and Zoom-In experiments, where the same visual problem is formulated as either an image-generation (I2I) or text-prediction (I2T) task. We evaluated all five training curricula across four I2I data budgets (3k–100k), together with an I2T-only baseline.
The results reproduce our main finding: generation-first training substantially outperforms directly mixing I2I and I2T supervision, even under a different model architecture. Both I2I$\rightarrow$I2T and I2I$\rightarrow$Mixed outperform Mixed at every tested budget from 10k to 100k on both tasks. The advantage becomes particularly pronounced at larger I2I budgets:
Moreover, both generation-first curricula improve consistently as the I2I data budget increases, reproducing the scaling behavior observed with BAGEL. In contrast, directly mixing I2I and I2T supervision yields much smaller gains even with 100k I2I samples.
The complete results are reported below. We measure I2T exact-match accuracy (%), reporting mean ± SEM across three training seeds, consistent with our BAGEL experiments.
Jigsaw on Janus-Pro-1B. I2T-only baseline: 21.3±10.1.
Zoom-In on Janus-Pro-1B. I2T-only baseline: 15.9±3.9.
The complete learning curves are provided in Appendix A.1, Fig. 6.
Together, these experiments provide cross-architecture evidence for our central conclusion: the benefit of I2I supervision for visual understanding depends strongly on the training curriculum. The same generation-first advantage and I2I data-scaling trends emerge in both BAGEL and Janus-Pro, despite their different architectures and visual generation mechanisms. This demonstrates that our findings are not specific to BAGEL's MoT design.
References
[1] Liang et al. Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR, 2025. https://arxiv.org/abs/2411.04996
[2] Aditi et al. Cosmos 3: Omnimodal World Models for Physical AI. arXiv:2606.02800, 2026. https://arxiv.org/abs/2606.02800
[3] Chen et al. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv:2501.17811, 2025. https://arxiv.org/abs/2501.17811