acceptodds
Under review as a conference paper at ICLR 2027

OmniTaskonomy: When does Visual Generation Improve Visual Understanding?

Abstract

Training a model to generate visual content can encourage it to learn rich perceptual capabilities related to geometry, spatial relationships, and objectness; yet, its benefits for visual understanding remain unclear. We ask: when and how does visual generation supervision improve visual understanding? We study controlled pairs of image-to-image (I2I) generation and image-to-text (I2T) understanding tasks that express the same underlying problem in different output modalities. We find that under the correct recipe, I2I training improves downstream I2T performance, with larger gains as the amount of I2I training data increases. We next ask which generation tasks benefit which understanding capabilities. To study transfer beyond paired tasks, we introduce OmniTaskonomy, a unified taxonomy spanning 19 I2I generation tasks and 25 I2T understanding capabilities. The resulting transfer map reveals selective, task-dependent benefits. Some follow intuitive correspondences, e.g., depth prediction improving metric 3D reasoning, object pointing improving counting, and jigsaw reconstruction improving 2D ordering. Interestingly, we also uncover surprising connections: 2.5D segmentation improving category recognition and Z-depth prediction improving localization. To probe these patterns, we analyze gradient alignment between generation and understanding tasks and find that stronger alignment is associated with larger downstream transfer gains. Together, our results highlight visual generation as a rich source of supervision for visual understanding and provide a roadmap for unlocking its benefits through the right training curriculum and task selection.

Primary area:
interpretability and explainable AI
Keywords:
unified multimodal models, visual generation, visual understanding, transfer learning, gradient alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wxPyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when visual generation supervision improves visual understanding. The authors first compare training recipes using paired Jigsaw and Zoom-In tasks that share inputs but produce image or text answers. They then introduce OmniTaskonomy to measure transfer from 19 I2I tasks to 25 understanding capabilities. On BAGEL, I2I training followed by I2T fine-tuning supports positive transfer, but the gains are selective and some capabilities get worse. Gradient alignment is also positively correlated with transfer gains, suggesting a possible signal for task selection.

Critical strengths
  1. The paired Jigsaw and Zoom-In tasks use the same inputs and require the same ordering operation, with only the answer format changing (Section 3). This makes the comparison easier to interpret and reduces differences in task content.

  2. The authors compare six training recipes and vary both I2I and I2T data budgets (Figure 2). These experiments help show how training order and supervision amounts affect transfer, rather than only showing that an additional generation stage improves a score.

  3. Figure 4 breaks down transfer by source task and target capability, reports both gains and drops, and averages results over three seeds. Readers can see which generation objectives help which capabilities, as well as where this supervision fails to help.

Critical weaknesses
  1. All core training experiments use BAGEL-7B-MoT (Section 3 and Appendices A.1 and D.1). The preferred curriculum, transfer map, and gradient-alignment findings therefore come from one architecture. Since generation-to-understanding transfer depends on how the two pathways interact, I cannot tell whether these findings would hold in another omni model. This matters because the paper presents them as guidance for training omni models more broadly; the current evidence does not establish how far that guidance extends.

  2. In the cross-task study, each source model receives roughly 50k I2I examples before the same 50k I2T stage, while the baseline receives only the I2T stage (Section 5.1 and Appendix D.1). The I2I tasks also use different datasets, including Taskonomy, VisGym, COCO, and ConceptEdit (Appendix C.4). Without controls for the additional training budget and source data, it is hard to separate the contribution of the generation objective from extra visual supervision or dataset differences. This directly affects how I interpret the transfer map: it shows which training pipelines help, but does not yet isolate which generation objectives are responsible. Together with the single-architecture evaluation, this leaves the paper’s main recommendations insufficiently supported for acceptance.

Questions for authors
  1. Were any of the training recipes tested on an omni model other than BAGEL? If you have such results, even for the paired tasks, please share them. If not, how much of the advantage of I2I → I2T do you think depends on BAGEL’s architecture? I’m trying to understand how broadly the training recommendation applies.

  2. For the cross-task study, did you run any controls that match the extra training budget or use the same source images with a different supervision objective? If so, what did they show? If these controls were not run, which existing results help separate the effect of the I2I objective from extra training and differences between source datasets?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer wxPy

Thank you for the questions about architecture generality, the additional training stage, and source-data choice. We added experiments for each concern: a controlled replication on Janus-Pro-1B which uses a substantially different architecture, an I2I-versus-I2T pre-stage comparison, and matched-objective source-domain swaps. We give the settings and numerical results below.

[W1, Q1]. Generalization across model architectures

Our original study focuses on BAGEL's Mixture-of-Transformers (MoT) architecture [1], an important design for unified multimodal models that has also been adopted in physical AI systems such as Cosmos 3 [2]. Beyond this original scope, we further demonstrate that our central findings on training recipes generalize to a substantially different model architecture. Specifically, we conducted additional experiments on Janus-Pro-1B [3], an autoregressive unified multimodal model. Unlike BAGEL, which adopts a MoT design with flow-based image generation, Janus-Pro uses separate visual encoders for understanding and generation, a shared autoregressive Transformer, and discrete visual tokens for image generation.

We reproduced the controlled Jigsaw and Zoom-In experiments, where the same visual problem is formulated as either an image-generation (I2I) or text-prediction (I2T) task. We evaluated all five training curricula across four I2I data budgets (3k–100k), together with an I2T-only baseline.

The results reproduce our main finding: generation-first training substantially outperforms directly mixing I2I and I2T supervision, even under a different model architecture. Both I2I$\rightarrow$I2T and I2I$\rightarrow$Mixed outperform Mixed at every tested budget from 10k to 100k on both tasks. The advantage becomes particularly pronounced at larger I2I budgets:

  • Jigsaw (100k I2I): I2I$\rightarrow$I2T reaches 73.8% and I2I$\rightarrow$Mixed reaches 78.3%, compared with only 25.8% for Mixed, corresponding to gains of +48.0 and +52.5 percentage points.
  • Zoom-In (100k I2I): I2I$\rightarrow$I2T reaches 92.4% and I2I$\rightarrow$Mixed reaches 95.0%, compared with 19.5% for Mixed, corresponding to gains of +72.9 and +75.5 percentage points.

Moreover, both generation-first curricula improve consistently as the I2I data budget increases, reproducing the scaling behavior observed with BAGEL. In contrast, directly mixing I2I and I2T supervision yields much smaller gains even with 100k I2I samples.

The complete results are reported below. We measure I2T exact-match accuracy (%), reporting mean ±\pm SEM across three training seeds, consistent with our BAGEL experiments.

Jigsaw on Janus-Pro-1B. I2T-only baseline: 21.3±10.121.3\pm10.1.

Training recipe3k I2I10k I2I30k I2I100k I2I
I2I$\rightarrow$I2T36.1±5.836.1\pm5.840.8±10.240.8\pm10.250.2±5.950.2\pm5.973.8±8.473.8\pm8.4
I2I$\rightarrow$Mixed7.7±0.97.7\pm0.920.2±4.320.2\pm4.351.1±0.751.1\pm0.778.3±0.878.3\pm0.8
Mixed15.3±1.515.3\pm1.510.5±2.210.5\pm2.221.4±2.121.4\pm2.125.8±1.725.8\pm1.7
Mixed$\rightarrow$I2T12.7±2.412.7\pm2.418.7±4.918.7\pm4.927.3±3.927.3\pm3.925.0±5.025.0\pm5.0
Frozen I2I$\rightarrow$Mixed17.1±4.517.1\pm4.516.0±2.416.0\pm2.419.8±0.919.8\pm0.915.0±3.015.0\pm3.0

Zoom-In on Janus-Pro-1B. I2T-only baseline: 15.9±3.915.9\pm3.9.

Training recipe3k I2I10k I2I30k I2I100k I2I
I2I$\rightarrow$I2T17.9±5.817.9\pm5.875.0±3.275.0\pm3.287.9±2.687.9\pm2.692.4±0.292.4\pm0.2
I2I$\rightarrow$Mixed13.4±4.313.4\pm4.323.6±4.523.6\pm4.590.9±0.790.9\pm0.795.0±1.495.0\pm1.4
Mixed5.6±0.65.6\pm0.616.6±1.816.6\pm1.821.7±1.921.7\pm1.919.5±2.219.5\pm2.2
Mixed$\rightarrow$I2T10.3±3.510.3\pm3.518.1±3.118.1\pm3.122.3±6.022.3\pm6.025.2±5.525.2\pm5.5
Frozen I2I$\rightarrow$Mixed5.3±0.45.3\pm0.414.8±2.614.8\pm2.632.0±2.332.0\pm2.371.5±1.171.5\pm1.1

The complete learning curves are provided in Appendix A.1, Fig. 6.

Together, these experiments provide cross-architecture evidence for our central conclusion: the benefit of I2I supervision for visual understanding depends strongly on the training curriculum. The same generation-first advantage and I2I data-scaling trends emerge in both BAGEL and Janus-Pro, despite their different architectures and visual generation mechanisms. This demonstrates that our findings are not specific to BAGEL's MoT design.

References

[1] Liang et al. Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR, 2025. https://arxiv.org/abs/2411.04996

[2] Aditi et al. Cosmos 3: Omnimodal World Models for Physical AI. arXiv:2606.02800, 2026. https://arxiv.org/abs/2606.02800

[3] Chen et al. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv:2501.17811, 2025. https://arxiv.org/abs/2501.17811

[W2, Q2a]. Additional training controls

First, we would like to clarify that our question is whether I2I supervision can compensate for limited task-specific I2T data. For example, colorization and inpainting pairs can be constructed at scale from existing images, without collecting additional annotations for the downstream understanding task. Also in many real settings there might be a large amount of I2I data (e.g. video) while I2T data is very scarce. We are not trying to say that I2I data is better or more efficient than I2T data, but are trying to leverage them for better I2T performance.

Second, we compare I2I and I2T pre-stage supervision using the same amount of counting data, followed by the same downstream LLaVA stage. OOD counting accuracy is reported as mean ±\pm SEM over three training seeds:

Training conditionOOD counting accuracy (%)
LLaVA only63.58±1.2163.58\pm 1.21
I2T VisGym →\rightarrow LLaVA64.18±0.8064.18\pm 0.80
I2I VisGym →\rightarrow LLaVA64.85±0.3464.85\pm 0.34

We show that I2I training can even achieve a slightly higher mean accuracy than using an I2T pre-stage using the same amount of data, supporting generation supervision as an alternative to additional I2T pretraining (Appendix A.2.1, Table 3).

[W2, Q2b]. Robustness to source-data choice

Thanks for the suggestion. To understand the effect of the data source domain, we tested whether each objective's transfer pattern persists when we replace its source images with images from another domain while keeping the downstream I2T setup fixed. We reconstructed 2×22\times2 Jigsaw on Taskonomy instead of VisGym, and colorization on COCO instead of Taskonomy.

The results show that the transfer patterns remain highly consistent across source domains. The original and replacement transfer profiles have Pearson correlations of 0.81 for Jigsaw and 0.80 for colorization, measured across 19 downstream capabilities. In particular, the strong positive transfer to 2D Ordering remains statistically significant across both source domains for both objectives:

I2I objectiveSource replacementProfile Pearson rr2D Ordering: original (pp)2D Ordering: replacement (pp)
2×22\times 2 JigsawVisGym →\rightarrow Taskonomy0.81+6.80∗+6.80^{*}+11.58∗+11.58^{*}
ColorizationTaskonomy →\rightarrow COCO0.80+5.30∗+5.30^{*}+7.89∗+7.89^{*}

Transfer gains are changes in I2T accuracy relative to I2T-only, averaged over three training seeds. ∗^* denotes p<0.05p<0.05 under the exact paired permutation test.

These findings provide evidence that the observed transfer patterns are not simply artifacts of the original source datasets. While the magnitude of individual gains can vary with the source domain, the overall transfer patterns remain strongly correlated, and the strongest positive transfer directions persist. This supports our central finding that I2I objectives exhibit characteristic transfer patterns across I2T capabilities.

We have updated the full source-swap results in Appendix A.2, Fig. 7. We provide the complete numerical results below.

2×22\times 2 Jigsaw: Complete source-swap results.

Target capabilityNNVisGym source (pp)Taskonomy source (pp)
Category recognition962+0.8+0.8−0.35-0.35
Appearance understanding687+1.5+1.5−0.24-0.24
Visual similarity248+0.3+0.3+1.08+1.08
State recognition340+0.4+0.4−0.39-0.39
Activity understanding307−2.3∗-2.3^{*}−0.65-0.65
Anomaly detection348+1.2+1.20.000.00
Situation understanding216−1.1-1.1−2.31∗-2.31^{*}
OCR and text recognition219+0.5+0.5+1.37+1.37
Lighting understanding156−2.6-2.6+0.21+0.21
Depth understanding820+0.4+0.4−1.50∗-1.50^{*}
Metric 3D relation723+2.6∗+2.6^{*}+0.37+0.37
Orientation understanding201−0.3-0.3+0.66+0.66
2D spatial relation927−0.2-0.2−0.58-0.58
Multi-view Reasoning149−6.9-6.9−5.82-5.82
Localization468−0.1-0.1+0.93+0.93
Connectivity134−1.2-1.2−0.25-0.25
Counting1542+1.0+1.0−1.36∗-1.36^{*}
2D Ordering190+6.8∗+6.8^{*}+11.58∗+11.58^{*}
Visual Correspondence508−2.9∗-2.9^{*}−1.38-1.38

Colorization: Complete source-swap results.

Target capabilityNNTaskonomy source (pp)COCO source (pp)
Category recognition962+0.2+0.2+0.45+0.45
Appearance understanding687−0.2-0.2+0.97+0.97
Visual similarity248−0.5-0.5+1.08+1.08
State recognition340−0.7-0.7+0.49+0.49
Activity understanding307+0.8+0.8+1.09+1.09
Anomaly detection348−0.5-0.5−0.38-0.38
Situation understanding216−0.2-0.2−1.39-1.39
OCR and text recognition219−0.2-0.2−0.30-0.30
Lighting understanding156+3.4+3.4+2.35+2.35
Depth understanding8200.00.0−1.67∗-1.67^{*}
Metric 3D relation723+0.5+0.5+1.11+1.11
Orientation understanding201−0.2-0.2−0.50-0.50
2D spatial relation927−1.2∗-1.2^{*}−0.43-0.43
Multi-view Reasoning149−5.8-5.8−2.68-2.68
Localization468+1.6+1.6+0.07+0.07
Connectivity134−0.2-0.2+0.50+0.50
Counting1542+0.7+0.7−0.52-0.52
2D Ordering190+5.3∗+5.3^{*}+7.89∗+7.89^{*}
Visual Correspondence508−0.9-0.9+0.72+0.72

NN denotes the target evaluation-set size. Each entry reports the change in I2T accuracy relative to the corresponding I2T-only baseline, in percentage points, averaged over three training seeds. ∗^{*} denotes p<0.05p<0.05 under the exact paired permutation test used in Fig. 7.

Reviewer_u61rOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Whether visual understanding improves visual generation is a rather explored research question in the literature, yet this paper proposed to investigate the opposite, questioning how visual generation contributes to visual understanding. In particular, the paper investigates which generation tasks help which understanding capability, hence providing an in-depth analysis of this research question. For doing so, the authors introduce OmniTaskonomy, a unified taxonomy of visual capabilities integrating both I2I and I2T tasks. Finally, the authors also investigate whether the contribution between generation tasks and understanding tasks can be explained by gradient alignment in controlled paired tasks.

Critical strengths
  • The paper has an interesting set of research questions that are tackled with extensive experiments.
  • The appendix is very informative and allows reproducibility
  • The OmniTaskonomy taxonomy is a relevant effort for answering the proposed research questions, but also has the potential of contributing beyond this paper
Critical weaknesses

While I do hope to see this paper published, I do believe that the manuscript is not particularly understandable from the way it is written. I have noted suggestions for improving it.

  1. The introduction is extremely long and it seems more like a summary of the entire paper rather than motivating the work and, indeed, introducing it. It takes a lot of space that could be given to other things (suggestion in point 2). In particular I believe that the part from line 67 to 104 could be substantially shortened.
  2. The paper lacks any sort of discussion of the results or acknowlegements of the limitations. This really contributes negatively to my understanding of the work overall.
  3. The abstract is quite confusing and contains a lot of details. I believe this could be shortened as well.
  4. In lines 64 to 66 it could be good to clearly add numbers to distinguish the three research questions.
  5. In line 67 the research question has the word "curriculum" (singular) while in line 64 the word is "curricula" (plural). Please be consistent.
  6. The title of the section 3 is not fully consistent with how RQ1 is stated. Having clear consistency would help the reader. This is also valid for Section 5 and Section 6.
  7. Section 3 starts with "We first ask". This is used a lot also in the introduction and I would suggest to find some synonyms to avoid such repetitions.
  8. In line 196-199, before the end of Section 3, a "finding" box is provided. However, no finding box is provided after the end of the other paragraphs of Section 3, which is confusing. Do those paragraphs not have findings? In addition, in Section 5, the finding box is provided at the end of the section, while in Section 6 there are two boxes in the middle of the Section. I believe, once again, that consistency would improve the quality of the manuscript.
  9. I have doubts about Section 4 being separated from Section 5. In the introduction, OmniTaskonomy has been introduced in RQ2. While it is true that this is a self-standing contribution of the paper and, in addition, it is also used for RQ3 in Section 6; the fact that it is not integrated in Section 5 creates some confusion while reading. In particular, the beginning of Section 5 refers directly to the end of Section 3, but this is confusing, because there is Section 4 in the middle. It prompts me to think that it would be better if there was Section 3 on RQ1 --> Section 5 (intro to RQ2) + Section 4 (omnitaskonomy) inside Section 5 and then experiments on RQ2 --> Section 6 on RQ3.
Questions for authors
  1. It is great that OmniTaskonomy also included human annotation, however, very little information is provided about this both in Section 4 and in Appendix B.5. Are the four annotators part of the team of authors? If so, could this impact/bias the annotation? Are these four annotator external? If so, was an IRB approval obtained? If not, why?

  2. What are the limitations of the study? The manuscript, as it stands, really suffers from the lack of such a discussion.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer u61r

Thank you for recognizing the research questions, experimental coverage, reproducibility material, and value of OmniTaskonomy. We appreciate your suggestions.

W1 & W3. Introduction and abstract

The introduction now gives an explicit RQ1–RQ3 roadmap, and the abstract presents the central question, OmniTaskonomy, and the three main findings in the same order. Please see the updated paper.

W4, W6, W7 & W9. Research questions and organization

We have explicitly labeled RQ1–RQ3 and aligned the section titles and transitions with these questions. Following your suggestion, we have presented OmniTaskonomy construction and the transfer experiments together under RQ2, followed by the gradient analysis under RQ3. We have also removed repetitive “We first ask” openings and corrected the transition that previously skipped Section 4.

W5 & W8. Terminology and finding boxes

We have standardized the use of “curriculum” and “curricula,” as well as the format and placement of finding boxes.

W2 & Q2. Discussion and limitations

We have added a discussion that connects the findings to curriculum design and source-task selection and explicitly states their empirical scope. The controlled curriculum findings are replicated on Janus-Pro-1B, while the full capability-transfer and gradient analyses are conducted on BAGEL. Extending these analyses across additional architectures and training settings remains a direction for future work.

Q1. Independent validation of the capability labels

The four annotators in the original validation were authors. To address the concern about author-specific judgments, we conducted an additional validation round with four AI researchers who are not authors of this paper.

The new audit covers 250 examples. We first randomly selected one example from each of the 25 capability leaves, sampled another 225 without replacement from the remaining retained examples, and shuffled the resulting set. Each researcher assessed the same 50 shared examples and 50 distinct examples, giving 100 judgments per researcher and 400 judgments in total. The shared set supports inter-reviewer agreement estimates, while the distinct sets increase coverage. Reviewers saw the image(s), question, answer choices, reference answer, assigned capability, and the catalogue of capability definitions and boundary rules. They could accept the assigned label, select a replacement, or mark the case as unsure.

Validation measureResult
Non-author AI researchers4
Unique examples audited250
Total judgments400
VLM label accepted386/400 (96.50%)
Replacement label selected9/400 (2.25%)
Unsure5/400 (1.25%)
Acceptance with equal weight per example96.50%
Acceptance among definite judgments386/395 (97.72%)
Human–VLM Cohen's κ\kappa, range across auditors0.968 to 0.978
Human–VLM Cohen's κ\kappa, mean across auditors0.975
Shared examples accepted by at least 3 of 4 auditors48/50 (96.00%)
Shared examples accepted unanimously45/50 (90.00%)
Shared examples with a two–two split2/50
Shared examples rejected by a majority0/50
Krippendorff's α\alpha on shared examples0.966

The Human–VLM agreement values compare each auditor with the assigned capability labels. The separate pairwise auditor table reports agreement among auditors on the shared examples.

Agreement is also high when comparing the researchers' final capability labels.

Reviewer pairShared examples with definite labelsExact agreementCohen's κ\kappa
Auditor 1 / Auditor 24897.92%0.98
Auditor 1 / Auditor 34895.83%0.95
Auditor 1 / Auditor 448100.00%1.00
Auditor 2 / Auditor 34993.88%0.93
Auditor 2 / Auditor 44997.96%0.98
Auditor 3 / Auditor 44895.83%0.95

Across all four researchers, Krippendorff's α\alpha is 0.966 on the shared examples. The high label acceptance and inter-reviewer agreement provide independent support for the reliability of the VLM-assigned capabilities. Sec. 4 and Appendix C.5 report the non-author provenance and audit protocol, including example allocation, review inputs, and decision options. The tables above provide the full validation summary and all six pairwise comparisons.

Regarding the nature of this activity, the researchers served as annotation auditors, and the object being evaluated was the VLM-assigned dataset label. We collected no demographic, behavioral, or personal data about the auditors and analyzed their annotations only in aggregate to assess label quality. This was dataset annotation quality control. Expert checks of automated annotation or evaluation are established practice used in prior work, including ECHO [1], MT-Bench/Chatbot Arena [2], and LLMScore [3].

References

[1] Jiaxin Ge et al. Constantly Improving Image Models Need Constantly Improving Benchmarks. ICLR, 2026. Appendix A.5. Paper.

[2] Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023. Section 4 and Appendix C. Paper.

[3] Yujie Lu et al. LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation. NeurIPS, 2023. Section 4. Paper.

Reviewer_u61rOpenReview ↗

Response to Authors

Dear authors, thank you for addressing my comments. I think the paper reads much better now and I am increasing my score. However, I am keeping my confidence rather low, because of little direct experience in this specific field.

Reviewer_JApvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when and how visual generation (I2I) can help visual understanding (I2T) in unified multimodal models. Previous works usually shows that understanding helps generation, but not much the other way around. The authors study this problem on BAGEL-7B-MoT in three parts. First, in a controlled setting with paired Jigsaw and Zoom-In tasks, they compare six training recipes and find that two-stage training (I2I first, then I2T finetuning) scales well as I2I data increase, while mixing I2I and I2T data together does not work well. Second, they build OmniTaskonomy based on the 3Rs of computer vision, covering 19 I2I tasks and 25 I2T capabilities collected from 7 benchmarks (9,444 samples validated by VLMs and humans). Testing all 19x25 pairs shows that transfer is selective, mostly helping capabilities like counting, metric 3D relation, and 2D ordering. Third, they measure gradient alignment at the pretrained checkpiont and find that I2I and I2T gradients align best in early pre-attention RMSNorm layers of the understanding branch, and this alignment correlates well with downstream transfer gains.

Critical strengths
  1. The experimental design from controlled task pairs to the broad taxonomy is very clear and solid. By starting with Jigsaw and Zoom-In where the input image and task logic are exactly the same, the paper convincingly shows that the training recipe (two-stage I2I to I2T instead of mixed training) is a key factor for positive transfer. Extending this to OmniTaskonomy gives a much more detail view of which specific capability benefits from which generation task, rather than just looking at coarse benchmark averages.

  2. The gradient alignment analysis in Section 6 provides a good optimization-level explanation for why transfer happens. Showing that gradient similarity in early pre-attention RMSNorm layers correlates with actual transfer performance (r = 0.795 across capabilities) connects the empirical transfer map to model optimization, and offers a useful signal for selecting tasks without running full training.

Critical weaknesses
  1. In the OmniTaskonomy transfer experiments (Section 5), the image domain and the task objective are mixed together, and negative transfer is not discussed enough. Unlike Section 3, the 19 I2I tasks come from very different datasets (Table 10), such as 12 tasks from Taskonomy (mostly indoor scenes) and others from COCO, VisGym, or ConceptEdit. It is hard to tell how much of the transfer gains comes from the I2I task objective itself versus the dataset domain. Also, in Figure 4, most pairs show no gain, and some capabilities like Visual Correspondence and Multi-view Reasoning have consistent negative transfer across almost all I2I sources, which is not well explained.

  2. The gradient analysis in Section 6.3 only covers 7 out of 25 capabilities (those with more than 500 samples), which leaves out some of the most important capability from Section 5, such as 2D Ordering (N=190, which has the largest gain up to +7.9 pp) and Localization (N=468). Also, all experiments are only done on one model, BAGEL-7B-MoT. Since MoT uses modality-specific expert parameters with shared attention, it is unclear if findings like pre-attention RMSNorm alignment or the failure of mixed training also hold for other unified model architectures.

Questions for authors
  1. In Section 6.3, why did you only include capabilities with N > 500 for the gradient correlation analysis, especailly since you sample minibatches of 64 with replacement? If you include capabilities with N > 100 (especially 2D Ordering and Localization which show strong transfer in Figure 4), does the correlation between gradient alignment and transfer gain still hold?

  2. In Figure 4, Visual Correspondence and Multi-view Reasoning drop across almost all I2I sources. Is this because the task objectives conflict with each other, or because Stage 1 training on 50k narrow-domain images like Taskonomy causes representation forgetting that Stage 2 cannot recover?

  3. In Table 3, when scaling I2I data from 3k to 100k in Mixed training, the I2I to I2T batch ratio changes from 4:64 to 212:64 without a loss weight coefficient. Could the poor scaling of Mixed training in Figure 2b be caused by the large I2I batch dominating the gradient, or did you also observe similar results when keeping the batch ratio balanced?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer JApv

Thank you for the careful review and for recognizing the controlled experimental design and the value of the gradient analysis. We address the source-domain concern with matched-objective source swaps, reproduce the training-recipe result on Janus-Pro-1B, and extend the gradient analysis with transfer-variance and sensitivity analyses.

[W1]. Robustness to source-domain choice

To address the concern that the transfer-map results may be driven by differences in source datasets rather than I2I objectives, we conducted additional source-swap experiments that hold the I2I objective fixed while changing its source domain. Specifically, we reproduced 2×22\times 2 Jigsaw on the Taskonomy data instead of VisGym, and also colorization on the COCO dataset instead of Taskonomy. We kept the downstream I2T setup fixed and evaluated transfer relative to the I2T-only baseline of the corresponding evaluation setting.

The results show that the transfer patterns remain highly consistent across source domains. The original and replacement transfer profiles have Pearson correlations of 0.81 for Jigsaw and 0.80 for colorization, measured across 19 downstream capabilities. In particular, the strong positive transfer to 2D Ordering remains statistically significant across both source domains for both objectives:

I2I objectiveSource replacementProfile Pearson rr2D Ordering: original (pp)2D Ordering: replacement (pp)
2×22\times 2 JigsawVisGym →\rightarrow Taskonomy0.81+6.80∗+6.80^{*}+11.58∗+11.58^{*}
ColorizationTaskonomy →\rightarrow COCO0.80+5.30∗+5.30^{*}+7.89∗+7.89^{*}

Transfer gains are changes in I2T accuracy relative to I2T-only, averaged over three training seeds. ∗^* denotes p<0.05p<0.05 under the exact paired permutation test.

These findings provide evidence that the observed transfer patterns are not simply artifacts of the original source datasets. While the magnitude of individual gains can vary with the source domain, the overall transfer patterns remain strongly correlated, and the strongest positive transfer directions persist. This supports our central finding that I2I objectives exhibit characteristic transfer patterns across I2T capabilities.

Importantly, OmniTaskonomy aims to characterize which I2I objectives benefit which I2T capabilities, rather than suggesting that all I2I objectives universally improve visual understanding. Negative transfer is also part of this characterization. For example, Multi-view Reasoning shows negative transfer under both source domains for both tested objectives, whereas Visual Correspondence exhibits more source-dependent behavior. We discuss these negative-transfer patterns and their potential explanations separately in Q2. We have updated the full source-swap results in Appendix A.2, Fig. 7. The complete numerical results are provided in the next comment.

[W1]. (2/2): Full source-swap results

We have updated the full source-swap results in Appendix A.2, Fig. 7. The complete numerical results are provided in the next comment.

2×22\times 2 Jigsaw: Complete source-swap results.

Target capabilityNNVisGym source (pp)Taskonomy source (pp)
Category recognition962+0.8+0.8−0.35-0.35
Appearance understanding687+1.5+1.5−0.24-0.24
Visual similarity248+0.3+0.3+1.08+1.08
State recognition340+0.4+0.4−0.39-0.39
Activity understanding307−2.3∗-2.3^{*}−0.65-0.65
Anomaly detection348+1.2+1.20.000.00
Situation understanding216−1.1-1.1−2.31∗-2.31^{*}
OCR and text recognition219+0.5+0.5+1.37+1.37
Lighting understanding156−2.6-2.6+0.21+0.21
Depth understanding820+0.4+0.4−1.50∗-1.50^{*}
Metric 3D relation723+2.6∗+2.6^{*}+0.37+0.37
Orientation understanding201−0.3-0.3+0.66+0.66
2D spatial relation927−0.2-0.2−0.58-0.58
Multi-view Reasoning149−6.9-6.9−5.82-5.82
Localization468−0.1-0.1+0.93+0.93
Connectivity134−1.2-1.2−0.25-0.25
Counting1542+1.0+1.0−1.36∗-1.36^{*}
2D Ordering190+6.8∗+6.8^{*}+11.58∗+11.58^{*}
Visual Correspondence508−2.9∗-2.9^{*}−1.38-1.38

Colorization: Complete source-swap results.

Target capabilityNNTaskonomy source (pp)COCO source (pp)
Category recognition962+0.2+0.2+0.45+0.45
Appearance understanding687−0.2-0.2+0.97+0.97
Visual similarity248−0.5-0.5+1.08+1.08
State recognition340−0.7-0.7+0.49+0.49
Activity understanding307+0.8+0.8+1.09+1.09
Anomaly detection348−0.5-0.5−0.38-0.38
Situation understanding216−0.2-0.2−1.39-1.39
OCR and text recognition219−0.2-0.2−0.30-0.30
Lighting understanding156+3.4+3.4+2.35+2.35
Depth understanding8200.00.0−1.67∗-1.67^{*}
Metric 3D relation723+0.5+0.5+1.11+1.11
Orientation understanding201−0.2-0.2−0.50-0.50
2D spatial relation927−1.2∗-1.2^{*}−0.43-0.43
Multi-view Reasoning149−5.8-5.8−2.68-2.68
Localization468+1.6+1.6+0.07+0.07
Connectivity134−0.2-0.2+0.50+0.50
Counting1542+0.7+0.7−0.52-0.52
2D Ordering190+5.3∗+5.3^{*}+7.89∗+7.89^{*}
Visual Correspondence508−0.9-0.9+0.72+0.72

NN denotes the target evaluation-set size. Each entry reports the change in I2T accuracy relative to the corresponding I2T-only baseline, in percentage points, averaged over three training seeds. ∗^{*} denotes p<0.05p<0.05 under the exact paired permutation test used in Fig. 7.

[W2]. Generalization across model architectures: Janus-Pro-1B

Our original study focuses on BAGEL's Mixture-of-Transformers (MoT) architecture [1], an important design for unified multimodal models that has also been adopted in physical AI systems such as Cosmos 3 [2]. Beyond this original scope, we further demonstrate that our central findings on training recipes generalize to a substantially different model architecture. Specifically, we conducted additional experiments on Janus-Pro-1B [3], an autoregressive unified multimodal model. Unlike BAGEL, which adopts a MoT design with flow-based image generation, Janus-Pro uses separate visual encoders for understanding and generation, a shared autoregressive Transformer, and discrete visual tokens for image generation.

We reproduced the controlled Jigsaw and Zoom-In experiments, where the same visual problem is formulated as either an image-generation (I2I) or text-prediction (I2T) task. We evaluated all five training curricula across four I2I data budgets (3k–100k), together with an I2T-only baseline.

The results reproduce our main finding: generation-first training substantially outperforms directly mixing I2I and I2T supervision, even under a different model architecture. Both I2I$\rightarrow$I2T and I2I$\rightarrow$Mixed outperform Mixed at every tested budget from 10k to 100k on both tasks. The advantage becomes particularly pronounced at larger I2I budgets:

  • Jigsaw (100k I2I): I2I$\rightarrow$I2T reaches 73.8% and I2I$\rightarrow$Mixed reaches 78.3%, compared with only 25.8% for Mixed, corresponding to gains of +48.0 and +52.5 percentage points.
  • Zoom-In (100k I2I): I2I$\rightarrow$I2T reaches 92.4% and I2I$\rightarrow$Mixed reaches 95.0%, compared with 19.5% for Mixed, corresponding to gains of +72.9 and +75.5 percentage points.

Moreover, both generation-first curricula improve consistently as the I2I data budget increases, reproducing the scaling behavior observed with BAGEL. In contrast, directly mixing I2I and I2T supervision yields much smaller gains even with 100k I2I samples.

The complete results are reported below. We measure I2T exact-match accuracy (%), reporting mean ±\pm SEM across three training seeds, consistent with our BAGEL experiments.

Jigsaw on Janus-Pro-1B. I2T-only baseline: 21.3±10.121.3\pm 10.1.

Training recipe3k I2I10k I2I30k I2I100k I2I
I2I$\rightarrow$I2T36.1±5.836.1\pm 5.840.8±10.240.8\pm 10.250.2±5.950.2\pm 5.973.8±8.473.8\pm 8.4
I2I$\rightarrow$Mixed7.7±0.97.7\pm 0.920.2±4.320.2\pm 4.351.1±0.751.1\pm 0.778.3±0.878.3\pm 0.8
Mixed15.3±1.515.3\pm 1.510.5±2.210.5\pm 2.221.4±2.121.4\pm 2.125.8±1.725.8\pm 1.7
Mixed$\rightarrow$I2T12.7±2.412.7\pm 2.418.7±4.918.7\pm 4.927.3±3.927.3\pm 3.925.0±5.025.0\pm 5.0
Frozen I2I$\rightarrow$Mixed17.1±4.517.1\pm 4.516.0±2.416.0\pm 2.419.8±0.919.8\pm 0.915.0±3.015.0\pm 3.0

Zoom-In on Janus-Pro-1B. I2T-only baseline: 15.9±3.915.9\pm 3.9.

Training recipe3k I2I10k I2I30k I2I100k I2I
I2I$\rightarrow$I2T17.9±5.817.9\pm 5.875.0±3.275.0\pm 3.287.9±2.687.9\pm 2.692.4±0.292.4\pm 0.2
I2I$\rightarrow$Mixed13.4±4.313.4\pm 4.323.6±4.523.6\pm 4.590.9±0.790.9\pm 0.795.0±1.495.0\pm 1.4
Mixed5.6±0.65.6\pm 0.616.6±1.816.6\pm 1.821.7±1.921.7\pm 1.919.5±2.219.5\pm 2.2
Mixed$\rightarrow$I2T10.3±3.510.3\pm 3.518.1±3.118.1\pm 3.122.3±6.022.3\pm 6.025.2±5.525.2\pm 5.5
Frozen I2I$\rightarrow$Mixed5.3±0.45.3\pm 0.414.8±2.614.8\pm 2.632.0±2.332.0\pm 2.371.5±1.171.5\pm 1.1

The complete learning curves are provided in Appendix A.1, Fig. 6.

Together, these experiments provide cross-architecture evidence for our central conclusion: the benefit of I2I supervision for visual understanding depends strongly on the training curriculum. The same generation-first advantage and I2I data-scaling trends emerge in both BAGEL and Janus-Pro, despite their different Transformer architectures and visual generation mechanisms. This demonstrates that our findings are not specific to BAGEL's MoT design.

References

[1] Liang et al. Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR, 2025. https://arxiv.org/abs/2411.04996

[2] Aditi et al. Cosmos 3: Omnimodal World Models for Physical AI. arXiv:2606.02800, 2026. https://arxiv.org/abs/2606.02800

[3] Chen et al. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv:2501.17811, 2025. https://arxiv.org/abs/2501.17811

Q1. Target coverage and robustness of gradient-transfer correlation

Why N>500N>500? We focused on capabilities with more stable transfer signals. Although we sample gradient minibatches of size 64 with replacement, this does not reduce variability in the downstream transfer estimates. Our additional variance analysis supports the original threshold: the 12 capabilities newly included under N>100N>100 exhibit 7.05× higher training-seed variance in transfer gains than the original seven capabilities (6.276.27 vs. 0.89pp20.89\mathrm{pp}^{2}). In particular, Multi-view Reasoning (N=149N=149) and 2D Ordering (N=190N=190) have the highest variances (30.6330.63 and 12.47pp212.47\mathrm{pp}^{2}). The complete variance analysis is shown in Appendix A.3, Fig. 8.

Target groupCapabilitiesMean transfer variance (pp2\mathrm{pp}^{2})
Original (N>500N>500)70.89
Newly included (100<N≤500100<N\leq 500)126.27

What happens when expanding to N>100N>100? We extended the analysis to all 19 capabilities, including 2D Ordering and Localization. The overall correlation becomes close to zero in the expanded set. However, given the substantially higher transfer variability of the newly included capabilities, we further examined the sensitivity of the correlation to high-variance targets.

Target coverageCapabilitiesPearson rr: capability meansPearson rr: source-target pairs
Original (N>500N>500)7+0.795+0.529
Expanded (N>100N>100)19-0.050-0.010
Excluding the two highest-variance targets17+0.415+0.240

Excluding Multi-view Reasoning and 2D Ordering, the two highest-variance targets, yields positive correlations of r=0.415r=0.415 across capability means and r=0.240r=0.240 across source-target pairs. We additionally examined pairs with statistically significant transfer effects across all N>100 capabilities (including both positive and negative transfer). Their correlation is r=0.18r=0.18 across 56 pairs, increasing to r=0.55r=0.55 across 43 pairs when excluding 2D Ordering. These supplementary analyses suggest that high-variability targets may contribute to the instability of the expanded correlation. We have added the expanded analysis and its variance diagnostics to Appendix A.3, clarifying the scope of our gradient-alignment findings.

Q2. Understanding negative transfer

We conducted additional source-domain controls to investigate whether the negative transfer on Visual Correspondence and Multi-view Reasoning arises from objective conflict or domain-induced representation forgetting. Specifically, we reconstructed 12 I2I objectives using alternative image sources (nine on COCO and three on Taskonomy) while keeping the downstream I2T setup fixed, and measured their transfer gains.

Negative transfer persists across source domains, although its magnitude changes. Replacing the source images does not eliminate the negative transfer. Among the 12 source-swapped objectives, 9 still produce negative transfer on Visual Correspondence and 11 on Multi-view Reasoning.

Target capabilityOriginal transfer (pp)Source-swapped transfer (pp)
Visual Correspondence-2.59-1.76
Multi-view Reasoning-4.50-2.27

These results indicate that source-domain choice affects the severity of negative transfer, but the negative transfer cannot be attributed solely to the original source datasets. Even when changing to more diverse data the negative transfer still exists. We have added the complete source-by-source transfer results and source-domain replacements to Appendix A.2, along with an expanded discussion of negative transfer.

Q3. Loss normalization under changing I2I:I2T batch ratios

Our implementation already normalizes the two objectives separately. At each Mixed update, we average I2I MSE over the I2I batch and I2T cross-entropy over the I2T batch, then add the two means with coefficients 1.0 and 1.0. We have made the reduction and coefficients explicit in Sec. 3 and Appendix B.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.