acceptodds
Under review as a conference paper at ICLR 2027

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Abstract

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data “complexity” largely determines whether modalities are synergistic, identify architectural choices that promote synergy—such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We strongly recommend a visit to our project page (https://anonymouspaperweb.github.io/) for an interactive reading.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal pretraining, unified models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7T3dOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically investigates the interactions among language modeling, image understanding, and image generation during unified pretraining. Combining training mixture experiments on real-world data with concept removal experiments on synthetic CLEVR data, the authors analyze the directionality and concept dependence of knowledge transfer across tasks. They further examine how data complexity, parameter sharing, and the timing of visual integration affect cross-modal synergy. The paper reports asymmetric knowledge transfer, the role of modality-specific FFNs, and a “vision laziness” phenomenon associated with delayed visual integration. Based on these findings, the authors propose a training mixture that prioritizes language and visual understanding and validate the design choices using MoE models with 13.5B total parameters trained on 2T tokens. The main contributions lie in the systematic empirical analysis and the resulting recommendations for unified multimodal pretraining.

Critical strengths

The question-driven structure and clearly stated findings make the connections between research questions, experiments, and conclusions easy to follow. The analyses of knowledge transfer, modality synergy, and unification timing provide insightful perspectives on unified multimodal pretraining. Experiments spanning real and synthetic data, multiple visual representations, and larger models provide broad empirical support for the main findings.

Critical weaknesses

W1. The paper should clarify what new mechanistic insights its vision-laziness analysis provides beyond established early-fusion practices such as those adopted in Kimi K3 [Kimi K3].

W2. The architectural comparisons insufficiently control for capacity and routing differences or clarify how split_ffn maps to the final MoE, limiting attribution of the performance gains.

W3. Data complexity lacks a reproducible quantitative definition, and its analysis lacks a directly validated connection to the final pretraining recipe.

W4. The spatial-relation gains in Figure 5 (5 percentage points on 100 samples per concept) need confidence intervals and evaluator reliability checks, while Figure 6 should extend its recovery analysis to spatial relations, size, and counting.

W5. The correspondence between experimental settings and results in Figure 7 is unclear, and “5% compute” should be explicitly distinguished from the generation token allocation.

[Kimi K3] Bai, Kimi Team Yifan et al. “Kimi K3: Open Frontier Intelligence.” (2026).

Questions for authors

Q1. Are the structural-concept transfer gains robust, and can the authors provide fine-tuning recovery results and validate the evaluator’s reliability?

Q2. How is complexity quantified, and how are the conditions and transition point between modality synergy and competition determined?

Q3. How do the complexity findings concretely inform real-data selection, training mixtures, or parameter allocation?

Q4. What independently testable mechanistic insights does this work add beyond existing native multimodal studies such as Kimi K2.5 [Kimi K2.5] and K3 [Kimi K3]?

Q5. How does split_ffn map to expert allocation and routing in the final MoE, and can its benefits be verified through comparisons with comparable capacity and compute?

Q6. What do the red X markers in Figure 9 indicate, and, if they denote clipping or anomalous results, can the actual values and criteria be reported?

[Kimi K2.5] Bai, Kimi Team Yifan et al. “Kimi K2.5: Visual Agentic Intelligence.” (2026).

[Kimi K3] Bai, Kimi Team Yifan et al. “Kimi K3: Open Frontier Intelligence.” (2026).

4 comments

We thank the reviewer for the constructive comments and the positive assessment of the paper's clarity and empirical breadth. We respond to the concerns point by point below.

Point 1 Vision laziness and prior early fusion studies (W1 and Q4)

Q What independently testable mechanistic insights does the analysis of vision laziness add beyond the early-fusion studies and practices in Kimi K2.5 and Kimi K3?

Thank you. Kimi K2.5 demonstrates an advantage of early fusion during continued pretraining with fixed text and visual budgets, while Kimi K3 adopts native multimodal training from scratch. We extend these findings by studying the problem from three complementary angles.

Figure 8 examines how much warmup on language alone is useful under a fixed total training budget. Figure 9 compares joint training with all six sequential orders of the three objectives, holding total tokens and modality allocation fixed, and demonstrates the advantage of training them together. Figure 10 quantifies how increasing language warmup affects subsequent visual pathway activity under the same amount of joint training, directly examining the development of vision laziness.

These experiments examine the training dynamics behind the advantage of early fusion.

Point 2 Capacity controls and the mapping from split FFNs to MoE (W2 and Q5)

Q How does split_ffn map to expert allocation and routing in the final MoE? Are its benefits verified through architectural comparisons with comparable model capacity and training compute?

The scaled MoE has 256 experts and 16 active per token, with two experts designated for language and vision and the remaining 14 active experts dynamically routed. Attention and normalization remain shared. This implements modality-specific capacity within the shared backbone.

The component ablation uses identical data and token budgets. Fully separated split_all approximately recovers unimodal performance: Delta PPL = −0.005 and Delta diffusion loss approximately −0.002. Selective sharing in split_ffn gives larger gains of −0.211 and −0.0168. More separated capacity alone does not explain the additional synergy.

For the scaled comparison, we deliberately selected a 3.5B dense model to account for the lower hardware utilization of the MoE (13.5B total, 1.5B active parameters) and obtain comparable training GPU hours. After basic MFU optimizations, both models used approximately 40,000 to 50,000 GPU hours on identical H200 hardware. Both use the same budget of 2T tokens, L70/U25/G5 mixture, and schedule for early unification. The matching criterion is therefore total training cost.

Point 3 The meaning and quantification of data difficulty (W3 and Q2)

Q What is the reproducible definition of data complexity, and how are the conditions and transition point between modality synergy and competition determined?

Here, complexity refers to empirical learning difficulty under the fixed training setup, assessed through diffusion loss and convergence. The study uses an approximate ordering of tasks, as stated in the original description, rather than a universal numerical complexity score.

Synergy and competition are measured by changes in performance on the target task relative to its unimodal baseline, with the target distribution held fixed. In Fig. 7(a), simple visual tasks improve language PPL, while video and SSTK increase it by +0.052 and +0.075. The observed transition is this change of sign across the tested conditions.

Point 4 How the difficulty study informs the final recipe (W3 and Q3)

Q How do the complexity findings concretely inform selection of real training data, training mixture weights, or parameter allocation in the final recipe?

Thank you. Simpler tasks make cross-modal synergy easier to observe and help distinguish the effects of different ways of sharing parameters. For example, pure backgrounds benefit language more than SSTK images. These experiments reveal the benefit of sharing attention and normalization while separating modality-specific FFNs, informing the final architecture.

Learning difficulty alone does not determine selection of real training data or mixture weights: practical pretraining prioritizes data quality and diversity. The L70/U25/G5 allocation follows the directional transfer findings and explicit mixture search, and is then evaluated at scale.

Point 5 CLEVR uncertainty and evaluator reliability (W4 and Q1)

Q Are the structural concept transfer gains robust? In particular, can confidence intervals and checks of evaluator reliability support the spatial gains of five percentage points measured on 100 samples per concept?

We thank the reviewer for this valuable suggestion. We agree that confidence intervals and evaluator calibration would strengthen the assessment of the spatial gains of five percentage points. We are unable to add further experiments in this revision.

Our experiments on structural concepts support the conclusion that understanding can transfer these capabilities to generation. Independent work reports similar findings. For example, Transferability Between Understanding and Generation in Unified Multimodal Models demonstrates understanding-to-generation transfer for counting and spatial relations. Similar observations have also appeared in a few recent works that we do not identify here to preserve anonymity. For evaluator reliability, VQA ground truth comes from the metadata used to generate CLEVR scenes. Generation evaluation checks explicit visual properties. We manually checked the judge's decisions on more than 100 CLEVR samples and found them reliable on these simple, constrained tasks.

Point 6 The scope of concept recovery experiments (W4 and Q1)

Q Can the recovery analysis after fine-tuning in Fig. 6 be extended beyond color and shape to spatial relations, size, and counting, to test whether those concepts also acquire usable priors?

Yes. We performed these recovery experiments, and spatial relations, size, and counting also show usable priors. This is consistent with their understanding-to-generation zero-shot transfer. Zero-shot transfer is the more demanding test: the concept must be usable immediately, whereas a prior can assist subsequent learning. Figure 6 focuses on color and shape to highlight the more informative case: zero-shot transfer fails, yet exposure through the other objective still leaves a useful prior. The structural recovery results were omitted to keep the presentation concise.

Point 7 The settings and matching values in Figure 7 (W5)

Q Which experimental settings correspond to the results in Fig. 7(a) and Fig. 7(b)? How do the split_ffn architecture results relate to the experiments with simplified data, given the DCLM/SSTK description in Sec. 4.2?

Thank you. As clarified in the General Response, the matching split_ffn values in Fig. 7(a) and Fig. 7(b) come from identical experimental configurations. The simplified data settings and 50B/50B token budget are now explicitly specified. The omission was in the experimental description; the reported results and conclusions are unchanged.

Point 8 Five percent tokens versus five percent compute (W5)

Q Does the claim of 5% compute refer to training tokens for visual generation or to an actual share of total training FLOPs?

The 5% refers to training tokens for visual generation. The wording should read 5% of training tokens. L70/U25/G5 uses five times fewer generation tokens than L50/U25/G25 while improving the alignment metrics and retaining competitive FID.

Point 9 The red X markers in Figure 9 (Q6)

Q What do the red X markers in Fig. 9 indicate? If they represent clipping or anomalous results, what are the underlying values and the criteria used to mark them?

The red X markers indicate severe performance collapse, with metric values far beyond the plotted axis range; for example, language perplexity can reach hundreds.

Thank you again for your helpful review and time. We welcome any further questions or discussion and look forward to your feedback.

The authors' responses have adequately addressed my previous concerns, and I have no further questions. I appreciate the authors' thoughtful and thorough clarifications.

I believe this work has meaningful research value and contributes to the ongoing development of unified multimodal large language models. Therefore, I have decided to increase my overall rating to 4. However, given my relatively limited expertise in multimodal large language models, I will keep my confidence rating unchanged. Good Luck!

We thank the reviewer for taking the time to read our rebuttal and are glad that your concerns have been addressed. We also appreciate your willingness to update your assessment and your positive recognition of our work.

Thank you once again for your thoughtful review and helpful suggestions, which have greatly improved this paper!

Reviewer_VPveOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper empirically studies unified multimodal pretraining, concentrating on knowledge flow, modality synergy and unification timing across language, visual understanding and visual generation. Through real-world and synthetic experiments, the author report asymmetric cross-modal transfer, architecture choices that promote synergy, and an advantage of early joint training compared to late or sequential training. Based on these findings, the authors propose a 70/25/5 language/understanding/generation data mixture recipe and evaluate several of these choices at scale with 13.5B MoE models trained on 2T tokens.

Critical strengths

This research provides a comprehensive empirical study with practical relevance. The authors examine several crucial points of unified multimodal pretraining, through extensive experiments. Controlled synthetic studies and large-scale runs are included, which is the main strength. Finally this paper provides some useful empirical insights on how to train unified models better.

Critical weaknesses
  1. The authors overstate mechanistic interpretations and generality of their findings. The paper turns empirical observations into broad claims on underlying mechanisms or necessities, such as language being a "universal booster" of visual capabilities and the "necessity" of early unification, even though other explanations could account for the same results.
  2. There are insufficient controls and inconsistencies in key experiments. In Fig. 8, since delaying unification also reduces the amount of visual training data, the drop in visual performance cannot be attributed to timing alone. In Fig. 7(b), the split_ffn results match the background-image and simple-text results in Fig. 7(a), even though Section 4.2 states that this experiment uses DCLM and SSTK. This leaves the actual experimental setup unclear.
Questions for authors
  1. Which datasets and baselines were used for Fig. 7(b)? The split_ffn results match the simple-data settings in Fig. 7(a).
  2. Could the authors provide the updated results with matched visual training data, to isolate the effect of unification timing?
3 comments

We thank the reviewer for the time and effort spent reviewing our work. Below, we address the concerns point by point and clarify the relevant experimental settings and controls.

Point 1 The scope of mechanistic and generality claims (W1)

Q Do the claims that language is a universal booster and that early unification is necessary exceed the evidence? What controls distinguish the proposed mechanisms from alternative explanations?

Thank you for raising this concern. Our conclusions draw on several controlled experiments that examine these effects from different perspectives. The benefits of language training are examined across four visual representations. Unification timing is studied through language warmup sweeps, all six sequential training orders, experiments with matched visual exposure, and validation at 2T tokens. Together, these experiments examine both performance and the development of the visual pathway.

We consider this combined evidence sufficient to support our conclusions under the studied conditions. No single experiment rules out every alternative explanation. The combined evidence from these controlled comparisons supports our conclusions under the studied conditions. Beyond Language Modeling, Scaling Laws for Native Multimodal Models, and Kimi K2.5 also report findings supporting joint or early multimodal training. Our study extends this line of work through multiple investigations. Together, these investigations help explain why timing matters and how delayed integration affects visual learning.

Point 2 Clarifying the architectural settings in Figure 7 (W2 and Q1)

Q Which datasets and unimodal baselines are used in Fig. 7(b)? Why do its split_ffn values match the settings with background images and simple text in Fig. 7(a), although Sec. 4.2 says DCLM and SSTK?

As clarified in the General Response, Fig. 7(b) uses DCLM with pure backgrounds, and simple repetitive text with SSTK. The corresponding unimodal baselines are language training on DCLM alone and generation training on SSTK alone. The split_ffn values match Fig. 7(a) because they report the same configurations. The setting description has been corrected; the numerical results and related conclusions are unchanged.

Point 3 Visual exposure controls and unification timing (W2 and Q2)

Q Because Fig. 8 reduces visual exposure when unification is delayed, can results with matched visual training data isolate the effect of integration timing?

Yes. As clarified in the General Response, Late-Fusion is compared with Full at the same budget of 2T tokens and L70/U25/G5 allocation. Total visual exposure is identical.

Multiple experiments also hold visual exposure fixed. Figure 9 fixes both the 1T total budget and the L50/U25/G25 allocation across joint training and all six sequential orders. Figure 10 gives every checkpoint after language warmup the same joint continuation of 200B tokens, including 100B visual tokens.

Figure 8 addresses a different question: how much warmup on language alone is useful within a fixed total budget. It shows diminishing language gains despite a subsequent 50/50 mixture of language and vision. The timing conclusion rests on all three experiments.

We hope these clarifications address your concerns about soundness. If any substantive concern remains unresolved, please feel free identify the specific issue so that we can try to answer it.

Thank you for the detailed rebuttal and the manuscript revisions. I appreciate the clarification of the experimental settings in Fig. 7(b) and the corrected comparisons in Sec. 6.2. In particular, the matched visual exposure between Full and Late-Fusion provides stronger evidence for the benefits of early unification. These revisions address part of my concerns regarding the experimental setup.

However, my main concern about the strength of the mechanistic claims remains. The architectural analysis in Fig. 7(b) relies on deliberately simplified auxiliary data, which limits how broadly its conclusions about modality synergy can be generalized. Moreover, the language-data experiments in Sec. 3.1 still lack compute-matched controls, and the activation and attention analyses do not fully establish the proposed causal explanation for "vision laziness."

While the revisions clarify several experimental details and strengthen the interpretation of existing comparisons, they do not provide new controlled evidence addressing these remaining concerns. I would also have appreciated a more substantial revision of the discussion and conclusions to reflect these limitations, particularly as the paper still uses strong statements such as "prove the necessity of early unification."

Overall, I acknowledge the improvements but remain unconvinced that the evidence fully supports the paper's central mechanistic claims. I therefore maintain my original score.

Thank you for following up. We appreciate your feedback and are glad the clarifications addressed some of your concerns. Please allow us to reply for one more round regarding 3 remaining concerns.

Scope of the architecture ablations in Fig. 7(b).

In Fig. 7(b), we vary attention and FinalNorm sharing on top of split_ffn in two auxiliary-data settings selected for strong cross-modal transfer. Under those same settings, Appendix H reproduces the synergy trend with split_ffn across four visual representation designs. On real data, Beyond Language Modeling (https://arxiv.org/pdf/2603.03276, Fig. 3 and Appendix C) shows that modality-specific FFNs outperform shared FFNs at matched FLOPs. Our 2T comparison provides complementary evidence at scale. The modality-specialized MoE improves language and visual understanding over the dense baseline at comparable GPU hours, with the same token budget, data mixture, and unification schedule.

Token budgets and FLOPs. in 3.1

For a fixed model size, matching total tokens approximately matches compute FLOPs. Sec. 3.1.1 deliberately holds visual exposure fixed at 50B tokens while varying the amount of added language data. Table 2 complements this design with a comparison at a fixed total budget of 1T tokens. Holding the understanding-to-generation ratio at 1:1, increasing the language share from 10% to 70% reduces visual exposure from 900B to 300B tokens while improving average visual understanding from 32.7 to 38.1 and GenEval from 0.148 to 0.219.

Vision laziness conclusion too strong.

We use vision laziness to describe the observed reduction in visual pathway activity. All four probes in Fig. 10 show the same overall trend with longer language warmup, with every condition receiving the same 200B joint continuation. We agree that “prove the necessity” may overstates the result and have revised the conclusion to read “Early unification mitigates vision laziness.”

Reviewer_5ohwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic empirical study of multimodal pretraining focusing on the interaction between language, visual understanding, and visual generation. The authors study four main questions: how knowledge transfers between modalities, when modalities compete or benefit each other, how early they should be jointly trained, and how to allocate training data efficiently. The experiments use a architecture combining autoregressive language modeling with flow matching for visual generation, primarily using SigLIP-2 representations. The study includes synthetic CLEVR experiments, large scale data mixture ablations, different parameter-sharing strategies, and several visual tokenization methods.

The authors find that language and visual understanding generally benefit visual generation, while visual generation has limited direct benefits for the other tasks. They also report that separating modality specific FFNs while sharing attention and normalization improves joint training, and that introducing vision early gives better results than delayed or sequential training. Based on these observations they propose an asymmetric training mixture with 70% language, 25% visual understanding, and 5% visual generation. The findings are further evaluated using 13.5B MoE models trained on 2T tokens. While the experiments offer several useful observations, I think some of the stronger causal conclusions are not fully supported by the experimental design.

Critical strengths

Broad empirical studies: The paper studies multimodal pretraining from several complementary perspectives, including knowledge transfer, architectural parameter sharing, training schedules, and data mixtures. The combination of synthetic experiments and large-scale pretraining is valuable. I particularly appreciate that the authors go beyond a single benchmark and try to understand why certain design choices work. The 21-run mixture study and validation at 2T tokens make the findings more practically relevant.

Interesting insights on concept level transfer and training recipes: The CLEVR experiments provide an interesting way to study which visual concepts transfer between understanding and generation. The observation that structural concepts may transfer differently from low-level attributes is worth investigating. I also find the asymmetric data mixture results useful from a systems perspective. Achieving competitive generative alignment with a relatively small allocation of generatin tokens is potentialy important, even if the efficiency claims need more careful analysis.

Critical weaknesses
  1. Several key causal claims are not adequately supported by controlled comparisons in my opinion: This is my biggest concern. In Section 3.1, the authors keep 50B tokens of one modality fixed while adding up to 200B tokens from another. This increases the total training budget from 50B to 250B tokens. Without compute-matched or token-matched controls, it is difficult to conclude that the improvements are specifically caused by knowledge transfer. A similar problem appears in Section 5.1. The total budget is fixed at 1T tokens, but increasing the language only warmup from 0B to 800B reduces the visual training allocation from 500B to 100B tokens. The observed visual performance degradation could therefore be explained by substantially less visual training, rather than the proposed vision laziness mechanism. The activation and attention analyses are suggestive, but do not establish this causal explanation. The large scale early vs late comparison also changes the amount of visual exposure. I have related concerns about the architecture study. Comparing a 13.5B MoE with a 3.5B dense model does not cleanly isolate the benefits of modality-specific FFNs, since model capacity and active computation differ. The small-scale parameter-sharing ablation is more useful, but the results need to be reconciled with the stated data setup. Overall, I think the paper frequently interprets empirical correlations as stronger mechanistic findings than the controls justify.
  1. The novelty and efficiency claims are overstated relative to prior work and the reported results. The paper has significant overlap with existing work, particularly Beyond Language Modeling: An Exploration of Multimodal Pretraining, which already studies multimodal synergy, modality-specific FFNs, unified visual representations, and MoE architectures. Prior work on native multimodal scaling has also examined early versus late fusion, while recent work directly investigates capability-level transfer between understanding and generation. The authors cite much of the related literature, but I think the paper needs a clearer explanation of what new conclusions it establishes beyond these studies.

I am also not fully convinced by the claim that visual generation requires only 5% of the compute budget. The experiments demonstrate that 5% of training tokens can be assigned to visual generation, which is not necessarily equivalent to 5% of total training FLOPs. The large-scale results are also somewhat mixed. Compared with the balanced recipe, GenEval improves from 0.467 to 0.482, but diffusion loss worsens from 0.261 to 0.272, and the reported FID slightly worse (5.13 to 5.23). This is still a potentially useful tradeoff, but it does not establish uniformly comparable generation quality.

Questions for authors
  1. In Section 3.1, the modality transfer experiments appear to add training tokens rather than keeping the total training budget fixed. It would be useful to see iso-token or iso-FLOP comparisons including a baseline where the additional compute is allocated to the original modality instead. Without these controls it's hard to tell how much of the improvement comes from genuine cross-modal transfer versus simply training longer.

  2. Figure 8 shows that the amount of visual training decreases substantially as language warmup increases. Can the authors compare early and late visual integration while keeping the number of visual training tokens and total compute fixed? It would also be helpful to see more direct evidence that late-trained models actually rely less on visual information. Maybe through image corruption or visual token ablation experiments. The smaller activation norms alone don't seem sufficient to support this conclusion.

  3. It's unclear how much of the improvement from split FFNs comes from the architecture itself versus the additional parameter capacity. Comparisons with matched active parameters and total capacity at similar FLOPs would help isolate the contribution. Also Section 4.2 mentions using DCLM and SSTK for the architecture ablation but some of the reported language modeling results in Figure 7(b) appear very similar to those in Figure 7(a), despite the different experimental settings described. Can the authors clarify if these experiments are using the same settings or if there is some overlap in the reported results?

  4. The efficiency claims around the different training recipes would be more convincing with actual training FLOPs that account for modality specific token processing and generation costs. I'm also curious how the authors interpret the worse diffusion loss and FID despite improvements in other generation quality benchmarks. It would be good to see comparisons at matched compute and similar generation quality to better understand if there is a real effeciency advantage here.

  5. It would be useful to understand how consistent the CLEVR transfer results are across random seeds and different visual backbones. Since the default setup uses a frozen pretrained SigLIP-2 encoder it's not entirely clear how much of the observed transfer comes from multimodal pretraining versus representations already learned by the encoder. Another concern is that Qwen3-VL is used both to generate synthetic captions and to evaluate generated images which could introduce correlated biases. An independent evaluator or some judge calibration would help address this.

3 comments

We thank the reviewer for the detailed comments and for recognizing the breadth and practical value of the study. We respond to the concerns point by point below.

Point 1 Knowledge transfer versus simply training longer (W1 and Q1)

Q In Sec. 3.1, adding auxiliary tokens raises the total budget from 50B to as much as 250B. Can iso-token or iso-FLOP controls allocate the additional budget to the original modality, to distinguish knowledge transfer from simply training longer?

Table 2 in Appendix B already compares allocations at a fixed total budget of 1T tokens. In the “Fix MM” sweep, understanding and generation maintain a 1:1 ratio while the budget is reallocated between language and vision.

Changing from L10/U45/G45 to L70/U15/G15 reduces visual training from 900B to 300B tokens, yet improves average visual understanding from 32.7 to 38.1, DPG from 0.326 to 0.399, GenEval from 0.148 to 0.219, and CLIP similarity from 0.256 to 0.273.

The models therefore train for the same number of tokens, yet the model receiving one-third as many visual tokens performs better on visual understanding and generation alignment. These results show that allocating budget to language can benefit visual capabilities more than additional visual training. The gain in this comparison cannot be attributed to an increase in total training tokens.

Point 2 Timing and fixed visual exposure (W1 and Q2)

Q Can early and late integration be compared with both total compute and the number of visual training tokens fixed? Does the scaled comparison of early and late integration also match visual exposure, rather than inheriting the exposure change in Fig. 8?

As clarified in the General Response, the scaled comparison between Full and Late-Fusion uses matched total and visual-token budgets.

Figures 8–10 address three distinct questions. Figure 8 studies the budget trade-off of language warmup. Figure 9 tests joint training against all six sequential orders with total tokens and modality counts fixed. Figure 10 measures how language warmup affects subsequent visual pathway activity after identical joint training, including 100B visual tokens. Figures 9 and 10 both fix visual exposure. All three experiments support the timing conclusion.

Point 3 Evidence for reduced reliance on vision (W1 and Q2)

Q Do the activation and attention analyses demonstrate that models trained with late integration rely less on visual information? Could image corruption or ablation of visual tokens provide more direct evidence than smaller activation norms?

We use four complementary measurements to characterize vision laziness from different angles: visual activations during training, image wrapper embeddings, visual activations during inference, and attention to image tokens, all under the same amount of joint training. While these do not exhaust every possible analysis, all four show a consistent trend: longer language warmup leads to reduced visual pathway activity. We consider the agreement across all four measurements strong evidence of vision laziness.

Figure 19 additionally compares the generated outputs. Early fusion better preserves requested attributes and geometric structure. For example, it renders a cloud in the shape of a Bitcoin symbol, while late fusion produces an unstructured cloud. These differences in the generated images support the interpretation of vision laziness and agree with the internal measurements.

Point 4 Capacity and practical MoE versus dense comparisons (W1 and Q3)

Q How much of the gain from split_ffn comes from additional capacity? Can comparisons match active parameters and total capacity at similar FLOPs, and how should the 13.5B MoE versus 3.5B dense comparison be interpreted?

Additional separated capacity alone does not explain the component ablation results. Fully separated split_all approximately recovers the unimodal baselines (Delta PPL = −0.005; Delta diffusion loss approximately −0.002), whereas split_ffn yields larger gains of −0.211 and −0.0168 under the same data and token budget.

For the scaled comparison, the MoE has 13.5B total and 1.5B active parameters. Comparing it with a roughly 1.5B dense model would approximately match FLOPs, but would not match GPU hours because MoE training generally achieves lower MFU than dense training. We therefore chose a 3.5B dense baseline to match training GPU hours.

After basic MFU optimizations, both models use approximately 40,000 to 50,000 GPU hours on identical H200 hardware, with the same token budget, recipe, and schedule for early unification. We compare total training cost, accounting for the utilization difference.

Point 5 The matching Figure 7 results (W1 and Q3)

Q Does Fig. 7(b) actually use DCLM and SSTK, as Sec. 4.2 states? Why do its split_ffn results match the points with simplified data in Fig. 7(a), and are those experimental settings shared?

The matching split_ffn values are correct: they come from the same experimental configurations in Fig. 7(a) and Fig. 7(b). The actual data settings are specified in the General Response. We have corrected the omitted setting description; the reported numbers and conclusions are unchanged.

Point 6 New findings and practical training insights (W2)

Q Which new findings go beyond Beyond Language Modeling and earlier studies of multimodal synergy, modality-specific FFNs, unified representations, MoE, early fusion, and transfer of individual capabilities between understanding and generation?

With full respect, we disagree with the assessment that our novelty claims are overstated relative to prior work. Our contribution is a detailed investigation of why modalities cooperate or interfere and how to improve unified joint training.

Beyond Language Modeling studies representations, data, architectures, and scaling. Our study adds distinct questions and conclusions: directional transfer among language, understanding, and generation; the distinction between zero-shot concept transfer and recoverable priors; how learning difficulty changes synergy and which shared components enable it; and how unification timing and training order affect visual pathway development. Together, these investigations of knowledge flow, modality synergy, and unification timing yield a practical pretraining recipe, which we validate at scale.

Beyond Language Modeling does not report these experiments on concept removal and recovery, these controls for unification timing, or the resulting practical training recommendations. This is the scientific understanding and practical know-how our physics study contributes.

Point 7 Generation token allocation and the meaning of efficiency (W3 and Q4)

Q Is 5% generation-token allocation actually 5% of total training FLOPs? Does compute accounting include modality-specific token processing and generation costs?

Thank you. The 5% refers to the allocation of generation tokens. It is not a measured share of total training FLOPs. The wording should read 5% of training tokens. At the same 2T total token budget, L70/U25/G5 uses five times fewer generation tokens than L50/U

Point 8 Generation alignment versus diffusion loss and FID (W3 and Q4)

Q How do you interpret higher diffusion loss and slightly worse FID despite better GenEval and other alignment scores? Does the comparison demonstrate an efficiency advantage at matched compute and similar generation quality?

Yes. With the same model and a total budget of 2T tokens, the asymmetric recipe improves multiple generation benchmarks while using five times fewer generation tokens. GenEval improves from 0.467 to 0.482, DPG from 0.676 to 0.689, and CLIP similarity from 0.310 to 0.312.

FID, evaluated on 50,000 images, changes only from 5.131 to 5.234, a difference of 0.103, or about 2%. The consistent gains across alignment and compositional benchmarks, together with this small FID difference, support an efficiency advantage with similar generation quality.

Point 9 CLEVR seeds backbones and pretrained visual representations (Q5)

Q How consistent are the CLEVR transfer findings across random seeds and visual backbones? Could the observed behavior come from the frozen pretrained SigLIP-2 representation rather than the multimodal training objectives?

CLEVR uses the SigLIP-2 RAE, which performed best in our experiments. The encoder and its weights are identical across all conditions. What changes is whether understanding or generation training includes the held-out concept. The differences between conditions therefore depend on where the concept is learned, while the encoder remains fixed.

Our broader experiments cover four common settings: RAE with SigLIP-2; Raw Pixels; CLIP + VAE uses SigLIP-2 for understanding and an SD3 VAE for generation; and AR (UniTok) predicts discrete image tokens autoregressively. These cover the main choices of visual representation and generation objective.

We evaluate all four settings in both the knowledge flow analysis (Sec. 3.1 and Appendix E) and the modality synergy analysis (Sec. 4.2 and Appendix H), obtaining highly consistent conclusions across representations. The full study includes 21 training runs of 1T tokens each and several runs of 2T tokens. Although repeating every CLEVR experiment across all four settings was beyond our computational budget, the breadth and consistency of these comparisons provide strong support for the robustness and generality of our main findings.

Point 10 Using Qwen for captions and generation evaluation (Q5)

Q Does using Qwen3-VL to generate training captions and judge generated images introduce correlated biases? Can an independent evaluator or judge calibration establish reliability?

Using Qwen3-VL for both roles does not mean that evaluation measures agreement with its own captions. The judge receives the generated image and the original prompt, then checks the requested color, shape, relation, size, or count. VQA ground truth comes directly from the metadata used to generate CLEVR scenes.

As part of our original evaluation, we manually checked Qwen3-VL's judgments on more than 100 CLEVR samples and found them reliable on these simple, constrained tasks. We thank the reviewer for suggesting independent evaluation and calibration as further checks.

If any of your concerns remain unresolved after these responses, please follow up with the specific point. We would welcome the opportunity to discuss it further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the breadth and practical value of the study, its clear organization around research questions, and the combination of controlled synthetic experiments with pretraining at scale. Their comments also highlight the value of the concept transfer analysis, the asymmetric training recipe, and our effort to explain why design choices work.

Before addressing the individual concerns, we clarify two experimental settings that were incompletely or inaccurately described in the nine-page submission.

Figure 7(b). The architectural comparisons pair DCLM with pure backgrounds for language evaluation, and simple repetitive text with SSTK for generation evaluation. Each configuration uses 50B language and 50B visual tokens. For each target task, we simplify the auxiliary modality to maximize cross-modal synergy and make the effects of parameter sharing clearer.

Large-scale trainings. Full is the common reference for all three comparisons. Dense and Late-Fusion retain Full's L70/U25/G5 allocation, changing only the architecture or unification timing, respectively. Balanced Recipe changes only the allocation to L50/U25/G25. All runs use 2T tokens.

We have corrected both descriptions in the updated manuscript. The underlying experiments, reported numerical results, and associated conclusions are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.