- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper systematically investigates the interactions among language modeling, image understanding, and image generation during unified pretraining. Combining training mixture experiments on real-world data with concept removal experiments on synthetic CLEVR data, the authors analyze the directionality and concept dependence of knowledge transfer across tasks. They further examine how data complexity, parameter sharing, and the timing of visual integration affect cross-modal synergy. The paper reports asymmetric knowledge transfer, the role of modality-specific FFNs, and a “vision laziness” phenomenon associated with delayed visual integration. Based on these findings, the authors propose a training mixture that prioritizes language and visual understanding and validate the design choices using MoE models with 13.5B total parameters trained on 2T tokens. The main contributions lie in the systematic empirical analysis and the resulting recommendations for unified multimodal pretraining.
Critical strengths
The question-driven structure and clearly stated findings make the connections between research questions, experiments, and conclusions easy to follow. The analyses of knowledge transfer, modality synergy, and unification timing provide insightful perspectives on unified multimodal pretraining. Experiments spanning real and synthetic data, multiple visual representations, and larger models provide broad empirical support for the main findings.
Critical weaknesses
W1. The paper should clarify what new mechanistic insights its vision-laziness analysis provides beyond established early-fusion practices such as those adopted in Kimi K3 [Kimi K3].
W2. The architectural comparisons insufficiently control for capacity and routing differences or clarify how split_ffn maps to the final MoE, limiting attribution of the performance gains.
W3. Data complexity lacks a reproducible quantitative definition, and its analysis lacks a directly validated connection to the final pretraining recipe.
W4. The spatial-relation gains in Figure 5 (5 percentage points on 100 samples per concept) need confidence intervals and evaluator reliability checks, while Figure 6 should extend its recovery analysis to spatial relations, size, and counting.
W5. The correspondence between experimental settings and results in Figure 7 is unclear, and “5% compute” should be explicitly distinguished from the generation token allocation.
[Kimi K3] Bai, Kimi Team Yifan et al. “Kimi K3: Open Frontier Intelligence.” (2026).
Questions for authors
Q1. Are the structural-concept transfer gains robust, and can the authors provide fine-tuning recovery results and validate the evaluator’s reliability?
Q2. How is complexity quantified, and how are the conditions and transition point between modality synergy and competition determined?
Q3. How do the complexity findings concretely inform real-data selection, training mixtures, or parameter allocation?
Q4. What independently testable mechanistic insights does this work add beyond existing native multimodal studies such as Kimi K2.5 [Kimi K2.5] and K3 [Kimi K3]?
Q5. How does split_ffn map to expert allocation and routing in the final MoE, and can its benefits be verified through comparisons with comparable capacity and compute?
Q6. What do the red X markers in Figure 9 indicate, and, if they denote clipping or anomalous results, can the actual values and criteria be reported?
[Kimi K2.5] Bai, Kimi Team Yifan et al. “Kimi K2.5: Visual Agentic Intelligence.” (2026).
[Kimi K3] Bai, Kimi Team Yifan et al. “Kimi K3: Open Frontier Intelligence.” (2026).
4 comments
Point 5 CLEVR uncertainty and evaluator reliability (W4 and Q1)
Q Are the structural concept transfer gains robust? In particular, can confidence intervals and checks of evaluator reliability support the spatial gains of five percentage points measured on 100 samples per concept?
We thank the reviewer for this valuable suggestion. We agree that confidence intervals and evaluator calibration would strengthen the assessment of the spatial gains of five percentage points. We are unable to add further experiments in this revision.
Our experiments on structural concepts support the conclusion that understanding can transfer these capabilities to generation. Independent work reports similar findings. For example, Transferability Between Understanding and Generation in Unified Multimodal Models demonstrates understanding-to-generation transfer for counting and spatial relations. Similar observations have also appeared in a few recent works that we do not identify here to preserve anonymity. For evaluator reliability, VQA ground truth comes from the metadata used to generate CLEVR scenes. Generation evaluation checks explicit visual properties. We manually checked the judge's decisions on more than 100 CLEVR samples and found them reliable on these simple, constrained tasks.
Point 6 The scope of concept recovery experiments (W4 and Q1)
Q Can the recovery analysis after fine-tuning in Fig. 6 be extended beyond color and shape to spatial relations, size, and counting, to test whether those concepts also acquire usable priors?
Yes. We performed these recovery experiments, and spatial relations, size, and counting also show usable priors. This is consistent with their understanding-to-generation zero-shot transfer. Zero-shot transfer is the more demanding test: the concept must be usable immediately, whereas a prior can assist subsequent learning. Figure 6 focuses on color and shape to highlight the more informative case: zero-shot transfer fails, yet exposure through the other objective still leaves a useful prior. The structural recovery results were omitted to keep the presentation concise.
Point 7 The settings and matching values in Figure 7 (W5)
Q Which experimental settings correspond to the results in Fig. 7(a) and Fig. 7(b)? How do the split_ffn architecture results relate to the experiments with simplified data, given the DCLM/SSTK description in Sec. 4.2?
Thank you. As clarified in the General Response, the matching split_ffn values in Fig. 7(a) and Fig. 7(b) come from identical experimental configurations. The simplified data settings and 50B/50B token budget are now explicitly specified. The omission was in the experimental description; the reported results and conclusions are unchanged.
Point 8 Five percent tokens versus five percent compute (W5)
Q Does the claim of 5% compute refer to training tokens for visual generation or to an actual share of total training FLOPs?
The 5% refers to training tokens for visual generation. The wording should read 5% of training tokens. L70/U25/G5 uses five times fewer generation tokens than L50/U25/G25 while improving the alignment metrics and retaining competitive FID.
Point 9 The red X markers in Figure 9 (Q6)
Q What do the red X markers in Fig. 9 indicate? If they represent clipping or anomalous results, what are the underlying values and the criteria used to mark them?
The red X markers indicate severe performance collapse, with metric values far beyond the plotted axis range; for example, language perplexity can reach hundreds.
Thank you again for your helpful review and time. We welcome any further questions or discussion and look forward to your feedback.
The authors' responses have adequately addressed my previous concerns, and I have no further questions. I appreciate the authors' thoughtful and thorough clarifications.
I believe this work has meaningful research value and contributes to the ongoing development of unified multimodal large language models. Therefore, I have decided to increase my overall rating to 4. However, given my relatively limited expertise in multimodal large language models, I will keep my confidence rating unchanged. Good Luck!
We thank the reviewer for taking the time to read our rebuttal and are glad that your concerns have been addressed. We also appreciate your willingness to update your assessment and your positive recognition of our work.
Thank you once again for your thoughtful review and helpful suggestions, which have greatly improved this paper!
We thank the reviewer for the constructive comments and the positive assessment of the paper's clarity and empirical breadth. We respond to the concerns point by point below.
Point 1 Vision laziness and prior early fusion studies (W1 and Q4)
Q What independently testable mechanistic insights does the analysis of vision laziness add beyond the early-fusion studies and practices in Kimi K2.5 and Kimi K3?
Thank you. Kimi K2.5 demonstrates an advantage of early fusion during continued pretraining with fixed text and visual budgets, while Kimi K3 adopts native multimodal training from scratch. We extend these findings by studying the problem from three complementary angles.
Figure 8 examines how much warmup on language alone is useful under a fixed total training budget. Figure 9 compares joint training with all six sequential orders of the three objectives, holding total tokens and modality allocation fixed, and demonstrates the advantage of training them together. Figure 10 quantifies how increasing language warmup affects subsequent visual pathway activity under the same amount of joint training, directly examining the development of vision laziness.
These experiments examine the training dynamics behind the advantage of early fusion.
Point 2 Capacity controls and the mapping from split FFNs to MoE (W2 and Q5)
Q How does split_ffn map to expert allocation and routing in the final MoE? Are its benefits verified through architectural comparisons with comparable model capacity and training compute?
The scaled MoE has 256 experts and 16 active per token, with two experts designated for language and vision and the remaining 14 active experts dynamically routed. Attention and normalization remain shared. This implements modality-specific capacity within the shared backbone.
The component ablation uses identical data and token budgets. Fully separated split_all approximately recovers unimodal performance: Delta PPL = −0.005 and Delta diffusion loss approximately −0.002. Selective sharing in split_ffn gives larger gains of −0.211 and −0.0168. More separated capacity alone does not explain the additional synergy.
For the scaled comparison, we deliberately selected a 3.5B dense model to account for the lower hardware utilization of the MoE (13.5B total, 1.5B active parameters) and obtain comparable training GPU hours. After basic MFU optimizations, both models used approximately 40,000 to 50,000 GPU hours on identical H200 hardware. Both use the same budget of 2T tokens, L70/U25/G5 mixture, and schedule for early unification. The matching criterion is therefore total training cost.
Point 3 The meaning and quantification of data difficulty (W3 and Q2)
Q What is the reproducible definition of data complexity, and how are the conditions and transition point between modality synergy and competition determined?
Here, complexity refers to empirical learning difficulty under the fixed training setup, assessed through diffusion loss and convergence. The study uses an approximate ordering of tasks, as stated in the original description, rather than a universal numerical complexity score.
Synergy and competition are measured by changes in performance on the target task relative to its unimodal baseline, with the target distribution held fixed. In Fig. 7(a), simple visual tasks improve language PPL, while video and SSTK increase it by +0.052 and +0.075. The observed transition is this change of sign across the tested conditions.
Point 4 How the difficulty study informs the final recipe (W3 and Q3)
Q How do the complexity findings concretely inform selection of real training data, training mixture weights, or parameter allocation in the final recipe?
Thank you. Simpler tasks make cross-modal synergy easier to observe and help distinguish the effects of different ways of sharing parameters. For example, pure backgrounds benefit language more than SSTK images. These experiments reveal the benefit of sharing attention and normalization while separating modality-specific FFNs, informing the final architecture.
Learning difficulty alone does not determine selection of real training data or mixture weights: practical pretraining prioritizes data quality and diversity. The L70/U25/G5 allocation follows the directional transfer findings and explicit mixture search, and is then evaluated at scale.