- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the contribution of visual content in native multimodal pre-training via controlled experiments with matched IMG/PAD comparisons across model sizes and training budgets. The paper finds that the visual content improves multimodal text prediction while largely preserving text-loss scaling and language performance. They further report selective improvements in text-only tasks like spatial reasoning, particularly localization, navigation, and symbolic planning.
Critical strengths
- This paper studies an important and interesting question about the role of visual content in native multimodal pre-training.
- The experiments are comprehensive, spanning multiple model sizes and training configs.
- The task-level analysis offers some interesting insights about the role of vision in multimodal training.
Critical weaknesses
- The PAD construction may introduce adverse training effects beyond removing visual information. Some text in multimodal examples depends heavily on the accompanying image. Keeping such text while replacing the image with uninformative padding creates an under-specified prediction task, requiring the model to predict image-specific details without access to their source. This could potentially encourage unsupported predictions or introduce unfavorable learning dynamics. Consequently, the IMG–PAD difference may reflect both the benefits of visual information and the disadvantages introduced by the PAD construction.
- The advantage of native multimodal training over staged training remains unclear. The experiments demonstrate the value of visual content within a native multimodal setup, but do not establish whether introducing both modalities from the beginning provides an advantage over first training on text and subsequently incorporating multimodal data, especially for multimodal tasks.
- A text-only baseline excluding multimodal data is missing.
Questions for authors
- Is the selective transfer pattern broadly shared across benchmarks within localization and navigation, or primarily driven by particular tasks?
- Can the planning results be broken down by the number of blocks, rather than only the pooled 5–12-block category?
3 comments
Response to Reviewer uQet [2/2]
W2. Native versus staged training
The advantage of native multimodal training over staged training is unclear. The experiments do not establish whether introducing both modalities from the beginning is better than training on text first and then incorporating multimodal data.
This is an important question, though adjacent to ours: we ask what visual content contributes given native pre-training. Prior work compares early-fusion with late-fusion native architectures, the latter connecting separately pre-trained components and continuing multimodal training, and reports no inherent advantage for late fusion (Shukor et al., 2025). That comparison varies the architecture rather than the training schedule, so a text-first curriculum under a matched protocol remains open. We have added this discussion to the related-work section and do not claim our results settle it.
W3. A text-only baseline
A text-only baseline excluding multimodal data is missing.
We report one below: a text-only () run family at all six scales, with the same architecture, the same 250B text tokens and the same optimization settings. It matches the text budget of IMG/PAD, not their total budget, which includes 75B multimodal tokens. Our spatial suite (5-shot CoT, avg@4) was not run on these models, so we compare on the 15 language benchmarks. At A3B:
| Text-only | PAD | IMG | IMG$-$Text-only | |
|---|---|---|---|---|
| Mean (15) | 45.36 | 45.75 | 46.33 | |
| MBPP+ | 48.68 | 52.12 | 54.23 | |
| GSM8K | 48.75 | 53.83 | 52.39 | |
| MMLU-Pro | 24.88 | 25.49 | 27.68 | |
| SIQA | 60.24 | 58.65 | 62.95 | |
| BBH | 44.97 | 45.04 | 40.84 |
IMG is above the text-only model on 10 of the 15 benchmarks and equal on one. The clear exception is BBH, where IMG is about 4 pp below both the text-only and the PAD model. BBH aggregates a broad set of largely non-spatial tasks; its three spatial subtasks are evaluated separately in our spatial suite, where geometric shapes, despite pp at the final checkpoint, favours IMG at only 4/13 checkpoints and is not a sustained gain.
For spatial reasoning, the one benchmark available for the models is SpatialEval, under a different (1-shot) protocol and without a PAD run: at A3B, text-only scores 35.95 versus 41.43 for , and is lower at five of six scales. We treat this as corroboration only, and note in the Appendix that evaluating our spatial suite on the text-only models would fully separate the two components.
Q1. Is the selective transfer shared across benchmarks?
Is the selective transfer pattern broadly shared across benchmarks within localization and navigation, or primarily driven by particular tasks?
It is shared by most, though not all, benchmarks (new Appendix table). Mean IMG$-$PAD gap (pp) over the 13 A3B checkpoints, with the number of checkpoints favouring IMG:
- Localization. SpatialMap (12/13) and BBH colored objects (10/13) drive the gain; SpartQA is weaker at (9/13).
- Navigation. MazeNav (12/13), BBH navigate (12/13) and SpatialUnderstanding (10/13); bAbI path finding shows no gain (, 6/13), with both conditions near the floor ().
- Other groups. None of their benchmarks reaches ; examples are StepGame (9/13), FoREST (4/13) and PIQA-spatial (5/13), while Geometry3K favours IMG at 11/13.
Q2. Planning results by number of blocks
Can the planning results be broken down by the number of blocks, rather than only the pooled 5–12-block category?
Thank you for the suggestion. We report the breakdown below; the table for all six scales is in the Appendix. Final-checkpoint success (avg@4, %):
| Size | Cond. | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| A874M | IMG | 40.1 | 31.9 | 5.5 | 0.5 | 0 | 0 | 0 | 0 | 0 | 0 |
| PAD | 58.9 | 21.6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| A3B | IMG | 77.6 | 59.1 | 41.0 | 27.1 | 13.9 | 12.5 | 16.9 | 10.6 | 2.7 | 4.2 |
| PAD | 51.6 | 49.1 | 40.5 | 31.8 | 11.1 | 9.6 | 1.2 | 4.2 | 0.7 | 0.7 |
- At A3B, IMG leads by pp on 3-block problems (77.6 vs 51.6), and pooled over 9–12 blocks success rises from to ( pp, about ); on 5–8 blocks the conditions are close ( vs ). The gain is therefore not monotonic in problem size.
- Success on blocks appears only at A3B. At A874M, IMG solves some 5- and 6-block problems while PAD solves none.
- Below A3B, short-plan differences are noisy, for example at A590M IMG leads on 4 blocks but trails on 3, so we draw no conclusions from them.
We thank the reviewer again for the insightful questions, and we would be happy to provide any further details.
References
- Shukor et al. Scaling Laws for Native Multimodal Models. ICCV, 2025.
Thank you for the detailed response and additional analyses. The response addresses most of my concerns, and I maintain my weak acceptance rating.
Response to Reviewer uQet [1/2]
We sincerely thank the reviewer for the careful reading, and for recognizing that the role of visual content in native multimodal pre-training is an important and interesting question, that the experiments are comprehensive across model sizes and training configurations, and that the task-level analysis offers insight into the role of vision. The question in W1 led to our most informative new analysis. All changes are marked in blue in the revised manuscript. We respond to each comment below.
W1. Possible adverse effects of the PAD construction
This is a well-founded concern, and we agree that the IMG$-$PAD difference should be read as the benefit of visual content relative to a blind-context baseline. We now state this definition explicitly and report where the effect appears.
(a) The effect is concentrated where the text depends on the image (new paragraph in Sec. 3.3). Our multimodal objective combines two sources: image-caption data and interleaved image-text documents. Caption text describes its image directly, so under PAD it becomes exactly the under-specified task the reviewer describes; our interleaved documents, mainly papers, long documents and web pages, rarely refer to their images. Relative PAD-over-IMG loss increase at the final checkpoint:
The gap is large where the text depends on the image and small where it does not, which is the pattern a blind-context baseline would produce. Sec. 3.1 now defines the anchor gap in these terms: PAD must predict image-dependent text without the image, rather than never having seen that text.
(b) Text learning stays close to a text-only reference. We compare against a text-only (r=0) run family at all six scales with the same architecture, the same 250B text tokens and the same optimization; it matches the text budget, not the total budget, of IMG/PAD:
(c) The gap grows with the amount of visual data. At a fixed 250B text budget, the caption gap at A3B is +8.68%, +12.23% and +16.26% for r=0.1, 0.2 and 0.3, and it increases with r at every scale.
(d) Remaining caveat. Our spatial suite was not run on the text-only models, so on spatial tasks the benefit of visual content and any cost of blind context are not separated. W3 (part 2/2) gives the comparison that is available.