acceptodds
Under review as a conference paper at ICLR 2027

More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

Abstract

Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controlled visual-content ablations across model sizes and training budgets. Specifically, we replace image embeddings with content-free padding token embeddings. Retaining visual content improves text prediction on image-text examples without materially changing the lowest text-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores. The relative multimodal loss benefit grows approximately as a power law of compute. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone. Both conditions improve with model and data scaling. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multimodal Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uQetOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the contribution of visual content in native multimodal pre-training via controlled experiments with matched IMG/PAD comparisons across model sizes and training budgets. The paper finds that the visual content improves multimodal text prediction while largely preserving text-loss scaling and language performance. They further report selective improvements in text-only tasks like spatial reasoning, particularly localization, navigation, and symbolic planning.

Critical strengths
  1. This paper studies an important and interesting question about the role of visual content in native multimodal pre-training.
  2. The experiments are comprehensive, spanning multiple model sizes and training configs.
  3. The task-level analysis offers some interesting insights about the role of vision in multimodal training.
Critical weaknesses
  1. The PAD construction may introduce adverse training effects beyond removing visual information. Some text in multimodal examples depends heavily on the accompanying image. Keeping such text while replacing the image with uninformative padding creates an under-specified prediction task, requiring the model to predict image-specific details without access to their source. This could potentially encourage unsupported predictions or introduce unfavorable learning dynamics. Consequently, the IMG–PAD difference may reflect both the benefits of visual information and the disadvantages introduced by the PAD construction.
  2. The advantage of native multimodal training over staged training remains unclear. The experiments demonstrate the value of visual content within a native multimodal setup, but do not establish whether introducing both modalities from the beginning provides an advantage over first training on text and subsequently incorporating multimodal data, especially for multimodal tasks.
  3. A text-only baseline excluding multimodal data is missing.
Questions for authors
  1. Is the selective transfer pattern broadly shared across benchmarks within localization and navigation, or primarily driven by particular tasks?
  2. Can the planning results be broken down by the number of blocks, rather than only the pooled 5–12-block category?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uQet [1/2]

We sincerely thank the reviewer for the careful reading, and for recognizing that the role of visual content in native multimodal pre-training is an important and interesting question, that the experiments are comprehensive across model sizes and training configurations, and that the task-level analysis offers insight into the role of vision. The question in W1 led to our most informative new analysis. All changes are marked in blue in the revised manuscript. We respond to each comment below.

W1. Possible adverse effects of the PAD construction

Keeping image-dependent text while replacing the image with uninformative padding creates an under-specified prediction task. The IMG$-$PAD difference may therefore reflect both the benefit of visual information and a disadvantage introduced by the PAD construction.

This is a well-founded concern, and we agree that the IMG$-$PAD difference should be read as the benefit of visual content relative to a blind-context baseline. We now state this definition explicitly and report where the effect appears.

(a) The effect is concentrated where the text depends on the image (new paragraph in Sec. 3.3). Our multimodal objective combines two sources: image-caption data and interleaved image-text documents. Caption text describes its image directly, so under PAD it becomes exactly the under-specified task the reviewer describes; our interleaved documents, mainly papers, long documents and web pages, rarely refer to their images. Relative PAD-over-IMG loss increase at the final checkpoint:

SourceA71MA340MA874MA3B
Image-caption+4.32%+4.32\%+7.50%+7.50\%+10.61%+10.61\%+16.26%+16.26\%
Interleaved document−0.02%-0.02\%−0.21%-0.21\%+0.04%+0.04\%+0.18%+0.18\%
Text-web (reference)+0.01%+0.01\%−0.02%-0.02\%−0.01%-0.01\%−0.02%-0.02\%

The gap is large where the text depends on the image and small where it does not, which is the pattern a blind-context baseline would produce. Sec. 3.1 now defines the anchor gap in these terms: PAD must predict image-dependent text without the image, rather than never having seen that text.

(b) Text learning stays close to a text-only reference. We compare against a text-only (r=0r = 0) run family at all six scales with the same architecture, the same 250B text tokens and the same optimization; it matches the text budget, not the total budget, of IMG/PAD:

  • The paired pure-text loss gap between PAD and IMG stays between −0.10%-0.10\% and +0.13%+0.13\% (Sec. 3.2); at the end of training, PAD is within −0.50%-0.50\% to +0.55%+0.55\% of the text-only model on every text source and scale.
  • Averaged over 15 language benchmarks, PAD$-$text-only is −0.41-0.41 and IMG$-$text-only is −0.18-0.18 across the six scales, both within scale-to-scale variation.

(c) The gap grows with the amount of visual data. At a fixed 250B text budget, the caption gap at A3B is +8.68%+8.68\%, +12.23%+12.23\% and +16.26%+16.26\% for r=0.1r = 0.1, 0.20.2 and 0.30.3, and it increases with rr at every scale.

(d) Remaining caveat. Our spatial suite was not run on the text-only models, so on spatial tasks the benefit of visual content and any cost of blind context are not separated. W3 (part 2/2) gives the comparison that is available.

AuthorsOpenReview ↗

Response to Reviewer uQet [2/2]

W2. Native versus staged training

The advantage of native multimodal training over staged training is unclear. The experiments do not establish whether introducing both modalities from the beginning is better than training on text first and then incorporating multimodal data.

This is an important question, though adjacent to ours: we ask what visual content contributes given native pre-training. Prior work compares early-fusion with late-fusion native architectures, the latter connecting separately pre-trained components and continuing multimodal training, and reports no inherent advantage for late fusion (Shukor et al., 2025). That comparison varies the architecture rather than the training schedule, so a text-first curriculum under a matched protocol remains open. We have added this discussion to the related-work section and do not claim our results settle it.

W3. A text-only baseline

A text-only baseline excluding multimodal data is missing.

We report one below: a text-only (r=0r = 0) run family at all six scales, with the same architecture, the same 250B text tokens and the same optimization settings. It matches the text budget of IMG/PAD, not their total budget, which includes 75B multimodal tokens. Our spatial suite (5-shot CoT, avg@4) was not run on these models, so we compare on the 15 language benchmarks. At A3B:

Text-onlyPADIMGIMG$-$Text-only
Mean (15)45.3645.7546.33+0.97+0.97
MBPP+48.6852.1254.23+5.55+5.55
GSM8K48.7553.8352.39+3.64+3.64
MMLU-Pro24.8825.4927.68+2.80+2.80
SIQA60.2458.6562.95+2.71+2.71
BBH44.9745.0440.84−4.13-4.13

IMG is above the text-only model on 10 of the 15 benchmarks and equal on one. The clear exception is BBH, where IMG is about 4 pp below both the text-only and the PAD model. BBH aggregates a broad set of largely non-spatial tasks; its three spatial subtasks are evaluated separately in our spatial suite, where geometric shapes, despite +10.45+10.45 pp at the final checkpoint, favours IMG at only 4/13 checkpoints and is not a sustained gain.

For spatial reasoning, the one benchmark available for the r=0r = 0 models is SpatialEval, under a different (1-shot) protocol and without a PAD run: at A3B, text-only scores 35.95 versus 41.43 for r=0.3r = 0.3, and is lower at five of six scales. We treat this as corroboration only, and note in the Appendix that evaluating our spatial suite on the text-only models would fully separate the two components.

Q1. Is the selective transfer shared across benchmarks?

Is the selective transfer pattern broadly shared across benchmarks within localization and navigation, or primarily driven by particular tasks?

It is shared by most, though not all, benchmarks (new Appendix table). Mean IMG$-$PAD gap (pp) over the 13 A3B checkpoints, with the number of checkpoints favouring IMG:

  • Localization. SpatialMap +2.60+2.60 (12/13) and BBH colored objects +5.16+5.16 (10/13) drive the gain; SpartQA is weaker at +0.89+0.89 (9/13).
  • Navigation. MazeNav +6.45+6.45 (12/13), BBH navigate +4.24+4.24 (12/13) and SpatialUnderstanding +2.23+2.23 (10/13); bAbI path finding shows no gain (−0.85-0.85, 6/13), with both conditions near the floor (≤16%\leq 16\%).
  • Other groups. None of their benchmarks reaches ≥12/13\geq 12/13; examples are StepGame +0.57+0.57 (9/13), FoREST −0.76-0.76 (4/13) and PIQA-spatial −0.31-0.31 (5/13), while Geometry3K favours IMG at 11/13.

Q2. Planning results by number of blocks

Can the planning results be broken down by the number of blocks, rather than only the pooled 5–12-block category?

Thank you for the suggestion. We report the breakdown below; the table for all six scales is in the Appendix. Final-checkpoint success (avg@4, %):

SizeCond.3456789101112
A874MIMG40.131.95.50.5000000
PAD58.921.600000000
A3BIMG77.659.141.027.113.912.516.910.62.74.2
PAD51.649.140.531.811.19.61.24.20.70.7
  • At A3B, IMG leads by +26.0+26.0 pp on 3-block problems (77.6 vs 51.6), and pooled over 9–12 blocks success rises from 2.06%2.06\% to 9.20%9.20\% (+7.14+7.14 pp, about 4.5×4.5\times); on 5–8 blocks the conditions are close (23.28%23.28\% vs 22.78%22.78\%). The gain is therefore not monotonic in problem size.
  • Success on ≥7\geq 7 blocks appears only at A3B. At A874M, IMG solves some 5- and 6-block problems while PAD solves none.
  • Below A3B, short-plan differences are noisy, for example at A590M IMG leads on 4 blocks but trails on 3, so we draw no conclusions from them.

We thank the reviewer again for the insightful questions, and we would be happy to provide any further details.

References

  • Shukor et al. Scaling Laws for Native Multimodal Models. ICCV, 2025.

Thank you for the detailed response and additional analyses. The response addresses most of my concerns, and I maintain my weak acceptance rating.

Reviewer_5rofOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the contribution of visual content to native multimodal pretraining through controlled experiments across model sizes and training budgets. The authors compare models trained with image embeddings against models in which those embeddings are replaced by content-free padding, keeping the architecture, training text, token budgets, and optimization settings fixed within each pair. Visual content reduces text-prediction loss on multimodal examples, with the relative benefit increasing with compute, while leaving pure-text loss scaling, compute-optimal allocations, and aggregate language performance largely unchanged. Text-only evaluations also reveal selective spatial-reasoning gains, most sustained in localization, navigation, and symbolic planning during training of the largest model.

Critical strengths
  • The IMG and PAD conditions are matched in architecture, training text, token budgets, and optimization settings, with only the visual content replaced by content-free padding. Preserving the accompanying text and sequence structure helps isolate the contribution of visual information from changes in language supervision or training resources.
  • The paper examines training losses, compute-optimal allocations, general language performance, and text-only spatial reasoning across multiple model sizes and training checkpoints. The task-level comparisons and planning-error analysis provide a more detailed picture of where visual pretraining helps and where limitations remain.
Critical weaknesses
  • The paper presents extensive evaluations under carefully controlled training conditions, but the main narrative makes it difficult to identify the central conceptual takeaways. Several results sections emphasize numerical comparisons without sufficiently explaining how they advance our understanding beyond prior work. Although the scaling analysis and task-specific spatial evaluations identify interesting patterns, their broader implications remain underdeveloped. In particular, the proposed interpretation of selective spatial transfer is not directly tested, and the applicability of the observed scaling relationship beyond the studied configuration remains uncertain. A clearer synthesis of what is newly learned, why it matters, and how strongly the evidence supports it would strengthen the paper. While this paper has strong experimental results, it is not yet convincing that this paper has clear and meaningful conclusion.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5rof [1/2]

We sincerely thank the reviewer for the thoughtful feedback, and for noting that matching the architecture, training text, token budgets and optimization, with only the visual content replaced, isolates the contribution of visual information from changes in language supervision or training resources. We also appreciate the reviewer's recognition of the task-level comparisons and the planning-error analysis. We respond to each comment below.

Q1. What the study adds to our understanding

The main narrative makes it difficult to identify the central conceptual takeaways, and several results sections emphasize numerical comparisons without sufficiently explaining how they advance our understanding beyond prior work.

We agree this was not stated plainly enough. One sentence captures what the study adds: the contribution of visual content is selective, both across training objectives and across downstream capabilities. Prior native multimodal scaling studies vary the data mixture, which changes language supervision and compute allocation at the same time; the matched ablation is what lets these effects be attributed to visual content, and the attribution turns out to be uneven in a specific and measurable way.

The three findings, each with what follows from it:

  • The language-side scaling relationship is largely unchanged. The text loss frontier (β=0.198\beta = 0.198 for both conditions; offsets 1.8291.829 vs 1.8311.831), the compute-optimal allocation (IsoFLOP a=0.649a = 0.649 vs 0.6450.645; envelope 0.5780.578 for both) and aggregate language performance are closely matched. What this means: within the mixture we train on, retaining visual content does not visibly change the compute-optimal trade-off between model size and text data, so a text-side allocation recipe can be reused.
  • The benefit to multimodal prediction strengthens with budget. What this means: the context an image provides is still being exploited at larger training budgets rather than being substituted by what the model learns from text. We state this specifically for multimodal text prediction; it is not a claim about downstream task value in general.
  • Text-only transfer is concentrated in part of the spatial suite. What this means: the benefit of visual pre-training can extend beyond tasks that receive images, but it is not summarized by either the language average or the aggregate spatial score. Which tasks show sustained gains has to be read off the task structure.

The Introduction now states the contributions in these terms, with the scope of the claims given immediately after.

Q2. Evidence for the selective pattern

The proposed interpretation of selective spatial transfer is not directly tested.

This is a fair criticism, and we have separated the empirical patterns from the interpretation. Four analyses, each answering a different question:

AnalysisWhat it supports
Caption vs. interleaved breakdownThe loss benefit is concentrated where the text depends on the image
Varying the data ratio rrThe anchor gap grows with the amount of multimodal training data
Per-benchmark persistenceWhether the selective transfer rests on several tasks or a few
Blocksworld by problem sizeHow the gain varies within a single task

(a) The loss benefit is concentrated where the text depends on the image. Our multimodal objective combines image-caption data and interleaved image-text documents. Caption text describes its image directly; our interleaved documents, mainly papers, long documents and web pages, rarely refer to their images. Relative PAD-over-IMG loss increase at the final checkpoint:

SourceA71MA340MA874MA3B
Image-caption+4.32%+4.32\%+7.50%+7.50\%+10.61%+10.61\%+16.26%+16.26\%
Interleaved document−0.02%-0.02\%−0.21%-0.21\%+0.04%+0.04\%+0.18%+0.18\%
Text-web (reference)+0.01%+0.01\%−0.02%-0.02\%−0.01%-0.01\%−0.02%-0.02\%

We read the small interleaved gap as indicating that the training loss is a weak probe on that source, not that visual context is unused there.

(b) The gap grows with the amount of multimodal data. At a fixed 250B text budget, the caption-source gap at A3B is +8.68%+8.68\%, +12.23%+12.23\% and +16.26%+16.26\% for r=0.1r = 0.1, 0.20.2 and 0.30.3, and it increases with rr at every model size.

Analyses (c) and (d), at the level of downstream tasks, continue in part 2/2.

AuthorsOpenReview ↗

Response to Reviewer 5rof [2/2]

Q2 (continued)

(c) The selective transfer rests on several benchmarks, not one. Over the 13 A3B checkpoints, every benchmark favouring IMG at ≥12/13\geq 12/13 checkpoints lies in localization, navigation or symbolic planning: SpatialMap (12/13, mean +2.60+2.60 pp), MazeNav (12/13, +6.45+6.45), BBH-navigate (12/13, +4.24+4.24) and Blocksworld (12/13, +5.19+5.19). The full per-benchmark table is in the Appendix.

(d) Within one task, the gain varies with problem size. In Blocksworld at A3B, success on 9–12-block problems rises from 2.06%2.06\% to 9.20%9.20\% (+7.14+7.14 pp, about 4.5×4.5\times), whereas on 5–8-block problems the two conditions are close (23.28%23.28\% vs 22.78%22.78\%).

Taken together, these are empirical patterns rather than a mechanism. Our reading, that the tasks with sustained gains share a demand for maintaining or composing object configurations, remains an interpretation that is consistent with them; we present it as such in the revision.

Q3. How strongly the evidence supports the conclusions

While this paper has strong experimental results, it is not yet convincing that it has a clear and meaningful conclusion; a clearer statement of how strongly the evidence supports it would strengthen the paper.

Two controls bear on this, and we state the scope of each.

A text-only (r=0r = 0) control at all six scales, with the same architecture, the same 250B text tokens and the same optimization settings. On the measures compared, the three conditions stay close: at A3B the 15-benchmark language averages are 45.36 (text-only), 45.75 (PAD) and 46.33 (IMG), and the paired pure-text loss gap between PAD and IMG stays between −0.10%-0.10\% and +0.13%+0.13\%. This control shares the text budget with IMG/PAD but not the total token budget, since IMG/PAD additionally train on 75B multimodal tokens; we report it as a complementary reference rather than a matched-compute baseline.

A paired item-level bootstrap for the spatial result: resampling problems with replacement within each benchmark (10k resamples), then averaging benchmarks equally within each task group and groups equally, as in the paper, gives an A3B IMG$-$PAD gap of +4.38+4.38 pp [95% CI +3.45+3.45, +5.29+5.29]. This quantifies uncertainty from the choice of problems; decoding variability is controlled separately by reporting avg@4. All runs use a single seed per condition, so the interval does not cover variation between training runs.

As a reference for the size of the effect, the same models over the same checkpoints give a mean IMG$-$PAD gap of +1.93+1.93 pp on the spatial suite against +0.38+0.38 pp on the 15 language benchmarks.

Q4. Applicability beyond the studied configuration

The applicability of the observed scaling relationship beyond the studied configuration remains uncertain.

We agree, and the scope is now stated explicitly in the Introduction and the Conclusion: the main analyses use r=0.3r = 0.3, with the anchor-gap trend also checked at r=0.1r = 0.1 and 0.20.2; one architecture family; activated sizes up to 3B; one seed per condition. The fitted exponents describe this range, and we do not claim extrapolation beyond it. Within the range, the anchor-gap trend is observed along two axes, model scale and data mixture, which is the broadest check we can offer without larger runs.

We hope this makes both the conclusions and their limits clearer. We thank the reviewer again for the constructive feedback, and we would be grateful to know if any point remains unclear.

Thank you for providing additional experimental results and clarifications. The authors' rebuttal have addressed most of my concerns through the revision, thus I'm raising my score to weak accept.

Reviewer_hjkxOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks what visual content contributes to native multimodal pre-training beyond visual understanding: does it change scaling behavior or improve capabilities evaluated without images? The authors train matched decoder-only MoE models from scratch at six activated scales, where the IMG condition receives real image patch embeddings and the PAD condition replaces them with content-free embeddings, keeping sequences, text supervision, token budgets, and optimization identical. Using IsoFLOP and training-curve-envelope estimators on text and multimodal loss frontiers, plus 15 general-language and 15 text-only spatial benchmarks, they find that visual content leaves text-loss scaling, compute-optimal text allocation, and aggregate language performance nearly unchanged, but lowers multimodal loss, with a relative anchor gap that widens with multimodal compute. On text-only spatial tasks, both conditions improve with scale, but IMG gains selectively in localization, navigation, and symbolic planning, with less consistent gains elsewhere. The paper concludes that visual content acts as a scalable anchor for multimodal learning and transfers selectively to spatial reasoning that requires maintaining or composing configurations, rather than broadly improving language ability.

Critical strengths
  1. The paper studies an important question: what role does visual content play in native multimodal pre-training?
  2. The matched IMG/PAD controlled study is carefully designed and evaluated across multiple model scales and compute budgets.
  3. The finding that visual pre-training improves text-only spatial reasoning is interesting.
Critical weaknesses
  1. The largest evaluated scale is still relatively small. The largest model has 3B activated non-embedding parameters, which limits how strongly the reported scaling trends support claims about substantially larger native multimodal models.
  2. The downstream spatial-reasoning gains are modest and selective. The aggregate A3B IMG-PAD gap is 4.38 percentage points, sustained improvements are concentrated in localization, navigation, and symbolic planning, and long-plan Blocksworld success remains low. This limits the broader claim that visual pre-training transfers to text-only reasoning.
  3. The PAD baseline preserves image positions and sequence length by inserting content-free embeddings. A no-image-token baseline with matched compute, obtained by adjusting model size or training-token count, would help test whether the conclusions depend on retaining these padding positions and the associated multimodal sequence structure.
Questions for authors
  1. Why does the study use a single patch-embedding layer rather than a pretrained vision encoder such as DINO or SigLIP? Is this choice intended to isolate native multimodal pre-training, and how might the encoder choice affect the visual-anchor scaling and text-only transfer results?
  2. Why is the multimodal-to-text token ratio fixed at r=0.3r=0.3? Do the authors expect the visual anchor gap and text-only transfer results to hold under other data mixtures, and can they provide evidence or justification for this choice?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hjkx [1/2]

We sincerely thank the reviewer for the careful reading, and for recognizing that the role of visual content in native multimodal pre-training is an important question, that the matched IMG/PAD study is carefully designed across model scales and compute budgets, and that the text-only spatial transfer result is interesting. The comments led us to add several new analyses. All changes are marked in blue in the revised manuscript. We respond to each comment below.

W1. Largest evaluated scale

The largest model has 3B activated non-embedding parameters, which limits how strongly the reported scaling trends support claims about substantially larger native multimodal models.

We agree, and the scope is now stated in the Introduction and Conclusion. The ladder spans ∼42×\sim 42\times in activated non-embedding parameters, and the power laws are presented as describing the fitted range only (Sec. 3.4).

Within the studied range the gap continues to grow. Our multimodal objective combines image-caption data and interleaved image-text documents, and separating the two sources shows that the growth is carried by the caption source, whose text describes the image directly. Relative PAD-over-IMG loss increase at the final checkpoint:

SourceA71MA340MA874MA3B
Image-caption+4.32%+4.32\%+7.50%+7.50\%+10.61%+10.61\%+16.26%+16.26\%
Interleaved document−0.02%-0.02\%−0.21%-0.21\%+0.04%+0.04\%+0.18%+0.18\%
Text-web (reference)+0.01%+0.01\%−0.02%-0.02\%−0.01%-0.01\%−0.02%-0.02\%

The gap also grows with the multimodal data ratio (Q2, part 2/2).

W2. Size and selectivity of the spatial gains

The downstream spatial gains are modest and selective: the aggregate A3B gap is 4.38 points, sustained improvements are concentrated in localization, navigation and symbolic planning, and long-plan Blocksworld success remains low.

We agree that the transfer is selective, and the paper does not claim a broad improvement in text-only reasoning. Four analyses indicate that the selective gains are systematic rather than noise.

Item sampling. A paired item-level bootstrap over problems (10k resamples) gives an A3B gap of +4.38+4.38 pp [95% CI +3.45+3.45, +5.29+5.29]. Decoding variability is controlled separately, since we report avg@4.

Size of the effect against a reference suite. Over the 13 A3B checkpoints, the gap averages +1.93+1.93 pp (positive at 10/13) on the spatial suite against +0.38+0.38 pp (range −0.33-0.33 to +0.80+0.80) on the 15 language benchmarks, measured on the same models at the same checkpoints.

Benchmark-level persistence (mean gap over 13 A3B checkpoints; checkpoints favouring IMG):

GroupBenchmarks
LocalizationBBH-colored-objects +5.16+5.16 (10/13); SpatialMap +2.60+2.60 (12/13); SpartQA +0.89+0.89 (9/13)
NavigationMazeNav +6.45+6.45 (12/13); BBH-navigate +4.24+4.24 (12/13); SpatialUnderstanding +2.23+2.23 (10/13); bAbI-path −0.85-0.85 (6/13)
PlanningBlocksworld +5.19+5.19 (12/13)
RelationsStepGame +0.57+0.57 (9/13); SpaRTUN −0.34-0.34 (6/13); bAbI-pos +1.38+1.38 (9/13); FoREST −0.76-0.76 (4/13)
Geometry / PhysicalBBH-shapes −1.13-1.13 (4/13); Geometry3K +2.29+2.29 (11/13); PIQA-spatial −0.31-0.31 (5/13)

Every benchmark favouring IMG at ≥12/13\geq 12/13 checkpoints lies in localization, navigation or planning. Since checkpoints within a run are correlated, we report these counts descriptively.

Within-task breakdown. At A3B, success on 9–12-block Blocksworld problems rises from 2.06%2.06\% to 9.20%9.20\% (+7.14+7.14 pp, about 4.5×4.5\times), whereas on 5–8-block problems the two conditions are close (23.28%23.28\% vs 22.78%22.78\%). The full table by block count, including a large gain on 3-block problems, is in the Appendix together with the per-benchmark table above.

We agree that absolute long-plan success remains low; the paper identifies execution as the bottleneck (Sec. 5.4).

AuthorsOpenReview ↗

Response to Reviewer hjkx [2/2]

W3. A no-image-token baseline

The PAD baseline preserves image positions and sequence length by inserting content-free embeddings. A no-image-token baseline with matched compute would help test whether the conclusions depend on retaining these padding positions.

First, to be precise about PAD: the replacement covers the whole image region, including its delimiter tokens, so the ablation is of the full image span rather than of the patch embeddings alone, while sequence length, token positions and the text-loss mask are preserved. This is now stated explicitly in Sec. 3.1.

Second, we have added a text-only (r=0r = 0) control at all six scales, with the same architecture, the same 250B text tokens and the same optimization settings. We should be explicit about what it is and is not: it removes the image positions and the multimodal text together, but it shares only the text budget with IMG/PAD, which additionally train on 75B multimodal tokens. It is therefore a complementary reference at a matched text budget, not the matched-compute baseline the reviewer asks for.

15-benchmark language avgA71MA128MA340MA590MA874MA3B
Text-only (r=0r = 0)22.3824.7730.7033.8837.6045.36
PAD21.9624.2529.2534.3636.6445.75
IMG21.9524.3430.0334.0036.9646.33
  • The paired pure-text loss gap between PAD and IMG stays between −0.10%-0.10\% and +0.13%+0.13\% (Sec. 3.2), and at the end of training both lie within ±0.55%\pm 0.55\% of the text-only model on every text source.
  • At every scale, the language differences are smaller than the gain between adjacent scales.

On these two measures, the pure-text loss and the language average, the three conditions stay close. We keep PAD as the primary control because it holds everything else fixed, including the text tokens, positions, data order and FLOPs; removing the image positions while keeping the multimodal text would require re-balancing NN or DD, which is the allocation our analysis measures. A baseline that matches total compute by adjusting model size or token count, as the reviewer suggests, would be a useful further control and we note it as such.

Q1. Patch embedding instead of a pretrained vision encoder

Why does the study use a single patch-embedding layer rather than a pretrained vision encoder such as DINO or SigLIP, and how might the encoder choice affect the visual-anchor scaling and the text-only transfer?

The rationale is now in Sec. 2.4:

  1. It keeps the scaling ladder interpretable. An encoder adds parameters outside the activated non-embedding count NN on which the allocation analysis is defined, so the fitted exponents would depend on how encoder parameters are counted.
  2. Encoder-free designs are competitive. Gemma 4, for example, introduces a unified encoder-free architecture for its 12B model that ingests raw image patches.

On the effect: a language-supervised encoder such as SigLIP would also bring knowledge from its own image–text pre-training, confounding learning from visual content with transferring a pretrained representation. Whether the exponent and the spatial transfer persist in that setting remains an open question, which we flag as future work.

Q2. Choice of the multimodal-to-text ratio

Why is the ratio fixed at r=0.3r = 0.3? Would the visual anchor gap and the text-only transfer hold under other data mixtures?

In prior scaling work on this setup, r=0.3r = 0.3 gave good multimodal performance; it is also the largest ratio in that range, which amplifies the multimodal signal and makes the anchor gap easier to measure. Fixing rr also kept the matched ablation, 2 conditions ×\times 6 from-scratch scales, feasible.

To test other mixtures, we have run the matched ablation at r=0.1r = 0.1 and 0.20.2 with the same 250B text budget (new paragraph in Sec. 3.4; full table in the Appendix). Caption-source gap at the final checkpoint:

rrA71MA340MA590MA874MA3B
0.1+3.33%+3.33\%+4.99%+4.99\%+6.02%+6.02\%+6.76%+6.76\%+8.68%+8.68\%
0.2+4.06%+4.06\%+6.39%+6.39\%+7.69%+7.69\%+9.02%+9.02\%+12.23%+12.23\%
0.3+4.32%+4.32\%+7.50%+7.50\%+9.30%+9.30\%+10.61%+10.61\%+16.26%+16.26\%

The gap increases with rr at every scale and with scale at every rr, so the anchor-gap trend is not specific to r=0.3r = 0.3, and we now describe the fitted exponents as specific to the studied mixtures. These runs cover the anchor gap only; the spatial suite was not evaluated at r=0.1r = 0.1 and 0.20.2.

We thank the reviewer again for these constructive suggestions.

References

  • Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770, 2026.
  • Zhai et al. Sigmoid Loss for Language Image Pre-Training. ICCV, 2023.
  • Tschannen et al. SigLIP 2. arXiv:2502.14786, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.