acceptodds
Under review as a conference paper at ICLR 2027

Scaling Native Multimodal Pre-Training From Scratch

Abstract

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain incompletely characterized. To address this gap, we investigate the optimal model size and token count for training a Transformer-based vision-language model under a fixed computational budget. Our study demonstrates that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct allocation trends. The language allocation exponents lie in a similar range across the different data mixtures. The multimodal model-allocation exponent decreases modestly with the multimodal token ratio, indicating a relative shift toward token allocation. Additionally, our scaling analysis yields a budget-compensation rule. Specifically, an additional multimodal-token budget can offset the text-efficiency penalty caused by incorporating visual information into native multimodal pre-training under a fixed compute budget. Downstream evaluations further reveal that native multimodal pre-training is associated with improved spatial reasoning and multimodal few-shot learning. Generally, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multimodal Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_158MOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors train a "native multimodal" Mixture-of-Experts (MoE) model from scratch, eschewing a pre-trained vision encoder in favor of a single-layer patch embedding. Using both Chinchilla-style IsoFLOP analysis and training-curve envelope fitting, they determine the compute-optimal allocation—the balance between model size N and the number of tokens D—for both text-only and multimodal loss. Based on this, they propose a "budget compensation rule." Finally, downstream evaluations demonstrate three key outcomes: text-processing capabilities remain unimpaired, spatial reasoning within the text-only domain is enhanced, and the model emerges with multimodal few-shot learning abilities.

Critical strengths
  1. The problem is significant: there is a lack of systematic characterization regarding computational allocation in natively multimodal models—a domain where many teams are currently shifting from late-fusion approaches toward native training.
  2. The scale of the experiment is substantial overall; fitting language objectives and multimodal objectives separately represents a sound improvement over using a single, aggregate loss function.
Critical weaknesses
  1. All models, including the 71M one, use the same 8M-token batch. That is likely well above the critical batch size of the small models. It penalizes them at low compute and pushes N_opt upward. The paper does not say how the per-size learning rates were tuned.
  2. There are only six model sizes, with an uneven gap between 874M and 3B. The quadratic fits clip N_opt to the observed range, and no confidence intervals or bootstrap estimates are given for a, b, E, β or Γ.
  3. "In-context learning emerges" rests on a +2.4-point gain at the largest scale. The few-shot setting also changes the visual token budget per image, and no chance-level reference is given per benchmark. The wording is stronger than the evidence.
Questions for authors
  1. How many loss-bearing text tokens does the multimodal corpus contain? Do the Γ_r < 1 and SpatialEval results hold against a text-only model trained on the same total number of loss-bearing tokens?
  2. How do the joint and separate fits compare on held-out compute, e.g., the A874M/A3B segments beyond the fitted range? Please give bootstrap CIs for Γ_r and all exponents.
  3. How do the exponents change with annealed (cooldown) losses and full FLOP accounting? Do the few-shot gains hold when 0-shot also uses 512 visual tokens?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 158M [1/5]

We sincerely thank the reviewer for an exceptionally precise review, and for noting that compute allocation in natively multimodal models lacks a systematic characterization and that the scale of the experiments is substantial. Every point below is answered with a new analysis, and where the evidence does not support our original wording we revise the claim. All changes are marked in blue in the revised manuscript.

One correction first. The review credits us for fitting the language and multimodal objectives separately. Table 2 of Shukor et al. (2025b) already fits the law per evaluation data type, so per-objective fitting is established practice. Its value here lies in the design: we hold the pure-text budget fixed and add multimodal tokens on top, so each objective's change across rr is attributable to the multimodal data rather than to a simultaneous change in the pure-text budget.

Q1. How the per-size learning rates were tuned

The paper does not say how the per-size learning rates were tuned.

The per-size architecture, the batch size and the peak learning rate were taken from a scaling ladder fitted during an earlier architecture exploration and reused unchanged for every mixture, so that the available compute could be spent on the data-mixture sweep. We now report them explicitly in Appendix B. The rates follow a single power law:

lr∝N−0.570,R2=0.997,\mathrm{lr} \propto N^{-0.570}, \qquad R^2 = 0.997,

with every size within 6.5%6.5\% of the rule (A71M 0.97×0.97\times, A128M 0.99×0.99\times, A340M 1.01×1.01\times, A590M 1.03×1.03\times, A874M 1.07×1.07\times, A3B 0.94×0.94\times). This is not a μ\muP parameterization, since lr×width\mathrm{lr} \times \mathrm{width} falls from 1.661.66 to 0.610.61 across the family. Every run warms up over 2,0002{,}000 steps and then holds the rate constant.

Scope. Our allocation estimates characterize this training recipe. We did not re-tune the learning rate for this study; Appendix F lists this as a limitation.

Q2. The shared 88M-token batch

All models, including the 7171M one, use the same 88M-token batch. That is likely well above the critical batch size of the small models. It penalizes them at low compute and pushes NoptN^{\text{opt}} upward.

We agree that a shared batch can bias NoptN^{\text{opt}} upward, and three observations bound the concern.

  1. Steps. The first fitted point is at 2020B tokens, i.e., step ≈2,400\approx 2{,}400, just after the 2,0002{,}000-step warmup; every model then trains for 3030k–3939k steps.
  2. Optimizer and data scaling. With Muon, data efficiency is reported to be retained "far beyond the so-called critical batch size" (Essential AI, 2025), and the critical batch size "scales primarily with data size rather than model size" (Zhang et al., 2024). Within an IsoFLOP slice the smaller models are the ones that have seen the most data, so a shared batch is not expected to penalize them preferentially.
  3. A separate sensitivity. Independently of the batch, the compute convention moves the exponents: under fuller FLOP accounting the text exponents fall from ≈0.65\approx 0.65 to 0.530.53–0.580.58 (Q17 below, Appendix D.7). Porian et al. (2024) identify the last-layer cost, the warmup duration and scale-dependent optimizer tuning as causes of the Kaplan–Chinchilla discrepancy. We did not measure the batch-size effect itself; a batch ablation at the two smallest sizes would settle it, and Appendix F states this.

References

  • Shukor et al. (2025b) — M. Shukor, E. Fini, V. G. T. da Costa, et al. Scaling Laws for Native Multimodal Models. ICCV 2025.
  • Essential AI (2025) — Essential AI. Practical Efficiency of Muon for Pretraining. arXiv:2505.02222.
  • Zhang et al. (2024) — H. Zhang, D. Morwani, N. Vyas, et al. How Does Critical Batch Size Scale in Pre-training? arXiv:2410.21676.
  • Porian et al. (2024) — T. Porian, M. Wortsman, J. Jitsev, et al. Resolving Discrepancies in Compute-Optimal Scaling of Language Models. arXiv:2406.19146.
AuthorsOpenReview ↗

Response to Reviewer 158M [2/5]

Q3. Clipping of the IsoFLOP vertices

The quadratic fits clip NoptN^{\text{opt}} to the observed range.

Of the 139139 IsoFLOP budgets, 33 have a vertex clipped to the observed size range and 11 profile is non-convex. All four lie just below A3B's first checkpoint, where A874M is the largest available model. Excluding them changes any exponent by at most 0.0110.011 (e.g., text at r=0r=0: 0.650→0.6390.650 \rightarrow 0.639). Appendix D.6 reports the audit.

Q4. Confidence intervals for the fitted quantities

No confidence intervals or bootstrap estimates are given for aa, bb, EE, β\beta or Γ\Gamma. Please give bootstrap CIs for Γr\Gamma_r and all exponents.

We added two analyses (Appendix D.6, Table 5). First, a moving-block bootstrap of within-run noise: each run's smoothed curve is split into a trend and residuals (SD 0.0060.006 for text, 0.0140.014 for multimodal), residual blocks are resampled jointly across sizes because all sizes see the same data order and their residuals are correlated at about 0.990.99, and the entire pipeline is rerun 1,0001{,}000 times, including the budget grid, the envelopes, the vertices with clipping, the regressions and both frontier fits. Second, a leave-one-model-size-out jackknife. Neither covers seed variance, since each configuration was trained once.

Quantityr=0r=0r=0.1r=0.1r=0.2r=0.2r=0.3r=0.3
atexta_{\text{text}}0.650 [0.631, 0.694]0.645 [0.615, 0.670]0.661 [0.640, 0.680]0.649 [0.622, 0.668]
amma_{\text{mm}}–0.686 [0.613, 0.704]0.649 [0.624, 0.683]0.622 [0.615, 0.661]
Γr\Gamma_r–0.952 [0.930, 0.985]0.915 [0.901, 0.950]0.899 [0.884, 0.936]
(1+r)Γr−1(1+r)\Gamma_r - 1–+4.7%+4.7\% [+2.3+2.3, +8.3+8.3]+9.8%+9.8\% [+8.1+8.1, +14.0+14.0]+16.9%+16.9\% [+14.9+14.9, +21.7+21.7]

Leave-one-size-out ranges: atexta_{\text{text}} within 0.600.60–0.690.69, amma_{\text{mm}} within 0.590.59–0.760.76, and Γr\Gamma_r within [0.943,0.955][0.943, 0.955], [0.894,0.939][0.894, 0.939] and [0.870,0.929][0.870, 0.929].

  • Resolved. Γr<1\Gamma_r < 1 and a positive total-compute penalty at every ratio, in every interval, every refit and every accounting; Γ0.3−Γ0.1=−0.053\Gamma_{0.3} - \Gamma_{0.1} = -0.053 [−0.079,−0.022][-0.079, -0.022]; and no detectable difference among the text exponents across mixtures, all pairwise intervals being within ±0.06\pm0.06.
  • Not resolved. Γ0.3−Γ0.2=−0.016\Gamma_{0.3} - \Gamma_{0.2} = -0.016 [−0.044,+0.011][-0.044, +0.011].
  • The separate multimodal offsets are weakly identified, with EmmE_{\text{mm}} intervals as wide as [0.58,2.15][0.58, 2.15], so we do not interpret them and use the multimodal frontiers only within their fitted range. Table 5 also reports intervals for EsE_s and βs\beta_s, for instance E‾text=1.838\overline E_{\text{text}} = 1.838 [1.808,1.905][1.808, 1.905] and β‾text=0.204\overline\beta_{\text{text}} = 0.204 [0.194,0.229][0.194, 0.229] for the joint fit; intervals for bsb_s follow from those for asa_s, since bs=1−asb_s = 1 - a_s.

Q5. The multimodal allocation trend

(Bears on the claim in Sec. 3.2.)

Under the main compute accounting, both estimators show a decreasing point estimate across the three mixtures, and the decrease is negative in all six leave-one-size-out refits of the IsoFLOP estimate:

Estimatorr=0.1r=0.1r=0.2r=0.2r=0.3r=0.3
IsoFLOP (primary)0.6860.6490.622
Training-curve envelope0.5610.5580.551

The size of the decrease is not statistically resolved: the envelope falls by 0.0100.010 against the IsoFLOP decrease of 0.0640.064; the 95%95\% interval of the IsoFLOP difference, [−0.072,+0.033][-0.072, +0.033], includes zero; and under the three alternative accountings the difference ranges from −0.027-0.027 to +0.004+0.004. The revision reports this trend together with these intervals (Sec. 3.2, Appendices D.6 and D.7).

Q6. The two estimators

(Related to Q5.)

The envelope is 22–3×3\times noisier than IsoFLOP, with bootstrap SD 0.0260.026–0.0400.040 against 0.0100.010–0.0240.024, because it switches between near-tied sizes whose losses differ by only 0.0010.001–0.0060.006 at the switching budgets. We therefore take absolute levels from the IsoFLOP estimates, as stated in Sec. 2.2, and use the envelope for the direction of change; the IsoFLOP text exponents lie within the envelope's leave-one-size-out ranges. Both estimators reuse the same runs, so their agreement is a sensitivity check rather than independent evidence.

Q7. Six model sizes and the 874874M–33B gap

There are only six model sizes, with an uneven gap between 874874M and 33B.

We agree the spacing is uneven. The frontier is not destabilized by it: removing every A3B run before fitting, freezing the parameters and predicting A3B about one decade beyond the fitted range gives a text-loss MAPE of 0.380.38–0.63%0.63\% (Appendix D.5). The allocation exponents are the sensitive quantity, shifting by up to 0.070.07 when A3B is removed, which the revision now reports explicitly.

AuthorsOpenReview ↗

Response to Reviewer 158M [3/5]

Q8. The evidence behind the in-context-learning claim

"In-context learning emerges" rests on a +2.4+2.4-point gain at the largest scale, and the wording is stronger than the evidence.

We agree the wording was too strong. Recomputing the 33-shot gain over 00-shot from the last three checkpoints of each of the three mixtures gives nine evaluations per size, drawn from three training trajectories, on the paper's 2121-benchmark set. The at-chance set below contains the six benchmarks on which A3B is within three points of chance at 00-shot: MME, HallusionBench, CV-Bench, OmniSpatial, SEAM and SpatialEval (Appendix E.1, Table 8).

SizePaper (final ckpt)Replicates: mean ±\pm SD (#>0>0)At-chance set (66)Other benchmarks (1515)
A71M−0.02-0.02+0.15±0.59+0.15 \pm 0.59 (66/99)−0.36±1.37-0.36 \pm 1.37+0.35±0.44+0.35 \pm 0.44
A128M+0.96+0.96+0.79±0.34+0.79 \pm 0.34 (99/99)+0.55±0.70+0.55 \pm 0.70+0.89±0.48+0.89 \pm 0.48
A340M+1.31+1.31+1.27±0.41+1.27 \pm 0.41 (99/99)+1.44±1.32+1.44 \pm 1.32+1.20±0.43+1.20 \pm 0.43
A590M+0.98+0.98+1.02±0.57+1.02 \pm 0.57 (99/99)+2.25±1.19+2.25 \pm 1.19+0.52±0.49+0.52 \pm 0.49
A874M+1.80+1.80+1.12±0.47+1.12 \pm 0.47 (99/99)+1.33±1.38+1.33 \pm 1.38+1.04±0.29+1.04 \pm 0.29
A3B+2.43+2.43+2.46±0.64\mathbf{+2.46 \pm 0.64} (99/99)+5.42±0.99+5.42 \pm 0.99+1.28±0.77+1.28 \pm 0.77
  1. The A3B gain is robust across the evaluated checkpoints and mixtures. It is positive in all nine evaluations, with a mean of +2.46+2.46 points; the standard deviation of 0.640.64 reflects variation across checkpoints and mixtures rather than across independent training runs, and a conservative item-level 95%95\% interval on the final-checkpoint gain is ±0.78\pm0.78.
  2. The increase with scale is not monotone. A128M through A874M are flat at about +1+1, and the single-checkpoint value of +1.80+1.80 at A874M is the largest of that model's nine evaluations, whose mean is +1.12+1.12. The revision reports these means and no longer implies a monotone increase.
  3. Where the gain lives. On the remaining fifteen benchmarks, demonstrations add +0.5+0.5 to +1.3+1.3 at every size from A128M upward, with no trend. The scale dependence comes from the six at-chance benchmarks, where demonstrations appear mainly to convey the answer format: A3B scores exactly 50.0%50.0\% on five of the fourteen MME categories at 00-shot, the signature of a constant binary answer.

Revised wording (Sec. 4.3). "Few-shot prompting yields small but consistent gains from 128128M upward that roughly double at 33B (+2.5±0.6+2.5 \pm 0.6), largest where the model is at chance without demonstrations. We read this as the onset of multimodal in-context learning in natively pre-trained base models, rather than as a fully emerged capability."

Q9. Chance-level references per benchmark

No chance-level reference is given per benchmark.

We now report, for every benchmark, the expected accuracy of a uniform random guess under our protocol, computed from the exact scored item pools, together with a constant-answer baseline for the open-ended tasks (Table 21, Appendix E.1). Eight of the 2323 benchmarks are at or near chance at 00-shot for every model size.

This also leads us to withdraw the claim that few-shot gains concentrate on spatial reasoning. Four of our five spatial benchmarks (CV-Bench, OmniSpatial, SEAM, SpatialEval) are among the eight at or near their reference; the one that is above it, RealWorldQA, gains little; and the non-spatial MME, also at chance for A3B, gains +9.5+9.5. Being at chance without demonstrations and belonging to the spatial category are not separable in our suite.

Q10. The visual-token budget in the few-shot setting

The few-shot setting also changes the visual token budget per image. Do the few-shot gains hold when 00-shot also uses 512512 visual tokens?

The confound does not arise; our original sentence was ambiguous. Across all 2323 benchmarks the test image is encoded with up to 15361536 tokens in 00-, 11- and 33-shot alike, and only the in-context demonstration images are capped at 512512 tokens so that the templates fit the context window. The 00-shot and kk-shot runs therefore see identical test inputs. On five benchmarks (MMBench-en, VQAv2, ScienceQA, MMBench-cc, SEAM) no test image exceeds 512512 tokens at all. We have corrected the sentence in Sec. 4.1.

Reviewer_wP9ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates early-fusion native multimodal pre-training and analyze the scaling trends with more compute (data D and number of parameters N). For this, they separate text and multimodal objectives and analyze their scaling behaviours independently. To estimate the compute-optimal allocation of D and N, they consider IsoFLOP profiles and training curve envelope, similar to previous established approaches in the scaling laws literature. As for the model size, they investigate models from 71M to 3B active parameters. Downstream evaluations are performed on a suite of text and multimodal benchmarks to track performance.

Critical strengths

The paper tackles a timely and important problem. Scaling laws for multimodal models are lagging behind their language-only counterparts, and this is even a bigger gap for native multimodal models. Thus, I believe there is a lot of potential here. I also find separating the two objectives (text and multimodal) to be a reasonable one, however, its actual importance needs to be better demonstrated (please see weaknesses).

Critical weaknesses

I have three main concerns about the paper in terms of literature positioning, empirical demonstration and writing.

  • The main difference of this work from Shukor et al. 2025 (Scaling Laws for Native Multimodal Models) is the decoupling of the two training objectives, which remains to be incremental. What exactly is enabled by the new scaling law fits here? A careful comparison and positioning of the two works is necessary. Similarly, Shukor et al. 2026 investigates the problem from the data mixtures aspect. Given these two works, the contribution from this paper remains incremental.
  • I found the empirical results reported in Table 13 to be quite weak, thus limiting the practical relevance of the paper to practitioners. In connection to that, results at a higher compute scale would be helpful as a demonstration of the predictive power of the fitted scaling laws.
  • In its current state, the paper feels rather unfinished with writing making it hard to interpret the findings.
Questions for authors

None, in its current stance I think the paper requires a major revision and resubmission.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wP9R [1/3]

We sincerely thank the reviewer for the candid review, and for noting that the problem is timely and important, that scaling laws for native multimodal models lag behind their language-only counterparts, and that separating the two objectives is a reasonable choice whose importance needs to be demonstrated. Answering that last point led us to sharpen how we position the work. All changes are marked in blue in the revised manuscript. We respond to each comment below.

Q1. Per-objective fitting is established practice

The main difference from Shukor et al. (2025) is the decoupling of the two objectives, which remains incremental.

We agree that per-objective fitting is established practice, and we have corrected our Related Work, which described Shukor et al. (2025b) as modeling only a single aggregate loss. Alongside the averaged loss, their Table 2 fits the law per evaluation data type, with model-allocation exponents of 0.52460.5246, 0.52030.5203 and 0.53150.5315 for text, image–caption and interleaved data, and Shukor et al. (2025a) fit one law per target domain. We likewise claim no novelty for held-out validation, which both studies perform well.

Q2. What the work contributes

What exactly is enabled by the new scaling law fits here?

Our contribution is to quantify the incremental compute cost of adding multimodal data while retaining a fixed pure-text token budget, using a pure-text reference within the same MoE family. This complements studies of mixture proportions under a fixed total budget. In those designs the pure-text token count varies with the mixture (text shares of 1010–40%40\% in Shukor et al., 2025b; simplex weights with a 0.10.1 minimum in Shukor et al., 2025a) and no pure-text run is trained. We instead hold the pure-text budget at 250250B tokens and add r⋅250r\cdot250B multimodal tokens, which supplies the reference against which the cost is measured (Sec. 2.1, Appendix A). Four results follow.

  1. A compute price for language parity. Adding 10/20/30%10/20/30\% multimodal tokens requires +4.7/+9.8/+16.9%+4.7/+9.8/+16.9\% total compute to match the pure-text loss, less than the +10/20/30%+10/20/30\% the tokens add (Sec. 3.3). With all A3B runs held out, the predicted crossing compute differs from the observed one by 2.72.7–5.1%5.1\% on average (Appendix D.5).
  2. Per-objective allocation for a sparse family. The per-data-type exponents of Shukor et al. (2025b) are reported for dense early fusion, with MoE given as a single averaged frontier, and Shukor et al. (2025a) train no MoE. Our family is MoE throughout.
  3. Where the compensation comes from. Only 15.5%15.5\% of multimodal-stream positions carry text loss once image tokens are masked, and in padding control runs at all six sizes and three ratios visual content leaves Γr\Gamma_r and the text frontier unchanged (Appendix D.8).
  4. Downstream behavior of native base models. We evaluate few-shot prompting of natively pre-trained base models and text-only transfer, under matched-token controls (Sec. 4, Appendix E).

References

  • Shukor et al. (2025b) — M. Shukor, E. Fini, V. G. T. da Costa, et al. Scaling Laws for Native Multimodal Models. ICCV 2025.
  • Shukor et al. (2025a) — M. Shukor, L. Bethune, D. Busbridge, et al. Scaling Laws for Optimal Data Mixtures. NeurIPS 2025.
AuthorsOpenReview ↗

Response to Reviewer wP9R [2/3]

Q3. Positioning relative to both prior works

A careful comparison and positioning of the two works is necessary.

The two prior studies and ours ask complementary questions under complementary designs, and the revision states this in the Related Work (Appendix A).

Shukor et al. (2025b) characterize scaling for native multimodal models and compare early with late fusion, over four mixtures that partition a fixed total budget; Shukor et al. (2025a) model how the loss depends on mixture proportions on the simplex, with a minimum weight of 0.10.1 per domain. Both fit a law per data type or target domain, and both report held-out validation. Because the proportions are of a fixed total, the pure-text token count varies with the mixture, between 1010 and 40%40\% in Shukor et al. (2025b), and no pure-text run is trained.

Our design fixes the pure-text budget at 250250B tokens and adds r⋅250r\cdot250B multimodal tokens on top. The pure-text run at r=0r=0 then serves as a reference, which is what lets us denominate the cost of adding multimodal data in compute: +4.7/+9.8/+16.9%+4.7/+9.8/+16.9\% total compute to match the pure-text loss at r=0.1/0.2/0.3r=0.1/0.2/0.3 (Sec. 3.3). Within this design we also report per-objective allocation for a sparse family, whereas the per-data-type exponents of Shukor et al. (2025b) are for dense early fusion with MoE given as a single averaged frontier and Shukor et al. (2025a) train no MoE; we measure the loss-bearing fraction of the multimodal stream, 15.5%15.5\%, which neither study reports; and we evaluate few-shot prompting of the base models rather than after supervised fine-tuning (Sec. 4, Appendices D.8 and E).

Q4. Our exponents and those of prior work

(Bears on how the contribution compares with prior estimates.)

Our headline atext≈0.65a_{\text{text}} \approx 0.65 counts activated non-embedding parameters, whereas Shukor et al. (2025b) count all decoder parameters including embeddings, with active parameters for their MoE models, and neglect attention. We therefore refit everything under three fuller accountings, rebuilding the budget grid in each unit (Appendix D.7, Table 6).

Accountingatexta_{\text{text}} (r=0r=0 / .1.1 / .2.2 / .3.3)amma_{\text{mm}} (r=.1r=.1 / .2.2 / .3.3)Γ0.1\Gamma_{0.1} / Γ0.2\Gamma_{0.2} / Γ0.3\Gamma_{0.3}
6ND6ND, non-embedding (main text).650 / .645 / .661 / .649.686 / .649 / .622.952 / .915 / .899
6N′D6N'D, N′=N+2dVN' = N + 2dV.571 / .562 / .549 / .532.546 / .540 / .519.953 / .923 / .895
MDMD incl. attention, ctx 20482048.543 / .551 / .539 / .538.557 / .516 / .561.952 / .913 / .887
MDMD incl. attention, ctx 512512.560 / .569 / .579 / .559.588 / .573 / .567.948 / .902 / .877
  • Much of the gap is a matter of convention. The text exponents fall by 0.0760.076–0.1220.122 and the multimodal exponents by 0.0550.055–0.1400.140, with the shift negative in every cell and well outside the bootstrap noise. Under 6N′D6N'D, the convention closest to theirs, our text exponents of 0.5320.532–0.5710.571 move toward their dense value of 0.52460.5246, and the per-token rows approach the 0.5240.524 of DeepSeek LLM, whose accounting also includes attention.
  • The mechanism is known. Pearce & Song (2024) attribute much of the Kaplan–Chinchilla discrepancy to "counting non-embedding rather than total parameters, combined with their analysis being performed at small scale", and Porian et al. (2024) isolate the last-layer cost, the warmup duration and scale-dependent optimizer tuning. In our family, (N+2dV)/N(N + 2dV)/N is 3.193.19 at A71M but 1.171.17 at A3B, so the omitted terms are far larger for the small models.
  • Γr\Gamma_r, on which our contribution rests, moves by at most 0.0220.022 across all four accountings, so every compensation conclusion is unchanged.

The revision states the accounting next to every main-text exponent and reports the alternatives in Appendix D.7. Data and training recipe also differ between the studies, so accounting does not explain the gap entirely.


References

  • Shukor et al. (2025b) — M. Shukor, E. Fini, V. G. T. da Costa, et al. Scaling Laws for Native Multimodal Models. ICCV 2025.
  • Shukor et al. (2025a) — M. Shukor, L. Bethune, D. Busbridge, et al. Scaling Laws for Optimal Data Mixtures. NeurIPS 2025.
  • DeepSeek LLM — DeepSeek-AI. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv:2401.02954.
  • Pearce & Song (2024) — T. Pearce, J. Song. Reconciling Kaplan and Chinchilla Scaling Laws. arXiv:2406.12907.
  • Porian et al. (2024) — T. Porian, M. Wortsman, J. Jitsev, et al. Resolving Discrepancies in Compute-Optimal Scaling of Language Models. arXiv:2406.19146.
AuthorsOpenReview ↗

Response to Reviewer wP9R [3/3]

Q5. Table 13 and practical relevance

The empirical results in Table 13 are quite weak, which limits the practical relevance of the paper to practitioners.

Table 13 of the submission, which the revision renumbers Table 21, reports 00-shot accuracy of base models: no instruction tuning, no pre-trained vision encoder, a 44K context, and answers scored by lowest perplexity. Our scaling claims are stated in terms of losses and do not rest on it. We now provide that context (Appendix E.1).

  • Reference levels change how the table reads. Computed from the exact scored item pools, 88 of the 2323 benchmarks stay within about three points of their random-guess reference, or below it, at 00-shot for every model size: ScienceQA-CoT, HallusionBench, LogicVista-CoT, CountQA, CV-Bench, OmniSpatial, SEAM and SpatialEval. Several are binary tasks with a 50%50\% reference, which also dilutes the 2323-benchmark average.
  • On the remaining 1515 the scaling is clear. The 00-shot average rises from 24.724.7 (A71M) to 35.735.7 (A3B), e.g., MMBench-en 27.1→54.127.1 \rightarrow 54.1, AI2D 25.8→47.025.8 \rightarrow 47.0, TextVQA 14.9→41.214.9 \rightarrow 41.2, CountBench 8.7→34.18.7 \rightarrow 34.1. We keep all 2323 benchmarks in the main average and report this subset as a supplementary view.
  • The absolute level is expected under this design. Multimodal tokens are deliberately a small share of the budget (r≤0.3r \le 0.3), and multimodal understanding depends substantially on language ability, which itself scales gradually at these budgets. Prior work is consistent with this: Shukor et al. (2025b, Sec. 3.3) report that training from scratch needs considerably longer to reach the loss of an LLM-initialized model.
  • Revision. The table now carries a Chance column, marks those eight benchmarks, and reports the average over the remaining fifteen beside the 2323-benchmark average.

Q6. The encoder-free choice

(Bears on the absolute scores in Table 13.)

Three reasons, now stated in the Related Work. First, a separately parameterized vision tower changes the parameter count in a way that interferes with fitting a scaling ladder, which this study is built on. Second, encoder-free designs are used at frontier scale: the Gemma 4 suite proposes "a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches", alongside encoder-based models at its other sizes. Third, in our own preliminary experiments a pre-trained vision tower raised absolute benchmark scores but did not change the shape of the benchmark scaling curves, which returns to the first reason.

Q7. Results at a higher compute scale

Results at a higher compute scale would help demonstrate the predictive power of the fitted scaling laws.

We agree. With all A3B runs removed before fitting and the parameters then frozen, the text frontiers predict A3B over 1.0×10211.0\times10^{21}–4.4×10214.4\times10^{21} FLOPs, about one decade beyond the fitted range, with a mean absolute error of 0.380.38–0.63%0.63\% and a final-checkpoint error of +0.19%+0.19\% to +1.27%+1.27\% (Appendix D.5). New, larger runs could not be completed within the discussion period, and we state that limitation in Appendix F.

Q8. Writing

In its current state, the paper feels rather unfinished, with writing that makes it hard to interpret the findings.

The revision states each headline result with its uncertainty where it is first claimed, in the abstract, the Introduction and Sec. 3; rewrites the positioning in the Introduction and Related Work; adds a notation table (Table 2) and a Limitations paragraph; and adds Appendices D.5–D.8, E.1, E.2 and F on predictive validation, uncertainty, compute accounting, the sources of the compensation, chance levels and limitations. Claims are recalibrated wherever the new analyses require it.

We thank the reviewer again for the directness of this review; it changed how we present several results. If parts remain hard to follow, we would be grateful to know which ones most affect your assessment, so that we can address them within the discussion period.


References

  • Shukor et al. (2025b) — M. Shukor, E. Fini, V. G. T. da Costa, et al. Scaling Laws for Native Multimodal Models. ICCV 2025.
  • Gemma 4 — Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770.

Thank you for the detailed rebuttal. It addressed most of my concerns, so I am happy to increase my rating. I still encourage the authors to showcase the predictability at a higher compute scale. Furthermore, I encourage to also showcase the results after post-training besides the current zero-shot evals.

Reviewer_U4xPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies native multimodal pretraining from scratch using a family of mixture-of-experts models and several multimodal-to-text token ratios. It fits compute-optimal frontiers for language and multimodal losses, analyzes the compute required to compensate for adding multimodal data, and evaluates downstream language, multimodal, and in-context-learning behavior.

Critical strengths
  1. The controlled model family and substantial training sweep provide useful empirical evidence on an important and expensive design choice. Avoiding a pretrained visual encoder makes the native-pretraining setting particularly interesting.
  2. The distinction between matched text-token budgets and total compute is valuable. The paper explicitly states that its fitted compensation factors do not imply lower total cost than pure-text training: the reported total requirements are approximately 4.7%, 9.8%, and 16.9% higher for the three mixtures (Section 3, p. 6).
  3. The downstream evaluation is broad and provides evidence that adding multimodal data can preserve aggregate text performance at matched text-token budgets. The appendices also document the frontier-fitting procedure and evaluation settings.
Critical weaknesses
  1. The scaling laws are fitted retrospectively but lack independent predictive validation. The paper emphasizes predictable scaling, yet does not demonstrate that laws fitted on a subset of training runs accurately predict the loss of models excluded from fitting, particularly at larger model sizes or higher compute budgets. Agreement between the IsoFLOP and training-curve-envelope estimates does not establish this: Appendix D.2 explicitly notes that both estimators use the same runs and that their agreement is a sensitivity check rather than independent validation. Consequently, the results establish descriptive trends within the observed training sweep, but provide limited evidence that the fitted laws can reliably guide a new training run.

    This limitation also affects the practical claims about compute-optimal allocation and budget compensation. The paper does not independently test whether training at the predicted model-size/token allocation achieves the predicted loss, or whether the predicted additional compute actually recovers a pure-text reference loss. I would like to see the authors fit on smaller-scale runs, freeze the fitted parameters, and evaluate predictions on held-out larger-scale runs, reporting predicted versus observed losses and prediction errors. For the allocation and compensation claims, a particularly useful test would train configurations selected by the fitted laws and compare them with nearby allocations at the same compute budget. This would establish whether the proposed laws offer actionable predictive guidance beyond summarizing the existing experiments.

Questions for authors
  1. Can scaling laws fitted only to smaller models or lower-budget runs predict losses for held-out larger models or higher-budget runs without refitting? Please report predicted versus observed values and quantitative prediction errors. Held-out checkpoints from a run used in fitting would be a weaker test than holding out entire training runs.
  2. Do independently trained configurations selected by the predicted compute-optimal allocation and compensation rule achieve the predicted losses? In particular, does the predicted compensated budget recover the target pure-text loss, and how does the selected allocation compare with nearby alternatives at equal compute?
3 comments
AuthorsOpenReview ↗

Response to Reviewer U4xP [1/2]

We sincerely thank the reviewer for the careful reading, and for recognizing that the controlled model family bears on an expensive design choice, that the encoder-free setting is of particular interest, and that the distinction between matched text-token budgets and total compute matters, including that our compensation factors do not imply a lower total cost. The comments led us to add the predictive tests the paper was missing. All changes are marked in blue in the revised manuscript. We respond to each comment below.

Q1. Predictive validation on held-out runs

The laws are fitted retrospectively and lack independent predictive validation; agreement between the IsoFLOP and envelope estimators is a sensitivity check, not validation. Can laws fitted only on smaller or lower-budget runs predict held-out runs without refitting? Holding out entire runs is the stronger test.

We agree. Entire runs are held out, not checkpoints. Every A3B run, for all four mixtures, is removed before the budget grid and the envelopes are built, and all fitted parameters are then frozen. We predict A3B over 1.0×10211.0\times10^{21}–4.4×10214.4\times10^{21} text FLOPs, about one decade beyond the fitted range of 3.3×10193.3\times10^{19}–5.1×10205.1\times10^{20}, and evaluate the errors on the segments of the empirical frontier where A3B is the lowest-loss model (Appendix D.5, Table 3, Fig. 14).

ObjectiverrSeparate fit: MAPE / max. errorJoint fit: MAPE / max. errorFinal A3B ckpt: separate / joint
text00.61% / 0.0320.43% / 0.024+1.27%+1.27\% / +0.85%+0.85\%
text0.10.63% / 0.0310.44% / 0.023+1.24%+1.24\% / +0.71%+0.71\%
text0.20.38% / 0.0310.43% / 0.026+0.28%+0.28\% / +1.01%+1.01\%
text0.30.38% / 0.0280.42% / 0.026+0.19%+0.19\% / +0.43%+0.43\%
mm0.11.89% / 0.125–+3.45%+3.45\%
mm0.21.04% / 0.077–−0.03%-0.03\%
mm0.31.12% / 0.066–+2.73%+2.73\%

Errors are prediction minus observation; the final-checkpoint text losses are 2.102.10–2.112.11. Every final-checkpoint text error is positive, so the frozen fits are slightly conservative.

  • Two-size holdout. Fitting only A71M–A590M and predicting both A874M and A3B, 1.41.4 decades ahead, gives a MAPE of 1.01.0–2.8%2.8\% for text and 0.650.65–4.6%4.6\% for the multimodal objective.
  • Compute holdout. Truncating every run at Ctotal≤6×1020C_{\text{total}} \le 6\times10^{20} (1.11.1 decades fitted) and predicting the next decade gives a text MAPE of 0.410.41–0.67%0.67\%.
  • Scope. With about one decade of fitted compute, the text frontiers extrapolate about one decade with a MAPE of about 1%1\%. The multimodal frontiers are weaker, because their small decay exponents (β≈0.11\beta \approx 0.11–0.130.13) leave the offset weakly identified. With only half a decade fitted, the separate fits at r=0.2r=0.2 and 0.30.3 become ill-posed (negative offset, β\beta at its bound), whereas the joint fit stays at 1.061.06–1.22%1.22\%. Above 1.3×10211.3\times10^{21} FLOPs only A3B exists, so the observed frontier there is a single run.
AuthorsOpenReview ↗

Response to Reviewer U4xP [2/2]

Q2. Does the predicted allocation beat nearby allocations at equal compute?

How does the allocation selected by the fitted laws compare with nearby alternatives at the same compute budget?

It selects the best size among those we trained, and departing from it is costly (Appendix D.5).

  • Inside the fitted range. The size nearest the IsoFLOP-predicted NoptN^{\text{opt}} is the best trained size at equal compute in 1717–1919 of 1919–2020 budgets, and the adjacent sizes are worse by about 0.040.04 (text) and 0.080.08 (multimodal) in loss.
  • Extrapolated. With NoptN^{\text{opt}} fitted without any A3B run, the prediction identifies the best available size in 77–88 of 99–1010 text budgets and 55–88 of 99 multimodal budgets. All misses occur at the A874M/A3B crossover, where the fit switches to A3B about 0.10.1 decade early; the mean loss gap to the best size over these budgets is 0.00050.0005–0.00330.0033 (text) and 0.0060.006–0.0180.018 (multimodal).
  • Scope. This validates the choice among discrete candidate sizes; a newly trained configuration at the predicted optimum is discussed in Q4. Removing A3B shifts the allocation exponents by up to 0.070.07, so every exponent is now reported with bootstrap and leave-one-size-out intervals (Appendix D.6), and the NoptN^{\text{opt}} predicted at the final A3B budget, 3.33.3–5.35.3B, lies beyond our runs.

Q3. Does the predicted compensated budget recover the pure-text loss?

Does the compensated budget actually recover the target pure-text loss?

Yes, within the loss noise. Fitted without any A3B run, the joint model predicts Γr=0.950/0.925/0.903\Gamma_r = 0.950 / 0.925 / 0.903 (all runs: 0.952/0.915/0.8990.952 / 0.915 / 0.899). On the held-out budgets we measure the text compute at which each mixture first reaches the pure-text loss and compare it with the prediction (Appendix D.5, Table 4, Fig. 15).

rrPredicted Γr\Gamma_rObserved median Γ\Gamma: envelope / A3BMean abs. error of predicted computeΓobs≤1\Gamma_{\text{obs}} \le 1
0.10.9500.914 / 0.9415.1% / 3.5%8/8, 7/8
0.20.9250.920 / 0.9223.3% / 2.7%8/8, 8/8
0.30.9030.902 / 0.8833.3% / 3.4%8/8, 8/8
  • The predicted budget. The predicted crossing compute differs from the observed crossing by 2.72.7–5.1%5.1\% on average, within the loss noise, since a per-run noise of about 0.0060.006 in loss corresponds to 66–7%7\% in compute. At the predicted compute, the loss of the mixture differs from the pure-text target by a mean of −0.002-0.002 to +0.003+0.003.
  • The added-token budget. Separately, Γobs=Ctext,obs/C0≤1\Gamma_{\text{obs}} = C_{\text{text,obs}}/C_0 \le 1 in 4747 of 4848 comparisons (eight budgets, three mixtures and two definitions of the observed frontier), so the compute supplied by the added multimodal tokens suffices to reach the pure-text target.
  • Stability. Across the full fit and the five held-out fitting subsets (withholding A3B, withholding A874M and A3B, and three compute cuts), Γr\Gamma_r stays within 0.9450.945–0.9600.960, 0.9110.911–0.9250.925 and 0.8820.882–0.9090.909. Across leave-one-size-out refits it stays within [0.943,0.955][0.943, 0.955], [0.894,0.939][0.894, 0.939] and [0.870,0.929][0.870, 0.929], and across the compute accountings of Appendix D.7 it moves by at most 0.0220.022.
  • Limits. Individual budgets are too noisy to resolve the sign of the total-compute penalty at r=0.1r=0.1, which is positive at only 44 of 88 envelope budgets; the pooled joint fit resolves it (+4.7%+4.7\%, 95%95\% interval [+2.3%,+8.3%][+2.3\%, +8.3\%]). The held-out budgets come from overlapping trajectories of the same runs and are therefore not independent replicates.

Q4. Independently trained configurations

Do independently trained configurations selected by the laws achieve the predicted losses?

We agree this would be the strongest test. A new intermediate-size run at an informative budget costs roughly as much as a full A3B run and could not be completed within the discussion period; the whole-run holdout above is what the existing sweep supports, and Appendix F states the limitation.

We thank the reviewer again for suggesting these tests; they have substantially strengthened the paper.

Reviewer_U4xPOpenReview ↗

Response

I thank the author for the experiments. This resolves my concern, and I'll keep my positive score

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors' response. Evaluate whether your concerns are addressed or if anything needs more attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.