- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors train a "native multimodal" Mixture-of-Experts (MoE) model from scratch, eschewing a pre-trained vision encoder in favor of a single-layer patch embedding. Using both Chinchilla-style IsoFLOP analysis and training-curve envelope fitting, they determine the compute-optimal allocation—the balance between model size N and the number of tokens D—for both text-only and multimodal loss. Based on this, they propose a "budget compensation rule." Finally, downstream evaluations demonstrate three key outcomes: text-processing capabilities remain unimpaired, spatial reasoning within the text-only domain is enhanced, and the model emerges with multimodal few-shot learning abilities.
Critical strengths
- The problem is significant: there is a lack of systematic characterization regarding computational allocation in natively multimodal models—a domain where many teams are currently shifting from late-fusion approaches toward native training.
- The scale of the experiment is substantial overall; fitting language objectives and multimodal objectives separately represents a sound improvement over using a single, aggregate loss function.
Critical weaknesses
- All models, including the 71M one, use the same 8M-token batch. That is likely well above the critical batch size of the small models. It penalizes them at low compute and pushes N_opt upward. The paper does not say how the per-size learning rates were tuned.
- There are only six model sizes, with an uneven gap between 874M and 3B. The quadratic fits clip N_opt to the observed range, and no confidence intervals or bootstrap estimates are given for a, b, E, β or Γ.
- "In-context learning emerges" rests on a +2.4-point gain at the largest scale. The few-shot setting also changes the visual token budget per image, and no chance-level reference is given per benchmark. The wording is stronger than the evidence.
Questions for authors
- How many loss-bearing text tokens does the multimodal corpus contain? Do the Γ_r < 1 and SpatialEval results hold against a text-only model trained on the same total number of loss-bearing tokens?
- How do the joint and separate fits compare on held-out compute, e.g., the A874M/A3B segments beyond the fitted range? Please give bootstrap CIs for Γ_r and all exponents.
- How do the exponents change with annealed (cooldown) losses and full FLOP accounting? Do the few-shot gains hold when 0-shot also uses 512 visual tokens?
6 comments
Response to Reviewer 158M [2/5]
Q3. Clipping of the IsoFLOP vertices
The quadratic fits clip to the observed range.
Of the IsoFLOP budgets, have a vertex clipped to the observed size range and profile is non-convex. All four lie just below A3B's first checkpoint, where A874M is the largest available model. Excluding them changes any exponent by at most (e.g., text at : ). Appendix D.6 reports the audit.
Q4. Confidence intervals for the fitted quantities
No confidence intervals or bootstrap estimates are given for , , , or . Please give bootstrap CIs for and all exponents.
We added two analyses (Appendix D.6, Table 5). First, a moving-block bootstrap of within-run noise: each run's smoothed curve is split into a trend and residuals (SD for text, for multimodal), residual blocks are resampled jointly across sizes because all sizes see the same data order and their residuals are correlated at about , and the entire pipeline is rerun times, including the budget grid, the envelopes, the vertices with clipping, the regressions and both frontier fits. Second, a leave-one-model-size-out jackknife. Neither covers seed variance, since each configuration was trained once.
| Quantity | ||||
|---|---|---|---|---|
| 0.650 [0.631, 0.694] | 0.645 [0.615, 0.670] | 0.661 [0.640, 0.680] | 0.649 [0.622, 0.668] | |
| – | 0.686 [0.613, 0.704] | 0.649 [0.624, 0.683] | 0.622 [0.615, 0.661] | |
| – | 0.952 [0.930, 0.985] | 0.915 [0.901, 0.950] | 0.899 [0.884, 0.936] | |
| – | [, ] | [, ] | [, ] |
Leave-one-size-out ranges: within –, within –, and within , and .
- Resolved. and a positive total-compute penalty at every ratio, in every interval, every refit and every accounting; ; and no detectable difference among the text exponents across mixtures, all pairwise intervals being within .
- Not resolved. .
- The separate multimodal offsets are weakly identified, with intervals as wide as , so we do not interpret them and use the multimodal frontiers only within their fitted range. Table 5 also reports intervals for and , for instance and for the joint fit; intervals for follow from those for , since .
Q5. The multimodal allocation trend
(Bears on the claim in Sec. 3.2.)
Under the main compute accounting, both estimators show a decreasing point estimate across the three mixtures, and the decrease is negative in all six leave-one-size-out refits of the IsoFLOP estimate:
| Estimator | |||
|---|---|---|---|
| IsoFLOP (primary) | 0.686 | 0.649 | 0.622 |
| Training-curve envelope | 0.561 | 0.558 | 0.551 |
The size of the decrease is not statistically resolved: the envelope falls by against the IsoFLOP decrease of ; the interval of the IsoFLOP difference, , includes zero; and under the three alternative accountings the difference ranges from to . The revision reports this trend together with these intervals (Sec. 3.2, Appendices D.6 and D.7).
Q6. The two estimators
(Related to Q5.)
The envelope is – noisier than IsoFLOP, with bootstrap SD – against –, because it switches between near-tied sizes whose losses differ by only – at the switching budgets. We therefore take absolute levels from the IsoFLOP estimates, as stated in Sec. 2.2, and use the envelope for the direction of change; the IsoFLOP text exponents lie within the envelope's leave-one-size-out ranges. Both estimators reuse the same runs, so their agreement is a sensitivity check rather than independent evidence.
Q7. Six model sizes and the M–B gap
There are only six model sizes, with an uneven gap between M and B.
We agree the spacing is uneven. The frontier is not destabilized by it: removing every A3B run before fitting, freezing the parameters and predicting A3B about one decade beyond the fitted range gives a text-loss MAPE of – (Appendix D.5). The allocation exponents are the sensitive quantity, shifting by up to when A3B is removed, which the revision now reports explicitly.
Response to Reviewer 158M [3/5]
Q8. The evidence behind the in-context-learning claim
"In-context learning emerges" rests on a -point gain at the largest scale, and the wording is stronger than the evidence.
We agree the wording was too strong. Recomputing the -shot gain over -shot from the last three checkpoints of each of the three mixtures gives nine evaluations per size, drawn from three training trajectories, on the paper's -benchmark set. The at-chance set below contains the six benchmarks on which A3B is within three points of chance at -shot: MME, HallusionBench, CV-Bench, OmniSpatial, SEAM and SpatialEval (Appendix E.1, Table 8).
| Size | Paper (final ckpt) | Replicates: mean SD (#) | At-chance set () | Other benchmarks () |
|---|---|---|---|---|
| A71M | (/) | |||
| A128M | (/) | |||
| A340M | (/) | |||
| A590M | (/) | |||
| A874M | (/) | |||
| A3B | (/) |
- The A3B gain is robust across the evaluated checkpoints and mixtures. It is positive in all nine evaluations, with a mean of points; the standard deviation of reflects variation across checkpoints and mixtures rather than across independent training runs, and a conservative item-level interval on the final-checkpoint gain is .
- The increase with scale is not monotone. A128M through A874M are flat at about , and the single-checkpoint value of at A874M is the largest of that model's nine evaluations, whose mean is . The revision reports these means and no longer implies a monotone increase.
- Where the gain lives. On the remaining fifteen benchmarks, demonstrations add to at every size from A128M upward, with no trend. The scale dependence comes from the six at-chance benchmarks, where demonstrations appear mainly to convey the answer format: A3B scores exactly on five of the fourteen MME categories at -shot, the signature of a constant binary answer.
Revised wording (Sec. 4.3). "Few-shot prompting yields small but consistent gains from M upward that roughly double at B (), largest where the model is at chance without demonstrations. We read this as the onset of multimodal in-context learning in natively pre-trained base models, rather than as a fully emerged capability."
Q9. Chance-level references per benchmark
No chance-level reference is given per benchmark.
We now report, for every benchmark, the expected accuracy of a uniform random guess under our protocol, computed from the exact scored item pools, together with a constant-answer baseline for the open-ended tasks (Table 21, Appendix E.1). Eight of the benchmarks are at or near chance at -shot for every model size.
This also leads us to withdraw the claim that few-shot gains concentrate on spatial reasoning. Four of our five spatial benchmarks (CV-Bench, OmniSpatial, SEAM, SpatialEval) are among the eight at or near their reference; the one that is above it, RealWorldQA, gains little; and the non-spatial MME, also at chance for A3B, gains . Being at chance without demonstrations and belonging to the spatial category are not separable in our suite.
Q10. The visual-token budget in the few-shot setting
The few-shot setting also changes the visual token budget per image. Do the few-shot gains hold when -shot also uses visual tokens?
The confound does not arise; our original sentence was ambiguous. Across all benchmarks the test image is encoded with up to tokens in -, - and -shot alike, and only the in-context demonstration images are capped at tokens so that the templates fit the context window. The -shot and -shot runs therefore see identical test inputs. On five benchmarks (MMBench-en, VQAv2, ScienceQA, MMBench-cc, SEAM) no test image exceeds tokens at all. We have corrected the sentence in Sec. 4.1.
Response to Reviewer 158M [1/5]
We sincerely thank the reviewer for an exceptionally precise review, and for noting that compute allocation in natively multimodal models lacks a systematic characterization and that the scale of the experiments is substantial. Every point below is answered with a new analysis, and where the evidence does not support our original wording we revise the claim. All changes are marked in blue in the revised manuscript.
One correction first. The review credits us for fitting the language and multimodal objectives separately. Table 2 of Shukor et al. (2025b) already fits the law per evaluation data type, so per-objective fitting is established practice. Its value here lies in the design: we hold the pure-text budget fixed and add multimodal tokens on top, so each objective's change across r is attributable to the multimodal data rather than to a simultaneous change in the pure-text budget.
Q1. How the per-size learning rates were tuned
The per-size architecture, the batch size and the peak learning rate were taken from a scaling ladder fitted during an earlier architecture exploration and reused unchanged for every mixture, so that the available compute could be spent on the data-mixture sweep. We now report them explicitly in Appendix B. The rates follow a single power law:
lr∝N−0.570,R2=0.997,
with every size within 6.5% of the rule (A71M 0.97×, A128M 0.99×, A340M 1.01×, A590M 1.03×, A874M 1.07×, A3B 0.94×). This is not a μP parameterization, since lr×width falls from 1.66 to 0.61 across the family. Every run warms up over 2,000 steps and then holds the rate constant.
Scope. Our allocation estimates characterize this training recipe. We did not re-tune the learning rate for this study; Appendix F lists this as a limitation.
Q2. The shared 8M-token batch
We agree that a shared batch can bias Nopt upward, and three observations bound the concern.
References