- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how removing a pretrained visual encoder affects the scaling behavior of multimodal language models. The authors compare encoder-free and encoder-based architectures across a shared ladder of 11 sparse MoE decoders, controlling the training data, optimization settings, and visual-token granularity. They find that encoder-free models favor a larger decoder allocation as multimodal training compute increases, while the text-loss scaling trends remain similar. Although encoder-free models underperform throughout the measured multimodal range, fitted scaling laws predict a crossover at approximately FLOPs. The paper also examines attention patterns, layerwise representation changes, and expert routing to understand how the decoder adapts to processing raw visual inputs.
Critical strengths
-
Understanding when a pretrained visual encoder remains beneficial is relevant to the design and compute allocation of future multimodal models. The paper moves beyond demonstrating encoder-free feasibility and attempts to quantify its relative efficiency.
-
The shared decoder ladder, matched visual-token granularity, and common training setup help isolate architectural differences. Separating text and multimodal objectives provides a useful control, while the causal-attention ablation helps assess whether the allocation shift is primarily attributable to the attention mask.
-
The appendix includes an alternative frontier estimator, held-out extrapolation checks, sensitivity analysis for the irreducible loss, conditional bootstrap intervals, and additional compute accounting. These analyses strengthen the evidence for the observed scaling trends, although they do not fully establish the predicted crossover.
Critical weaknesses
-
The fixed visual encoder substantially limits the scope of the architectural conclusion. Throughout the experiments, the encoder-based models use an approximately 400M-parameter SigLIP 2 encoder while the decoder continues to grow. This setup establishes scaling behavior relative to a particular fixed visual front end. It does not establish when encoder-free models become preferable to an encoder-based system whose encoder size, decoder size, and training duration are jointly optimized under the same total compute budget. The conclusion may also depend on the encoder’s pretraining objective and representations. Larger encoders or encoders with different visual priors—for example, the self-supervised DINOv3-7B—could change the relative scaling behavior. DINOv3 provides a concrete example of a substantially larger encoder trained under a different supervision regime. Whether such an alternative improves the compute–performance tradeoff remains an empirical question. The paper acknowledges the fixed-encoder setting, but this qualification should be more prominent in its central claims.
-
The predicted crossover is insufficiently validated to support the paper’s strongest conclusion. The main multimodal fits use six compute budgets spanning FLOPs, whereas the compute-optimal crossover is estimated at FLOPs. Thus, the crossover lies approximately six times beyond the largest fitted budget and is not directly observed. The held-out multimodal check validates a smaller, 2.5-fold extrapolation. Furthermore, the prediction assumes that the fitted power laws continue to hold and that both architectures share an irreducible loss floor. The bootstrap intervals characterize uncertainty conditional on this modeling framework; they do not fully capture uncertainty from the functional form, training randomness, or the estimation of IsoFLOP minima. For overtraining, the fitted multipliers are held fixed during bootstrap resampling. Consequently, the evidence supports faster improvement within the studied regime more strongly than it supports a reliable crossover near FLOPs.
-
Downstream results support a narrowing gap but do not establish practical parity. The downstream evaluation is a useful addition, yet substantial differences remain at the largest reported model size. In Table 6, the average score is 57.7 for encoder-based models versus 51.2 for encoder-free models, with larger gaps on ChartQA and DocVQA. Matching aggregate validation loss therefore should not be assumed to imply matching visual capabilities. Task-specific comparisons at matched total compute would provide stronger evidence for the practical significance of the predicted crossover.
-
Important details needed to reproduce the scaling estimates are missing. The manuscript describes learning-rate and batch-size selection through an internal scaling law without providing the full per-model settings. More complete architecture configurations, data-mixture specifications, and raw measurements would make it easier to assess whether the estimated differences are robust to implementation and optimization choices.
Questions for authors
-
How would the results change if encoder size and decoder size were jointly optimized under a fixed total training-compute budget? Could a limited sweep over multiple encoder sizes establish whether the current fixed-encoder baseline approximates this optimum?
-
Have you evaluated encoders with different pretraining objectives or visual representations, such as DINOv3, in addition to SigLIP 2? Which conclusions do you expect to transfer across encoder families, and what evidence supports that expectation?
15 comments
Response to Reviewer GFhd (2/5)
W1 & Q2: Encoders with different pretraining objectives.
Following the reviewer's suggestion, we have evaluated two encoders with different pretraining objectives, the generative GenLIP and the self-supervised DINOv3 (Appendix H). Specifically, we replace SigLIP2 in the encoder-based 6.1B-A425M model with GenLIP or DINOv3, keeping all other settings identical to the main ladder, and report the multimodal validation loss:
| Visual encoder | Objective | Params | MM loss (6.6B tokens) | MM loss (14.7B tokens) | MM loss (29.5B tokens) |
|---|---|---|---|---|---|
| None (encoder-free) | – | – | 1.773 | 1.499 | 1.367 |
| SigLIP 2 So400m/16 | Contrastive | 400M | 1.398 | 1.084 | 0.953 |
| GenLIP So400m/16 | Generative | 470M | 1.289 | 1.016 | 0.903 |
| DINOv3 ViT-L/16 | Self-supervised | 300M | 1.361 | 1.055 | 0.925 |
Both alternative encoders yield lower multimodal loss than SigLIP 2, but the differences among encoders are small compared with their lead over the encoder-free model. At 29.5B tokens, the three encoders differ by at most 0.05, whereas all of them lead the encoder-free model by 0.41–0.46. Their advantage over SigLIP 2 also narrows with training, from 0.109 to 0.050 for GenLIP and from 0.036 to 0.028 for DINOv3 between 6.6B and 29.5B tokens. Notably, DINOv3 outperforms SigLIP 2 despite being pretrained without language supervision. A possible explanation is that, because the encoder is trained jointly with the decoder, language alignment can be acquired during multimodal pretraining, so the quality of patch-level features matters more than prior alignment. Based on these results and the mechanisms analyzed in the paper, we expect the conclusions to transfer across encoder families as follows:
- Text scaling does not involve the encoder and transfers directly.
- The shift toward larger decoders and the faster-decreasing frontier arise because the decoder must learn visual representations by itself, which does not depend on the encoder used as the baseline. We therefore expect these trends to transfer.
- The location of the crossover depends on the encoder: a stronger visual prior, whether from a better pretraining objective or a larger encoder, delays the crossover. The results above suggest that this delay is modest for encoders.
Response to Reviewer GFhd (3/5)
W2: Validation of the predicted crossover.
We understand the reviewer's concern that the crossover is extrapolated rather than directly observed. Directly observing it would require training both encoder-free and encoder-based models at the crossover scale, which we estimate to cost about 491,000 GPU hours. Predicting such large-scale behavior from small-scale runs is precisely the purpose of scaling laws. As the reviewer kindly noted in the strengths, we follow prior scaling studies such as Chinchilla [1] and extensively cross-check the extrapolation in Appendix B, using an alternative frontier estimator, held-out extrapolation checks, a sensitivity analysis of the irreducible loss, and conditional bootstrap intervals.
Moreover, the predicted crossover lies at a practically relevant scale. Its point estimate of FLOPs is comparable to the compute required to pretrain a typical visual encoder alone: about FLOPs for GenLIP So400m/16 (8B samples) and about FLOPs for SigLIP 2 So400m/16 (40B samples), estimated as from their reported training setups. In other words, the compute that the encoder-free model needs to catch up is of the same order as the compute already spent on the visual prior it replaces.
Response to Reviewer GFhd (1/5)
We sincerely thank you for the constructive comments and valuable suggestions. Thank you for the time and effort you have devoted to assessing our manuscript and helping us improve its quality. Our detailed responses to your concerns are provided below.
We thank the reviewer for this insightful suggestion. We agree that the choice of encoder affects the results. Following the reviewer's suggestion, we have revised Section 3.2.1 (Lines 263-264) to clarify that our conclusions are obtained with a fixed SigLIP 2 So400m encoder. Below, we explain why we fix an encoder of about 400M parameters rather than scaling it jointly with the decoder, and how the encoder affects our conclusions.
(1) Unlike language models, visual encoders have not been shown to scale. Many efforts have pushed ViTs to billions of parameters (e.g., InternViT-6B), but the gains saturate quickly, no stable power law has emerged, and such encoders are rarely adopted in practice. Co-scaling the encoder would therefore add a saturating component to our fits: a larger ViT still helps at the small decoder scales where the sweep has to be run, but brings little gain at the large scales we extrapolate to, so the fitted law would not extrapolate reliably. Fixing the encoder keeps the decoder, whose scaling is well characterized, as the only extrapolated dimension.
(2) The encoder affects our conclusions through the strength of its visual prior rather than its size, since we count only decoder compute. A larger encoder is therefore expected to delay the crossover only as it provides a stronger visual prior, similar to the stronger encoders in Q2 (Appendix H). We exclude the encoder from compute accounting because its size is negligible at practical scales (e.g., about 1% of a decoder with 32B activated parameters), whereas in our small runs it is comparable to or larger than the activated decoder (e.g., 2.3B-A150M) and would dominate the fitted laws. Counting the ViT cost would only move the crossover earlier (Appendix C).
(3) Consistent with this, current practice suggests that a 400M encoder is close to the practical optimum. If the optimal encoder size grew with the decoder, recent MLLMs would already pair their largest decoders with much larger encoders; instead, systems such as Qwen3-VL and Kimi K2.5 keep encoders of about 400M parameters. An empirical sweep would also require a family of encoders pretrained with a uniform recipe across a wide range of sizes, whereas public families provide only a few discrete sizes, which is too coarse to match our 11-rung decoder ladder. Fixing a 400M encoder is therefore the most representative setting available for our study.