acceptodds
Desk rejected at ICLR 2027

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: **(1)** Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. **(2)** The two architectures exhibit nearly overlapping loss–compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around FLOPs, well within practical pretraining budgets. **(3)** Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Encoder-Free, Scaling Laws, Multimodal Pretraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GFhdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how removing a pretrained visual encoder affects the scaling behavior of multimodal language models. The authors compare encoder-free and encoder-based architectures across a shared ladder of 11 sparse MoE decoders, controlling the training data, optimization settings, and visual-token granularity. They find that encoder-free models favor a larger decoder allocation as multimodal training compute increases, while the text-loss scaling trends remain similar. Although encoder-free models underperform throughout the measured multimodal range, fitted scaling laws predict a crossover at approximately 102210^{22} FLOPs. The paper also examines attention patterns, layerwise representation changes, and expert routing to understand how the decoder adapts to processing raw visual inputs.

Critical strengths
  1. Understanding when a pretrained visual encoder remains beneficial is relevant to the design and compute allocation of future multimodal models. The paper moves beyond demonstrating encoder-free feasibility and attempts to quantify its relative efficiency.

  2. The shared decoder ladder, matched visual-token granularity, and common training setup help isolate architectural differences. Separating text and multimodal objectives provides a useful control, while the causal-attention ablation helps assess whether the allocation shift is primarily attributable to the attention mask.

  3. The appendix includes an alternative frontier estimator, held-out extrapolation checks, sensitivity analysis for the irreducible loss, conditional bootstrap intervals, and additional compute accounting. These analyses strengthen the evidence for the observed scaling trends, although they do not fully establish the predicted crossover.

Critical weaknesses
  1. The fixed visual encoder substantially limits the scope of the architectural conclusion. Throughout the experiments, the encoder-based models use an approximately 400M-parameter SigLIP 2 encoder while the decoder continues to grow. This setup establishes scaling behavior relative to a particular fixed visual front end. It does not establish when encoder-free models become preferable to an encoder-based system whose encoder size, decoder size, and training duration are jointly optimized under the same total compute budget. The conclusion may also depend on the encoder’s pretraining objective and representations. Larger encoders or encoders with different visual priors—for example, the self-supervised DINOv3-7B—could change the relative scaling behavior. DINOv3 provides a concrete example of a substantially larger encoder trained under a different supervision regime. Whether such an alternative improves the compute–performance tradeoff remains an empirical question. The paper acknowledges the fixed-encoder setting, but this qualification should be more prominent in its central claims.

  2. The predicted crossover is insufficiently validated to support the paper’s strongest conclusion. The main multimodal fits use six compute budgets spanning 1020–102110^{20} – 10^{21} FLOPs, whereas the compute-optimal crossover is estimated at 6.1×10216.1\times10^{21} FLOPs. Thus, the crossover lies approximately six times beyond the largest fitted budget and is not directly observed. The held-out multimodal check validates a smaller, 2.5-fold extrapolation. Furthermore, the prediction assumes that the fitted power laws continue to hold and that both architectures share an irreducible loss floor. The bootstrap intervals characterize uncertainty conditional on this modeling framework; they do not fully capture uncertainty from the functional form, training randomness, or the estimation of IsoFLOP minima. For overtraining, the fitted multipliers are held fixed during bootstrap resampling. Consequently, the evidence supports faster improvement within the studied regime more strongly than it supports a reliable crossover near 102210^{22} FLOPs.

  3. Downstream results support a narrowing gap but do not establish practical parity. The downstream evaluation is a useful addition, yet substantial differences remain at the largest reported model size. In Table 6, the average score is 57.7 for encoder-based models versus 51.2 for encoder-free models, with larger gaps on ChartQA and DocVQA. Matching aggregate validation loss therefore should not be assumed to imply matching visual capabilities. Task-specific comparisons at matched total compute would provide stronger evidence for the practical significance of the predicted crossover.

  4. Important details needed to reproduce the scaling estimates are missing. The manuscript describes learning-rate and batch-size selection through an internal scaling law without providing the full per-model settings. More complete architecture configurations, data-mixture specifications, and raw measurements would make it easier to assess whether the estimated differences are robust to implementation and optimization choices.

Questions for authors
  1. How would the results change if encoder size and decoder size were jointly optimized under a fixed total training-compute budget? Could a limited sweep over multiple encoder sizes establish whether the current fixed-encoder baseline approximates this optimum?

  2. Have you evaluated encoders with different pretraining objectives or visual representations, such as DINOv3, in addition to SigLIP 2? Which conclusions do you expect to transfer across encoder families, and what evidence supports that expectation?

15 comments
AuthorsOpenReview ↗

Response to Reviewer GFhd (1/5)

We sincerely thank you for the constructive comments and valuable suggestions. Thank you for the time and effort you have devoted to assessing our manuscript and helping us improve its quality. Our detailed responses to your concerns are provided below.


W1 & Q1: Fixed encoder vs. joint encoder–decoder scaling.

We thank the reviewer for this insightful suggestion. We agree that the choice of encoder affects the results. Following the reviewer's suggestion, we have revised Section 3.2.1 (Lines 263-264) to clarify that our conclusions are obtained with a fixed SigLIP 2 So400m encoder. Below, we explain why we fix an encoder of about 400M parameters rather than scaling it jointly with the decoder, and how the encoder affects our conclusions.

(1) Unlike language models, visual encoders have not been shown to scale. Many efforts have pushed ViTs to billions of parameters (e.g., InternViT-6B), but the gains saturate quickly, no stable power law has emerged, and such encoders are rarely adopted in practice. Co-scaling the encoder would therefore add a saturating component to our fits: a larger ViT still helps at the small decoder scales where the sweep has to be run, but brings little gain at the large scales we extrapolate to, so the fitted law would not extrapolate reliably. Fixing the encoder keeps the decoder, whose scaling is well characterized, as the only extrapolated dimension.

(2) The encoder affects our conclusions through the strength of its visual prior rather than its size, since we count only decoder compute. A larger encoder is therefore expected to delay the crossover only as it provides a stronger visual prior, similar to the stronger encoders in Q2 (Appendix H). We exclude the encoder from compute accounting because its size is negligible at practical scales (e.g., about 1% of a decoder with 32B activated parameters), whereas in our small runs it is comparable to or larger than the activated decoder (e.g., 2.3B-A150M) and would dominate the fitted laws. Counting the ViT cost would only move the crossover earlier (Appendix C).

(3) Consistent with this, current practice suggests that a 400M encoder is close to the practical optimum. If the optimal encoder size grew with the decoder, recent MLLMs would already pair their largest decoders with much larger encoders; instead, systems such as Qwen3-VL and Kimi K2.5 keep encoders of about 400M parameters. An empirical sweep would also require a family of encoders pretrained with a uniform recipe across a wide range of sizes, whereas public families provide only a few discrete sizes, which is too coarse to match our 11-rung decoder ladder. Fixing a 400M encoder is therefore the most representative setting available for our study.

AuthorsOpenReview ↗

Response to Reviewer GFhd (2/5)

W1 & Q2: Encoders with different pretraining objectives.

Following the reviewer's suggestion, we have evaluated two encoders with different pretraining objectives, the generative GenLIP and the self-supervised DINOv3 (Appendix H). Specifically, we replace SigLIP2 in the encoder-based 6.1B-A425M model with GenLIP or DINOv3, keeping all other settings identical to the main ladder, and report the multimodal validation loss:

Visual encoderObjectiveParamsMM loss (6.6B tokens)MM loss (14.7B tokens)MM loss (29.5B tokens)
None (encoder-free)––1.7731.4991.367
SigLIP 2 So400m/16Contrastive400M1.3981.0840.953
GenLIP So400m/16Generative470M1.2891.0160.903
DINOv3 ViT-L/16Self-supervised300M1.3611.0550.925

Both alternative encoders yield lower multimodal loss than SigLIP 2, but the differences among encoders are small compared with their lead over the encoder-free model. At 29.5B tokens, the three encoders differ by at most 0.05, whereas all of them lead the encoder-free model by 0.41–0.46. Their advantage over SigLIP 2 also narrows with training, from 0.109 to 0.050 for GenLIP and from 0.036 to 0.028 for DINOv3 between 6.6B and 29.5B tokens. Notably, DINOv3 outperforms SigLIP 2 despite being pretrained without language supervision. A possible explanation is that, because the encoder is trained jointly with the decoder, language alignment can be acquired during multimodal pretraining, so the quality of patch-level features matters more than prior alignment. Based on these results and the mechanisms analyzed in the paper, we expect the conclusions to transfer across encoder families as follows:

  1. Text scaling does not involve the encoder and transfers directly.
  2. The shift toward larger decoders and the faster-decreasing frontier arise because the decoder must learn visual representations by itself, which does not depend on the encoder used as the baseline. We therefore expect these trends to transfer.
  3. The location of the crossover depends on the encoder: a stronger visual prior, whether from a better pretraining objective or a larger encoder, delays the crossover. The results above suggest that this delay is modest for encoders.
AuthorsOpenReview ↗

Response to Reviewer GFhd (3/5)

W2: Validation of the predicted crossover.

We understand the reviewer's concern that the crossover is extrapolated rather than directly observed. Directly observing it would require training both encoder-free and encoder-based models at the crossover scale, which we estimate to cost about 491,000 GPU hours. Predicting such large-scale behavior from small-scale runs is precisely the purpose of scaling laws. As the reviewer kindly noted in the strengths, we follow prior scaling studies such as Chinchilla [1] and extensively cross-check the extrapolation in Appendix B, using an alternative frontier estimator, held-out extrapolation checks, a sensitivity analysis of the irreducible loss, and conditional bootstrap intervals.

Moreover, the predicted crossover lies at a practically relevant scale. Its point estimate of 102210^{22} FLOPs is comparable to the compute required to pretrain a typical visual encoder alone: about 6.5×10216.5\times10^{21} FLOPs for GenLIP So400m/16 (8B samples) and about 2.5×10222.5\times10^{22} FLOPs for SigLIP 2 So400m/16 (40B samples), estimated as 6ND6ND from their reported training setups. In other words, the compute that the encoder-free model needs to catch up is of the same order as the compute already spent on the visual prior it replaces.

Reviewer_1QQoOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores removing visual encoders from MoE-based MLLMs. The paper conducts compute-optimal allocation analysis by varying training FLOPS and concludes that removing visual encoders leads to more computation for multimodal objective. It further display the trends of multimodfal objective's catching up with model scaling. Additional analysis shows that removing visual encoders triggers different visual attention patterns, layer-wise representations, and expert routing behaviours. The paper overall serves as a further demonstration that encoder-free MLLMs have potentials for catching up with encoder-based MLLMs.

Critical strengths
  1. The controlled analysis is sufficient for the main claims of the paper, with fixed decoder ladder, data curation, and optimization settings, providing a fair and clear comparison between encoder-based and encoder-free MLLMs.
  2. The comparison specifically for the multimodal objective and with various topics is informative, leading the readers to take a closer look at the difference between the two architectures.
  3. The experiments are overall sufficient and thorough, including the robust test, results on multiple benchmarks across different sizes of models, etc.
Critical weaknesses
  1. Motivation: While this paper conducts extensive experiments to show the trend that by scaling the computation, the encoder-free MLLM can possibly catch up with its encoder-based variant, it fails to directly demonstrate the superiority of removing visual encoders from training/inference perspectives. How will the model behave if we continue scaling the computation after catching up? The authors may also discuss the inference differences between the two architectures. Will the encoder-free architectures be more efficient or stable at inference time?
  2. Only one vision encoder is included. The SigLIP 2 is itself trained on a large burden of visual data. There are also other kinds of vision encoders. The authors may discuss the differences between vision encoders with varied capacities. Will the catch-up point arrive early or late when compared with inferior/superior vision encoders?
  3. It will be more interesting and convincing if Table 6 in the Appendix shows that the encoder-free model finally surpasses or reaches a comparable level to encoder-based models.
Questions for authors

Please see the Critical Weaknesses. The rating will be increased if some of the concerns are addressed.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1QQo (1/3)

We sincerely thank you for the constructive comments and valuable suggestions. Thank you for the time and effort you have devoted to assessing our manuscript and helping us improve its quality. Our detailed responses to your concerns are provided below.


W1: Motivation for removing the visual encoder.

We thank the reviewer for this insightful question. We agree that the original manuscript focused on whether encoder-free models can catch up with encoder-based models, but did not explicitly discuss why removing the visual encoder is desirable once they do. Making this discussion explicit indeed clarifies the motivation of our study. We think that once encoder-free models catch up, removing the visual encoder no longer sacrifices performance and brings practical benefits in the following aspects:

  1. Training and inference infrastructure. The ViT and the decoder differ in architecture, sequence length, and parallelism strategy. During training, the two modules must share the same devices through time-division multiplexing, which complicates parallelization and load balancing. During inference, the ViT is usually deployed separately and must finish before decoder prefill, which lengthens the time to first token. An encoder-free model is a single Transformer and reuses the infrastructure of language models.
  2. Scaling laws. Scaling laws are fitted on many small-scale runs and then extrapolated to large scales. The ViT cost does not shrink with the decoder, so it dominates the training time of exactly these small runs. The ViT also requires its own hyperparameters, which must be scaled separately from those of the decoder. Without the encoder, existing scaling law methodology for language models applies directly.

Moreover, the reviewer raises an interesting question on how the models behave when scaling beyond the crossover. This question is challenging to answer, as this regime lies far beyond our experimental range. Nevertheless, since the encoder-free model has a larger scaling exponent, our fitted laws imply that it would attain a slightly lower loss than the encoder-based model after the crossover (by about 6.8×10−36.8\times10^{-3} in loss at 102510^{25} FLOPs). However, since extrapolation becomes less reliable further from our experimental range, we do not claim this as a conclusion of our paper and leave its verification to future large-scale experiments.

We have added this discussion as a new appendix section (Appendix G, "Discussion of Encoder-Free Advantages"). We sincerely appreciate the reviewer's comments, which help improve the quality of our paper.

AuthorsOpenReview ↗

Response to Reviewer 1QQo (2/3)

W2: Effect of the vision encoder on the catch-up point.

We chose SigLIP because encoders of about 400M parameters are a representative setting in recent MLLMs such as Qwen3-VL and Kimi K2.5. Following the reviewer's suggestion, we further compare encoders with different pretraining objectives (Appendix H). Specifically, we replace SigLIP in the encoder-based 6.1B-A425M model with GenLIP (generative next-token prediction) and DINO (self-supervised), keeping all other settings identical to the main ladder, and report the multimodal validation loss:

Visual encoderObjectiveParamsMM loss (6.6B tokens)MM loss (14.7B tokens)MM loss (29.5B tokens)
None (encoder-free)––1.7731.4991.367
SigLIP2 So400m/16Contrastive400M1.3981.0840.953
GenLIP So400m/16Generative470M1.2891.0160.903
DINOv3 ViT-L/16Self-supervised300M1.3611.0550.925

Both alternative encoders yield lower multimodal loss than SigLIP 2, so they would delay the crossover, but we expect this delay to be modest. First, at 29.5B tokens the three encoders differ by at most 0.05 in loss, whereas all of them lead the encoder-free model by 0.41–0.46. Second, their advantage over SigLIP 2 narrows with training: from 6.6B to 29.5B tokens, it shrinks from 0.109 to 0.050 for GenLIP and from 0.036 to 0.028 for DINOv3. Hence, we expect the choice of encoder to shift the location of the crossover moderately rather than change our conclusions.

We have added the above comparison of encoders with different pretraining objectives as a new appendix section (Appendix H), and have clarified in Section 3.2.1 (Lines 263-264) that the crossover estimate is conditional on the choice of encoder. We sincerely thank the reviewer for this comment, which has greatly helped us improve the quality of our paper!

AuthorsOpenReview ↗

Response to Reviewer 1QQo (3/3)

W3: Downstream results in Table 6.

We thank the reviewer for this suggestion. We agree that encoder-free models in Table 6 have not yet reached the level of encoder-based models. This is expected from our scaling laws: all checkpoints in Table 6 are trained with less than 102110^{21} FLOPs, which is still below the predicted crossover. In this regime, encoder-free models also have a higher multimodal loss than encoder-based models.

Nevertheless, the downstream gap already narrows as the model scales up, following the same trend as the loss. The table below reports the difference in average score between encoder-free and encoder-based models at 100B training tokens. As the model grows from 8B-A577M to 33B-A2.2B, the gap shrinks from −11.4 to −6.4.

Model8B-A577M13B-A922M16B-A1B22B-A1.6B33B-A2.2B
Free − Based−11.4−11.0−9.8−9.0−6.4

Moreover, we fully agree with the reviewer that showing encoder-free models reaching or surpassing encoder-based models in Table 6 would make our conclusion more interesting and convincing. However, this requires a long-horizon training run of about 491,000 GPU hours, which is far beyond our current resources, especially within the rebuttal period. Running this experiment is our top priority for future work, and we will report the results in a future version of the paper once the run is completed. We thank the reviewer again for this valuable suggestion!


We would be more than happy to discuss any further questions!

Reviewer_4CPbOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares multimodal pre-training with and without a visual encoder under a shared 11-expert sparse MoE decoder, mixed training setup, and visual token granularity. The authors fit IsoFLOP-optimal configurations and loss curves for text-only and multimodal objectives respectively. They find that the multimodal optimal configuration for the encoder-free scheme leans toward a larger decoder. Its multimodal loss remains higher within the observed range, yet fitted curves predict convergence at approximately the 10²² FLOPs scale. The paper further analyzes overtraining, topic divergence, bidirectional visual attention, inter-layer representation shifts, and expert routing.

Critical strengths

The paper reframes the question of "whether the visual encoder can be removed" into a quantitative problem concerning the allocation of model capacity, training data, and compute budget. Even if the long-term convergence prediction is not provisionally accepted, the shift in optimal configurations after removing visual priors still offers useful insights for the design and training planning of native multimodal models.

Critical weaknesses

Equal average pre-training loss does not necessarily imply concurrent parity in OCR, document understanding, or other perceptual capabilities. The paper’s own topic analysis also demonstrates markedly different rates of gap convergence. Downstream experiments provide supporting evidence, yet capability gaps persist within the observed regime, and it remains inconclusive under which task requirements removing the visual encoder is worthwhile.

Questions for authors

Please address the key concerns raised above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4CPb (1/1)

We sincerely thank you for the positive evaluation and the insightful comments. Thank you for the time and effort you have devoted to assessing our manuscript and helping us improve its quality. Our detailed responses to your concerns are provided below.


W1: Equal average loss does not imply parity on perceptual capabilities.

We agree that equal aggregate loss does not imply parity on every capability, and our topic analysis (Section 3.2.3) shows that topics catch up at different rates. Nevertheless, our fitted laws indicate that encoder-free models catch up on every topic, including perception-intensive ones. These topics require more compute to catch up, but the required compute is still well below that of current flagship pretraining.

Following the reviewer's comment, we have added the extrapolated crossover of each topic to Section 3.2.3 of the revised version (Lines 360–362). In the training-optimal setting, the crossover ranges from 1.5×10211.5\times10^{21} FLOPs on STEM, the earliest topic, to 6.1×10226.1\times10^{22} FLOPs on Caption, the latest. Under 5×5\times overtraining, it ranges from 2.8×10212.8\times10^{21} to 9.8×10229.8\times10^{22} FLOPs. Perception-intensive topics therefore require about an order of magnitude more compute than the aggregate crossover (6.1×10216.1\times10^{21} FLOPs). Even the latest topic crossover is still well below the pretraining compute of recent flagship models (about 102510^{25} FLOPs, e.g., Kimi K2.5). Hence, whether the visual encoder can be removed depends mainly on the available compute rather than on the target task.

We thank the reviewer for this valuable comment, which has helped us improve the quality of our paper!


We would be more than happy to discuss any further questions!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Relation to the concurrent work "Unbundling Multimodal Vision Encoding"

Dear Reviewers, ACs and readers,

We post this note for everyone interested in this topic. The concurrent work "Unbundling Multimodal Vision Encoding" [1] studies a question closely related to ours. Since the two works may appear to reach different conclusions at first glance, we briefly clarify how they relate.

Q1: Why do the two works seem to reach different conclusions?

The difference lies mainly in the comparison protocol. Our work and Scaling Laws for Native Multimodal Models [2] compare encoder-free and encoder-based architectures under a training-optimal protocol, where the N/DN/D allocation of each architecture is chosen to minimize its loss at every compute budget. [1] compares them at a fixed TPP (Figure 7) or with fixed 1.7B and 4B backbones (Figure 3), which are generally not training-optimal for either architecture. Re-solving the crossover from the fits reported in [1] under a training-optimal allocation, we obtain approximately 2.5×10212.5\times10^{21} FLOPs (our own estimate, for reference only). This is earlier than our estimate of 102210^{22} FLOPs, which is the expected direction: the encoder in [1] is trained from scratch, whereas ours is initialized from pretrained SigLIP 2 and thus provides a stronger encoder-based baseline.


Q2: Where do the two works agree?

Both works find that encoder-free models favour more parameters: [1] reports this directly, and our work shows that the training-optimal allocation of encoder-free models shifts toward larger decoders (Takeaway 1). We also agree with [1] that vision benefits from dedicated parameters. When parameters are shared, part of the capacity is taken up by text, so adding 400M parameters to the decoder is not necessarily equivalent to adding a 400M ViT.

Sincerely, Paper #8136 Authors


[1] Brown et al. Unbundling Multimodal Vision Encoding. arXiv:2611.17334, 2026.

[2] Shukor et al. Scaling Laws for Native Multimodal Models. ICCV, 2025.

AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, ACs, SACs and PCs,

We sincerely thank the reviewers for the time and effort dedicated to reviewing our paper. Your constructive feedback has significantly helped us improve the quality of our work. We also extend our gratitude to the ACs, SACs and PCs for handling our submission. To facilitate your assessment, we provide a consolidated summary of the rebuttal outcomes below.

Sincerely, Paper #8136 Authors


1. Summary of Our Paper

We compare the scaling laws of encoder-free and encoder-based MLLMs on a shared ladder of 11 MoE decoders with identical data and training settings. Our key takeaways are:

  • (1) Removing the visual encoder leaves the compute-optimal allocation on the text objective unchanged, but shifts multimodal training toward larger models.
  • (2) Both architectures nearly overlap on text loss frontiers, while encoder-free models initially lag on multimodal loss but are predicted to catch up within practical budgets.
  • (3) Without a visual encoder, the decoder learns to take over its role: bidirectional interactions among visual tokens become increasingly beneficial with compute, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated.

2. Summary of Strengths

  • Insightful Findings: All three reviewers found the study insightful, with 4CPb noting its useful insights for designing native MLLMs, and 1QQo and GFhd describing the analyses as informative.
  • Rigorous Experimental Design: Reviewers 1QQo and GFhd commended the controlled comparison, where both architectures share the same 11-rung MoE decoder ladder, training data, optimization settings and visual-token granularity.
  • Solid Results with Comprehensive Experiments: Reviewers 1QQo and GFhd praised the thorough experiments, including downstream evaluations across model sizes and extensive robustness checks.
  • Clear Presentation: Reviewer 1QQo noted the paper is clear and easy to follow.

3. Summary of Concerns and Responses

Reviewer 4CPb (Score: 4): The reviewer suggested discussing how the crossover varies across capabilities. We added the crossover of each topic (Section 3.2.3), all well below the pretraining compute of current flagship models.

Reviewer 1QQo (Score: 3→4): The reviewer asked about the motivation for removing the encoder, the effect of encoder choice, and the downstream gap. We added a discussion of encoder-free advantages (Appendix G) and a comparison of encoders with different pretraining objectives (Appendix H), and showed that the downstream gap narrows with scale. The reviewer confirmed all concerns were addressed and raised the score to 4.

Reviewer GFhd (Score: 2): We responded to the initial concerns on the fixed encoder, crossover extrapolation, downstream parity, and reproducibility with new experiments and appendices. The subsequent discussion focused on two points:

  • Request to train the encoder from scratch: Our encoder is trained jointly with the decoder, so a from-scratch encoder differs only in initialization. While initialization may shift the crossover, our contribution is not a precise crossover point but the scaling trends we identify: encoder-free models favour larger decoders, improve faster on the multimodal objective, and learn to take over visual encoding within the decoder.
  • Request to scale the encoder with the backbone: There is no clear evidence that enlarging the encoder yields sustained returns as the backbone grows. Open-source MLLMs keep a ~400M encoder across a ~300x range of backbone sizes (e.g., Qwen3.5 from 9B to 397B, Kimi K3 at 2.78T). By following flagship open-source models, our setting makes the conclusions more informative for practical MLLM development.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.