acceptodds
Under review as a conference paper at ICLR 2027

Unbundling the Vision Encoder in Multimodal LLMs

Abstract

How multimodal LLMs should process images remains unsettled. Most rely on a vision encoder optimized separately for its own objective, while recent recipes train the encoder from scratch alongside the language model or remove it entirely. We view the vision encoder as a bundle of three choices that these recipes keep or discard: a visual prior from separate pretraining, dedicated vision-only capacity, and placement of that capacity before the multimodal backbone (pre-fusion) rather than inside it (in-fusion). Because published models also differ in data, backbone, and compute, we test each choice in a controlled study of 1.7B–8B multimodal LLMs trained from scratch on common data, evaluated on downstream visual question answering and language benchmarks at matched lifetime compute. An encoder trained from scratch with the backbone matches pretrained ones, so separate pretraining offers little advantage once its compute is counted. What matters instead is dedicated capacity: encoder-free models trail on visual understanding even with the encoder's parameters added to their backbone. Yet that capacity need not precede fusion, as vision-only modules in early backbone layers (in-fusion) match or slightly exceed the encoder given longer training, while lowering time to first token. For multimodal LLMs trained from scratch, our findings favor a single training stage with dedicated vision capacity early in the network.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal pretraining, representation learning, visual understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1qeWOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the question of how images should be encoded and processed in multimodal models, specifically in models that can receive image/text inputs and give textual outputs.

Different strategies are compared, constructing a clear "architectural flow-chart" of possibilities. The relative strengths of the strategies are measured by evaluating performance in textual and visual understanding benchmarks.

On the whole, the paper provides a clean picture of the different methods of including a vision input into a multimodal backbone, and provides interesting and timely guidance on the relative strenghts and weaknesses.

The experimental setups are quite remarkable, retraining from scratch models of modest sizes (up to 8B) in each configuration.

The paper's concludes that - at constant training compute - a fine-tuned encoder is not particularly advantageous. However, it shows that some vision-dedicated capacity, in the form of a parameter budget for vision-only operations, is beneficiary. This added capacity performs best when located before or at the beginning of the model, which is consistent with the literature empirically showing that images are "perceived" early in the model backbone.

Critical strengths

The question of exactly how and where introducing a visual encoder is timely and very interesting. While the question itself has been asked before, the structural clarity of the paper makes the approach novel.

The computing effort behind the work is impressive.

Critical weaknesses

Some of the findings rely on relatively small differences in performance, but the confidence intervals of the benchmarks are never discussed in the main text. If the authors were to address this in some form (see questions below) I believe the paper would be stronger as a result.

Questions for authors

I have some questions, mainly on the robustness of the results:

The trends are constructed on the difference of percentage in the benchmarks. Most of the trends present gaps of multiple percentage points, such as the decay of performance of VQA for encoder-free vs encoder models, but some conclusions are based on smaller differences.

It would be then interesting to have a better understanding of what could be the underlying noise in the benchmarks. The paper currently does not comment much on this.

First of all, all training runs are single seed. I understand that additional seeds of training would be prohibitedly expensive, but the possible magnitude of noise due to the random seed in training is never commented. Can the authors comment on this?

Additionally, some measure of error in the benchmarks should be reasonably cheap to obtain. Having an estimate of the confidence estimate of the different benchmarks would allow the readers to better separate the robust trends with those that may be inside the noise.

The models are compared at fixed compute, and the relative strengths are discussed consequently. This is clear and acceptable for a controlled study. However, the training at this compute time is not yet saturated. The benchmarks are still moving quite a bit, as shown for example in Figure 12. Some benchmarks have very low performances, e.g. the OCR benchmarks, well below the actual capability of similar models of that size. It is unclear to me that the reported trend will survive at larger compute time. In the appendix there is a longer training for the 4B model (but the encoder-free version is missing!), which shows that the relative performance is not entirely settled even at 30 ZFlops. I think that the authors could comment on this somewhere in the text.

Some additional minor points:

I find the color-coding of the results in Table 1 confusing. Should the bold not be across all modalities? Why are the encoder-free and encoder lines considered separately? In the current form it is quite difficult to understand what message the table is trying to convey.

Table 1's caption title "Enlarging the encoder-free backbone does not close the gap to the encoder recipe." is vague. The gap exists only when considering the Avg. VQA, but this is unclear from the statement. Moreover, for the 1.7B most of the difference come from the OCR benchmark, where all models perform very poorly. Can the authors comment on this?

In Figure 4, which models are the two left plots referring to? It is unclear if this is one size, 1.7B or 4B or 8B, or the average of the three. From the caption one would infer that it is 1.7B, but it is never explicitly stated.

Finally, is there a similar figure such as Figure 12 for the models at larger size?

2 comments

Thank you for the positive review and the constructive feedback! Table and figure numbers below refer to the revised PDF.

W1. the confidence intervals of the benchmarks are never discussed in the main text

See general response B. The main text (Section 2) now discusses benchmark error and run-to-run variation, with paired intervals in Tables 1, 3, and 22. The gaps our recommendations rest on, the encoder's lead over encoder-free and early over late placement, are larger than both kinds of noise. Two comparisons differ by about one point (in-fusion against the encoder, and the from-scratch encoder against MetaCLIP), so the revision describes both comparisons as matches.

Q1. the possible magnitude of noise due to the random seed in training is never commented

See general response B. Repeating instruction tuning changes little, and separately trained runs of a recipe differ by at most 1.3 Avg. VQA points.

Source of variationSpread (VQA points)Where
Five instruction tunings of one encoder checkpoint0.19Table 14
Separately trained repeats of a recipe at our longest budget (Table 3)at most 1.3Table 13
One benchmark, binomial 95% CI half-width±1.0 (DocVQA) to ±5.5 (MMVP)Section 2
Paired difference in Avg. VQA between two models±0.7 to 0.8Tables 1, 3 and 22

Q2. some measure of error in the benchmarks should be reasonably cheap to obtain

Averaged over the VQA suite, the error on a difference between two models stays under one point (last row above). That is well below the encoder's lead over encoder-free and the gap between early and late placement of the vision-only FFN modules at 1.7B (38.8 against 31.9 Avg. VQA at 2.6 ZFLOPs, Table 30).

Q3. the encoder-free version is missing!

Apologies for the omission. We trained the missing 4B encoder-free run to 29.0 ZFLOPs and added it to Table 3. Section 4 now notes that training is not saturated. The encoder's lead over encoder-free holds through the longest budgets at 1.7B and 4B: it leads at all seven 1.7B and all five 4B budgets. The point gap narrows as both curves flatten, but encoder-free still needs about 2.5x the encoder's compute for the same Avg. VQA at both sizes, and the scaling laws agree (general response B).

Avg. VQAEncoder-freeEncoderEncoder leadEncoder matches it at
1.7B, 13.0 ZFLOPs40.242.7+2.5 [1.8, 3.2]5.2 ZFLOPs (Table 1)
4B, 29.0 ZFLOPs45.448.8+3.4 [2.7, 4.2]11.6 ZFLOPs (45.5)

Brackets are paired 95% bootstrap intervals over VQA questions.

Q3. Some benchmarks have very low performances, e.g. the OCR benchmarks, well below the actual capability of similar models of that size.

We agree. At 224 × 224 px, small text covers only a few pixels, which caps OCR scores for every model; for text and charts, 448 × 448 px is enough (Beyer, 2024). To check that our conclusion does not depend on this, we instruction-tuned the same checkpoints at 448 × 448 px. This raises OCR and chart by 12 to 14 points, and the encoder stays ahead (Appendix I, Table 42). At 4B, longer training moves most of its lead to the other three VQA categories.

SettingEncoder lead over encoder-free
1.7B / 4B, 448 × 448 px instruction tuning+4.1 [3.3, 4.8] / +2.6 [1.9, 3.3] Avg. VQA
4B, 2.9 ZFLOPs (224 px)+10.0 on OCR & chart
4B, 29.0 ZFLOPs (224 px)+1.8 on OCR & chart, ahead in all four categories

M1. Should the bold not be across all modalities?

Tables 1–3 now bold the best value per block across all rows, and Table 1 adds a Δ Avg. VQA column against the encoder.

M2. Table 1's caption title [...] is vague. The gap exists only when considering the Avg. VQA

We have revised Table 1's caption to "Parameter-matched encoder-free variants raise Avg. Text but leave Avg. VQA below the encoder, mostly on OCR and chart questions." At the Table 1 budgets, the encoder's lead is largest on OCR and chart. At 1.7B it also leads on general questions and is level on knowledge and vision-centric ones, and at 4B it leads in all four categories. Our answer on OCR above shows that the lead holds when OCR scores rise.

Encoder minus encoder-free, Table 1 budgetGeneralKnowledgeOCR & chartVision-centric
1.7B (5.2 ZFLOPs)+3.8+0.2+9.4−0.3
4B (11.6 ZFLOPs)+2.1+1.0+7.5+2.3

M3. In Figure 4, which models are the two left plots referring to?

The left plots show 1.7B models over training, and the right plots show 1.7B, 4B, and 8B (13.0, 11.6, and 10.6 ZFLOPs respectively). We have revised the caption to state this more clearly.

M4. is there a similar figure such as Figure 12 for the models at larger size?

Yes, Figure 14 provides per-category scores at the five 4B budgets for both encoder runs and encoder-free.

I thank the authors for answering my concerns and questions.

I find in particular that the discussion on errors and confidence levels makes the paper more robust.

I stand by my decision to support acceptance.

Reviewer_81XXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper decomposes the vision encoder in multimodal LLMs into three choices, including separate visual pretraining, vision-only capacity, and placement before rather than inside the multimodal backbone. Controlled experiments with randomly initialized 1.7B–8B models suggest that same-data encoder pretraining offers little benefit over joint training, while early additive branches inside the backbone can match or slightly exceed a separate encoder.

Critical strengths
  • The three-way decomposition turns several choices that are usually changed together into concrete experimental questions. Training all stages with shared data, backbones, and optimization makes the comparisons useful for understanding how visual compute should be allocated.

  • The same-data pretraining result is consistent across objectives, budgets, and model scales. Table 17 is particularly informative because longer MetaCLIP pretraining raises ImageNet accuracy from 80.4% to 84.4% while Avg. VQA falls from 38.7 to 36.9, showing that better recognition features do not necessarily produce better multimodal understanding.

  • The early in-fusion design is a useful architecture result. In Table 3, FFN+QKVO exceeds the separate encoder by 1.2 Avg. VQA points at both 1.7B and 4B, while the FFN-only version nearly matches it.

Critical weaknesses
  • The experiments do not fully isolate vision-only routing as the reason that added capacity helps. In Table 28, an all-token separate branch recovers 4.3 of the 5.0 Avg. VQA points gained by the vision-only branch, leaving only 0.7 points at matched compute and 1.0 at matched exposure attributable to the routing difference. The current evidence supports the value of an early separate branch more clearly than the stronger conclusion that parameters help only when dedicated to vision.

  • The encoder's benefit is strongly task- and scale-dependent. In Table 1 at 1.7B, its gains over encoder-free processing are 9.4 points on OCR and chart and 3.8 on general VQA, but only 0.2 on knowledge VQA and -0.3 on vision-centric VQA. Appendix D.1 shows that the OCR gap then narrows at 8B while vision-centric tasks become the largest remaining gap, so Avg. VQA alone does not establish a general need for dedicated capacity across visual capabilities.

  • Lifetime compute measures total theoretical FLOPs, but treating encoder-pretraining compute and multimodal-training compute as interchangeable changes the question from the value of a visual prior to the best allocation of a fixed budget. Figures 2 and 8 provide useful fixed-compute and fixed-update controls, although both stages still draw on the same MetaCLIP images. The results therefore support the narrower conclusion that separate same-data pretraining is not more compute-efficient in this setup, but provide less evidence about a visual prior learned from additional data or reused across multiple models.

Questions for authors
  • It would be helpful to compare the all-token and vision-only versions of the same separate branch at the full training budget, using the same initialization and data order. This would show whether the small routing advantage in Table 28 persists after longer training.

  • Why does the encoder advantage concentrate on OCR and chart tasks at 1.7B and 4B, then shift toward vision-centric tasks at 8B? Per-category results for the all-token and vision-only branches would help determine whether modality-specific routing explains this pattern.

  • Is the pretraining conclusion intended to apply only to same-data pretraining under a single-model lifetime-compute budget? It would be helpful to clarify how Figures 2 and 8 should be interpreted when a pretrained encoder contributes data unavailable during multimodal training or is reused across several models.

2 comments

Thank you for your careful review and helpful questions. Table, figure and appendix numbers below refer to the revised PDF.

W1. The experiments do not fully isolate vision-only routing as the reason that added capacity helps.

Q1. "at the full training budget, using the same initialization and data order"

See general response A. The vision-only branch beats the all-token branch at both 1.7B and 4B, leading at every budget (Table 31). The table below gives the longest budget of each pair.

Vision-only minus all-token branch [95% CI]

Model size / Training BudgetAvg. VQAOCR & chart
1.7B, 13.0 ZFLOPs+2.2 [1.4, 3.0]+5.0 [4.3, 5.8]
4B, 5.8 ZFLOPs+2.5 [1.8, 3.2]+5.9 [5.2, 6.7]

Brackets are paired 95% bootstrap intervals over VQA questions. Both models use the same separate branch with the same parameters, matched in total compute, and differ in which tokens use it, so these gaps measure routing at matched compute. The 1.7B pair shares seed and initialization and reaches the full 13.0 ZFLOPs; the 4B pair stops at 5.8 ZFLOPs but also shares the data stream, as you ask. The two pairs agree at every budget. At the short budget of Table 30 the branch itself gives most of the gain over encoder-free, but that part fades: at the longest budget of each pair the all-token branch ends within 0.2 points of encoder-free, while routing keeps its lead at every budget (Table 31). As you suggest, the conclusion now says the encoder's parameters help visual understanding "most when dedicated to vision" rather than "only".

W2. "Avg. VQA alone does not establish a general need for dedicated capacity"

We agree that the lead is not uniform, and the abstract, Takeaway 2 and Section 4 now say where it is largest at each size. What holds throughout is its sign: the encoder leads encoder-free on Avg. VQA at every size and budget and is never reliably behind it in any category (Table 1, Figure 13, Appendix D.1). With the 4B encoder-free run we added at 29.0 ZFLOPs, it leads in all four categories (Figure 14), and with 448 px instruction tuning, which raises OCR and chart scores by 12 to 14 points, it still leads by 4.1 and 2.6 Avg. VQA at 1.7B and 4B (Table 42).

Q2. "Why does the encoder advantage concentrate on OCR and chart tasks at 1.7B and 4B, then shift toward vision-centric tasks at 8B?"

The shift also appears at 4B with longer training, while at 1.7B the lead stays on OCR and chart through 13.0 ZFLOPs. Our intuition is that reading small text needs fine visual detail, which dedicated capacity supplies early in training. Smaller or shorter-trained encoder-free models lack it, so the encoder's lead is largest on OCR and chart. With more parameters and training, the encoder-free model learns to read text too, and the OCR and chart gap narrows.

Per-category routing results (Table 31) follow the same pattern as the encoder's lead. At 1.7B, routing gains most on OCR and chart through 13.0 ZFLOPs. At 4B, longer training moves routing's gain from OCR and chart toward vision-centric questions, as it does for the encoder's lead. We have not isolated why vision-centric questions become the largest gap with scale.

Vision-only minus all-token branchOCR & chartVision-centricGeneralKnowledge
1.7B, 13.0 ZFLOPs+5.0+3.3+2.8−2.4
4B, 2.9 ZFLOPs+7.9+1.3+1.6−0.4
4B, 5.8 ZFLOPs+5.9+3.9+1.3−1.0

Knowledge questions depend more on the language model than on the image (Cambrian-1), and the all-token branch also adds capacity for text tokens, which likely explains its lead there.

W3, Q3. "only to same-data pretraining under a single-model lifetime-compute budget?"

Same data: yes, by design. Single-model budget: no, and our free-pretraining experiment answers that part of your question. An encoder reused across many models costs each of them almost nothing, which is the free-pretraining setting (Table 19 and Figure 8, bottom). There, pretraining still gives no consistent gain: longer free pretraining raises ImageNet accuracy but lowers Avg. VQA from 38.7 to 36.9, while a from-scratch encoder given the same 2.7 ZFLOPs of multimodal training scores 39.1 (Table 19). When pretraining compute is charged, it gains at most 0.7 Avg. VQA (Table 18).

Holding the images fixed is what makes the study controlled, since an encoder pretrained on more data would mix the value of pretraining with that of extra data. Every pretrained encoder in the paper is ours, pretrained on the MetaCLIP images used in multimodal training (Section 2), so Figures 2 and 8 are same-data results, as the abstract and Takeaway 1 state. Appendix H leaves open whether released encoders help through what they add beyond our setting, such as new data.

We hope these results and clarifications resolve your concerns.

Reviewer_81XXOpenReview ↗

Response to the Authors

Thank you for the detailed response and additional experiments. These address my previous concerns, and I will raise my recommendation to weak acceptance. I find the paper’s controlled comparisons and insights into the placement of dedicated visual capacity valuable.

One interesting direction for future work is to clarify the role of the additive parameterization. For Q/K/V/O, the authors use WsW_s for text and Ws+ΔWvW_s+\Delta W_v for vision. A useful comparison would independently parameterize WtW_t and WvW_v, keeping joint attention and the FFNs unchanged. Initializing Wt=Wv=Ws(0)W_t=W_v=W_s^{(0)} would match the additive model’s initial effective projections when ΔWv=0\Delta W_v=0.

Because these full-rank formulations have the same projection expressivity, this ablation would primarily test whether the shared-base parameterization and its coupled updates help optimization in from-scratch multimodal training. Accounting for learning-rate and regularization effects would make the comparison informative.

This is a suggestion for future exploration, not a remaining concern affecting my recommendation.

Reviewer_cxNZOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies three choices commonly bundled into a vision encoder: separate pretraining, dedicated visual parameters, and placement before the multimodal backbone. The authors train 1.7B–8B models from scratch using common data sources and compare recipes at matched training compute, including encoder pretraining. Evaluation covers 16 visual question-answering benchmarks and 12 language benchmarks.

The main findings are that jointly trained encoders are competitive with separately pretrained encoders, enlarging an encoder-free backbone does not fully recover the encoder’s visual performance, and additional vision-only modules in early backbone layers can match an encoder after sufficient training. The paper also reports lower time to first token for these in-fusion designs.

Critical strengths
  1. The controlled comparisons provide useful evidence about multimodal training choices. Training the components directly, comparing four encoder-pretraining objectives, and accounting for upstream compute make the pretraining comparison informative. The shared-capacity controls are also stronger than simply removing the encoder: Table 1 matches parameter budgets, while Table 24 additionally matches token exposure and allows the expanded backbones more compute. Together, these experiments support the practical finding that simply enlarging the shared backbone is an inefficient substitute for the encoder under the tested recipe.

  2. The evidence that visual computation can move inside the backbone is convincing. Table 3 shows competitive performance at both 1.7B and 4B after longer training. The placement ablation is particularly useful: moving the same vision-only branches from early to late layers substantially reduces VQA performance. This gives a concrete architectural insight, and the measured latency reductions add practical value. I find this contribution worthwhile even without interpreting the small final accuracy gains as a decisive advantage over an encoder.

Critical weaknesses
  1. I do think that the paper does not fully establish that modality-exclusive capacity is the key ingredient. In Table 28, ordinary shared FFN widening reaches 33.3 Avg. VQA, a separate branch used by all tokens reaches 36.5, and restricting that branch to vision tokens reaches 37.2. Most of the improvement therefore appears before.

  2. In Table 20, the scratch encoder trails MetaCLIP by 0.7Avg. VQA points at 1.7B and leads by 0.6 and 1.3 points at 4B and 8B. The final in-fusion gains over the encoder are also roughly one point. The paper reports useful question-level bootstrap intervals for selected ablations, but these do not measure variation across training or instruction-tuning runs. Section E.4 additionally shows that the relative performance is sensitive to the final learning-rate decay. I find the evidence for comparable performance more persuasive than the evidence for a reliable advantage. This distinction matters when turning the experiments into general training recommendations.

Questions for authors
  1. Does the all-token separate FFN branch remain behind the vision-only branch when trained through the longer horizon and final decay used in Table 3, with matched initialization and data order? Without this comparison, what evidence supports attributing the benefit primarily to vision-only routing rather than the branch parameterization?

  2. Do the reported lifetime budgets include the three epochs of Cambrian-7M instruction tuning applied before VQA evaluation? Section A.6 appears to count only upstream and multimodal training. If instruction tuning is excluded, how large is that cost for each architecture, and could including it affect the comparisons?

  3. How much do the main endpoint differences vary across repeated instruction tuning or independent training runs? Paired endpoint intervals and any available repeated runs would help establish whether the roughly one-point gains are reliable.

I'll raise the scores based on authors' feedbacks if the feedbacks solve my concerns.

1 comment

Thank you for taking the time to review our work and provide valuable feedback. We are glad you found our evidence for moving visual capacity inside the backbone to be "convincing" and "a concrete architectural insight." We would like to address your two remaining concerns below. Table, section and appendix numbers refer to the revised PDF.

Q1. Does the all-token separate FFN branch remain behind the vision-only branch when trained through the longer horizon and final decay used in Table 3

See general response A. The vision-only branch stays ahead of the all-token branch through the final decay at the longest budget of each pair, with its largest gains on OCR and chart questions (Table 31).

Vision-only minus all-token branch [95% CI]

Model size / Training budgetAvg. VQAOCR & chart
1.7B, 13.0 ZFLOPs+2.2 [1.4, 3.0]+5.0 [4.3, 5.8]
4B, 5.8 ZFLOPs+2.5 [1.8, 3.2]+5.9 [5.2, 6.7]

Brackets are paired 95% bootstrap intervals over VQA questions. Both pairs are matched in total compute and share seed and initialization; the 1.7B pair reaches the Table 3 horizon, and the 4B pair also shares the data stream.

W1, Q1. what evidence supports attributing the benefit primarily to vision-only routing rather than the branch parameterization?

See general response A. The two models use the same separate branch with the same parameters at matched total compute and differ in which tokens use it. As you note, at the short budget of Table 30 the branch itself gives most of the gain over encoder-free. That part fades with training: at the longest budget of each pair the all-token branch ends level with encoder-free (40.4 against 40.2 Avg. VQA at 1.7B and 40.6 against 40.4 at 4B), while the vision-only branch still leads it by 2.2 and 2.5 points (Table 31). The all-token branch also trains on fewer tokens at matched compute, and matching tokens instead does not shrink the routing gap (Table 30).

W2, Q3. How much do the main endpoint differences vary across repeated instruction tuning or independent training runs?

See general response B. The differences our recommendations rely on, such as the encoder's lead over encoder-free and vision-only routing's lead on OCR and chart, are larger than both run-to-run variation and benchmark noise.

Source of variationAvg. VQAWhere
Five instruction tunings of one checkpoint0.19Table 14
Separately trained repeats of a recipe (at our longest budget, Table 3)at most 1.3Table 13
Paired interval over questions (difference in Avg. VQA)±0.7 to 0.8Tables 1, 3 and 22

For the roughly one-point differences you mention, in-fusion (FFN+QKVO) finishes ahead of the encoder at both sizes and the from-scratch encoder leads MetaCLIP at 4B and 8B, but the leader changes across the scaling-law sizes for in-fusion (Table 37) and between 1.7B and 4B for pretraining. We agree that this supports comparable performance rather than a reliable advantage, so the revision describes both comparisons as "matches". Our conclusion needs no more than a match: it shows that the encoder's pre-fusion placement and its separate pretraining on the same data can each be removed without a consistent loss. The decay sensitivity you point to in Section E.4 is part of why we claim only a match, so the revision compares recipes only after the final decay, where two separately trained encoders score 48.8 and 48.7.

Q2. Do the reported lifetime budgets include the three epochs of Cambrian-7M instruction tuning applied before VQA evaluation?

Lifetime compute counts encoder pretraining and multimodal training, excluding instruction tuning because it is a fixed downstream step applied to every checkpoint to measure VQA (Section B.1). Instruction tuning costs 0.8 to 0.9% of the Table 3 budgets, and counting it changes compute ratios between compared recipes by at most 0.9%. The revision states this in Section 2, and Section A.6 lists its cost per recipe.

We hope these results resolve both concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to the area chair and all reviewers

We thank the area chair and the reviewers for their time and constructive feedback.

Multimodal LLMs differ in how they use a vision encoder: most attach a separately pretrained one, Kimi K3 trains it from scratch, and Gemma 4 and Inkling drop it. Published models differ in many choices at once, so they cannot show which part of the encoder matters, a question 1qeW calls "timely and very interesting". We train every component from scratch on the same images at matched lifetime compute, which lets us remove the encoder's separate pretraining, its pre-fusion placement and its vision-only capacity one at a time. The result is a map of which parts of the encoder a model needs: matching the encoder without a part shows that the part can be removed. For a multimodal LLM trained from scratch, the results favor a single training stage with vision-only capacity in the backbone's first layers.

ClaimEvidenceWhere
Removing vision-only capacity is costlyThe encoder leads encoder-free at every size and budget, with longer training, with 448 px instruction tuning, and in new scaling laws from 588M to 6.4B. Encoder-free needs about 2.5x the encoder's compute.Tables 1, 3, 37 and 38, Appendix I
Separate encoder pretraining gives no consistent gainA from-scratch encoder matches encoders pretrained on the same images, across objectives and up to 8B. Even free pretraining raises ImageNet accuracy but not Avg. VQA.Tables 17, 19 and 22
Vision capacity can move into the backbone and works best in its early layersVision-only modules in the backbone's first layers match the encoder at 1.7B and 4B, with a lower time to first token.Tables 2, 3 and 30, Figure 6
Vision-only routing beats an all-token branchWith the same branch parameters and compute, routing only vision tokens through the branch wins at every budget at 1.7B and 4B, most on OCR and chart questions.Table 31

Updates to the paper

As cxNZ suggested, we now call the two one-point comparisons (in-fusion against the encoder, and the from-scratch encoder against MetaCLIP) matches.

During the discussion we ran the longer and larger experiments the reviewers asked for, and repeated the key comparisons with 448 × 448 px instruction tuning, where the encoder's lead and the routing gain both hold (Appendix I). General responses A to C answer the common questions in detail. Table and figure numbers in all our comments refer to the revised PDF.

QuestionAsked byHow we address it
Does vision-only routing still beat an all-token branch with longer training?cxNZ, 81XXYes, at 1.7B and 4B. We extended the 1.7B pair to our longest budget and trained a new 4B pair (general response A)
Are gaps of about one point reliable?cxNZ, 1qeWWe added tables of repeated training runs and repeated instruction tuning, and paired intervals for the main comparisons (general response B)
Does the encoder's lead hold at larger scale and with longer training?81XX, 1qeWYes. We trained a new scaling ladder from 588M to 6.4B and fit scaling laws, and trained the missing 4B encoder-free run to 29.0 ZFLOPs (general responses B and C)
Is the pretraining result limited to same-data pretraining under a single-model budget?81XXSame data: yes, by design. Single-model budget: no. Within our same-data setting, our free-pretraining experiment, the setting of an encoder reused across models, still shows no gain (Table 19, reply to 81XX)
AuthorsOpenReview ↗

General response A: Vision-only routing (cxNZ, 81XX)

cxNZ and 81XX ask whether the vision-only branch still beats an all-token branch with longer training. The vision-only branch leads at every budget at both 1.7B and 4B (Table 31). The table below gives the longest budget of each pair. Both models use the same separate branch with the same parameters, matched in total compute, and differ in which tokens use it, so the gap measures vision-only routing at matched compute. The 1.7B pair shares seed and initialization and runs through the final decay at the Table 3 horizon (13.0 ZFLOPs), as cxNZ asked; the 4B pair stops at 5.8 ZFLOPs but also shares the data stream, as 81XX asked. The two pairs agree at every budget.

Vision-only minus all-token branch [95% CI]

Model size / Training budgetAvg. VQAOCR & chart
1.7B, 13.0 ZFLOPs+2.2 [1.4, 3.0]+5.0 [4.3, 5.8]
4B, 5.8 ZFLOPs+2.5 [1.8, 3.2]+5.9 [5.2, 6.7]
1.7B, 13.0 ZFLOPs, 448 px instruction tuning (Appendix I)+2.6 [1.8, 3.3]+3.7 [3.0, 4.5]
4B, 5.8 ZFLOPs, 448 px instruction tuning (Appendix I)+3.3 [2.5, 4.0]+7.3 [6.5, 8.1]

Brackets are paired 95% bootstrap intervals over VQA questions.

With 224 px tuning, the vision-only branch's lead is largest on OCR and chart questions at every budget. At the short budget of Table 30, the branch itself gives most of the gain over encoder-free, even when all tokens use it. That part fades with training: by the longest budget of each pair the all-token branch ends within 0.2 points of encoder-free, while vision-only routing keeps its lead at every budget (Table 31). The all-token branch also processes text, so at matched compute it trains on fewer tokens; matching tokens instead does not shrink the routing gap (Table 30).

AuthorsOpenReview ↗

General response B: Reliability and scaling laws (cxNZ, 1qeW, 81XX)

cxNZ and 1qeW ask how large benchmark and training noise are, and 81XX and 1qeW ask how the encoder's lead changes with scale and training. Benchmark and training variation (Tables 13 and 14) are small relative to the encoder's lead over encoder-free and the early-versus-late placement gap, and about the size of the one-point gaps, which is why the revision calls those matches. To test our conclusions across scale, we added a size ladder that trains three recipes at five backbone sizes from 588M to 6.4B and fit scaling laws, which show the encoder ahead of encoder-free at every size (Table 37).

CheckSetupAvg. VQA spread / errorWhere
Benchmark noisePaired 95% bootstrap intervals over VQA questions±0.7 to 0.8 for a difference between modelsTables 1, 3 and 22
Instruction-tuning noiseFive instruction tunings of one checkpoint0.19Table 14
Training-run noiseSeparately trained repeats of a recipe at our longest budget (Table 3)At most 1.3 differenceTable 13
Scaling-law fitEncoder, encoder-free and in-fusion (FFN+QKVO) from 588M to 6.4B (13 checkpoints per recipe)Predicts Avg. VQA at each left-out size within 0.8 on averageTables 36 to 39, Figure 16

The ladder grows the vision module with the backbone, and at 6.4B the fixed 0.4B encoder of the main runs also leads encoder-free by 3.6 to 5.2 Avg. VQA at every budget.

The scaling laws summarize the size trend. For the same Avg. VQA, encoder-free needs 2.4 to 3.4 times the encoder's compute at every size from 1.7B to 6.4B, while in-fusion needs about as much as the encoder (Table 38). In-fusion against the encoder and the from-scratch encoder against MetaCLIP differ by about one point and the leader changes across sizes, so the revision describes both comparisons as matches.

Compute for the same Avg. VQA, relative to the scratch encoder1.7B3.2B6.4B
Encoder-free2.37×2.84×3.44×
In-fusion (FFN+QKVO)1.10×1.05×1.00×
AuthorsOpenReview ↗

General response C: Revisions in the uploaded PDF

The revised PDF, uploaded with this response, makes the following changes:

TopicAsked byChanges in the revised PDF
Encoder vs. encoder-free81XX, 1qeWThe abstract, Takeaway 2 and Section 4 state where the encoder helps most by size and category. Table 3 adds the 4B encoder-free run at the longest budget, and Section 4 summarizes the 448 px results.
ReliabilitycxNZ, 1qeWSection 2 notes run-to-run variation and benchmark error, citing Tables 13 (repeated training) and 14 (repeated instruction tuning), with paired intervals in Tables 1, 3 and 22.
RoutingcxNZ, 81XXSection 5.2 adds the two routing pairs, the all-token branch at our longest budget, and placement after the final decay (Table 30). Table 31 reports every budget with per-category scores, and the conclusion says the encoder's parameters help "most when dedicated to vision".
In-fusion vs. encodercxNZ, 1qeWThe abstract, contributions, takeaways, Section 5.3 and the Table 3 and 34 captions state that in-fusion matches the encoder, with 4B recipes compared at the end of training.
Pretraining81XX, cxNZThe abstract, Takeaway 1 and the conclusion state that the pretrained encoders use the same data. Section 3 and the Figure 3 and Table 22 captions state that the from-scratch encoder matches them at every size, and Section 3 adds the never-pretrained encoder (Table 19).
Scaling laws1qeW, 81XXAppendix F adds the size ladder, fitted laws (Figure 16) and leave-one-size-out error (Tables 36 to 39), summarized in Section 5.3.
448 px tuning1qeWAppendix I (Table 42) adds 448 px instruction-tuning results.
ComputecxNZSection 2 states that instruction tuning changes compute ratios by at most 0.9%, and Section A.6 gives its cost.
Presentation1qeWOne bold rule for Tables 1–3, a Δ column in Table 1, clearer Table 1 and Figure 4 captions, and a 4B version of Figure 12 (Figure 14).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.