- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses the question of how images should be encoded and processed in multimodal models, specifically in models that can receive image/text inputs and give textual outputs.
Different strategies are compared, constructing a clear "architectural flow-chart" of possibilities. The relative strengths of the strategies are measured by evaluating performance in textual and visual understanding benchmarks.
On the whole, the paper provides a clean picture of the different methods of including a vision input into a multimodal backbone, and provides interesting and timely guidance on the relative strenghts and weaknesses.
The experimental setups are quite remarkable, retraining from scratch models of modest sizes (up to 8B) in each configuration.
The paper's concludes that - at constant training compute - a fine-tuned encoder is not particularly advantageous. However, it shows that some vision-dedicated capacity, in the form of a parameter budget for vision-only operations, is beneficiary. This added capacity performs best when located before or at the beginning of the model, which is consistent with the literature empirically showing that images are "perceived" early in the model backbone.
Critical strengths
The question of exactly how and where introducing a visual encoder is timely and very interesting. While the question itself has been asked before, the structural clarity of the paper makes the approach novel.
The computing effort behind the work is impressive.
Critical weaknesses
Some of the findings rely on relatively small differences in performance, but the confidence intervals of the benchmarks are never discussed in the main text. If the authors were to address this in some form (see questions below) I believe the paper would be stronger as a result.
Questions for authors
I have some questions, mainly on the robustness of the results:
The trends are constructed on the difference of percentage in the benchmarks. Most of the trends present gaps of multiple percentage points, such as the decay of performance of VQA for encoder-free vs encoder models, but some conclusions are based on smaller differences.
It would be then interesting to have a better understanding of what could be the underlying noise in the benchmarks. The paper currently does not comment much on this.
First of all, all training runs are single seed. I understand that additional seeds of training would be prohibitedly expensive, but the possible magnitude of noise due to the random seed in training is never commented. Can the authors comment on this?
Additionally, some measure of error in the benchmarks should be reasonably cheap to obtain. Having an estimate of the confidence estimate of the different benchmarks would allow the readers to better separate the robust trends with those that may be inside the noise.
The models are compared at fixed compute, and the relative strengths are discussed consequently. This is clear and acceptable for a controlled study. However, the training at this compute time is not yet saturated. The benchmarks are still moving quite a bit, as shown for example in Figure 12. Some benchmarks have very low performances, e.g. the OCR benchmarks, well below the actual capability of similar models of that size. It is unclear to me that the reported trend will survive at larger compute time. In the appendix there is a longer training for the 4B model (but the encoder-free version is missing!), which shows that the relative performance is not entirely settled even at 30 ZFlops. I think that the authors could comment on this somewhere in the text.
Some additional minor points:
I find the color-coding of the results in Table 1 confusing. Should the bold not be across all modalities? Why are the encoder-free and encoder lines considered separately? In the current form it is quite difficult to understand what message the table is trying to convey.
Table 1's caption title "Enlarging the encoder-free backbone does not close the gap to the encoder recipe." is vague. The gap exists only when considering the Avg. VQA, but this is unclear from the statement. Moreover, for the 1.7B most of the difference come from the OCR benchmark, where all models perform very poorly. Can the authors comment on this?
In Figure 4, which models are the two left plots referring to? It is unclear if this is one size, 1.7B or 4B or 8B, or the average of the three. From the caption one would infer that it is 1.7B, but it is never explicitly stated.
Finally, is there a similar figure such as Figure 12 for the models at larger size?
2 comments
I thank the authors for answering my concerns and questions.
I find in particular that the discussion on errors and confidence levels makes the paper more robust.
I stand by my decision to support acceptance.
Thank you for the positive review and the constructive feedback! Table and figure numbers below refer to the revised PDF.
See general response B. The main text (Section 2) now discusses benchmark error and run-to-run variation, with paired intervals in Tables 1, 3, and 22. The gaps our recommendations rest on, the encoder's lead over encoder-free and early over late placement, are larger than both kinds of noise. Two comparisons differ by about one point (in-fusion against the encoder, and the from-scratch encoder against MetaCLIP), so the revision describes both comparisons as matches.
See general response B. Repeating instruction tuning changes little, and separately trained runs of a recipe differ by at most 1.3 Avg. VQA points.
Averaged over the VQA suite, the error on a difference between two models stays under one point (last row above). That is well below the encoder's lead over encoder-free and the gap between early and late placement of the vision-only FFN modules at 1.7B (38.8 against 31.9 Avg. VQA at 2.6 ZFLOPs, Table 30).
Apologies for the omission. We trained the missing 4B encoder-free run to 29.0 ZFLOPs and added it to Table 3. Section 4 now notes that training is not saturated. The encoder's lead over encoder-free holds through the longest budgets at 1.7B and 4B: it leads at all seven 1.7B and all five 4B budgets. The point gap narrows as both curves flatten, but encoder-free still needs about 2.5x the encoder's compute for the same Avg. VQA at both sizes, and the scaling laws agree (general response B).
Brackets are paired 95% bootstrap intervals over VQA questions.
We agree. At 224 × 224 px, small text covers only a few pixels, which caps OCR scores for every model; for text and charts, 448 × 448 px is enough (Beyer, 2024). To check that our conclusion does not depend on this, we instruction-tuned the same checkpoints at 448 × 448 px. This raises OCR and chart by 12 to 14 points, and the encoder stays ahead (Appendix I, Table 42). At 4B, longer training moves most of its lead to the other three VQA categories.
Tables 1–3 now bold the best value per block across all rows, and Table 1 adds a Δ Avg. VQA column against the encoder.
We have revised Table 1's caption to "Parameter-matched encoder-free variants raise Avg. Text but leave Avg. VQA below the encoder, mostly on OCR and chart questions." At the Table 1 budgets, the encoder's lead is largest on OCR and chart. At 1.7B it also leads on general questions and is level on knowledge and vision-centric ones, and at 4B it leads in all four categories. Our answer on OCR above shows that the lead holds when OCR scores rise.
The left plots show 1.7B models over training, and the right plots show 1.7B, 4B, and 8B (13.0, 11.6, and 10.6 ZFLOPs respectively). We have revised the caption to state this more clearly.
Yes, Figure 14 provides per-category scores at the five 4B budgets for both encoder runs and encoder-free.