- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces FEINER, an efficient framework for integrating visual information into pretrained language decoders. It freezes a pretrained vision encoder and uses an intermediate-layer readout, an MLP projector with text-based bias initialization, and Gram regularization to preserve visual relationships while adapting features to the decoder. Learned register tokens with a gated image-only re-read further support visual information integration. Experiments with 450M and 3B models show improved visual understanding across multiple benchmarks at reduced training cost, supported by ablations and analyses of representation preservation and decoder attention.
Critical strengths
-
Clear presentation and organization. The paper structures its analysis around visual representation preservation, projection, and decoder integration, making the motivation and experimental progression easy to follow.
-
Informative analysis. The ablations combine task performance with representation and attention analyses, clarifying design choices and documenting both improvements and negative results.
-
Training efficiency. FEINER-450M improves average visual benchmark performance over nanoVLM while using approximately 60% fewer reported H100 GPU-hours.
Critical weaknesses
-
Backbone diversity. Both model variants use SmolLM-family decoders. Experiments across additional LLM families are needed to establish the generality of the proposed approach.
-
The benchmarks primarily assess single-image understanding, with no direct evaluation of visual grounding or video understanding, two important multimodal tasks that test fine-grained spatial alignment and temporal reasoning, respectively.
Questions for authors
- Please see weakness above.
- Table 2 shows that FEINER-450M improves average visual performance over nanoVLM, but ChartQA and language scores drop slightly. Could the authors give us some insight or explanation?
4 comments
Thank you for the clarification and the updated ChartQA results. However, the rebuttal is strong on defending design choices and fixing the ChartQA anomaly, but light on new experimental validation. could you provide at least one experiment using a small backbone from a different LLM family (e.g., Qwen or Llama)?
We are happy that we clarified some of your points.
Regarding additional backbone results: we initially did evaluate on other decoder families, in particular different LFM2.5 and Qwen3 variants. The table below shows results on the feiner-train dataset from our experiments on LFM2.5-350M and Qwen3-0.6B, starting from the baseline setup (base decoder with SigLIP2 encoder and linear projector), updating SigLIP to a frozen EUPE encoder, and the final setup with frozen EUPE and our proposed improvements.
| Group accuracy | LFM SigLIP (baseline) | LFM frozen EUPE | LFM final (ours) | Qwen SigLIP (baseline) | Qwen frozen EUPE | Qwen final (ours) | |
|---|---|---|---|---|---|---|---|
| Winoground | 2.75 | 3.75 | 7.25 | 7.50 | 4.50 | 8.50 | |
| BiVLC | 0.48 | 6.21 | 7.91 | 10.40 | 10.57 | 15.65 |
Ultimately, we decided that the baseline scores of these decoder families were not strong enough to justify promoting them to costly downstream benchmarks and further analyses. An additional problem is the lack of information about (and therefore control over) the data mix, training stages etc. of the additional decoders, that made it difficult to adequately address this baseline underperformance. Nonetheless, as seen above, our proposed interventions substantially improved the performance over the baseline across both additional decoder families, similar to what we see for the SmolLM family. We are thus confident that our proposed interventions generalize, and will include these additional results in the manuscript. Thank you again for this constructive point.
The author has addressed my concerns, so I raise my score.
Rebuttal by Authors
Thank you for your positive and constructive feedback.
We select the SmolLM-decoder family for two main reasons:
The scientific focus of our work is on how visual information is preserved, projected, and integrated into a language decoder. Our proposed improvements are not formulated around SmolLM-specific components. Within the compute budget, we decided to focus on a deeper evaluation of this representative architecture instead of spreading the evaluation across multiple LLM decoder families.
Thank you for your comment. Please note that visual grounding is in fact investigated through e.g., benchmarks on object localization (+8.57 on MMBench's Object-Localization over nanoVLM-460M), relation understanding (+8.82 on CRPE Relation), and spatial reasoning (+3.52 on SpatialEval). Single-image experiments provide a focused setting to establish the improvements in visual information integration that form the contribution of this work. Regarding multi-image and video tasks, these still rely on the model's visual integration interface. Prior works have shown that multi-image and video benefit directly from improve single image understanding, and thus our design motivation also applies to multi-image and video tasks [1,2].
The slight differences in ChartQA and language scores are likely related to difference in the exact data composition and training set size. For example, single-image filtering leads to a reduced number of OCR documents in our FineVision data mix. To address this inconsistency, we retrained our models on a larger training data corpus, which will be part of the final release. On this extended corpus, feinerVLM-450M reaches 59.96 on ChartQA, above nanoVLM-460M's 57.72.
[1] Li et al. LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models. https://arxiv.org/abs/2407.07895
[2] Li et al. LLaVA-OneVision: Easy Visual Task Transfer. https://arxiv.org/abs/2408.03326