acceptodds
Under review as a conference paper at ICLR 2027

feiner: Preserve, Project, and Let the Decoder Integrate Visual Information

Abstract

Extending language models to reason over additional modalities is an important step towards general-purpose AI models. Since not all modalities can be efficiently trained natively, integrating multimodal information into pre-trained language decoders is an important task. In this work, we focus on vision, one of the most important non-text modalities, and use LLaVA-style vision-language models to study how information from vision encoders can be preserved and utilized in pre-trained decoders. To this end, we introduce a framework for efficient, controlled training and evaluation. Our resulting design preserves strong modality-specific representations and effectively projects them into the decoder's latent space to increase the availability of vision-specific information. We find that representation preservation and decoder-side information flow are complementary requirements for strong visual understanding. Our *feiner* approach rivals state-of-the-art models at comparable scale while requiring less than half the compute. These results provide an efficient basis for studying how pre-trained decoders can incorporate additional modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UhX2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces FEINER, an efficient framework for integrating visual information into pretrained language decoders. It freezes a pretrained vision encoder and uses an intermediate-layer readout, an MLP projector with text-based bias initialization, and Gram regularization to preserve visual relationships while adapting features to the decoder. Learned register tokens with a gated image-only re-read further support visual information integration. Experiments with 450M and 3B models show improved visual understanding across multiple benchmarks at reduced training cost, supported by ablations and analyses of representation preservation and decoder attention.

Critical strengths
  1. Clear presentation and organization. The paper structures its analysis around visual representation preservation, projection, and decoder integration, making the motivation and experimental progression easy to follow.

  2. Informative analysis. The ablations combine task performance with representation and attention analyses, clarifying design choices and documenting both improvements and negative results.

  3. Training efficiency. FEINER-450M improves average visual benchmark performance over nanoVLM while using approximately 60% fewer reported H100 GPU-hours.

Critical weaknesses
  1. Backbone diversity. Both model variants use SmolLM-family decoders. Experiments across additional LLM families are needed to establish the generality of the proposed approach.

  2. The benchmarks primarily assess single-image understanding, with no direct evaluation of visual grounding or video understanding, two important multimodal tasks that test fine-grained spatial alignment and temporal reasoning, respectively.

Questions for authors
  1. Please see weakness above.
  2. Table 2 shows that FEINER-450M improves average visual performance over nanoVLM, but ChartQA and language scores drop slightly. Could the authors give us some insight or explanation?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your positive and constructive feedback.

Backbone diversity. Both model variants use SmolLM-family decoders. Experiments across additional LLM families are needed to establish the generality of the proposed approach.

We select the SmolLM-decoder family for two main reasons:

  1. They arguably represent the template for modern VLM architectures based on the general Llava-style principles. This includes the modular encoder-projector-decoder architecture with a standard dense decoder.
  2. SmolLM2 is fully open, including weights, public data sources and training recipes. This is rare and allows us to support fully reproducible research.

The scientific focus of our work is on how visual information is preserved, projected, and integrated into a language decoder. Our proposed improvements are not formulated around SmolLM-specific components. Within the compute budget, we decided to focus on a deeper evaluation of this representative architecture instead of spreading the evaluation across multiple LLM decoder families.

The benchmarks primarily assess single-image understanding, with no direct evaluation of visual grounding or video understanding, two important multimodal tasks that test fine-grained spatial alignment and temporal reasoning, respectively.

Thank you for your comment. Please note that visual grounding is in fact investigated through e.g., benchmarks on object localization (+8.57 on MMBench's Object-Localization over nanoVLM-460M), relation understanding (+8.82 on CRPE Relation), and spatial reasoning (+3.52 on SpatialEval). Single-image experiments provide a focused setting to establish the improvements in visual information integration that form the contribution of this work. Regarding multi-image and video tasks, these still rely on the model's visual integration interface. Prior works have shown that multi-image and video benefit directly from improve single image understanding, and thus our design motivation also applies to multi-image and video tasks [1,2].

Table 2 shows that FEINER-450M improves average visual performance over nanoVLM, but ChartQA and language scores drop slightly. Could the authors give us some insight or explanation?

The slight differences in ChartQA and language scores are likely related to difference in the exact data composition and training set size. For example, single-image filtering leads to a reduced number of OCR documents in our FineVision data mix. To address this inconsistency, we retrained our models on a larger training data corpus, which will be part of the final release. On this extended corpus, feinerVLM-450M reaches 59.96 on ChartQA, above nanoVLM-460M's 57.72.

[1] Li et al. LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models. https://arxiv.org/abs/2407.07895

[2] Li et al. LLaVA-OneVision: Easy Visual Task Transfer. https://arxiv.org/abs/2408.03326

Thank you for the clarification and the updated ChartQA results. However, the rebuttal is strong on defending design choices and fixing the ChartQA anomaly, but light on new experimental validation. could you provide at least one experiment using a small backbone from a different LLM family (e.g., Qwen or Llama)?

We are happy that we clarified some of your points.

Regarding additional backbone results: we initially did evaluate on other decoder families, in particular different LFM2.5 and Qwen3 variants. The table below shows results on the feiner-train dataset from our experiments on LFM2.5-350M and Qwen3-0.6B, starting from the baseline setup (base decoder with SigLIP2 encoder and linear projector), updating SigLIP to a frozen EUPE encoder, and the final setup with frozen EUPE and our proposed improvements.

Group accuracyLFM SigLIP (baseline)LFM frozen EUPELFM final (ours)Qwen SigLIP (baseline)Qwen frozen EUPEQwen final (ours)
Winoground2.753.757.257.504.508.50
BiVLC0.486.217.9110.4010.5715.65

Ultimately, we decided that the baseline scores of these decoder families were not strong enough to justify promoting them to costly downstream benchmarks and further analyses. An additional problem is the lack of information about (and therefore control over) the data mix, training stages etc. of the additional decoders, that made it difficult to adequately address this baseline underperformance. Nonetheless, as seen above, our proposed interventions substantially improved the performance over the baseline across both additional decoder families, similar to what we see for the SmolLM family. We are thus confident that our proposed interventions generalize, and will include these additional results in the manuscript. Thank you again for this constructive point.

The author has addressed my concerns, so I raise my score.

Reviewer_qYosOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces feiner, a framework combining curated training data, diagnostic evaluation, and an efficient training recipe for modular vision-language models built from pretrained vision encoders and language models. The approach focuses on preserving visual representations while enabling the language decoder to use them effectively. It combines a frozen vision encoder with an MLP projector, text-centroid-based bias initialization, and Gram regularization to preserve patch relationships, alongside register tokens with an additional image-attention step to improve visual information use. At a comparable model size, feinerVLM improves average visual benchmark performance over nanoVLM while using approximately 60% less reported multimodal training GPU time, although language benchmark scores decline slightly.

Critical strengths
  • The paper connects its analysis of visual information degradation to concrete design choices. It examines changes in visual representations across the encoder, projector, and decoder, and motivates its interventions through analyses of patch relationships and representation shifts. These empirical findings provide useful guidance for designing compact VLMs.
  • The proposed recipe demonstrates practical improvements in performance and training efficiency. At a comparable model size, feinerVLM improves the average visual benchmark score over nanoVLM, while reducing reported training costs considerably.
Critical weaknesses

I'm willing to raise my rating if the authors adequately address the concerns below:

  • The distinction from and advantages over related representation alignment methods are insufficiently established. VIRAL preserves visual information by aligning intermediate visual tokens with features from pretrained vision models [1]. Although feiner instead preserves patch relationships at the projector output, the benefits of this design choice remain unclear. In the sense that both works aim to preserve the pretrained visual representation in MLLMs, incorporating visual representation alignment method as their baseline training recipe and provide comparisons of performance and training cost will make the proposed method's effectiveness clearer.
  • The independent contributions of individual components and the robustness of their improvements are insufficiently validated. The cumulative ablations make it difficult to assess each component’s necessity and its interactions with other components. In particular, adding registers and residual re-reading yields relatively modest gains: relative binding increases from 17.23 to 17.70, and Winoground group accuracy rises from 10.25% to 11.25%. Ablations that remove individual components from the final model, together with results, would strengthen the evidence.

[1] Visual Representation Alignment for Multimodal Large Language Models, Arxiv 2025

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your positive assessment and your insightful questions.

The distinction from and advantages over related representation alignment methods are insufficiently established. VIRAL preserves visual information by aligning intermediate visual tokens with features from pretrained vision models [1]. Although feiner instead preserves patch relationships at the projector output, the benefits of this design choice remain unclear. In the sense that both works aim to preserve the pretrained visual representation in MLLMs, incorporating visual representation alignment method as their baseline training recipe and provide comparisons of performance and training cost will make the proposed method's effectiveness clearer.

VIRAL shares our motivation to retain visual information. We appreciate the opportunity to connect these approaches more clearly. Our approach preserves the encoder representation before decoder processing, aimed at reducing the need for external alignment.

We ran preliminary VIRAL-style comparisons with EUPE, DINOv3, and SigLIP2 teachers on our base feinerVLM without centroid bias, Gram loss and re-read registers. Interestingly, using the same EUPE encoder as teacher to align the vision embeddings did not improve our model. DINOv3 and SigLIP2 performed better than EUPE alignment, but both trailed centroid-plus-Gram preservation on average group accuracy. Initial analysis of the vision embedding indicates that our preservations have a positive effect on CKNNA throughout the earlier decoder blocks, potentially shifting the need for external alignment. We see combining projector-side preservation with alignment within the decoder as a promising direction for future work. We will include these comparisons in the camera ready.

The independent contributions of individual components and the robustness of their improvements are insufficiently validated. The cumulative ablations make it difficult to assess each component’s necessity and its interactions with other components. In particular, adding registers and residual re-reading yields relatively modest gains: relative binding increases from 17.23 to 17.70, and Winoground group accuracy rises from 10.25% to 11.25%. Ablations that remove individual components from the final model, together with results, would strengthen the evidence.

Thank you for the recommendation to clarify the interaction of the different proposed components. We will add further comparisons to the paper. The table below shows relative binding for the main components to improve performance of our VLM with frozen EUPE encoder, with linear vs. MLP and re-read registers choices as columns and text-centroid bias and Gram regularization as rows.

ComponentsLinearLinear + re-readMLPMLP + re-read
Neither8.3113.8612.7512.74
Bias9.6916.8815.6910.41
Gram5.6513.579.2913.17
Bias + Gram11.7217.0617.2317.70

As seen in the table, the text-centroid bias is helpful across most setups. Gram is mainly helpful when used in combination with centroid bias. This is likely due to the separate bias allowing the model to adjust centroid direction independently of the embedding geometry of the vision tokens regularized by Gram. This trend is similar for both linear and MLP projectors. Re-reading registers yields larger gains with linear projection, yet remains useful when using an MLP projector if Gram is applied as well.

Please note that the different components have coupled effects, which is also reflected in Figure 5 of the paper. For example, the MLP increases patch separation but distorts encoder relationships and produces larger early vision embedding adjustments. Centroid bias reduces early decoder adaptation pressure, while Gram additionally aids preserving the patch relationships. While the individual analysis of each component might not describe their cumulutative effect fully, we agree that they provide additional clarity and will include them in the camera ready.

Thank you for providing additional experimental results. Additional ablations study resolved my concern on the second point.

Regarding the comparison between representation alignment method like VIRAL, could you provide any quantitative results to further compare the effectiveness of each method? Also, could you provide the comparison on the training cost? This will further clarify the effectiveness of proposed method.

We are glad that we could resolve parts of your questions. Below is table of the experiments we have run to compare against VIRAL. Note that the training times are not fully comparable, as our setups without EUPE were previously trained on 8x H100 while we only had 8x RTX PRO 6000 available during the rebuttal.

SetupWG groupRelative bindingTime (min)
EUPE frozen8.7512.7582.6
EUPE frozen + VIRAL EUPE-B7.007.3896.4
EUPE frozen + VIRAL SigLIP2-B7.7511.8496.7
EUPE frozen + VIRAL DINOv3-B10.0010.21104.3
EUPE frozen + centroid bias + Gram10.2517.2385.4

We follow VIRAL and use decoder block 16 for alignment, as well as a three-layer MLP with SiLU activations and hidden width 2048. Compared to using our proposed text-based centroid bias and Gram for embedding preservation, VIRAL with DINOv3-B slightly underperforms (-0.25 on WG group and -7.02 for relative binding). However, VIRAL indeed does improve on our baseline (EUPE + MLP) with their proposed DINOv3-B teacher. We were also interested to test other encoders to evaluate how differences in their pre-training impact their benefit for alignment. We see that aligning to the same encoder as already used in the VLM (EUPE) or to the contrastive pre-trained SigLIP2 does not improve performance. This seems to be similar to the behavior reported in Table 2 in VIRAL.

Thank you for providing additional, interesting quantitative comparison with VIRAL method. This resolves all of my points, thus I’m raising my score to weak accept. Because there are plenty of methods studying the visual representation alignment on MLLMs, I believe incorporating additional analysis to compare those methods together will further validate the proposed training recipe and strengthen the claim.

Reviewer_3PEXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Feiner, a small vision-language model built on nanoVLM (SigLIP2-B + SmolLM2-360M), and studies how visual information can be preserved and effectively used inside a pretrained language decoder. The authors also provide a lightweight training and evaluation recipe in which a single feiner-train run takes about 23 H100 hours and feiner-eval measures relative binding using calibrated Yes/No log-odds. Starting from this base, they incrementally add a series of modifications, including freezing the EUPE encoder and taking its 10th-layer output, replacing the projector with an MLP, adding a bias initialized from text centroids, applying a Gram structure-preserving loss, and finally adding register and image re-read modules. The resulting 450M model improves over nanoVLM by 2.8 points on average across 14 vision benchmarks.

Critical strengths
  1. The paper separates whether visual information is available from whether the decoder actually uses it, and measures each stage of the pipeline separately, turning a vague claim into a concrete diagnostic question.
  2. The frozen-encoder recipe delivers a real efficiency gain, making controlled ablations affordable while still improving visual understanding over the base model.
Critical weaknesses
  1. The metric is unvalidated and used circularly. feiner-eval is never checked against downstream benchmarks, agreement with group accuracy, or a null distribution, yet the encoder, block etc. are all selected on it before being reported as evidence.
  2. The work is an assembly of existing modules. The encoder, intermediate-layer readout, MLP projector, centroid bias, Gram loss, and register tokensall come from prior work. The paper treats this combination as a scientific contribution without showing that the assembly produces an insight .
  3. The "comparable scale" claim is contradicted by the paper table 2. FastVLM-0.5B, at about the same size as feinerVLM-450M , scores 50.72 on the benchmark against 44.37.
Questions for authors
  1. The paper shows that end-to-end training shifts encoder features, which is expected, but feature drift does not by itself imply worse VLM performance. Is freezing the encoder beneficial beyond efficiency?
  2. All ablations are reported only on feiner-eval and FineWeb perplexity, and Table 2 evaluates only the final models.could the authors provide per-component ablations on several downstream benchmarks ?
  3. The paper presents decoder-side information flow as complementary to representation preservation.Could the authors provide direct evidence of improved fusion compared with the baseline? Possible analyses include layer-wise linear probes of visual attributes.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for your constructive feedback.

The metric is unvalidated and used circularly. feiner-eval is never checked against downstream benchmarks, agreement with group accuracy, or a null distribution, yet the encoder, block etc. are all selected on it before being reported as evidence.

To avoid a misunderstanding, let us briefly clarify the pipeline again. feiner-eval combines Winoground and BiVLC, established benchmarks for compositional visual understanding [1,2,3]. The standard metric in these benchmarks is group accuracy, which we complement with relative binding.

While group accuracy asks whether the model correctly matches both images to their corresponding captions (Fig. 2a. Point 3 and 4), relative binding additionally takes the confidence of the model's choices into consideration, capturing improvements even if the prediction remains incorrect. Both metrics are connected (Spearman correlation of 0.635 with Winoground group accuracy and 0.791 with BiVLC group accuracy) but complementary. We thus report both metrics and use them to improve the model's visual understanding (see e.g. line 324).

However, we do not use them circularly as a final test target. Instead, we use the downstream benchmarks in Table 2 as a final hold out validation of our model to test if our interventions actually improve performance.

The work is an assembly of existing modules. The encoder, intermediate-layer readout, MLP projector, centroid bias, Gram loss, and register tokensall come from prior work. The paper treats this combination as a scientific contribution without showing that the assembly produces an insight .

We would like to point out that the paper’s core contribution is an analysis of regressions in visual representations throughout the VLM. The controlled ablations test how targeted interventions address these failures and produce cumulative improvements in visual understanding.

For example, with a plain MLP, the decoder’s initial blocks have substantial adaptation pressure to the detriment of visual tokens (Figure 2b). Concretely, the decoder rotates the centroid of the vision tokens in embedding space and increase its magnitude. In contrast, training with our proposed text-centroid bias reduces early centroid magnitude growth by 45.3% and rotation from 21.07∘21.07^{∘} to 10.43∘10.43^{∘} resulting in the improvements detailed in Section 4.4.

Our paper therefore does not combine arbitrary methods from prior work but performs targeted model improvements based on the insights from this quantitative interpretability analysis.

The "comparable scale" claim is contradicted by the paper table 2. FastVLM-0.5B, at about the same size as feinerVLM-450M , scores 50.72 on the benchmark against 44.37.

Table 2 actually supports competitiveness with several models of comparable size. While FastVLM-0.5B has the higher aggregate score, their naming is slightly misleading: Apple’s released FastVLM-0.5B checkpoint actually contains 758M parameters [4], about 1.7 times our 445M total parameters. In terms of models with size closer to ours, feinerVLM-450M outperforms nanoVLM-460M and SmolVLM-500M on the reported comparisons.

[1] Diwan et al. Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. https://arxiv.org/pdf/2211.00768

[2] Ossowski et al. Prompting Large Vision-Language Models for Compositional Reasoning. https://arxiv.org/pdf/2401.11337

[3] Tamir et al. Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models. https://arxiv.org/pdf/2610.22602

[4] https://huggingface.co/apple/FastVLM-0.5B

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Questions:

The paper shows that end-to-end training shifts encoder features, which is expected, but feature drift does not by itself imply worse VLM performance. Is freezing the encoder beneficial beyond efficiency?

We would first like to address the premise of your statement that feature drift does not by itself imply worse VLM performance. Specifically, the ADE20K results (Fig. 3c) indicate that, while VLM downstream performance might seem unaffected, there are specific properties that downstream benchmarks can miss and that are negatively affected. This has been previously shown in [5,6] and is corroborated by our findings on ADE20K. Here, the model suffers from the reduced access to dense labels in the vision embeddings, resulting in the semantic segmentation performance dropping from 52.67 to 48.15 mIoU.

Moreover, while freezing the encoder does not directly lead to general improvements (Fig. 4a), we do see an improvement on benchmarks that require fine-grained understanding. For example, SpatialEval increases from 24.19 with trainable EUPE to 26.39 when freezing the encoder. Therefore, the core question underlying our paper is how we can counteract the disadvantages of encoder freezing such that our proposed improvements can get the best of both worlds.

All ablations are reported only on feiner-eval and FineWeb perplexity, and Table 2 evaluates only the final models.could the authors provide per-component ablations on several downstream benchmarks ?

We investigated per-component ablations on the benchmarks of Table 2, but found them noisy and unable to capture subtle improvements in visual understanding (see also Section 3.3) because their scores mix visual discrimination with task-specific knowledge and answer preferences. This issue was also described in Chen et al. [7]. We therefore use the controlled image–caption comparisons in Winoground and BiVLC, together with representation analyses, to guide the component ablations.

The paper presents decoder-side information flow as complementary to representation preservation.Could the authors provide direct evidence of improved fusion compared with the baseline? Possible analyses include layer-wise linear probes of visual attributes.

Please note that the paper already includes specific evidence of improved fusion because the feiner-eval tasks (Winoground and BiVLC) already probe the effect of fusion through compositional image-text matching. Nonetheless, we welcome your recommendation and have extended this analysis to intermediate decoder layers on Winoground (WG). In addition, we also directly probe visual attributes using the GQA benchmark and its color attributes on 2,000 held-out questions following [8]. The table below shows layer-wise Winoground group accuracy and GQA color-probe accuracy (%):

SetupWG groupblock 24block 28finalGQA colourblock 24block 28final
No registers6.509.7510.2532.2633.9632.70
Re-read registers6.2510.2511.2531.9235.1533.80

As seen in the table, re-read registers at block 28 improve the score compared to the baseline consistently in the final layers. Please also compare the positive effects of re-read registers in our response to reviewer qYos.

[5] Qiu et al. Refining CLIP's Spatial Awareness: A Visual-Centric Perspective. https://arxiv.org/pdf/2504.02328

[6] Tong et al. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. https://arxiv.org/pdf/2401.06209

[7] Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models?. https://arxiv.org/abs/2403.20330

[8] Michalkiewicz et al. Foveated Probes Recover Localized Binding Information in Vision Foundation Models. https://arxiv.org/abs/2608.00726

AuthorsOpenReview ↗

Gentle ping

Dear Reviewer 3PEX,

We would appreciate if you could engage with our rebuttal and let us know if your concerns have been addressed so that we can use the remaining reviewer-author discussion period to address any outstanding points.

Thank you in advance!

Thank you for the response. It partially addresses my concerns. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.