acceptodds
Withdrawn from ICLR 2027

Beyond Homogeneous Visual Representations: Complementary Visual Inductive Biases for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern vision-language models (VLMs) achieve strong performance across a broad range of multimodal tasks, yet still struggle with visual questions that require fine-grained discrimination and spatial understanding. These limitations motivate investigating whether supplementary visual representations can improve existing VLMs without replacing their native visual encoders. Pretrained convolutional networks offer a candidate feature source, motivated by their local connectivity and spatial weight sharing. We introduce CONVCUE, which augments the native visual representations of a pretrained VLM with final-stage features from a parallel, frozen pretrained CNN. A learnable adapter maps convolutional features to the native visual feature dimension, while gated cross-attention allows the original visual tokens to retrieve information from the CNN features. The enhanced tokens are passed through the original visual-to-language projector, and the model is adapted through a two-stage training procedure. We evaluate CONVCUE on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B across 13 multimodal benchmarks covering visual question answering, document and chart understanding, and multimodal reasoning. CONVCUE improves average benchmark performance over both the original models and matched two-stage fine-tuning controls on all three backbones. On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks and raises the average score from 75.00 to 78.82 relative to the matched fine-tuning control. These results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, CNN, Visual Ability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o1bGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This study proposes CONVCUE, a method that improves VLM performance by adding a frozen CNN (ConvNeXt-base) to the VLM backbone. It fuses the features from the ViT in the VLM backbone and those from the CNN through cross-attention. Then, the model is trained in two stages using image-text pairs and visual instruction data. Extensive experiments on three VLM backbones and 13 benchmarks show that the proposed method consistently improves performance.

Critical strengths
  • The proposed method shows consistent improvements on three VLM backbones and 13 benchmarks.
  • The study provides various analyses, including pilot experiments, representation magnitude analysis, and layer-wise analysis.
Critical weaknesses

Major Weaknesses

  1. The distinction and novelty of the proposed method compared with existing approaches seem limited. Approaches including CNN + ViT dual encoders, supplementary CNN features, multi-encoder fusion, and cross-attention-based fusion have already been studied in prior works [1–5]. In particular, Mini-Gemini proposed a cross-attention-based approach that uses low-resolution CLIP tokens as Q and high-resolution ConvNeXt features as K/V [4]. FOCUS [5] also combines QwenViT and ConvNeXt-L dual branches using cross-attention and gated cross-attention. This study cites Cambrian-1 [1], LLaVA-HR [2], and ConvLLaVA [3], but does not discuss Mini-Gemini [4] and FOCUS [5], which are closely related methods. Therefore, the remaining methodological distinction seems limited to the specific integration design that injects frozen CNN features into native VLM visual tokens through a gated residual connection.

[1] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.

[2] Luo, Gen, et al. "Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Ge, Chunjiang, et al. "Convllava: Hierarchical backbones as visual encoder for large multimodal models." arXiv preprint arXiv:2405.15738 (2024).

[4] Li, Yanwei, et al. "Mini-gemini: Mining the potential of multi-modality vision language models." IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).

[5] Yang, Fan, et al. "Focus: Unified vision-language modeling for interactive editing driven by referential segmentation." Advances in Neural Information Processing Systems 38 (2026): 33408-33437.

  1. The effect of additional model capacity should be controlled. CONVCUE adds a pretrained CNN branch, CNN adapter, and cross-attention module to the baseline, increasing the model size and representational capacity. However, the study does not provide a corresponding parameter or capacity-matched control. Thus, it is difficult to determine whether the performance gains come from the proposed heterogeneous representations, the convolutional inductive bias, or simply the additional model capacity.

  2. The core claim of complementary convolutional inductive bias is not experimentally isolated. CONVCUE simultaneously introduces a pretrained ConvNeXt, additional adapter parameters, additional pretrained visual knowledge, and additional inference compute. The proposed Direct SFT only controls for the effect of additional training, but does not control for these factors. In particular, there is no comparison with a comparable pretrained ViT auxiliary branch. Thus, it is difficult to determine whether the performance gain comes from the convolutional inductive bias or simply an additional pretrained visual representation.

  3. The training process substantially retrains the existing VLM itself. Only the CNN module is frozen, the original visual encoder and projector are trained from Stage 1, and the LLM is additionally unfrozen in Stage 2. Therefore, it is difficult to interpret the final performance improvement as the effect of CNN feature augmentation alone.

  4. The performance comparisons with representative prior methods are not controlled methodological comparisons. For a fair comparison with alternatives such as Cambrian-1, LLaVA-UHD-v3, Eagle, and Florence-VL in Table 2, the same backbone, training data, and training recipe should be used while varying only the fusion method. However, this study does not provide such a comparison.

  5. The average values in Table 1 are not directly comparable. They simply average the available benchmark scores, although the benchmark subsets differ across methods. The study should use a common subset or remove the average for incomplete rows.

Minor Weaknesses

  • The meaning of the superscript I for SEEDI\text{SEED}^\text{I} and SQAI\text{SQA}^\text{I} in Table 2 is not defined.
  • CONVCUE's expression "Scale = 4B" in Table 1 may be misread as the total model size. Since CONVCUE includes an additional ConvNeXt, adapter, and fusion module, it would be clearer to use LLM Scale or Backbone Scale.
  • The exact pretrained ConvNeXt-Base checkpoint or source should be specified for reproducibility.
  • This study does not contain any proofs. Please revise the reproducibility statement.
  • L716: GPUS → GPUs. This sentence is also redundant with the previous one.
  • L719: uhd → LLaVA-UHD.
Questions for authors
  1. Considering "All the tests follow their paper" in L303 and "tested from the same split" in L760, I am wondering whether the reported results are taken from the original papers or reevaluated using the official checkpoints.
Reviewer_tb7ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CONVCUE, a framework that augments the native visual representations of a pretrained vision-language model (VLM) with features from a parallel, frozen pretrained CNN (ConvNeXt-Base), without replacing the original ViT encoder. The motivation is that VLMs often fail on questions requiring fine-grained discrimination and spatial understanding, and that the local connectivity and spatial weight sharing of CNNs may provide a complementary inductive bias to the ViT-based native encoder. Concretely, a learnable adapter maps the CNN's final-stage features to the native visual token dimension; a gated cross-attention module then lets the original visual tokens (as queries) retrieve information from the CNN features (as keys/values), with a channel-wise sigmoid gate controlling a residual update. The enhanced tokens pass through the original visual-to-language projector, so the token count sent to the LLM is unchanged. The model is adapted via a two-stage procedure (vision-language alignment with a frozen LLM, then joint supervised fine-tuning). Design choices (cross-attention over weighted fusion; adapting the visual-side components) are first justified through pilot experiments on a COCO-derived Spatial QA task. CONVCUE is evaluated on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B across 13 multimodal benchmarks, where it improves average performance over both the original models and matched Direct SFT controls; on Qwen3-VL-4B it improves on all 13 benchmarks and surpasses Qwen3-VL-8B on each. The paper adds representation-update analyses, a case study of feature sensitivity and layer-wise answer readouts, and an inference-efficiency study showing modest overhead.

Critical strengths
  1. Clean, practical, and deployment-friendly design. The method keeps the original encoder frozen-path intact, uses a frozen CNN, does not increase the number of visual tokens passed to the LLM, and adds only a small inference cost (+7–10 ms time-to-first-token and +0.41 GiB peak memory in the reported setting). This makes it an attractive plug-in augmentation that can be applied to existing VLMs without retraining or replacing their visual backbones.

  2. The matched Direct SFT control is a genuine methodological strength. By rerunning the exact two-stage procedure—same data, same unfreezing schedule, same hyperparameters—but without the CNN branch, the paper separates the benefit of the added architecture from the benefit of simply training more. The contrast is most convincing on LLaVA-OneVision-7B, where Direct SFT slightly hurts the average (−0.39) while CONVCUE substantially improves it (+9.75), indicating the gains are not merely an artifact of additional supervised training on the same data.

  3. The gain distribution is consistent with the stated hypothesis. The largest improvements appear on GQA (+11), RealWorldQA, HallusionBench, and MMStar—precisely the spatial, compositional, and fine-grained tasks that the "complementary convolutional locality" story predicts, rather than a uniform or noise-like lift across benchmarks. This coherence between the mechanism claimed and the empirical pattern observed lends the results additional credibility.

Critical weaknesses

The central scientific claim (convolutional inductive bias is the key ingredient) is not fully validated. The entire narrative rests on the complementary nature of convolutional locality, yet the supplementary branch is never swapped for a non-convolutional frozen encoder (e.g., DINOv2, SAM, or a second ViT with a different training objective). As a result, what the experiments actually establish is that "adding a frozen heterogeneous feature source via gated cross-attention helps," not that convolution specifically is responsible. Without a CNN-vs-non-CNN supplementary-encoder ablation, the title's framing around "convolutional inductive biases" is an overclaim.

Capacity/parameter confound is not ruled out. The Direct SFT control adds no extra parameters, whereas CONVCUE introduces an adapter plus an 8-head cross-attention fusion module. It therefore remains unclear how much of the gain comes from the content of the CNN features versus the additional trainable fusion capacity. A control in which the cross-attention attends to randomly-initialized / noise "CNN" features, or attends back to the ViT's own features, would be needed to cleanly separate "useful CNN information" from "extra parameters and an extra attention module."

Novelty is incremental. Combining CNN and ViT pathways and fusing features from multiple visual encoders is a well-explored direction (Cambrian-1, Eagle, LLaVA-HR, ConvLLaVA, Florence-VL), as the related-work section itself acknowledges. The specific delta here—a frozen CNN as cross-attention keys/values, retaining the original pathway, with a channel-wise gated residual—is a sensible but modest combination of existing components rather than a conceptually new mechanism.

The closed-source baselines are a generation or two out of date, inflating the "competitive with larger models" framing. The commercial reference points—GPT-4V (0409), GPT-4.1, Claude-3.7-Sonnet, and Gemini-1.5-Pro—are all from substantially earlier model generations relative to an ICLR 2027 submission, and current frontier systems have moved well beyond them. Comparing a 2026-era method against these stale references makes the gap look more favorable than it is and weakens any claim of being competitive at the frontier. Since the core contribution is actually an open-backbone improvement validated against matched Direct SFT controls and Qwen3-VL-8B, the closed-source numbers add little and, as presented, risk overstating the result. The authors should either refresh the comparison with contemporary closed-source models or explicitly demote these entries to historical context rather than peer baselines.

Questions for authors

See Weaknesses.

Reviewer_rkBHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ConvCue, which augments the visual tokens of a pretrained VLM with features from a frozen ConvNeXt-Base. A trainable adapter maps the CNN features to the ViT width, and the original ViT tokens query them through gated cross-attention before the original projector. The model is trained in two stages: alignment, then full SFT on LLaVA-UHD data. On Qwen3-VL-2B/4B and LLaVA-OneVision-7B, ConvCue improves the 13-benchmark average over both the original models and a matched Direct SFT control.

Critical strengths
  • The design is simple and non-invasive: the original encoder and projector are kept, and the token count reaching the LLM does not change. Inference overhead is small (about 8 ms and 0.4 GB).
  • Including a matched Direct SFT control with the same data and schedule is good practice.
  • The method is evaluated on three backbones across two model families.
Critical weaknesses
  • The author evaluates the baseline model with a different configuration (i.e generated token limit), which leads to a huge gap between the official report of Qwen3VL and the reported figures in this paper. The reviewer highly suggests that the authors should use the same evaluation protocol for fair comparison. HallusionBench also mixes question-level accuracy (ConvCue, 76.08) with the official averaged metric (InternVL3.5-4B, 44.8).
  • In Table 2, the untouched Qwen3-VL-4B already averages 81.31 on the nine benchmarks, above every listed method (max 79.69). The table therefore measures the ability of backbone models, not ConvCue. Within-backbone controls are also missing. The paper does not compare against existing fusion designs on the same backbone, such as Mini-Gemini-style ViT-to-ConvNeXt cross-attention, Cambrian’s SVA, or a second ViT branch.
  • The gains are unusually large and unexplained. On LLaVA-OV-7B, ConvCue adds about 12–13 points on MMMU, RealWorldQA, ERQA, and GQA, while Direct SFT with identical data slightly drops. The CNN only sees a frozen, center-cropped 336px view (less than the ViT sees), so it is unclear how it improves knowledge-heavy benchmarks like MMMU.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.