- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This study proposes CONVCUE, a method that improves VLM performance by adding a frozen CNN (ConvNeXt-base) to the VLM backbone. It fuses the features from the ViT in the VLM backbone and those from the CNN through cross-attention. Then, the model is trained in two stages using image-text pairs and visual instruction data. Extensive experiments on three VLM backbones and 13 benchmarks show that the proposed method consistently improves performance.
Critical strengths
- The proposed method shows consistent improvements on three VLM backbones and 13 benchmarks.
- The study provides various analyses, including pilot experiments, representation magnitude analysis, and layer-wise analysis.
Critical weaknesses
Major Weaknesses
- The distinction and novelty of the proposed method compared with existing approaches seem limited. Approaches including CNN + ViT dual encoders, supplementary CNN features, multi-encoder fusion, and cross-attention-based fusion have already been studied in prior works [1–5]. In particular, Mini-Gemini proposed a cross-attention-based approach that uses low-resolution CLIP tokens as Q and high-resolution ConvNeXt features as K/V [4]. FOCUS [5] also combines QwenViT and ConvNeXt-L dual branches using cross-attention and gated cross-attention. This study cites Cambrian-1 [1], LLaVA-HR [2], and ConvLLaVA [3], but does not discuss Mini-Gemini [4] and FOCUS [5], which are closely related methods. Therefore, the remaining methodological distinction seems limited to the specific integration design that injects frozen CNN features into native VLM visual tokens through a gated residual connection.
[1] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.
[2] Luo, Gen, et al. "Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models." International Conference on Learning Representations. Vol. 2025. 2025.
[3] Ge, Chunjiang, et al. "Convllava: Hierarchical backbones as visual encoder for large multimodal models." arXiv preprint arXiv:2405.15738 (2024).
[4] Li, Yanwei, et al. "Mini-gemini: Mining the potential of multi-modality vision language models." IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).
[5] Yang, Fan, et al. "Focus: Unified vision-language modeling for interactive editing driven by referential segmentation." Advances in Neural Information Processing Systems 38 (2026): 33408-33437.
-
The effect of additional model capacity should be controlled. CONVCUE adds a pretrained CNN branch, CNN adapter, and cross-attention module to the baseline, increasing the model size and representational capacity. However, the study does not provide a corresponding parameter or capacity-matched control. Thus, it is difficult to determine whether the performance gains come from the proposed heterogeneous representations, the convolutional inductive bias, or simply the additional model capacity.
-
The core claim of complementary convolutional inductive bias is not experimentally isolated. CONVCUE simultaneously introduces a pretrained ConvNeXt, additional adapter parameters, additional pretrained visual knowledge, and additional inference compute. The proposed Direct SFT only controls for the effect of additional training, but does not control for these factors. In particular, there is no comparison with a comparable pretrained ViT auxiliary branch. Thus, it is difficult to determine whether the performance gain comes from the convolutional inductive bias or simply an additional pretrained visual representation.
-
The training process substantially retrains the existing VLM itself. Only the CNN module is frozen, the original visual encoder and projector are trained from Stage 1, and the LLM is additionally unfrozen in Stage 2. Therefore, it is difficult to interpret the final performance improvement as the effect of CNN feature augmentation alone.
-
The performance comparisons with representative prior methods are not controlled methodological comparisons. For a fair comparison with alternatives such as Cambrian-1, LLaVA-UHD-v3, Eagle, and Florence-VL in Table 2, the same backbone, training data, and training recipe should be used while varying only the fusion method. However, this study does not provide such a comparison.
-
The average values in Table 1 are not directly comparable. They simply average the available benchmark scores, although the benchmark subsets differ across methods. The study should use a common subset or remove the average for incomplete rows.
Minor Weaknesses
- The meaning of the superscript I for and in Table 2 is not defined.
- CONVCUE's expression "Scale = 4B" in Table 1 may be misread as the total model size. Since CONVCUE includes an additional ConvNeXt, adapter, and fusion module, it would be clearer to use LLM Scale or Backbone Scale.
- The exact pretrained ConvNeXt-Base checkpoint or source should be specified for reproducibility.
- This study does not contain any proofs. Please revise the reproducibility statement.
- L716: GPUS → GPUs. This sentence is also redundant with the previous one.
- L719: uhd → LLaVA-UHD.
Questions for authors
- Considering "All the tests follow their paper" in L303 and "tested from the same split" in L760, I am wondering whether the reported results are taken from the original papers or reevaluated using the official checkpoints.