- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes post-connector distillation, a two-stage training recipe for developing efficient vision encoders for VLMs. In the first stage, a student encoder and connector are trained to reconstruct the post-connector visual tokens of a pretrained VLM using unlabeled images and videos. This enables direct evaluation on VLM benchmarks without additional instruction tuning. The second stage jointly fine-tunes the encoder, connector, and language backbone using visual instruction data. Experiments demonstrate the effectiveness of this recipe through training objective, architecture, and data mixture ablations. The resulting 164M hybrid encoder matches the Qwen3.5-2B vision encoder across 13 VLM benchmarks when paired with a common language backbone, while requiring 2.2× fewer encoder and connector FLOPs.
Critical strengths
Practical training recipe. The proposed post-connector distillation provides a simple way to evaluate vision encoders on downstream VLM benchmarks during training, without repeated instruction tuning. The experiments in Figure 4 show that these early evaluations can help distinguish between different architectures and data mixtures, making encoder development more efficient.
Critical weaknesses
-
Limited methodological novelty and missing key comparisons. The proposed method largely consists of feature reconstruction using standard L1 and cosine losses, followed by conventional supervised fine-tuning. While distilling post-connector features is a reasonable design choice, the paper does not directly compare it with pre-connector distillation under matched training conditions. As a result, it is unclear whether the proposed distillation target offers meaningful advantages over existing approaches.
-
Limited evidence for general encoder quality. The strongest result comes from integrating the encoder with a relatively small 230M language backbone, where it matches Qwen3.5-2B's vision encoder after additional fine-tuning. However, when evaluated with the original Qwen3.5-2B backbone (Table 2), the student still falls behind the teacher, particularly on MMMU-Pro (27.1 vs. 39.7), LogicVista (37.6 vs. 48.8), and RefCOCO-M (86.0 vs. 91.8). The evaluation also focuses almost entirely on downstream VLM benchmarks, without examining standard visual representation transfer tasks. These results leave me unconvinced that the proposed recipe produces a broadly competitive vision encoder beyond the specific evaluated settings.
Questions for authors
-
Post-connector vs. pre-connector distillation. The paper motivates post-connector distillation as a simpler alternative to conventional feature distillation, but does not provide a direct comparison. How does the proposed approach compare with pre-connector distillation when using the same student architecture, training data, and compute budget? This seems important for establishing the benefit of the proposed method.
-
Choice of student architecture. Why did the authors choose a convolution-attention hybrid encoder rather than a smaller ViT? Would a ViT-Small or similarly sized ViT trained with the same distillation recipe achieve comparable performance and efficiency? Such a comparison would help clarify the role of the architecture in the reported results.
-
Performance gaps with the original teacher. In Table 2, the student model still shows substantial degradation on MMMU-Pro (27.1 vs. 39.7), LogicVista (37.6 vs. 48.8), and RefCOCO-M (86.0 vs. 91.8). What explains these gaps, particularly given that the student is trained to reproduce the teacher's post-connector visual representations? How should these results be reconciled with the competitive performance reported using the smaller LFM2.5-230M backbone?
-
Generalization of the learned representations. The evaluation focuses primarily on downstream VLM performance after instruction tuning. How well does the learned encoder transfer to standard visual representation tasks, such as ImageNet classification or dense prediction? Such evidence would help establish whether the proposed recipe learns broadly useful visual features beyond the evaluated VLM settings.