acceptodds
Withdrawn from ICLR 2027

Post-Connector Distillation: An Efficient Recipe for Training Vision Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different efficient vision encoder architectures are needed for different hardware, but developing a new vision encoder requires a large amount of compute, elaborate training pipelines, and repeated downstream fine-tuning to measure its quality on VLM tasks. For fast iteration, we introduce post-connector distillation, a simple two-stage recipe for learning from an existing VLM. The first stage is a simple student-teacher reconstruction loss over the final multimodal connector output embeddings, which requires only unlabeled images or videos. This stage enables direct evaluation on downstream VQA benchmarks without additional instruction tuning, significantly accelerating model development. The second stage jointly refines the encoder, connector, and language backbone through supervised fine-tuning (SFT). It closes the remaining capability gaps between the student vision encoder and teacher vision encoder. In our ablations, training loss is a poor indicator of encoder quality, while the early benchmark scores predict which data mixture and architecture perform better after fine-tuning. Paired with a different language backbone, it matches Qwen3.5-2B's ViT across 13 VLM benchmarks while requiring 2.2 fewer encoder-plus-connector FLOPs, and outperforms all other evaluated encoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Encoder, Distillation, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BQTqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes post-connector distillation, a two-stage training recipe for developing efficient vision encoders for VLMs. In the first stage, a student encoder and connector are trained to reconstruct the post-connector visual tokens of a pretrained VLM using unlabeled images and videos. This enables direct evaluation on VLM benchmarks without additional instruction tuning. The second stage jointly fine-tunes the encoder, connector, and language backbone using visual instruction data. Experiments demonstrate the effectiveness of this recipe through training objective, architecture, and data mixture ablations. The resulting 164M hybrid encoder matches the Qwen3.5-2B vision encoder across 13 VLM benchmarks when paired with a common language backbone, while requiring 2.2× fewer encoder and connector FLOPs.

Critical strengths

Practical training recipe. The proposed post-connector distillation provides a simple way to evaluate vision encoders on downstream VLM benchmarks during training, without repeated instruction tuning. The experiments in Figure 4 show that these early evaluations can help distinguish between different architectures and data mixtures, making encoder development more efficient.

Critical weaknesses
  1. Limited methodological novelty and missing key comparisons. The proposed method largely consists of feature reconstruction using standard L1 and cosine losses, followed by conventional supervised fine-tuning. While distilling post-connector features is a reasonable design choice, the paper does not directly compare it with pre-connector distillation under matched training conditions. As a result, it is unclear whether the proposed distillation target offers meaningful advantages over existing approaches.

  2. Limited evidence for general encoder quality. The strongest result comes from integrating the encoder with a relatively small 230M language backbone, where it matches Qwen3.5-2B's vision encoder after additional fine-tuning. However, when evaluated with the original Qwen3.5-2B backbone (Table 2), the student still falls behind the teacher, particularly on MMMU-Pro (27.1 vs. 39.7), LogicVista (37.6 vs. 48.8), and RefCOCO-M (86.0 vs. 91.8). The evaluation also focuses almost entirely on downstream VLM benchmarks, without examining standard visual representation transfer tasks. These results leave me unconvinced that the proposed recipe produces a broadly competitive vision encoder beyond the specific evaluated settings.

Questions for authors
  1. Post-connector vs. pre-connector distillation. The paper motivates post-connector distillation as a simpler alternative to conventional feature distillation, but does not provide a direct comparison. How does the proposed approach compare with pre-connector distillation when using the same student architecture, training data, and compute budget? This seems important for establishing the benefit of the proposed method.

  2. Choice of student architecture. Why did the authors choose a convolution-attention hybrid encoder rather than a smaller ViT? Would a ViT-Small or similarly sized ViT trained with the same distillation recipe achieve comparable performance and efficiency? Such a comparison would help clarify the role of the architecture in the reported results.

  3. Performance gaps with the original teacher. In Table 2, the student model still shows substantial degradation on MMMU-Pro (27.1 vs. 39.7), LogicVista (37.6 vs. 48.8), and RefCOCO-M (86.0 vs. 91.8). What explains these gaps, particularly given that the student is trained to reproduce the teacher's post-connector visual representations? How should these results be reconciled with the competitive performance reported using the smaller LFM2.5-230M backbone?

  4. Generalization of the learned representations. The evaluation focuses primarily on downstream VLM performance after instruction tuning. How well does the learned encoder transfer to standard visual representation tasks, such as ImageNet classification or dense prediction? Such evidence would help establish whether the proposed recipe learns broadly useful visual features beyond the evaluated VLM settings.

Reviewer_fgW9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose post-connector distillation, with a novel recipe to train small/efficient vision encoders for VLMs. The first stage is distillation pre-training w/ student-teacher reconstruction loss, which avoids the need for image-text pairs (only unlabeled images), and is able to be ready for VQA benchmarking after this stage. The second stage is more traditional, fine-tuning on the image-text pair. The end product is an interesting development path/roadmap that was produced a lightweight 164M encoder, that has on-par performance w/ Qwen3.5-2B 306M encoder. The end output is competitive on multiple major benchmarks, and requires significantly fewer (2.2x less) FLOPS.

Critical strengths

The work proposes a way to evaluate early development in a model w/out the need for instruction fine-tuning. The early stage could theoretically be used to narrow the experimentation space and save cost for model development. Experimental results show a comparable smaller encoder can be comparable to a larger Qwen2B encoder w/ less FLOPs.

Critical weaknesses

While the possible benefits of an “early stop” to evaluate to save FLOPs on further, costly instruction tuning is a strong benefit. It’s hard to claim that the recipe itself is novel/powerful. There is should be additional experimentation in a controlled setting w/ a fixed architecture and similar training constraints to other recipes to see if the ultimate encoder quality on tasks are still beneficial. For example, could another recipe also produce a just as performant 164M encoder, given same architecture and training data? It’s unclear from the current results.

Questions for authors
  1. Any thoughts/followup on the novelty of the recipe itself in a controlled setting
  2. It’s unclear why RefCOCO-M has such performance gaps - additional empirical and qualitative explanations would help the reader on the strengths/shortcoming of the method.
Reviewer_SqgKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on building an efficient recipe for vision encoder training used in a typical VLM architecture (vision encoder -> connector -> language model). They make the observation that the current prototyping process for building strong vision encoders is quite expensive and is slow to iterate, as each time you’d need to train the full VLM to measure downstream VLM scores. They instead propose a post-connector distillation stage, in which they align the representation of a teacher encoder and a student encoder at the output of the connector. This way, the student encoder could directly use the language model backbone of the teacher VLM. Once a good student encoder is found, the entire VLM can be finetuned in an additional SFT stage. For this, they consider a Qwen3.5-2B as teacher and a 165M param hybrid architecture as the student. Evaluations are performed on a set of standard VLM tasks to show the effectiveness of the distillation method.

Critical strengths

The paper tackles a timely and practical problem and explores a reasonable approach. The recipe is a somewhat minor modification from previous distillation works like AM-RADIO (post connector instead of pre connector) which is simple and does not require labels.

Critical weaknesses

While I acknowledge the usefulness of evaluating the VLM checkpoints during pretraining, I found the overall applicability of this approach to be rather limited and empirical results to be inadequate.

  • Can we expect meaningful iteration speed when we consider more practical settings with strong & large teachers? The paper only considers a 2B encoder, so empirical evidence here is weak. What are the boundary conditions and scalability of this distillation strategy precisely?
  • The paper’s main point being able to predict which data mixture and architecture perform better post SFT is not convincingly shown. I found Figure 4 to be a rather weak demonstration. First of all, why do we compare different objective losses or mixtures? Aren’t we supposed to track a “validation loss” that should be a good proxy for the post-SFT performance? And, to demonstrate this, we’d need to have several data points with different training recipes and configurations to fit a trend. Without a strategy similar to this, I am not sure if we can claim this main point.

Thus, I am leaning towards rejection and recommend a major revision & resubmission.

Questions for authors
  • Can we expect meaningful iteration speed when we consider more practical settings with strong & large teachers? The paper only considers a 2B encoder, so empirical evidence here is weak. What are the boundary conditions and scalability of this distillation strategy precisely?
  • Please see my comment about the predictability of post-SFT trends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.