acceptodds
Under review as a conference paper at ICLR 2027

Connectors as Channels: Usable Information between Frozen Vision and Language

Abstract

Vision language models couple a frozen visual encoder to a frozen language model through a small trainable connector. The connector family and the number of retained visual tokens are still chosen by ablation. Published comparisons disagree on which of these choices matters, and they still lack a common account of when a given design is appropriate. The information bottleneck is the classical theory of this kind of compression. It prices a learnable code, while a connector sits between two frozen spaces and is read only through the directions that move the next token loss. In this paper, we model the connector as a channel and measure usable information, the mutual information that remains after the frozen language model has read the mapped tokens. On released Qwen3-VL checkpoints, the readable set recovered from last layer hidden states is much thinner than the leading principal subspace of the embedding table. Identifying this set exposes a concrete cost of query mixing, because lengthening the encoder without new signal dilutes the attention mass on a signal set and lowers usable information. We then estimate canonical spectra from teacher forcing hidden states and from caption vectors, and we train linear, two layer, and query connectors on frozen CLIP features that feed a frozen language model. These results account for the large token counts retained by published designs, for the sufficiency of a single matrix when the encoder is already contrastively aligned, and for the behaviour of a query bottleneck as a mixing mechanism rather than an integer token budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Usable information, Vision-language connectors, Multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r2HyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In Vision-Language Models (VLMs), a trainable connector bridges the gap between the frozen vision encoder and the language decoder. The paper investigates the mutual information of connector families using single Linar layer, two layers and query connectors applied in frozen encoder and decoder scenario.

Critical strengths

The paper analyzes information transfer from frozen vision encoders to language decoders across diverse VLM architectures like LLaVA, BLIP-2, Flamingo, and Qwen3-VL. Their study systematically compares how various connector types facilitate this process.

Critical weaknesses
  • The experimental setup and key findings are hard to verify, as core definitions and parameters are insufficiently explained, see questions below.
  • Figures are not clearly explained what to be seen. For instance Figure 1, ist firstly refered in the introduction without symbold explanation in text or caption. The symbols are eather not explained or only explained in later text passages or appendix. In the experimental section the textual description is inprecise, as only the overall figure is refered, while a specific subfigure is been meant.
  • Equation 2 suffers from structural inconsistency. While X,Z, and Y represent features connected by functions (represented by arrows), T represents the language model, which is a function. Yet it is incorrectly rendered as a feature node.
  • the training settings are not reproducably explained.
  • The experimental results could be extended further. For example, in Figure 4a, the sample size is evaluated at only two data points, whereas Figure 4c omits the query-based approach.
Questions for authors
  • how do you define the readable set U and how is it related to token pruning?
  • How do you verify the LLM-decoder alignment given the mutual feature-subspace from the connector, as the decoder is non-retrainable. Simply maximizing the mutual information in Z^~ is no guaratnee for small channel decoder alignment.
  • How exactly is the connector been trained? I assume only the caption loss form the teacher results are applied to Z^~, so before LLM-decoder?
  • Regrading the training-settings: On what captions in specific do you train? How many samples are generated for training?
  • How is the ortogonal Projector \PI_U been created? Is the projector a channel masking or is the feature been converted into leading subspace of PCA?
  • How is the "usable Information"-function I(Z,Y) Page 4 been defined? Later on page 6 you define the mutual information function, but it is arguably not clear if these functions are the same.
  • What is the difference between m and me in signal mass Eta? I cannot find a definition of me. Why do you have 2 definitions for signal mass eta (1. Sum over alpha and 2. formula to number of keys me, N and m)?
  • in Theorem 4: What is meant by risk? Is it the computational complexity?
Reviewer_EbgWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why different connector architectures and visual-token budgets behave differently when the vision encoder and language model are frozen. It characterizes information accessible to the language model, compares representations derived from hidden states and caption embeddings, and studies linear, two-layer, and query-based connectors under controlled conditions. Released-model analyses find concentrated hidden-state variation, while small-scale connector training shows that architecture rankings depend on the evaluation target. The experiments also investigate how adding irrelevant keys changes query attention and information retention, aiming to inform connector and token-budget choices.

Critical strengths
  1. A useful controlled comparison. Keeping the encoder and reader frozen while varying connector family, token count, and sample size helps separate factors that are often changed together in complete VLM recipes. Importantly, the connector is also evaluated using the frozen language model's caption loss, connecting the comparison to an actual prediction objective. The resulting family-by-token grid makes the sensitivity of architectural rankings visible.
  2. Several complementary empirical views. Released Qwen3-VL analyses, alternative caption representations, and controlled noise-key experiments make the study more informative than a single connector-ranking table. For instance, the reported 90%-variance hidden widths are 104/111 on the 2B/8B readers, compared with embedding widths of 1692/2594 at the same threshold. These concrete measurements highlight how strongly the observed geometry depends on which representation is inspected.
Critical weaknesses
  1. Direct connector-training evidence is too limited for broad design guidance. Appendix I uses one CLIP ViT-B/32–Qwen3-VL-2B pairing, 400 COCO images with 80 held out, and two seeds only where recorded. The evaluation mainly reports teacher-forced caption loss, with retrieval near chance. The 0.044-nat linear–MLP gap at n=320 is also given without run-level variability, making its stability difficult to assess. These results do not yet establish that the recommended family/token choices improve generated captions, VQA, or larger-reader performance.
  2. The query-family interpretation contradicts the empirical rankings. Appendix H states that queries are the weakest caption-vector reader in both Tables 9–10. However, Table 9 at k=32 gives query 5.82 versus two-layer 3.56 nats, and Table 10 at k=1 gives 3.46 versus 2.62. These are counterexamples within the same metric, affecting a central comparison used to motivate the query-mixing account. The blanket ranking should be corrected, with the query disadvantage limited to the comparisons that actually support it.
Questions for authors
  1. Please reconcile the “weakest reader” statement with Tables 9–10. Which specific comparisons support a query disadvantage, and which favor queries?
  2. Which existing held-out task results connect the proposed information diagnostics to better connector/token-budget selection? Please also identify the runs with repeated seeds and provide their variability, especially for the 0.044-nat linear–MLP gap.
Reviewer_9ersOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the connectors between a frozen vision encoder and a frozen language model. It defines usable information by projecting connector outputs onto an estimated language-readable subspace, and then develops claims about linear sufficiency, query mixing, and also the sample cost of nonlinear connectors. its experiments measure embedding and hidden-state geometry on released checkpoints, estimate canonical correlation spectra, as well as train three connector families using a frozen CLIP encoder and Qwen3-VL reader.

Critical strengths
  1. The paper explicitly distinguishes the connector family, token count, and frozen reader, as well as task representation. This is a useful approach to organize the problem, since a spectrum computed against a caption embedding do not need to describe the task measured by caption-generation loss. Comparing several target representations on the same image features makes that distinction concrete.

  2. The experiments train linear, two-layer, and query connectors, while keeping the CLIP encoder and Qwen3-VL language model frozen. They evaluate caption loss on held-out images and vary token count and training-set size. Measuring the actual reader loss in addition to representation statistics, is a useful part of the study, especially because the two evaluations rank the connector families differently.

Critical weaknesses
  1. The theory treats the frozen language model as reading connector outputs only through a fixed projection onto U. The operational choice of U, however, is PCA of final-layer hidden states on teacher-forced captions. Concentration of output-state variance does not establish that input directions outside that subspace have no effect on the loss.

  2. The proof of theorem2 writes the correlation of a mixture as the weighted average of its component correlations in line 848 of Appendix C. This omits normalization by the mixture variance and its dependence on the covariance between s and b. Consequently, the claimed monotonicity is false under the stated assumptions, even in a fixed-weight jointly Gaussian example.

Questions for authors
  1. What additional assumptions make the information claims valid?

  2. Does the estimated subspace predict actual loss sensitivity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.