acceptodds
Under review as a conference paper at ICLR 2027

Learn and Pin: A Transferable JEPA-Style Visual Token Compressor for Frozen VLM Hosts

Abstract

In vision–language models, the image dominates the input: LLaVA-1.5 renders one image as tokens, far more than a typical prompt, so most image compute is spent inside the language model. Existing visual token compressors either hand-specify the pre-designed method or learn it with task supervision tied to one host. None learns label-free model inside a frozen encoder and transfers it, unmodified, to a new host. In this paper, we train a small compressor inside a frozen vision encoder without captions. It uses a JEPA-style predictive term and a distributional pin. The pin is a kernel two-sample statistic that keeps the distribution of the compressed tokens close to the encoder's own. The compressor then runs frozen in the encoder's forward pass and is evaluated zero-shot. We do not quote published numbers. We re-run all competitors, both published methods and training-free anchors, in one setting. They are matched on token count, or on total cost for decoder-side pruners. At (a quarter of the tokens) on the LLaVA + SmolLM2-135M parameter host, the proposed method leads the field by CIDEr zero-shot and one-shot at of the uncompressed analytic cost. The compressor transfers unchanged into LLaVA-1.5-7B. It retains – of the uncompressed ceiling on five of six in-context tasks and – on five standard benchmarks, at of the analytic cost. It also transfers to an unseen Q-Former bridge at retention, over the strongest competitor. On CLIP retrieval, it retains of recall at and leads by .

Primary area:
foundation or frontier models, including LLMs
Keywords:
visual token compression, vision-language models, JEPA, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KtMdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a visual token compression method by predicting masked content in latent space of a frozen vision encoder to force it to generate fewer tokens while preserving the original semantics. After training the compressor, it can adapt to other downstream Vision-Language models (VLMs). Experimental results show a certain level of effectiveness, but not quite interesting. The paper's organization and writing is poor and reading or understanding it is difficult. It might be an auto-research paper.

Critical strengths

Developing visual token compression methods that can adapt to different VLMs without requiring task-specific annotations is an important research direction. This paper proposes a rather effective method to address this issue. The method is motivated by JEPA (Joint-Embedding Predictive Architecture). The authors propose or adopt simple interpretable training objectives to learn a compressor for a frozen vision encoder.

Critical weaknesses
  1. The paper's organization and writing is a disaster. It doesn't look like a formal conference paper, and might be generated by AI. For example, there are so many unnecessary detailed experimental result numbers in abstract, meaningless italics, unusual wording like 'competitor', 'host', etc. The presentation of experiments is also poor. I expect that there should be a clear table that shows the proposed method's results and other compared methods, while the main table is organized in a chaos.
  2. The method is not novel. The authors simply combine existing methods without proposing any new one.
  3. The experimental results are not good. In table 3, the proposed method cannot exceed ToMe (ICLR 2023), and the compared methods are not the latest algorithms.
Questions for authors

Please do research by yourself. It is obvious that this paper cannot match the ICLR-level. It is meaningless to submit your AI-generated paper to a top-tier venue, because there is no chance to be accepted. You should read your submission at first to understand what and how it is.

2 comments

We thank the reviewer for recognising that label-free visual token compression that adapts to different VLMs is an important direction, and that our training objectives are simple and interpretable.

The goal of this paper is a transferable, self-supervised compressor that is trained once and inserted into a frozen host VLM, so the host never has to be retrained. Our aim is performance comparable to the uncompressed host and to existing token-compression methods, with little or no degradation, not necessarily to outperform trained compressors that co-adapt the host. The saving comes from transfer: one training run serves every host that shares the same vision encoder.

We acknowledge that our wording overstated the scope of this transferability. Our experiments cover hosts built on CLIP-based vision transformers, not all vision encoders. CLIP-based VLMs are widely used in research and practice, but we will revise the abstract and Introduction to state this scope precisely, and we plan to extend the approach to other encoder architectures in future work. Below, we address the reviewer’s comments from this perspective.

While the individual components of our method are conventional, we claim novelty in combining them to address a new problem. We show that this combination produces a compressor that is trained once and transfers across host models without modification, while performing comparably to state-of-the-art methods.

Regarding the comment that the paper may be AI-generated: we confirm that the work, including the method, experiments, and analysis, is our own. As stated in the AI assistance disclosure, we used an AI tool only to help improve the wording. All authors reviewed and edited the manuscript, and the paper was proofread several times. We take the feedback on readability seriously and we can further improve the paper’s readability considering the reviewer’s suggestions about the choice of some words. We respectfully ask that the discussion focus on the technical content.

Comparison with recent methods. Table 3 already includes VisionZip (CVPR 2025). To compare with the latest methods without re-implementing them, Table R1 uses the numbers published by VisionZip and EvoComp (CVPR 2026). Individual benchmarks move in both directions: against VisionZip at 128 tokens we lead on GQA (+0.83) and POPE (+1.77) and trail on MMBench (−1.74); against EvoComp we lead on POPE (+1.77) and trail on GQA (−0.79) and MMBench (−3.44). The mean over the three benchmarks gives the overall picture: at K=144K=144 our compressor retains 95.53%, within one point of VisionZip (95.25%) and EvoComp (96.35%) at 128 tokens, their nearest published budget. EvoComp, however, is trained for each host with labelled data, whereas our compressor uses no labels, never sees LLaVA-1.5-7B and is inserted unchanged. It therefore performs comparably to the state of the art without any training in the host. We will add this table to the paper.

*Table R1. LLaVA-1.5-7B, zero-shot, retention (%) against the uncompressed host of each evaluator. Published rows are from the VisionZip and EvoComp papers; our row uses our evaluator (Table 3 of the submission). Mean: mean of GQA, MMBench and POPE.

MethodTraining signalKKGQAMMBenchPOPEMeanVQAv2SEED
VisionZip (CVPR 2025)none (CLS attention)12893.0595.8396.8695.2596.3193.69
VisionZip (CVPR 2025)none (CLS attention)19295.8097.3799.3097.4997.8396.25
EvoComp (CVPR 2026)host loss, labelled data12894.6797.5396.8696.3595.92–
EvoComp (CVPR 2026)host loss, labelled data19296.4597.9998.9597.8097.58–
Ours, gridlabel-free, no decoder14493.8894.0998.6395.5393.7694.54

ToMe in Table 3. We agree that ToMe is at par with or slightly ahead of our compressor in Table 3, and we will state this in the abstract. ToMe leads by 0.45–1.81 points on POPE, GQA, MMBench and MME, our compressor leads on SEED (+2.45), and over the eight tasks of Tables 2–3 outside TextVQA the two match (94.7% against 94.9% mean retention). ToMe's strength, however, is specific to this host. No training-free heuristic is strong on every host, whereas our compressor has the highest retention of the four methods on every host outside TextVQA (Table R2). ToMe keeps only 50.3% of the uncompressed captioning score on the 135M decoder, against 84.9% for ours, and trails ours by 5.0 points on the Q-Former bridge. Choosing a heuristic for a new host therefore requires a labelled evaluation on that host, whereas our compressor is trained once per encoder, without labels. ToMe is also not cheaper at inference: it uses 31.6% of the uncompressed GFLOPs, against 31.1% for ours. On TextVQA, the 2×2 grid averaging discards the fine detail that reading text requires; our attention-pool variant raises retention from 45.0% to 66.1%, close to ToMe (67.6%), and we will add it to Table 3.

Table R2. Retention (% of the uncompressed host) at matched token count, on every host in the paper. 7B: mean over the nine tasks of Tables 2–3 except TextVQA. *Below ToMe, the strongest competitor in this cell (Table 2). †Grid / attention-pool.

HostTaskKKToMeToMe 1-shotVisionZipOurs
LLaVA-1.5-7B8 tasks, mean14494.790.991.494.9
BLIP-2 host (Q-Former)COCO CIDEr14491.6<91.6*<91.6*96.6
Research host (135M)COCO CIDEr14450.373.674.084.9
CLIP ViT-B/16Flickr30k rˉ\bar{r}4993.373.863.894.1
CLIP ViT-B/16Flickr30k rˉ\bar{r}9898.696.597.699.5
Worst case over the five rows above50.373.663.884.9
LLaVA-1.5-7BTextVQA14467.634.971.945.0 / 66.1†
Reviewer_ZA3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes "Learn and Pin," a label-free visual token compressor installed inside a frozen vision encoder. It trains two pooling variants solely on the encoder's internal activations using two objectives: a JEPA-style masked feature prediction loss L_{pred} to preserve visual content, and a multi-bandwidth RBF kernel MMD loss (L_{pin}) to keep the compressed tokens on the original feature manifold. Once trained, the module transfers zero-shot to different frozen VLM hosts that share the same vision backbone.

Critical strengths

The paper identifies why self-supervised token compressors fail on frozen VLM hosts and resolves this cleanly by combining masked feature prediction with an MMD distributional pin. Additionally, the evaluation is thorough and fair, reimplementing multiple baselines under strict matched-geometry protocols and providing insightful ablations that decouple content retention from manifold alignment.

Critical weaknesses

Despite claiming zero-shot transferability as its main contribution, the method frequently underperforms training-free baselines like ToMe and VisionZip when transferred to LLaVA-1.5-7B, and suffers a severe collapse on fine-grained OCR tasks such as TextVQA. Furthermore, the claim of complete host-independence is contradicted by the experimental setup's reliance on SmolLM2-135M as a research host for training and tuning, and the paper omits foundational references for its core architectural components such as MAE and Perceiver.

Questions for authors
  1. Upon transferring to LLaVA-1.5-7B, the method frequently underperforms training-free baselines. At K=144 (Table 3), it trails ToMe or random selection on POPE, GQA, MMBench, and MME (Δ from -0.66 to -1.81). At K=288 (Table 2), it also lags behind CLS top-k and VisionZip across VQAv2, OK-VQA, and COCO.
  2. Since zero-cost, training-free heuristics like ToMe and VisionZip achieve comparable or superior results on the 7B model, the practical necessity of pre-training this compressor is questionable.
  3. On TextVQA (Table 3), Grid-pool (K=144) plummets from 49.30 to 22.16—23.45 points below the best baseline and far behind CLS top-k (37.10) and VisionZip (35.44). The 2×2 spatial averaging blurs local high-frequency details, exposing a critical flaw in text-dense scenarios.
  4. The Attention-pool variant is omitted from the standard LLaVA-1.5-7B benchmarks in Table 3. Please provide these results and clarify whether Attention-pool mitigates the severe degradation on TextVQA.
  5. Section 3 claims complete independence from language models during training, yet Sections 4.1 and 5.1 state that SmolLM2-135M is used "to train the proposed compressor and transfer it." Please clarify if SmolLM2-135M validation metrics were used for hyperparameter tuning or checkpoint selection, which would imply implicit host/task supervision.
3 comments

We thank the reviewer ZA3w for the careful review.

Question 1)

Reply. We agree, and we will state in the abstract that training-free baselines are at par with or slightly ahead of our compressor on LLaVA-1.5-7B. Our claim is transfer with comparable performance, not a lead. The compressor never saw this host and is inserted unchanged, yet in the cells the reviewer cites it trails the strongest baseline by 0.55–1.81 points, except on VQAv2 at K=288K=288 (−3.33). It leads on SEED (+1.84, Table 3) and on five of the six in-context tasks at K=144K=144 (Table 2), and it keeps 94–99% of the uncompressed host on every benchmark of Table 3 except TextVQA (Questions 3 and 4). Its mean retention on GQA, MMBench and POPE at K=144K=144 (95.53%) is within one point of two recent compressors at 128 tokens, their nearest published budget: VisionZip (95.25%) and EvoComp (96.35%), which is trained for each host with labelled data (Table R1).

Table R1. LLaVA-1.5-7B, zero-shot, retention (%) against the uncompressed host of each evaluator. Published rows are from the VisionZip and EvoComp papers; our row uses our evaluator (Table 3 of the submission). Mean is over GQA, MMBench and POPE.

MethodTraining signalKKGQAMMBenchPOPEMeanVQAv2SEED
VisionZip (CVPR 2025)none (CLS attention)12893.0595.8396.8695.2596.3193.69
VisionZip (CVPR 2025)none (CLS attention)19295.8097.3799.3097.4997.8396.25
EvoComp (CVPR 2026)host loss, labelled data12894.6797.5396.8696.3595.92–
EvoComp (CVPR 2026)host loss, labelled data19296.4597.9998.9597.8097.58–
Ours, gridlabel-free, no decoder14493.8894.0998.6395.5393.7694.54

Question 2) "Since zero-cost, training-free heuristics like ToMe and VisionZip achieve comparable or superior results on the 7B model, the practical necessity of pre-training this compressor is questionable."

Reply. Pre-training is what makes the compressor reliable across hosts: no training-free heuristic is strong on every host, whereas our compressor has the highest retention of the four methods on every host outside TextVQA, at the same inference cost (Table R2).

On LLaVA-1.5-7B at K=144K=144, ToMe matches our compressor (94.7% against 94.9% mean retention without TextVQA), and we will state this in the abstract. VisionZip does not: we lead it on seven of the nine tasks. On other hosts the ranking changes. ToMe keeps 50% on the 135M decoder and VisionZip 64% on CLIP at K=49K=49, against a worst case of 84.9% for ours. Choosing a heuristic for a new host therefore requires a labelled evaluation on that host; our compressor is trained once per encoder, without labels. The heuristics are also not cheaper at inference: 31.1% of the uncompressed GFLOPs for ours and VisionZip, 31.6% for ToMe. Finally, our advantage grows as the budget shrinks: +13.44 CIDEr over the strongest competitor at K=72K=72 on the research host (Figure 1), and +12.6 retention points over CLS top-kk at K=36K=36 on the 7B (Table 6).

Table R2. Retention (% of the uncompressed host) at matched token count, on every host in the paper. 7B: mean over the nine tasks of Tables 2–3 except TextVQA. *Below ToMe, the strongest competitor in this cell (Table 2). †Grid / attention-pool.

HostTaskKKToMeToMe 1-shotVisionZipOurs
LLaVA-1.5-7B8 tasks, mean14494.790.991.494.9
BLIP-2 host (Q-Former)COCO CIDEr14491.6<91.6*<91.6*96.6
Research host (135M)COCO CIDEr14450.373.674.084.9
CLIP ViT-B/16Flickr30k rˉ\bar{r}4993.373.863.894.1
CLIP ViT-B/16Flickr30k rˉ\bar{r}9898.696.597.699.5
Worst case over the five rows above50.373.663.884.9
LLaVA-1.5-7BTextVQA14467.634.971.945.0 / 66.1†

Question 3)

Reply. We agree. Uniform 2×2 averaging discards the fine detail that reading text requires, and we have listed this as a limitation of the grid variant (Section 7). The attention-pool variant, whose output tokens are selected patches plus a learned offset, recovers about 10 of the 27 lost points (Question 4).

Question 4)

Reply. We thank the reviewer. Table R3 gives the requested results at the budget of Table 3 (K=144K=144, zero-shot, no host adaptation). The attention-pool variant is within 0.8 points of the grid on POPE, GQA, MMBench and MME, and 1.73 points below it on SEED. It mitigates the TextVQA degradation but does not remove it: accuracy rises from 22.16 to 32.58 (retention from 45% to 66%), on par with ToMe (33.33) but still below CLS top-kk (37.10). We will add the attention-pool row to Table 3.

Table R3. LLaVA-1.5-7B, zero-shot, K=144K=144, no host adaptation. All rows except attention-pool are from Table 3 of the submission.

ModePOPE accPOPE F1GQAMMBenchMMESEEDTextVQA
Uncompressed ceiling86.3784.8162.4469.5178.8360.7949.30
ToMe86.2685.1459.5665.8576.4155.0233.33
CLS top-kk84.4882.6356.3465.6573.5954.7737.10
Ours, grid85.1983.3358.6265.4075.5057.4722.16
Ours, attention-pool85.1283.2558.0165.8574.7255.7432.58

Question 5)

Reply. SmolLM2-135M metrics were not used for checkpoint selection or for setting λ\lambda or wpredw_{\text{pred}}. The checkpoint is the minimiser of the held-out compressor loss on a 5% image split, computed in the vision tower with no decoder, and λ\lambda and wpredw_{\text{pred}} were fixed on the same split. The phrase "which we use to train the proposed compressor" in Section 4.1 is therefore inaccurate. We will correct it in the revision.

Missing references. We will cite MAE (He et al., 2022) for masked prediction with a lightweight decoder, and Perceiver, Perceiver IO (Jaegle et al., 2021; 2022) and the Perceiver Resampler of Flamingo (Alayrac et al., 2022) for cross-attention pooling into a small set of tokens, and state in Section 3.2 how our modules differ from them.

Thank you for the candid response. While your answers address my concerns about experimental clarity, I will maintain my rating of Weak Rejection for two main reasons: (1) pairing a JEPA-style predictive loss with an MMD regularizer over standard pooling remains a relatively conventional combination of existing components, lacking novelty; and (2) on the primary deployment host, incurring additional pre-training compute only achieves performance comparable to ToMe, which limits the practical motivation of the proposed approach.

Reviewer_SZDPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Learn and Pin, a label-free, JEPA-style visual token compression framework for frozen vision-language models (VLMs). The method combines masked feature prediction to preserve visual information with MMD-based distribution alignment to maintain compatibility with frozen downstream models. The trained compressor can be transferred across different VLM hosts without further adaptation, provided they share the same vision encoder. Experiments on LLaVA, BLIP-2 variants, and CLIP demonstrate substantial token reduction while retaining competitive performance across multiple vision-language tasks.

Critical strengths
  1. Label-free training provides a useful way to reuse a compressor across frozen downstream hosts sharing one visual representation. Sections 4.1 and 5.1 report unchanged transfer from the research setup to LLaVA-1.5-7B and a controlled Q-Former bridge variant. At grid K=144, Table 2 reports 96.51 CIDEr on the Q-Former host, retaining 96.6% of its uncompressed score and exceeding the strongest competitor by 5.02.

  2. The evaluation includes useful controls for the effects of training and the two objective terms. Table 5 compares learned refinement with parameter-free pooling at matched geometry, and Section 6.3, Figure 2, and Appendix B compare predictive-only, pin-only, and joint training across token budgets. Appendix B.2 further reports recovery of small-host captioning after a pin-only coordinate repair.

Critical weaknesses
  1. Limited methodological novelty. Evidence: Section 3.2 adopts conventional token selection, pooling, and attention mechanisms. Eqs. (6)–(8) combine established JEPA-style feature prediction with MMD-based distribution alignment. Table 6 further shows that pin-only training outperforms the joint objective at K=144 on ICL-0 (100.3 vs. 99.1 CIDEr).

  2. Limited transferability and insufficient performance gains. Evidence: Sections 4.1 and 5.1 only demonstrate transfer across downstream models sharing the same CLIP vision encoder. The CLIP ViT-B/16 experiment requires retraining. Evaluation relies on relatively outdated VLMs, and Table 3 shows improvements over the strongest token-matched baseline only on SEED. TextVQA drops from 49.30 to 22.16 after compression.

Questions for authors

(A) Clarifications that could change my assessment

  1. Does the claimed transferability extend to different vision encoders? The current experiments only demonstrate transfer across downstream models sharing the same vision encoder, which is not fully consistent with the claims in the Introduction. Demonstrating transfer across different vision encoders without retraining, particularly on more recent VLMs, would significantly strengthen the contribution and could improve my rating.
  2. Do the reported performance gains still hold after excluding overlapping training and evaluation images? Appendix A reports that some evaluation images in VQAv2, OK-VQA, and POPE overlap with the compressor's pretraining images, with approximately 75% overlap for POPE. Re-evaluating the method on overlap-free test sets and reporting appropriate paired uncertainty estimates would strengthen the reliability of the experimental conclusions and could lead me to raise my rating.

(B) Suggestions / future work that do not independently affect this rating Investigate spatially adaptive compression or retention of high-resolution patches for text reading, given the failure analysis in Section 7.

2 comments

We thank the reviewer SZDP for the comments.

The goal of this paper is a transferable, self-supervised compressor that is trained once and inserted into a frozen host VLM, so the host never has to be retrained. Our aim is performance comparable to the uncompressed host and to existing token-compression methods, with little or no degradation, not necessarily to outperform trained compressors. The saving comes from transfer: one training run serves every host that shares the same vision encoder. The compressor is trained once in the vision tower, with no labels and no decoder, and is then inserted unchanged into a 135M decoder, a Q-Former bridge, LLaVA-1.5-7B and LLaVA-1.6-Vicuna-7B. While the individual components of our method are conventional, we claim novelty in combining them to address this problem.

We acknowledge that our wording overstated the scope of this transferability. Our experiments cover hosts built on CLIP-based vision transformers, not all vision encoders. CLIP-based VLMs are widely used in research and practice, but we will revise the abstract and Introduction to state this scope precisely, and we plan to extend the approach to other encoder architectures in future work.

Comparable performance. Table R1 compares our grid compressor with two recent compressors on LLaVA-1.5-7B. Individual benchmarks move in both directions: against VisionZip at 128 tokens we lead on GQA (+0.83) and POPE (+1.77) and trail on MMBench (−1.74); against EvoComp we lead on POPE (+1.77) and trail on GQA (−0.79) and MMBench (−3.44). The mean over the three benchmarks gives the overall picture. At K=144K=144, between the two published budgets, our compressor retains 95.53%: +0.28 above VisionZip and −0.82 below EvoComp at 128 tokens, within one point of both. EvoComp, however, is trained for each host with labelled data, whereas our compressor uses no labels, never sees LLaVA-1.5-7B, and is inserted unchanged. It therefore performs comparably to the state of the art without any training in the host. We will add this table to the paper.

Table R1. LLaVA-1.5-7B, zero-shot, retention (%) against the uncompressed host of each evaluator. Published rows are from the VisionZip and EvoComp papers; our row uses our evaluator (Table 3 of the submission). Mean: is the mean of GQA, MMBench and POPE.

MethodTraining signalKKGQAMMBenchPOPEMeanVQAv2SEED
VisionZip (CVPR 2025)none (CLS attention)12893.0595.8396.8695.2596.3193.69
VisionZip (CVPR 2025)none (CLS attention)19295.8097.3799.3097.4997.8396.25
EvoComp (CVPR 2026)host loss, labelled data12894.6797.5396.8696.3595.92–
EvoComp (CVPR 2026)host loss, labelled data19296.4597.9998.9597.8097.58–
Ours, gridlabel-free, no decoder14493.8894.0998.6395.5393.7694.54

Weakness 1) "Table 6 further shows that pin-only training outperforms the joint objective at K=144 on ICL-0 (100.3 vs. 99.1 CIDEr)."

Reply. The benefit of the joint objective is transfer across tasks. Neither single-term model transfers: each wins one task family and loses the other by a wide margin, whereas pred + pin holds up across captioning, few-shot classification and a host it never saw. Table 6 is an ablation study: its purpose is to isolate what each term contributes, not to show that the joint objective leads on every cell, so a single captioning cell cannot show this benefit.

Each term alone serves one task family. The pin serves tasks that read the tokens through a decoder: pin-only reaches 95.2 / 66.3 / 25.5 ICL-0 CIDEr at KK = 144 / 72 / 36, where pred-only collapses to 40.0 / 26.0 / 4.7. The predictive term serves tasks that use token content directly: on 5-shot E-600, pred-only retains 110.7 / 106.9 / 82.6% at the same budgets, against 101.8 / 88.9 / 60.4% for pin-only.

The joint objective transfers across both. On the attention-pool ladder, pred + pin keeps pin-only's captioning (94.4 vs. 95.2 at K=144K=144) and exceeds it as the budget tightens (68.8 vs. 66.3 at K=72K=72; 34.5 vs. 25.5 at K=36K=36). It also recovers part of the content that pin-only lacks, leading pin-only on E-600 retention at every budget (106.4 vs. 104.5, 103.0 vs. 101.8, 93.3 vs. 88.9, 68.9 vs. 60.4). On the unseen 7B host it matches pin-only at K≥144K \geq 144 (within 0.5 points) and leads at K=72K=72 and 36 (82.5 vs. 81.7; 70.9 vs. 67.3).

Weakness 2)

Reply. We agree that Table 3 shows a lead over the strongest baseline only on SEED. As stated above, our claim is comparable performance after transfer, not a lead, and Table R1 shows mean retention within one point of recent compressors. We evaluate on LLaVA-1.5-7B because it is the standard host for these compressors (VisionZip, CVPR 2025; EvoComp, CVPR 2026), so our numbers compare directly with theirs. The TextVQA drop in Table 3 comes from the 2×2 grid averaging; the attention-pool variant scores 32.58 on TextVQA, on par with ToMe (33.33).

A newer host, and training in the host. Table R2 shows transferability to a newer CLIP-based host, LLaVA-1.6-Vicuna-7B, and compares our transferred compressors with the same architectures trained inside that host by next-token prediction. With architecture, budget and host held fixed, our compressors, which never saw this host, retain more on seven of eight benchmarks and raise mean retention by +4.94 (grid) and +8.85 (attention pool) points. Training in the host is therefore not only more costly but also less effective here. We will add this table to the paper.

Table R2. LLaVA-1.6-Vicuna-7B (frozen, one 577-token view), K=144K=144, zero-shot. Next-token in host: the same architecture, trained inside the frozen host with its next-token loss on COCO captions. Mean ret.: mean retention over the eight benchmarks.

ModeVQAv2OK-VQATextVQAPOPE accGQAMMEMMBenchSEEDMean ret.
Uncompressed ceiling77.8157.9254.0085.0863.6078.2772.2165.73100.00
Grid, transferred (ours)71.2254.3620.1377.3459.5675.6367.4860.6386.19
Grid, next-token in host65.9945.8616.9982.8356.6268.7965.4558.9081.25
Δ transferred − next-token+5.23+8.50+3.14−5.49+2.94+6.84+2.03+1.73+4.94
Attention-pool, transferred (ours)70.1353.5633.2977.5957.6775.8968.5059.4688.55
Attention-pool, next-token in host62.6336.5632.1183.0852.7467.9762.5052.9879.70
Δ transferred − next-token+7.50+17.00+1.18−5.49+4.93+7.92+6.00+6.48+8.85

Question 1)

Reply. No. As stated above, our experiments show transfer across hosts built on CLIP-based vision transformers, not across encoders, and we will narrow the claim in the abstract and Introduction accordingly. The limit follows from the method: the pin matches the compressed tokens to the frozen encoder's own token distribution, which is what a host built on that encoder reads. A host on a different encoder expects a different distribution, so the compressor would be retrained for that encoder, still without labels or a decoder.

Question 2)

Reply. Yes. We re-scored our compressor and all baselines on the VQAv2, OK-VQA and POPE images that do not appear in the compressor's pretraining set, with 95% paired bootstrap intervals over images (Table R3). The conclusions are unchanged: we remain behind ToMe on VQAv2 0-shot and POPE, and ahead of VisionZip on OK-VQA 0-shot. No difference between the seen and overlap-free margins is significant, so we find no evidence that overlap inflated our results. All other headline results (COCO captioning, CUB, Aircraft, Flickr30k, TextVQA) were already overlap-free. We will report the overlap-free numbers and intervals in the revision.

Table R3. Margin over the strongest published method on LLaVA-1.5-7B, 0-shot. Negative means the published method is ahead.

BenchmarkPublished methodFull poolOverlap-free95% interval
VQAv2ToMe−0.87−1.49[−2.24, −0.72]
OK-VQAVisionZip+1.55+2.28[0.24, 4.34]
POPEToMe−1.07−4.53[−6.69, −2.54]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.