Learn and Pin: A Transferable JEPA-Style Visual Token Compressor for Frozen VLM Hosts

In vision–language models, the image dominates the input: LLaVA-1.5 renders one image as $576$ tokens, far more than a typical prompt, so most image compute is spent inside the language model. Existing visual token compressors either hand-specify the pre-designed method or learn it with task supervision tied to one host. None learns label-free model inside a frozen encoder and transfers it, unmodified, to a new host. In this paper, we train a small compressor inside a frozen vision encoder without captions. It uses a JEPA-style predictive term and a distributional pin. The pin is a kernel two-sample statistic that keeps the distribution of the compressed tokens close to the encoder's own. The compressor then runs frozen in the encoder's forward pass and is evaluated zero-shot. We do not quote published numbers. We re-run all competitors, both published methods and training-free anchors, in one setting. They are matched on token count, or on total cost for decoder-side pruners. At $K=144$ (a quarter of the tokens) on the LLaVA + SmolLM2-135M parameter host, the proposed method leads the field by $+9.02\pm 1.23$ CIDEr zero-shot and $+11.08\pm 2.76$ one-shot at $43.3\%$ of the uncompressed analytic cost. The compressor transfers unchanged into LLaVA-1.5-7B. It retains $92$–$97\%$ of the uncompressed ceiling on five of six in-context tasks and $94$–$99\%$ on five standard benchmarks, at $31\%$ of the analytic cost. It also transfers to an unseen Q-Former bridge at $96.6\%$ retention, $+5.02$ over the strongest competitor. On CLIP retrieval, it retains $99.5\%$ of recall at $K=98$ and leads by $+0.72$.