acceptodds
Withdrawn from ICLR 2027

Tokenizing the Unseen: Compact Visual Prefix Cache for Overcoming VLM Input Bandwidth

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained visual understanding faces an input-bandwidth tradeoff: low-resolution inputs discard small details such as text rendered at a small size, while high-resolution inputs occupy hundreds of decoder positions, a burden that compounds when the same image must answer many questions. Existing approaches each pay a different cost: agentic methods search, zoom, or crop for every query, so latency grows with the number of questions; raising the input resolution inflates the visual sequence; and token-reduction methods lose much of their accuracy under aggressive compression. We propose a compact visual prefix that decouples a high-resolution branch from the visual input budget of a frozen language decoder. This branch reads the image at full resolution and compresses it into the prefix, so the decoder needs only a fixed number of visual positions, while the pretrained backbone and its native path are left unchanged. Computed from the image alone, the prefix can be cached once and reused across questions. Under a budget of 24 decoder positions, the learned prefix reaches 0.386 TextOCR word F1 against 0.154 for the strongest of five tested token reducers. On DocVQA, a single prefix per page serves all 4.68 questions on average. These results show that a compact visual prefix can extract and compress useful information from a high-resolution image, and that the result is reusable across questions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, high-resolution perception, visual token compression, soft prefix tokens, KV cache, OCR, scene text, frozen decoder, input bandwidth, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZZMzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper explore the visual input problem in vision-language models (VLMs), where low-resolution inputs may miss details such as text, while high-resolution inputs inflate the decoder's sequence length and compute.

To address this, the authors propose a compact visual prefix adapter that reads dense high-resolution features from a frozen vision encoder and compresses them into a small, fixed number of continuous prefix tokens, optionally paired with a low-resolution image. The paper formalizes sufficient conditions for cross-question causal KV-cache reuse and a frozen-decoder population log-loss decomposition, and evaluates the method across several benchmarks.

Critical strengths
  1. The paper addresses a practical challenge in deploying high-resolution VLMs under strict token budgets and persistent KV-cache storage constraints, demonstrating that learned query-agnostic prefixes can substantially outperform training-free token pruning and merging methods under extreme compression.

  2. The authors maintain commendable transparency in their appendices by providing thorough diagnostic audits--including white-image and wrong-image content controls, decoding repetition-penalty audits, and explicit disclosures of negative results and historical architectural discrepancies--which help clarify when the learned prefix relies on genuine visual evidence versus task-level format adaptation.

  3. Within matched task formulations such as scene-text transcription, the image-only adapter demonstrates encouraging zero-shot domain transfer to unseen benchmarks (improving ICDAR15 word F1 from 0.134 to 0.424 and TotalText from 0.467 to 0.546) and consistent gains across multiple VLM backbones (Qwen3-VL-4B, Qwen3-VL-8B, and GLM-4.1V-9B).

Critical weaknesses
  1. The paper reads as heavily generated. The text contain LLM watermarks and detected as generated, the figures are heavily generated and references duplicated (for example: V* and LORA appear twice).

  2. The paper's framing risks conflating cross-question KV-cache reuse with visual sequence compression, and omits direct empirical comparisons against standard full-image KV prefix caching (which already reuses native visual KV tensors across questions without recomputation in engines like vLLM and SGLang), offline visual token caching frameworks, and training-free post-prefill multimodal KV-cache compressors that achieve large memory reductions without severe accuracy collapse ("SGLang: Efficient Execution of Structured Language Model Programs", 2024; "Efficient Memory Management for Large Language Model Serving with PagedAttention", 2023; "Compress & Cache: Vision token compression for efficient generation and retrieval", 2025; "LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference", 2024; "VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration", 2025).

  3. Because the proposed prefix adapter introduces 92.5M to 251.1M trainable parameters trained with in-domain supervision on high-resolution features, comparisons against frozen VLMs, a 10x smaller projector atop statically pre-pruned 1-D VisionZip tokens, and a 12M-parameter LoRA SFT baseline restricted to physically illegible low-resolution thumbnails (~2.16px text height) are confounded; the evaluation would benefit from capacity- and resolution-matched LoRA/PEFT controls and established learned coarse-to-fine projectors (TokenPacker: Efficient Visual Projector for Multimodal LLM, 2025; Honeybee: Locality-Enhanced Projector for Multimodal LLM, 2024).

  4. The learned prefix exhibits severe out-of-distribution (OOD) task overfitting and instruction-following collapse on unseen VQA tasks—scoring 0.000 ANLS on zero-shot DocVQA and 0.010–0.047 generation accuracy on VBench, and still underperforming the untrained frozen low-resolution native baseline on VBench generation even after mixing in GQA supervision.

  5. The paper contrasts its method against heavy multi-step tree search (ZoomEye) but omits discussion and empirical comparison with lightweight, single-pass context-aware resolution selectors and adaptive cropping methods for frozen VLMs, which dynamically select minimal sufficient resolutions or crops per query without multi-call overhead or OOD format degradation (CARES: Context-Aware Resolution Selector for VLMs, 2026).

  6. Discrepancy between formulated 3-stage adapter and evaluated legacy 1-stage checkpoints (Page 4)

"The queries first read the dense visual features, which then attend to the updated queries. A final query readout carries the refined visual evidence into z."

Section 3.2 and Figure 1(b) present a three-stage bidirectional cross-attention adapter as the unified prefix construction methodology. However, the architectural audit in Appendix D reveals that all "single-block" ("legacy") blind adapters--which serve as the basis for the multi-backbone generalization, zero-shot scene-text transfer, legacy budget-allocation, image-content controls, and reusable DocVQA cache experiments across Sections 4.3, 4.4, 5, and Tables 10–12--return the initial query branch P1P_1 directly before performing any readout from the updated visual branch V1V_1. Consequently, in all of those experiments, the visual refinement stage (V1V_1) and second readout stage (P2P_2) are disconnected from the training loss and receive zero gradients, leaving 158,612,480 unused parameters out of 251,089,920 registered parameters and reducing the active module to a 92,477,440-parameter one-way query resampler.

Furthermore, as noted in Appendix D, while the primary budget experiments (Section 4.2, Table 1, and Appendix B) execute the full three-stage readout in Equation 4, no accuracy-matched one-way resampler or original-vision-readout control accompanies that budget curve. Conversely, on the legacy cohort where a one-way resampler control was tested, both the "One-way learned-query resampler" (92M, 0.305 F1) and "Legacy blind" (251M registered / 92.5M active, 0.313 F1) execute the exact same active one-way forward mapping (P1P_1), differing only in parameter registration, and comparing the pure 24-token prefix across cohorts shows nearly identical F1 scores. So, the empirical necessity of the visual-refinement and second-readout stages (V1V_1 and P2P_2, adding 158.6M parameters) is never isolated.

Questions for authors

In the multi-question DocVQA evaluation and systems microbenchmarks, the 12-token cached prefix suffers a steep accuracy drop compared to full-resolution input, as well as somehow got the highest Collision rate. How would the authors explain that?

Reviewer_JbWTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a compact visual prefix framework for improving fine-grained visual understanding under limited decoder visual-token budgets. The method uses a frozen high-resolution vision encoder and a trainable learned-query adapter to compress dense visual features into a small number of continuous prefix tokens, while keeping the language decoder frozen. By placing the image-dependent prefix before question-dependent inputs, the framework also enables KV-cache reuse across multiple questions about the same image. Experiments on OCR, visual question answering, and document understanding benchmarks show substantial improvements over several token-reduction baselines under constrained visual budgets. While the motivation and empirical results on text-oriented tasks are encouraging, my main concerns relate to the architectural justification, generalization beyond OCR, practical efficiency, and the contribution of the theoretical analysis.

Critical strengths
  1. The paper addresses a relevant problem in high-resolution visual understanding, where preserving fine-grained information often requires a large number of visual tokens. Decoupling high-resolution visual encoding from the decoder's input budget is a reasonable approach, particularly for applications involving repeated queries over the same image.

  2. The proposed framework combines learned visual compression with question-independent prefix caching. Placing the visual prefix before question-dependent inputs provides a clear mechanism for KV-cache reuse, and the paper explicitly discusses the conditions under which this reuse is valid.

  3. The paper provides extensive experiments across different visual-token budgets, datasets, and model backbones, along with several ablation studies and image-content controls.

Critical weaknesses
  1. The paper emphasizes reducing decoder visual bandwidth and enabling cross-question KV-cache reuse, but the current systems evaluation does not fully establish the practical efficiency benefits. In Appendix F.1, the online prefix incurs higher prefill latency (90.5 ms vs. 82.4 ms) and peak GPU memory (10.108 GB vs. 9.645 GB) than the full-resolution baseline. Although the separate cache-reuse experiment reports reduced query latency, it is conducted on only 12 images with eight questions each, and the results for different numbers of questions are estimated from measured components rather than directly benchmarked. It remains unclear under what image resolutions and query frequencies the cost of high-resolution encoding and prefix construction can be amortized.

  2. While the proposed method achieves promising results on OCR and scene-text recognition tasks, its generalization to broader visual understanding remains insufficiently demonstrated. In Sections 4.3–4.5 and Appendices E and K, the OCR-trained prefix shows limited improvements on the standard V*Bench benchmark and even underperforms native visual inputs in some configurations. Moreover, Appendix E reports a negative result on GQA small-object understanding, where the trained prefix achieves 0.564 compared with 0.586 for the approximately matched frozen baseline. These findings suggest that the learned representation may be particularly effective at preserving task-specific textual information rather than general fine-grained visual evidence. Although the authors explore mixed OCR and QA supervision, the resulting improvements do not fully resolve concerns about generalization across different visual reasoning tasks.

  3. The theoretical analysis in Sections 3.3–3.4 and Appendix J is mathematically reasonable, but its connection to the specific architectural design appears somewhat limited. Proposition 1 primarily formalizes the question-independent KV-cache reuse property under causal attention, while Proposition 2 specializes a standard log-loss decomposition to distinguish information loss from frozen-decoder mismatch. Although these results help motivate the overall framework, they do not directly explain why the proposed alternating cross-attention adapter should outperform simpler learned compression architectures or which components are essential to its effectiveness. Given that the theoretical analysis is presented as one of the paper's contributions, I would appreciate further clarification on its novelty relative to existing work on prefix caching and information bottleneck formulations, as well as how these theoretical insights specifically motivate or justify the proposed architectural choices.

  4. I find some of the experimental descriptions in Appendix F.1 difficult to interpret and verify. For example, the authors state that the original report claims fp32 logit agreement within 1.1×10−41.1 \times 10^{-4}, while the recovered JSON only verifies 91/96 bf16 greedy-output matches, without establishing the fp32 claim or explaining the remaining discrepancies. The appendix also notes that several configuration details are absent from the raw records and that no hardware benchmarks were rerun during the revision. These descriptions read more like a retrospective summary of archived experimental records than a clearly documented and reproducible evaluation protocol.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_dh3SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a compact visual prefix for reducing the visual input bandwidth of frozen VLMs. The method introduces a learned-query adapter compresses high-resolution visual features into a small number of prefix tokens, while keeping the vision encoder and language decoder frozen. By placing the image-only prefix before question-dependent inputs, its decoder KV cache can be reused across multiple questions.

Critical strengths
  1. The paper shows strong performance under limited visual token budgets. Table 1 shows that the learned prefix achieves 0.386 TextOCR word F1 with only 24 visual tokens, significantly outperforming the strongest tested training-free reducer.

  2. The KV cache reuse mechanism is a practically valuable design. The proposed method compute the visual prefix once per image and caching KV states across all decoder layers, achieving linear-speedup in multi-question scenarios without re-running the vision encoder. It's highly relevant for real-world VLM deployment.

Critical weaknesses
  1. The learned-query adapter is closely related to existing visual bottleneck architectures, such as Perceiver Resampler and BLIP-2's Q-Former, but direct comparisons are missing. Although the authors include an OCR-supervised VisionZip projector, its parameter count is substantially smaller than that of the proposed adapter.

  2. The main compression experiments rely on TextOCR supervision. Although the authors further explore OCR+GQA and multi-task supervision in Tables 3 and 4, these results suggest that the learned representation is highly dependent on the training objectives. Evaluating prefixes trained with broader visual-language objectives on unseen, non-text-centric benchmarks would help establish the generalizability of the proposed method.

Questions for authors
  1. Could the authors compare the proposed adapter with capacity-matched learned-query compressors, such as a one-way resampler or a Perceiver-style adapter, under the same visual token budgets.

  2. Could the authors evaluate whether prefixes trained on general VLM alignment data? This would help clarify whether the proposed method can learn general-purpose visual representations beyond OCR-centric tasks.

  3. Could the authors provide end-to-end latency and memory comparisons for different numbers of questions per image?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.