acceptodds
Under review as a conference paper at ICLR 2027

READ: Query-Relevant Evidence Retrieval for Visual Token Compression in Edge–Cloud VLM Inference

Abstract

Multimodal foundation models enable edge-cloud collaborative intelligence by transmitting visual tokens from edge devices to a cloud vision-language model for downstream reasoning. However, uploading dense token sequences incurs substantial communication overhead and cloud-side computation and memory costs. Vision-only selectors operate before transmission but cannot adapt to specific user queries. Conversely, query-aware methods typically require executing heavy language or multimodal networks on resource-constrained edge devices, incurring substantial local overhead. We present READ (Retrieving Evidence via Attention Distillation), a framework that reformulates pre-transmission token selection as an evidence retrieval task. During offline training, READ leverages prompt-to-visual attention from the frozen cloud model to supervise a compact cross-modal retriever that selects query-relevant visual tokens for accurate and efficient collaborative inference. By jointly training the retriever with prefix-ranking and score-calibration objectives, READ produces well-calibrated relevance scores that enable edge devices to identify task-critical visual tokens for efficient collaborative inference without running large models locally. Experiments demonstrate that READ achieves a competitive efficiency-accuracy trade-off across diverse architectures and transmission budgets, effectively preserving reasoning performance even under aggressive compression with minimal edge overhead.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Visual token compression, Query-aware token selection, Edge-cloud collaborative inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vhcWOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual token selection for edge–cloud VLM inference, where an edge device encodes an image and uploads visual tokens to a cloud VLM. READ caches the cloud VLM's prompt-to-visual attention offline and distills it into a small cross-modal retriever that runs on the device. The retriever combines a visual salience prior with the prompt embeddings, is trained with a prefix-ranking loss over log-uniformly sampled budgets plus a rank-survival calibration loss, and selects K tokens with relevance-weighted coverage before upload. Experiments on LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and InternVL2.5 show better accuracy retention than most baselines, especially at tight budgets.

Critical strengths
  1. The problem is practical and the gap is clearly identified. Vision-only selectors can run before transmission but ignore the question, while query-aware methods either prune inside the cloud model after everything has been uploaded or need a heavy model on the device. Asking for query-aware selection before transmission, without running a language model on the device, is a reasonable and useful target.
  2. Transferring the cloud VLM's token preference into a small model is a clean way to save cost. Instead of running a proxy VLM on the device, READ caches the target VLM's attention once offline and distills which visual tokens it prefers into a compact retriever. This keeps the selection aligned with the model that will actually answer, while cutting the selector from billions of parameters to tens of millions and more than halving the edge latency compared with SGP. The training objective also lets a single checkpoint serve any token budget.
  3. The experiments are extensive and contain some interesting findings. The method is evaluated on four VLM families, including dynamic-resolution models, on up to ten benchmarks and several budgets, and the advantage over baselines grows as the budget shrinks. The analyses of the teacher signal are also interesting, for example that attention from the last prompt token works better for direct selection but worse as a distillation target than attention aggregated over the whole prompt.
Critical weaknesses
  1. The edge–cloud setting is weakly motivated. Without the deployment story, READ is essentially a learned visual token selector, and the edge–cloud framing is what makes selection before transmission necessary. However, the paper measures communication only in token counts (§5) and never in bytes. Projected visual tokens are high-dimensional, so even the reduced set may be larger than a compressed image. A simpler design, uploading the compressed image and running the selection in the cloud, would then give a smaller uplink and the same prefill savings. The paper neither discusses this option nor gives a reason, such as privacy, why the image cannot leave the device.
  2. The query-aware ability claimed in the title and abstract is not directly verified. No experiment isolates how much the selection actually depends on the question. The comparison with query-agnostic baselines does not settle this, because READ also differs from them in training data, the query-independent visual prior, and the coverage step, and the prior alone carries a large part of the result. The gains are also not concentrated on the text-heavy tasks that the motivation emphasizes, and the only direct evidence is a single qualitative example.
  3. The technical novelty is limited, and it is unclear how much of the gain comes from distillation. Each component is close to existing work: the attention signal is what FastV and SGP already use, and relevance-weighted diversity selection is close to DivPrune, VisPruner, and IDPruner. The reformulation as evidence retrieval is mostly a new name for budgeted top-K selection. The paper's own results also suggest that a substantial part of the improvement comes from the components that are not distilled, since the distilled retriever outperforms direct selection with the teacher attention it imitates, and removing the prior or the coverage step causes large drops.
Questions for authors
  1. Can you report the uplink payload in bytes rather than tokens, stating which features are transmitted and at what dimension and precision, and compare it with uploading a compressed image and pruning in the cloud?
  2. Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?
  3. What are the edge memory and latency once the LLM word-embedding table needed for the prompt embeddings is included, and on what hardware were the edge numbers in Table 4 measured?
  4. Since every cloud VLM needs its own teacher cache and retriever, what does it cost to support a new or updated cloud model, and does a retriever trained for one VLM transfer to another?

More than this, please see the weaknesses above. If my concerns can be resolved, I would consider raising my score.

13 comments

Q1. Can you report the uplink payload in bytes rather than tokens, stating which features are transmitted and at what dimension and precision, and compare it with uploading a compressed image and pruning in the cloud?

Response:

We thank the reviewer for highlighting this important comparison. Traditional image/video codecs are highly efficient, and uploading a compressed image for cloud-side pruning is a viable alternative, particularly when uplink bandwidth is the dominant constraint. We do not claim that READ uses fewer bytes than JPEG or other conventional codecs, or necessarily reduces total edge-plus-cloud computation. Our focus is edge–cloud computation partitioning: the edge performs vision encoding and query-conditioned feature selection, allowing the cloud to bypass vision encoding and process a shorter LLM prefill sequence. We have incorporated this detailed discussion and interface comparison into Appendix D (Table 5).

For LLaVA-1.5-7B at 336 × 336 resolution, the byte accounting and architectural alternatives are:

Transport OptionUplink PayloadCloud ViTCloud Visual Sequence
Original Image~339 KB (Raw RGB)Required576 tokens
Dense Visual embeddings4.72 MBBypassed576 tokens
Compressed Image + Cloud pruning~16–26 KB (JPEG Compression)Required32 tokens
Selected embeddings262.1 KB (proj.) / 65.5 KB (pre-proj.)Bypassed32 tokens
Compressed Image + Edge selected indexImage + 64 bytes indexRequired32 tokens

Under the evaluated LLaVA-1.5-7B setting (336 × 336 input, bf16 precision, projected hidden dimension D=4096D = 4096), transmitting 32 selected projected visual tokens requires 262.1 KB plus 64 bytes of spatial position indices, reducing the uncompressed 576-token payload (4.72 MB) by 94.4%. If transmitting 1024-dimensional pre-projector features, the payload is further reduced to 65.5 KB, reflecting both token selection and lower transmitted feature dimensions.

This system design connects naturally to the philosophy of Feature Coding for Machines (FCM) and collaborative edge–cloud paradigms such as the Digital Retina [4]. Conventional codecs (e.g., JPEG, AVC/HEVC) are optimized for human perceptual viewing, whereas downstream VLM reasoning operates directly on deep visual features. If raw or compressed images are naively streamed to the cloud, the central server must execute vision encoders for every concurrent query. In high-throughput, multi-tenant serving, this visual encoding creates a severe server-side bottleneck. By shifting visual front-end encoding and query-conditioned evidence filtering to edge devices with local compute, the cloud completely bypasses vision encoding and processes a substantially shortened sequence during prefill.

Crucially, token selection and traditional feature compression are not mutually exclusive: token pruning compresses across the sequence length (token count) dimension, which is fully complementary to subsequent feature quantization and entropy coding (e.g., TOFC [1]). READ addresses a different operating point from conventional image compression: rather than minimizing image transmission bitrate alone, it enables query-conditioned visual evidence selection before transmission, shifting visual processing to the edge and reducing the computational burden of cloud-side multimodal inference. This approach is complementary to feature coding techniques that further compress the selected representations.

Because READ predicts a compact, discrete index set of query-relevant spatial patches, our framework naturally accommodates two complementary dynamic transport options depending on which system bottleneck dominates:

  • Embedding Transmission Mode (Default): In cloud-compute or latency-critical scenarios, the edge directly uploads selected visual token embeddings. This completely bypasses server-side vision encoding, avoiding ViT execution overhead and delivering uncompressed visual evidence directly to the cloud LLM.
  • Index Offload Mode (Bandwidth-Constrained): Under severely bandwidth-constrained links, the edge can select tokens locally and offload a standard compressed image accompanied solely by predicted token indices (requiring merely 64 bytes of index metadata for K=32K=32). The cloud server encodes the image and extracts the indexed tokens for prefill, minimizing uplink payload while still achieving full cloud-side sequence acceleration.

In summary, evaluating communication cost in concrete bytes rather than abstract token counts highlights the practical flexibility of our framework across different system bottlenecks. We thank the reviewer again for raising this thoughtful architectural question, which has helped us significantly clarify the edge–cloud deployment trade-offs and broaden the practical scope of READ.

Q2. Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?

Response:

We sincerely thank the reviewer for proposing these constructive evaluations to assess query conditioning. To address the reviewer's questions, we conducted two comprehensive experiments: first, systematically ablating and perturbing the user question—by completely removing the query, replacing it with a generic prompt, and randomly shuffling questions across images; and second, comparing directly against lightweight on-device query-aware selection via CLIP patch-to-text cosine similarity. In these diagnostic evaluations, query perturbations are applied strictly to the edge retriever's query input to assess visual evidence selection, and the downstream cloud VLM always receives the original user question to generate its response. The query-conditioning ablation is reported in Appendix I.3 (Table 9); the additional CLIP comparison is provided below.

To examine how prompt semantics influence token selection, we evaluated our model under three diagnostic query conditions: removing the question text entirely (w/o Query, retaining the learned visual pathway, visual prior, and coverage but removing question conditioning), replacing the specific question with a fixed, task-agnostic prompt (Generic Query: "Describe the image."), and randomly pairing each image with another question drawn from the same benchmark (Shuffled Query). We benchmarked all conditions across five multimodal benchmarks under three retention budgets (K∈{128,64,32}K \in \{128, 64, 32\}):

BudgetQuery ConditionTextVQAPOPEMMEMMBVizWizAvg.Rel.
K=576K=576LLaVA-1.5-7B58.285.31514.864.350.266.7100.0%
K=128K=128w/o Query56.783.51429.458.551.764.496.6%
Generic Query56.682.41436.458.651.764.296.3%
Shuffled Query57.184.41459.658.551.865.097.5%
Matched Query57.285.91440.562.252.365.998.8%
K=64K=64w/o Query54.677.41360.556.651.561.692.4%
Generic Query54.775.51351.656.751.661.291.8%
Shuffled Query56.278.31361.258.851.962.794.0%
Matched Query56.186.41450.661.353.265.998.8%
K=32K=32w/o Query51.565.91221.852.250.756.384.4%
Generic Query51.667.21228.355.250.157.185.6%
Shuffled Query54.570.71251.656.950.859.188.6%
Matched Query55.386.61348.159.753.664.596.7%

As demonstrated in the table, query conditioning is essential: under aggressive compression (K=32K=32, retaining only 5.6% of tokens), matched queries achieve 96.7% relative retention, outperforming query-free selection by +12.3 percentage points (64.5 vs. 56.3 average). When the query is removed or replaced with a generic prompt, the compressed model still preserves a respectable baseline (retaining 84.4% to 85.6% at K=32K=32) with the learned visual pathway and salience prior still available, together with coverage based on visual-feature diversity. Nevertheless, both conditions perform substantially worse than matched queries because many queries require localizing subtle, question-specific visual evidence—such as verifying tiny physical entities in POPE (where matched queries beat query-free selection by +20.7 points), reading localized text in TextVQA, or discerning fine-grained attributes—which query-independent salience alone cannot anticipate.

Interestingly, shuffled queries achieve intermediate performance (88.6% relative retention at K=32K=32) and surpass the query-free baseline on several benchmarks. One plausible explanation is that within-dataset questions share task-level semantics and formatting, so even a mismatched question can provide useful guidance (e.g., towards scene text in TextVQA). This mechanism has not been isolated experimentally. Matched queries nevertheless achieve higher aggregate performance than shuffled queries at every evaluated budget, although not on every individual benchmark.

Q2. Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?

Response:

To directly answer the reviewer's question regarding how much the selected tokens change across different questions on the same image, we evaluated the token selection divergence between the original Matched Query and each perturbed condition on identical images across all three budgets (K∈{128,64,32}K \in \{128, 64, 32\}). Specifically, we compute the Jaccard Similarity between the selected token sets for each sample:

BudgetPerturbation ConditionTextVQAPOPEMMEMMBVizWizMean JaccardMean Change Ratio (1−J1 - J)
K=128K=128vs. w/o Query64.1%68.4%68.9%70.4%68.4%68.1%31.9%
vs. Generic Query71.1%78.0%80.1%74.7%77.6%76.3%23.7%
vs. Shuffled Query84.7%88.6%90.7%89.6%90.6%88.8%11.2%
K=64K=64vs. w/o Query55.1%61.8%60.6%61.7%61.4%60.1%39.9%
vs. Generic Query58.9%72.9%72.6%66.2%70.1%68.2%31.8%
vs. Shuffled Query80.3%84.5%87.3%85.9%87.4%85.1%14.9%
K=32K=32vs. w/o Query51.0%55.0%54.9%57.5%59.1%55.5%44.5%
vs. Generic Query49.9%66.7%65.6%61.1%64.9%61.6%38.4%
vs. Shuffled Query77.4%81.6%84.5%83.5%85.9%82.6%17.4%

These quantitative similarity measurements provide several key takeaways:

  1. Dynamic Reallocation under Tight Budgets: Under aggressive compression (K=32K=32), completely removing the query causes the Jaccard similarity to drop to 55.5%. This confirms that nearly half of the retained token capacity is actively and dynamically steered by the text question rather than statically dictated by query-agnostic visual priors.
  2. Semantic Sensitivity Continuum: The degree of token divergence strictly tracks semantic specificity across all three budgets: removing queries causes the largest deviation (J=55.5%∼68.1%J = 55.5\% \sim 68.1\%), generic prompts yield intermediate divergence (J=61.6%∼76.3%J = 61.6\% \sim 76.3\%), while shuffled prompts share broad formatting cues and thus maintain higher overlap (J=82.6%∼88.8%J = 82.6\% \sim 88.8\%). Crucially, even with high task formatting overlap, the remaining ~11%--17% query-specific token reallocation is what enables matched queries to consistently outperform shuffled queries across all budgets.
Reviewer_kidKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes READ (Retrieving Evidence via Attention Distillation), a query-aware visual token compression framework for efficient edge–cloud vision-language model (VLM) inference. The key idea is to identify and transmit only query-relevant visual tokens from edge devices to cloud VLMs, reducing communication and computational overhead without requiring large language models to run on the edge. Experiments across four VLM architectures and ten multimodal benchmarks show that READ achieves strong accuracy retention under aggressive token compression. The paper also reports substantial reductions in cloud computation and inference latency.

Critical strengths
  1. The paper addresses the important challenge of reducing communication and computation costs in edge-cloud VLM inference, particularly under limited bandwidth.

  2. The method is well motivated. Distilling cloud-model attention into a lightweight query-aware retriever is a reasonable approach to avoiding expensive on-device VLM inference. The combination of prefix ranking, calibration, and diversity-aware selection also makes the method reasonable.

  3. The evaluation covers multiple VLM architectures, benchmarks, and compression budgets. READ generally performs well under aggressive compression.

Critical weaknesses
  1. The ablation is insufficient. The paper does not isolate the contribution of dynamic prefix-ranking supervision by removing the ranking objective while retaining the other components. It also lacks a quantitative experiment comparing READ with an otherwise identical retriever that does not use query information.

  2. The paper should provide more evidence that validation of attention as a reliable evidence signal. Qualitative analysis is enough.

Questions for authors
  1. Can the authors include an ablation that removes the dynamic prefix-ranking objective while keeping calibration and coverage selection unchanged?

  2. How much performance does READ lose when the query is removed, replaced with a generic prompt, or randomly shuffled across images?

  3. How closely do the teacher's attention-based token rankings align with the visual regions genuinely necessary for correct predictions? Can the authors provide qualitative or quantitative evidence-localization or token-importance evaluations?

5 comments

Q1. Can the authors include an ablation that removes the dynamic prefix-ranking objective while keeping calibration and coverage selection unchanged?

Response:

We sincerely thank the reviewer for suggesting this precise single-factor ablation study, which directly measures how much dynamic prefix-ranking supervision contributes to our framework. To provide a definitive answer, we ablated the dynamic prefix-ranking objective while keeping all other modules completely intact. The complete ablation results across five multimodal benchmarks and three retention budgets are reported in the table below. We have incorporated this single-factor ablation into Table 6 and Appendix E of the revised manuscript.

BudgetMethodTextVQAPOPEMMEMMBMM-VetAvg.Rel.
K=576K=576LLaVA-1.5-7B58.285.31514.864.329.762.6100.0%
K=128K=128w/o Visual Prior57.385.61430.258.831.560.997.3%
w/o Calibration Loss57.285.91447.358.731.161.197.6%
w/o Ranking Loss57.184.81422.158.430.360.396.3%
w/ KL Divergence Loss56.884.21412.058.229.859.995.7%
w/o Coverage57.285.41445.558.831.461.097.4%
READ (Ours)57.285.91440.562.232.662.099.0%
K=64K=64w/o Visual Prior56.183.01364.158.427.458.693.6%
w/o Calibration Loss56.283.81374.057.627.058.793.8%
w/o Ranking Loss56.080.41373.657.827.458.192.7%
w/ KL Divergence Loss55.379.81354.257.126.657.391.5%
w/o Coverage56.584.81420.358.128.959.995.7%
READ (Ours)56.186.41450.661.329.361.197.6%
K=32K=32w/o Visual Prior54.677.11278.155.424.555.188.0%
w/o Calibration Loss54.376.91280.055.423.554.887.5%
w/o Ranking Loss54.271.01207.355.023.352.884.2%
w/ KL Divergence Loss52.969.21192.653.622.551.682.4%
w/o Coverage54.879.51312.257.226.956.890.7%
READ (Ours)55.386.61348.159.729.259.695.2%

As shown in the table, dynamic prefix-ranking is critical to READ's performance, particularly under tight bandwidth constraints (K=32K=32). Removing the ranking objective causes overall relative retention to collapse from 95.2% to 84.2%, with POPE hallucination resistance dropping precipitously from 86.6% to 71.0% (−15.6-15.6 points) and MME perception declining by 140.8 points.

This ablation supports the benefit of pairwise ranking supervision beyond pointwise calibration in our training setup. Rank-survival calibration uses soft-label binary cross-entropy to fit inclusion probabilities, whereas prefix-ranking directly encourages a positive logit difference between candidates inside and outside sampled teacher prefixes. Such ordering matters because coverage starts from the highest-scoring patch and weights visual-feature cosine-distance novelty by relevance. The results are consistent with improved ordering near retention cutoffs, although we have not separately measured rank-inversion rates.

Q2. How much performance does READ lose when the query is removed, replaced with a generic prompt, or randomly shuffled across images?

Response:

We appreciate the reviewer for suggesting to isolate the exact role of query conditioning. To thoroughly examine how prompt semantics influence token selection, we conducted a systematic evaluation comparing our model across three controlled query conditions: completely removing the question text, replacing the question with a fixed, task-agnostic prompt ("Describe the image."), and randomly shuffling questions across mismatched images within each benchmark. Crucially, these diagnostic perturbations are applied strictly to the edge retriever's query input to assess visual evidence selection; the downstream cloud VLM always receives the original, unperturbed user question alongside the selected visual tokens to generate its final response. We evaluated all conditions across five multimodal benchmarks under three retention budgets (K∈{128,64,32}K \in \{128, 64, 32\}). The full quantitative results and relative preservation rates are reported in the table below. We have incorporated this query-conditioning ablation and in-depth analysis into Appendix I of the revised manuscript.

BudgetQuery ConditionTextVQAPOPEMMEMMBVizWizAvg.Rel.
K=576K=576LLaVA-1.5-7B58.285.31514.864.350.266.7100.0%
K=128K=128w/o Query56.783.51429.458.551.764.496.6%
Generic Query56.682.41436.458.651.764.296.3%
Shuffled Query57.184.41459.658.551.865.097.5%
Matched Query57.285.91440.562.252.365.998.8%
K=64K=64w/o Query54.677.41360.556.651.561.692.4%
Generic Query54.775.51351.656.751.661.291.8%
Shuffled Query56.278.31361.258.851.962.794.0%
Matched Query56.186.41450.661.353.265.998.8%
K=32K=32w/o Query51.565.91221.852.250.756.384.4%
Generic Query51.667.21228.355.250.157.185.6%
Shuffled Query54.570.71251.656.950.859.188.6%
Matched Query55.386.61348.159.753.664.596.7%

As demonstrated in the table, query conditioning is essential, and READ actively leverages prompt semantics to improve retention accuracy, achieving 96.7% relative performance even under aggressive compression (K=32K=32).

When the query is removed or replaced with a generic prompt, the compressed model still maintains a competitive baseline (retaining 84.4% to 85.6% at K=32K=32). The learned visual pathway and salience prior remain active in these conditions, together with coverage based on visual-feature diversity. Their individual contributions are not isolated by the query ablation. Nevertheless, both conditions perform substantially worse than matched queries because many queries require localizing subtle, question-specific visual evidence—such as verifying tiny physical entities in POPE, reading localized text, or discerning fine-grained attributes—which query-independent salience alone cannot anticipate.

Interestingly, shuffled queries achieve intermediate performance and surpass the query-free baseline on several benchmarks. A possible explanation is within-dataset query similarity and shared task-level formatting; these mechanisms have not been isolated experimentally. For example, in TextVQA, although phrasing varies across samples, all prompts explicitly require reading scene text and include the "Reference OCR token:" prefix; this shared prompt structure effectively steers the retriever to extract text-bearing image patches, directly assisting the LLM in reading the target text. Matched queries achieve higher aggregate performance at every evaluated budget, although not on every individual benchmark, supporting the benefit of instance-level question alignment.

Q3. How closely do the teacher's attention-based token rankings align with the visual regions genuinely necessary for correct predictions? Can the authors provide qualitative or quantitative evidence-localization or token-importance evaluations?

Response:

We appreciate the reviewer for raising this essential point, which is fundamental to validating our attention distillation framework. To rigorously evaluate the validity of teacher attention, we provide both quantitative token-selection experiments and grounded qualitative spatial localizations on LLaVA-1.5-7B:

BudgetMethodRetainTextVQAPOPEMMEMMBVizWizAvg.Rel.
K=576K=576LLaVA-1.5-7B57658.285.31514.864.350.266.7100.0%
K=128K=128 (↓77.8%\downarrow 77.8\%)Random Selection12851.281.21378.753.752.561.592.2%
Teacher Attention12857.386.51461.458.852.065.598.2%
K=64K=64 (↓88.9%\downarrow 88.9\%)Random Selection6448.275.41315.049.652.858.387.4%
Teacher Attention6456.886.21407.958.551.964.897.2%
K=32K=32 (↓94.4%\downarrow 94.4\%)Random Selection3245.867.31130.044.552.053.279.8%
Teacher Attention3254.884.91326.556.650.562.693.9%

As demonstrated in the table, directly retaining tokens ranked highest by the teacher's full-prompt question-to-visual cross-attention scores robustly preserves reasoning capability across all budgets. Under severe compression (K=32K=32), teacher-attention-based selection achieves 93.9% relative retention across benchmarks, substantially outperforming uniform random selection, with decisive advantages on fine-grained verification, localized text reading, and multi-modal perception. This performance gap supports teacher attention as a useful token-importance proxy, but does not establish causal evidence localization.

Besides, in the revised manuscript, we qualitatively show teacher-ranked patches overlapping answer-relevant text, entities, and diagram details. The quantitative selection comparison and qualitative visualization are reported in Appendix F (Table 7 and Figure 3).

Reviewer_cvbCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Query aware visual token compression requires cross modal attention from LLM to select relevant tokens. To reduce the communication overhead this selection should happen at local device before transmission. This paper trains a light weight cross attention model that distills the server side large VLM attention behavior. Here what we need is the ranking only for the selection. So the distill loss function tries to keep the rank maintained and also score values in proper scale. The lightweight retriever provides relevance scores to patches which are combined with visual saliency score to make a score for token selection. finally, this score is weighted by coverage score to provide diversity. This method showed big improvement over other competing methods, especially in severe compression cases. Also more efficient as the edge model size is small. When compared another proxy-guided method, SGP,

Critical strengths

Overall system is clearly designed and shows big improvements. Components such as ranking distillation loss, calibration loss, visual prior, coverage are well defined and improve the overall performance as shown in the ablation study. Depending on used VLM model, sometimes SGP shows better performance. Still READ model size is 32x smaller than SGP, has practical meaning at edge devices. Not only latency, the model size is also an important factor for edge deployment where storage is limited.

Critical weaknesses

NONE

Questions for authors

Why does the performance vary significantly on different VLM models? What is the main cause?

1 comment

Q1. Why does the performance vary significantly on different VLM models? What is the main cause?

Response:

We thank the reviewer for raising this thoughtful question. We distinguish differences in the original backbones' capabilities from differences in their sensitivity to visual token compression. On the same six benchmarks in Table 2, the full-token average scores are 71.8 for Qwen2.5-VL-7B and 77.3 for InternVL2.5-8B, so absolute compressed scores should not be compared without accounting for this baseline difference. Relative to each backbone's full-token performance, READ retains 95.7% and 95.9%, respectively, at the 25% token budget, and 88.6% and 90.3% at the 10% budget. Thus, READ's average relative retention is close across these two backbones, although aggressive compression exposes task-dependent differences. For example, at the 10% budget, TextVQA decreases from 75.5 to 70.7 on Qwen and from 75.5 to 67.0 on InternVL, whereas MM-Vet decreases from 65.1 to 44.6 and from 61.0 to 39.8, respectively. The compression sensitivity therefore varies by both backbone and task, rather than one backbone being uniformly more robust.

The architectures also differ in how visual evidence is represented and positioned. As described in Appendix A, READ uses a CLS-to-patch visual prior for InternVL's InternViT encoder and an averaged visual self-attention prior for Qwen's encoder without a CLS token. InternVL uses pixel-unshuffle-based visual token reduction, while Qwen uses dynamic-resolution tokenization with multimodal rotary position embeddings. Consequently, the same retained percentage need not correspond to the same absolute token count or spatial coverage. These differences provide plausible reasons for architecture-dependent compression behavior, but our current experiments do not isolate their individual causal contributions.

The change in READ's relative ranking against SGP is a separate consideration. At the 10% budget, READ exceeds SGP by 7.0 percentage points in relative retention on Qwen, while SGP exceeds READ by 1.4 points on InternVL. This may partly reflect how well the auxiliary model's importance rankings transfer to the target backbone; the cross-model scores alone do not establish proxy mismatch as the sole cause. READ instead learns from the frozen target VLM's own attention, avoiding dependence on rankings transferred from a separate proxy model. We do not claim that this removes all compression error or guarantees higher accuracy than SGP on every backbone. Its contribution is target-specific, query-conditioned selection without executing an auxiliary language model online.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.