- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies visual token selection for edge–cloud VLM inference, where an edge device encodes an image and uploads visual tokens to a cloud VLM. READ caches the cloud VLM's prompt-to-visual attention offline and distills it into a small cross-modal retriever that runs on the device. The retriever combines a visual salience prior with the prompt embeddings, is trained with a prefix-ranking loss over log-uniformly sampled budgets plus a rank-survival calibration loss, and selects K tokens with relevance-weighted coverage before upload. Experiments on LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and InternVL2.5 show better accuracy retention than most baselines, especially at tight budgets.
Critical strengths
- The problem is practical and the gap is clearly identified. Vision-only selectors can run before transmission but ignore the question, while query-aware methods either prune inside the cloud model after everything has been uploaded or need a heavy model on the device. Asking for query-aware selection before transmission, without running a language model on the device, is a reasonable and useful target.
- Transferring the cloud VLM's token preference into a small model is a clean way to save cost. Instead of running a proxy VLM on the device, READ caches the target VLM's attention once offline and distills which visual tokens it prefers into a compact retriever. This keeps the selection aligned with the model that will actually answer, while cutting the selector from billions of parameters to tens of millions and more than halving the edge latency compared with SGP. The training objective also lets a single checkpoint serve any token budget.
- The experiments are extensive and contain some interesting findings. The method is evaluated on four VLM families, including dynamic-resolution models, on up to ten benchmarks and several budgets, and the advantage over baselines grows as the budget shrinks. The analyses of the teacher signal are also interesting, for example that attention from the last prompt token works better for direct selection but worse as a distillation target than attention aggregated over the whole prompt.
Critical weaknesses
- The edge–cloud setting is weakly motivated. Without the deployment story, READ is essentially a learned visual token selector, and the edge–cloud framing is what makes selection before transmission necessary. However, the paper measures communication only in token counts (§5) and never in bytes. Projected visual tokens are high-dimensional, so even the reduced set may be larger than a compressed image. A simpler design, uploading the compressed image and running the selection in the cloud, would then give a smaller uplink and the same prefill savings. The paper neither discusses this option nor gives a reason, such as privacy, why the image cannot leave the device.
- The query-aware ability claimed in the title and abstract is not directly verified. No experiment isolates how much the selection actually depends on the question. The comparison with query-agnostic baselines does not settle this, because READ also differs from them in training data, the query-independent visual prior, and the coverage step, and the prior alone carries a large part of the result. The gains are also not concentrated on the text-heavy tasks that the motivation emphasizes, and the only direct evidence is a single qualitative example.
- The technical novelty is limited, and it is unclear how much of the gain comes from distillation. Each component is close to existing work: the attention signal is what FastV and SGP already use, and relevance-weighted diversity selection is close to DivPrune, VisPruner, and IDPruner. The reformulation as evidence retrieval is mostly a new name for budgeted top-K selection. The paper's own results also suggest that a substantial part of the improvement comes from the components that are not distilled, since the distilled retriever outperforms direct selection with the teacher attention it imitates, and removing the prior or the coverage step causes large drops.
Questions for authors
- Can you report the uplink payload in bytes rather than tokens, stating which features are transmitted and at what dimension and precision, and compare it with uploading a compressed image and pruning in the cloud?
- Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?
- What are the edge memory and latency once the LLM word-embedding table needed for the prompt embeddings is included, and on what hardware were the edge numbers in Table 4 measured?
- Since every cloud VLM needs its own teacher cache and retriever, what does it cost to support a new or updated cloud model, and does a retriever trained for one VLM transfer to another?
More than this, please see the weaknesses above. If my concerns can be resolved, I would consider raising my score.
13 comments
Q2. Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?
Response:
We sincerely thank the reviewer for proposing these constructive evaluations to assess query conditioning. To address the reviewer's questions, we conducted two comprehensive experiments: first, systematically ablating and perturbing the user question—by completely removing the query, replacing it with a generic prompt, and randomly shuffling questions across images; and second, comparing directly against lightweight on-device query-aware selection via CLIP patch-to-text cosine similarity. In these diagnostic evaluations, query perturbations are applied strictly to the edge retriever's query input to assess visual evidence selection, and the downstream cloud VLM always receives the original user question to generate its response. The query-conditioning ablation is reported in Appendix I.3 (Table 9); the additional CLIP comparison is provided below.
To examine how prompt semantics influence token selection, we evaluated our model under three diagnostic query conditions: removing the question text entirely (w/o Query, retaining the learned visual pathway, visual prior, and coverage but removing question conditioning), replacing the specific question with a fixed, task-agnostic prompt (Generic Query: "Describe the image."), and randomly pairing each image with another question drawn from the same benchmark (Shuffled Query). We benchmarked all conditions across five multimodal benchmarks under three retention budgets ():
| Budget | Query Condition | TextVQA | POPE | MME | MMB | VizWiz | Avg. | Rel. |
|---|---|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | 58.2 | 85.3 | 1514.8 | 64.3 | 50.2 | 66.7 | 100.0% | |
| w/o Query | 56.7 | 83.5 | 1429.4 | 58.5 | 51.7 | 64.4 | 96.6% | |
| Generic Query | 56.6 | 82.4 | 1436.4 | 58.6 | 51.7 | 64.2 | 96.3% | |
| Shuffled Query | 57.1 | 84.4 | 1459.6 | 58.5 | 51.8 | 65.0 | 97.5% | |
| Matched Query | 57.2 | 85.9 | 1440.5 | 62.2 | 52.3 | 65.9 | 98.8% | |
| w/o Query | 54.6 | 77.4 | 1360.5 | 56.6 | 51.5 | 61.6 | 92.4% | |
| Generic Query | 54.7 | 75.5 | 1351.6 | 56.7 | 51.6 | 61.2 | 91.8% | |
| Shuffled Query | 56.2 | 78.3 | 1361.2 | 58.8 | 51.9 | 62.7 | 94.0% | |
| Matched Query | 56.1 | 86.4 | 1450.6 | 61.3 | 53.2 | 65.9 | 98.8% | |
| w/o Query | 51.5 | 65.9 | 1221.8 | 52.2 | 50.7 | 56.3 | 84.4% | |
| Generic Query | 51.6 | 67.2 | 1228.3 | 55.2 | 50.1 | 57.1 | 85.6% | |
| Shuffled Query | 54.5 | 70.7 | 1251.6 | 56.9 | 50.8 | 59.1 | 88.6% | |
| Matched Query | 55.3 | 86.6 | 1348.1 | 59.7 | 53.6 | 64.5 | 96.7% |
As demonstrated in the table, query conditioning is essential: under aggressive compression (, retaining only 5.6% of tokens), matched queries achieve 96.7% relative retention, outperforming query-free selection by +12.3 percentage points (64.5 vs. 56.3 average). When the query is removed or replaced with a generic prompt, the compressed model still preserves a respectable baseline (retaining 84.4% to 85.6% at ) with the learned visual pathway and salience prior still available, together with coverage based on visual-feature diversity. Nevertheless, both conditions perform substantially worse than matched queries because many queries require localizing subtle, question-specific visual evidence—such as verifying tiny physical entities in POPE (where matched queries beat query-free selection by +20.7 points), reading localized text in TextVQA, or discerning fine-grained attributes—which query-independent salience alone cannot anticipate.
Interestingly, shuffled queries achieve intermediate performance (88.6% relative retention at ) and surpass the query-free baseline on several benchmarks. One plausible explanation is that within-dataset questions share task-level semantics and formatting, so even a mismatched question can provide useful guidance (e.g., towards scene text in TextVQA). This mechanism has not been isolated experimentally. Matched queries nevertheless achieve higher aggregate performance than shuffled queries at every evaluated budget, although not on every individual benchmark.
Q2. Can you add an ablation that removes or replaces the question, a measure of how much the selected tokens change across different questions on the same image, and a comparison with lightweight query-aware selectors that can also run on the device, such as TRIM or CLIP text–image similarity?
Response:
To directly answer the reviewer's question regarding how much the selected tokens change across different questions on the same image, we evaluated the token selection divergence between the original Matched Query and each perturbed condition on identical images across all three budgets (). Specifically, we compute the Jaccard Similarity between the selected token sets for each sample:
| Budget | Perturbation Condition | TextVQA | POPE | MME | MMB | VizWiz | Mean Jaccard | Mean Change Ratio () |
|---|---|---|---|---|---|---|---|---|
| vs. w/o Query | 64.1% | 68.4% | 68.9% | 70.4% | 68.4% | 68.1% | 31.9% | |
| vs. Generic Query | 71.1% | 78.0% | 80.1% | 74.7% | 77.6% | 76.3% | 23.7% | |
| vs. Shuffled Query | 84.7% | 88.6% | 90.7% | 89.6% | 90.6% | 88.8% | 11.2% | |
| vs. w/o Query | 55.1% | 61.8% | 60.6% | 61.7% | 61.4% | 60.1% | 39.9% | |
| vs. Generic Query | 58.9% | 72.9% | 72.6% | 66.2% | 70.1% | 68.2% | 31.8% | |
| vs. Shuffled Query | 80.3% | 84.5% | 87.3% | 85.9% | 87.4% | 85.1% | 14.9% | |
| vs. w/o Query | 51.0% | 55.0% | 54.9% | 57.5% | 59.1% | 55.5% | 44.5% | |
| vs. Generic Query | 49.9% | 66.7% | 65.6% | 61.1% | 64.9% | 61.6% | 38.4% | |
| vs. Shuffled Query | 77.4% | 81.6% | 84.5% | 83.5% | 85.9% | 82.6% | 17.4% |
These quantitative similarity measurements provide several key takeaways:
- Dynamic Reallocation under Tight Budgets: Under aggressive compression (), completely removing the query causes the Jaccard similarity to drop to 55.5%. This confirms that nearly half of the retained token capacity is actively and dynamically steered by the text question rather than statically dictated by query-agnostic visual priors.
- Semantic Sensitivity Continuum: The degree of token divergence strictly tracks semantic specificity across all three budgets: removing queries causes the largest deviation (), generic prompts yield intermediate divergence (), while shuffled prompts share broad formatting cues and thus maintain higher overlap (). Crucially, even with high task formatting overlap, the remaining ~11%--17% query-specific token reallocation is what enables matched queries to consistently outperform shuffled queries across all budgets.
Q1. Can you report the uplink payload in bytes rather than tokens, stating which features are transmitted and at what dimension and precision, and compare it with uploading a compressed image and pruning in the cloud?
Response:
We thank the reviewer for highlighting this important comparison. Traditional image/video codecs are highly efficient, and uploading a compressed image for cloud-side pruning is a viable alternative, particularly when uplink bandwidth is the dominant constraint. We do not claim that READ uses fewer bytes than JPEG or other conventional codecs, or necessarily reduces total edge-plus-cloud computation. Our focus is edge–cloud computation partitioning: the edge performs vision encoding and query-conditioned feature selection, allowing the cloud to bypass vision encoding and process a shorter LLM prefill sequence. We have incorporated this detailed discussion and interface comparison into Appendix D (Table 5).
For LLaVA-1.5-7B at 336 × 336 resolution, the byte accounting and architectural alternatives are:
Under the evaluated LLaVA-1.5-7B setting (336 × 336 input, bf16 precision, projected hidden dimension D=4096), transmitting 32 selected projected visual tokens requires 262.1 KB plus 64 bytes of spatial position indices, reducing the uncompressed 576-token payload (4.72 MB) by 94.4%. If transmitting 1024-dimensional pre-projector features, the payload is further reduced to 65.5 KB, reflecting both token selection and lower transmitted feature dimensions.
This system design connects naturally to the philosophy of Feature Coding for Machines (FCM) and collaborative edge–cloud paradigms such as the Digital Retina [4]. Conventional codecs (e.g., JPEG, AVC/HEVC) are optimized for human perceptual viewing, whereas downstream VLM reasoning operates directly on deep visual features. If raw or compressed images are naively streamed to the cloud, the central server must execute vision encoders for every concurrent query. In high-throughput, multi-tenant serving, this visual encoding creates a severe server-side bottleneck. By shifting visual front-end encoding and query-conditioned evidence filtering to edge devices with local compute, the cloud completely bypasses vision encoding and processes a substantially shortened sequence during prefill.
Crucially, token selection and traditional feature compression are not mutually exclusive: token pruning compresses across the sequence length (token count) dimension, which is fully complementary to subsequent feature quantization and entropy coding (e.g., TOFC [1]). READ addresses a different operating point from conventional image compression: rather than minimizing image transmission bitrate alone, it enables query-conditioned visual evidence selection before transmission, shifting visual processing to the edge and reducing the computational burden of cloud-side multimodal inference. This approach is complementary to feature coding techniques that further compress the selected representations.
Because READ predicts a compact, discrete index set of query-relevant spatial patches, our framework naturally accommodates two complementary dynamic transport options depending on which system bottleneck dominates:
In summary, evaluating communication cost in concrete bytes rather than abstract token counts highlights the practical flexibility of our framework across different system bottlenecks. We thank the reviewer again for raising this thoughtful architectural question, which has helped us significantly clarify the edge–cloud deployment trade-offs and broaden the practical scope of READ.