- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LEAF, a training-free visual token selection method that combines adaptive spatial partitioning with a language-conditioned token selection mechanism. The method utilizes an auxiliary visual encoder to construct a saliency-guided hierarchical structure while extracting query-relevant scores through the forward pass of a truncated Large Language Model (LLM). Subsequently, a hierarchy-aware selector allocates the token budget based on spatial coverage, task relevance, and local feature heterogeneity. Experiments across various Vision-Language Models (VLMs) and visual understanding benchmarks demonstrate highly competitive performance, particularly in high-compression scenarios.
Critical strengths
-
Strong empirical performance under aggressive compression. LEAF achieves the highest normalized average performance among the evaluated methods across four frozen VLMs and the tested token budgets. On Qwen2.5-VL at 10% token retention, it preserves 96.08% of full-token performance, compared with 90.76% for the strongest evaluated baseline. These results provide substantive evidence that the method better preserves task performance under stringent visual-token budgets.
-
Comprehensive and coherent framework design. LEAF integrates saliency-guided spatial partitioning, question-aware relevance probing, and hierarchy-aware budget allocation into a unified training-free pipeline. These components address three complementary decisions: how to partition the image, which visual evidence is relevant, and how much detail each region requires. This coordinated design provides a systematic framework for balancing regional coverage and task-relevant detail under constrained token budgets.
Critical weaknesses
-
Insufficient justification for adaptive spatial partitioning. LEAF uses regional average saliency to determine whether further partitioning is needed, although high saliency does not necessarily indicate greater within-region complexity. Comparisons among QTMT split types do not isolate the benefit of adaptive partitioning, as simpler alternatives, such as fixed uniform partitioning with comparable granularity, are not evaluated.
-
Weak baselines for hierarchical budget allocation. The w/o S3 ablation exhausts higher-ranked leaves before considering others, encouraging excessive budget concentration. Its substantial degradation therefore does not convincingly establish the superiority of the proposed allocation mechanism over competitive alternatives, such as global relevance-based top-k selection, uniform regional allocation, or relevance-weighted regional budgets.
-
Unclear practical efficiency and unverified serving integration. LEAF adds an auxiliary DeiT-S encoder and nine full-sequence decoder blocks before rerunning inference on the compressed sequence, resulting in higher FLOPs than competing methods at matched token budgets. Its probing requires explicit Q/K replay beyond standard FlashAttention execution, although the backbone can still use fused attention kernels. Integration with vLLM/SGLang, including scheduling and KV-cache handling, is not demonstrated.
-
Insufficient validation of spatial coverage. LEAF measures regional coverage through feature similarity, potentially treating visually similar but spatially distinct regions as substitutes. The coverage objective does not explicitly account for their different locations or instance identities. Without targeted quantitative analysis of repeated objects and spatial relationships, it remains unclear whether this mechanism reliably preserves the fine-grained evidence required for counting and spatial reasoning.
Questions for authors
-
Adaptive spatial partitioning. What justifies using regional average saliency to determine whether further subdivision is needed, given that saliency does not necessarily reflect within-region complexity? Can you compare QTMT with fixed uniform partitioning?
-
Practical efficiency and serving integration. Can you clarify which stages retain fused attention kernels, quantify the additional cost of Q/K replay, and indicate whether the two-pass pipeline has been tested in vLLM or SGLang, including scheduling and KV-cache handling?
-
Preservation of spatially distinct evidence. How does the feature-similarity-based coverage mechanism avoid treating visually similar but spatially distinct object instances as interchangeable? Can you provide targeted quantitative evidence on counting or spatial reasoning that isolates the effect of the coverage term on preserving these distinct instances?
Other questions please refer to weakness.
5 comments
Response to Reviewer 23ft (Part 2/3): Fused Kernels, Replay Cost, and Serving
W3 / Q2. Fused kernels, Q/K replay cost, and serving integration.
Which stages use fused attention kernels. All of them, except for one small matrix product.
- Probe (layers 0–8, full sequence): standard fused attention (PyTorch SDPA) with
output_attentions=False, so no attention matrix is materialized. - Compressed pass (all 32 layers, prefill and decoding): unchanged backbone, fused attention.
- Q/K replay: forward hooks capture the layer-8 query and key projections that the fused probe already computes, so nothing is recomputed. The replay applies RoPE and computes only the question-row logits at that single layer ( per head instead of ), followed by a softmax. This is the only non-fused computation.
- DeiT-S saliency and QTMT: run on the CPU, overlapped with the GPU probe.
Cost of Q/K replay. LLaVA-1.5-7B, fp16, batch 1, A100, median of 100 runs; = 35 system + 576 visual + question tokens:
| Replay (ms) | Probe w/o replay (ms) | Probe w/ replay (ms) | Full prefill (ms) | Replay GFLOPs | |
|---|---|---|---|---|---|
| 32 | 0.28 | 21.91 | 22.14 | 79.08 | 0.17 |
| 64 | 0.30 | 22.74 | 23.01 | 80.90 | 0.35 |
| 128 | 0.41 | 23.62 | 24.03 | 84.55 | 0.78 |
| 256 | 0.52 | 29.10 | 29.79 | 102.62 | 1.82 |
Replay adds 0.2–0.7 ms (1.0–2.4%) to the probe and less than 0.1% of its FLOPs. The main cost of LEAF is the truncated probe itself (about 28% of a full-length prefill), not replay.
FLOPs versus latency. We agree that at a matched token budget LEAF costs more FLOPs than lighter pruning methods. In the revision we report Bjøntegaard deltas against random retention over 2–40% retention on Qwen2.5-VL-7B (new table in Appendix C.3):
| VisionZip | DivPrune | ZOO-Prune | LEAF | |
|---|---|---|---|---|
| BD-FLOPs (%) ↓ | −2.09 | −4.31 | −3.39 | +33.51 |
| BD-Latency (%) ↓ | +0.18 | +2.75 | +16.26 | −0.14 |
LEAF is the only method with lower end-to-end latency than random retention at matched performance, because the CPU-side hierarchy is hidden behind the probe and LEAF needs far fewer tokens for the same accuracy (BD-Token −75.94%). On LLaVA-1.5-7B (Table 14), LEAF with 64 tokens is both more accurate on MMBench and faster in prefill than VisionZip, DivPrune and ZOO-Prune with 128 tokens (62.11% / 73.24 ms vs. 62.00–62.03% / 77.43–94.32 ms).
vLLM/SGLang. We have not integrated LEAF into vLLM or SGLang, and we state this as a limitation in the revised Section 5 and Appendix D. The two-pass design maps onto these engines without changing their attention kernels or KV-cache manager:
- The probe is a truncated 9-layer prefill whose KV is not written to the paged cache, and the replay is a small kernel on projections it already produces.
- The selected visual embeddings then enter the engine as an ordinary, shorter multimodal request, so its paged KV cache shrinks in proportion to the compression and decoding is unchanged.
- For the scheduler, the probe is one extra prefill-like job (about 9/32 of a full prefill) that can be batched or chunked with other prefills.
The main engineering work is the two-phase request lifecycle; no change to fused attention or paged KV handling is needed.
Response to Reviewer 23ft (Part 3/3): Spatially Distinct Instances
W4 / Q3. Preserving visually similar but spatially distinct instances.
LEAF's coverage term does not, by itself, decide whether a second instance is selected. Three properties of the selector keep distinct instances from being treated as interchangeable:
- Candidates are spatial units. The selector chooses among QTMT leaves, which are contiguous regions; instances at different locations generally fall into different leaves, scored separately.
- Each leaf keeps its own relevance term. An inactive leaf has gain (Eq. 9). If a look-alike instance was already selected, only the coverage part shrinks; the importance comes from the LLM probe on position-encoded tokens, so a second relevant instance still competes on its own relevance.
- Concentrating on one instance is discouraged. Extra patches from an opened leaf receive only the decayed detail gain with , so opening another relevant instance's leaf is usually preferred.
Targeted evidence. We evaluate SEED-Bench categories that require distinct instances and their locations: Instance Counting (2,447 questions), Instance Location (978) and Spatial Relation (657), on LLaVA-1.5-7B with the same evaluator for all methods (Count / Loc. / Spatial rel.):
| Method | 128 tokens | 64 tokens |
|---|---|---|
| ZOO-Prune | 54.80 / 54.60 / 45.97 | — |
| CDPruner | 56.40 / 54.91 / 47.03 | 54.52 / 53.89 / 45.81 |
| MMTok | 54.64 / 54.91 / 47.34 | 53.45 / 52.86 / 45.05 |
| Global top- (no hierarchy, no coverage) | 56.48 / 56.44 / 47.79 | — |
| LEAF, coverage only | 57.13 / 55.83 / 48.10 | — |
| LEAF w/o coverage | 56.76 / 56.85 / 47.34 | 56.44 / 54.81 / 46.73 |
| LEAF | 56.60 / 56.34 / 47.34 | 56.44 / 54.50 / 47.34 |
Coverage on vs. off on the main benchmarks (GQA / MMB / MME / POPE)*:
| 128 tokens | 64 tokens | |
|---|---|---|
| LEAF w/o coverage | 59.38 / 63.32 / 1756.34 / 86.42 | 57.69 / 62.20 / 1691.41 / 85.95 |
| LEAF | 59.45 / 62.97 / 1760.18 / 86.41 | 58.18 / 62.37 / 1699.28 / 86.08 |
Three observations. (i) Coverage does not merge instances: removing it changes counting by at most 0.16 points, and the coverage-only selector has the highest counting and spatial-relation accuracy at 128 tokens. (ii) Coverage matters more under tight budgets: at 128 tokens the two variants are within noise, while at 64 tokens coverage improves all four main benchmarks and spatial relation (+0.61). (iii) LEAF matches or exceeds ZOO-Prune, CDPruner and MMTok on all three categories, with the largest margins at 64 tokens (+1.9 to +3.0 counting, +1.5 to +2.3 spatial relation).
We agree that coverage does not model instance identity explicitly: two instances inside the same leaf, or a distant instance with low question relevance, can still be under-represented. We have added this discussion and the tables to Appendix C.6, and note position-aware similarity (adding normalized leaf coordinates to ) as a natural extension.
*Both variants re-run in the same environment; full-LEAF numbers are within 0.3 points (5 on MME) of Table 1.
We hope these responses address the reviewer's concerns, and we would be happy to clarify anything further.
Response to Reviewer 23ft (Part 1/3): Adaptive Partitioning and Budget Allocation
We thank Reviewer 23ft for the thoughtful and detailed review, and for recognizing LEAF's strong empirical performance under aggressive compression and its coherent framework design. We address each question below.
W1 / Q1. Why mean saliency for splitting, and comparison with a uniform partition.
What the split criterion measures. LEAF's saliency is not an attention or objectness score. It is the normalized projection of each location's centered DeiT-S feature onto the leading principal direction of feature variation (Eqs. 10–13). Low mean saliency means a region is close to the image's dominant background mode and is treated as homogeneous; high mean saliency indicates structured content, where finer leaves help. We agree that mean saliency is a proxy for within-region complexity, so LEAF does not rely on it alone: the split geometry follows the saliency variation inside the region (anisotropy and center concentration, Eqs. 15–16), and the number of tokens per leaf is set by its feature heterogeneity in the S3 selector, independently of saliency.
Uniform partition. As suggested, we replace QTMT with a fixed uniform 18×18 grid (324 regions, matched to QTMT's average of 332 leaves), keeping S2 and S3 unchanged (LLaVA-1.5-7B, 128 tokens):
At matched granularity, adaptive partitioning beats the uniform grid on all four benchmarks (+0.37 GQA, +1.69 MMB, +26.3 MME, +1.43 POPE). Building the hierarchy from pixel gradients, a direct measure of local complexity, is also worse than DeiT saliency (58.83 / 62.21 / 1746.26 / 85.22; Fig. 4(a)), which suggests semantic structure is a better split cue than texture. LEAF is also stable in the threshold α that turns mean saliency into a split decision: across α∈{0.1,0.15,0.2,0.3,0.4}, GQA stays within 59.02–59.34, MMB 63.06–63.20, MME 1754.6–1765.0 and POPE 85.59–86.41, all above the uniform grid. We have added both results to Appendix C.5 and Table 7 in Section 4.2.
W2. Stronger alternatives to hierarchy-aware budget allocation.
We agree that "w/o S3" is a step-necessity ablation, not a strong allocation baseline. As suggested, we compare allocation rules that use the same QTMT leaves and the same question-aware patch scores as LEAF and change only how the 128-token budget is distributed (LLaVA-1.5-7B; Avg. = mean retention relative to the full-token model):
Uniform allocation gives every leaf an equal share of the budget; relevance-weighted budgets make each leaf's share proportional to its relevance rj=maxi∈Ljqi. Both apportion the budget with the D'Hondt rule, cap each leaf at its size, and keep the highest-scoring patches within each leaf.
All three alternatives are far stronger than "w/o S3", confirming the reviewer's point. LEAF still has the best average and is best on GQA and MME (+16 MME over the best alternative). Relevance-weighted budgets are slightly better on MMB (+0.35) and global top-k on POPE (+0.63). POPE asks about a single named object, so the most relevant patches suffice; LEAF's advantage comes from questions that need evidence from several regions, which is what the coverage term targets. We have added this comparison to Appendix C.5 and Table 7 in Section 4.2.