acceptodds
Under review as a conference paper at ICLR 2027

When Pruning Rankings Reverse: State-Conditioned Visual Token Compression across Architectures

Abstract

The high cost of visual tokens makes token compression key to accelerating VLM inference. Existing methods mainly select tokens by token-wise task relevance or inter-token diversity, and typically apply a fixed rule to all inputs. However, we find that the relative effectiveness of these two representative approaches reverses across model architectures and shifts with task and compression ratio; even under the same architecture and task, preferences differ across samples. This indicates that token selection should adapt to each sample rather than follow a fixed rule. We therefore propose SATOR (Sample-Adaptive Token Reduction), a training-free, sample-adaptive token selection method that requires no architecture- or task-specific parameters. SATOR estimates the task contribution of each token from attention and Value norms. Before selection, it uses the visual redundancy and task support of the current input to balance two marginal gains: a distinctiveness gain that suppresses redundancy with retained tokens, and a representativeness gain that encourages coverage of unretained task-relevant information. On LLaVA-NeXT-7B, SATOR retains only 11.1% of visual tokens while preserving 97.5% of the original average performance, achieving a 3.24 prefill speedup, a 2.19 end-to-end speedup, and a 6.02 reduction in KV-cache size.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Compression, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WAJgOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the relative performance of relevance-based and diversity-based visual token pruning changes across models and tasks.

It proposes SATOR, a training-free method that combines determinant-based distinctiveness with task-weighted coverage. Task contribution is estimated from relative attention and value norms. A routing weight computed from input redundancy, task support, and the token budget balances the two gains.

The experiments cover 4 VLM backbones, with routing and component ablations on three models and inference-time measurements on the LLaVA models.

Critical strengths
  1. The cross-model analysis and paired sample results address a useful practical problem. They show that average method rankings can conceal different outcomes across inputs, providing evidence against choosing a pruning method solely from aggregate performance.

  2. The routing rule uses the current input without additional training or external calibration statistics. Combining redundancy reduction with coverage of unretained task information gives the method a coherent design and makes it potentially easier to transfer across settings.

  3. The routing ablations provide meaningful evidence beyond the main benchmark tables. Table 6 compares individual gains, fixed weights, stage-wise mean weights, and reversed state dependencies. SATOR improves the mean normalized score from 95.57 for the best shared constant to 96.01, supporting its value over the tested controls.

  4. The efficiency results include selection overhead. On LLaVA-NeXT, the reported 2x total inference speedup and 6x KV-cache reduction accompany 97.5% average performance retention. These measurements support a useful accuracy-efficiency trade-off in the tested setting.

Critical weaknesses
  1. The novelty is not sufficiently distinguished from some closely related work. AgilePruner already studies input-dependent attention/diversity preferences and image-aware adaptation. CDPruner uses relevance-weighted determinant selection, while SCOPE combines saliency and marginal coverage. SATOR differs in its contribution scores and state-dependent mixture, but the paper needs to explain what additional insight these differences provide. Section 2.2 also incorrectly attributes AgilePruner to Wang et al. CrisPrune, which combines text relevance, visual saliency, and determinant-based diversity, is not discussed. To summarize, I found the differences of this work with previous works listed here are mostly technical designs rather than insights.

  2. The performance comparisons omit AgilePruner, CDPruner, and SCOPE. Since these methods directly address the proposed adaptation and selection mechanisms, their absence leaves the practical value of SATOR's added machinery unresolved. Competitive quality with reduced calibration would be a meaningful contribution, but comparisons with the current baselines do not establish that advantage.

  3. The compression schedule and budget accounting are unclear. Tables 5 and 10 use three stages, but their decoder layers and retained counts are not specified. Table 2 describes retained token counts, whereas Table 7 explicitly uses average per-layer retention. These quantities are different, and the schedule affects both accuracy and computation. Without the layer-wise counts and matching rules for baselines, the main comparisons are difficult to assess.

  4. The evidence for the necessity of sample conditioning is weaker than the conclusion suggests. Complementary errors and an oracle gain show potential headroom, but do not establish that the proposed states predict the useful selection behavior. Table 6 supports the tested adaptive rule, yet its best fixed weight is shared across architectures and its stage-wise constants are means rather than optimized controls. The results therefore leave some uncertainty about the benefit of matching each routing weight to its own sample.

Questions for authors
  1. Please clarify the pruning layers, retained counts at each stage, and the definition of the reported retention ratio for every backbone. How were one-shot and multi-stage baselines matched?

  2. Please clarify the methodological differences from AgilePruner, CDPruner, and SCOPE. Could a focused comparison under matched budgets establish the added quality, efficiency, or calibration benefit? One representative setting would help resolve this concern without requiring a broader benchmark suite.

  3. To validate the sample-conditioning claim, does the proposed routing outperform weights shuffled among samples within the same model, task, and stage? Alternatively, can existing evidence show that the states predict the relative utility of the two gains? This would help distinguish sample matching from differences in average routing behavior.

13 comments
AuthorsOpenReview ↗

Response to Reviewer WAJg: Q1

Q1: Please clarify the pruning layers, retained counts at each stage, and the definition of the reported retention ratio for every backbone. How were one-shot and multi-stage baselines matched?

We thank the reviewer for pointing this out; the original paper did not describe these configurations clearly enough. Below we explain the three items in the question one by one, and the revised version will add all configurations to Section 5.1 and the Appendix.

1. Pruning layers and per-stage retention counts for each backbone. SATOR adopts three-stage progressive pruning on all backbones, and all three pruning steps are completed in the first half of the LLM. Taking LLaVA-1.5-7B as an example (32 LLM layers in total, layer indices 0–31), pruning is performed at the inputs of layers 1, 10, and 15. The original number of visual tokens is 576. In the 64-token setting, the three stages retain 95, 52, and 21 tokens in turn: layer 0 processes 576, layers 1–9 process 95, layers 10–14 process 52, and layers 15–31 process 21. The per-layer average is exactly 64:

576+9×95+5×52+17×2132=64\frac{576 + 9 \times 95 + 5 \times 52 + 17 \times 21}{32} = 64 The other backbones (LLaVA-NeXT, Qwen3-VL, InternVL3) likewise follow the mainstream setting in this field (e.g., SparseVLM and PDrop) and complete the three-stage progressive pruning in the first half of the network (detailed experimental settings will be added to the Appendix of the revised version). There are two reasons:

  • Image information mainly flows into text tokens in the shallow to middle layers. Completing pruning in the first half both preserves the key cross-modal interactions and reduces computation as early as possible.
  • Visual redundancy increases as the network deepens. Decreasing the retained amount stage by stage avoids discarding useful visual information too early in the shallow layers.

2. Definition of the retention ratio, and why percentages are reported for some backbones. All backbones and all methods throughout the paper use the same definition: the fraction of the original visual tokens retained at each LLM layer, averaged over the layers. Let the original number of visual tokens of a sample be NvN_v, the LLM have LL layers in total, and the number of visual tokens input to layer ℓ\ell be NℓN_\ell; then

Retention ratio=1L∑ℓ=0L−1NℓNv\text{Retention ratio} = \frac{1}{L}\sum_{\ell=0}^{L-1}\frac{N_\ell}{N_v}

The two reporting formats correspond to the same definition:

  • LLaVA-1.5: NvN_v is fixed at 576, so token counts and retention ratios correspond one to one (192 / 128 / 64 means 33.3% / 22.2% / 11.1%). For convenient direct comparison with existing work, we follow the convention of reporting by token count.
  • LLaVA-NeXT, Qwen3-VL, InternVL3: these three backbones use dynamic resolution (AnyRes, native dynamic resolution, and dynamic tiling, respectively), and NvN_v varies with the resolution and aspect ratio of each image. In this case, the budget cannot be controlled with a fixed token count: retaining the same 64 tokens is about 11% for a small image but only a few percent for a large image, so the compression strength varies from image to image. Therefore, on these backbones the budget can only be controlled proportionally: each sample retains Bk=round(ρkNv)B_k = \mathrm{round}(\rho_k N_v) tokens at stage kk, so that each sample's per-layer average retention ratio equals the target value. Results are accordingly reported as percentages.

Therefore, 64 tokens on LLaVA-1.5 and 11.1% on Qwen3-VL represent the same compression strength.

3. How one-shot pruning baselines and multi-stage pruning baselines are matched. All baselines are aligned to the same average retention ratio defined above, i.e., the same average number of visual tokens processed per LLM layer. There are three specific cases:

  • (1) Baselines whose original papers report results on the same model at the same budget: we directly cite the numbers from the original papers. The revised version will mark the source of each in the table notes.
  • (2) Baselines whose original papers do not report this setting and that prune inside the LLM: we re-evaluate them with exactly the same pruning layers and per-stage retention counts as SATOR, replacing only the scoring or selection criterion. In this way, the number of tokens at every layer is the same as SATOR's, and differences in results come only from the selection criterion itself.
  • (3) Baselines whose original papers do not report this setting and that prune once before entering the LLM (at the vision encoder or connector): we compress directly to the target value before entering the LLM. Every LLM layer processes the same number of visual tokens, and the per-layer average retention ratio naturally equals the target value. On LLaVA-1.5, 11.1% corresponds to a fixed retention of 64 tokens; on dynamic-resolution backbones, round(11.1%×Nv)\mathrm{round}(11.1\% \times N_v) tokens are retained per sample at the same ratio, consistent with SATOR's proportional control.
AuthorsOpenReview ↗

Response to Reviewer WAJg: Q2(1/3)

Q2: Please clarify the methodological differences from AgilePruner, CDPruner, and SCOPE. Could a focused comparison under matched budgets establish the added quality, efficiency, or calibration benefit? One representative setting would help resolve this concern without requiring a broader benchmark suite.

We thank the reviewer for the suggestion. We would like to emphasize that the fundamental difference between SATOR and these three methods lies in "what determines the balance between the two types of criteria":

  • AgilePruner determines it by image complexity, relying on training-set statistics for calibration;
  • CDPruner and SCOPE use a fixed combination for all inputs;
  • SATOR, inside the LLM, computes λ = R^{1+s} in closed form from the current sample's redundancy R and task support s, routing continuously between the two corrections per sample and per stage, without any calibration.

All of SATOR's other design differences serve this point:

  • the contribution score q must be comparable across images and stages so that Σq/B can define the support s;
  • the state must be read from inside the model to reflect differences across architectures;
  • pruning must be performed in stages inside the LLM so that the state can be re-estimated as redundancy changes.
AgilePrunerCDPrunerSCOPESATOR
Importance signalVision encoder attention (CLS attention on LLaVA)Cosine between visual and instruction embeddings before the LLM, min-max normalizedCLS attention from the penultimate ViT layer (question-agnostic)Text→visual relative attention inside the LLM × Value norm, weighted by head informativeness, RMS saturation normalized
Selection operatorRank by attention, removing neighbors within a similarity thresholdConditional DPP greedy MAPGreedy selection by coverage gain × saliency^αGreedy selection mixing the distinctiveness gain (DPP) and the representativeness gain (q-weighted coverage) by λ
BalancingAdjust the threshold by erank, or linearly allocate the numbers of the two token types by erankFixedFixed (multiplicative combination)λ = R^{1+s}, computed per sample and per stage
Calibration / hyperparametersAverage erank of the LLaVA training set, constant 0.01, upper bound τ_maxNone in the main experiments; the balance factor θ needs hand-tuning per dataset; the text embedding source needs to be chosen per architectureα (default 1.0, selected via ablation)None
Structural dependencyVision encoder attentionA paired text encoder, or LLM word embeddingsCLS token of the vision encoderOnly LLM-internal attention and hidden states
Pruning positionVision-encoder side, onceBefore the LLM, onceAfter the vision encoder, onceMulti-stage inside the LLM, re-estimating the state at each stage

Differences from AgilePruner

  • Commonality: both let the balance between attention and diversity vary with the input.
  • The adaptation signal differs. AgilePruner relies on image complexity (the erank of visual features or attention entropy), which is question-agnostic; the same image paired with different questions yields the same selection strategy. SATOR's support s is obtained from text→visual attention and is question-dependent. When s is removed and only visual redundancy is kept (λ = R), performance drops from 96.01% to 95.35% (Table 6).
  • The object of adaptation differs. AgilePruner adjusts the deduplication strength in "attention ranking + threshold deduplication", where the threshold grows with the token's rank and with the ratio erank_input / erank_avg, subject to an upper bound; it can also linearly allocate the numbers of the two token types by erank, i.e., a hard split. SATOR continuously weights the two marginal gains at each greedy step, and includes the dimension of "covering not-retained task evidence" that AgilePruner lacks.
  • The calibration requirements differ. AgilePruner needs the average erank computed from LLaVA training-set statistics as a reference, as well as hand-set constants and an upper bound, and the statistics must be recomputed when changing models or data distributions. SATOR's R, s, and λ are all obtained in closed form from the current input.
AuthorsOpenReview ↗

Response to Reviewer WAJg: Q2(2/3)

Differences from CDPruner

  • Commonality: the distinctiveness gain (Eq. 6) has the same form as CDPruner's conditional DPP greedy gain, i.e., a relevance-weighted kernel plus greedy MAP. The revised version will explicitly state this connection.
  • The role of DPP differs. In CDPruner, the conditional DPP is the entire objective; in SATOR, it is only one of two marginal gains, with its weight determined per sample by λ. Setting λ ≡ 1, i.e., degenerating into a CDPruner-style objective, gives 95.39%, 0.62 pp lower than SATOR (Table 6).
  • The definition and scale of the relevance score differ. CDPruner computes the cosine between visual and instruction embeddings before the LLM and then applies min-max normalization, so every image necessarily contains tokens taking 1 and 0, and only the relative order is preserved. SATOR's q comes from inside the LLM: it uses relative attention to avoid attention sinks, the Value norm to capture the actual amount written in, and head entropy weights to highlight heads that vary with the text, and it is RMS saturation normalized so that it is comparable across images and stages; only then can Σq/B serve as the support s. Switching to max normalization lowers performance to 95.81% (Table 12).
  • The kernel function differs. SATOR first mean-centers the hidden states to eliminate anisotropy, and then uses ((1+cos)/2)² to suppress weak similarities; CDPruner uses the raw cosine.
  • The balancing mechanism differs. CDPruner notes in its appendix that the optimal balance factor θ varies across benchmarks and needs to be hand-tuned per dataset; SATOR turns it into a state quantity computed in closed form per sample.

Differences from SCOPE

  • Commonality: the representativeness gain and SCOPE's coverage gain are both facility-location-style coverage gains. The revised version will add the citation.
  • The source of saliency differs. SCOPE uses ViT CLS attention, which is question-agnostic; SATOR uses the question-dependent task contribution q.
  • The object and form of coverage differ. SCOPE covers all visual tokens by the maximum cosine. SATOR weights by q, covering "the not-yet-retained task-relevant information", and adopts a squared-residual form (Eq. 7) so that high-contribution tokens far from the selected set receive larger gains.
  • The combination differs. SCOPE combines the two multiplicatively via Δ(v;S)·A_v^α, with α fixed to 1.0 for all inputs. SATOR first normalizes the two gains separately over the candidate set and then mixes them additively by λ; 1−λ increases with the support s, so the weight of coverage is increased only when the task evidence exceeds the budget. Pure coverage (λ ≡ 0) yields only 93.74%, and on InternVL3 it is even lower than pure importance ranking (92.92% vs. 95.05%), indicating that coverage is not beneficial for all inputs.
  • The scope of applicability differs. SCOPE relies on the CLS token and cannot be directly applied to architectures whose vision encoder has no CLS token, such as Qwen3-VL. Although InternVL3's vision encoder has a CLS token, each dynamic tile has its own, and the visual tokens are further merged via a 2×2 pixel shuffle, requiring additional modification. SATOR uses only LLM-internal signals and can be used directly on all three architectures.

In summary, the three methods either adjust the balance by image complexity with the help of training-set statistics (AgilePruner), or adopt a fixed combination for all inputs (CDPruner, SCOPE). SATOR instead reads the current sample's redundancy and task support from inside the LLM and routes continuously between redundancy removal and coverage per sample and per stage, without any dataset statistics or hyperparameter calibration.

Reviewer_8csuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual token pruning by comparing FastV and DART, showing that the relative performance of relevance-oriented and diversity-oriented selection varies across model architectures, tasks, and compression ratios. Based on these observations, the authors propose SATOR, a training-free method that estimates token-level task contribution using relative text-to-visual attention, attention-head weights, and value-vector norms. It then adaptively balances distinctiveness and representativeness during token selection according to the input’s visual redundancy and task support.

Critical strengths
  1. The paper systematically analyzes how pruning preferences vary across models and tasks, motivating adaptive combinations of selection strategies.
  2. Extensive experiments and ablations provide strong empirical support for the method’s effectiveness.
  3. The method achieves substantial inference speedups while largely preserving model performance.
Critical weaknesses
  1. The performance and computational benefits of multi-stage pruning depend not only on the final number of retained tokens, but also on where pruning occurs and how many tokens are retained at each stage. The paper does not clearly specify these configurations or adequately explain whether the baselines use the same pruning layers, average per-layer token budgets, or comparable computational costs.

  2. The efficiency evaluation in Table 8 compares only against Random and SparseVLM, omitting the main competing methods in the performance tables. Actual inference times for Qwen3-VL and InternVL3 are also not reported.

  3. AgilePruner already adjusts the balance between attention-based and diversity-based token selection according to image complexity and demonstrates that adaptive mixing outperforms fixed mixing. The paper’s claim to be the first to adopt this approach is therefore not adequately justified. An experimental comparison with AgilePruner is also missing.

  4. The citation for AgilePruner appears incorrect: the corresponding bibliography entry is instead When Token Pruning Is Worse Than Random: Understanding Visual Token Information in VLLMs.

Questions for authors

Please refer to weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8csu: Q1

TO Reviewer 8csu

We sincerely thank Reviewer 8csu for the careful review, the clear summary, and the positive assessment of our systematic analysis of pruning preferences, the thorough experimental ablations, and the significant inference speedup.

Q1: The performance and computational benefits of multi-stage pruning depend not only on the final number of retained tokens, but also on where pruning occurs and how many tokens are retained at each stage. The paper does not clearly specify these configurations or adequately explain whether the baselines use the same pruning layers, average per-layer token budgets, or comparable computational costs.

1. Pruning layer positions and per-stage budget allocation. SATOR adopts three-stage progressive pruning. Taking LLaVA-1.5-7B (32 Transformer layers in total, with layer indices 0–31) as an example, pruning is performed at layers 1, 10, and 15. The initial number of visual tokens is 576. In the 64-token setting, the layer-averaged token retention satisfies:

576+9×95+5×52+17×2132=64\frac{576 + 9 \times 95 + 5 \times 52 + 17 \times 21}{32} = 64

The other backbones (LLaVA-NeXT, Qwen3-VL, InternVL3) likewise follow the mainstream setting in this field (e.g., SparseVLM and PDrop) and complete the three-stage progressive pruning within the first half of the network (detailed experimental settings are given in the Appendix of the revised version). On the one hand, image information mainly flows into text tokens in the shallow-to-middle layers, so completing pruning within the first half of the network reduces computational cost early while preserving the key cross-modal interactions; on the other hand, since visual redundancy gradually increases with network depth, we adopt progressive pruning so that the number of retained tokens decreases stage by stage with depth, avoiding premature loss of useful visual information in the shallow layers.

2. Definition of the retention ratio and baseline alignment. Throughout the paper, the visual token retention ratio follows a single definition, namely "the average number of visual tokens retained across the layers of the model ÷ the original number of visual tokens". All compared baselines are strictly aligned to the same average retention ratio in one of the following three ways:

  • (1) For baselines whose original papers already report results on the same model at the same average retention ratio, we directly cite the numbers from the original papers (we will mark the source of each entry in the table notes of the revised version);
  • (2) For baselines whose original papers do not report the corresponding setting and that prune inside the LLM, we re-evaluate them using exactly the same pruning layer positions and per-stage retention counts as SATOR, replacing only the scoring or selection criterion, thereby ensuring that the per-layer token counts and theoretical computation are strictly identical;
  • (3) For baselines whose original papers do not report the corresponding setting and that compress once before entering the LLM (at the connector or vision encoder), we directly retain tokens down to the target average before entering the LLM (e.g., a fixed 64 tokens at 11.1%), so that every LLM layer processes the same number of visual tokens and the cross-layer average retention ratio is naturally aligned.
AuthorsOpenReview ↗

Response to Reviewer 8csu: Q2

Q2: The efficiency evaluation in Table 8 compares only against Random and SparseVLM, omitting the main competing methods in the performance tables. Actual inference times for Qwen3-VL and InternVL3 are also not reported. We thank the reviewer for pointing out this omission. Strictly following the evaluation protocol of Table 8 of the main text—measuring the end-to-end total time (Total time) and prefill time (Prefill time) over all 8,910 POPE questions on a single RTX 4090 (FP16, batch size = 1), including the token selection overhead, with all pruning methods aligned to an 11.1% per-layer average retention ratio—we extend the efficiency comparison to all four backbone architectures (adding Qwen3-VL-8B and InternVL3-8B), and include the representative competing baselines CDPruner, AgilePruner, and SCOPE (these methods will also be added to the performance comparison tables in the revised version).

Measured inference time comparison on POPE across the four backbone architectures (mm:ss / h:mm:ss, 11.1% average retention ratio)

LLaVA-1.5-7B

MethodTotal timePrefill time
Full16:399:55
CDPruner12:186:09
AgilePruner12:065:59
SCOPE11:495:42
SATOR (ours)11:465:37

LLaVA-NeXT-7B

MethodTotal timePrefill time
Full46:1137:21
CDPruner22:1612:40
AgilePruner21:5112:18
SCOPE21:1411:42
SATOR (ours)21:0511:31

Qwen3-VL-8B

MethodTotal timePrefill time
Full36:0019:39
CDPruner31:4412:09
AgilePruner31:2611:50
SCOPE30:4911:17
SATOR (ours)30:4311:08

InternVL3-8B

MethodTotal timePrefill time
Full1:25:581:08:19
CDPruner30:1312:33
AgilePruner29:5012:13
SCOPE29:1411:37
SATOR (ours)29:0611:27

The supplementary measurements support two conclusions:

  • Significant practical speedup is also achieved on dynamic high-resolution architectures such as Qwen3-VL and InternVL3: in addition to LLaVA-1.5-7B (Prefill speedup 1.77×1.77\times, end-to-end speedup 1.42×1.42\times) and LLaVA-NeXT-7B (Prefill speedup 3.24×3.24\times, end-to-end speedup 2.19×2.19\times), SATOR compresses the Prefill time on Qwen3-VL-8B from 19:39 to 11:08 (1.76×1.76\times speedup, with the end-to-end time decreasing from 36:00 to 30:43, a 1.17×1.17\times speedup); the benefit is especially prominent on InternVL3-8B, whose dynamic tiling produces longer visual sequences, where the Prefill time is drastically shortened from 1:08:19 to 11:27 (5.97×5.97\times speedup) and the end-to-end total time decreases from 1:25:58 to 29:06 (2.95×2.95\times speedup).
  • Inference latency is better than that of competing methods across all four architectures: at the same 11.1% average retention ratio, SATOR has the shortest Prefill and end-to-end total time on all four backbones—consistently faster than CDPruner (saving 32–71 seconds end to end) and AgilePruner (saving 20–46 seconds), and comparable to and slightly faster than the lightweight SCOPE (saving 3–9 seconds). As shown in Appendix D.1, this benefits from incremental Cholesky updates reducing the distinctiveness gain to O(B2N)O(B^2N), incremental residual updates keeping the representativeness gain at O(BN2)O(BN^2), and the closed-form routing λ=R1+s\lambda = R^{1+s} fully reusing existing intermediate quantities, so that selection is completed with only a single greedy pass.

The revised version will formally extend Table 8 in Section 5.3 with the complete timing comparison of the above four backbones and competing baselines.

AuthorsOpenReview ↗

Response to Reviewer 8csu: Q3(1/2)

Q3: AgilePruner already adjusts the balance between attention-based and diversity-based token selection according to image complexity and demonstrates that adaptive mixing outperforms fixed mixing. The paper’s claim to be the first to adopt this approach is therefore not adequately justified. An experimental comparison with AgilePruner is also missing.

We thank the reviewer for pointing out AgilePruner. We would like to emphasize that the difference between SATOR and AgilePruner lies not in "whether to adapt", but in "based on what state and at what granularity to adapt":

  • AgilePruner adapts according to image complexity (the erank of visual features or the CLS attention entropy), which is question-agnostic image-level adaptation, and it must be calibrated with statistics on the training set.
  • SATOR adapts according to the joint image–question state, including the visual redundancy R and the task support s obtained from text→visual attention. It routes continuously per sample and per stage inside the LLM, without any dataset statistics or calibration constants.

Two phenomena in our analysis cannot be explained by the single signal of image complexity:

  • Cross-architecture reversal: on MME, POPE, SQA, and TextVQA, with exactly the same images and questions, at 33.3% retention, DART leads on all four on LLaVA-1.5, whereas FastV leads on all four on Qwen3-VL (Fig. 1, Tab. 9).
  • Sample-level differences: within a fixed "architecture × task" configuration, preferences still vary from sample to sample (Tab. 1).

On the "first" claim. We accept the reviewer's comment. AgilePruner has already revealed on LLaVA-1.5 that the relative merits of attention-based and diversity-based pruning vary with image complexity, and has verified that adaptive mixing outperforms fixed mixing. The revised version will delimit our contributions more precisely:

  • (i) revealing the ranking reversal of preferences across architectures, as well as per-sample preference differences under a fixed architecture and task, quantified with an oracle;
  • (ii) proposing calibration-free, stage-wise continuous routing inside the LLM based on the joint image–question state.

Meanwhile, we will explicitly acknowledge AgilePruner's prior findings in the introduction and related work. In addition, the reference entry corresponding to AgilePruner in §2.2 is incorrect and will be corrected in the revised version.

AgilePrunerSATOR
Adaptation signalImage complexity: erank of visual features or CLS attention entropy, question-agnosticVisual redundancy R, plus task support s obtained from text→visual attention, question-dependent
Calibrationerank_avg taken from LLaVA training-set statistics, plus a constant 0.01 and an upper bound τ_maxNo dataset statistics, no constants; λ = R^{1+s} is computed in closed form directly from the current input
Adaptation acts onThe similarity threshold τ (removing neighbors within the threshold after attention ranking); or linearly allocating the numbers of the two token types by erankThe continuous weights of the two marginal gains at each greedy step
Selection criterionAttention + deduplicationTask contribution + redundancy removal (distinctiveness) + coverage of not-yet-retained task-relevant information (representativeness)
Pruning positionVision-encoder side, onceMulti-stage inside the LLM, re-estimating the state at each stage
Reviewer_KxMpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training-free visual token pruning for VLMs. Comparing FastV (relevance) and DART (diversity), finding that the best rule changes across architectures, tasks, and budgets. It also finds that each rule wins a different subset of samples in every configuration. The paper then proposes SATOR, scoring tokens by head-weighted relative attention times value norm and greedily selects them with a mix of a DPP-style "distinctiveness gain" and a coverage-style "representativeness gain", weighted per sample and stage from token redundancy and task support. SATOR gets the best average on four VLMs at every budget tested, with the biggest gains at 11.1% retention.

Critical strengths
  1. The finding that neither relevance nor diversity wins everywhere is useful, and it's a fair point that most pruning methods commit to one. Figure 1 shows the preference moving with model, task and budget, and Table 1 shows each rule gets some samples right that the other doesn't.

  2. The method is training-free, doesn't need tuning per model, and is best on average across all 4 models and budgets tested, with measured wall-clock speedups.

Critical weaknesses
  1. Per-sample adaptation: I'm not fully convinced that selection needs to adapt at per-sample level. The Table 1 oracle doesn't have noise baseline, at 11.1% retention any two pruning rules will flip many borderline samples even without a real preference, so a per-sample oracle over them will always show some gain. There's also no check that RR, ss or λ\lambda actually predict which rule a sample prefers, which is the core link of the method. And in Table 6 the per-sample λ\lambda adds less than a point over the best fixed λ\lambda.

  2. Novelty: the distinctiveness gain follows the same conditional DPP formulation as CDPruner [1] (relevance weighted kernel + greedy MAP), using LLM attention as the relevance, and the representativeness gain is a coverage gain similar to SCOPE [2]. MoB [3] also already argued that the balance between prompt alignment and visual preservation should vary by task, and reports similar LLaVA-1.5 numbers at the larger budgets. None of them are compared against. So the main new piece is the per-sample routing weight, which is weakened by the weakness above.

[1] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

[2] SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

[3] Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering

Questions for authors
  1. Which layers does SATOR prune at and how is the budget split across stages? And how are the baselines matched to the same average retention? FastV in Table 9 is significantly worse than SATOR's own importance-only variant at the same retention, so I'm wondering if the Section 3 analysis uses the same pruning setup as SATOR.
  2. Does a sample's DART vs FastV preference actually correlate with its λ\lambda, e.g. DART win rate by λ\lambda quantile? And what's the oracle gain between 2 random pruning seeds? Those would directly test the main claim.
10 comments
AuthorsOpenReview ↗

Response to Reviewer KxMp: Q1(1/2)

TO Reviewer KxMp

We sincerely thank Reviewer KxMp for the careful review, the clear summary, and the positive assessment of our finding on the complementarity between relevance and diversity, the training-free and tuning-free nature of our method, its best performance across models, and its significant practical speedup.

Q1: Which layers does SATOR prune at and how is the budget split across stages? And how are the baselines matched to the same average retention? FastV in Table 9 is significantly worse than SATOR's own importance-only variant at the same retention, so I'm wondering if the Section 3 analysis uses the same pruning setup as SATOR.

1. Pruning layer positions and per-stage budget allocation. SATOR adopts three-stage progressive pruning. Taking LLaVA-1.5-7B (32 Transformer layers in total, with layer indices 0–31) as an example, pruning is performed at layers 1, 10, and 15. The initial number of visual tokens is 576. In the 64-token setting, the layer-averaged token retention satisfies:

576+9×95+5×52+17×2132=64\frac{576 + 9 \times 95 + 5 \times 52 + 17 \times 21}{32} = 64

The other backbones (LLaVA-NeXT, Qwen3-VL, InternVL3) likewise follow the mainstream setting in this field (e.g., SparseVLM and PDrop) and complete the three-stage progressive pruning within the first half of the network (detailed experimental settings are given in the Appendix of the revised version). On the one hand, image information mainly flows into text tokens in the shallow-to-middle layers, so completing pruning within the first half of the network reduces computational cost early while preserving the key cross-modal interactions; on the other hand, since visual redundancy gradually increases with network depth, we adopt progressive pruning so that the number of retained tokens decreases stage by stage with depth, avoiding premature loss of useful visual information in the shallow layers.

2. Definition of the retention ratio and baseline alignment. Throughout the paper, the visual token retention ratio follows a single definition, namely "the average number of visual tokens retained across the layers of the model ÷ the original number of visual tokens". All compared baselines are strictly aligned to the same average retention ratio in one of the following three ways:

  • (1) For baselines whose original papers already report results on the same model at the same average retention ratio, we directly cite the numbers from the original papers (we will mark the source of each entry in the table notes of the revised version);
  • (2) For baselines whose original papers do not report the corresponding setting and that prune inside the LLM, we re-evaluate them using exactly the same pruning layer positions and per-stage retention counts as SATOR, replacing only the scoring or selection criterion, thereby ensuring that the per-layer token counts and theoretical computation are strictly identical;
  • (3) For baselines whose original papers do not report the corresponding setting and that compress once before entering the LLM (at the connector or vision encoder), we directly retain tokens down to the target average before entering the LLM (e.g., a fixed 64 tokens at 11.1%), so that every LLM layer processes the same number of visual tokens and the cross-layer average retention ratio is naturally aligned.

3. FastV in Table 9 and "Importance ranking only" in Table 6 do not use the same importance criterion. FastV directly uses the raw softmax attention weights and averages all attention heads with equal weights, making it highly susceptible to interference from task-irrelevant factors such as Attention Sink, noise from non-task-adaptive heads, and tiny vector norms. In contrast, "Importance ranking only" in Table 6 is a controlled internal ablation of SATOR itself (i.e., directly taking the top-BB by qjq_j), sharing the same three-stage schedule and task contribution qjq_j; and qjq_j integrates four key improvements:

  • (i) relative attention normalized only within the visual-token span and averaged over all text positions, to eliminate Attention Sink and positional bias;
  • (ii) entropy-based head informativeness weights ωh\omega_h;
  • (iii) multiplication by the Value vector norm ∥Vj(h)∥2\Vert{}V_j^{(h)}\Vert{}_2 to capture the actual write-in contribution;
  • (iv) within-image RMS saturation normalization.

In fact, FastV in Table 9 is itself re-run under exactly the same pruning schedule and parameter settings as SATOR. Given identical pruning layers, stage budgets, and parameter settings, "Importance ranking only" achieves a significant advantage over FastV; this gap stems entirely from the difference in how the importance signal is defined, directly validating the effectiveness of the qjq_j design (Table 7 and Appendix Table 12 also quantify the gain of each sub-component of qjq_j item by item).

AuthorsOpenReview ↗

Response to Reviewer KxMp: Q1(2/2)

4. FastV and DART are introduced as diagnostic probes to demonstrate preference reversal. As described in Section 3, FastV and DART represent typical public rules oriented toward relevance and diversity, respectively; in the paper they serve only as diagnostic probes for revealing the phenomenon that "the preference for fixed criteria reverses across architectures, tasks, and samples", rather than as SATOR's own importance baseline (that role is played by "Importance ranking only" in Table 6). Under every model, task, and retention-ratio configuration, FastV and DART are always compared under symmetrically matched pruning settings, so the win/loss relation between them and the conclusion of cross-architecture ranking reversal are fully fair and reliable. We will explicitly add the above pruning layer indices, stage budgets, and baseline alignment details to Section 5.1 and the Appendix of the revised version.

AuthorsOpenReview ↗

Response to Reviewer KxMp: Q2(1/3)

Q2: Does a sample's DART vs FastV preference actually correlate with its λ, e.g. DART win rate by λ quantile? And what's the oracle gain between 2 random pruning seeds? Those would directly test the main claim.

Correlation test between a sample's DART vs. FastV preference and the routing weight λ\lambda A sample's preference for the diversity rule (DART) versus the relevance rule (FastV) has a clear and monotonic positive correlation with the pre-pruning routing weight λ\lambda: the higher λ\lambda, the more DART dominates; the lower λ\lambda, the more FastV dominates. This conclusion is doubly corroborated by the existing cross-architecture, cross-task, and causal-reversal ablation evidence in the main text, and by a sample-level win-rate test by λ\lambda quantile within the same configuration:

1. Existing multi-granularity and causal evidence in the main text and Appendix.

  • Cross-architecture level (as shown in Fig. 1(a) and Table 5 of the main text, and Appendix Table 9): Table 5 of the main text shows that in Stage 1, where the pruning magnitude is the largest, LLaVA-1.5 has extremely high visual redundancy (R=0.95R=0.95), with a corresponding routing weight as high as λ=0.90\lambda=0.90 (three-stage mean 0.630.63), significantly higher than Qwen3-VL (Stage-1 λ=0.57\lambda=0.57, mean 0.530.53) and InternVL3 (Stage-1 λ=0.52\lambda=0.52, mean 0.530.53). Correspondingly, as shown in Section 3, Fig. 1(a) of the main text, and Appendix Table 9, the diversity-oriented DART leads FastV by a large margin of 9.3–18.8 points on the high-λ\lambda LLaVA-1.5; on the low-λ\lambda InternVL3, its advantage drops sharply to 0.8–1.9 points, and on Qwen3-VL (at 33.3% and 22.2% retention) it even reverses, with the relevance-oriented FastV leading by 5.8 and 3.0 points.
  • Cross-task level within the same architecture (as shown in Table 1 and Fig. 1(b) of the main text, and Appendix B, Table 10): Taking Qwen3-VL-8B as an example, the three-stage average λ\lambda of the four benchmarks computed from Appendix Table 10 increases strictly in the order TextVQA (0.5000.500) << POPE (0.5130.513) << MME (0.5270.527) << ScienceQA (0.5570.557) (as pointed out in Appendix B, the chart-type images in ScienceQA contain large uniform backgrounds, so RR and λ\lambda are the highest at every stage). Meanwhile, the DART win rate on decisive samples (i.e., DART/(DART+FastV)\text{DART}/(\text{DART}+\text{FastV})) computed directly from Table 1 of the main text at 11.1% retention increases strictly monotonically in exactly the same order, 44.2% << 49.2% << 51.6% << 63.2% (rank correlation coefficient 1.001.00), and the task score differences in Fig. 1(b) of the main text likewise flip from FastV-favored (TextVQA −4.5-4.5, POPE −5.7-5.7) to DART-favored (MME +3.2+3.2, ScienceQA +4.2+4.2).
  • Causal ablation of the routing direction (as shown in Section 5.3 and Table 6 of the main text): If sample preference were unrelated to λ\lambda, flattening or reversing λ\lambda should not affect performance. However, as shown in Table 6 of the main text, merely flattening the between-sample differences in λ\lambda by fixing it to the stage mean from Table 5 (Stage-wise constant) already lowers the average performance by 0.63 points (by 1.04 points on LLaVA-1.5); if the monotonic direction of λ\lambda with respect to redundancy RR or support ss is reversed (Reversed R / s direction), performance degrades consistently on all three architectures, with the mean dropping to 94.84% and 95.50% (significantly below SATOR's 96.01%; reversing RR even falls below the 95.39% of the pure-deduplication variant), confirming at the causal level that sample preference is consistent with the direction of λ\lambda.

2. Sample-level DART win rate by λ\lambda quartile within a fixed architecture and task.

As shown in Appendix B and Table 10, after fixing the architecture, stage, and task, the 5th–95th percentile span of λ\lambda across samples within a single benchmark still reaches 0.09–0.34, which is uniformly larger than the span of cross-task means at the same stage (only 0.04–0.09). To directly test whether the true preference of samples within these configurations varies with λ\lambda, on the same 12 (architecture, task) configurations as Table 1 of the main text, totaling 54,889 paired samples (11.1% retention), we independently divided the samples within each (architecture, task) configuration, from low to high by their stage-averaged routing weight λ(x)\lambda(x), into four equal quantile bins, Q1 (lowest 25%) to Q4 (highest 25%), and computed the DART win rate among decisive samples (uDART(x)≠uFastV(x)u_{\text{DART}}(x) \neq u_{\text{FastV}}(x)), where the full-set baseline is derived from Table 1 of the main text (the per-architecture rows are consistent with Table 6 of the main text and Appendix C, taking an equally weighted average over the four benchmarks).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.