- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how the relative performance of relevance-based and diversity-based visual token pruning changes across models and tasks.
It proposes SATOR, a training-free method that combines determinant-based distinctiveness with task-weighted coverage. Task contribution is estimated from relative attention and value norms. A routing weight computed from input redundancy, task support, and the token budget balances the two gains.
The experiments cover 4 VLM backbones, with routing and component ablations on three models and inference-time measurements on the LLaVA models.
Critical strengths
-
The cross-model analysis and paired sample results address a useful practical problem. They show that average method rankings can conceal different outcomes across inputs, providing evidence against choosing a pruning method solely from aggregate performance.
-
The routing rule uses the current input without additional training or external calibration statistics. Combining redundancy reduction with coverage of unretained task information gives the method a coherent design and makes it potentially easier to transfer across settings.
-
The routing ablations provide meaningful evidence beyond the main benchmark tables. Table 6 compares individual gains, fixed weights, stage-wise mean weights, and reversed state dependencies. SATOR improves the mean normalized score from 95.57 for the best shared constant to 96.01, supporting its value over the tested controls.
-
The efficiency results include selection overhead. On LLaVA-NeXT, the reported 2x total inference speedup and 6x KV-cache reduction accompany 97.5% average performance retention. These measurements support a useful accuracy-efficiency trade-off in the tested setting.
Critical weaknesses
-
The novelty is not sufficiently distinguished from some closely related work. AgilePruner already studies input-dependent attention/diversity preferences and image-aware adaptation. CDPruner uses relevance-weighted determinant selection, while SCOPE combines saliency and marginal coverage. SATOR differs in its contribution scores and state-dependent mixture, but the paper needs to explain what additional insight these differences provide. Section 2.2 also incorrectly attributes AgilePruner to Wang et al. CrisPrune, which combines text relevance, visual saliency, and determinant-based diversity, is not discussed. To summarize, I found the differences of this work with previous works listed here are mostly technical designs rather than insights.
-
The performance comparisons omit AgilePruner, CDPruner, and SCOPE. Since these methods directly address the proposed adaptation and selection mechanisms, their absence leaves the practical value of SATOR's added machinery unresolved. Competitive quality with reduced calibration would be a meaningful contribution, but comparisons with the current baselines do not establish that advantage.
-
The compression schedule and budget accounting are unclear. Tables 5 and 10 use three stages, but their decoder layers and retained counts are not specified. Table 2 describes retained token counts, whereas Table 7 explicitly uses average per-layer retention. These quantities are different, and the schedule affects both accuracy and computation. Without the layer-wise counts and matching rules for baselines, the main comparisons are difficult to assess.
-
The evidence for the necessity of sample conditioning is weaker than the conclusion suggests. Complementary errors and an oracle gain show potential headroom, but do not establish that the proposed states predict the useful selection behavior. Table 6 supports the tested adaptive rule, yet its best fixed weight is shared across architectures and its stage-wise constants are means rather than optimized controls. The results therefore leave some uncertainty about the benefit of matching each routing weight to its own sample.
Questions for authors
-
Please clarify the pruning layers, retained counts at each stage, and the definition of the reported retention ratio for every backbone. How were one-shot and multi-stage baselines matched?
-
Please clarify the methodological differences from AgilePruner, CDPruner, and SCOPE. Could a focused comparison under matched budgets establish the added quality, efficiency, or calibration benefit? One representative setting would help resolve this concern without requiring a broader benchmark suite.
-
To validate the sample-conditioning claim, does the proposed routing outperform weights shuffled among samples within the same model, task, and stage? Alternatively, can existing evidence show that the states predict the relative utility of the two gains? This would help distinguish sample matching from differences in average routing behavior.
13 comments
Response to Reviewer WAJg: Q2(1/3)
Q2: Please clarify the methodological differences from AgilePruner, CDPruner, and SCOPE. Could a focused comparison under matched budgets establish the added quality, efficiency, or calibration benefit? One representative setting would help resolve this concern without requiring a broader benchmark suite.
We thank the reviewer for the suggestion. We would like to emphasize that the fundamental difference between SATOR and these three methods lies in "what determines the balance between the two types of criteria":
- AgilePruner determines it by image complexity, relying on training-set statistics for calibration;
- CDPruner and SCOPE use a fixed combination for all inputs;
- SATOR, inside the LLM, computes λ = R^{1+s} in closed form from the current sample's redundancy R and task support s, routing continuously between the two corrections per sample and per stage, without any calibration.
All of SATOR's other design differences serve this point:
- the contribution score q must be comparable across images and stages so that Σq/B can define the support s;
- the state must be read from inside the model to reflect differences across architectures;
- pruning must be performed in stages inside the LLM so that the state can be re-estimated as redundancy changes.
| AgilePruner | CDPruner | SCOPE | SATOR | |
|---|---|---|---|---|
| Importance signal | Vision encoder attention (CLS attention on LLaVA) | Cosine between visual and instruction embeddings before the LLM, min-max normalized | CLS attention from the penultimate ViT layer (question-agnostic) | Text→visual relative attention inside the LLM × Value norm, weighted by head informativeness, RMS saturation normalized |
| Selection operator | Rank by attention, removing neighbors within a similarity threshold | Conditional DPP greedy MAP | Greedy selection by coverage gain × saliency^α | Greedy selection mixing the distinctiveness gain (DPP) and the representativeness gain (q-weighted coverage) by λ |
| Balancing | Adjust the threshold by erank, or linearly allocate the numbers of the two token types by erank | Fixed | Fixed (multiplicative combination) | λ = R^{1+s}, computed per sample and per stage |
| Calibration / hyperparameters | Average erank of the LLaVA training set, constant 0.01, upper bound τ_max | None in the main experiments; the balance factor θ needs hand-tuning per dataset; the text embedding source needs to be chosen per architecture | α (default 1.0, selected via ablation) | None |
| Structural dependency | Vision encoder attention | A paired text encoder, or LLM word embeddings | CLS token of the vision encoder | Only LLM-internal attention and hidden states |
| Pruning position | Vision-encoder side, once | Before the LLM, once | After the vision encoder, once | Multi-stage inside the LLM, re-estimating the state at each stage |
Differences from AgilePruner
- Commonality: both let the balance between attention and diversity vary with the input.
- The adaptation signal differs. AgilePruner relies on image complexity (the erank of visual features or attention entropy), which is question-agnostic; the same image paired with different questions yields the same selection strategy. SATOR's support s is obtained from text→visual attention and is question-dependent. When s is removed and only visual redundancy is kept (λ = R), performance drops from 96.01% to 95.35% (Table 6).
- The object of adaptation differs. AgilePruner adjusts the deduplication strength in "attention ranking + threshold deduplication", where the threshold grows with the token's rank and with the ratio erank_input / erank_avg, subject to an upper bound; it can also linearly allocate the numbers of the two token types by erank, i.e., a hard split. SATOR continuously weights the two marginal gains at each greedy step, and includes the dimension of "covering not-retained task evidence" that AgilePruner lacks.
- The calibration requirements differ. AgilePruner needs the average erank computed from LLaVA training-set statistics as a reference, as well as hand-set constants and an upper bound, and the statistics must be recomputed when changing models or data distributions. SATOR's R, s, and λ are all obtained in closed form from the current input.
Response to Reviewer WAJg: Q2(2/3)
Differences from CDPruner
- Commonality: the distinctiveness gain (Eq. 6) has the same form as CDPruner's conditional DPP greedy gain, i.e., a relevance-weighted kernel plus greedy MAP. The revised version will explicitly state this connection.
- The role of DPP differs. In CDPruner, the conditional DPP is the entire objective; in SATOR, it is only one of two marginal gains, with its weight determined per sample by λ. Setting λ ≡ 1, i.e., degenerating into a CDPruner-style objective, gives 95.39%, 0.62 pp lower than SATOR (Table 6).
- The definition and scale of the relevance score differ. CDPruner computes the cosine between visual and instruction embeddings before the LLM and then applies min-max normalization, so every image necessarily contains tokens taking 1 and 0, and only the relative order is preserved. SATOR's q comes from inside the LLM: it uses relative attention to avoid attention sinks, the Value norm to capture the actual amount written in, and head entropy weights to highlight heads that vary with the text, and it is RMS saturation normalized so that it is comparable across images and stages; only then can Σq/B serve as the support s. Switching to max normalization lowers performance to 95.81% (Table 12).
- The kernel function differs. SATOR first mean-centers the hidden states to eliminate anisotropy, and then uses ((1+cos)/2)² to suppress weak similarities; CDPruner uses the raw cosine.
- The balancing mechanism differs. CDPruner notes in its appendix that the optimal balance factor θ varies across benchmarks and needs to be hand-tuned per dataset; SATOR turns it into a state quantity computed in closed form per sample.
Differences from SCOPE
- Commonality: the representativeness gain and SCOPE's coverage gain are both facility-location-style coverage gains. The revised version will add the citation.
- The source of saliency differs. SCOPE uses ViT CLS attention, which is question-agnostic; SATOR uses the question-dependent task contribution q.
- The object and form of coverage differ. SCOPE covers all visual tokens by the maximum cosine. SATOR weights by q, covering "the not-yet-retained task-relevant information", and adopts a squared-residual form (Eq. 7) so that high-contribution tokens far from the selected set receive larger gains.
- The combination differs. SCOPE combines the two multiplicatively via Δ(v;S)·A_v^α, with α fixed to 1.0 for all inputs. SATOR first normalizes the two gains separately over the candidate set and then mixes them additively by λ; 1−λ increases with the support s, so the weight of coverage is increased only when the task evidence exceeds the budget. Pure coverage (λ ≡ 0) yields only 93.74%, and on InternVL3 it is even lower than pure importance ranking (92.92% vs. 95.05%), indicating that coverage is not beneficial for all inputs.
- The scope of applicability differs. SCOPE relies on the CLS token and cannot be directly applied to architectures whose vision encoder has no CLS token, such as Qwen3-VL. Although InternVL3's vision encoder has a CLS token, each dynamic tile has its own, and the visual tokens are further merged via a 2×2 pixel shuffle, requiring additional modification. SATOR uses only LLM-internal signals and can be used directly on all three architectures.
In summary, the three methods either adjust the balance by image complexity with the help of training-set statistics (AgilePruner), or adopt a fixed combination for all inputs (CDPruner, SCOPE). SATOR instead reads the current sample's redundancy and task support from inside the LLM and routes continuously between redundancy removal and coverage per sample and per stage, without any dataset statistics or hyperparameter calibration.
Response to Reviewer WAJg: Q1
Q1: Please clarify the pruning layers, retained counts at each stage, and the definition of the reported retention ratio for every backbone. How were one-shot and multi-stage baselines matched?
We thank the reviewer for pointing this out; the original paper did not describe these configurations clearly enough. Below we explain the three items in the question one by one, and the revised version will add all configurations to Section 5.1 and the Appendix.
1. Pruning layers and per-stage retention counts for each backbone. SATOR adopts three-stage progressive pruning on all backbones, and all three pruning steps are completed in the first half of the LLM. Taking LLaVA-1.5-7B as an example (32 LLM layers in total, layer indices 0–31), pruning is performed at the inputs of layers 1, 10, and 15. The original number of visual tokens is 576. In the 64-token setting, the three stages retain 95, 52, and 21 tokens in turn: layer 0 processes 576, layers 1–9 process 95, layers 10–14 process 52, and layers 15–31 process 21. The per-layer average is exactly 64:
32576+9×95+5×52+17×21=64 The other backbones (LLaVA-NeXT, Qwen3-VL, InternVL3) likewise follow the mainstream setting in this field (e.g., SparseVLM and PDrop) and complete the three-stage progressive pruning in the first half of the network (detailed experimental settings will be added to the Appendix of the revised version). There are two reasons:
2. Definition of the retention ratio, and why percentages are reported for some backbones. All backbones and all methods throughout the paper use the same definition: the fraction of the original visual tokens retained at each LLM layer, averaged over the layers. Let the original number of visual tokens of a sample be Nv, the LLM have L layers in total, and the number of visual tokens input to layer ℓ be Nℓ; then
Retention ratio=L1∑ℓ=0L−1NvNℓ
The two reporting formats correspond to the same definition:
Therefore, 64 tokens on LLaVA-1.5 and 11.1% on Qwen3-VL represent the same compression strength.
3. How one-shot pruning baselines and multi-stage pruning baselines are matched. All baselines are aligned to the same average retention ratio defined above, i.e., the same average number of visual tokens processed per LLM layer. There are three specific cases: