acceptodds
Under review as a conference paper at ICLR 2027

ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning

Abstract

Recent advances in Vision-Language Models (VLMs) enable large language models (LLMs) to process high-resolution images, significantly improving real-world multimodal understanding. However, this capability introduces a large number of vision tokens, incurring substantial computational overhead. To mitigate this issue, various vision token compression methods have been proposed. Existing methods often estimate different sources of visual redundancy using learned representations or fixed pruning schedules. We propose ERASE, an adaptive two-stage framework that separates image-level redundancy removal from instruction-dependent token pruning. Stage 1 derives image-dependent token retention from lightweight raw-image statistics, while Stage 2 progressively removes instruction-irrelevant tokens across decoder layers. Experiments demonstrate substantial token reduction while preserving accuracy: on Qwen2.5-VL-7B, ERASE retains 95.70% of the original model's accuracy at 25% token retention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision token pruning, Efficient multimodal inference, Vision language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wrQyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ERASE is a training-free, two-stage visual-token pruner. Stage 1, before the LLM, smooths a luminance map and keeps a patch if any pixel's Sobel gradient exceeds τE = (1 - τ)·median(Ỹ) or if its 256-bin intensity entropy exceeds the mean patch entropy H̄ (Eqs 1-5), so its retention r1 varies per image. Stage 2 ranks survivors by instruction-to-vision attention a_v (Eq 7), adding the Stage-1 cues with weight w at layer 2 (Eq 8) and using attention alone near two-thirds depth; Algorithm 1 sets both keep ratios from r1 and the target layer-averaged retention. With τ = 0.45 and w = 0.2 fixed, ERASE is tested on five Qwen and InternVL3 backbones, eight benchmarks and up to nine baselines at 45/35/25% retention, reporting the best average at every tier.

Critical strengths
  1. ERASE avoids the encoder-attention memory cost at native resolution. Because Stage 1 scores tokens from luminance statistics rather than encoder attention, ERASE prunes Qwen2.5-VL-7B inputs of up to 16K visual tokens at 19.15 GB peak memory, while the evaluated implementations of AgilePruner, VScan and Nüwa need 37.38 to 39.59 GB even on downscaled inputs (Table 8). On encoders without a [CLS] token, like Qwen2.5-VL's, that is the clearest practical separation from [CLS]-dependent pruners such as these three.

  2. The accuracy lead over the [CLS]-dependent baselines survives the one comparison free of the resolution confound. On InternVL3-8B, whose [CLS] token lets AgilePruner, VScan and Nüwa run without OOM (Sec 4.2), ERASE has the best average at all three tiers, 72.39 against 69.51 for Nüwa at 25% retention, a 2.88-point margin (Table 3). This is the decisive reference point for me, because on Qwen those baselines see downscaled inputs, which mixes pruning quality with input resolution.

  3. The evaluation is broad and its numbers check out. ERASE has the top average at every tier on all five backbones against up to nine baselines (Tables 1-3, 6-7), and every average and stated margin I recomputed matches the printed value to rounding. The result holds at the most aggressive tier: at 25% retention on Qwen2.5-VL-7B, ERASE keeps 95.70% of base accuracy (Table 1). A lead this consistent and this checkable is what makes the efficiency-accuracy claim credible.

Critical weaknesses
  1. The novelty is a combination of known parts, and the new Stage-2 piece adds under a point in the full system. Stage 1 builds on Sobel and Canny-style thresholding (Eqs 1-4), and iLLaVA, VScan and Nüwa already prune before and inside the LLM (Sec 2.2.2). PruneSID, a compared baseline, already adapts its compression ratio to image complexity. The 3.14 points credited to visual guidance (Sec 3.3) are retained accuracy with Stage 1 disabled (Fig 7 caption). With Stage 1 on, task accuracy moves from 70.10 to 71.01, +0.91 (Table 9). Moving Table 9 into the main text and scoping the claim costs no experiments.

  2. The paper claims matched retention, but ERASE's per-tier realized retention is never reported and sometimes falls below the nominal tier. Per A.1 (L675-677), Stage 2 is skipped whenever Stage 1 already meets the budget, so "the reported retention ratios serve as upper bounds for ERASE." On Qwen2.5-VL-7B, Stage 1 alone keeps 41.28% of DocVQA tokens on average (Table 4), so ERASE's 45% DocVQA cell in Table 1 must run below budget on average. This may understate ERASE rather than flatter it, but readers cannot compare the operating points at equal cost. Reporting realized retention per cell, and matching baselines to it, would close this.

  3. The Qwen headline margin leaves out the strongest baseline, which the appendix shows is nearly tied. Sec 4.2 reports +3.95 over iLLaVA as the margin over the strongest Qwen2.5-VL-7B baseline. Nüwa, the strongest baseline on InternVL3 at every tier (Table 3), appears on Qwen only in appendix Table 8, on downscaled inputs, because its evaluated implementation needs full attention maps without a [CLS] token (A.3.2). Even so it reaches 74.85 at 25%, 0.72 below ERASE (Table 8). Reporting this margin in the main text, and scoring Nüwa without the full map as iLLaVA does (Sec 2.2.2), would show whether the gap survives.

Questions for authors
  1. Please report ERASE's realized layer-averaged retention per cell in Tables 1-3. How far below each nominal tier does it fall (for example DocVQA at 45%), and does the ranking hold when baselines run at the realized value?
  2. Did Table 8 follow Nüwa's own Qwen2.5-VL setup (its paper reports Qwen2.5-VL-7B results) or your adaptation? Could its attention scores be computed via the FlashAttention-2 normalizer, as for iLLaVA, to run it at full resolution?
  3. On Qwen3-VL-8B, ERASE trails PruneSID on MathVista at all three tiers (47.10 vs 50.40 at 25%, Table 2). Is there a systematic cost on math and reasoning tasks, and would you scope the OCR-gains statement in Sec 4.2 accordingly?
  4. SparseVLM (ICML 2025, arXiv 2410.04417) rates visual tokens from decoder self-attention and needs no encoder [CLS] token, so it fits both Qwen and InternVL3. It is neither cited nor compared; can you add it on one backbone?
  5. Algorithm 1 holds each image at the target unless Stage 1 falls below it (L714-720), so Tables 1-3 adapt only the stage split. Table 4's +0.52 adaptive gain omits Stage 2 (L406-407); can Table 9 test a fixed Stage 1?
7 comments

We appreciate the reviewer for the thoughtful feedback. We address each concern and suggestion below with additional experiments.

W1/Q5. Component contributions and image-adaptive pruning

We agree that Table 9 should be moved to the main text to clarify the contribution of each component in the full system. We have moved it to the main text as Table 6 (Table wrQy-1 below) and revised Sec. 3.3 to distinguish the two results: visual guidance improves retained accuracy by 3.14 points with Stage 1 disabled (Fig. 7c), while improving task accuracy by 0.91 points in the full system (L309-311). As shown in Table wrQy-1, visual guidance provides smaller gains when Stage 1 is enabled, likely because Stage 1 has already removed visually redundant tokens using the same image statistics. We now clarify this distinction in the revised manuscript (L512–514).

As the reviewer correctly notes, ERASE adapts how the token budget is distributed across pruning stages rather than the overall target budget, except when Stage 1 alone falls below the target. To evaluate the contribution of this adaptive allocation in the full system, we added a fixed Stage 1 variant (†) to Table 6 (Table wrQy-1). Adaptive Stage 1 improves average accuracy from 70.70% to 71.01% (+0.31 points). Together with the Stage 1-only improvement of 0.52 points in Table 4, these results demonstrate a consistent, albeit modest, benefit from image-adaptive retention. We have revised the manuscript to clarify this contribution (L519–524).

Regarding novelty, we agree that Stage 1 builds on established techniques such as edge detection and patch entropy. However, ERASE does not simply apply these operators independently. Instead, it combines complementary visual cues with image-relative thresholds to determine both which tokens and how many tokens to retain before the LLM. This adaptive selection is further integrated with hierarchical instruction-aware pruning, where Stage 1 retention determines the subsequent pruning schedule. Unlike iLLaVA, VScan, and Nüwa, which rely on vision-encoder attention or learned visual representations, ERASE uses lightweight raw-image statistics for early pruning. This avoids the additional latency observed with iLLaVA (2671 vs. 2941 ms, Table 5) and the encoder-attention memory overhead encountered by VScan and Nüwa on Qwen2.5-VL, which lacks an encoder [CLS] token (Table 9 of the revised manuscript). While PruneSID also adapts compression to image complexity, ERASE achieves higher Stage 1 accuracy at matched retention ratios (Table 4) with lower selection overhead (Fig. 8c and Table 5).

Table wrQy-1. End-to-end component ablation on Qwen2.5-VL-7B

Stage 1ShallowVGDeepChartQAInfoVQARWQAMMStarAvg.
×✓××73.8458.5365.2356.6063.55
×✓✓×75.8061.7466.8058.3365.67
×✓×✓73.6465.2367.7158.0066.15
×✓✓✓76.6467.2367.7161.8768.36
✓✓××79.6867.2267.9760.0068.72
✓✓✓×79.6868.0867.4561.0069.05
✓✓×✓79.4871.4868.2461.2070.10
✓✓✓✓80.2071.8369.4162.6071.01
✓†✓✓✓79.6872.0569.1561.9370.70

W2/Q1. Realized retention and comparison fairness (1/2)

We thank the reviewer for this careful observation. As the reviewer correctly notes, Stage 2 is skipped when Stage 1 already retains fewer tokens than the target, so ERASE's realized retention can fall below the nominal retention ratio. We agree that evaluation at these retention is needed to compare baselines at equal cost, and we address it below.

To clarify the actual operating points, Tables wrQy-2 to wrQy-4 report ERASE's realized layer-averaged retention for each benchmark and target ratio in Tables 1–3, respectively. Bold values indicate deviations of at least 2 percentage points from the target.

Table wrQy-2. Realized layer-averaged retention (%) per cell of ERASE in Table 1, Qwen2.5-VL-7B

TargetTextVQAChartQADocVQAInfoVQAMathVistaMMBenchRWQAMMStar
45%44.7043.8937.0844.4140.2143.4944.8742.23
35%34.9334.8432.2634.9033.6934.6134.9734.28
25%24.9624.9624.2924.9924.7824.9124.9724.88

Table wrQy-3. Realized layer-averaged retention (%) per cell of ERASE in Table 2, Qwen3-VL-8B

TargetTextVQAChartQADocVQAInfoVQAMathVistaMMBenchRWQAMMStar
45%44.7544.7437.5844.4242.1044.1244.8842.93
35%34.9134.8732.4134.9134.2534.6134.9634.40
25%24.9324.8824.3224.9924.8224.7024.9624.76

Table wrQy-4. Realized layer-averaged retention (%) per cell of ERASE in Table 3, InternVL3-8B

TargetTextVQAChartQADocVQAInfoVQAMathVistaMMBenchRWQAMMStar
45%44.7644.2240.1444.9341.3643.9744.9243.24
35%34.9434.9433.6434.9934.0134.7834.9734.60
25%24.9724.9724.7124.9924.8124.9424.9724.92

As anticipated by the reviewer, the largest deviation occurs on DocVQA at 45% target retention, where the realized retention is 37.08%, 37.58%, and 40.14% on Qwen2.5-VL-7B, Qwen3-VL-8B, and InternVL3-8B, respectively. Most other configurations remain close to their target ratios, with all cells at 25% retention deviating by less than 1 percentage point.

W2/Q1. Realized retention and comparison fairness (2/2)

To assess the impact of these differences on comparison fairness, we re-evaluated the baselines at ERASE's realized retention for all cells with deviations of at least 2 percentage points. Tables wrQy-5 to wrQy-7 summarize the results. Values in parentheses indicate the original nominal-tier results, while the other values represent the re-evaluated results. Cells marked with † retain their original values because ERASE's realized retention differs from the target by less than 2 percentage points. The average accuracy (Avg.) is recomputed across all eight benchmarks using the updated results.

Table wrQy-5. Re-evaluation of Table 1 with matched realized layer-averaged retention

RetainMethodDocVQAMathVistaMMStarAvg.
100%Base94.8265.1065.5378.97
45%FastV91.28 (93.19)63.20 (63.20)61.20 (62.00)76.00 (76.34)
DART79.68 (85.35)61.00 (62.00)62.67 (62.53)71.43 (72.25)
CDPruner82.53 (85.41)56.20 (56.70)61.73 (63.07)69.76 (70.35)
PruneSID85.88 (90.32)61.20 (61.30)63.80 (62.73)73.39 (73.82)
iLLaVA91.38 (93.09)63.20 (64.30)62.47 (62.27)76.31 (76.64)
SparseVLM87.95 (91.05)61.40 (62.90)61.73 (62.47)74.03 (74.70)
ERASE94.2264.2064.1377.63
35%FastV88.58 (90.27)62.30†60.20†73.85 (74.06)
DART75.25 (78.17)57.70†61.33†67.72 (68.09)
CDPruner80.44 (81.57)55.40†62.00†67.68 (67.82)
PruneSID83.85 (86.07)60.40†62.20†70.87 (71.14)
iLLaVA90.20 (90.91)60.70†62.40†74.82 (74.91)
SparseVLM84.23 (86.75)59.40†60.40†70.49 (70.80)
ERASE94.1263.7063.9377.18

Table wrQy-6. Re-evaluation of Table 2 with matched realized layer-averaged retention

RetainMethodDocVQAMathVistaMMStarAvg.
100%Base95.7569.1066.6079.59
45%FastV74.04 (81.92)47.10 (49.00)53.07 (55.07)62.93 (64.40)
DART75.91 (82.69)55.80 (56.10)56.00 (59.53)66.93 (68.26)
CDPruner65.89 (72.63)40.50 (42.60)50.60 (52.93)60.06 (61.45)
PruneSID91.53 (93.46)58.60 (59.90)61.20 (60.40)73.45 (73.76)
iLLaVA91.65 (93.02)44.00 (45.40)60.07 (60.53)70.94 (71.35)
ERASE94.7754.7061.5374.88
35%FastV66.05 (70.07)43.60†49.00†57.05 (57.55)
DART69.35 (73.30)49.80†55.07†62.14 (62.64)
CDPruner61.12 (63.57)38.00†49.33†55.92 (56.23)
PruneSID88.59 (90.46)52.80†60.33†68.96 (69.19)
iLLaVA90.64 (90.74)44.30†57.93†68.81 (68.82)
ERASE94.2951.0059.0772.82

Table wrQy-7. Re-evaluation of Table 3 with matched realized layer-averaged retention

RetainMethodDocVQAMathVistaAvg.
100%Base91.9868.8078.77
45%FastV77.83 (82.17)64.50 (64.00)74.06 (74.54)
DART73.84 (77.59)67.80 (66.30)71.51 (71.80)
CDPruner61.25 (66.87)56.30 (57.80)64.53 (65.42)
PruneSID69.40 (76.02)66.00 (66.10)70.42 (71.26)
iLLaVA65.92 (68.65)59.00 (58.40)64.81 (65.08)
VScan87.22 (88.73)60.30 (60.90)75.19 (75.45)
Nüwa85.98 (87.13)65.00 (65.90)75.82 (76.07)
AgilePruner80.97 (83.83)62.00 (63.40)73.31 (73.85)
ERASE87.9566.4076.37

As expected, most baseline accuracies decrease under the reduced token budgets. Importantly, ERASE continues to achieve the highest average accuracy across all three models and retention tiers after accounting for the realized retention differences. The relative rankings among baselines also remain nearly unchanged, except for a minor swap between iLLaVA and CDPruner on InternVL3-8B. These results confirm that the main conclusions of Tables 1–3 remain valid when the larger retention discrepancies are accounted for. We have incorporated these analyses into Appendix A.3.8 (Tables 15–18) of the revised paper.

Reviewer_Tw9DOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ERASE, a training-free, adaptive two-stage framework for visual token pruning in vision-language models. The first stage uses image-level edge and luminance-entropy statistics to select informative visual tokens before they enter the language model, adapting the retention ratio to each image. The second stage performs hierarchical, instruction-aware pruning within the decoder: shallow-layer selection combines instruction attention with visual cues, while deep-layer selection relies on instruction relevance alone. The pruning schedule is adjusted to meet a target layer-averaged token budget. Experiments across Qwen2.5-VL, Qwen3-VL, and InternVL3 demonstrate favorable accuracy–efficiency trade-offs on eight multimodal benchmarks. On Qwen2.5-VL-7B, ERASE preserves 95.70% of the baseline’s average benchmark score at 25% layer-averaged visual token retention, while achieving a 1.78× end-to-end speedup in the evaluated 16K-token setting.

Critical strengths

Strengths

  1. Well-motivated and interpretable design. ERASE separates image-level redundancy removal from instruction-dependent token selection. Its use of lightweight edge and luminance-entropy statistics provides an interpretable, model-independent criterion for initial pruning, while hierarchical instruction-aware selection progressively refines the retained visual information.

  2. Broad and systematic evaluation. The paper evaluates multiple model families and sizes across eight benchmarks covering text recognition, document and chart understanding, visual reasoning, and general visual question answering. Comparisons with recent baselines, component ablations, and analyses of pruning locations and hyperparameters provide substantial empirical support for the design choices.

  3. Strong accuracy–efficiency trade-offs. ERASE achieves the highest average benchmark score among the evaluated methods at the reported matched retention budgets across the tested architectures. The latency and KV-cache measurements complement the accuracy results with practical evidence of efficiency, particularly for high-resolution inputs.

  4. Training-free operation and clear presentation. The method requires no additional model training and avoids materializing full visual-encoder attention maps for its initial pruning stage. The paper is well organized, and the equations, pseudocode, and supplementary implementation details make the approach accessible and facilitate reproduction.

Critical weaknesses

The evaluation focuses on layer-averaged token retention ratios of 25%–45%. Exploring more aggressive compression, such as 10% retention where feasible, would help characterize the method’s operating limits and robustness, particularly on detail-sensitive tasks.

Questions for authors

Described in weakness

3 comments

We sincerely thank the reviewer for the careful assessment and helpful suggestion. Following the suggestion, we conducted additional experiments at more aggressive retention ratios.

W1. Performance under aggressive token compression

We conducted additional experiments on two detail-sensitive, high-resolution benchmarks, DocVQA and InfoVQA, using Qwen2.5-VL-7B. Table Tw9D-1 reports accuracy at 15% and 10% layer-averaged token retention. ERASE consistently achieves the highest accuracy at both retention ratios. Notably, even at 10% retention, ERASE outperforms the strongest baseline, CDPruner, by 18.28 points on DocVQA and 4.76 points on InfoVQA, while FastV and DART suffer substantial accuracy degradation. These results demonstrate that ERASE maintains a favorable accuracy–compression trade-off even under aggressive token pruning. We have incorporated these results into Appendix A.3.7 (Table 14) of the revised paper.

Table Tw9D-1. Accuracy of detail-sensitive tasks at aggressive retention ratios

RatioMethodDocVQAInfoVQA
15%FastV50.8735.10
15%DART44.0131.32
15%CDPruner66.1843.67
15%PruneSID67.1238.76
15%iLLaVA69.1349.31
15%ERASE88.3358.42
10%FastV21.4427.12
10%DART24.5925.38
10%CDPruner58.2837.91
10%PruneSID56.6533.87
10%iLLaVA45.7335.21
10%ERASE76.5642.67

The authors have satisfactorily resolved my concerns. I have no remaining issues and will keep my evaluation unchanged.

Thank you for your response. We are glad that our rebuttal has addressed your concerns, and we sincerely appreciate your time and valuable feedback.

Reviewer_ne2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ERASE, an adaptive two-stage framework for vision token compression in Vision-Language Models. The first stage performs image-level token pruning using lightweight raw-image statistics—edge detection via Sobel filters and patch-wise entropy—to retain informative regions while discarding smooth, low-information areas. The second stage applies instruction-aware pruning across decoder layers, ranking vision tokens by their relevance to the text instruction and optionally incorporating visual guidance from Stage 1. Experiments on Qwen2.5-VL-7B, Qwen3-VL-8B, and InternVL3-8B demonstrate competitive accuracy retention with substantial token reduction and favorable latency compared to several baselines.

Critical strengths

The primary strength is the intuitive two-stage decomposition that cleanly separates image-level redundancy removal from instruction-dependent token pruning, providing a clear and practical design rationale.

Building on this structure, the integration of Stage 1 visual guidance into Stage 2's token ranking offers a simple yet effective accuracy improvement with minimal additional computational cost.

Critical weaknesses
  1. A notable limitation is the limited technical novelty of the core visual cues. The edge detection relies on classical Sobel filters and the entropy measure is a straightforward histogram-based statistic—both are well-established computer vision techniques rather than novel machine learning contributions. For a venue focused on representation learning, this reliance on handcrafted cues without any learnable component may feel underwhelming, and the paper would benefit from discussing how learned alternatives might further improve performance.

  2. Beyond the novelty concern, the architectural choices throughout the pipeline are predominantly heuristic. The shallow pruning layer is fixed at layer 2, the deep pruning layer at approximately two-thirds of the decoder depth, and hyperparameters such as τ = 0.45 and w = 0.2 are held constant across all models and benchmarks. While the paper provides some empirical justification for these choices, a more systematic sensitivity analysis would strengthen the claim that ERASE is a generally applicable framework rather than a carefully tuned recipe for specific models.

  3. Compounding these issues, the efficiency gains over strong baselines like FastV are marginal in practice. As shown in Table 5, ERASE achieves a task latency of 2671 ms compared to FastV's 2689 ms on 4K images, a difference of less than 1%. Given that ERASE introduces additional computation in Stage 1 (edge detection, entropy computation) that FastV does not incur, the near-parity in latency raises questions about whether the two-stage design provides meaningful practical advantages beyond accuracy, or whether a well-tuned single-stage method could achieve similar efficiency with greater simplicity.

Questions for authors
  1. Have you considered replacing the handcrafted visual cues (Sobel edges and histogram entropy) with lightweight learnable modules? If so, how much additional accuracy or compression efficiency could be gained, and at what computational cost?

  2. How sensitive is ERASE's performance to deviations from the fixed pruning layers (layer 2 and ~2/3 depth) and hyperparameters (τ = 0.45, w = 0.2)? Could you provide a sensitivity analysis across different values to clarify the robustness of these design choices?

  3. The paper reports marginal latency improvements over FastV despite the additional Stage 1 computation. Under what conditions—e.g., different hardware configurations, batch sizes, or image resolutions—does ERASE meaningfully outperform FastV in latency, and when might FastV be preferable?

4 comments

We thank the reviewer for the valuable comments. We address each concern and question below.

W1/Q1. Technical novelty and learnable alternatives

We agree that Sobel filtering and histogram-based entropy are well-established techniques. However, ERASE does not simply apply these operators to visual token pruning. Instead, we design a complementary edge–entropy retention criterion with image-adaptive thresholds, allowing Stage 1 to determine both which tokens and how many tokens to retain based on each image's characteristics. This adaptive retention further guides the hierarchical instruction-aware pruning in Stage 2, where the pruning schedule is dynamically adjusted according to the number of tokens retained in Stage 1. Thus, our contribution lies in the adaptive pruning mechanism and its integration across the two stages, rather than in the individual visual operators.

The use of raw-image statistics is a deliberate design choice. Much of the redundancy in visual inputs arises from spatial structure that can be identified directly from pixels, without relying on learned representations. By exploiting this property, ERASE performs early redundancy removal without additional training or dependence on vision-encoder features. This model-agnostic design allows the same hyperparameters (τ = 0.45 and w = 0.2) to be applied across Qwen2.5-VL, Qwen3-VL, and InternVL3. In contrast, learned feature-based approaches may introduce additional training requirements or dependencies on the target vision encoder. Instruction relevance, however, cannot be determined from the image alone, so Stage 2 uses the LLM's instruction-to-vision attention without requiring additional training.

Regarding learnable alternatives, we have not directly evaluated a learned replacement for Stage 1. However, Table 4 compares ERASE's Stage 1 with existing methods that rely on learned visual representations. At matched retention ratios, ERASE's Stage 1 achieves 77.70% accuracy with a fixed budget, outperforming PruneSID (76.06%) and iLLaVA (76.89%). Image-adaptive retention further improves accuracy to 78.22%, preserving 99.04% of the base model's accuracy at 57.09% token retention. These results demonstrate that simple raw-image statistics can provide effective token selection without additional training or model-specific adaptation. While learnable alternatives may offer further improvements, exploring their accuracy–efficiency trade-offs remains an interesting direction for future work.

W2/Q2. Sensitivity to pruning layers and hyperparameters

We thank the reviewer for suggesting a more systematic sensitivity analysis. As the reviewer notes, Fig. 7b evaluates deep-layer selection with Stage 1 disabled. We therefore conducted additional sensitivity analyses of the full ERASE framework on Qwen2.5-VL-7B under the same 25% layer-averaged token budget.

Sensitivity to pruning layers: Table ne2U-1 evaluates shallow-layer selection while fixing the deep pruning layer at 19. Accuracy remains comparable at layers 1 and 2 but generally decreases as shallow pruning moves deeper, particularly on ChartQA and InfoVQA. This trend aligns with the decline in vision–text similarity observed in Fig. 7a, supporting the importance of pruning before this decline. Table ne2U-2 evaluates deep-layer selection while fixing the shallow layer at 2. Accuracy substantially improves when the deep pruning point moves beyond the middle layers, reaching its highest average at layer 19, approximately two-thirds of the decoder depth. Performance remains relatively stable across nearby deep layers, while pruning too late gradually reduces accuracy because satisfying the same token budget requires more aggressive late-stage pruning (L289–290 of the paper), consistent with Fig. 7b. Together, these results support our choice of pruning locations based on the observed layer-wise vision–text dynamics, rather than relying solely on empirically tuned layer indices.

Table ne2U-1: Impact of selecting shallow layer

LayersChartQAInfoVQARWQAMMStarAvg.
1+1980.0070.9070.0761.3370.58
2+1980.2071.8369.4162.6071.01
3+1978.9668.5967.8461.6069.25
4+1977.5663.1267.7161.6767.52
5+1976.1660.0468.3761.4766.51
6+1975.7261.0569.6761.8767.08

Table ne2U-2: Impact of selecting deep layer

LayersChartQAInfoVQARWQAMMStarAvg.
2+762.0852.0967.1959.3360.17
2+1162.6056.4368.6360.6062.07
2+1575.0470.9969.5461.5369.28
2+1980.2071.8369.4162.6071.01
2+2380.7270.2668.6361.9370.39
2+2779.9268.3267.4561.6069.32

Sensitivity to τ: We conducted an experiment to evaluate the sensitivity of τ in the full ERASE framework. While Appendix A.3.3 already analyzes the impact of τ on edge-based pruning, we additionally examine its impact on end-to-end performance. Table ne2U-3 shows that average accuracy generally increases with τ up to 0.45, consistent with Fig. 9, but decreases at τ = 0.50. As shown in Table 6 of the revised paper (previously Table 9), disabling Stage 1 reduces accuracy, suggesting that removing visual redundancy before Stage 2 benefits instruction-aware pruning. Since increasing τ retains more tokens in Stage 1, it may require more aggressive pruning in Stage 2 to meet the same overall token budget, potentially reducing end-to-end accuracy.

Table ne2U-3: End-to-end analysis on τ

τChartQAInfoVQARWQAMMStarAvg.
0.3579.7271.8568.3762.1370.52
0.4079.9271.5768.5062.5370.63
0.4580.2071.8369.4162.6071.01
0.5080.0471.6867.9761.9370.41

Sensitivity to w: Fig. 8b of the paper already analyzes the visual-guidance weight w, showing that moderate guidance improves accuracy, with the best average performance at w = 0.2. Excessive guidance can weaken instruction-aware token selection.

Overall, these results provide further empirical justification for our pruning-layer choices and demonstrate limited sensitivity to τ within the evaluated range. Furthermore, the same configuration (shallow layer 2, deep layer at approximately two-thirds of the decoder depth, τ = 0.45, and w = 0.2) is applied consistently across Qwen2.5-VL, Qwen3-VL, and InternVL3 without model-specific tuning. We have incorporated the additional sensitivity analyses into Appendix A.3.5–A.3.6 (Tables 11–13) of the revised paper.

W3. Practical advantages of the two-stage design

We agree that ERASE achieves latency comparable to FastV rather than a substantial additional speedup. However, efficiency and accuracy should be considered jointly when evaluating vision token pruning methods. To make this trade-off clearer, we summarize the relevant accuracy and latency results from Tables 1 and 5 of the manuscript in Table W3-1 below.

Table ne2U-4: Accuracy–latency comparison on Qwen2.5-VL-7B at 25% layer-averaged token retention. Accuracy is averaged across eight benchmarks (Table 1), while latency is measured on 16K-token inputs (Table 5).

MethodAvg. Acc. (%)Task Latency (ms)Speedup
Base78.974762-
FastV68.7826891.77x
iLLaVA71.6229411.62x
ERASE75.5726711.78x

As shown in Table ne2U-4, ERASE achieves 6.79 points higher average accuracy than FastV at comparable latency. Compared with iLLaVA, the strongest baseline in accuracy, ERASE improves accuracy by 3.95 points while reducing task latency from 2941 to 2671 ms.

Furthermore, a multi-stage design is not inherently less efficient than a single-stage approach. Latency depends on the computational cost of token selection and the number of tokens processed at each layer, rather than simply the number of pruning stages. For example, the single-stage methods CDPruner and PruneSID are approximately 1.5× slower than ERASE due to their selection overhead (Table 5). As further explained in our response to Q3, ERASE reduces attention FLOPs compared with FastV by removing redundant visual tokens before the LLM, helping offset the additional pruning overhead.

Importantly, the two-stage design allows ERASE to address complementary sources of redundancy at different points in the VLM pipeline. Stage 1 removes image-level redundancy before the LLM, while Stage 2 progressively removes instruction-irrelevant tokens based on layer-wise vision–text interactions. Consequently, ERASE achieves substantially higher accuracy than existing single-stage methods without sacrificing inference efficiency.

Q3. Latency comparison with FastV

To explain why ERASE achieves latency comparable to FastV despite its additional pruning stages, we provide an analytical comparison of decoder attention FLOPs, together with the measured latency results from Table 5.

At a matched layer-averaged token retention ratio, different pruning methods process approximately the same total number of vision tokens across decoder layers. Thus, the FLOPs of linear operations, including projections and MLPs, are comparable. The main computational differences arise from token-selection overhead and attention, whose FLOPs scale quadratically with sequence length. Although ERASE introduces additional selection steps compared with FastV, it reduces attention computation by removing redundant visual tokens before the LLM.

Following the FLOPs count of FlashAttention-2 [1], the causal attention forward pass requires approximately 2·n²·D FLOPs per layer, where D=3584 for Qwen2.5-VL-7B. We approximate the sequence length n by the number of vision tokens, which dominates the input sequence. At 25% retention, FastV processes all visual tokens in the first two layers and retains 19.23% thereafter, resulting in (2·n²·D)·2 + (2·(0.1923n)²·D)·26 ≈ 5.92n²D attention FLOPs. In contrast, ERASE retains 60.77% of visual tokens after Stage 1, followed by shallow- and deep-layer keep ratios of 48.32% and 30%, respectively, resulting in (2·(0.6077n)²·D)·2 + (2·(0.6077·0.4832n)²·D)·17 + (2·(0.6077·0.4832·0.3n)²·D)·9 ≈ 4.55n²D attention FLOPs. Here, 60.77% is the average Stage 1 retention ratio on InfoVQA (Table 4), and 48.32% is the corresponding shallow-layer keep ratio used in our FLOPs approximation.

Under this approximation, FastV requires approximately 1.30× the attention FLOPs of ERASE, corresponding to a difference of approximately 1.3 TFLOPs for 16K vision tokens. This reduction helps offset ERASE's additional token-selection overhead. Stage 1 computes edge and entropy cues using lightweight, parallelizable GPU operations on a single-channel luminance image, while Stage 2 performs instruction-aware selection on the reduced token set. Consequently, ERASE achieves latency comparable to FastV despite its additional pruning stages.

Furthermore, Table 10 in Appendix A.3.4 reports average latency across image resolutions corresponding to 1K–16K visual tokens, where ERASE achieves task latency comparable to FastV (1205 vs. 1206 ms). The reduction in attention computation discussed above is not specific to a particular hardware configuration or batch size, suggesting that ERASE has the potential to maintain competitive efficiency across different inference settings.

[1] Dao, Tri. "Flashattention-2: Faster attention with better parallelism and work partitioning." International Conference on Learning Representations (ICLR), 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.