acceptodds
Under review as a conference paper at ICLR 2027

WinD: Mitigating Pruning-Induced Hallucinations in MLLMs via Window-Aware Diversity Selection

Abstract

Visual token pruning effectively accelerates Multimodal Large Language Models (MLLMs), yet its impact on model reliability remains under-explored. In this paper, we identify a critical side effect: prevalent pruning strategies significantly exacerbate hallucinations, particularly in complex multi-object scenes. Our analysis associates this degradation with the interaction between global CLS-guided selection and attention-sink behavior. Such selection disproportionately preserves sink-favored tokens, which occupy the limited preservation budget and reduce the grounding evidence available to the model, causing the model’s focus to collapse onto a few sink tokens that cannot fully represent the whole visual context, while disregarding other essential visual information. To handle this problem, we shift the pruning paradigm from sink-seeking to preserving diverse important information, proposing the Window-aware Diversity selection (WinD) framework. WinD decentralizes the pruning process by adaptively partitioning the visual field into local windows based on visual content. It dynamically allocates pruning budgets based on local information density and introduces a semantic diversity-aware selection mechanism to suppress redundant sink features. Extensive experiments demonstrate that WinD strikes a superior balance between efficiency and reliability. On LLaVA-1.5-7B, our method reduces 88.9% of visual tokens and 77.5% of TFLOPS while retaining 96.7% of original performance, and effectively mitigates pruning-induced hallucinations. Our codes will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, Token Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_toA7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free, window-aware visual-token pruning method. It uses a minimum quota for each spatial region, adaptive budget allocation, and diversity-aware selection to preserve visual evidence while reducing computation. The authors evaluate the efficiency–performance trade-off on three 7B vision-language models and multiple benchmarks. However, direct cross-model evidence for hallucination mitigation remains incomplete.

Critical strengths

The three base models are LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B.

Critical weaknesses
  1. The paper should clarify how CLS attention is obtained for Qwen2.5-VL, how it allocates tokens when the number of windows exceeds the budget, and what the end-to-end latency is. Direct hallucination results across all three backbones are also incomplete.

  2. At the same 64-token budget, WinD improves aggregate performance retention only from 96.0% to 96.7%, while prefill is slightly slower (64.3 vs. 62.6 ms). Its advantage at matched end-to-end latency remains unclear.

  3. The research could aim to prune tokens that contribute to hallucinations and thereby substantially reduce the hallucination rate. If the method merely ensures that pruning does not increase hallucinations, I see limited value in its hallucination-mitigation contribution.

Questions for authors

See weakness

7 comments
AuthorsOpenReview ↗

Response to Reviewer toA7 (1/2): Implementation clarifications and cross-backbone hallucination evaluation

W1. Implementation clarifications and cross-backbone hallucination evaluation.

We thank the reviewer for identifying several details that were insufficiently described. First, Qwen2.5-VL does not provide a CLS token. For this architecture, we compute each visual token's importance by averaging the attention from the global tokens to that visual token. Thus, WinD uses a model-appropriate global-attention signal rather than requiring a specific CLS-token architecture.

Second, the entropy-derived window size s^l\hat{s}_{l} is an initial proposal rather than an unconditional choice. At pruning layer ll, the implementation selects sl⋆=min⁡{s∈S∣s≥s^l,;Ml(s)≤Npresl}s_{l}^{\star} = \min\lbrace s \in \mathcal{S} \mid s \geq \hat{s}_{l},\\; M_{l}(s) \leq N_{\mathrm{pres}}^{l} \rbrace, where Ml(s)M_{l}(s) counts only windows containing at least one currently surviving token. Therefore, if the proposed partition produces more active windows than the available budget, neighboring regions are merged by selecting the closest larger feasible window size before applying the quota allocation in Eq. (4). This guarantees Ml(sl⋆)≤NpreslM_{l}(s_{l}^{\star}) \leq N_{\mathrm{pres}}^{l}, so assigning at least one token to every active window remains feasible. We will add this omitted feasibility step to the method description.

Third, the prefill latency reported in Table 2 includes the complete pruning procedure, including dynamic partitioning, quota allocation, and diversity-aware selection. It measures the model-side inference pipeline and excludes disk I/O and CPU image loading. We will clarify this timing boundary and provide the component-level overhead breakdown.

Finally, to provide direct hallucination evidence across all three evaluated VLM backbones, we report matched POPE F1 results under the most aggressive compression setting using the same evaluation protocol:

POPE F1 (↑)

BackboneBudgetVanillaFlowCutWinD
LLaVA-1.5-7B6485.981.982.9
LLaVA-NeXT-7B16086.479.985.1
Qwen2.5-VL-7B88.9% reduction86.982.182.4

Under matched token budgets, WinD improves POPE F1 over FlowCut by 1.0, 5.2, and 0.3 points on LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, respectively. These results complement the detailed CHAIR and HallusionBench analyses on LLaVA-1.5-7B and show that the mitigation of pruning-induced hallucination is not limited to a single VLM backbone.

AuthorsOpenReview ↗

Response to Reviewer toA7 (2/2): Practical latency–quality trade-off and scope of hallucination mitigation

W2. Practical latency–quality trade-off.

At 64 retained visual tokens, WinD preserves 96.7% of the unpruned model's average performance, compared with 96.0% for FlowCut, at an additional prefill cost of 1.7 ms. The table summarizes this trade-off for both LLaVA backbones. All differences are relative to FlowCut, and prefill includes the complete token-selection procedure.

BackboneTokensΔAvg. (pp)FlowCut prefill (ms)WinD prefill (ms)Overhead (ms)
LLaVA-1.5-7B64+0.762.664.3+1.7 (2.7%)
LLaVA-NeXT-7B160+0.163.865.3+1.5 (2.4%)

The captioning results show the practical benefit at the 64-token budget. On LLaVA-1.5-7B, the share of captions containing at least one hallucinated object (CHAIR-s) falls from 58.1% with FlowCut to 51.3% with WinD, a reduction of 6.8 percentage points. CHAIR-i also falls from 14.9% to 12.2%, and POPE F1 rises from 81.9 to 82.9. On LLaVA-NeXT-7B, the aggregate scores are nearly identical at 160 tokens, while POPE F1 rises from 79.9 to 85.1 with WinD. These comparisons quantify the change in quality and prefill latency at a fixed token budget.

W3. Scope and value of hallucination mitigation.

We agree that WinD is not designed to eliminate the intrinsic hallucinations of the unpruned model. Our claim is narrower: it mitigates the additional hallucination introduced by aggressive visual-token pruning. We will revise the paper to consistently use the term “pruning-induced hallucination mitigation” and avoid suggesting that WinD is a general-purpose hallucination-removal method.

Visual tokens cannot generally be divided into a fixed set of hallucination-causing and non-hallucination-causing tokens. Their usefulness depends on the image, prompt, and interactions within the model. In particular, attention-sink tokens are not necessarily harmful by themselves; the problem is that global top-k pruning can over-preserve them, thereby discarding spatially distributed evidence required to ground the response. Directly removing sink tokens may therefore also eliminate useful contextual information. WinD instead reduces this selection bias by preserving regional coverage and semantic diversity.

We believe this remains practically important: an efficiency method should not amplify hallucination as a side effect of compression. As shown by the matched-budget results above, WinD better preserves hallucination-related performance than global pruning while removing up to 88.9% of visual tokens. Thus, the contribution is not that token pruning itself solves all model hallucinations, but that substantial computational savings can be obtained with less additional reliability degradation. Methods for reducing the intrinsic hallucinations of the original model are complementary to, rather than replaced by, WinD.

AuthorsOpenReview ↗

Follow-up on our author response

Thank you again for your feedback. We added POPE F1 results across all three backbones and clarified the latency–quality trade-off. At 160 retained tokens on LLaVA-NeXT-7B, WinD improves POPE F1 by 5.2 points over FlowCut with 1.5 ms of additional prefill latency. If you have a chance to read our response, we would appreciate hearing whether the added results address your concerns about the practical benefit of WinD.

Reviewer_xkcYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates hallucinations induced by visual token pruning and attributes them to the over-preservation of attention-sink tokens. It proposes WinD, which combines entropy-based window partitioning, adaptive token allocation, and diversity-aware selection to preserve visual evidence under aggressive compression. Experiments evaluate its performance and efficiency on three 7B vision-language models.

Critical strengths
  1. The paper addresses a relevant problem: token pruning should be evaluated for its impact on hallucinations, beyond aggregate task accuracy.
  2. Combining spatial coverage with semantic diversity is intuitively reasonable, and the paper includes hallucination evaluations, diagnostic analyses, and measured prefill latency.
Critical weaknesses
  1. All experiments use LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B. While these models remain useful for comparison with earlier pruning work, they are insufficient to establish the method’s relevance to current VLMs. More recent families, such as Qwen3-VL, and substantially larger variants have been available since 2025. The absence of such evaluations leaves it unclear whether the observed sink behavior and the benefits of WinD persist with stronger visual representations and larger language backbones. In particular, the claim of robustness “across model scales” is not supported by experiments exclusively on 7B models.
  2. Although the paper includes competitive baselines such as FlowCut and VScan, several particularly relevant methods are missing. PruneSID combines importance–diversity selection with image-adaptive compression; Nüwa explicitly addresses spatial integrity lost through pruning; and HiPrune/HiPrune++ exploits hierarchical attention and spatial neighborhoods, with additional hallucination evaluation. MMTok is cited but not evaluated. These methods directly overlap with the motivations behind WinD, making their omission consequential for both novelty assessment and the claimed performance–efficiency trade-off. Matched-protocol comparisons are needed; results reported separately in these papers cannot establish their relative ranking.
  3. Using the arithmetic mean of per-benchmark performance normalized by the vanilla model—a calculation consistent with Table 3 and the headline results in Table 1—WinD obtains approximately 95.46% at 77.8% token reduction, rather than the reported 96.9%. The corresponding FlowCut result is approximately 94.12%, rather than 96.5%. At 88.9% reduction, WinD’s listed scores yield approximately 90.30%, rather than 90.9%. The authors need to specify the aggregation formula and reconcile these discrepancies. Since normalized average performance is central to the paper’s conclusions, this is a substantive reporting concern.
  4. The incremental gains are small, and a convincing practical advantage over the strongest baseline is not established. On LLaVA-NeXT, WinD improves normalized average performance over FlowCut by only 0.3, 0.2, and 0.1 percentage points at the three reported budgets. No uncertainty estimates are provided to assess these margins. Meanwhile, Table 2 shows slightly higher prefill latency for WinD: 64.3 vs. 62.6 ms on LLaVA-1.5 and 65.3 vs. 63.8 ms on LLaVA-NeXT, alongside slightly higher FLOPs. These results suggest a modest trade-off whose significance requires further justification. Paired confidence intervals and end-to-end measurements accounting for selection overhead would make the practical contribution easier to assess.
Questions for authors
  1. Can the authors evaluate WinD on a newer VLM family and at least one substantially larger model? Do the attention-sink findings and hallucination improvements remain consistent across architectures and parameter scales?
  2. How does WinD compare with PruneSID, Nüwa, HiPrune/HiPrune++, and MMTok under identical evaluation protocols and comparable computational budgets, particularly on CHAIR, POPE, and HallusionBench?
  3. How are the “Avg.” values in Table 4 calculated? Can the authors reconcile the reported averages with the individual benchmark scores, including the discrepancies at 77.8% and 88.9% reduction?
  4. Are the small improvements over FlowCut statistically reliable under paired example-level analysis? What is the end-to-end latency–quality trade-off when all pruning overhead is included?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xkcY (1/2): Qwen3-VL evaluation and comparisons with related methods

W1/Q1. Evaluation on a more recent VLM architecture.

Our original evaluation covers only 7B models. We additionally evaluate Qwen3-VL-8B under the same training-free pruning protocol. Avg. is the mean of the six benchmark scores normalized by their respective unpruned scores.

Method (reduction)GQAMMBMMEPOPESQATextVQAAvg.
Vanilla (0%)61.984.5237588.594.285.2100.00%
FlowCut (66.7%)61.480.9227886.186.381.195.82%
WinD (66.7%)61.183.1230987.588.682.797.38%
FlowCut (77.8%)59.377.1225684.584.979.393.45%
WinD (77.8%)60.379.4229485.185.580.194.82%
FlowCut (88.9%)56.873.7219882.679.370.288.57%
WinD (88.9%)57.375.6222183.281.674.190.53%

At 88.9% reduction, WinD scores 83.2 on POPE versus 88.5 for the unpruned model, retaining 94.0% of its score. We also observe attention concentration on Qwen3-VL. The 8B experiment extends our evaluation to a newer architecture. Generalization to substantially larger models remains open. We will replace “across model scales” with “across model architectures” in the revision.

W2/Q2. Comparisons with closely related pruning methods.

We re-evaluate the official implementations of PruneSID, Nüwa, HiPrune++, and MMTok using the same LLaVA-1.5-7B checkpoint, token budgets, prompts, decoding settings, and evaluation code. Avg. (%) covers all 12 benchmarks from our original evaluation. The 192-, 128-, and 64-token budgets correspond to reductions of 66.7%, 77.8%, and 88.9%, respectively.

TokensMethodGQAMMBMMB-CNMMEPOPESQAVQAv2TextVQAVizWizSEEDMM-VetLLaVA-BAvg.
576Vanilla61.964.758.1186285.969.578.558.250.059.331.166.9100.0
192FlowCut60.063.257.8183686.168.677.157.551.857.534.368.4100.1
192PruneSID60.163.757.1179186.968.576.856.752.457.033.265.199.2
192Nüwa60.963.356.9183486.468.275.957.452.157.730.564.298.5
192HiPrune++60.363.557.5181886.968.876.957.552.757.134.166.199.9
192MMTok60.163.457.1177486.469.077.156.951.157.233.266.299.1
192WinD60.163.857.5183186.969.577.157.651.457.736.768.9101.0
128FlowCut58.562.156.5179285.268.676.057.352.056.232.567.598.4
128PruneSID58.862.156.8174985.268.375.354.753.855.631.966.397.8
128Nüwa60.261.856.3182885.567.875.157.051.956.129.865.897.6
128HiPrune++59.061.357.0178085.968.576.256.052.654.932.764.797.9
128MMTok58.361.756.0177986.369.076.457.051.355.533.165.298.0
128WinD58.862.157.0176185.769.476.456.752.256.234.767.799.2
64FlowCut55.660.854.9174481.969.172.855.652.953.232.365.196.0
64PruneSID57.158.855.1173382.767.871.954.254.253.430.265.295.2
64Nüwa58.360.054.3170682.367.572.854.953.152.828.262.994.3
64HiPrune++56.460.353.8176782.668.972.854.554.751.731.964.895.8
64MMTok55.960.353.6171582.468.972.355.152.353.031.163.794.9
64WinD56.160.755.4173282.969.873.255.453.453.533.465.396.7

WinD achieves the highest 12-benchmark average at all three budgets. At 64 tokens it retains 96.7% of unpruned performance, compared with 96.0% for FlowCut, the strongest baseline by this measure.

Hallucination evaluation on LLaVA-1.5-7B at 64 tokens.

MethodCHAIR-s ↓CHAIR-i ↓POPE ↑HallusionBench ↑
FlowCut58.114.981.939.4
PruneSID55.613.282.740.1
Nüwa56.413.182.339.9
HiPrune++52.712.982.640.1
MMTok58.314.282.438.9
WinD51.312.282.940.9

WinD achieves the best score on all four metrics. Compared with the strongest baseline for each metric, CHAIR-s falls from 52.7 to 51.3 and CHAIR-i from 12.9 to 12.2. POPE rises from 82.7 to 82.9 and HallusionBench from 40.1 to 40.9. We will include the full results and implementation details in the revision.

AuthorsOpenReview ↗

Response to Reviewer xkcY (2/2): Correction of Table 4 averages and the latency–quality trade-off

W3/Q3. Correction of the averages in Table 4.

Thank you for identifying this error. When we shortened Table 4, we retained the “Avg.” values from an earlier version that included more benchmarks. The individual benchmark scores were unchanged, but the reported averages corresponded to the earlier benchmark set.

We have recomputed the averages for all four pruning methods on Qwen2.5-VL-7B using the six benchmarks displayed in Table 4. Each benchmark contributes equally to the average after normalization by its unpruned score:

Avg⁡(m)=1006∑k=16Sm,kSvanilla,k\operatorname{Avg}(m) = \frac{100}{6}\sum_{k=1}^{6}\frac{S_{m,k}}{S_{\mathrm{vanilla},k}},

where Sm,kS_{m,k} and Svanilla,kS_{\mathrm{vanilla},k} are the scores of method mm and the unpruned model on benchmark kk, respectively. We round only the final average.

Qwen2.5-VL-7B: visual-token reduction

Method66.7%77.8%88.9%
FastV92.59%89.94%84.10%
HoloV95.30%92.32%87.17%
FlowCut97.41%94.12%89.74%
WinD97.75%95.46%90.30%

These values confirm the reviewer's calculations: at 77.8% reduction, the corrected averages are 95.46% for WinD and 94.12% for FlowCut. At 88.9% reduction, WinD retains 90.30% of the unpruned performance. WinD has the highest normalized average among the four methods at all three budgets. We will correct Table 4 and the corresponding values in the main text, and include the aggregation formula in the table caption.

W4/Q4. Reliability and latency–quality trade-off.

We agree that the aggregate gains on LLaVA-NeXT-7B are modest. WinD's normalized average exceeds FlowCut's by 0.3, 0.2, and 0.1 percentage points at 640, 320, and 160 retained tokens, respectively.

To assess whether the trend extends to a newer architecture, we also evaluated FlowCut and WinD on Qwen3-VL-8B using the same checkpoint, prompts, token budgets, decoding settings, and evaluation code. As shown in the table in our response to W1/Q1, WinD scores higher on 17 of the 18 benchmark–budget pairs. Its normalized-average gains are approximately 1.6, 1.4, and 2.0 percentage points at 66.7%, 77.8%, and 88.9% token reduction, respectively.

The table below pairs the quality gains at the most aggressive LLaVA budgets with their measured prefill costs. Both latency measurements include the complete pruning procedure.

BackboneTokensΔAvg. (pp)FlowCut prefill (ms)WinD prefill (ms)Overhead (ms)
LLaVA-1.5-7B64+0.762.664.3+1.7 (2.7%)
LLaVA-NeXT-7B160+0.163.865.3+1.5 (2.4%)

On LLaVA-NeXT-7B, POPE provides a complementary measure of this trade-off because it is evaluated separately from the six benchmarks in the normalized average. At 160 tokens, FlowCut and WinD achieve similar six-benchmark averages (91.8% and 91.9%, respectively). On POPE, however, the unpruned model scores 86.4 F1, compared with 79.9 for FlowCut and 85.1 for WinD. Thus, at this budget, the additional 1.5 ms of prefill latency accompanies a 5.2-point gain in POPE F1.

AuthorsOpenReview ↗

Follow-up on our author response

Thank you again for your detailed feedback. Our response includes new results on Qwen3-VL-8B, matched comparisons with the four baselines you suggested, and corrected averages for Table 4. If you have a chance to review these updates, we would appreciate your thoughts on whether they address your main concerns and whether any point would benefit from further clarification.

Reviewer_2kHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WinD, a window-aware diversity selection framework designed to mitigate pruning-induced hallucinations in Multimodal Large Language Models (MLLMs). The authors identify that conventional global CLS-based token pruning tends to over-preserve attention sink tokens, potentially discarding important visual evidence and exacerbating hallucinations. To address this issue, WinD introduces three components: entropy-based dynamic window partitioning, adaptive token budget allocation, and diversity-promoting semantic selection. These components aim to preserve spatial coverage and semantic diversity while reducing the number of visual tokens.

Critical strengths
  1. The paper investigates a practically relevant problem. The authors highlight the overlooked relationship between visual token pruning and hallucination, which is important for developing efficient and reliable MLLMs. The analysis of attention sink behaviors provides an interesting perspective on the limitations of conventional pruning strategies.

  2. The proposed framework achieves a great efficiency-performance trade-off. WinD maintains competitive performance under aggressive visual token compression while reducing computational costs.

Critical weaknesses
  1. The methodological novelty are relatively limited. The proposed method mainly combines existing concepts, including attention entropy, adaptive spatial partitioning, and diversity-based token selection. Moreover, the improvements over strong baselines such as FlowCut are relatively modest in several settings. For example, the performance retention on LLaVA-NeXT-7B increases only from 91.8% to 91.9% under the highest compression ratio. These results raise questions about whether the additional methodological complexity is sufficiently justified.

  2. Continued reliance on global CLS attention. Although WinD aims to mitigate the limitations of CLS-based pruning, its dynamic window partitioning still relies on global CLS attention entropy. This global statistic may not adequately capture heterogeneous local visual complexity, especially when small but important objects appear in low-attention regions. Moreover, potential biases in CLS attention could propagate into the partitioning decisions, limiting the effectiveness of the proposed adaptation mechanism.

  3. Limited architectural coverage and compression granularity. The experimental evaluation could be strengthened by incorporating a broader range of model architectures, including video-oriented MLLMs such as Video-LLaVA, to better assess the generalizability of WinD. Furthermore, each evaluated model is tested at only three compression ratios, providing a relatively coarse characterization of the efficiency–performance trade-off.

Questions for authors
  1. Could the authors clarify how the dynamic quota allocation handles cases where the number of windows exceeds the token preservation budget? Equation (4) guarantees at least one retained token per window, but this constraint may become infeasible under aggressive compression and fine-grained partitioning. For example, a 576-token grid partitioned into 2×2 windows would produce 144 windows, exceeding a preservation budget of 64 tokens. How does WinD handle such cases in practice?

  2. Could the authors provide a more comprehensive analysis of the practical computational overhead introduced by WinD? Although the method substantially reduces TFLOPs, Table 2 shows only a 1.6× prefill speedup on LLaVA-1.5-7B, with slightly higher latency than FlowCut. Could the authors further quantify the overhead of dynamic partitioning, quota allocation, and semantic diversity selection, particularly for high-resolution inputs and different batch sizes?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2kHP (1/2): Window budgets and computational overhead

Q1. Feasibility of quota allocation.

Thank you for pointing out this boundary case. Our original description of dynamic window partitioning was incomplete. The entropy-derived window size s^l\hat{s}_l is an initial proposal. At layer ll, let Ml(s)M_l(s) denote the number of windows of size ss containing at least one surviving token. We select sl⋆=min⁡{s∈S∣s≥s^l,;Ml(s)≤Npresl}s_{l}^{\star} = \min\lbrace s \in \mathcal{S} \mid s \geq \hat{s}_{l},\\; M_{l}(s) \leq N_{\mathrm{pres}}^{l} \rbrace,

where S\mathcal{S} contains the admissible window sizes and NpreslN_{\mathrm{pres}}^l is the layer-specific budget. Empty windows are excluded from the count. If the proposed partition is infeasible, we select the smallest larger window size that satisfies the budget constraint. In the reviewer's example, if 2×22 \times 2 windows produce 144 active windows under a 64-token budget, we select a coarser partition with at most 64 active windows. Equation (4) then allocates the non-negative residual budget Npresl−Ml(sl⋆)N_{\mathrm{pres}}^l - M_l(s_l^\star). We will clarify this feasibility step and the definition of MlM_l in the revision.

Q2. Practical computational overhead.

The prefill latency in Table 2 includes the complete pruning procedure. Compared with FlowCut, WinD adds 1.7 ms on LLaVA-1.5-7B (64.3 vs. 62.6 ms, +2.7%) and 1.5 ms on LLaVA-NeXT-7B (65.3 vs. 63.8 ms, +2.4%).

We profile three WinD components at each pruning layer and accumulate their latencies across layers. Partition covers attention-entropy computation, feasible window-size determination, and token-to-window assignment. Quota covers window-level importance aggregation and adaptive budget allocation. Diversity covers anchor selection, similarity-penalty computation, intra-window top-kk selection, and index gathering. We independently time the complete selection block to obtain the selection subtotal. Full prefill covers the vision encoder, token selection, multimodal projector, and LLM prefill. All values in the table are in milliseconds.

ModelPartitionQuotaDiversitySelection subtotalFull prefill
LLaVA-1.5-7B0.280.111.231.6264.3
LLaVA-NeXT-7B0.460.372.343.1765.3

All measurements use batch size 1 and each model's native input resolution. We use identical image–prompt inputs, exclude disk I/O and CPU image loading, and report the mean over 100 runs after 20 warm-up runs on an NVIDIA RTX A6000 using FP16 precision. Synchronized wall-clock timing includes both GPU computation and CPU–GPU synchronization.

Selection takes 1.62 ms on LLaVA-1.5-7B and 3.17 ms on LLaVA-NeXT-7B, accounting for 2.52% and 4.85% of full prefill, respectively. The wall-clock speedup is smaller than the FLOP reduction because prefill also includes fixed costs, memory operations, kernel launches, and computations unaffected by the number of retained visual tokens. We will add the profiling protocol and component-level breakdown to the revision.

AuthorsOpenReview ↗

Response to Reviewer 2kHP (2/2): Methodological novelty, global attention, and evaluation scope

W1. Methodological novelty.

We agree that attention entropy, spatial partitioning, and diversity-aware selection each have precedents. Our contribution lies in identifying how global top-k pruning can amplify attention-sink bias and contribute to pruning-induced hallucination. Our analysis shows that global top-k selection can spend a limited token budget on semantically redundant tokens while discarding evidence from other regions. WinD addresses both effects through entropy-adaptive partitioning, a minimum quota per active window, and diversity-aware selection within each window.

W2. Reliance on global attention.

Global attention provides the initial importance signal, and its entropy guides the choice of window size. We agree that this global statistic can miss local complexity. Once a feasible partition is selected, each active window receives at least one token slot, even if the window has low global attention. WinD then allocates the remaining budget adaptively and considers semantic diversity within each window. This guarantees window coverage. If a small object shares a window with background, however, the retained token may still represent the background. We will clarify this limitation. For Qwen2.5-VL, which has no CLS token, we use the mean attention from the global tokens to each visual token as its importance signal.

W3. Architectural coverage and compression granularity.

As detailed in our response to Reviewer 2 (W1/Q1), we additionally evaluate WinD on Qwen3-VL-8B, extending the evaluation to a more recent VLM architecture. We chose three token budgets to cover moderate, high, and aggressive compression and to permit comparisons with prior methods at the same operating points. The results show a consistent trend across these three budgets, while the behavior at intermediate budgets remains unmeasured. Our evaluation focuses on images. Extending WinD to video would also require evaluating temporal redundancy and cross-frame consistency, and we will clarify this scope in the revision.

AuthorsOpenReview ↗

Follow-up on our author response

Thank you again for your review. In our response, we clarified how WinD handles cases where the number of active windows exceeds the token budget and added a component-level latency breakdown for both LLaVA backbones. If you have a chance to read our response, we would appreciate hearing whether it addresses your questions or whether anything remains unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.