acceptodds
Under review as a conference paper at ICLR 2027

Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs

Abstract

Training-free visual token pruning allocates a fixed budget among candidate tokens, and existing methods keep refining the scoring function, hovering near the ceiling of the general benchmarks, yet most collapse on fine-grained tasks such as grounding. We instead ask what visual evidence the LLM needs at inference, and find two answers. First, the evidence can come from only two sources, the image and the question, and no single score serves every task. Second, keeping only a subset of native patches breaks the integrity of the visual sequence in both content and position. Building on both findings, we present **Bifocal**, a training-free, two-stage visual token pruning method built on mixed-granularity evidence. Before the LLM, it keeps two kinds of fine evidence and a coarse context. Query-focused evidence keeps the tokens the question refers to, image-intrinsic evidence maximises how well the token set represents and covers the image, and a pooled coarse context preserves the dual integrity of the retained tokens. Inside the LLM, it further keeps the top-K tokens by text-to-image attention. Evaluated across three models and several budgets, spanning general VQA, reasoning, OCR, hallucination and grounding, Bifocal achieves state-of-the-art results: it attains the best general average in every setting while, at the tightest budget, leading the strongest baseline on grounding by to percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Vision Token Pruning, Token Compression, Efficiency and Compression, Interpretability and Analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VAWvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-free visual token pruning for multimodal LLMs. The key motivation is that different tasks rely on complementary image-driven and query-driven evidence, while sparse subset pruning can additionally disrupt the positional and content integrity of the visual sequence. To address these issues, the paper proposes Bifocal, which combines query-focused fine evidence, image-intrinsic fine evidence, and pooled coarse context before the LLM, followed by a second-stage attention-based pruning inside the LLM. Experiments across multiple VLM backbones and token budgets show strong results, especially on fine-grained grounding tasks.

Critical strengths
  1. The paper is well-written and organized, and the figures are helpful for understanding the motivation and the proposed mixed-granularity design.

  2. The motivation is clear and intuitive, especially the observation that sparse pruning can further damage both the positional and content integrity of the original visual sequence.

  3. The additional visualizations and ablation studies are helpful for understanding the behavior of the proposed method and the contribution of its individual components.

Critical weaknesses
  1. My main concern is the generality of the proposed design and whether the current formulation fully follows from the paper’s own observations. The current method and analysis are largely developed around single-image setting, particularly with the three evidence types relying on separate scoring functions tailored to their respective roles. However, in more complex multimodal scenarios (e.g. multi-image, video, text-intensive multimodal reasoning tasks), the composition and interaction of visual evidence may differ substantially across images, text contexts, or temporal contexts, making it unclear whether the method design can generalize effectively.
  2. The evaluation could be further extended to more recent reasoning-oriented VLMs. The current experiments focus on LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, while it remains unclear how well the proposed pruning strategy transfers to models with stronger multimodal reasoning behavior, such as Qwen3-VL-Thinking. Particularly, evaluating whether the selected visual evidence remains sufficient throughout longer decoding trajectories would provide a more complete picture of the method’s generality.
  3. The experimental results are not analyzed sufficiently. In Table 1, all baselines are reported for the 192- and 128-token settings, whereas FastV, PDrop, PruMerge+, and VisionZip are omitted from the 64-token setting without explanation. The paper should clarify the reason for these omissions and provide a more consistent comparison across budgets and backbones.
Questions for authors
  1. Have the authors considered using task-dependent token allocations for different types of visual evidence? Since the paper shows that different tasks favor different proportions of image-driven and query-driven evidence, it would be interesting to know whether allocating different numbers of coarse, image-intrinsic, and query-focused tokens according to the task type could further improve performance.
7 comments
AuthorsOpenReview ↗

Response to Reviewer VAWv (Part 1/4)

We thank the reviewer for the thorough review and insightful suggestions. We address each concern below. Newly added or substantially revised text in the manuscript is highlighted in blue. Below, QQ, II and CC denote query-focused evidence, image-intrinsic evidence and coarse context. With the additional experiments, Bifocal shows its generalization across 4 models, multiple budgets and 26 benchmarks covering 7 types of multimodal tasks, although some of these tasks go beyond the scope of this paper.

W1: My main concern is the generality of the proposed design and whether the current formulation fully follows from the paper's own observations. The current method and analysis are largely developed around single-image setting, particularly with the three evidence types relying on separate scoring functions tailored to their respective roles. However, in more complex multimodal scenarios (e.g. multi-image, video, text-intensive multimodal reasoning tasks), the composition and interaction of visual evidence may differ substantially across images, text contexts, or temporal contexts, making it unclear whether the method design can generalize effectively.

We understand the reviewer's concern and explain it from three aspects: how our observations lead to the design, whether the roles of the three kinds of evidence still hold in multi-image, video and text-intensive scenarios, and how Bifocal performs on these tasks.

(1) The observations, the method design and the three scoring functions are not independent. Our observations revolve around splitting the budget between two kinds of tokens, selected by the query and by the CLS token. They overlap little and each has a clear target: query tokens focus on objects, while CLS tokens are spread out and cover the image widely. The method therefore keeps these two kinds of selection and strengthens each around its target. For query tokens, we filter out the background more precisely to select target tokens. Based on these target tokens, we use the CLS attention to select further tokens that cover more information. The coarse tokens are obtained by aggregating the remaining tokens. Accordingly, query tokens aim at objects and improve REC, while CLS tokens help with questions about objects absent from the image and reduce hallucination.

(2) We thank the reviewer for this inspiring point and add further experiments on the role of token evidence in multimodal scenarios. On Qwen2.5-VL and Qwen3-VL, which support multi-image and video inputs, we select four benchmarks whose ground-truth target regions let us examine whether QQ still keeps the content the question is about, and whether II still covers diverse content: the group grounding subset of MIG-Bench for multi-image (target: the target image and its box), Charades-STA for video (target: the frames within the queried time span), and TreeBench (target: the boxes of all instances in the question) and TextVQA for text-intensive tasks.

  • Experiment 1: where QQ and II point

For each sample, we rank the tokens of all images or frames by the QQ score and by the II score separately, take 25% of all visual tokens from each, and measure the fraction of the ground-truth region that is selected, ∣S∩G∣/∣G∣|S\cap G|/|G|.

ModelMetricMIG-BenchCharades-STATreeBenchTextVQA
Qwen2.5-VL-7BQQ covers target0.420.270.620.44
II covers target0.250.250.560.81
Q∩IQ \cap I overlap0.250.220.240.31
Qwen3-VL-8BQQ covers target0.410.270.620.44
II covers target0.250.250.530.74
Q∩IQ \cap I overlap0.250.210.220.28

In these additional scenarios, QQ still covers a larger fraction of the target, and the two selections share few tokens, so they are two different kinds of evidence. TextVQA is the exception because such questions rarely point to where the answer is (e.g., "what does the sign say?"). This is consistent with the analysis in Appendix B and does not contradict the conclusion.

  • Experiment 2: II improves coverage

On the multi-image and video tasks, under the same 25% budget, we compare giving the whole budget to QQ, splitting it evenly between QQ and II, and further adding CC. We report the fraction of images or frames that receive at least half of their proportional share of tokens:

ModelAllocationMIG-BenchCharades-STA
Qwen2.5-VL-7BQQ0.770.96
QQ + II0.940.99
QQ + II + CC1.001.00
Qwen3-VL-8BQQ0.780.97
QQ + II0.950.99
QQ + II + CC1.001.00

As the II evidence is added, the information coverage increases substantially.

AuthorsOpenReview ↗

Response to Reviewer VAWv (Part 2/4)

  • Experiment 3: Performance on multi-image, video and text-intensive tasks

We further evaluate Qwen2.5-VL and Qwen3-VL on the following tasks:

  • Multi-image: MuirBench
  • Video: MVBench, Charades-STA
  • Text-intensive reasoning: MMMU val, MathVista, TreeBench

Table 1. Qwen2.5-VL-7B, 25% tokens.

MethodMuirBenchMVBenchCharades-STA [email protected]Charades-STA mIoUMMMU valMathVistaTreeBenchAvg
Baseline45.865.631.530.650.768.140.2100.0
FastV40.662.228.028.247.951.735.389.0
VisionZip44.062.926.927.447.862.938.892.9
DART43.460.128.028.146.350.636.389.0
HiPrune44.262.927.027.648.762.839.893.7
EADP42.261.926.626.947.758.339.090.8
VLM-Pruner43.262.425.525.947.359.440.590.9
Bifocal44.263.428.228.648.965.441.295.9

Table 2. Qwen3-VL-8B-Instruct, 25% tokens.

MethodMuirBenchMVBenchCharades-STA [email protected]Charades-STA mIoUMMMU valMathVistaTreeBenchAvg
Baseline52.268.657.750.552.164.744.0100.0
FastV47.663.150.545.648.746.040.288.3
VisionZip50.767.055.749.151.658.041.795.9
DART49.065.851.946.949.850.840.791.6
HiPrune50.467.555.448.950.359.041.795.7
EADP51.665.254.648.551.353.742.594.7
VLM-Pruner51.365.553.748.550.350.041.292.9
Bifocal50.968.256.649.950.858.743.097.0

On the additional multi-image, video, and text-intensive tasks, Bifocal still leads on both models. However, for multi-image and video tasks, acceleration involves not only token redundancy. Key-image selection, memory, and other issues are equally critical, but these go beyond the scope of our method.

AuthorsOpenReview ↗

Response to Reviewer VAWv (Part 3/4)

W2: The evaluation could be further extended to more recent reasoning-oriented VLMs. The current experiments focus on LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, while it remains unclear how well the proposed pruning strategy transfers to models with stronger multimodal reasoning behavior, such as Qwen3-VL-Thinking. Particularly, evaluating whether the selected visual evidence remains sufficient throughout longer decoding trajectories would provide a more complete picture of the method's generality.

We thank the reviewer for this very interesting suggestion. We explore it from two angles. First, we have added comparisons of Bifocal on Qwen3-VL-8B-Instruct in W1, and here we further add Qwen3-VL-8B-Thinking with thinking enabled. Second, to explore the generalization of our method during long decoding, we compare CoT reasoning with direct answering on MathVista for Qwen2.5-VL-7B and Qwen3-VL-8B-Instruct.

  • Experiment 1: Qwen3-VL-8B-Thinking

With 25% tokens and at most 2048 generated tokens, the results are:

MethodGQATextVQADocVQAPOPEGenRefCOCOg testREC
Baseline57.476.792.990.8100.088.4100.0
FastV49.064.454.283.179.859.667.4
VisionZip53.673.280.587.893.073.683.3
HiPrune50.474.383.387.592.675.485.3
EADP55.472.676.388.992.864.472.9
Bifocal56.675.484.889.696.780.491.0
  • Experiment 2: Long decoding

All results are on the full MathVista testmini with 25% tokens. Direct outputs only the option letter or a number, Reasoning reasons first and then answers (at most 4096 tokens), and Thinking uses Qwen3-VL-8B-Thinking (at most 8192 tokens):

MethodQwen2.5-VL-7B DirectQwen2.5-VL-7B ReasoningQwen3-VL-8B DirectQwen3-VL-8B ReasoningQwen3-VL-8B ThinkingAvg
Baseline68.169.964.772.375.8100.0
FastV51.750.346.045.552.170.1
DART50.647.850.853.755.373.7
EADP58.356.353.758.660.782.1
VLM-Pruner59.454.750.055.258.079.1
Bifocal65.460.958.759.859.887.1

In thinking mode, as the output grows longer, the overall ability of pruning methods drops, while Bifocal remains competitive.

W3: The experimental results are not analyzed sufficiently. In Table 1, all baselines are reported for the 192- and 128-token settings, whereas FastV, PDrop, PruMerge+, and VisionZip are omitted from the 64-token setting without explanation. The paper should clarify the reason for these omissions and provide a more consistent comparison across budgets and backbones.

FastV, PDrop, PruMerge+, and VisionZip are relatively early methods. Due to the page limit, we removed some of the earlier methods with relatively weaker performance from the 64-token setting. We have added these four methods to the 64-token setting of Table 1 in the revised paper. Below is the complete table with the venue of each method. Gen and REC follow Tables 1 and 2 of the paper. They average the following benchmarks relative to the unpruned model (%), with [email protected] for REC:

  • Gen: VQAv2, GQA, MMB, OK-VQA, SQA-I, AI2D, TextVQA, OCRBench, DocVQA, ChartQA, POPE.
  • REC: RefCOCO val, testA and testB, RefCOCO+ val, testA (R+-tA) and testB (R+-tB), RefCOCOg val and test (Rg-t).

Table 1. General and grounding benchmarks on LLaVA-1.5-7B, Average Tokens = 64. Underlined‾\underline{\text{Underlined}} methods are those omitted from Table 1 of the paper.

MethodVenueVQA$^{\mathrm{v2}}$GQAMMBVQAOK^{\mathrm{OK}}VQA^{\mathrm{T}}\DocVQAPOPER+-tARg-tGenREC
No Pruning–78.561.964.153.546.021.585.959.448.4100.0100.0
FastV‾\underline{\text{FastV}}ECCV 202456.446.450.537.031.210.536.40.20.470.40.8
PDrop‾\underline{\text{PDrop}}CVPR 202568.652.455.542.638.914.473.93.42.285.75.3
VisionZip‾\underline{\text{VisionZip}}CVPR 202572.555.160.247.740.915.677.03.73.490.17.4
PruMerge+‾\underline{\text{PruMerge+}}ICCV 202571.455.259.146.937.813.075.81.92.087.14.4
DARTEMNLP 202565.451.154.842.529.311.759.00.81.177.82.5
NüwaICLR 202674.658.662.049.439.915.579.833.924.387.454.9
SIDICLR 202674.257.359.548.438.013.884.32.62.289.85.4
VLM-PrunerCVPR 202669.954.457.743.032.811.969.11.21.382.23.3
HiPruneACL Findings 202669.153.659.546.339.916.173.01.10.987.92.6
EADPECCV 202675.358.160.747.937.614.485.63.22.390.06.2
Bifocal–75.358.562.746.840.917.081.741.930.490.666.7

Across backbones, all baselines are reproduced with their official code, so on Qwen2.5-VL we include those whose official implementations support Qwen. We further compare them on Qwen3-VL-8B (Appendix I.2), and compare the pre-LLM baselines with Bifocal using only its Stage 1 under the same per-layer budget (Appendix I.6).

Reviewer_TZeROpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Bifocal, a training-free visual token pruning framework for multimodal LLMs. It combines query-focused evidence, image-intrinsic evidence, and pooled coarse tokens to preserve both fine-grained details and global visual context. A second pruning stage further removes redundant tokens using attention scores. Experiments across three models and multiple benchmarks demonstrate strong performance, particularly on visual grounding tasks under aggressive token compression.

Critical strengths
  1. The paper identifies an important limitation of existing pruning methods, namely their poor performance on fine-grained grounding despite competitive results on general VQA benchmarks.
  2. The mixed-granularity framework is well motivated, and the analysis of positional and content integrity provides useful insights into the problem.
  3. The experiments are comprehensive, covering multiple models, token budgets, and multimodal tasks. The substantial improvements on grounding benchmarks, supported by ablation studies, make the empirical contribution convincing.
Critical weaknesses
  1. The method introduces nontrivial preprocessing overhead through saliency estimation and greedy coverage selection.
  2. The efficiency analysis is relatively limited and does not fully establish the accuracy-latency trade-off across different settings.
Questions for authors

1.Could the authors provide more detailed ablations isolating the effects of word weighting, geometric-mean aggregation, regression-based saliency correction, and density-aware pruning? This would help clarify whether the proposed mechanisms are individually necessary. 2.How does the computational overhead of Bifocal scale with image resolution and token budget? A breakdown of preprocessing costs and a more comprehensive accuracy-latency comparison would help assess its practical efficiency.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TZeR (Part 1/4)

We thank the reviewer for the positive assessment and constructive comments, and for recognizing our mixed-granularity framework. We address each concern below, and newly added or substantially revised text in the manuscript is highlighted in blue.

W1: The method introduces nontrivial preprocessing overhead through saliency estimation and greedy coverage selection.

Thank you for the suggestion. The saliency estimation and greedy coverage in Bifocal can indeed raise concerns about latency. We took this into account and optimized their implementation with fused kernels and CUDA Graph:

(1) Reusing attention. On LLaVA-1.5, saliency directly reuses the CLS attention already computed by the penultimate ViT layer, so no extra forward pass is needed.

(2) Fused saliency kernel. On Qwen2.5-VL, saliency estimation only computes the column mass on one full-attention block of the ViT, and the softmax and column sum are compiled into a fused kernel.

(3) CUDA Graph for greedy coverage. Greedy coverage replays all rounds at once with a CUDA Graph. On Qwen2.5-VL, the gain computation of each round is fused into one kernel, and the selection and update of each round are fused into another, so each round launches only two kernels.

(4) Cached query word embeddings. The embedding of each query word is computed once and then reused.

To show the cost of the pruning operations more clearly, we add a stage-wise latency breakdown in Table 1 and compare the effect of the greedy optimization separately in Table 2. For a fair comparison, all methods are measured on a single NVIDIA A100 (40 GB) GPU. The ViT uses eager attention and the LLM uses SDPA attention throughout. The data is TextVQA val.

Table 1. Latency breakdown of Bifocal in ms. Columns in parentheses are parts of the stage to their left.

MethodViTStage 1(Saliency + poolingGreedy coverage)LLM prefill(Stage 2)TTFT
LLaVA-1.5-7B
No Pruning8.70–––51.80–61.39
Bifocal, 192 tokens8.707.283.533.7534.12 (1.52×)1.4150.60 (1.21×)
Bifocal, 128 tokens8.705.323.531.7929.41 (1.76×)1.4143.84 (1.40×)
Bifocal, 64 tokens8.704.083.530.5526.44 (1.96×)1.4239.61 (1.55×)
Qwen2.5-VL-7B
No Pruning51.52–––91.75–146.36
Bifocal, 50%51.7911.617.494.1252.80 (1.74×)0.49121.96 (1.20×)
Bifocal, 25%51.7410.077.492.5834.78 (2.64×)0.49101.98 (1.44×)
Bifocal, 15%50.719.247.441.8031.07 (2.95×)0.4796.27 (1.52×)

Table 2. Effect of the greedy coverage optimization. ΔGen and ΔREC are accuracy changes relative to the optimized greedy coverage.

Image-intrinsic selectionGreedy time (ms)TTFT (ms)SpeedupΔGenΔREC
LLaVA-1.5-7B, 128 tokens
Greedy coverage8.4951.081.20×00
Greedy coverage, optimized1.7943.841.40×00
Top-K043.191.42×−0.4−1.3
Qwen2.5-VL-7B, 25%
Greedy coverage24.13125.141.17×00
Greedy coverage, optimized2.58101.981.44×00
Top-K099.121.48×−0.6−1.5

Overall, the time that pruning saves in inference is larger than the cost of the selection operations. After optimization, greedy coverage adds little latency, and the smaller the budget, the faster it runs.

AuthorsOpenReview ↗

Response to Reviewer TZeR (Part 2/4)

W2: The efficiency analysis is relatively limited and does not fully establish the accuracy-latency trade-off across different settings.

Thank you for pointing this out. To show the accuracy–latency trade-off more completely across settings, we take three budgets each on LLaVA-1.5-7B and Qwen2.5-VL-7B and measure the TTFT of Bifocal and representative baselines. Tables 1 and 2 list the TTFT together with general and REC accuracy, and a more intuitive accuracy–latency plot is given in Figure 12 (Appendix H). In addition, we report the maximum measured speedup that each method reaches under the same accuracy requirement. The hardware and measurement settings are the same as in W1.

Table 1. Accuracy and latency on LLaVA-1.5-7B. TTFT of No Pruning is 61.39 ms.

BudgetMetricHiPruneSIDEADPDARTVLM-PrunerBifocal, Stage 1 onlyBifocal
192TTFT1.45×0.64×0.80×1.31×1.00×1.38×1.21×
Gen98.193.996.796.297.298.298.5
REC10.69.514.060.281.991.2111.0
128TTFT1.53×0.68×0.97×1.38×1.13×1.53×1.40×
Gen95.091.594.592.694.895.896.3
REC5.85.98.331.050.872.1101.8
64TTFT1.66×1.36×1.20×1.43×1.43×1.72×1.55×
Gen87.989.890.077.882.289.190.6
REC2.65.46.22.53.330.166.7

Table 2. Accuracy and latency on Qwen2.5-VL-7B. TTFT of No Pruning is 146.36 ms.

BudgetMetricVisionZipHiPruneEADPDARTVLM-PrunerBifocal
50%TTFT1.27×1.21×0.85×1.40×1.13×1.20×
Gen91.591.793.493.197.498.4
REC89.689.491.585.594.098.3
25%TTFT1.47×1.39×1.17×1.62×1.44×1.44×
Gen84.885.186.880.390.394.9
REC69.967.967.929.949.190.2
15%TTFT1.63×1.51×1.43×1.72×1.62×1.52×
Gen77.077.781.468.077.889.0
REC43.840.340.96.99.567.9

Table 3. Maximum TTFT speedup under an accuracy requirement on Qwen2.5-VL-7B. For each method, the fastest of the three ratios that meets the requirement is reported, and – means that none of them meets it

Accuracy requirementVisionZipHiPruneEADPDARTVLM-PrunerBifocal
Gen ≥ 95––––1.13×1.20×
Gen ≥ 901.27×1.21×0.85×1.40×1.44×1.44×
REC ≥ 90––0.85×–1.13×1.44×
Gen ≥ 90 and REC ≥ 90––0.85×–1.13×1.44×

Overall, smaller budgets bring larger speedups for all methods but also lower accuracy. Under the same accuracy requirement, Bifocal is faster. On Qwen2.5-VL, under Gen ≥ 90 and REC ≥ 90, Bifocal reaches 1.44×, while the best of the other methods reaches only 1.13×.

AuthorsOpenReview ↗

Response to Reviewer TZeR (Part 3/4)

Q1: Could the authors provide more detailed ablations isolating the effects of word weighting, geometric-mean aggregation, regression-based saliency correction, and density-aware pruning? This would help clarify whether the proposed mechanisms are individually necessary.

Some of these components were analyzed in Appendices D and E, but due to time constraints the analysis was not clear or complete enough. We now add complete ablations, and the full table is given in Appendix G.2.

Table 1. Ablations of the four mechanisms on LLaVA-1.5-7B at 128 tokens. Each row changes one component of Bifocal, and Rg-t is RefCOCOg test.

VariantTextVQAOCRBenchDocVQAPOPER+-tAR+-tBRg-tGenREC
Bifocal45.726120.484.761.142.049.496.4104.4
Word weighting
word weighting → uniform45.526420.484.661.341.448.596.4 (0.0)103.4 (−1.0)
Geometric-mean aggregation
geometric → arithmetic mean45.623121.084.860.441.448.995.7 (−0.7)103.2 (−1.2)
geometric mean → per-cell max45.522720.784.960.141.148.395.4 (−1.0)102.3 (−2.1)
geometric mean → mean word embedding45.523520.984.661.041.849.395.7 (−0.7)104.1 (−0.3)
Regression-based saliency correction
w/o norm-regression correction45.626120.584.459.239.347.296.4 (−0.1)99.4 (−5.0)
Density-aware pruning
w/o density prior45.324520.684.860.141.248.695.9 (−0.6)102.5 (−1.9)

For token selection, removing any of the mechanisms causes a drop, and the effect is larger on REC. Word weighting mainly affects REC, as we expected. In addition, these components are not assembled arbitrarily. Following the pruning framework derived from our observations, each kind of token has a clear goal: query-focused tokens retain the content the question points to, and image-intrinsic tokens add diversity beyond the question.

AuthorsOpenReview ↗

Response to Reviewer TZeR (Part 4/4)

Q2: How does the computational overhead of Bifocal scale with image resolution and token budget? A breakdown of preprocessing costs and a more comprehensive accuracy-latency comparison would help assess its practical efficiency.

Thank you for the helpful suggestion. The breakdown of the preprocessing cost is covered by the additional experiments in W1. Here we show how the cost of Bifocal scales with image resolution and token budget. Since LLaVA-1.5 does not support dynamic resolution, we run these experiments on Qwen2.5-VL-7B, add Qwen3-VL-8B to show generalization, and measure speed on DocVQA val, whose images have higher resolution.

Table 1. Scaling of Bifocal with resolution on Qwen2.5-VL-7B. Each side of the image must be a multiple of 28.

ResolutionVisual tokensNo Pruning TTFT (ms)LLM prefill (ms)TTFT (ms)Peak memory (GiB)
Keep 25%
448×44824370.4625.79 (1.38×)67.58 (1.04×)16.08
616×61648890.2226.97 (1.84×)74.60 (1.21×)16.08
896×896997143.0435.16 (2.62×)97.42 (1.47×)16.09
1260×12601981263.5953.97 (3.05×)172.45 (1.53×)16.21
1792×17923766526.9791.39 (3.63×)369.71 (1.43×)16.96
Keep 15%
448×44824370.4625.25 (1.41×)66.77 (1.06×)16.08
616×61648890.2225.77 (1.92×)72.39 (1.25×)16.08
896×896997143.0431.60 (2.91×)92.40 (1.55×)16.09
1260×12601981263.5937.90 (4.35×)155.68 (1.69×)16.21
1792×17923766526.9762.61 (5.29×)340.40 (1.55×)16.96

Table 2. Scaling of Bifocal with resolution on Qwen3-VL-8B. Each side of the image must be a multiple of 32.

ResolutionVisual tokensNo Pruning TTFT (ms)LLM prefill (ms)TTFT (ms)Peak memory (GiB)
Keep 25%
512×51224364.6335.99 (1.19×)64.50 (1.00×)16.98
704×70448894.9037.39 (1.68×)77.16 (1.23×)16.98
1024×1024997169.4044.74 (2.37×)120.66 (1.40×)16.99
1440×14401933348.3670.14 (2.86×)239.52 (1.45×)17.15
2048×20483509743.90102.46 (3.71×)540.02 (1.38×)17.93
Keep 15%
512×51224364.6336.62 (1.17×)65.23 (0.99×)16.98
704×70448894.9037.19 (1.69×)77.20 (1.23×)16.98
1024×1024997169.4041.61 (2.55×)116.91 (1.45×)16.99
1440×14401933348.3645.93 (4.37×)215.07 (1.62×)17.15
2048×20483509743.9073.72 (5.15×)509.55 (1.46×)17.93

As the resolution grows, the prefill speedup keeps increasing steadily and even exceeds 5×, and TTFT also improves. However, current pruning methods, including ours, operate after the pretrained vision encoder, so the ViT itself is not affected by pruning, and the TTFT speedup stops rising at very high resolutions.

Reviewer_6xSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Bifocal, a two-stage, training-free visual token pruning method. The motivation is that existing methods score similarly on general benchmarks but drop sharply on fine-grained tasks such as grounding.

The method has two stages:

  • Stage 1 (before the LLM): The token budget is split across three types of tokens, mixing fine and coarse granularity. Query-focused: Query-focused evidence keeps the regions the question points to. Per-word cosine saliency maps are computed and smoothed. Image saliency is the residual of CLS attention after linear regression on the norm. Each word is weighted by the correlation between its map and the image saliency, and the words are combined with a weighted geometric mean. Image-intrinsic: Image-intrinsic evidence covers the image itself, through greedy selection on an attention-weighted submodular coverage objective. Coarse context: The unselected patches are average-pooled into coarse tokens.
  • Stage 2 (intermediate LLM layer): Following FastV, the top-K tokens are kept according to text-to-image attention, multiplied by a local density prior.

Experiments cover three models (LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL), each at three budgets. The general-task average is slightly better than the baselines, while the grounding lead is large. For example, LLaVA-1.5 reaches 101.8% REC at a layer-averaged 128 tokens.

Critical strengths
  1. The finding in Fig. 3 is valuable. Keeping only the patches inside the GT box gives lower REC than two-source selection, and adding the pooled tokens recovers nearly unpruned performance. So grounding collapse is caused mainly by broken positional and content integrity, not by "selecting the wrong tokens." This is the most substantive finding of the paper.
  2. The grounding gains are large and consistent. Bifocal leads across all three models and every budget: on LLaVA-NeXT at 160 tokens it reaches 52.1%, versus 19.0% for the strongest baseline. In Appendix C, plugging HiPrune, VisionZip, and EADP into the framework raises their grounding by 10–20 points. This shows the framework itself is effective and the gains do not just come from hyperparameter tuning. 3。 The evaluation is broad: 20 benchmarks across five ability areas, with most baselines reproduced under a unified setup.
Critical weaknesses
  1. The budget comparison may be unfair. A "layer-averaged 128" budget means Bifocal actually keeps 224 tokens in the first 16 layers, while one-shot pruning baselines keep only 128 throughout. Grounding is highly sensitive to visual information in shallow layers, so this gap may explain a large part of the advantage. FLOPs are roughly matched, but a comparison with an equal Stage 1 budget is needed (e.g., 224 vs. 224, or no Stage 2 for any method).
  2. The ablations are insufficient. The C/I/Q allocation is ablated on only one model at one budget. There are no ablations for smoothing, norm-regression debiasing, word weighting, geometric vs. arithmetic mean, Stage 2 itself, or the choice of pruning layer L. For a method built by stacking several techniques, this is a serious gap. How was the 64/80/80 allocation chosen? Was it tuned on the test sets?
  3. The theory is oversold. The "exact second-order Taylor expansion" is just an algebraic identity, and removing a multiplicative gain with linear regression is only a first-order heuristic. The cross term in Eq. 3 is largest at sink patches, which deviate most from the mean, so the approximation fails exactly where the correction is most needed. Table 6 itself shows that high-norm patches still make up 46–50% of the top 10% after correction.
  4. The writing is overly rhetorical. Metaphors such as "invite the image back to sit on the jury" and "referee" get in the way of reading.
Questions for authors
  1. When reproducing the baselines, were the position ids of the retained tokens kept at their original grid positions, or were they renumbered consecutively? HiPrune and SID reach only single-digit REC. Is that also the case in the original papers or the official code?
  2. Why does REC on LLaVA-1.5 reach 111%, exceeding the unpruned model?
  3. Under an equal Stage 1 budget (e.g., all methods use 224 tokens followed by the same Stage 2, or Bifocal drops Stage 2 and uses only 128 tokens), how much of the grounding advantage remains?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6xSq (Part 1/5)

We thank the reviewer for the thorough review and constructive comments, and for recognizing our mixed-granularity pruning framework. We address each concern below.

For brevity, the tables below show only representative metrics, while the full results are provided in the revised manuscript. Newly added or substantially revised text is highlighted in blue for easy reference. Unless otherwise noted, Gen and REC are the average scores over the following benchmarks, each relative to the unpruned model (%), and REC uses [email protected]:

  • Gen: GQA, MMB, OK-VQA, SQA, AI2D, TextVQA, OCRBench, DocVQA, POPE.
  • REC: RefCOCO val, RefCOCO testA (R-tA), RefCOCO testB, RefCOCO+ val, RefCOCO+ testA (R+-tA), RefCOCO+ testB (R+-tB), RefCOCOg val, and RefCOCOg test (Rg-t).

W1: The budget comparison may be unfair. A "layer-averaged 128" budget means Bifocal actually keeps 224 tokens in the first 16 layers, while one-shot pruning baselines keep only 128 throughout. Grounding is highly sensitive to visual information in shallow layers, so this gap may explain a large part of the advantage. FLOPs are roughly matched, but a comparison with an equal Stage 1 budget is needed (e.g., 224 vs. 224, or no Stage 2 for any method).

  • Experiment 1: Fairness Comparison:

A fixed token count throughout the model only applies to one-shot projector-level pruning. Instead of the widely reported number of finally retained visual tokens, we report the layer-averaged token count to compare a broader range of pruning methods at approximately matched FLOPs, including LLM-level pruning and multi-stage pruning.

To address the reviewer's fairness concern, we compare Bifocal with projector-level baselines under an equal budget, where all methods prune only once before the LLM and Bifocal uses only its Stage 1.

Table 1. LLaVA-1.5-7B.

MethodGQAMMBTextVQAR-tAR+-tARg-tGenREC
Baseline61.964.146.064.459.448.4100.0100.0
192 Tokens
PruMerge+57.661.339.01.81.82.290.55.5
VisionZip59.363.844.55.44.64.397.710.3
HiPrune59.263.044.95.54.84.898.210.6
SID58.963.340.95.24.24.193.69.5
Bifocal60.563.845.060.954.943.398.291.2
128 Tokens
PruMerge+57.660.339.20.91.01.489.93.6
VisionZip57.762.243.83.33.02.795.36.8
HiPrune57.361.943.63.02.42.495.25.8
SID58.062.239.33.02.82.491.25.9
Bifocal59.563.644.351.045.033.395.872.1

Table 2. Qwen2.5-VL-7B.

MethodGQAMMBTextVQAR-tAR+-tARg-tGenREC
Baseline60.984.082.891.688.984.3100.0100.0
50% Tokens
VisionZip57.478.774.881.981.177.491.589.6
HiPrune57.379.274.881.280.677.891.789.4
EADP59.582.677.386.481.976.693.491.5
Bifocal59.983.281.789.585.681.098.896.3
25% Tokens
VisionZip55.177.068.666.665.059.084.869.9
HiPrune54.977.768.864.463.258.485.167.9
EADP58.679.570.868.664.956.186.867.9
Bifocal58.781.178.280.774.666.894.681.1

(1) Under the same pre-LLM budget, Bifocal still leads on both base models.

(2) All baselines are reproduced with their official code. Positional encoding and reproduction fidelity are addressed in Q1. In the Qwen comparison, all methods keep the original positional encoding.

  • Experiment 2: Layer-wise Budget Comparison:

We agree that shallow-layer visual information is important. Our paper also regards Bifocal as a two-stage method, which takes advantage of the two stages, and the multi-stage idea also draws on prior methods [1]. However, this is not the only source of Bifocal's improvement. We add a layer-wise budget comparison: with the same or even larger shallow-layer budgets, some methods still perform worse than Bifocal:

MethodL1–L2L3–L8L9–L16L17–L32GenREC
FastV57698989889.517.6
VLM-Pruner57698989894.850.8
DART57610010010092.631.0
PDrop576192966494.937.4
Nüwa2242242243293.691.1
Bifocal2242242243296.3101.8

References

[1] Long Xing, et al. Conical Visual Concentration for Efficient Large Vision-Language Models. CVPR 2025.

AuthorsOpenReview ↗

Response to Reviewer 6xSq (Part 2/5)

W2: The ablations are insufficient. The C/I/Q allocation is ablated on only one model at one budget. There are no ablations for smoothing, norm-regression debiasing, word weighting, geometric vs. arithmetic mean, Stage 2 itself, or the choice of pruning layer L. For a method built by stacking several techniques, this is a serious gap. How was the 64/80/80 allocation chosen? Was it tuned on the test sets?

  • Experiment 1: allocation of the three token types

64/80/80 is not obtained by search, nor tuned on the test sets. Taking LLaVA as an example, each image has 24×24 = 576 tokens, so non-overlapping pooling gives only a few token counts (e.g., 16, 36, 64, 144). We take 64, the closest to 1/3 of the budget, and split the rest evenly between the other two types, which gives 64/80/80.

We extend the allocation ablation to more budgets and models (Appendix G.1). CC, II, QQ: numbers of coarse-context, image-intrinsic and query-focused tokens.

Table 1. LLaVA-1.5-7B, 128 tokens, CC = 64.

II / QQ0 / 16020 / 14040 / 12060 / 10080 / 80100 / 60120 / 40140 / 20159 / 1
Gen94.794.795.295.496.095.395.495.194.9
REC105.0103.9103.6103.5102.1102.5101.8101.2100.3

Table 2. LLaVA-1.5-7B, 192 tokens.

CC / II / QQMMBPOPER+-tARg-tGenREC
0 / 0 / 33664.285.163.753.797.2111.0
0 / 168 / 16864.385.764.153.097.7109.4
0 / 335 / 164.485.864.552.597.4108.9
64 / 96 / 17664.485.665.253.697.6111.8
64 / 116 / 15664.385.765.653.497.5111.2
64 / 136 / 13664.485.766.254.298.3111.5
64 / 156 / 11664.385.564.853.497.6110.7
144 / 0 / 19264.085.465.353.997.5112.3
144 / 96 / 9664.185.764.952.897.2110.5
144 / 191 / 164.185.664.752.297.1109.1

Table 3. LLaVA-1.5-13B, 128 tokens.

CC / II / QQMMBPOPER+-tARg-tGenREC
0 / 0 / 22462.782.259.647.388.385.6
0 / 112 / 11266.985.965.951.597.595.3
0 / 223 / 167.485.464.951.897.794.3
64 / 0 / 16067.184.267.654.396.299.3
64 / 80 / 8067.585.566.553.497.297.4
64 / 159 / 168.185.466.053.297.696.1

Table 4. Qwen2.5-VL-7B, 25% tokens.

CC / II / QQDocVQAPOPEGenREC
29% / 0% / 100%86.685.993.884.8
29% / 50% / 50%90.286.695.087.9
29% / 62.5% / 37.5%90.387.095.188.1
29% / 75% / 25%90.486.795.088.2
29% / 99% / 1%90.286.594.988.3
50% / 0% / 100%86.486.093.484.6
50% / 50% / 50%88.986.794.388.0
50% / 99% / 1%89.486.594.688.4

The default allocation is not the best setting on every metric, as it would be if tuned on the test sets: on LLaVA-1.5-7B with CC fixed, REC generally rises as QQ grows, while on Qwen2.5-VL-7B giving all fine tokens to QQ is the worst setting. In both cases, the default stays close to the best rows on Gen and REC. On LLaVA-1.5-13B, removing CC lowers REC, and removing both CC and II also hurts GQA and POPE.

  • Experiment 2: component ablations

These components may look like a stack of tricks, but they are the score computation and merging along two main lines, query and image, with a clear goal: the query retains the relevant content, while the image adds diversity. Some of them are analyzed in the appendix, but not clearly or completely. We now add complete ablations:

VariantOCRBenchPOPER+-tAR+-tBGenREC
Bifocal (Stage 1: 224)26184.761.142.096.4104.4
Query scoring
w/o smoothing26684.961.541.696.5 (0.0)103.9 (−0.5)
w/o norm-regression correction26184.459.239.396.4 (−0.1)99.4 (−5.0)
word weighting → uniform26484.661.341.496.4 (0.0)103.4 (−1.0)
geometric → arithmetic mean23184.860.441.495.7 (−0.7)103.2 (−1.2)
geometric mean → per-cell max22784.960.141.195.4 (−1.0)102.3 (−2.1)
geometric mean → mean word embedding23584.661.041.895.7 (−0.7)104.1 (−0.3)
Stage 2
w/o density prior24584.860.141.295.9 (−0.6)102.5 (−1.9)
LL = 8 (Stage 1: 416)16269.06.43.178.3 (−18.1)8.7 (−95.7)
LL = 12 (Stage 1: 288)17085.347.334.989.9 (−6.6)83.8 (−20.6)
LL = 24 (Stage 1: 160)30183.851.632.896.6 (+0.1)83.9 (−20.5)
AuthorsOpenReview ↗

Response to Reviewer 6xSq (Part 3/5)

W3: The theory is oversold. The "exact second-order Taylor expansion" is just an algebraic identity, and removing a multiplicative gain with linear regression is only a first-order heuristic. The cross term in Eq. 3 is largest at sink patches, which deviate most from the mean, so the approximation fails exactly where the correction is most needed. Table 6 itself shows that high-norm patches still make up 46–50% of the top 10% after correction.

The high-norm correction aims to weaken the influence of the patch norm on the CLS attention, so that tokens that better reflect the visual content are selected. Some expressions in the paper, such as the "exact" Taylor expansion and the norm "bias", were imprecise and could cause misunderstanding, and we have revised them. We clarify the motivation, approximation and effectiveness of the correction below.

(1) Taylor expansion and the cross term. We agree that the mean-centered second-order Taylor expansion is essentially an algebraic identity. Its motivation is to decouple the norm-related gain from the content, rather than to piece together an overstated theory. As derived in Appendix E, the CLS attention can be written as aj=njcja_j=n_jc_j, where njn_j is the norm-related gain and cjc_j is the content score. Expanding the product xyxy around the means separates the gain from the content:

aj=cˉnj+nˉcj−nˉcˉ+(nj−nˉ)(cj−cˉ)a_j=\bar{c}n_j+\bar{n}c_j-\bar{n}\bar{c}+(n_j-\bar{n})(c_j-\bar{c})

Removing the gain-only term cˉnj\bar{c}n_j leaves rj=nj(cj−cˉ)r_j=n_j(c_j-\bar{c}). The cross term is therefore not discarded, but kept in the residual. The sign of rjr_j follows cj−cˉc_j-\bar{c}, while its magnitude is still scaled by njn_j, so a patch with a large gain but below-average content is still ranked low. Since token selection depends only on the ranking of the scores, the goal is not to remove the norm-related component from the score values, but to keep patches with low content from being ranked on top.

(2) Regression approximation. Since the gain nj=eGρjn_j=e^{G\rho_j} cannot be observed, we use the patch norm map ν\nu to approximate its effect and subtract the fitted component from the smoothed CLS attention:

Sj=a~j−(β^νj+b^)S_j=\tilde{a}_j-(\hat\beta\nu_j+\hat b)

This is a first-order approximation rather than an exact removal of the multiplicative gain. We choose it because it is simple and the most general across model architectures. A log-domain residual would be exact for a single softmax, where log⁡aj=Gρj+log⁡cj\log a_j=G\rho_j+\log c_j, but Qwen has no CLS token and its saliency map is a sum of softmaxes over typically thousands of queries, so the log domain is not exact either. Both forms carry some approximation error, which does not undermine the effectiveness shown in (3) below.

Overall, as the reviewer notes, some patches with extremely high norm may still remain. Our goal is not to remove all attention sinks, which may also aggregate useful global information, but to keep uninformative high-norm patches from dominating the selection.

(3) Possible misunderstanding and clarification. Table 6 of the original submission (Table 7 in the revision, Appendix E) takes the 5% of patches with the highest norm. The 46%–50% is the fraction of these patches that still rank in the top 10% after the correction, down from 64%–75% before it. It is not the share of high-norm patches among the top 10%, as the two use different denominators. The table below gives more intuitive metrics on LLaVA-1.5-7B. η\eta is the metric above. The third column is the share of high-norm patches among the top-28 patches. The last two columns count how many of the final top-16 and top-80 patches fall inside the ground-truth box:

Splitη\etaHigh-norm among top-28In-box among top-16In-box among top-80
GQA75% → 46%64% → 32%——
RefCOCO testA68% → 46%64% → 32%2.0 → 3.519.9 → 22.2
RefCOCO testB75% → 50%64% → 32%1.8 → 2.817.4 → 20.3

This gives more direct evidence. After the correction, the share of high-norm patches among the top-28 tokens drops sharply, while some sinks that carry content are still kept. More tokens inside the ground-truth box, which carry more useful content, are selected after the correction.

W4: The writing is overly rhetorical. Metaphors such as "invite the image back to sit on the jury" and "referee" get in the way of reading.

Thank you. We have revised the rhetorical expressions throughout the paper into plain technical language. For example, "we invite the image back to sit on the jury alongside the question: a word earns a louder voice when…" is now "we weight each word by how well its saliency map agrees with the image saliency map", and "referee" is replaced by "serve as the reference for word weighting". We hope the reviewer understands that these metaphors were originally intended to help readers grasp the intuition, and to make the paper a bit more engaging.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their constructive comments. We have uploaded a revised version of the paper, with all changes highlighted in blue. The main revisions are summarized below, where [Rn] refers to the n-th reviewer. Overall, the revision strengthens the paper in three aspects: generalization, with Bifocal now evaluated on 4 models, multiple budgets and 26 benchmarks covering 7 types of multimodal tasks; efficiency, with a more detailed analysis of the overhead of Bifocal and the accuracy–latency trade-off; and ablations, with comprehensive ablation studies that verify the completeness of Bifocal.

Experiments

  • [R1] Equal per-layer budgets. Appendix I.6 compares the methods under the same pre-LLM budget on LLaVA-1.5-7B and Qwen2.5-VL-7B.
  • [R1, R2] Ablations. Appendix G.1 extends the allocation ablation of CC, II and QQ to another budget, LLaVA-1.5-13B and Qwen2.5-VL-7B. Appendix G.2 adds detailed ablations of the individual components.
  • [R2] Efficiency. A new efficiency subsection in the main text compares TTFT and memory with the baselines. Appendix H adds the accuracy–latency trade-off at three budgets on two models, the latency of each stage, and the scaling with image resolution.
  • [R3] More models and tasks. Appendix I.2 adds Qwen3-VL-8B-Instruct. Appendix I.3 adds multi-image (MuirBench), video (MVBench and Charades-STA) and text-intensive reasoning tasks (MMMU, MathVista and TreeBench) on Qwen2.5-VL-7B and Qwen3-VL-8B-Instruct, where Bifocal achieves the best average on both models.
  • [R3] Reasoning models and long decoding. Appendix I.4 adds Qwen3-VL-8B-Thinking with thinking enabled, and compares direct answering, reasoning and thinking on MathVista.
  • [R3] Complete comparison at 64 tokens. FastV, PDrop, PruMerge+ and VisionZip are added to the 64-token setting of Table 1.

Writing

  • [R1] Theory. We have refined the wording of the high-norm correction, such as "exact" and "bias".
  • [R1] Metaphors. Rhetorical expressions such as "jury" and "referee" are replaced with plain and standard descriptions.
  • [R1] Reproduction details. We state that all baselines are reproduced with their official code, and that Bifocal keeps the original positional encoding of the retained tokens.
  • [R3] Experimental setup. The newly added models and benchmarks are listed in the experimental setup.

In the individual responses, we answer each question in detail. We hope that the responses and revisions address your concerns, and we are happy to discuss any further questions.

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

AuthorsOpenReview ↗

Summary of the Rebuttal for the AC

Dear AC and SAC,

We sincerely thank all reviewers for their constructive comments and the AC for handling our submission. All changes in the revised paper are highlighted in blue. To facilitate your assessment, we summarize the paper and the discussion below, and the details are given in the individual responses.

1. Overview of Bifocal

Our motivation is that existing visual token pruning methods suffer substantial performance degradation on fine-grained tasks, particularly visual grounding. Before the LLM, tokens can be selected based on two kinds of information, the image and the question. However, different benchmarks rely on different kinds of evidence, and a single set of selected tokens loses both content integrity and positional integrity. Based on these observations, we propose a mixed-granularity pruning framework that consists of coarse pooled tokens and fine-grained evidence. While the coarse tokens maintain integrity, the two kinds of fine-grained tokens have clear goals: query-focused tokens preserve the content that the question points to, and image-intrinsic tokens enhance coverage. We design targeted algorithms for each of them, which together form Bifocal. After the rebuttal, Bifocal has been evaluated on four model families, multiple budgets and 26 benchmarks covering 7 types of multimodal tasks, showing strong performance.

2. Discussion Status and Key Responses

Reviewer 6xSq (Rating 2, Confidence 3): has not responded to our rebuttal yet.

  1. Fairness. The reviewer was concerned that the comparison is unfair, especially against one-shot pruning methods, and asked how the positional encoding of the retained tokens is handled.

    (1) Budget. We add a comparison between one-shot methods and Stage-1-only Bifocal under the same pre-LLM budget, and further equip these methods with our Stage 2. Bifocal remains ahead on both LLaVA-1.5 and Qwen2.5-VL. In the paper, we report the layer-averaged token count at approximately matched FLOPs, instead of the commonly reported number of finally retained tokens, so that LLM-level and multi-stage methods can also be compared. As stated in the paper, Bifocal is a two-stage method following prior work.

    (2) Reproduction. All baselines strictly follow their official code. We reproduce them to obtain REC, which most of them do not report, and verify that our results are close to those reported in their papers and by third-party papers.

    (3) Positional encoding. Bifocal keeps the original encoding. On LLaVA, methods use different encoding tricks, so we add a comparison where the modifiable baselines adopt the same encoding. On Qwen, all methods keep the original encoding because of M-RoPE, so we further compare them on Qwen2.5-VL and Qwen3-VL. Bifocal still leads, especially on REC.

  2. Ablations. We add finer-grained allocation ablations on LLaVA-1.5-7B, allocation ablations on LLaVA-1.5-13B and Qwen2.5-VL-7B, and ablations of each component.

  3. Theory and writing. We re-explain the motivation and derivation of the norm-related correction, clarify a possible misunderstanding of the metric in the high-norm patch table, and replace rhetorical metaphors with plain technical descriptions. We also check and standardize the writing throughout the paper.

Reviewer TZeR (Rating 4, Confidence 3)

  1. Efficiency. (1) We give a clear stage-wise breakdown of the overhead and the gain on two models at three budgets each. (2) We add experiments on resolution and latency under multiple budgets on Qwen2.5-VL-7B and Qwen3-VL-8B. As the resolution increases, the prefill speedup of both models keeps rising and exceeds 5×. (3) We compare the accuracy and TTFT of Bifocal and the baselines at three budgets on two models, and report the maximum speedup each method reaches under the same accuracy requirement. Under Gen ≥ 90 and REC ≥ 90, Bifocal is the fastest.

  2. Ablations. Addressed together with Reviewer 6xSq above.

Reviewer VAWv (Rating 2 → 3, Confidence 3)

  1. Generalization. (1) We add experiments verifying that the analysis behind our framework still holds in more multimodal scenarios. (2) We additionally evaluate six multi-image, video and text-intensive benchmarks on Qwen2.5-VL-7B and Qwen3-VL-8B-Instruct. (3) We add the evaluation of Qwen3-VL-8B-Thinking. (4) We compare direct answering, CoT reasoning and thinking of up to 8192 tokens on MathVista. In all these settings, Bifocal still leads.

  2. Missing comparisons. The early classic methods omitted from the 64-token setting due to the page limit are added back.

  3. Dynamic allocation. The reviewer's idea coincides exactly with what we explored during development. From benchmark identification, benchmark-wise allocation and sample-wise allocation, we show that dynamic allocation can bring gains, but its upper bound is low and it does not generalize well to unseen tasks.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.