acceptodds
Under review as a conference paper at ICLR 2027

Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Visual Token Pruning in MLLMs

Abstract

Multimodal large language models (MLLMs) process hundreds or thousands of visual tokens per image, incurring substantial inference costs. While existing visual token pruning methods mitigate this overhead, they implicitly assume that a single fixed pruning strategy can be applied uniformly across all inputs. Our analysis reveals that ranking pruning methods by average benchmark accuracy conceals substantial sample-wise complementarity: although the average-best strategy excels overall, alternative strategies prove superior on a considerable fraction of individual samples. To harness this diversity, we propose **VIP-Router**, a lightweight **VI**sion **P**runing **Router** that, conditioned on low-cost visual and textual features, selects for each input the pruning strategy predicted to be most suitable at a specified pruning level, while retaining full-token inference as a fallback when pruning is predicted to be unfavorable. On VTC-Bench Group A, a pruning-sensitive suite spanning eight vision-language benchmarks, VIP-Router consistently outperforms the best fixed pruning strategy at every reduction ratio, with relative gains of 26.9% in average accuracy and 22.0% in average utility, a cost-aware metric that penalizes each method for the visual tokens it actually retains. Under stricter image-disjoint splits, VIP-Router still improves average utility over the best fixed strategy by 4.7% in relative terms, with gains in all 15 runs (three splits × five seeds). Moreover, VIP-Router is plug-and-play: it modifies neither the pruning algorithms nor the model weights and adds only 1.38M trainable parameters (0.017% of the backbone). It also remains effective on four additional MLLMs and yields gains on four unseen benchmarks, highlighting the potential of sample-adaptive routing for visual token pruning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Vision Token Pruning, Adaptive Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_aevdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method named VIP-Router, which selects a visual token pruning strategy for each input. The proposed method uses a lightweight router to predict the utility of existing pruning methods from visual and textual features. The router can also select full-token inference as a fallback. Experiments are conducted on VTC-Bench to evaluate the performance of VIP-Router compared to baselines. The results show improved accuracy and utility, with additional evaluations using image-disjoint splits and different backbones.

Critical strengths
  • The paper observes that different pruning methods work well on different samples. This provides a clear and reasonable motivation for selecting a strategy for each input.

  • The paper provides image-disjoint evaluation with three splits and five seeds. These experiments help show the performance of the method beyond the main evaluation setting.

Critical weaknesses
  • The main evaluation only uses samples that the full-token model answers correctly. Therefore, selecting Full always gives a correct answer in VTC-Bench Group A. However, the router is not evaluated on the unfiltered benchmark, where full-token inference can also fail. The appendix shows that different pruning strategies still work well on different samples in the unfiltered data. However, it does not show if the trained router can select the right strategy for these samples.

  • The performance gains are much smaller under image-disjoint evaluation. The average utility improves by 22.0% under record-disjoint evaluation, but only 4.7% under image-disjoint evaluation. These two settings use different test sets. Therefore, the difference cannot be explained only by image overlap.

  • The paper does not show if the performance gains will hold under the same computation budget. At 99% reduction, VIP-Router retains 7.6% of visual tokens on average, compared to 1% for the fixed baseline, because it can select Full. The proposed utility includes a penalty for using more tokens. However, the fraction of retained tokens does not directly measure inference latency or FLOPs.

  • QMoP already uses query-guided routing to select among different visual compression branches. VIP-Router selects among existing pruning methods, but the idea of routing is similar. Additionally, Figure 1 shows that different pruning strategies work best on different benchmarks. However, the paper does not compare with a simpler router that selects strategies based on task category and retention ratio.

  • The cost of training the router is not fully clear. Training requires offline correctness labels for several pruning methods at five retention ratios. The experiments with different backbones also train a separate router for each backbone. Without these results, the practical advantage over a strong fixed strategy is not clear, especially given the smaller gains under image-disjoint evaluation.

Questions for authors
  • Can the authors provide results for the complete VIP-Router on the unfiltered VTC-Bench test set? Full can also give incorrect answers outside Group A.

  • Can the authors show if VIP-Router still outperforms the strongest fixed pruning strategy under the same actual inference FLOPs or latency? Comparing the same nominal pruning ratios does not make this clear.

  • Can the authors report the offline computation needed to obtain correctness labels for each strategy and train the router for a new backbone? How does this cost compare to the computation saved during inference?

  • Can the authors clarify why the performance gains are much smaller under image-disjoint evaluation? It would be helpful to show if the remaining gains come from input features that also apply to other datasets, or strategy preferences specific to the current dataset.

10 comments
AuthorsOpenReview ↗

Response to Critical Weakness 1 & Question 1

Response to Reviewer aevd

We sincerely thank Reviewer aevd (and all reviewers) for your time, thoughtful evaluation, and constructive comments. We are encouraged that you find our observation of sample-level complementarity across pruning methods a clear and reasonable motivation for sample-adaptive strategy selection, and that you appreciate our multi-split, multi-seed image-disjoint evaluations demonstrating performance beyond the primary evaluation setting. Below, we address each of your Critical Weaknesses and Questions point by point.

1. Unfiltered Evaluation and the Role of Full Outside Group A (Critical Weakness 1 & Question 1)

Response:

We fully agree with the distinction you draw: showing that sample-level complementarity exists among candidate pruning strategies does not automatically guarantee that a learned router can predict and exploit it. To directly test VIP-Router on the complete, unfiltered data distribution, we retrained and evaluated a four-strategy VIP-Router (VIP-4) without Full fallback on the unfiltered VTC-Bench dataset. This model retains the exact same router architecture and the four pruning methods (FastV, VisionZip, PruMerge+, and DART), restricting the decision space strictly to the four compressed pruning strategies.

1. Empirical results on the complete, unfiltered VTC-Bench test set. Across the complete unfiltered held-out test set (5,681 test records across all five reduction ratios), VIP-4 achieves a macro-average accuracy of 64.92 ± 0.28%, significantly outperforming the Best Fixed strategy selected on the unfiltered source training split (60.95%) by +3.97 percentage points, with positive gains across all 5/5 training seeds. Relative to the four-strategy Oracle (74.92%), VIP-4 recovers 28.42% of the available Oracle headroom. These results demonstrate that even without Group A filtering, predictable and exploitable sample-level complementarity persists, and our trained router successfully translates it into accuracy gains.

2. Theoretical behavior of Full fallback on unfiltered data under U=A−CU = A - C. From the perspective of our utility definition, Full fallback behaves very differently on unfiltered data compared to Group A. Recall Equation (1): Ui(s,r)=ais−c(s,r),U_i(s, r) = a_i^s - c(s, r), where ais∈{0,1}a_i^s \in \{0, 1\} indicates whether strategy ss answers sample ii correctly, Full has a normalized cost of 11, and each compressed pruning strategy has a nominal cost of rr.

  • In Group A, Full is always correct by construction, so its utility is identically Ui(Full,r)=1−1=0U_i(\mathrm{Full}, r) = 1 - 1 = 0.
  • On unfiltered data, however, Full can also fail (aiFull=0a_i^{\mathrm{Full}} = 0). Whenever Full answers incorrectly, its utility is: Ui(Full,r)=0−1=−1,U_i(\mathrm{Full}, r) = 0 - 1 = -1, whereas a compressed pruning strategy that also answers incorrectly incurs a utility of only: Ui(s,r)=0−r=−r,r∈[0.01,0.25].U_i(s, r) = 0 - r = -r, \qquad r \in [0.01, 0.25]. Thus, whenever Full fails, its utility (−1-1) is strictly lower than that of any failed pruning strategy (−r-r). Even when Full is correct, it must avoid a pruning failure to offset its extra token cost.

In terms of expected utility, Full is preferred over a pruning strategy ss only when: P(aiFull=1)−P(ais=1)>1−r.P(a_i^{\mathrm{Full}} = 1) - P(a_i^s = 1) > 1 - r. This imposes a very high selection threshold on Full: for instance, at r=0.25r = 0.25, Full must have an expected accuracy advantage exceeding 75 percentage points; at r=0.01r = 0.01, this threshold rises to 99 percentage points. Consequently, on unfiltered distributions, our cost-aware objective naturally sets a very high bar for selecting Full, and simply adding Full as an option does not automatically inflate overall utility.

3. Why VIP-4 (excluding Full) provides an even stricter validation. By completely removing Full in VIP-4, we eliminate any contribution from Full fallback altogether. Because all four compressed strategies share the exact same nominal token cost rr at each specified retention ratio, VIP-4's improvement over Best Fixed comes 100% from sample-adaptive selection among the pruning criteria rather than using extra tokens. Even when stripped of Full fallback and evaluated across all unfiltered samples—including those where pruning is harmless and those where even Full fails—VIP-Router still achieves +3.97 percentage points higher accuracy than Best Fixed and recovers 28.42% of the Oracle headroom. This directly supports our core claim: sample-level heterogeneity across visual pruning criteria is both real and exploitable by a lightweight router on unfiltered data.

AuthorsOpenReview ↗

Response to Critical Weakness 2 & Question 4 part 1

2. Why Gains Narrow under Image-Disjoint Evaluation and What Signal Drives Remaining Gains (Critical Weakness 2 & Question 4)

Response:

We address your two questions in turn: (A) why the gains are smaller under the strict image-disjoint protocol, and (B) whether the learned routing gains stem from dataset-specific benchmark preferences or generalizable input features that transfer to other datasets.

Part A: Why Performance Gains Are Smaller under Strict Image-Disjoint Splits

Combining Table 2 and our diagnostic controls in Appendix A.5.1, we identify three key factors explaining the smaller gains under strict image-disjoint evaluation:

1. The drop occurs in the router's headroom recovery rate, not in the underlying Oracle complementarity. As shown in Table 2, the macro-average Oracle headroom (UOracle−UBestFixedU_{\mathrm{Oracle}} - U_{\mathrm{BestFixed}}) is 26.32 utility points under the primary record-disjoint split and 25.49 utility points averaged across the three strict image-disjoint splits (differing by only ~3.2%), while Best Fixed utility also remains stable (30.88 vs. 30.44). In contrast, VIP-Router's realized utility gain over Best Fixed narrows from +6.80 points to +1.44 points (OHR from 26.5% to 6.2%). Thus, substantial sample-level complementarity remains available under strict image-disjoint splits, but the retrained router recovers a smaller fraction of that headroom.

2. Unseen test images alone do not inevitably cause a drop in routing gain (Appendix A.5.1, Table 12). In Appendix A.5.1 (Table 12, page 24), we directly tested whether encountering unseen images at test time causes routing performance to collapse. Keeping the original record-disjoint trained routers and fixed baselines completely unchanged, we split the primary test set into Seen images and Unseen images based on exact RGB pixel hashes:

  • Seen images (3,565 test rows): VIP-Router improves utility over Best Fixed by +6.83 points (26.66% OHR, 5/5 seeds positive).
  • Unseen images (1,475 test rows from 461 unique unseen images): VIP-Router improves utility over Best Fixed by +6.82 points (27.08% OHR, 5/5 seeds positive), and exceeds Best Fixed@Ratio by +6.03 points.

This confirms that unseen test images by themselves do not cause a sharp loss in routing gains: under the same training support, the router achieves virtually identical absolute utility gains (+6.82 vs. +6.83) and OHR (27.08% vs. 26.66%) on unseen test images as on seen test images.

3. Strict image-disjoint re-partitioning simultaneously alters training support and benchmark distributions. Under the strict image-disjoint protocol, all records sharing an underlying image must be grouped into the same partition. Because several constituent benchmarks (e.g., GQA, POPE) pair many questions with shared images, image-level grouping alters both the diversity of unique training images—reducing the training support from 3,566 unique images in the primary split to 3,053–3,072 unique images in the strict splits (Table 9)—and the training/validation/test benchmark compositions. Thus, the difference in Table 2 reflects the combined effect of reduced training-image coverage, altered train/test distributions, and unseen-image generalization, rather than image memorization alone.

AuthorsOpenReview ↗

Response to Critical Weakness 2 & Question 4 part 2

Part B: Benchmark-Level Preferences and Sample-Level Decision Analysis under Strict Image-Disjoint Evaluation

We further examined to what extent VIP-Router's remaining gains under the strict image-disjoint setting can be explained by dataset-specific benchmark-level strategy preferences, and whether the router genuinely executes sample-level choices distinct from a fixed benchmark lookup table.

First, in terms of aggregate Utility, benchmark-level preferences indeed have strong explanatory power under strict image-disjoint splits. Across the three strict image-disjoint splits, we compared VIP-Router against Best Fixed@Benchmark, which selects a single fixed pruning strategy for each benchmark based on the corresponding source training split. The resulting Utility differences (VIP−BF@Benchmark\mathrm{VIP} - \mathrm{BF@Benchmark}) are −0.00038-0.00038, −0.00299-0.00299, and +0.00696+0.00696 across the three splits; when compared against the finer-grained Best Fixed@Benchmark,Ratio, the router likewise does not consistently achieve an additional net gain.

However, sample-wise behavioral analysis reveals another side that aggregate Utility masks: the router's decisions differ substantially from those of the benchmark-level fixed strategy, and these differences directly change sample-level correctness:

Strict SplitStrategy DisagreementRouter RescueRouter DamageVIP−BF@Benchmark ΔU\mathrm{VIP} - \mathrm{BF@Benchmark}\ \Delta U
82661.50%8.39%8.34%−0.00038
36058.79%8.04%8.27%−0.00299
26660.03%7.22%6.47%+0.00696

Here, Router Rescue denotes test samples where VIP-Router answers correctly while Best Fixed@Benchmark answers incorrectly, and Router Damage denotes samples where VIP-Router answers incorrectly while the fixed baseline is correct. Both metrics count only cases where the router selects a compressed pruning strategy, normalized by the total test samples at each ratio and macro-averaged equally across the five reduction ratios.

Across all three splits, the strategy disagreement rate between VIP-Router and Best Fixed@Benchmark is close to 60%. Meanwhile, the router selects Full on only 2.29%, 3.05%, and 1.29% of samples, respectively, indicating that these routing differences occur predominantly among the compressed pruning criteria rather than being driven by Full fallback.

More importantly, on approximately 7%–8% of test samples, VIP-Router corrects the failures of the benchmark-level fixed strategy by selecting a different pruning criterion, demonstrating that its dynamic routing produces tangible sample-level benefits. Under strict unseen-image evaluation, however, the router also causes approximately 6%–8% of samples answered correctly by the fixed baseline to become incorrect. Because the rates of rescue and damage are comparable in magnitude, the net additional gain across the three splits is small and not yet stable.

This analysis leads to three clear conclusions:

  • Benchmark-level preferences explain a substantial share of the aggregate Utility gain under strict image-disjoint evaluation.
  • VIP-Router's behavior is far from a static benchmark-level lookup table: it executes distinct sample-level strategy selections on ~60% of test inputs and rescues 7%–8% of fixed-strategy errors.
  • The current limitation of the router on strictly unseen images is that its valid sample-level rescues do not yet consistently exceed the losses from mistaken switches, so its per-sample error recovery does not yet translate into a stable net advantage over stronger coarse-grained fixed baselines.

Combined with our primary Group A results (+3.81 utility over Best Fixed@Benchmark,Ratio), our unfiltered evaluation (+2.94 pp accuracy over Best Fixed@Benchmark,Ratio), and our zero-shot cross-dataset transfer results in Section 4.4 (Table 3b, where VIP-Router gains +5.43, +5.14, and +2.07 utility points on completely unseen external datasets with zero image overlap), the evidence supports that input-dependent pruning strategy selection is learnable and partially transferable across datasets, while our strict image-disjoint behavioral analysis pinpoints its current boundary: the router can already identify a subset of samples where switching pruning criteria helps, but cannot yet avoid erroneous switches reliably enough on strictly unseen images.

Reviewer_S5HBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VIP-Router, a lightweight learned router that picks a visual token pruning strategy for each input instead of applying one fixed strategy to all inputs. The router takes frozen CLIP visual and text features plus a text-to-vision cross-attention module, and uses a ratio-conditioned MLP to predict a cost-aware utility for each candidate: FastV, VisionZip, PruMerge+, DART, and full-token inference as a fallback. The highest-utility candidate is then executed. The router is trained on the per-sample correctness labels released by VTC-Bench for Qwen2-VL-7B and is evaluated on VTC-Bench Group A, a pruning-sensitive subset, at 75%–99% reduction ratios. Additional results cover image-disjoint splits, four other MLLM backbones, and transfer to four unseen benchmarks.

Critical strengths
  • The motivation is interesting and well supported. The paper shows that the best strategy on average is not the best strategy for each sample, and that a large fraction of failures of the best fixed strategy can be recovered by another strategy.
  • The method is plug-and-play and lightweight. It doesn't modify the pruning methods or the MLLM and adds only a small number of trainable parameters.
  • The experimental analysis is fairly thorough, with multiple seeds, image-disjoint splits, a ratio-conditioned fixed baseline, and a bound on the contribution of the full-token fallback. The paper is clearly written and the limitations are honestly discussed.
Critical weaknesses
  • The evaluation is limited to a filtered subset. All main results are on VTC-Bench Group A, which only keeps samples where full-token inference is correct and image downsampling is wrong. The full-token inference is 100% correct on Group A by construction, which makes falling back to Full look better than it would be in practice. The paper acknowledges this in the limitations, but I think results on the full benchmark sets, including samples where pruning is harmless and samples where even Full is wrong, are necessary to support the claims.

  • The gains under the image-disjoint protocol are much smaller. The headline numbers in the abstract come from the record-disjoint split, where the same image can appear in both training and test with different questions. Since the router conditions directly on image features, this setting is likely optimistic. Under the image-disjoint splits, the utility gain over Best Fixed and the recovered oracle headroom both drop significantly. Some individual runs are barely above Best Fixed. This substantially weakens the claim that pruning-strategy suitability can be reliably predicted at the sample level.

  • The comparison with Best Fixed is not under the same token budget. Best Fixed always retains exactly a fraction r of the visual tokens, whereas VIP-Router sometimes selects Full and therefore uses more tokens. At 99% reduction, for example, VIP-Router retains 7.6% of the tokens on average versus 1% for Best Fixed. A fairer comparison would run the fixed strategies at the same realized token budget as VIP-Router at each ratio.

  • The method is hard to generalize and deploy. The router needs to be trained on correctness labels for the training data on a specific backbone, so it has to be retrained for every new backbone and every new pruning method. The paper's limitations section confirms that cross-backbone transfer fails. Deployment is also difficult in practice: a serving system would need to integrate multiple pruning implementations that operate at different places (before the LLM vs. inside the decoder), plus an extra CLIP encoder, and handle batches where different samples use different strategies.

Questions for authors
  1. Can you report results on the full VTC-Bench benchmarks (and the transfer benchmarks) without Group A filtering?
  2. Can you compare against the fixed strategies at the same realized token budget as VIP-Router at each reduction ratio, in terms of both accuracy and utility?
  3. How does VIP-Router compare with a simple baseline that selects the best strategy per benchmark or per domain? Is the router mostly learning task identity rather than sample-level preferences?
  4. What is the cost of building the routing data and retraining the router for a new backbone or a new pruning method? Have you measured wall-clock latency and batched serving throughput with multiple pruning strategies in the same deployment?
9 comments
AuthorsOpenReview ↗

Response to Critical Weakness 1 & Question 1 part 1

Response to Reviewer S5HB

We sincerely thank Reviewer S5HB (and all reviewers) for your time, careful assessment, and constructive suggestions. We are very encouraged by your positive feedback—specifically recognizing that our motivation regarding sample-level complementarity is interesting and well supported, that VIP-Router is lightweight and plug-and-play without modifying either the pruning methods or the MLLM backbone, that our experimental analysis is thorough (spanning multiple seeds, image-disjoint splits, a ratio-conditioned fixed baseline, and a theoretical bound on the full-token fallback), and that the manuscript is clearly written with an honest discussion of limitations. Below, we address each of your Critical Weaknesses and Questions point by point.

1. Unfiltered Evaluation on Source and Transfer Benchmarks (Critical Weakness 1 & Question 1)

Response:

Thank you for raising this important question regarding unfiltered evaluation.

1. Motivation and behavior of the Full fallback. First, we would like to clarify that the original purpose of introducing Full was to examine a question one step beyond selecting among pruning strategies: can a router recognize when a sample is genuinely unsuitable for pruning and retain all visual tokens? This design does not encourage frequent fallback. Under our utility definition U=A−CU = A - C, Full incurs a cost of 11, whereas each pruning strategy incurs a cost of rr; consequently, selecting Full benefits utility only when the expected accuracy gain is large enough to offset the cost penalty (1−r)(1 - r). As reported in Appendix A.5.6 of our paper, across the five reduction ratios, VIP-Router selects Full on only 4.92%, 3.72%, 3.51%, 3.48%, and 6.66% of samples (peaking at just 6.66%), confirming that Full serves strictly as an occasional fallback for a small minority of inputs rather than a routine inference path. Therefore, we respectfully disagree with interpreting the inclusion of Full as a mechanism designed to exploit the Group A construction for evaluation gains.

At the same time, we fully agree that outside Group A, Full is not guaranteed to be correct. To directly test our core claim—whether a router can learn which pruning criterion best suits each input on the complete, unfiltered data distribution—we completely remove Full and retrain a four-output router (VIP-4) over the four off-the-shelf pruning strategies (FastV, VisionZip, PruMerge+, and DART) following the exact same training pipeline.

2. Unfiltered evaluation on the full VTC-Bench source benchmarks. We evaluate VIP-4 on the complete, unfiltered held-out test set of VTC-Bench (retaining all samples, including those where pruning is harmless and those where even Full fails). Across 5,681 held-out test records and all five reduction ratios, the four-strategy VIP-Router achieves a macro-average accuracy of 64.92 ± 0.28%, outperforming the Best Fixed strategy selected on the unfiltered source training split (60.95%) by +3.97 percentage points, with positive gains across all 5/5 training seeds. Relative to the four-strategy Oracle (74.92%), VIP-Router recovers 28.42% of the available accuracy headroom.

To understand the magnitude of the overall gain on unfiltered data, we further decompose the correctness distribution across the four pruning strategies. Averaged equally across the five reduction ratios, the proportions of test samples where the four strategies are all correct, all wrong, and inconsistent (mixed) are 42.96%, 25.08%, and 31.95%, respectively. This distinction is critical:

  • On the first two categories—accounting for 68.04% of all unfiltered samples—all four pruning strategies yield the exact same binary outcome. Thus, regardless of which strategy a router selects, it cannot obtain any accuracy gain over a fixed strategy; the large prevalence of these strategy-invariant samples naturally dilutes the average margin across the full test set.
  • Actionable routing headroom exists only on the remaining 31.95% mixed subset, where different pruning criteria produce distinct correctness outcomes. On this mixed subset, VIP-Router achieves an accuracy gain of +11.99 percentage points over Best Fixed, demonstrating that the router extracts a strong decision signal from cross-strategy complementarity.

Thus, the +3.97 percentage point overall gain on the unfiltered test set should be understood in the context that nearly 70% of unfiltered samples offer zero room for strategy differentiation. Crucially, this result comes from a four-strategy router retrained directly on unfiltered data without Full fallback, proving that VIP-Router's effectiveness does not rely on Full being 100% correct in Group A, and that it reliably exploits strategy complementarity on the complete, unfiltered benchmark distribution.

AuthorsOpenReview ↗

Response to Critical Weakness 1 & Question 1 part 2

3. Zero-shot transfer on unfiltered target benchmarks. We also evaluate zero-shot transfer by directly applying the unfiltered-trained four-strategy router (VIP-4) to the four unseen target benchmarks without any target-domain training, calibration, or checkpoint selection. Best Fixed remains VisionZip (selected solely on the source training set). Macro-averaged across the five reduction ratios within each benchmark, the results are as follows (accuracy in %, gains in percentage points, ±\pm standard deviation across 5 training seeds):

Unfiltered Target BenchmarkTest RecordsBest Fixed Acc. (%)VIP-4 Acc. (%)VIP-4 − BF (pp)Positive Seeds
ScienceQA-IMG2,01774.2675.07 ± 0.09+0.825/5
MMMU86146.3446.31 ± 0.30−0.031/5
RealWorldQA76553.3951.72 ± 0.70−1.670/5
SEED-Bench Image14,23360.4660.13 ± 0.10−0.330/5

Since Full is excluded, both methods have an identical nominal macro token cost of 0.09472. Under this unfiltered zero-shot transfer setting, VIP-4 achieves consistent positive gains across all 5 seeds on ScienceQA-IMG, performs on par with Best Fixed on MMMU, and does not maintain an advantage on SEED-Bench Image (−0.33-0.33 pp) and RealWorldQA (−1.67-1.67 pp).

We believe it is important to distinguish between (i) the validity of sample-adaptive pruning routing and (ii) zero-shot out-of-distribution generalization across benchmarks:

  1. Our core contribution is showing that visual token pruning strategies exhibit predictable sample-level complementarity that a lightweight router can exploit to outperform fixed strategies. This claim is supported across multiple settings: (i) in-domain and cross-benchmark Group A evaluations (Table 1, Table 3b), (ii) strict image-disjoint splits (Table 2), and (iii) the full unfiltered source evaluation above (+3.97 pp across 5/5 seeds without Full). In contrast, unfiltered cross-benchmark zero-shot transfer tests a strictly stronger condition: whether a policy trained solely on unfiltered source data (where 68% of samples carry tied labels across strategies) can transfer its decision boundary directly to unseen task distributions without any target adaptation.
  2. The mixed zero-shot transfer results do not diminish the value of sample-adaptive strategy selection. Even under domain shift, the router succeeds on ScienceQA-IMG (5/5 seeds) and matches Best Fixed on MMMU. Where degradation occurs, it reflects shifts in question types, visual granularity demands, and the relative prior rankings of pruning operators across domains—highlighting that cross-domain stability of learned preference mappings is an open direction, while sample-level complementarity itself remains a clear and actionable source of improvement.
AuthorsOpenReview ↗

Response to Critical Weakness 2

2. Performance under Image-Disjoint Splits vs. Image Memorization (Critical Weakness 2)

Response:

We acknowledge that the utility gains under the strict image-disjoint splits are smaller in magnitude. However, regarding the key underlying concern—whether the strong gains under the primary record-disjoint protocol mainly stem from the router memorizing images seen during training—we actually provided a direct diagnostic experiment in Appendix A.5.1 (Table 12, page 24) of our paper.

1. Direct evaluation on Seen vs. Unseen images within the primary test set (Table 12). Specifically, we partitioned the original record-disjoint test set into Seen images and Unseen images based on whether each test image's decoded RGB pixel hash ever appeared in the router's training set. Crucially, we kept the original trained routers, train/validation splits, and fixed-strategy baselines completely unchanged (without any retraining, model reselection, or baseline reselection):

Test Subset (Primary Protocol)Test RowsBest Fixed UtilityVIP-Router UtilityΔU\Delta UOHRPositive Seeds
Seen images3,56535.5342.36 ± 0.76+6.8326.66%5/5
Unseen images1,47519.7226.54 ± 0.52+6.8227.08%5/5

Notably, the Unseen images subset comprises 461 unique images that never appeared in the training set (480 image–question records, 1,475 record–ratio test decisions). Even when training-seen images are completely excluded at test time, the original router achieves a gain of +6.82 utility points (+34.6% relative improvement over Best Fixed) and recovers 27.08% of the Oracle headroom. This is virtually identical to its performance on the Seen images subset (+6.83 utility points, 26.66% OHR). Furthermore, on the Unseen images subset, VIP-Router outperforms the training-selected Best Fixed@Ratio baseline by +6.03 utility points, with positive gains across all 5/5 training seeds.

This directly demonstrates that the large gains under the primary record-disjoint protocol are not confined to images seen during training; the router predicts and exploits sample-level pruning complementarity even on completely unseen visual content.

2. Why strict image-disjoint splits differ in magnitude. We also note in Appendix A.5.1 that the Seen and Unseen subsets in Table 12 have different benchmark compositions, so their comparison is descriptive rather than an isolated causal estimate of image overlap. Conversely, constructing the strict image-disjoint splits (Splits I, II, and III) does not merely remove image overlap between train and test—it simultaneously alters the training support (reducing unique training images and grouping heavily multi-question benchmarks such as GQA and POPE by image clusters) as well as the validation and test compositions. Thus, the difference in magnitude between the two protocols cannot be attributed solely to removing image memorization.

Even under this stricter re-partitioning, VIP-Router achieves mean utilities of 32.43, 30.62, and 32.60 across the three independent image-disjoint splits, consistently outperforming Best Fixed (30.70, 29.54, and 31.09) in all 15/15 independent training runs (3 splits ×\times 5 seeds).

In summary, the empirical evidence supports two conclusions:

  1. Image overlap is not a prerequisite for substantial sample-level routing gains, as directly demonstrated by the +6.82 utility gain (27.08% OHR) on the unseen-image test subset in Table 12 (as well as zero-shot transfer in Table 3b, where 0% of target images overlap with source data).
  2. Strict image-disjoint re-partitioning reveals that room remains to further improve out-of-distribution visual generalization, as the proportion of recovered headroom decreases when training support and split structures are altered—an open challenge we explicitly discuss in our Limitations section rather than a refutation of sample-level predictability.

Comments on the paper

13 comments
~Yusen_Peng1OpenReview ↗

Interesting work; have a couple of questions

Dear authors,

This is indeed very interesting work! I have a couple questions and would appreciate it if any of the authors could respond.

  1. The paper "QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression", as a pioneering work, applies a similar "MoE-like" idea to route into different compression strategies. Is it possible to do proper fair comparison with QMoP too?

  2. Both encoder-stage compression (visionzip, prumerge) and LLM-stage compression (FastV) are considered. But I would imagine that LLM-stage compression would be less effcient than encoder-stage ones (in terms of FLOPs, prefill latency, etc.) What's the benefit of considering FastV here?

Thanks for your work!

AuthorsOpenReview ↗

Response to Critical Weakness 1 part 1

Response to Reviewer xy2M

We sincerely thank Reviewer xy2M (and all reviewers) for your time, insightful comments, and constructive feedback. We are encouraged that you find our evidence of sample-level complementarity among existing pruning strategies and our direction of exploiting it beyond a uniform fixed criterion sensible and well-motivated, and that you appreciate the breadth of our empirical evaluations across multiple pruning ratios, MLLM backbones, unseen benchmarks, image-disjoint splits, and cost-aware comparisons. Below, we address each of your Critical Weaknesses and Questions point by point.

1. Methodological Distinction from MetaKV and QMoP (Critical Weakness 1)

Response:

We appreciate the opportunity to clarify the scope and nature of our contribution. Scientifically, while VIP-Router, MetaKV, and QMoP all involve input-conditioned adaptive selection, they differ substantially in the specific problem formulated, the objects operated upon, and their technical paradigms.

(1) Distinction from MetaKV: Selecting among heterogeneous visual pruning criteria vs. KV-cache compression configurations. MetaKV studies KV-cache compression in long-context LLM inference. It selects among configurations of KVQuant, H2O\text{H}_2\text{O}, and RocketKV by predicting accuracy, latency, and peak memory footprint to satisfy user-specified latency and memory constraints. Its goal is resource scheduling across KV-cache configurations under varying hardware budgets.

In contrast, VIP-Router investigates sample-level suitability differences across distinct visual token pruning criteria in MLLMs. Beyond compression budgets, we address a fundamental question that lacked systematic investigation: even at the exact same visual token retention ratio, different pruning criteria can yield markedly different accuracy outcomes depending on the input image and question; can this sample-level heterogeneity be predicted in advance to improve inference quality? Thus, rather than scheduling KV-cache budgets, VIP-Router exploits differences in which visual information is preserved by distinct pruning criteria at a specified pruning level. Through systematic comparisons against Best Fixed, the per-sample Oracle, and Oracle Headroom Recovery (OHR) across multiple reduction ratios, we quantify and validate the value of this complementarity—a research question that cannot be answered by KV-cache configuration selection in MetaKV.

(2) Distinction from QMoP: Discrete selection over off-the-shelf pruning operators vs. joint modeling and fusion of trainable compression branches. QMoP designs three custom compression branches (pooling, resampling, and pruning) within the visual projector and uses a Query-Guided Router to dynamically weight and fuse the visual representations produced by these branches. It relies on a two-stage training pipeline involving projector pretraining followed by joint fine-tuning of the projector and the LLM. Thus, QMoP focuses on learning a new multi-branch compression representation via joint backbone training.

VIP-Router follows a fundamentally different design principle: we do not redesign pruning operators, do not jointly train multiple compression branches, and do not modify or fine-tune the underlying MLLM; instead, we operate directly over off-the-shelf pruning methods independently proposed in the literature. Specifically, FastV, VisionZip, PruMerge+, and DART retain their original mechanisms, while VIP-Router predicts the most suitable strategy for each input sample and target reduction ratio, executing a single discrete choice rather than fusing representations across branches. This design offers two key advantages:

  • Methodological decoupling: It decouples the design of visual token pruning operators from sample-level strategy selection. New pruning methods can be developed independently, and VIP-Router can harness their complementarity in a plug-and-play manner without retraining the MLLM, building directly upon existing progress in visual token pruning.
  • Scientific insight: Because candidate pruning operators remain unmodified, we can directly compare their sample-wise behavior at identical compression ratios and quantify the headroom between fixed criteria and the sample-level Oracle—providing clear empirical evidence of predictable, exploitable sample-level heterogeneity across independent pruning criteria.

Therefore, whereas QMoP learns a multi-branch visual projector via joint MLLM fine-tuning, VIP-Router unlocks the complementarity among unmodified, off-the-shelf pruning operators through lightweight input-conditioned routing. Sharing the high-level concept of routing does not mean the two works follow the same technical route or solve the same problem.

AuthorsOpenReview ↗

Response to Critical Weakness 1 part 2

Summary of novelty and contemporaneous context.

In short, we do not claim to invent MLP routing, cross-attention, or utility regression as standalone neural building blocks. Rather, methodological innovation should not be judged solely by whether individual layer primitives are newly invented. Our core contribution lies in identifying and quantifying predictable sample-level heterogeneity across visual token pruning criteria, formulating a unified ratio-conditioned and cost-aware utility routing framework (which jointly decides which criterion to apply across ratios rr and whether to prune at all via Full), and achieving substantial accuracy–efficiency gains without modifying existing pruning operators or MLLM weights.

Finally, we note that under the ICLR 2027 Reviewer Guidelines, authors are not required to compare against contemporaneous work or non-peer-reviewed arXiv preprints, and the absence of such comparisons cannot serve as grounds for rejection. MetaKV was posted as an arXiv preprint on September 7, 2026 (contemporaneous with this ICLR submission), and QMoP was posted on March 22, 2026 (for which we also only found an arXiv preprint without a verifiable peer-reviewed venue record). Nevertheless, we proactively discussed both works in Section 2.2 and have detailed our substantive differences above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.