- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method named VIP-Router, which selects a visual token pruning strategy for each input. The proposed method uses a lightweight router to predict the utility of existing pruning methods from visual and textual features. The router can also select full-token inference as a fallback. Experiments are conducted on VTC-Bench to evaluate the performance of VIP-Router compared to baselines. The results show improved accuracy and utility, with additional evaluations using image-disjoint splits and different backbones.
Critical strengths
-
The paper observes that different pruning methods work well on different samples. This provides a clear and reasonable motivation for selecting a strategy for each input.
-
The paper provides image-disjoint evaluation with three splits and five seeds. These experiments help show the performance of the method beyond the main evaluation setting.
Critical weaknesses
-
The main evaluation only uses samples that the full-token model answers correctly. Therefore, selecting Full always gives a correct answer in VTC-Bench Group A. However, the router is not evaluated on the unfiltered benchmark, where full-token inference can also fail. The appendix shows that different pruning strategies still work well on different samples in the unfiltered data. However, it does not show if the trained router can select the right strategy for these samples.
-
The performance gains are much smaller under image-disjoint evaluation. The average utility improves by 22.0% under record-disjoint evaluation, but only 4.7% under image-disjoint evaluation. These two settings use different test sets. Therefore, the difference cannot be explained only by image overlap.
-
The paper does not show if the performance gains will hold under the same computation budget. At 99% reduction, VIP-Router retains 7.6% of visual tokens on average, compared to 1% for the fixed baseline, because it can select Full. The proposed utility includes a penalty for using more tokens. However, the fraction of retained tokens does not directly measure inference latency or FLOPs.
-
QMoP already uses query-guided routing to select among different visual compression branches. VIP-Router selects among existing pruning methods, but the idea of routing is similar. Additionally, Figure 1 shows that different pruning strategies work best on different benchmarks. However, the paper does not compare with a simpler router that selects strategies based on task category and retention ratio.
-
The cost of training the router is not fully clear. Training requires offline correctness labels for several pruning methods at five retention ratios. The experiments with different backbones also train a separate router for each backbone. Without these results, the practical advantage over a strong fixed strategy is not clear, especially given the smaller gains under image-disjoint evaluation.
Questions for authors
-
Can the authors provide results for the complete VIP-Router on the unfiltered VTC-Bench test set? Full can also give incorrect answers outside Group A.
-
Can the authors show if VIP-Router still outperforms the strongest fixed pruning strategy under the same actual inference FLOPs or latency? Comparing the same nominal pruning ratios does not make this clear.
-
Can the authors report the offline computation needed to obtain correctness labels for each strategy and train the router for a new backbone? How does this cost compare to the computation saved during inference?
-
Can the authors clarify why the performance gains are much smaller under image-disjoint evaluation? It would be helpful to show if the remaining gains come from input features that also apply to other datasets, or strategy preferences specific to the current dataset.
10 comments
Response to Critical Weakness 2 & Question 4 part 1
2. Why Gains Narrow under Image-Disjoint Evaluation and What Signal Drives Remaining Gains (Critical Weakness 2 & Question 4)
Response:
We address your two questions in turn: (A) why the gains are smaller under the strict image-disjoint protocol, and (B) whether the learned routing gains stem from dataset-specific benchmark preferences or generalizable input features that transfer to other datasets.
Part A: Why Performance Gains Are Smaller under Strict Image-Disjoint Splits
Combining Table 2 and our diagnostic controls in Appendix A.5.1, we identify three key factors explaining the smaller gains under strict image-disjoint evaluation:
1. The drop occurs in the router's headroom recovery rate, not in the underlying Oracle complementarity.
As shown in Table 2, the macro-average Oracle headroom () is 26.32 utility points under the primary record-disjoint split and 25.49 utility points averaged across the three strict image-disjoint splits (differing by only ~3.2%), while Best Fixed utility also remains stable (30.88 vs. 30.44). In contrast, VIP-Router's realized utility gain over Best Fixed narrows from +6.80 points to +1.44 points (OHR from 26.5% to 6.2%). Thus, substantial sample-level complementarity remains available under strict image-disjoint splits, but the retrained router recovers a smaller fraction of that headroom.
2. Unseen test images alone do not inevitably cause a drop in routing gain (Appendix A.5.1, Table 12). In Appendix A.5.1 (Table 12, page 24), we directly tested whether encountering unseen images at test time causes routing performance to collapse. Keeping the original record-disjoint trained routers and fixed baselines completely unchanged, we split the primary test set into Seen images and Unseen images based on exact RGB pixel hashes:
- Seen images (3,565 test rows): VIP-Router improves utility over
Best Fixedby +6.83 points (26.66% OHR, 5/5 seeds positive). - Unseen images (1,475 test rows from 461 unique unseen images): VIP-Router improves utility over
Best Fixedby +6.82 points (27.08% OHR, 5/5 seeds positive), and exceedsBest Fixed@Ratioby +6.03 points.
This confirms that unseen test images by themselves do not cause a sharp loss in routing gains: under the same training support, the router achieves virtually identical absolute utility gains (+6.82 vs. +6.83) and OHR (27.08% vs. 26.66%) on unseen test images as on seen test images.
3. Strict image-disjoint re-partitioning simultaneously alters training support and benchmark distributions. Under the strict image-disjoint protocol, all records sharing an underlying image must be grouped into the same partition. Because several constituent benchmarks (e.g., GQA, POPE) pair many questions with shared images, image-level grouping alters both the diversity of unique training images—reducing the training support from 3,566 unique images in the primary split to 3,053–3,072 unique images in the strict splits (Table 9)—and the training/validation/test benchmark compositions. Thus, the difference in Table 2 reflects the combined effect of reduced training-image coverage, altered train/test distributions, and unseen-image generalization, rather than image memorization alone.
Response to Critical Weakness 2 & Question 4 part 2
Part B: Benchmark-Level Preferences and Sample-Level Decision Analysis under Strict Image-Disjoint Evaluation
We further examined to what extent VIP-Router's remaining gains under the strict image-disjoint setting can be explained by dataset-specific benchmark-level strategy preferences, and whether the router genuinely executes sample-level choices distinct from a fixed benchmark lookup table.
First, in terms of aggregate Utility, benchmark-level preferences indeed have strong explanatory power under strict image-disjoint splits. Across the three strict image-disjoint splits, we compared VIP-Router against Best Fixed@Benchmark, which selects a single fixed pruning strategy for each benchmark based on the corresponding source training split. The resulting Utility differences () are , , and across the three splits; when compared against the finer-grained Best Fixed@Benchmark,Ratio, the router likewise does not consistently achieve an additional net gain.
However, sample-wise behavioral analysis reveals another side that aggregate Utility masks: the router's decisions differ substantially from those of the benchmark-level fixed strategy, and these differences directly change sample-level correctness:
| Strict Split | Strategy Disagreement | Router Rescue | Router Damage | |
|---|---|---|---|---|
| 826 | 61.50% | 8.39% | 8.34% | −0.00038 |
| 360 | 58.79% | 8.04% | 8.27% | −0.00299 |
| 266 | 60.03% | 7.22% | 6.47% | +0.00696 |
Here, Router Rescue denotes test samples where VIP-Router answers correctly while Best Fixed@Benchmark answers incorrectly, and Router Damage denotes samples where VIP-Router answers incorrectly while the fixed baseline is correct. Both metrics count only cases where the router selects a compressed pruning strategy, normalized by the total test samples at each ratio and macro-averaged equally across the five reduction ratios.
Across all three splits, the strategy disagreement rate between VIP-Router and Best Fixed@Benchmark is close to 60%. Meanwhile, the router selects Full on only 2.29%, 3.05%, and 1.29% of samples, respectively, indicating that these routing differences occur predominantly among the compressed pruning criteria rather than being driven by Full fallback.
More importantly, on approximately 7%–8% of test samples, VIP-Router corrects the failures of the benchmark-level fixed strategy by selecting a different pruning criterion, demonstrating that its dynamic routing produces tangible sample-level benefits. Under strict unseen-image evaluation, however, the router also causes approximately 6%–8% of samples answered correctly by the fixed baseline to become incorrect. Because the rates of rescue and damage are comparable in magnitude, the net additional gain across the three splits is small and not yet stable.
This analysis leads to three clear conclusions:
- Benchmark-level preferences explain a substantial share of the aggregate Utility gain under strict image-disjoint evaluation.
- VIP-Router's behavior is far from a static benchmark-level lookup table: it executes distinct sample-level strategy selections on ~60% of test inputs and rescues 7%–8% of fixed-strategy errors.
- The current limitation of the router on strictly unseen images is that its valid sample-level rescues do not yet consistently exceed the losses from mistaken switches, so its per-sample error recovery does not yet translate into a stable net advantage over stronger coarse-grained fixed baselines.
Combined with our primary Group A results (+3.81 utility over Best Fixed@Benchmark,Ratio), our unfiltered evaluation (+2.94 pp accuracy over Best Fixed@Benchmark,Ratio), and our zero-shot cross-dataset transfer results in Section 4.4 (Table 3b, where VIP-Router gains +5.43, +5.14, and +2.07 utility points on completely unseen external datasets with zero image overlap), the evidence supports that input-dependent pruning strategy selection is learnable and partially transferable across datasets, while our strict image-disjoint behavioral analysis pinpoints its current boundary: the router can already identify a subset of samples where switching pruning criteria helps, but cannot yet avoid erroneous switches reliably enough on strictly unseen images.
Response to Critical Weakness 1 & Question 1
Response to Reviewer aevd
We sincerely thank Reviewer aevd (and all reviewers) for your time, thoughtful evaluation, and constructive comments. We are encouraged that you find our observation of sample-level complementarity across pruning methods a clear and reasonable motivation for sample-adaptive strategy selection, and that you appreciate our multi-split, multi-seed image-disjoint evaluations demonstrating performance beyond the primary evaluation setting. Below, we address each of your Critical Weaknesses and Questions point by point.
1. Unfiltered Evaluation and the Role of
FullOutside Group A (Critical Weakness 1 & Question 1)Response:
We fully agree with the distinction you draw: showing that sample-level complementarity exists among candidate pruning strategies does not automatically guarantee that a learned router can predict and exploit it. To directly test VIP-Router on the complete, unfiltered data distribution, we retrained and evaluated a four-strategy VIP-Router (
VIP-4) withoutFullfallback on the unfiltered VTC-Bench dataset. This model retains the exact same router architecture and the four pruning methods (FastV,VisionZip,PruMerge+, andDART), restricting the decision space strictly to the four compressed pruning strategies.1. Empirical results on the complete, unfiltered VTC-Bench test set. Across the complete unfiltered held-out test set (5,681 test records across all five reduction ratios),
VIP-4achieves a macro-average accuracy of 64.92 ± 0.28%, significantly outperforming theBest Fixedstrategy selected on the unfiltered source training split (60.95%) by +3.97 percentage points, with positive gains across all 5/5 training seeds. Relative to the four-strategy Oracle (74.92%),VIP-4recovers 28.42% of the available Oracle headroom. These results demonstrate that even without Group A filtering, predictable and exploitable sample-level complementarity persists, and our trained router successfully translates it into accuracy gains.2. Theoretical behavior of
Fullfallback on unfiltered data under U=A−C. From the perspective of our utility definition,Fullfallback behaves very differently on unfiltered data compared to Group A. Recall Equation (1): Ui(s,r)=ais−c(s,r), where ais∈{0,1} indicates whether strategy s answers sample i correctly,Fullhas a normalized cost of 1, and each compressed pruning strategy has a nominal cost of r.Fullis always correct by construction, so its utility is identically Ui(Full,r)=1−1=0.Fullcan also fail (aiFull=0). WheneverFullanswers incorrectly, its utility is: Ui(Full,r)=0−1=−1, whereas a compressed pruning strategy that also answers incorrectly incurs a utility of only: Ui(s,r)=0−r=−r,r∈[0.01,0.25]. Thus, wheneverFullfails, its utility (−1) is strictly lower than that of any failed pruning strategy (−r). Even whenFullis correct, it must avoid a pruning failure to offset its extra token cost.In terms of expected utility,
Fullis preferred over a pruning strategy s only when: P(aiFull=1)−P(ais=1)>1−r. This imposes a very high selection threshold onFull: for instance, at r=0.25,Fullmust have an expected accuracy advantage exceeding 75 percentage points; at r=0.01, this threshold rises to 99 percentage points. Consequently, on unfiltered distributions, our cost-aware objective naturally sets a very high bar for selectingFull, and simply addingFullas an option does not automatically inflate overall utility.3. Why
VIP-4(excludingFull) provides an even stricter validation. By completely removingFullinVIP-4, we eliminate any contribution fromFullfallback altogether. Because all four compressed strategies share the exact same nominal token cost r at each specified retention ratio,VIP-4's improvement overBest Fixedcomes 100% from sample-adaptive selection among the pruning criteria rather than using extra tokens. Even when stripped ofFullfallback and evaluated across all unfiltered samples—including those where pruning is harmless and those where evenFullfails—VIP-Router still achieves +3.97 percentage points higher accuracy thanBest Fixedand recovers 28.42% of the Oracle headroom. This directly supports our core claim: sample-level heterogeneity across visual pruning criteria is both real and exploitable by a lightweight router on unfiltered data.