- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines how much performance loss in visual token pruning can be recovered by more expensive offline subset selection. It uses auxiliary question-answer pairs generated by a frontier VLM to score candidate subsets through the frozen target model's answer likelihoods. The proposed QA mini-batch greedy algorithm samples questions at each selection step and builds nested, image-specific subsets that are reused across evaluation prompts. Experiments with two model families on 900 images from nine benchmarks demonstrate substantial headroom over existing selectors on held-out questions, alongside transfer to long-form descriptions. The contribution is an empirical characterization of achievable pruning performance under an offline search protocol.
Critical strengths
-
The paper asks a useful diagnostic question that conventional accuracy-versus-budget comparisons do not answer. Producing feasible, unchanged-token subsets with consistently higher held-out performance demonstrates that some pruning loss is recoverable selector error. Reusing one image-specific subset across questions and description prompts makes this result more informative than a single-question oracle. The paper appropriately distinguishes achievable operating points from certified optima and explicitly treats search cost as part of the experimental protocol.
-
The empirical investigation is unusually substantial. It covers two architectures, strong direct subset baselines, paired image-level bootstrap analyses, and several controls for auxiliary supervision. At 20% retention on the small Qwen set, dataset-agnostic generation reaches 92.79% relative performance and overlap-filtered search reaches 91.36%, versus 83.47% for the strongest baseline. The three-seed QA-MBG comparison also consistently outperforms full greedy. Together, these results support the main existence-of-headroom finding.
Critical weaknesses
-
The strongest generalization controls remain limited in scope. Benchmark-agnostic generation and overlap removal are evaluated separately on the small Qwen cohort; the manuscript does not show their combination or analogous controls for Gemma. After overlap removal, the margin over the best baseline at 30% retention is only 0.46 relative-performance points (Table 20), and each removal condition is a single search run. Thus, the large main-experiment gaps are convincing under the stated auxiliary-supervision protocol, but a stronger claim about benchmark-independent, semantically disjoint general understanding is less firmly established. This limits the generality of the conclusion rather than invalidating the demonstrated feasible subsets.
-
The long-form transfer evidence relies on a single model judge whose scores depend on the comparison group. Appendix D uses joint multi-candidate judging, and the full-token reference receives scores ranging from 3.954 to 3.622 across budget groups in Table 15. Appendix G correctly states that its intervals condition on the realized judgments and omit judge-rerun variability. The RealWorldQA example on p.47 also assigns 5/5 overall and groundedness to descriptions calling the wheels identical, despite the benchmark explicitly distinguishing their sizes. Within-group paired gains remain informative, but these observations motivate an independently calibrated check of the long-form conclusions. A small human or second-judge evaluation would materially strengthen this part of the contribution.
Questions for authors
-
Does the headroom persist when dataset-agnostic generation and semantic-overlap filtering are applied jointly, with repeated search seeds? A representative check on Gemma would also clarify whether the current robustness conclusions extend beyond Qwen.
-
How stable are the long-form paired rankings under an independent judge or a small blinded human audit, particularly if the candidate group is held fixed? Please distinguish ranking stability from absolute score calibration.
-
Can the authors clarify how the reported approximately 0.5 H100 GPU-hours per image extrapolates beyond the single 945-token, 16-QA Qwen timing example in Appendix B? The main auxiliary pool averages 33.64 QAs, and Gemma uses a different architecture. Representative runtime statistics would make the released search targets easier to reproduce.
7 comments
Response to Reviewer VPHU (2/3)
Q2 / W2. Long-form stability: independent judges, fixed group, human audit
Second judges.
- We re-scored every original long-form request with two judges from other model families: the open-weight Qwen3.8-27B and Claude Haiku 5.5. That is 21,600 requests (900 images × 2 prompts × 6 budget groups × 2 backbones), with identical prompts, images, and anonymized candidate order.
- To remove group dependence, we also built a fixed-group set of 3,600 requests. For each image and prompt, one request contains the same ten candidates: the full-token answer, plus QA-MBG and the two strongest baselines at 5, 10, and 20%. All three judges scored this set.
Ranking stability. The method ranking is identical under all three judges in both sets, and QA-MBG ranks first at every budget for every judge. The table gives the paired gain over the strongest baseline: the six-budget mean for the original groups, and the 5/10/20% mean for the fixed group.
| GPT-5.6-Luna | Qwen3.8-27B | Haiku 5.5 | |
|---|---|---|---|
| Qwen3-VL-4B vs FLoC, original groups | 0.612 | 0.678 | 0.711 |
| Qwen3-VL-4B vs FLoC, fixed group | 0.458 | 0.552 | 0.620 |
| Gemma 4 E4B vs PruMerge, original groups | 0.450 | 0.456 | 0.509 |
| Gemma 4 E4B vs PruMerge, fixed group | 0.487 | 0.507 | 0.566 |
All 95% CI lower bounds are at least 0.42. The judges also agree closely with each other:
- When two judges both strictly prefer one candidate of a pair, they agree on 90.6–95.2% of pairs, and on 88.3–94.0% for QA-MBG vs the strongest baseline.
- Kendall's τ between judges over condition means is 0.89–0.95 for the original groups and 1.00 for the fixed group.
- Item-level Pearson r is 0.78–0.84.
Calibration. Absolute score levels differ across judges; Qwen3.8-27B is the most lenient. The offsets themselves change with the group (Haiku 5.5 is 0.25–0.29 below Luna in the original groups but within 0.04 in the fixed group). We therefore compare only within-judge paired differences.
Group dependence.
- The full-token drift the reviewer noted appears under all three judges. It is therefore a property of joint multi-candidate scoring, not of one judge.
- It does not affect same-budget comparisons. For the three compressed methods of the fixed group, every same-budget ordering is the same in the original and fixed groups, under all three judges.
- It does affect comparisons across budget groups. In the original groups, QA-MBG at 5% scores above the strongest baseline at 10% for both backbones and all three judges. In the fixed group it is slightly below in all six cases (e.g., Qwen3-VL under Luna: 2.633 vs 2.449 in the original groups, 1.963 vs 2.012 in the fixed group).
The revision states in Section 4.1 that long-form scores are compared only within a budget, and adds all of the above to the new Appendix D.5.
Blinded human audit.
- Design. Twelve raters each compared 10 pairs, giving 120 ratings over 66 distinct comparisons; 54 comparisons were rated by two raters. Each pair is QA-MBG versus the strongest same-budget baseline from the fixed-group set. Pairs are balanced across backbones and span budgets and image domains. Raters saw the image, the prompt, and the two anonymized descriptions in random order, and chose A, B, or tie.
- Preference. QA-MBG was preferred 68 times, the baseline 33 times, with 19 ties. The win rate (tie = ½) is 0.65 [0.55, 0.75]: 0.67 [0.55, 0.78] for Qwen and 0.62 [0.50, 0.75] for Gemma.
- Comparison with the judges. On the same items, the judges give 0.75 (Luna), 0.75 (Qwen3.8), and 0.80 (Haiku). The direction agrees, but the judges are more decisive than humans.
- Agreement. Where both a judge and a human state a preference, they agree on 61–70% of items (46/76, 49/70, 55/85). Two humans agree on 26/40 such items (65%), with κ = 0.21 overall. Judge–human agreement is thus close to human–human agreement on this task.
- Finer splits. Per-budget splits have 24–72 ratings each and wide intervals. Appendix D.5 (Table 13) reports them with the backbone and image-domain splits.
The p.47 example. Thank you for catching this. Giving 5/5 groundedness to descriptions that call the wheels identical is a judge error. We replaced all nine qualitative examples in Appendix Q: they now come from the fixed candidate groups, show the scores of all three judges, and mark each description by hand against the image (verified content in green, errors in red).
Response to Reviewer VPHU (3/3)
Q3. Representative runtime statistics
Thank you for pointing this out. The single-image example in Appendix B was meant to isolate the execution optimizations; it was not the source of the per-image cost.
We now report per-image statistics for the main runs of both backbones (revised Appendix B.3):
- They are measured from the per-step profiles logged during search.
- Each image was searched on one H100 80GB.
- They cover search only; model loading and evaluation are excluded.
| Qwen3-VL-4B | Gemma 4 E4B | |
|---|---|---|
| Visual tokens , mean (range) | 777 (196–980) | 1,083 (1,032–1,120) |
| Greedy steps (to ratio 0.3) | 233 | 325 |
| QA–candidate evaluations per image | 1.24M | 2.06M |
| Time per evaluation | 1.5 ms | 1.9 ms |
| GPU-hours per image, mean | 0.51 | 1.07 |
| 10th / 50th / 90th percentile | 0.04 / 0.66 / 0.77 | 0.99 / 1.07 / 1.17 |
| Maximum | 0.95 | 1.43 |
| Total for 900 images | ≈460 | ≈970 |
How the cost extrapolates.
- At step , all remaining candidates are scored on sampled QAs.
- Reaching therefore takes QA–candidate evaluations.
- With the main pool ( on average), .
- The logged counts match this formula, and search time is linear in . A single constant of 1.5 ms per evaluation explains the Qwen per-image times with .
- Cost is therefore quadratic in the number of visual tokens and linear in the number of QAs sampled per step. Other images, pool sizes, or sampling fractions can be extrapolated directly from and the per-evaluation constant.
Why the Appendix example understated the cost. Its 16-QA pilot pool gives 4 QAs per step, not about 7. A main-pool image of the same size therefore costs more: main-run DocVQA images (about 950 tokens) average 0.77 GPU-hours.
Variation across images and backbones.
- In Qwen3-VL, the token count follows image resolution, so the cost spans more than an order of magnitude. MMStar images (about 256 tokens) take 0.05 h, while DocVQA and V*Bench images (about 950 tokens) take 0.74–0.77 h. This explains the low 10th percentile.
- Gemma 4 caps images at 1,120 soft tokens, so its cost is nearly constant at about 1.1 h. That is about twice Qwen's, because Gemma has more visual tokens and a slightly higher time per evaluation.
Comparison with full greedy. On the 180-image small set, full greedy performs 4.7× as many evaluations. It takes 3.4× the GPU time for Qwen (1.7 vs 0.5 h) and 3.5× for Gemma (3.9 vs 1.1 h).
What the cost buys (new, Section 4.4). The search is a one-time cost per training image, and its subsets serve as training targets: a 0.73M-parameter selector distilled from the 900 searched images leads every compared selector on 15,410 held-out questions from unseen images, on both backbones, at the inference cost of a training-free selector.
Changes to the paper (marked in blue)
- Abstract and introduction. They state ≈0.5 (Qwen3-VL-4B) and ≈1.1 (Gemma 4 E4B) H100 GPU-hours per image. Appendix B.3 now contains the runtime tables, the cost model, and the per-benchmark breakdown. The single-image probe is kept only as an execution microbenchmark.
- New full-set experiment. The joint dataset-agnostic + overlap-filtered experiment on 900 images for both backbones is added to Section 4.3, with the protocol, per-benchmark, and long-form results in the new Appendix N. The conclusion states that held-out QA performance can drop as the auxiliary QAs depart from the target distribution, while long-form performance is largely maintained.
- New Appendix D.5. It reports the second judges (Qwen3.8-27B, Haiku 5.5), the fixed-group re-scoring, ranking vs. calibration, group dependence, and the blinded human audit. Section 4.1 states that long-form scores are compared only within a budget, and Section 4.2 summarizes the judge and human results.
- p.47. All nine qualitative examples in Appendix Q are replaced (fixed groups, three judges, errors marked by hand).
- New Section 4.4 (Figure 4) and Appendix P. A selector distilled from the searched subsets, compared with learned and training-free selectors on held-out questions, with per-benchmark scores and FLOPs.
Response to Reviewer VPHU (1/3)
We thank the reviewer for the careful reading and for questions that could be answered with experiments. We ran all three requested checks:
Revisions are marked in blue in the updated PDF.
Q1 / W1. Joint dataset-agnostic generation and overlap filtering, both backbones
Setup.
Results. Six-budget means over 5–30% retention. Actual QA is in native score; long-form uses the main long-form judge and protocol.
What this shows.
Interpretation. We agree that the QA headroom depends on the query distribution. It grows as the auxiliary questions more closely match the target questions:
When a deployment serves a specialized use case, such as document processing or driving assistance, matching the auxiliary QA distribution to that use case can maximize performance. The benchmarks play this role in our main experiments.
The revision adds this experiment to Section 4.3 ("Controlling the QA distribution"), with the protocol, per-benchmark, and long-form results in the new Appendix N.
Repeated seeds. We prioritized running the joint condition on all 900 images for both backbones. This covers five times as many images as the small set and costs more than three seeds on it.