acceptodds
Withdrawn from ICLR 2027

Hierarchical Compression of Vision-Language Model Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace. Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for language models, but for VLMs the question remains under-explored. We present **PRIMEBench** (**P**runing **R**edundant **I**tems for **M**ultimodal **E**valuation), a vision-aware hierarchical benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one benchmark per capability category, item pruning with Vision-Aware Variance (VAW), and category-count pruning. VAW combines inter-model variance with a vision-dependence score computed from multimodal embeddings alone, while encouraging coverage of diverse items within each benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side pruning.

Primary area:
datasets and benchmarks
Keywords:
Data selection, Efficiency, Vision-language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LMQ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRIMEBench, a four-stage framework for compressing VLM evaluation suites. It removes items deemed visually unnecessary or non-discriminative, selects representative benchmarks within manually defined capability categories, prunes items using a combination of inter-model variance and embedding-based coverage, and finally selects a small subset of capability categories whose aggregate ranking correlates with that of the larger suite.

Critical strengths
  1. The empirical infrastructure is unusually broad. The paper aggregates 43 benchmarks, more than 386,000 items, and 30 recent open-weight VLMs from several model families. It additionally considers external leaderboard. The paper makes a serious effort to study this topic.

  2. Hierarchical compression is a practically useful framing. Separating benchmark cleaning, within-category benchmark selection, item selection, and category-count reduction makes the trade-offs easier to inspect.

Critical weaknesses
  1. The headline claim is that the released suite preserves model rankings after removing more than 97% of items. However, the reported 0.959 correlation is not measured against the original 43-benchmark ranking defined in Section 3.

Firstly, Stage 0 changes the item distributions. The paper removes all-correct and text-only-solvable items and then uses this cleaned pool as the reference.

  • Removing all-correct items preserves the ordering within an individual benchmark because it applies an affine transformation to that benchmark’s model scores. It does not necessarily preserve the ordering after averaging across benchmarks, because benchmarks with different removal rates are rescaled by different factors. Removal rates range from 0% to 78.3%, so this effect may be substantial.
  • An item solvable without an image may still discriminate VLMs through language capability. Removing it may be normatively desirable when designing a vision-specific suite, but that is benchmark redesign rather than compression of the original ranking.

Secondly, the end-to-end evaluation changes benchmark weighting. Section 3 defines the full score as an equal average over benchmarks. Appendix D.3 states that the end-to-end reference instead averages category means, thereby weighting categories equally rather than benchmarks. This can change the target ranking.

Consequently, Table 2 seems combine correlations computed against different references and cannot establish that the released suite preserves the original full-suite ordering.

  1. The key metric used by this work, spearman correlation summarizes squared rank displacement but does not directly report how many model pairs are reversed, whether the top models change order, or whether differences among close competitors are preserved. A high Spearman score can also be dominated by easy separation between model families while failing among the strongest models.

The paper should report pairwise inversion rates, top-k overlap, maximum and median rank displacement, winner preservation, and performance specifically among statistically indistinguishable or top-performing models.

  1. Rank preservation is not capability preservation. The released suite keeps only 5 categories (Section 6). 13 capability categories are removed. A high aggregate rank correlation does not imply that these omitted capabilities are recoverable. It may instead show that the evaluated models share a dominant general-capability. A specialized model that improves substantially on an omitted capability could be ranked incorrectly. The paper needs capability-level distortion analyses.

Moreover, stage 3 constrains the number of categories rather than the number of items, inference FLOPs, latency, or monetary cost. Because category pools have very different sizes, reducing the category count does not reliably reduce evaluation cost. The method should optimize actual measured cost subject to fidelity and coverage constraints, rather than selecting k=5 from a heuristic.

  1. Stage 1 selects the benchmark most correlated with the mean ranking of its category. Correlation with a category mean does not imply that the selected benchmark covers the other benchmarks’ content, answer format, difficulty, cultural distribution, or evaluation metric. The paper should report within-category content coverage, model-by-benchmark residuals, and worst-case reconstruction errors rather than only median category-level rank correlation.

  2. The theoretical appendix is disconnected from the main objective. It does not establish how variance selection preserves model rankings, how selected items cover capabilities, etc.

Questions for authors

see the above "Critical Weaknesses" part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LMQ8 (1/2)

We thank Reviewer LMQ8 for the feedback. Below, we give the core findings on each Critical Weakness.


Response to Critical Weakness 1: Reference Definitions, Stage 0 Filtering, & Table 2 Unification

We compared the released suite across reference definitions and against unmodified pre-Stage-0 baselines.

  • Pre-Stage-0 vs. post-Stage-0 reference correlations:
    • Evaluated across the candidate reference definitions before and after Stage 0, the released suite achieves ρ=0.949\rho = 0.949 against the unmodified Section 3 pre-Stage-0 reference (see table below).
    • Under our paper's category-mean of benchmark zz-scores reference, pre-Stage-0 correlation reaches ρ=0.956\rho = 0.956. Crucially, the pre- and post-Stage-0 ranking targets correlate at ρ=0.990\rho = 0.990, and all reference definitions align closely with one another.
  • All-correct removal & rescaling rates:
    • Differing removal rates rescale raw benchmark scores differently. On ScienceQA, 78.378.3% is the combined Stage 0 removal rate. All-correct removal alone on ScienceQA is 28.328.3%, which leaves zz-scored references invariant.
  • Rationale for blind-solvable filtering:
    • We respectfully view removing blind-solvable items as preserving what a vision-language benchmark is intended to measure rather than redesigning it. Prior works, including MMStar (Chen et al., NeurIPS 2024), MMMU-Pro (Yue et al., ACL 2025), and VQA v2 (Goyal et al., CVPR 2017), demonstrate that text-solvable items evaluate LLM language priors rather than multimodal perception. While text-only capabilities can be measured far more cheaply without images, Stage 0 filtering barely alters the ranking target (ρ=0.990\rho = 0.990).
  • Unifying Section 3, Appendix D.3, and Table 2:
    • Weighting definitions across Section 3, Appendix D.3, and Table 2 align closely, and under the unified Section 3 reference the released suite reaches ρ=0.949\rho = 0.949 in-sample (see table below).
Reference definitionIn-sample pre-Stage-0In-sample post-Stage-0Held-out 20 splits (r=5r = 5% VAW, vs. Stage-0 ref.)
Section 3 (equal average of raw benchmark scores)0.9490.9490.9470.9470.86860.8686
Equal-benchmark zz-score0.9500.9500.9550.9550.88070.8807
Paper reference (category mean of benchmark zz; pre vs. post ρ=0.990\rho = 0.990)0.9560.9560.9590.9590.89250.8925
Range across 10 definitions & Inter-reference ρ\rho0.947–0.9700.947\text{–}0.9700.947–0.9660.947\text{–}0.966Inter-ref. ρ\rho: 0.980–0.9910.980\text{–}0.991
AuthorsOpenReview ↗

Response to Reviewer LMQ8 (2/2)

Response to Critical Weakness 2: Decision-Focused & Fine-Grained Ranking Metrics

  • Decision metrics against baselines: On held-out splits, the released suite reaches Kendall τ=0.759\tau = 0.759 and inverts 0.1200.120 of model pairs, in the same range as Random, Variance, and AnchorPoints (table below). Keeping all 18 categories halves the inversion rate to 0.0670.067.
  • Close pairs: We call a pair close when the 9595% item-bootstrap interval of its reference score difference contains zero. Close pairs make up 1111% of all pairs but 4141% of the suite's inversions, and every selector we tested inverts them at a similar rate (0.380.38 to 0.490.49). Near-tied models therefore need the full benchmarks to be separated, whichever selector is used.
Held-out selector (r=5r = 5%, 20 splits)k=5k = 5: Spearmank=5k = 5: inversionk=5k = 5: winner kept18 cat.: Spearman18 cat.: inversion18 cat.: winner kept
VAW0.8920.8920.1200.1200.450.450.9560.9560.0670.0670.400.40
Random0.8990.8990.1150.1150.430.430.9480.9480.0730.0730.450.45
AnchorPoints0.8930.8930.1200.1200.250.250.9500.9500.0740.0740.450.45
Variance0.8890.8890.1250.1250.300.300.9470.9470.0740.0740.400.40

Response to Critical Weakness 3: Capability Distortion & Cost-Budgeted Stage 3 Optimization

  • Capability distortion: The k=5k = 5 suite targets aggregate ranking under tight GPU budgets, and the 18-category suite (ρ=0.956\rho = 0.956 held-out) serves capability-level reporting. Dropped categories are predicted better than the build-mean baseline in every split (see our response to Reviewer k7i9).
  • GPU-hour budget in Stage 3: Category runtime varies widely (Math Reasoning alone takes 7.77.7 GPU-hours on Qwen3-VL-8B), so we replaced Stage 3's fixed category count with an explicit GPU-hour budget, keeping the search objective unchanged. Within the released suite's budget, cost-budgeted Stage 3 matches fixed-count fidelity at about 3030% of the compute.
Stage 3 variant (2020 splits, r=5r = 5%, Qwen3-VL-8B)Mean categoriesGPU-h (log-based)Held-out ρ\rho
Fixed k=5k = 55.05.01.431.430.8920.892
GPU budget ≤1.61\le 1.61 h7.17.10.430.430.8990.899
GPU budget ≤0.40\le 0.40 h5.755.750.210.210.8860.886

Response to Critical Weakness 4: Stage 1 Within-Category Coverage, Residuals, & Reconstruction Error

  • Category design and pooling: The 18 categories follow the capability groupings of recent frontier VLM reports (Qwen3-VL, InternVL3.5, GLM-4.1V) and were fixed before selection. Stage 1 checks each representative against its category mean and, when the correlation falls below 0.950.95, adds high-variance items from the other benchmarks in that category.

  • Within-category diagnostics: Omitted benchmark scores are reconstructed from the category representative on held-out models with the error below.

Stage 1 within-category diagnosticValue
Representative-to-benchmark correlation (median)0.7180.718
Held-out benchmark reconstruction MAE (points)6.066.06

Response to Critical Weakness 5: Scoping the Theoretical Appendix

The theoretical appendix targets a narrower question than rank preservation or capability coverage, as its closing remarks state. It shows that the vision tilt shrinks at the rate of the variance-estimation noise, so Stage 2 approaches plain variance selection on large panels, whereas a fixed tilt would not.

The General Response lists the planned revisions.

Thank you again for your feedback. It is very helpful for improving our paper.

Reviewer_k7i9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PRIMEBench is a four-stage pipeline that compresses a 43-benchmark VLM suite. It removes blind-solvable items, keeps one benchmark per category, and prunes items by Vision-Aware Variance (VAW). VAW adds an embedding-based vision term to item variance. The released suite keeps 9,377 of 386,324 items from 5 of 18 categories.

Critical strengths
  1. The protocol is careful and candid. Held-out tests use 20 random model splits and a chronological split, with spread reported.

  2. The study shows strong redundancy across VLM benchmarks. On 231 external leaderboard models, five benchmarks keep a rank correlation of 0.988.

Critical weaknesses
  1. The vision-aware part adds no measurable fidelity. Removing the vision term moves VAW from 0.925 to 0.921, and text-only cells score higher at 0.930. Overall, VAW leads AnchorPoints by only 0.005, about one rank swap among 15 held-out models.

  2. The released suite's 0.959 is measured on the 30 models that chose it. When refit on 15 build models, the pipeline reaches 0.892 on held-out models, below 0.901 for benchmark selection alone. The suite drops 13 of 18 categories, including math, document and spatial reasoning.

Questions for authors
  1. Can you correlate the cosine vision score with the accuracy drop without images? Also add uniform random sampling at equal per-benchmark budget, in Stage 2 and end to end. This would show whether the vision term carries signal and whether any selector beats random.

  2. Can you score the frozen suite on proprietary or thinking models, and hold out all 10 Qwen models? Prediction error on the 13 dropped categories would show what the suite misses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer k7i9

We thank Reviewer k7i9 for the feedback. Below, we give the core findings for each Critical Weakness and Question.


Response to Critical Weakness 1: Visual Components & AnchorPoints Comparison

  • Visual components: We agree. As stated in the General Response, the visual terms do not add held-out ranking fidelity, and the Stage 2 gain comes from embedding coverage cells (+0.0172+0.0172 over proportional benchmark shares).
  • AnchorPoints: VAW ties AnchorPoints on random splits (+0.005+0.005 within categories) and leads or ties it under distribution shift.
Distribution shiftVAWAnchorPoints
LoRA checkpoints0.9390.9390.8520.852
Chronological split0.9360.9360.9290.929
Leave-one-family-out0.9270.9270.9280.928

Response to Critical Weakness 2: In-Sample 0.9590.959, Stage 3 Instability, & Suite Scope

  • In-sample 0.9590.959: 0.9590.959 is the in-sample value on the construction panel. On held-out models, the 18-category suite reaches 0.9560.956.
  • Stage 3 instability: The variance comes from Stage 3 choosing different categories across splits, and it affects every selector, Random included. Without Stage 3, the 18-category suite keeps held-out fidelity at 0.9560.956, so we recommend it as the default operating point.
Suite configurationHeld-out ρ\rho
18 pruned categories (no Stage 3, default)0.9560.956
Stage 3 (k=5k = 5) on pruned pools0.8920.892
Stage 3 (k=5k = 5) on unpruned pools0.9010.901

Response to Question 1: Vision Score vs. Image-Free Drop & Stratified Random Baselines

  • δVL\delta^{\text{VL}} vs. image-free accuracy drop: δVL\delta^{\text{VL}} correlates positively with the full-image minus image-free accuracy drop in most benchmarks (pooled within-benchmark Spearman ρ=+0.195\rho = +0.195). Since the visual terms do not change rankings (General Response), selection rests on the coverage cells.
  • Stratified Random: Stratified and uniform Random behave alike. VAW is highest within categories, and all three are comparable end to end.
Selector (r=5r = 5%)Within-category ρ\rhoEnd-to-end ρ\rho
Stratified Random0.92240.92240.89590.8959
Uniform Random0.92170.92170.89860.8986
VAW0.92490.92490.89250.8925

Response to Question 2: Proprietary/Thinking Models, All-Qwen Holdout, & 13-Category Error

  • Proprietary models (OpenVLM): Subsets frozen on open-weight models transfer to proprietary models and outperform random sampling on overlapping MMMU items.
OpenVLM proprietary-model checkRank correlation ρ\rho
Frozen five-benchmark subset0.9620.962
Selected subset on overlapping MMMU items0.9410.941
  • All-Qwen holdout: With all 10 Qwen models held out as the test fold, VAW leads Random within categories and keeps strong end-to-end ranking.
All-Qwen holdout (1010 models, r=5r = 5%)VAW ρ\rho
Within-category0.9130.913
End-to-end0.9640.964
  • Error on the 13 dropped categories: Ridge regression from the retained categories predicts the dropped categories better than the build-mean baseline in every split, and a specialist model shifts by less than one rank on average. On 280280 fine-grained capability labels, VAW also improves per-capability ranking over Random in all 2020 splits. For category-level reporting we recommend the 18-category suite.
Dropped categories (2020 splits, held-out)Result
MAE (points)6.176.17 vs. build mean 7.967.96
Specialist shift (+1+1 / +2+2 SD)Mean 0.420.42 / 0.810.81 ranks
Fine-grained per-capability ρ\rhoVAW 0.6550.655 vs. Random 0.6280.628

The General Response lists the planned revisions.

Thank you again for your feedback. It is very helpful for improving our paper.

Reviewer_nBc5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRIMEBench compresses VLM evaluation suites through data cleaning, representative benchmark selection, item pruning, and category pruning. Its VAW selector combines inter-model variance, a vision-dependence score, and coverage cells. Experiments cover 30 models and 43 benchmarks. The released suite retains 9,377 items, a 97.6% reduction, with (\rho=0.959) against the cleaned reference on the construction panel. Refitting Stages 1–3 within each split yields a mean held-out end-to-end correlation of 0.892.

Critical strengths
  1. Joint benchmark and item compression has practical value, and the staged design accommodates different budgets.
  2. The evaluation includes repeated random splits, chronological evaluation, end-to-end refitting, and additional benchmark-level validation on OpenVLM.
  3. The authors report ablations and negative results candidly. Analyses such as checkpoint ranking help clarify the method’s scope.
Critical weaknesses
  1. The additional contribution of the visual components is not sufficiently established. Section 7 attributes most of the improvement to coverage cells. In Table 13, the default method achieves 0.925, while using text-only cells with α=0\alpha=0 removes both visual components of Stage 2 and still achieves 0.929. These means do not establish that text-only selection is better, but they do not demonstrate a clear benefit over text-only coverage with variance scoring. The panel-size scan in Section 7 offers some support for the visual tilt, but does not resolve this comparison on held-out models at the released setting. The contribution claims should distinguish the benefit of coverage from that of the visual components. Moreover, the embedding shift in Equation (6) needs direct validation as a measure of whether an answer requires the image; the existing image-free responses offer a way to test this assumption.
  2. The end-to-end comparisons leave the practical benefit of choosing VAW unclear. Figure 3 reports 0.925 for VAW versus 0.920 for AnchorPoints at 5% retention, without uncertainty estimates for their paired difference. In the held-out end-to-end comparison, VAW, AnchorPoints, and Variance achieve 0.892, 0.893, and 0.889. At 10%, VAW reaches 0.866 versus 0.917 for Variance (Table 12). These results do not show a consistent advantage under the reported protocol. However, different selectors may lead Stage 3 to choose different categories, so equal retention does not necessarily imply equal final cost. Reporting final item counts and adding uniform-random and benchmark-stratified random item selectors would help establish whether VAW offers a better cost–fidelity trade-off.
Questions for authors

See Critical Weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nBc5

We thank Reviewer nBc5 for the feedback. Below, we give the core findings on each Critical Weakness.


Response to Critical Weakness 1: Visual Components, Coverage Cells, and Eq. (6) Validation

We agree. As stated in the General Response, the visual terms do not add held-out ranking fidelity. The Stage 2 gain over proportional benchmark shares comes from embedding coverage cells (+0.0172+0.0172), and text-only cells reach 0.9290.929 against 0.9250.925 for VAW. The small-panel tilt gain likewise matches a permuted δ\delta.

For the direct check of Eq. (6) against image-free responses, δVL\delta^{\text{VL}} tracks the measured accuracy drop under a blank image positively on most benchmarks.

Eq. (6) check (152,852152{,}852 items)Value
δVL\delta^{\text{VL}} vs. image-minus-blank accuracy (Spearman ρ\rho)+0.195+0.195
Blind-solvable detection (mean AUC)0.6570.657

Response to Critical Weakness 2: End-to-End Fidelity, Uncertainty, Random Baselines, and Cost

  • Paired comparisons: Within categories, VAW leads Variance, IRT 2PL, DatBench, and DISCO and ties AnchorPoints and tinyBenchmarks.
Within-category vs. VAW (r=5r = 5%)VarianceIRT 2PLDatBenchDISCOAnchorPointstinyBenchmarks
VAW Δρ\Delta\rho+0.019+0.019+0.016+0.016+0.031+0.031+0.143+0.143+0.005+0.005+0.002+0.002
Paired outcomeVAW leadsVAW leadsVAW leadsVAW leadsTieTie
  • Random baselines: Item selection gains most in per-capability ranking, where the coverage and discrimination selector clearly exceeds Random. End to end, every selector including Random lands near 0.890.89, because the five categories Stage 3 picks change from split to split and this variance dominates all selectors alike.
Setting (r=5r = 5%)Coverage + discrimination selectorRandom
Per-capability ranking0.69440.69440.62790.6279
Cross-category general ranking0.94270.94270.93600.9360
Within-category0.92490.92490.92240.9224
End-to-end with Stage 30.89250.89250.89860.8986
  • Item counts and GPU hours: At comparable fidelity, VAW keeps fewer items and fewer GPU hours than Random, and both cut the cost of the selected categories by an order of magnitude. We recommend the 18-category suite as the default operating point and the kk-category suite for strict budgets.
Operating point (Qwen3-VL-8B, r=5r = 5%)Final itemsGPU-hHeld-out ρ\rho
18-category suite (default)8,9318{,}9316.546.540.95570.9557
Stage 3, VAW2,7102{,}7100.920.920.89250.8925
Stage 3, uniform Random3,1313{,}1311.481.480.89860.8986
Five categories, all items74,94474{,}94411.9511.950.90090.9009

The General Response lists the planned revisions.

Thank you again for your feedback. It is very helpful for improving our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their feedback. New analyses on our 3030-model panel, 2020 held-out splits, and 231231 OpenVLM models confirm the reviewers' main point.

  1. No gain from visual terms: The visual components of Stage 2 do not improve held-out ranking fidelity, and the gain over proportional benchmark shares comes from embedding coverage cells. Text-only cells without the vision tilt reach 0.9290.929 against 0.9250.925 for VAW.

  2. Why visual terms do not help: Measured vision dependence (accuracy with the image minus accuracy with a blank image) correlates at 0.770.77 with item accuracy and tracks difficulty, which inter-model variance already captures. At matched accuracy, high- and low-dependence items give near-identical held-out model rankings. On our panel, information that only the pixels carry concentrates in recognition, reading, and grounding, whereas a text description reproduces most of the image's effect on visual reasoning tasks. Spatial reasoning shows the largest per-category gain of a measured vision-dependence term over the text-only selector (+0.065+0.065), which suggests the hypothesis that image-derived terms may help where the image itself carries discriminative information, as in spatial reasoning. This gain is not yet robust, and we are testing the hypothesis.

  3. Withdrawal and revision: Because these results remove the vision-aware selector as a primary contribution, the paper needs a full revision, so we will withdraw this submission. The revision will:

    • present Stage 2 as variance selection over coverage cells;
    • report random baselines, paired intervals, and measured GPU cost for every selector;
    • add a direct check of Eq. (6) against image-free responses;
    • recommend the 18-category suite as the default;
    • add ranking diagnostics such as pairwise inversions and close pairs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.