- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PRIMEBench, a four-stage framework for compressing VLM evaluation suites. It removes items deemed visually unnecessary or non-discriminative, selects representative benchmarks within manually defined capability categories, prunes items using a combination of inter-model variance and embedding-based coverage, and finally selects a small subset of capability categories whose aggregate ranking correlates with that of the larger suite.
Critical strengths
-
The empirical infrastructure is unusually broad. The paper aggregates 43 benchmarks, more than 386,000 items, and 30 recent open-weight VLMs from several model families. It additionally considers external leaderboard. The paper makes a serious effort to study this topic.
-
Hierarchical compression is a practically useful framing. Separating benchmark cleaning, within-category benchmark selection, item selection, and category-count reduction makes the trade-offs easier to inspect.
Critical weaknesses
- The headline claim is that the released suite preserves model rankings after removing more than 97% of items. However, the reported 0.959 correlation is not measured against the original 43-benchmark ranking defined in Section 3.
Firstly, Stage 0 changes the item distributions. The paper removes all-correct and text-only-solvable items and then uses this cleaned pool as the reference.
- Removing all-correct items preserves the ordering within an individual benchmark because it applies an affine transformation to that benchmark’s model scores. It does not necessarily preserve the ordering after averaging across benchmarks, because benchmarks with different removal rates are rescaled by different factors. Removal rates range from 0% to 78.3%, so this effect may be substantial.
- An item solvable without an image may still discriminate VLMs through language capability. Removing it may be normatively desirable when designing a vision-specific suite, but that is benchmark redesign rather than compression of the original ranking.
Secondly, the end-to-end evaluation changes benchmark weighting. Section 3 defines the full score as an equal average over benchmarks. Appendix D.3 states that the end-to-end reference instead averages category means, thereby weighting categories equally rather than benchmarks. This can change the target ranking.
Consequently, Table 2 seems combine correlations computed against different references and cannot establish that the released suite preserves the original full-suite ordering.
- The key metric used by this work, spearman correlation summarizes squared rank displacement but does not directly report how many model pairs are reversed, whether the top models change order, or whether differences among close competitors are preserved. A high Spearman score can also be dominated by easy separation between model families while failing among the strongest models.
The paper should report pairwise inversion rates, top-k overlap, maximum and median rank displacement, winner preservation, and performance specifically among statistically indistinguishable or top-performing models.
- Rank preservation is not capability preservation. The released suite keeps only 5 categories (Section 6). 13 capability categories are removed. A high aggregate rank correlation does not imply that these omitted capabilities are recoverable. It may instead show that the evaluated models share a dominant general-capability. A specialized model that improves substantially on an omitted capability could be ranked incorrectly. The paper needs capability-level distortion analyses.
Moreover, stage 3 constrains the number of categories rather than the number of items, inference FLOPs, latency, or monetary cost. Because category pools have very different sizes, reducing the category count does not reliably reduce evaluation cost. The method should optimize actual measured cost subject to fidelity and coverage constraints, rather than selecting k=5 from a heuristic.
-
Stage 1 selects the benchmark most correlated with the mean ranking of its category. Correlation with a category mean does not imply that the selected benchmark covers the other benchmarks’ content, answer format, difficulty, cultural distribution, or evaluation metric. The paper should report within-category content coverage, model-by-benchmark residuals, and worst-case reconstruction errors rather than only median category-level rank correlation.
-
The theoretical appendix is disconnected from the main objective. It does not establish how variance selection preserves model rankings, how selected items cover capabilities, etc.
Questions for authors
see the above "Critical Weaknesses" part.
2 comments
Response to Reviewer LMQ8 (2/2)
Response to Critical Weakness 2: Decision-Focused & Fine-Grained Ranking Metrics
- Decision metrics against baselines: On held-out splits, the released suite reaches Kendall and inverts of model pairs, in the same range as Random, Variance, and AnchorPoints (table below). Keeping all 18 categories halves the inversion rate to .
- Close pairs: We call a pair close when the % item-bootstrap interval of its reference score difference contains zero. Close pairs make up % of all pairs but % of the suite's inversions, and every selector we tested inverts them at a similar rate ( to ). Near-tied models therefore need the full benchmarks to be separated, whichever selector is used.
| Held-out selector (%, 20 splits) | : Spearman | : inversion | : winner kept | 18 cat.: Spearman | 18 cat.: inversion | 18 cat.: winner kept |
|---|---|---|---|---|---|---|
| VAW | ||||||
| Random | ||||||
| AnchorPoints | ||||||
| Variance |
Response to Critical Weakness 3: Capability Distortion & Cost-Budgeted Stage 3 Optimization
- Capability distortion: The suite targets aggregate ranking under tight GPU budgets, and the 18-category suite ( held-out) serves capability-level reporting. Dropped categories are predicted better than the build-mean baseline in every split (see our response to Reviewer k7i9).
- GPU-hour budget in Stage 3: Category runtime varies widely (Math Reasoning alone takes GPU-hours on Qwen3-VL-8B), so we replaced Stage 3's fixed category count with an explicit GPU-hour budget, keeping the search objective unchanged. Within the released suite's budget, cost-budgeted Stage 3 matches fixed-count fidelity at about % of the compute.
| Stage 3 variant ( splits, %, Qwen3-VL-8B) | Mean categories | GPU-h (log-based) | Held-out |
|---|---|---|---|
| Fixed | |||
| GPU budget h | |||
| GPU budget h |
Response to Critical Weakness 4: Stage 1 Within-Category Coverage, Residuals, & Reconstruction Error
-
Category design and pooling: The 18 categories follow the capability groupings of recent frontier VLM reports (Qwen3-VL, InternVL3.5, GLM-4.1V) and were fixed before selection. Stage 1 checks each representative against its category mean and, when the correlation falls below , adds high-variance items from the other benchmarks in that category.
-
Within-category diagnostics: Omitted benchmark scores are reconstructed from the category representative on held-out models with the error below.
| Stage 1 within-category diagnostic | Value |
|---|---|
| Representative-to-benchmark correlation (median) | |
| Held-out benchmark reconstruction MAE (points) |
Response to Critical Weakness 5: Scoping the Theoretical Appendix
The theoretical appendix targets a narrower question than rank preservation or capability coverage, as its closing remarks state. It shows that the vision tilt shrinks at the rate of the variance-estimation noise, so Stage 2 approaches plain variance selection on large panels, whereas a fixed tilt would not.
The General Response lists the planned revisions.
Thank you again for your feedback. It is very helpful for improving our paper.
Response to Reviewer LMQ8 (1/2)
We thank Reviewer LMQ8 for the feedback. Below, we give the core findings on each Critical Weakness.
Response to Critical Weakness 1: Reference Definitions, Stage 0 Filtering, & Table 2 Unification
We compared the released suite across reference definitions and against unmodified pre-Stage-0 baselines.