acceptodds
Under review as a conference paper at ICLR 2027

Cells or Variant Families? Resampling-Unit Sensitivity in Repeated-Variant VLM Benchmarks

Abstract

Repeated-variant vision-language model (VLM) benchmarks score multiple prompts, option rotations, or counterfactual images derived from a shared source. Treating those cells as independent can change reported uncertainty while leaving the observed score, eligible cells, and weights unchanged. We conduct a version-pinned descriptive audit of 23 public response matrices from DynaMath, MMBench V11, and VB, comparing cell-IID and source-family percentile-bootstrap intervals. Median family-to-cell interval-width ratios are 2.329, 1.827, and 0.865, respectively; the MMBench result is a rotated-prompt cell diagnostic, not its official CircularEval score. Across ten fixed model pairs, one unadjusted pointwise comparison changes from positive to unresolved; the other nine are unchanged. A post-hoc Bonferroni analysis also changes one comparison, but a different one. We report a scorer correction to our earlier analysis and present a descriptive re-analysis rather than a prospectively verified confirmatory study. The contribution is a bounded VLM-specific cross-benchmark measurement, not a new bootstrap method or a field-wide prevalence claim. The results motivate declaring the target population, scoring and eligibility rules, and resampling unit, and testing sensitivity while keeping the observed statistic fixed.

Primary area:
datasets and benchmarks
Keywords:
vision-language model evaluation, benchmark uncertainty, clustered bootstrap, resampling unit, repeated-variant benchmarks, reproducibility audit

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_552fOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the choice of resampling unit affects uncertainty estimation and model comparison in repeated-variant VLM benchmarks. Using 23 publicly available response matrices from DynaMath, MMBench V11, and VB, the authors compare cell-IID bootstrap with family-cluster bootstrap while keeping the observed score, eligible cells, and cell weights fixed. Based on this analysis, the paper recommends explicitly reporting the target population, scoring rules, and resampling unit, and conducting fixed-statistic sensitivity analyses.

Critical strengths

S1. The paper addresses a practically relevant question by examining dependencies among prompts, option rotations, or counterfactual images derived from the same source.

S2. The experimental analysis is reasonably thorough and includes several targeted sensitivity analyses.

Critical weaknesses

Both the methodological novelty and the additional insights are relatively limited. Cluster bootstrap methods and the effects of the inferential unit on uncertainty have already been extensively studied, and the main contribution is to apply these established statistical ideas to three VLM benchmarks. Beyond this empirical application, the paper offers few substantially new insights or perspectives.

Questions for authors

The main concern is the limited contribution of this paper.

Reviewer_AQyrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper audits 23 public response matrices across three repeated-variant VLM benchmarks, comparing bootstrap uncertainty estimates based on individual variants versus source families while keeping observed scores and weights fixed. It finds that this choice can widen or narrow confidence intervals and changes one of ten pointwise model-comparison conclusions, highlighting the need to report and assess sensitivity to the resampling unit.

Critical strengths
  1. The paper studies a clear issue: whether repeated variants should be treated as separate samples or grouped by their original source.
Critical weaknesses
  1. The presentation is hard to follow. Terms such as "width-ratio equivalence band," "uncertainty contracts," and "family-cluster" do not have clear definitions.
  2. References are missing: the paper evaluates several existing VLMs, such as Gemma, InternVL, Qwen-VL, and LLaVA, but does not clearly cite them.
  3. Family resampling gives opposite effects across benchmarks: much wider intervals for DynaMath/MMBench, but narrower intervals for VB. The manuscript does not clearly explain why.
Questions for authors

"After Bonferroni correction, one comparison still changes, but it is a different model pair." Does this weaken the main result?

Reviewer_gtBKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits the evaluation practice on VLM benchmarks that include multiple questions (cells) as variants derived from the same source image or prompt (family). The authors show that scoring at family-level leads to confidence intervals larger than standard cell-level scores on two of the three benchmarks evaluated (DynaMath and MMBench V11), changing the statistical significance of model performance comparisons.

Critical strengths
  • LLM/MLLM evaluation often suffers from high variance due to the stochastic nature in generative models and the limited dataset size. Audit results from this paper show that the performance estimation and model ranking can potentially be affected by the score aggregation, whether micro (by cell) or macro (by family).
Critical weaknesses
  • The primary concern is the limited technical depth and scientific novelty. The paper largely extends the analysis of existing literature (Neuhof & Benjamini, 2026a) from LLM benchmarks to VLM, does not run any model evaluation and instead relies on public model outputs. It does not introduce additional insights, novel metrics or practical solutions to the underlying problem.
  • The choice of benchmarks seems arbitrary and somewhat narrow. The paper would be stronger if e.g. extended to video benchmarks where multiple questions are associated with the same video.
  • Writing appears entirely AI-generated and includes unclear references to audit runs like A0-A5 without clearly defining them, making the paper difficult to follow and incoherent at times.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.