- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how the choice of resampling unit affects uncertainty estimation and model comparison in repeated-variant VLM benchmarks. Using 23 publicly available response matrices from DynaMath, MMBench V11, and VB, the authors compare cell-IID bootstrap with family-cluster bootstrap while keeping the observed score, eligible cells, and cell weights fixed. Based on this analysis, the paper recommends explicitly reporting the target population, scoring rules, and resampling unit, and conducting fixed-statistic sensitivity analyses.
Critical strengths
S1. The paper addresses a practically relevant question by examining dependencies among prompts, option rotations, or counterfactual images derived from the same source.
S2. The experimental analysis is reasonably thorough and includes several targeted sensitivity analyses.
Critical weaknesses
Both the methodological novelty and the additional insights are relatively limited. Cluster bootstrap methods and the effects of the inferential unit on uncertainty have already been extensively studied, and the main contribution is to apply these established statistical ideas to three VLM benchmarks. Beyond this empirical application, the paper offers few substantially new insights or perspectives.
Questions for authors
The main concern is the limited contribution of this paper.