- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WorldBench is a benchmark of 2,000 image-question pairs that prioritizes visual diversity. An LLM-assisted taxonomy of 2,000 concepts across seven domains guides image curation, with a preference for contextual, non-iconic scenes. Annotators revise four-option questions until at least one frontier model answers incorrectly. The authors assess image diversity using two spectrum-based metrics with three vision encoders and 360 pairwise judgments from 12 volunteers. They evaluate 15 model configurations, with the best overall accuracy reaching 64.0%, and analyze reasoning budgets and correlations with other benchmarks. The main contribution is a curated resource for evaluating MLLMs across a broad range of visual inputs.
Critical strengths
- Useful visual coverage. The taxonomy spans natural scenes and objects, digital interfaces, academic figures, documents, and agent environments. Taxonomy-guided curation and the deliberate selection of contextual images provide a useful basis for testing models beyond narrow visual domains. This breadth gives WorldBench value as an additional evaluation resource.
- Substantial evaluation and complementary diversity measurements. The paper evaluates 15 model configurations, reports domain-level results, and investigates reasoning effort. Combining several vision encoders with human diversity judgments is a useful design, and the paper acknowledges representation-dependent rankings. The authors also state that the dataset and evaluation code are available for review, supporting the resource contribution.
Critical weaknesses
- Human solvability and residual annotation error are not independently quantified. Section 2.3 describes manual checking, volunteer audits, and Claude Code-assisted review, but reports no independent human answer accuracy, agreement, or residual ambiguity/error rate. The human experiment in Section 3.2 evaluates image-panel diversity rather than question correctness or solvability. This distinction matters for a benchmark whose questions are revised until a model fails: reliable labels and human solvability are needed to interpret those failures. A modest independent QA audit would strengthen the claim that the questions are intuitive for humans.
- Uncertainty in the human diversity ranking is not fully characterized. Figure 7 provides bootstrap confidence intervals, but does not report uncertainty for differences between WorldBench and the other leading benchmarks. Overlapping marginal intervals alone do not determine whether those differences are statistically significant. Moreover, 12 volunteers each contribute 30 judgments, while Appendix C.1 resamples comparisons rather than participants. If judgments from the same volunteer are correlated, this procedure may understate uncertainty. The highest point estimate is a valid descriptive result, but Section 3.2 goes further, stating that participants judge WorldBench "to be more visually diverse than the comparison benchmarks." The leading scores are close (1187.4 vs. 1157.8 for MMT-Bench and 1149.3 for MMBench) with heavily overlapping intervals, so this stronger claim requires additional analysis. Participant-clustered estimates of pairwise score differences would clarify the strength of the human evidence, while the broader conclusion that WorldBench is among the most diverse benchmarks may remain supported.
Questions for authors
- Reconciling the diversity metrics. Appendix D.1 defines ER and PR from the same normalized covariance eigenvalues, implying ER >= PR. Yet all six dataset rows in Table 3 have ER < PR for both SigLIP 2 and Perception Encoder. What explains this discrepancy? Both metrics are scale-invariant, so no normalization of a single matrix can yield ER < PR. Please specify the matrix, normalization, and sample used for each metric and whether corrections affect Figure 6. Eq. (3) uses the mean-centered covariance, whereas the cosine-kernel Vendi Score corresponds to the uncentered Gram matrix; please clarify the claimed equivalence.
- Independent QA validation. Can you provide a blinded human evaluation on a random sample, with answer accuracy, agreement, and the fraction of ambiguous or corrected items? This request concerns additional QA validation, including domain-level results where feasible, rather than the existing image-diversity study.
- Screening effects and evaluation variability. Screening effects. Please report model comparisons by screening cohort. For the 542-question cohort, Qwen3.5-VL-Plus-Instruct and GPT-5.4-Thinking (low) were used as screening models; Qwen3.5-VL-Plus-Thinking and other GPT-5.4 reasoning budgets were not listed as screening configurations. This may confound the Instruct/Thinking and reasoning-budget comparisons. How do these differences vary across cohorts, including those screened with predecessor models? Evaluation variability. Please distinguish uncertainty across benchmark items from variability across repeated generations. Paired comparisons on the same items and repeated runs for close comparisons would help assess whether reported model differences and reasoning-budget trends are stable.
- Human diversity comparisons. Can you report participant-clustered bootstrap intervals for Bradley-Terry score differences between WorldBench and the other leading benchmarks, and describe how comparison pairs were allocated? Please distinguish the observed ranking from any claim of statistically supported superiority.
- Reporting and interpretation. Why do Table 2 counts sum to 1,994 rather than 2,000? Please provide domain counts and clarify the weighting of Table 1 averages (for Gemini-3.1-Pro, the unweighted mean of the seven domain scores is 65.0, not the reported 64.0). Figure 9 uses six local models, as listed in Table 4, and reports a mean correlation of 0.94. Given this small model set and high correlation, please provide uncertainty estimates or qualify the inference that WorldBench measures a distinct dimension.
3 comments
Response to Reviewer GDAu (2/2)
W2, Q4: Can you report participant-clustered bootstrap intervals for Bradley-Terry score differences between WorldBench and the other leading benchmarks, and describe how comparison pairs were allocated? Please distinguish the observed ranking from any claim of statistically supported superiority.
To estimate uncertainty, we perform 5000 bootstrap resamples of participants, while keeping each selected participant's 30 judgements together and refitting the Bradley-Terry model.
| Comparison | Score difference [95% confidence interval] |
|---|---|
| WorldBench − MMT-Bench | +30.0 [−77.1, 135.1] |
| WorldBench − MMBench | +38.3 [−94.0, 171.7] |
WorldBench retains the highest point estimate, but has uncertain differences from the closest benchmarks. This supports our interpretation of the human study as evidence that WorldBench is among the most visually diverse benchmarks evaluated here.
Each participant judged 30 distinct pairs out of 36 possible pairs, with randomized left/right placement. Each pair received 7-12 judgements overall. We thank the reviewer for suggesting this participant-level bootstrap analysis, which better communicates uncertainty in the human-diversity comparison.
Q5: Why do Table 2 counts sum to 1,994 rather than 2,000? Please provide domain counts and clarify the weighting of Table 1 averages (for Gemini-3.1-Pro, the unweighted mean of the seven domain scores is 65.0, not the reported 64.0).
Table 2 retained counts from an outdated version of the benchmark, but these have since been corrected. The table reports 1,442 questions across the earlier phases and 558 in the final phase, totaling 2,000. The final domain counts are 245 Living Things, 506 Objects, 435 Scenes, 248 Digital World, 228 Academics, 172 Documents/Charts/Tables, and 166 Agents.
Overall accuracy weights each question equally. For Gemini-3.1-Pro, 1,279/2,000 = 63.95%, which is displayed as 64.0%, rather than the unweighted mean of domain scores. We have renamed the Table 1 column from "Average" to "Overall" and updated the accompanying sentence accordingly.
Q5: Figure 9 uses six local models, as listed in Table 4, and reports a mean correlation of 0.94. Given this small model set and high correlation, please provide uncertainty estimates or qualify the inference that WorldBench measures a distinct dimension.
Thank you for highlighting the uncertainty associated with the small model set used in this analysis. WorldBench's mean Pearson correlation is 0.936, with an estimated 95% bootstrap interval of [0.826, 0.997].
We have qualified the interpretation in the paper as such:
“This analysis is limited to six local models and does not necessarily establish a distinct underlying capability.”
WorldBench's contribution rests on its explicit visual coverage, remaining performance headroom, and transferable hard cases. Here, the correlation analysis simply provides supplementary evidence.
We thank the reviewer for the detailed feedback, which helped us correct reporting errors and strengthen the evidence supporting WorldBench's contribution.
The authors' response has addressed my concerns, I will maintain the positive rating.
Response to Reviewer GDAu (1/2)
Thank you for recognizing WorldBench's breadth and complementary evaluations, and for your suggestions to improve validation and reporting.
To evaluate human answerability, we evaluate a previously unexposed human participant on 100 questions randomly sampled from WorldBench. The participant achieves 77%, compared with 61% for Qwen3.5-35B-A3B Thinking, 46% for GLM-4.6V, and 39% for Gemma4-E4B Thinking.
Many mistakes the human participant makes (the remaining 23%) are avoidable rather than unanswerable. For example, one question presents an online forum with a post with 526 upvotes, and asks what the correct count is after one additional upvote. The participant selects 526, rather than answering with the correct count after one upvote. After providing explanations for the answers, the human participant agrees fully with the answer key.
To control question quality, different authors constructed different benchmark sections, and a final review was done, revising 128 questions from the initial set.
We thank the reviewer for pointing out the errors in our metrics. The former Appendix Table 3 contained erroneous calculations. We have since removed that table and the associated interpretation, and replaced it with a DINOv3-based comparison in Appendix B.
Figure 6 remains unaffected. For these results, both metrics use the same 1,000-image sample. We L2-normalize the embeddings, subtract their sample mean, and calculate C=XX⊤/N. ER and PR are computed from the same normalized covariance eigenvalues, which implies ER ≥ PR.
We have also corrected the statement that centered-covariance ER is equivalent to ordinary cosine-kernel Vendi in the main manuscript and appendix. Thank you for identifying these errors.
Thank you for highlighting the potential effect screening history can have on model comparisons.
We compare performance on the earlier group of 1442 questions to the final group of 558 questions, finding:
We find that the later group is harder for all models, meaning that construction history affects both difficulty and relative performance. Thus, selecting questions using Instruct and low-reasoning models may have some effect on observed gains, which we aim to present as an overall trend for the configurations evaluated rather than a general claim that more reasoning consistently improves performance across all models.
For GLM-InternVL, a paired bootstrap comparison over all 2000 questions gives an overall difference of 1.20 percentage points [-1.25, 3.70], meaning that for close comparisons we cannot confidently establish their rankings.