acceptodds
Under review as a conference paper at ICLR 2027

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

Abstract

In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. Quantitative and human evaluations provide complementary evidence that WorldBench is among the most visually diverse benchmarks we evaluate, while each evaluation has distinct limitations. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

Primary area:
datasets and benchmarks
Keywords:
multimodal datasets, multimodal LLMs, VLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GDAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldBench is a benchmark of 2,000 image-question pairs that prioritizes visual diversity. An LLM-assisted taxonomy of 2,000 concepts across seven domains guides image curation, with a preference for contextual, non-iconic scenes. Annotators revise four-option questions until at least one frontier model answers incorrectly. The authors assess image diversity using two spectrum-based metrics with three vision encoders and 360 pairwise judgments from 12 volunteers. They evaluate 15 model configurations, with the best overall accuracy reaching 64.0%, and analyze reasoning budgets and correlations with other benchmarks. The main contribution is a curated resource for evaluating MLLMs across a broad range of visual inputs.

Critical strengths
  1. Useful visual coverage. The taxonomy spans natural scenes and objects, digital interfaces, academic figures, documents, and agent environments. Taxonomy-guided curation and the deliberate selection of contextual images provide a useful basis for testing models beyond narrow visual domains. This breadth gives WorldBench value as an additional evaluation resource.
  2. Substantial evaluation and complementary diversity measurements. The paper evaluates 15 model configurations, reports domain-level results, and investigates reasoning effort. Combining several vision encoders with human diversity judgments is a useful design, and the paper acknowledges representation-dependent rankings. The authors also state that the dataset and evaluation code are available for review, supporting the resource contribution.
Critical weaknesses
  1. Human solvability and residual annotation error are not independently quantified. Section 2.3 describes manual checking, volunteer audits, and Claude Code-assisted review, but reports no independent human answer accuracy, agreement, or residual ambiguity/error rate. The human experiment in Section 3.2 evaluates image-panel diversity rather than question correctness or solvability. This distinction matters for a benchmark whose questions are revised until a model fails: reliable labels and human solvability are needed to interpret those failures. A modest independent QA audit would strengthen the claim that the questions are intuitive for humans.
  2. Uncertainty in the human diversity ranking is not fully characterized. Figure 7 provides bootstrap confidence intervals, but does not report uncertainty for differences between WorldBench and the other leading benchmarks. Overlapping marginal intervals alone do not determine whether those differences are statistically significant. Moreover, 12 volunteers each contribute 30 judgments, while Appendix C.1 resamples comparisons rather than participants. If judgments from the same volunteer are correlated, this procedure may understate uncertainty. The highest point estimate is a valid descriptive result, but Section 3.2 goes further, stating that participants judge WorldBench "to be more visually diverse than the comparison benchmarks." The leading scores are close (1187.4 vs. 1157.8 for MMT-Bench and 1149.3 for MMBench) with heavily overlapping intervals, so this stronger claim requires additional analysis. Participant-clustered estimates of pairwise score differences would clarify the strength of the human evidence, while the broader conclusion that WorldBench is among the most diverse benchmarks may remain supported.
Questions for authors
  1. Reconciling the diversity metrics. Appendix D.1 defines ER and PR from the same normalized covariance eigenvalues, implying ER >= PR. Yet all six dataset rows in Table 3 have ER < PR for both SigLIP 2 and Perception Encoder. What explains this discrepancy? Both metrics are scale-invariant, so no normalization of a single matrix can yield ER < PR. Please specify the matrix, normalization, and sample used for each metric and whether corrections affect Figure 6. Eq. (3) uses the mean-centered covariance, whereas the cosine-kernel Vendi Score corresponds to the uncentered Gram matrix; please clarify the claimed equivalence.
  2. Independent QA validation. Can you provide a blinded human evaluation on a random sample, with answer accuracy, agreement, and the fraction of ambiguous or corrected items? This request concerns additional QA validation, including domain-level results where feasible, rather than the existing image-diversity study.
  3. Screening effects and evaluation variability. Screening effects. Please report model comparisons by screening cohort. For the 542-question cohort, Qwen3.5-VL-Plus-Instruct and GPT-5.4-Thinking (low) were used as screening models; Qwen3.5-VL-Plus-Thinking and other GPT-5.4 reasoning budgets were not listed as screening configurations. This may confound the Instruct/Thinking and reasoning-budget comparisons. How do these differences vary across cohorts, including those screened with predecessor models? Evaluation variability. Please distinguish uncertainty across benchmark items from variability across repeated generations. Paired comparisons on the same items and repeated runs for close comparisons would help assess whether reported model differences and reasoning-budget trends are stable.
  4. Human diversity comparisons. Can you report participant-clustered bootstrap intervals for Bradley-Terry score differences between WorldBench and the other leading benchmarks, and describe how comparison pairs were allocated? Please distinguish the observed ranking from any claim of statistically supported superiority.
  5. Reporting and interpretation. Why do Table 2 counts sum to 1,994 rather than 2,000? Please provide domain counts and clarify the weighting of Table 1 averages (for Gemini-3.1-Pro, the unweighted mean of the seven domain scores is 65.0, not the reported 64.0). Figure 9 uses six local models, as listed in Table 4, and reports a mean correlation of 0.94. Given this small model set and high correlation, please provide uncertainty estimates or qualify the inference that WorldBench measures a distinct dimension.
3 comments
AuthorsOpenReview ↗

Response to Reviewer GDAu (1/2)

Thank you for recognizing WorldBench's breadth and complementary evaluations, and for your suggestions to improve validation and reporting.

W1, Q2: Can you provide a blinded human evaluation on a random sample, with answer accuracy, agreement, and the fraction of ambiguous or corrected items?

To evaluate human answerability, we evaluate a previously unexposed human participant on 100 questions randomly sampled from WorldBench. The participant achieves 77%, compared with 61% for Qwen3.5-35B-A3B Thinking, 46% for GLM-4.6V, and 39% for Gemma4-E4B Thinking.

Many mistakes the human participant makes (the remaining 23%) are avoidable rather than unanswerable. For example, one question presents an online forum with a post with 526 upvotes, and asks what the correct count is after one additional upvote. The participant selects 526, rather than answering with the correct count after one upvote. After providing explanations for the answers, the human participant agrees fully with the answer key.

To control question quality, different authors constructed different benchmark sections, and a final review was done, revising 128 questions from the initial set.

Q1: Please specify the matrix, normalization, and sample used for each metric and whether corrections affect Figure 6. Eq. (3) uses the mean-centered covariance, whereas the cosine-kernel Vendi Score corresponds to the uncentered Gram matrix; please clarify the claimed equivalence.

We thank the reviewer for pointing out the errors in our metrics. The former Appendix Table 3 contained erroneous calculations. We have since removed that table and the associated interpretation, and replaced it with a DINOv3-based comparison in Appendix B.

Figure 6 remains unaffected. For these results, both metrics use the same 1,000-image sample. We L2-normalize the embeddings, subtract their sample mean, and calculate C=XX⊤/NC=XX^\top/N. ER and PR are computed from the same normalized covariance eigenvalues, which implies ER ≥ PR.

We have also corrected the statement that centered-covariance ER is equivalent to ordinary cosine-kernel Vendi in the main manuscript and appendix. Thank you for identifying these errors.

Q3: Please report model comparisons by screening cohort. […] How do these differences vary across cohorts, including those screened with predecessor models? […] Please distinguish uncertainty across benchmark items from variability across repeated generations. Paired comparisons on the same items and repeated runs for close comparisons would help assess whether reported model differences and reasoning-budget trends are stable.

Thank you for highlighting the potential effect screening history can have on model comparisons.

We compare performance on the earlier group of 1442 questions to the final group of 558 questions, finding:

ModelEarlier group, n = 1,442Final group, n = 558
Qwen3.5-35B-A3B Thinking56.2%44.4%
GLM-4.6V45.6%34.4%
InternVL3.5-241B42.9%37.1%

We find that the later group is harder for all models, meaning that construction history affects both difficulty and relative performance. Thus, selecting questions using Instruct and low-reasoning models may have some effect on observed gains, which we aim to present as an overall trend for the configurations evaluated rather than a general claim that more reasoning consistently improves performance across all models.

For GLM-InternVL, a paired bootstrap comparison over all 2000 questions gives an overall difference of 1.20 percentage points [-1.25, 3.70], meaning that for close comparisons we cannot confidently establish their rankings.

AuthorsOpenReview ↗

Response to Reviewer GDAu (2/2)

W2, Q4: Can you report participant-clustered bootstrap intervals for Bradley-Terry score differences between WorldBench and the other leading benchmarks, and describe how comparison pairs were allocated? Please distinguish the observed ranking from any claim of statistically supported superiority.

To estimate uncertainty, we perform 5000 bootstrap resamples of participants, while keeping each selected participant's 30 judgements together and refitting the Bradley-Terry model.

ComparisonScore difference [95% confidence interval]
WorldBench − MMT-Bench+30.0 [−77.1, 135.1]
WorldBench − MMBench+38.3 [−94.0, 171.7]

WorldBench retains the highest point estimate, but has uncertain differences from the closest benchmarks. This supports our interpretation of the human study as evidence that WorldBench is among the most visually diverse benchmarks evaluated here.

Each participant judged 30 distinct pairs out of 36 possible pairs, with randomized left/right placement. Each pair received 7-12 judgements overall. We thank the reviewer for suggesting this participant-level bootstrap analysis, which better communicates uncertainty in the human-diversity comparison.

Q5: Why do Table 2 counts sum to 1,994 rather than 2,000? Please provide domain counts and clarify the weighting of Table 1 averages (for Gemini-3.1-Pro, the unweighted mean of the seven domain scores is 65.0, not the reported 64.0).

Table 2 retained counts from an outdated version of the benchmark, but these have since been corrected. The table reports 1,442 questions across the earlier phases and 558 in the final phase, totaling 2,000. The final domain counts are 245 Living Things, 506 Objects, 435 Scenes, 248 Digital World, 228 Academics, 172 Documents/Charts/Tables, and 166 Agents.

Overall accuracy weights each question equally. For Gemini-3.1-Pro, 1,279/2,000 = 63.95%, which is displayed as 64.0%, rather than the unweighted mean of domain scores. We have renamed the Table 1 column from "Average" to "Overall" and updated the accompanying sentence accordingly.

Q5: Figure 9 uses six local models, as listed in Table 4, and reports a mean correlation of 0.94. Given this small model set and high correlation, please provide uncertainty estimates or qualify the inference that WorldBench measures a distinct dimension.

Thank you for highlighting the uncertainty associated with the small model set used in this analysis. WorldBench's mean Pearson correlation is 0.936, with an estimated 95% bootstrap interval of [0.826, 0.997].

We have qualified the interpretation in the paper as such:

“This analysis is limited to six local models and does not necessarily establish a distinct underlying capability.”

WorldBench's contribution rests on its explicit visual coverage, remaining performance headroom, and transferable hard cases. Here, the correlation analysis simply provides supplementary evidence.

We thank the reviewer for the detailed feedback, which helped us correct reporting errors and strengthen the evidence supporting WorldBench's contribution.

The authors' response has addressed my concerns, I will maintain the positive rating.

Reviewer_QyyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldBench, a multimodal reasoning benchmark designed to achieve high visual diversity. The authors argue that existing vision-language benchmarks primarily emphasize task diversity rather than visual diversity. To address this limitation, they construct a visual taxonomy comprising 2,000 visual concepts across seven domains and collect diverse, context-rich images. Both model-based and human evaluations suggest that WorldBench achieves the highest visual diversity among the benchmarks considered.

The authors also adopt an iterative, manually designed question-generation process to increase the difficulty of multiple-choice questions for frontier MLLMs. An evaluation of 15 MLLMs demonstrates that the benchmark is highly challenging, with the best-performing model achieving only 64% accuracy.

Critical strengths
  1. The authors propose a benchmark emphasizing visual diversity and an interesting annotation approach.

  2. The authors perform an extensive evaluation of frontier models, demonstrating the existing weaknesses in visual reasoning.

Critical weaknesses
  1. Insufficient evidence for the added value of taxonomy-driven visual diversity. The paper's main argument is that existing benchmarks do not focus enough on visual diversity. Their proposed solution is to increase visual diversity by collecting images following a taxonomy. However, the authors do not isolate the effect of increased visual diversity on benchmark difficulty or diagnostic value. Furthermore, it is unclear how much of the models' poor performance comes from high visual diversity or adversarial labeling.
Questions for authors
  1. What role does the taxonomy play in the image collection process? More specifically, given that the authors evaluate visual diversity using model embeddings, would directly selecting images to maximize diversity in the embedding space yield comparable or greater diversity? How does the proposed taxonomy-driven approach compare with random image sampling? Such comparisons would help establish the added value of the taxonomy.

  2. How does image diversity reveal the models' performance gap? This is the main motivation of the paper, but the relationship between visual diversity and model performance is not sufficiently investigated. Could the authors provide empirical evidence that greater visual diversity reveals additional model weaknesses, independently of the adversarial question-generation process?

  3. What systemic weaknesses of existing models does WorldBench reveal that existing benchmarks do not already capture? The authors discuss fine-grained perception errors and ungrounded inferences, but previous multimodal benchmarks have also observed these weaknesses. For example, [1] investigates MLLMs' weaknesses in visual perception. [2] demonstrates that MLLMs can struggle with perceptual questions that are easy for humans. Could the authors clarify whether WorldBench reveals systematic failure modes that existing benchmarks do not adequately capture?

[1] Tong et al. "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs." CVPR, 2024.

[2] Fu et al. "BLINK: Multimodal Large Language Models Can See but Not Perceive." ECCV, 2024.

3 comments

Thank you for asking us to clarify the value of taxonomy-guided construction of a diverse visual benchmark. Our central contribution is visual coverage plus human-answerable, transferable difficulties.

W1, Q1: What role does the taxonomy play in the image collection process? More specifically, given that the authors evaluate visual diversity using model embeddings, would directly selecting images to maximize diversity in the embedding space yield comparable or greater diversity? How does the proposed taxonomy-driven approach compare with random image sampling?

The visual taxonomy provides control over which domains appear in the benchmark. Embedding-based selection, on the other hand, inherits preferences/biases from the encoder's training data. The taxonomy covers seven diverse visual domains, including photographs, interfaces, documents, academic figures, and agent environments.

To demonstrate this, we sample 100 images from the same 2,000-image WorldBench pool using two methods. Random sampling is uniform without replacement. DINOv3 starts with a random image, and then repeatedly greedily chooses the image which has the greatest minimum cosine distance from those already selected. We have added this to Appendix B, and also show a side-by-side comparison in Figure 11.

MeasureRandom samplingDINOv3 selection
DINOv3 effective rank88.997.7
Digital World images12%1%
Documents/Charts/Tables images10%2%

Despite achieving a higher effective rank, DINOv3's selection concentrates on photographs, representing pretraining biases of the vision encoder. Across three repetitions, it selects 1-3 Digital World images, compared with 10-14 under random sampling.

Therefore, a higher embedding diversity does not ensure broad domain coverage. A taxonomy provides explicit control over visual composition that embedding-based selection does not guarantee.

W1, Q2: Could the authors provide empirical evidence that greater visual diversity reveals additional model weaknesses, independently of the adversarial question-generation process?

WorldBench's visual diversity allows us to construct questions grounded across a broad set of domains. Some examples include navigating chat interfaces, comparing slopes across quadrants in a vector-field plot, and real-world manipulation tasks involving selecting a point to open a drawer. These require a broad set of applicable skills that traditional object-oriented photographs do not provide. These additionally illustrate how broader visual coverage can expose challenging questions across a variety of visual settings.

We now want to address the concern that benchmark difficulty arises from adversarial selection against GPT rather than generalizable challenges. We take 100 GPT construction failures, and evaluate three models outside the construction loop:

Outcome across the three comparison familiesQuestions
Missed by at least one96/100
Missed by at least two74/100
Missed by all three40/100

We find that 96/100 questions are missed by at least one comparison family. Additionally, out of 210 incorrect responses from the comparison models, 147 choose the same incorrect option as GPT, showing that failures recur.

Therefore, we ground the contribution of WorldBench in its demonstrated coverage and transferable hard cases.

Q3: What systemic weaknesses of existing models does WorldBench reveal that existing benchmarks do not already capture?

We agree that BLINK and Eyes Wide Shut establish important precedents for perceptual failures. WorldBench's distinguishing/complementary features are broad visual coverage, transferable hard cases, and substantial headroom for further progress.

We analyze 100 questions that GPT failed during construction. We characterize the failure mode of each problem, and find that they contain 43 perception, 41 spatial, 5 temporal, and 11 knowledge questions, showing that the challenge of WorldBench extends beyond fine-grained perception. Across the benchmark, the strongest model achieves only 64% accuracy, showing that there's significant room for future improvement.

We've revised the related work to describe visual diversity as WorldBench's organizing principle rather than one specific task or capability. WorldBench is not intended to discover new failure categories, but rather assess whether known limitations persist across a broad variety of visual contexts.

We would like to thank the reviewer for motivating the analysis of embedding models and sharpening the contribution of WorldBench.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to clarify the role of the visual taxonomy and provide further evidence of the benchmark's generalizable difficulty. However, some of my central concerns remain unresolved.

W1, Q1: The comparison with DINOv3-based selection is informative. However, the comparison is conducted by sampling from the existing WorldBench image pool, which has already been constructed using the taxonomy. Therefore, while the experiment demonstrates the limitations of embedding-based selection within this pool, it does not fully establish the advantage of taxonomy-driven collection over alternative image collection strategies.

W1, Q2: The examples from different visual domains are compelling, but a more systematic analysis would be helpful. For instance, comparing model performance or failure modes across different visual domains, or examining how benchmark coverage affects the diversity of observed failures, could provide stronger evidence for the paper's central motivation.

Q3: I appreciate the clarification that WorldBench is not intended to identify entirely new failure categories, but rather to assess whether known limitations persist across a broader range of visual contexts. This is a reasonable and more precise positioning of the contribution.

The authors have partially addressed my concerns, but some of the central questions remain unanswered. Therefore, I keep my score unchanged.

Thank you for the thoughtful follow-up and for recognizing WorldBench's value in evaluating known limitations across broader contexts.

W1, Q1: The comparison is conducted by sampling from the existing WorldBench image pool, which has already been constructed using the taxonomy.

We agree only using the WorldBench image pool restricted the initial experiment. Therefore, we've repeated the comparison with 10,000 deduplicated DataComp images (a web-scale, uncurated dataset), with no matches within the WorldBench image set. Across 10 repetitions of selecting 1,000 images, we find:

MeasureRandomDINOv3
Effective rank430.6523.5
Participation ratio278.0404.2
Real World images66.4%81.1%
Digital World images33.6%18.9%

Again, we see that a higher encoder-diversity score accompanies narrower coverage of the Digital World, now outside the initial WorldBench pool. This analysis supports the role of the taxonomy in providing explicit coverage over visual composition, rather than assuming an encoder-based score ensures coverage.

We thank the reviewer for prompting this external check.

W1, Q2: For instance, comparing model performance or failure modes across different visual domains, or examining how benchmark coverage affects the diversity of observed failures, could provide stronger evidence for the paper's central motivation.

Following your suggestion, we sampled 50 incorrect answers from each one of WorldBench's seven visual domains, using predictions from Qwen3.5-35B-A3B Thinking as the reference. We classify each failure into four categories: perception, spatial, temporal, and knowledge.

Visual domainPerceptionSpatialTemporalKnowledge
Living Things232511
Objects301505
Scenes341204
Digital World202226
Academics291308
Documents/Charts/Tables261518
Agents182516

The central challenge differs across these sampled errors. Perception accounts for 68% of Scene errors but only 36% of Agent errors; spatial understanding accounts for 50% of Agent errors but 24% of Scene errors. Additionally, knowledge becomes more of a requirement for Academics and Documents/Charts/Tables.

Additionally, looking at the performance of models on WorldBench across visual domains reveal strengths/weaknesses that an averaged score hides. The following values are from Table 1:

ModelObjectsDigital WorldDocuments/Charts/Tables
Gemini-3.1-Pro60.7%71.4%73.8%
Qwen3.5-VL-Plus-Thinking62.8%58.9%60.2%
GPT-5.4-Thinking (high)53.8%67.3%64.5%

We see that Qwen-Plus Thinking leads GPT high on Objects, but trails on Digital World. Similarly, Gemini-Pro leads Qwen-Plus in Digital World, but trails on Objects.

These differences show the value of evaluating models across varied settings. If we only evaluate on a single domain, the findings we draw would only narrowly apply to that specific domain and not generally. WorldBench extends coverage across a variety of visual domains, making these differences clear.

Our central contribution is a benchmark that combines broad visual coverage with recurring hard cases.

We thank the reviewer for helping us make this contribution and its supporting evidence clearer. We hope the additional results can address your concerns, if you have any remaining concerns, please let us know!

Reviewer_NhjUOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WorldBench, a multimodal benchmark emphasizing visual diversity, comprising 2,000 image-based multiple-choice questions across seven domains. The authors construct a taxonomy to guide image collection and iteratively design questions that challenge frontier models. Quantitative metrics and human comparisons suggest broad visual diversity relative to existing benchmarks. Evaluation of 15 multimodal models yields a best accuracy of 64.0%, highlighting persistent limitations in fine-grained visual understanding and showing that increased reasoning budgets do not consistently improve performance.

Critical strengths
  1. Taxonomy-guided image collection, model-assisted question refinement, and multiple rounds of human review support benchmark breadth and quality.
  2. Experiments cover 15 models, multiple diversity metrics, human comparisons, and reasoning-budget analysis.
Critical weaknesses
  1. The claim that questions are intuitive for humans lacks a quantitative human accuracy baseline, while diversity judgments involve only 12 participants.
Questions for authors
  1. What is human accuracy on WorldBench, and how robust are the diversity judgments given the small pool of 12 participants?
1 comment

Thank you for recognizing WorldBench's visual breadth, taxonomy-guided construction, and evaluation across model configurations.

W1, Q1: The claim that questions are intuitive for humans lacks a quantitative human accuracy baseline, while diversity judgments involve only 12 participants.

We evaluate a previously unexposed human participant on 100 questions randomly sampled from WorldBench:

Participant/modelAccuracy
Human participant77%
Qwen3.5-35B-A3B Thinking61%
GLM-4.6V46%
Gemma4-E4B Thinking39%

We additionally find that many mistakes the human participant makes are avoidable rather than evidence of unanswerable questions. For example, one question presents an online forum with a post with 526 upvotes, and asks what the correct count is after one additional upvote. The participant selects 526, rather than answering with the correct count after one upvote, 527. After providing explanations for the answers, the human participant agrees fully with the answer key.

This is preliminary evidence that despite significant model difficulty, questions remain human-answerable.

W1, Q1: What is human accuracy on WorldBench, and how robust are the diversity judgments given the small pool of 12 participants?

Our human-diversity study has 360 judgements across 12 participants. WorldBench retains the highest estimated diversity score when we account for differences between participants. We use participant-level bootstrapping, repeating the analysis 5000 times, sampling 12 participants with replacement while keeping their 30 judgments together. Then, we recompute diversity scores:

ComparisonScore difference [95% confidence interval]
WorldBench − MMT-Bench+30.0 [−77.1, 135.1]
WorldBench − MMBench+38.3 [−94.0, 171.7]

We find that WorldBench retains the highest point estimate. However, differences from the closest benchmarks remain uncertain. This supports our description of WorldBench as among the most visually diverse benchmarks evaluated here.

We thank the reviewer for motivating the human evaluation and the statistical uncertainty analysis.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response and Summary of Revisions

We would like to begin by thanking the reviewers for their thoughtful feedback. WorldBench's two intended, central contributions are diverse visual coverage and difficult questions that generalize across model families. The additional analyses we have performed strengthen these contributions in three ways:

  1. Human Answerability: A previously unexposed participant achieves 77% accuracy on 100 questions randomly sampled from WorldBench, compared with 61% for Qwen3.5-35B-A3B Thinking on the same questions. This supports the claim that questions are generally answerable by humans, despite still remaining challenging for models.

  2. Transferability of difficult questions: We evaluate 100 questions that GPT failed during construction using models from three other model families, and find that 96/100 are also missed by at least one other family. This is evidence that failure modes are transferable, and that difficulty isn't specific to the construction model.

  3. Taxonomy versus embedding-based diversity. To demonstrate the effectiveness of a visual taxonomy, we compare random sampling of 100 images from WorldBench to DINOv3-based selection. We find that while effective rank, an embedding-based diversity metric, increases, domain representation decreases. Digital World coverage falls from 12 images to 1, and Document/Charts/Tables from 10 to 2. Thus, a higher embedding diversity score doesn't imply broader visual coverage. A taxonomy allows explicit control over what visual domains are represented in the benchmark. The results of this experiment and a side-by-side comparison are added to Appendix B.

We have additionally fixed construction counts, corrected the ER-Vendi equivalence, replaced the erroneous appendix table, and qualified our correlation interpretation.

We thank the reviewers for helping us strengthen WorldBench as a benchmark that provides broad visual coverage, maintains transferable difficulty, and remains challenging for frontier models, leaving substantial room to measure future progress.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.