acceptodds
Under review as a conference paper at ICLR 2027

HumanSightBench: A Benchmark for Human Visual Perception in MLLMs

Abstract

Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation. Our dataset is available https://huggingface.co/datasets/Susan0803/visual-perception-benchmark-anonhere.

Primary area:
datasets and benchmarks
Keywords:
VLM,benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pEkUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HumanSightBench, a benchmark for evaluating human-easy visual perception in MLLMs. It contains 2,876 questions across 7 categories and 21 perceptual abilities, with strict filtering for clarity, unique answers, visual grounding, human solvability, and text-only shortcuts. Experiments on eight MLLMs reveal substantial gaps in basic visual abilities, especially counting, grounding, OCR, and spatial perception.

Critical strengths
  1. Clear benchmark motivation. The paper focuses on human-easy visual perception rather than entangled end-to-end multimodal reasoning.
  2. Rigorous item validation. Each question is filtered for clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcuts.
  3. Fine-grained diagnosis. The 21 ability groups reveal specific bottlenecks, such as counting, grounding, OCR, and spatial perception.
Critical weaknesses
  1. Substantial overlap with prior benchmarks. The central finding that current MLLMs still struggle on human-easy basic visual tasks closely resembles BabyVision, with considerable overlap in fine-grained discrimination, visual tracking, spatial perception, and orientation-related abilities. The paper should clarify what qualitatively new diagnostic insight HumanSightBench provides beyond broader coverage and stricter validation.
  2. Category imbalance. OCR and Structured Visual Perception account for most of the benchmark, which may bias the overall score toward these abilities.
  3. Perception is not fully isolated. Errors may still arise from task interpretation, reasoning, output formatting, or judge mistakes rather than visual perception itself.
  4. Model-based screening may affect representativeness. Removing saturated items makes the benchmark more diagnostic, but the retained set may not reflect the natural distribution of everyday visual tasks.
Questions for authors

see above

4 comments

Thank you for the careful review. We have clarified the benchmark's contribution and added human-solving measurements, a judge audit and alternative score summaries. We address the four concerns below.

W1. Novelty and distinction from prior benchmarks

Our response. HumanSightBench and BabyVision share an interest in basic visual skills. Our added value is combining direct-answer production, query-directed reading and multilingual tasks within one diagnostic collection, rather than proposing new visual primitives.

Evidence. Section 2.2/Table 1 describes 2,876 questions, including a 300-item multilingual subset, specified-path transcription and visible text revisions. Table 2 gives the following task-level results.

ModelOCR pass@3 (%)Text Content Recognition pass@3 (%)
Qwen-3.8-Max62.742.3
GPT-6-Astra58.045.3

Interpretation. The configuration with stronger OCR overall performs worse on Text Content Recognition. This identifies a specific weakness hidden by the broad-category ranking, giving a more concrete target for model improvement than the statement that visual errors persist.

Scope and limitation. These are complementary tasks and descriptive profiles under the reported input configurations, not a controlled model-only comparison or a claim of strict superiority over prior benchmarks.

W2. Category imbalance

Our response. Since OCR and Structured Visual Perception account for 70.9% of the questions, we now report several weighting schemes rather than relying on micro pass@3 alone.

Evidence. Table 3 reports the following summaries; Table 4 supplies all group sizes.

ModelMicro pass@3 (%)L1 macro (%)L2 macro (%)
GPT-5.6-Sol37.235.136.5
GPT-6-Astra65.664.967.2
GLM-5.3-Flash46.243.444.7
Seed-Evolving58.456.157.4
Qwen-3.8-Max63.960.164.9
GLM-4.6V24.021.525.2
Kimi K2.642.236.438.9
Qwen3VL-235B-Thinking32.426.230.0

Interpretation. Micro, L1 macro and L2 macro weight items, categories and ability groups differently. The eight configurations retain the same observed ordering, although their scores change. This makes weighting sensitivity visible without discarding retained questions.

Scope and limitation. Macro scores reweight existing reported group results; they are not new evaluations or a correction for selection bias. Ten-item groups remain uncertain: one additional success changes their score by ten percentage points.

W3. Perception attribution and judge effects

Our response. We distinguish measured human performance from judging consistency, rather than treating construction criteria as proof that every model error is perceptual.

Evidence. The new human study covers all 21 task groups. Appendix D/Table 6 reports frozen-reference text-answer agreement.

Question subsetQuestionsMatches / responsesAgreement (%)
All sampled questions63201/25279.76
Non-point questions61193/24479.10

The separate judge audit covers 126 model responses from 42 questions. Appendix B.4/Table 5 compares both scoring outputs with the same first-pass human labels.

Scoring outputMatches / decidable labelsAgreement (%)
Text-judge decision116/12394.31
Final score == 1111/12390.24

Interpretation. The human study supplies actual human answers; the judge audit examines how automatic decisions agree with human labels. Five point-and-count responses explain the difference between the judge rates because pass@3 uses the final decision, not textual matching alone. Section 4.4 discusses interpretation, reasoning and output conventions as additional error sources.

Scope and limitation. The coordinator confirms answer isolation and a 60-second cutoff; timing was not independently audited. Eighteen reported times exceed 60 seconds and two are missing; the discrepancy remains unresolved. Two point-and-count questions use count-only comparisons, so the human result is not full-task accuracy or a controlled human-model gap. The judge rates exclude three undecidable labels. Audited inputs contain reference/response text, not images or question content; disagreements remain unadjudicated, so agreement is not judge accuracy.

W4. Model-based selection and representativeness

Our response. We position HumanSightBench as a diagnostic challenge set, not a random sample of everyday visual tasks.

Evidence. Sections 3.2-3.3 and Appendix B.2 describe iterative, batch-dependent quality review and model-informed difficulty checks, rather than one uniform screening pass.

Interpretation. This supports comparing configurations on the same retained tasks and locating weaknesses for follow-up evaluation. Quality review and selecting difficult cases serve different purposes; difficulty alone does not establish item validity.

Scope and limitation. Complete candidate-to-retained records and the screening-model roster are unavailable. We cannot quantify all selection effects, and neither macro averaging nor the sampled human study removes them.

Closing

These revisions make the benchmark's added diagnostic value and the evidence behind its scores more explicit, while keeping the conclusions within what the results support.

AuthorsOpenReview ↗

Reminder for Rebuttal

We thank the reviewer again for their time and effort. As the discussion period approaches its end, we would be grateful if the reviewer could review our responses to the newly added comments. Please let us know whether our replies have adequately addressed your remaining concerns; if further clarification is needed, we are happy to provide it.

Thank you for the responses. While some concerns have been partially addressed, I remain significantly concerned about the benchmark's novelty and the disproportionate representation of OCR tasks. Therefore, I maintain my original rating of 2

Reviewer_4azBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces HumanSightBench, a 2,876-item benchmark designed to evaluate human visual perception abilities in multimodal LLMs. The dataset spans 7 top-level categories and 21 subcategories and applies five inclusion gates before optional model screening removes saturated items. Evaluating 8 representative MLLMs with pass@3 judged by an LLM-as-judge reveals some known gaps in Counting, Text Content Recognition, Visual Reasoning, and Grounding, with best overall pass@3 of 65.6%.

Critical strengths
  • Clear articulation of evaluation boundaries and known confounds, e.g., judge limitations, resizing differences.
  • The collection and validation workflow is well-structured, with explicit criteria and role separation between annotator groups A and B.
  • Tables and figures communicate both overall trends and ability-level failures, including illustrative qualitative cases that match the reported weaknesses.
Critical weaknesses
  1. Limited novelty because the task designs overlap heavily with related work, differentiation is weak, and the paper offers insufficient insight or methodological contribution. Almost all tasks in HumanSightBench are conventional computer vision problems, like counting, OCR, and similar abilities have already been evaluated extensively in BLINK, BabyVision, and PerceptionBench. However, it does not distinguish its protocol and coverage from these works, for example, PerceptionBench's atomic ability verification, HoloCount for counting, CC-OCRV2 for OCR grounding. The proposed inclusion criteria also overlap with existing human-validation and visual-grounding protocols. Last but not least, the paper does not sufficiently establish new insights or methodological advances.

  2. Although the benchmark is positioned as evaluating human-level visual perception, it does not report an independent human accuracy baseline, nor systematically investigate human perceptual behavior.

  3. Insufficient human annotation protocol and verification analysis. It lacks important statistics on rates of human annotator&verifier agreement, how candidates are rejected, reviewer allocation. Moreover, the DeepSeek-V3.2-based answer judge is not validated against human judgments. Also the full judging prompt is not provided.

  4. How was the LLM-as-judge validated? The benchmark heavily relie on a single LLM-as-judge for correctness without a reported human audit or cross-judge validation risks measurement bias, especially for short answers and OCR variants.

  5. Unbalanced benchmark coverage. OCR and Structured Visual Perception account for 70.9% of the benchmark, while several ability groups contain only 10 samples. Consequently, the overall score may not represent a broad visual perception capability, and small categories results are fragile.

  6. The resized vs non-resized image conditions differ across models and even human annotators, and are not controlled. It complicats comparisons and causal attribution of failures to perception vs input interface.

Questions for authors

Please refer to Critical Weaknesses part.

5 comments

The paper introduces HumanSightBench. I remain concerned about limited novelty and insufficient differentiation from prior benchmarks, the lack of an independent human accuracy baseline and annotation/verification statistics, an unvalidated LLM-as-judge, unbalanced category coverage, and uncontrolled resized/non-resized conditions that complicate comparisons. As its' 17 Nov now, no author rebuttal has been posted, so these concerns remain unaddressed. I therefore keep my original score.

Thank you for the detailed review. We have strengthened the benchmark comparison and added human-solving measurements, a judge audit and alternative score summaries. The results below address the six concerns separately.

W1. Contribution and diagnostic insight

Our response. HumanSightBench combines direct transcription/list/count outputs, query-directed reading and a 300-item multilingual subset within 2,876 questions. Specified-path reading and visible revisions test adherence to requested reading conditions. Table 1 contrasts this combination with BLINK, BabyVision, PerceptionBench, HoloCount and CC-OCR V2.

Evidence. Table 2 provides a task-level ranking reversal and the following performance ranges across the eight evaluated configurations.

ModelOCR pass@3 (%)Text Content Recognition pass@3 (%)
Qwen-3.8-Max62.742.3
GPT-6-Astra58.045.3
MetricMinimum (%)Maximum (%)
Overall pass@324.065.6
Counting pass@34.627.5

Interpretation. A broad OCR advantage can coexist with a weaker reading subtask, while counting remains difficult even for configurations with much higher overall scores. This identifies concrete improvement targets that aggregate rankings obscure.

Scope and limitation. The contribution is complementary coverage and diagnosis, not new visual primitives. These profiles describe the retained tasks under their reported input settings, not isolated causal mechanisms.

W2. Independent human baseline

Our response. We added actual human solving, distinct from the audit of model answers. Three questions per L2 group were locked before collection without using model or human outcomes.

Evidence. Rotating contributors supplied 252 answers covering all 21 task groups. Appendix D/Table 6 reports frozen-reference text-answer agreement.

Question subsetQuestionsMatches / responsesAgreement (%)
All sampled questions63201/25279.76
Non-point questions61193/24479.10

Interpretation. This supplies measured human-answer evidence across the taxonomy. The similar non-point result shows that the overall agreement level is not solely due to the count-only comparisons.

Scope and limitation. The coordinator confirmed answer isolation and a 60-second cutoff; timing was not independently audited. Eighteen reported times exceed 60 seconds and two are missing; the discrepancy remains unresolved. All rows remain, including one prior-exposure note. Two point-and-count questions contribute counts only, with coordinates missing from five of eight answers. This is not full-task accuracy, universal human success or a controlled human-model gap.

W3. Construction and human verification

Our response. We now distinguish construction-time quality review, model-informed difficulty checks and the new human-solving study.

Evidence. Section 3.3 and Appendices B.1-B.2 document revisions to ambiguous targets, occlusion, counted units, ordering and output formats. The construction and review roles are described separately from the new study in Appendix D.

Interpretation. Quality review concerns whether questions and references are well-defined; difficulty checks identify challenging tasks. Neither activity is itself an independent human-accuracy measurement.

Scope and limitation. Historical records do not establish uniform three-solver, 60-second validation with unanimous correctness for every item. Initial annotator-verifier agreement and complete review/attrition records are unavailable. The new study is not retrospective validation of all released questions.

W4. Judge validation and reproducibility

Our response. We audited both the textual judge and the final scoring decision used by pass@3, rather than treating them as interchangeable.

Evidence. The audit covers 42 questions across three models, with three responses from one assigned model per question (126 total). Appendix B.4/Table 5 compares them against decidable first-pass human labels.

Scoring outputMatches / decidable labelsAgreement (%)
Text-judge decision116/12394.31
Final score == 1111/12390.24

The five component-level differences have the following recorded values (Appendix B.4).

ResponsesHuman labelText-judge labelLocalization F1Final score
51100.5

Interpretation. Pass@3 requires final score == 1, so these responses fail the final criterion despite textual agreement. Reporting both rates exposes this scoring distinction. Appendix B also provides the full prompt, input format and recorded settings.

Scope and limitation. Three undecidable labels are excluded; disagreements and localization scores remain unadjudicated. Audited judge inputs contain reference/response text, not images or question content. Configured and returned model identifiers differ, and backend identity and the full evaluator remain unverified. These are label-agreement measurements, not judge accuracy or full backend reproduction.

W5. Imbalance and small groups

Our response. We retain the full task inventory and expose the effect of weighting, rather than reducing large groups without adding evidence to sparse ones.

Evidence. Table 3 reports micro pass@3 and equal-weight L1/L2 macro summaries; Table 4 lists all group sizes.

ModelMicro pass@3 (%)L1 macro (%)L2 macro (%)
GPT-5.6-Sol37.235.136.5
GPT-6-Astra65.664.967.2
GLM-5.3-Flash46.243.444.7
Seed-Evolving58.456.157.4
Qwen-3.8-Max63.960.164.9
GLM-4.6V24.021.525.2
Kimi K2.642.236.438.9
Qwen3VL-235B-Thinking32.426.230.0

Interpretation. The observed ordering remains unchanged across these three summaries, although score magnitudes differ. Readers can inspect item-, category- and ability-weighted performance instead of relying on one aggregate.

Scope and limitation. Macro averages reweight existing group scores, not add data. This is not general ranking robustness: ten-item groups remain coarse, and their rankings are descriptive rather than population-level estimates.

W6. Resizing and input conditions

Our response. We describe the comparisons as model-input configurations, not a controlled resizing experiment.

Evidence. Section 4.1 documents Resize/No Resize; Appendix D records that humans could zoom on original images. No Resize refers to the image supplied to the interface, not a guarantee against internal preprocessing.

Interpretation. The reported results locate weaknesses under these settings, which is useful for diagnosis but does not separate model and interface effects.

Scope and limitation. We have not established a controlled resolution effect or a matched-viewing human-model gap, and do not attribute all failures solely to perception.

Closing

The revised manuscript now provides direct human-answer evidence, a scoring-component audit and explicit weighting comparisons, alongside a clearer account of the benchmark's complementary diagnostic contribution.

Reviewer_CFRvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing evaluation benchmarks often conflate visual perception with other cognitive abilities, making it difficult to pinpoint a model's true visual bottlenecks. To address this, we introduce HumanSightBench—a benchmark specifically designed to assess the foundational visual perception capabilities of Multimodal Large Language Models (MLLMs). Guided by the core principle of being "effortless for humans yet highly challenging for models," the benchmark employs a rigorous five-tier filtering process: ensuring clear question prompts, unique correct answers, explicit grounding in visual evidence, a 100% accuracy rate among human participants under time constraints, and the elimination of "text-only" shortcuts. The resulting dataset comprises 2,876 short-answer questions spanning 7 primary categories—such as OCR, spatial understanding, structural perception, and visual tracking—and 21 distinct capability dimensions. Evaluations of eight state-of-the-art models, including GPT-6-Astra and Qwen-3.8-Max, reveal that even the top-performing closed-source model achieves a pass@3 score of only 65.6%. These findings not only expose fundamental, pervasive perceptual flaws in current models but also provide a highly diagnostic tool for measuring the core "visual acuity" of multimodal systems.

Critical strengths
  1. A rigorous screening mechanism enhances the validity of the test items: By applying filters—such as ensuring prompt clarity, visual feasibility, and the elimination of text-based "shortcuts"—and requiring that an average person can answer correctly with 100% consistency within 60 seconds, the framework guarantees that any errors stem solely from fundamental flaws in visual perception. This achieves the objective of being "trivial for humans, yet extremely challenging for models."
  2. A fine-grained classification system with high diagnostic value: The evaluation is broken down into 7 major categories and 21 specific perceptual sub-capabilities, effectively revealing the distinct weaknesses and issues inherent in different models.
  3. Clear, foundational analysis: By examining the models' chains of thought, the framework identifies typical mechanisms behind "perceptual collapse," offering valuable guidance for improving multimodal data and model architectures.
Critical weaknesses
  1. Highly imbalanced category distribution, with limited samples in certain subcategories leading to significant volatility: OCR and structured visual perception together account for 70.9% of the entire dataset. Conversely, subcategories under semantic recognition—such as "basic recognition," "actions and interactions," and "semantic relationships"—contain only 10 questions each. In a group of just 10 questions, the outcome of a single item can cause a 10% fluctuation in the score for that specific capability, indicating a lack of statistical robustness.
  2. Asymmetry in observational input conditions between humans and models: Human evaluators are permitted to zoom in on specific areas and pan across the image to examine details at the original resolution. In contrast, model evaluation is conducted using either a fixed, resized full image or the original, unscaled image; models are not provided with interactive tools for dynamic, localized zooming. Consequently, some instances of model failure may be attributable to resolution constraints rather than inherent limitations in the models' own capabilities.
Questions for authors
  1. Could we consider a detailed classification and categorization of the error types across different models—moving beyond mere illustrative examples—to provide concrete guidance for model iteration?
  2. Some current multimodal models incorporate tools to evaluate performance on certain benchmarks (for instance, the "BabyVision + CI" setup is now used by some models as a reporting metric). It would be worth investigating whether providing a model with access to a richer suite of tools leads to performance improvements on such benchmarks.
  3. I am curious whether varying the "thinking effort" of a single model results in performance differences on these benchmarks.
2 comments

Thank you for these suggestions. We have added the human study and clarified the evaluation settings.

  1. Small categories (W1). We agree that scores from small groups are unstable. Table 4 now gives every group denominator, and Table 3 adds equal-weight L1 and L2 macro averages, computed from reported group scores, alongside micro pass@3. For a ten-item group, one more solved question changes the score by ten percentage points. We therefore treat small-group rankings as descriptive, not stable population rankings (Section 3.4).
  2. Viewing conditions (W2). The new 63-question human study used original images with zoom. The study coordinator confirmed answer isolation and a 60-second cutoff. Of 252 answers, 201/252 (79.76%) matched the frozen reference text (Appendix D/Table 6). Two point-and-count questions were checked on counts only, so this is not full-task accuracy. We kept all responses, including 18 reported times over 60 seconds and two missing times; this discrepancy is unresolved and timing was not independently audited. Model inputs also differ: No Resize describes the image sent to the interface, not a guarantee against internal preprocessing. This is neither a matched human-model comparison nor a resizing ablation (Section 4.1).
  3. Error analysis (Q1). Table 2 compares all 21 task groups across eight configurations. Counting averages 16.0% pass@3 and Text Content Recognition 31.6%; Section 4.3/Figure 4 examines the six weakest groups. This gives a systematic account of which tasks are difficult. Figure 5 illustrates possible failure patterns, but we have not annotated error causes across the full dataset (Section 4.4).
  4. Tool-assisted evaluation (Q2). We have not run a matched tool-enabled versus tool-disabled comparison. Table 2 should therefore not be read as evidence for the benefit of tool use. Tool access and call budgets would need to be specified in a separate comparison.
  5. Thinking effort (Q3). Pass@3 uses repeated attempts, not three different thinking levels (Appendix B.3). We have not run a within-model thinking-effort sweep. We therefore do not claim that more thinking helps or hurts. The reasoning traces in Section 4.4 are illustrative cases, not an effort ablation.

Thank you for the additional clarifications. I will keep the current score as is; however, I look forward to seeing further experiments addressing the three points I raised: first, a categorized analysis of the errors made by each model; second, an evaluation of how models perform when integrated with simple external tools (such as Python code); and third, an assessment of how single models perform across different levels of "thinking effort" (e.g., medium, high, and xhigh).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.