- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HumanSightBench, a benchmark for evaluating human-easy visual perception in MLLMs. It contains 2,876 questions across 7 categories and 21 perceptual abilities, with strict filtering for clarity, unique answers, visual grounding, human solvability, and text-only shortcuts. Experiments on eight MLLMs reveal substantial gaps in basic visual abilities, especially counting, grounding, OCR, and spatial perception.
Critical strengths
- Clear benchmark motivation. The paper focuses on human-easy visual perception rather than entangled end-to-end multimodal reasoning.
- Rigorous item validation. Each question is filtered for clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcuts.
- Fine-grained diagnosis. The 21 ability groups reveal specific bottlenecks, such as counting, grounding, OCR, and spatial perception.
Critical weaknesses
- Substantial overlap with prior benchmarks. The central finding that current MLLMs still struggle on human-easy basic visual tasks closely resembles BabyVision, with considerable overlap in fine-grained discrimination, visual tracking, spatial perception, and orientation-related abilities. The paper should clarify what qualitatively new diagnostic insight HumanSightBench provides beyond broader coverage and stricter validation.
- Category imbalance. OCR and Structured Visual Perception account for most of the benchmark, which may bias the overall score toward these abilities.
- Perception is not fully isolated. Errors may still arise from task interpretation, reasoning, output formatting, or judge mistakes rather than visual perception itself.
- Model-based screening may affect representativeness. Removing saturated items makes the benchmark more diagnostic, but the retained set may not reflect the natural distribution of everyday visual tasks.
Questions for authors
see above
4 comments
W3. Perception attribution and judge effects
Our response. We distinguish measured human performance from judging consistency, rather than treating construction criteria as proof that every model error is perceptual.
Evidence. The new human study covers all 21 task groups. Appendix D/Table 6 reports frozen-reference text-answer agreement.
| Question subset | Questions | Matches / responses | Agreement (%) |
|---|---|---|---|
| All sampled questions | 63 | 201/252 | 79.76 |
| Non-point questions | 61 | 193/244 | 79.10 |
The separate judge audit covers 126 model responses from 42 questions. Appendix B.4/Table 5 compares both scoring outputs with the same first-pass human labels.
| Scoring output | Matches / decidable labels | Agreement (%) |
|---|---|---|
| Text-judge decision | 116/123 | 94.31 |
| Final score == 1 | 111/123 | 90.24 |
Interpretation. The human study supplies actual human answers; the judge audit examines how automatic decisions agree with human labels. Five point-and-count responses explain the difference between the judge rates because pass@3 uses the final decision, not textual matching alone. Section 4.4 discusses interpretation, reasoning and output conventions as additional error sources.
Scope and limitation. The coordinator confirms answer isolation and a 60-second cutoff; timing was not independently audited. Eighteen reported times exceed 60 seconds and two are missing; the discrepancy remains unresolved. Two point-and-count questions use count-only comparisons, so the human result is not full-task accuracy or a controlled human-model gap. The judge rates exclude three undecidable labels. Audited inputs contain reference/response text, not images or question content; disagreements remain unadjudicated, so agreement is not judge accuracy.
W4. Model-based selection and representativeness
Our response. We position HumanSightBench as a diagnostic challenge set, not a random sample of everyday visual tasks.
Evidence. Sections 3.2-3.3 and Appendix B.2 describe iterative, batch-dependent quality review and model-informed difficulty checks, rather than one uniform screening pass.
Interpretation. This supports comparing configurations on the same retained tasks and locating weaknesses for follow-up evaluation. Quality review and selecting difficult cases serve different purposes; difficulty alone does not establish item validity.
Scope and limitation. Complete candidate-to-retained records and the screening-model roster are unavailable. We cannot quantify all selection effects, and neither macro averaging nor the sampled human study removes them.
Closing
These revisions make the benchmark's added diagnostic value and the evidence behind its scores more explicit, while keeping the conclusions within what the results support.
Reminder for Rebuttal
We thank the reviewer again for their time and effort. As the discussion period approaches its end, we would be grateful if the reviewer could review our responses to the newly added comments. Please let us know whether our replies have adequately addressed your remaining concerns; if further clarification is needed, we are happy to provide it.
Thank you for the responses. While some concerns have been partially addressed, I remain significantly concerned about the benchmark's novelty and the disproportionate representation of OCR tasks. Therefore, I maintain my original rating of 2
Thank you for the careful review. We have clarified the benchmark's contribution and added human-solving measurements, a judge audit and alternative score summaries. We address the four concerns below.
W1. Novelty and distinction from prior benchmarks
Our response. HumanSightBench and BabyVision share an interest in basic visual skills. Our added value is combining direct-answer production, query-directed reading and multilingual tasks within one diagnostic collection, rather than proposing new visual primitives.
Evidence. Section 2.2/Table 1 describes 2,876 questions, including a 300-item multilingual subset, specified-path transcription and visible text revisions. Table 2 gives the following task-level results.
Interpretation. The configuration with stronger OCR overall performs worse on Text Content Recognition. This identifies a specific weakness hidden by the broad-category ranking, giving a more concrete target for model improvement than the statement that visual errors persist.
Scope and limitation. These are complementary tasks and descriptive profiles under the reported input configurations, not a controlled model-only comparison or a claim of strict superiority over prior benchmarks.
W2. Category imbalance
Our response. Since OCR and Structured Visual Perception account for 70.9% of the questions, we now report several weighting schemes rather than relying on micro pass@3 alone.
Evidence. Table 3 reports the following summaries; Table 4 supplies all group sizes.
Interpretation. Micro, L1 macro and L2 macro weight items, categories and ability groups differently. The eight configurations retain the same observed ordering, although their scores change. This makes weighting sensitivity visible without discarding retained questions.
Scope and limitation. Macro scores reweight existing reported group results; they are not new evaluations or a correction for selection bias. Ten-item groups remain uncertain: one additional success changes their score by ten percentage points.