- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HSS is a benchmark for "intuitive visual reasoning": inferring what is implied but not depicted in an image or video—past causes, future trajectories, affordances, social roles and intent, occluded structure. The benchmark contains 522 human-authored tasks (288 image, 234 video) organized into 4 domains and 11 subdomains, produced by a highly lossy curation pipeline (3,466 authored → 522 released, 15.1% acceptance) with three independent review rounds requiring unanimous consensus. Answers are free-form and graded against atomic rubrics by an LLM judge, with an item counting as solved only if every rubric criterion is met. Across 24 frontier MLLMs, the best model (GPT-6-astra, max reasoning effort) reaches 53.6% vs. 93.1% human accuracy; a blind control drops the top model to 6.6%; social understanding is the weakest domain (23.9% avg.); failures concentrate in perception (53%) and latent inference (41%) rather than deliberate reasoning (5%); agentic harnesses (Claude Code, Codex) add +5 to +20 points but do not close the gap; and higher reasoning effort sometimes hurts (e.g., −14 points on retrodiction for GPT-6-astra).
Critical strengths
- A genuinely under-served evaluation axis, cleanly delimited. The explicit-perception (V*, BLINK) vs. implicit-inference distinction, and the positioning against MMMU-style "images as givens for academic problems," is accurate and useful. Complementing BabyVision ("perceive what is explicit") with "infer what is implicit" is a crisp conceptual contribution.
- Curation rigor is the strongest part of the paper. A 15.1% end-to-end acceptance rate with documented attrition at each of three independent unanimous-consensus gates, plus explicit exclusion principles (no pure perception, no specialized expertise), is well above typical benchmark practice and directly targets the subjectivity risk inherent to "intuition" tasks.
- The evaluation is unusually well-audited. Blind control (6.6% residual, with a concrete analysis of why—two-way questions + invented detail), three-vendor judge cross-grading (r ≥ 0.98, κ ≥ 0.92, >95% agreement, self-preference shift of only −0.44), a 20-person human baseline with confidence tracking, and bootstrap CIs on all main numbers. This is the standard other benchmark papers should meet.
- The failure analysis yields non-obvious, actionable findings. That 94% of failures are perceptual/latent rather than reasoning failures; that on multi-model-failed tasks a median of 80% fail for the same cause; that agentic re-inspection fixes missed cues but cannot fix 2D-overlap-read-as-3D errors (102→101); and that failed agent runs take more steps than successful ones—each of these is a concrete signal for where modeling effort should go.
Critical weaknesses
- Subdomain-level statistics are thin. With 522 tasks across 11 subdomains, many subdomains hold roughly 40–50 items; the subdomain-level claims (e.g., per-subdomain failure signatures like "extrapolation fails on facing direction 42%", reasoning-effort swings of −14 to +16 points) rest on dozens of items and a handful of models, yet are presented without per-subdomain uncertainty. The item-level ±3–4 bootstrap intervals on the main table are good practice, but the more interesting subdomain narratives need the same treatment.
- The human baseline may not be independent of the curation pool. The paper says twenty annotators answered tasks "under the same free-form conditions," but does not state whether these annotators had previously authored or reviewed any HSS tasks. Given the pipeline's unanimity filtering, anyone involved in review has seen the reference answers, which would inflate the 93.1%. This needs to be stated explicitly.
- Judge mediation of the human baseline is unclear. It is not stated whether human answers were also graded by Claude-Opus-5 under the same rubric protocol, or scored by human reviewers. If judge-graded, residual judge strictness affects both sides; if human-graded, the comparison mixes grading regimes. Either is workable, but the choice should be reported.
Questions for authors
See weaknesses above
Response to Reviewer TB9N
We thank the reviewer for the detailed review. We address the reviewer's comments point by point below.
W1. Subdomain-level uncertainty.
We agree that the subdomain claims need the same treatment as the main table. The new Appendix H puts 95% intervals on every subdomain-level claim, and Secs. 3.2 and 3.4 now report them.
W2. Independence of the human baseline.
The human participants are independent of the curation pool. None of them authored or reviewed any HSS task, so none had seen a task or its reference answer. Sec. 3.1 and App. D now state this explicitly.
W3. How human answers were graded.
Human answers were graded by the same LLM judge (Claude-Opus-5) with the same rubrics and prompt as the models, so any residual judge strictness applies equally to both sides. To check the judge itself, human reviewers who saw the media audited 150 of its verdicts and agreed with all of them (App. J.1), and the reference answers pass all 522 tasks under the same grading (App. J.2). Sec. 3.1 and App. D now state the grading protocol.
We hope our responses address the reviewer's concerns and that the reviewer will reevaluate our work based on these responses and the updated manuscript.