acceptodds
Under review as a conference paper at ICLR 2027

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

Abstract

Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns like unwritten rules or hidden labels. This capacity reflects a form of humanity's *sixth sense*: an intuitive reasoning mechanism that recovers implicit information beyond raw sensory perception. Crucially, this rapid, zero-shot visual intuition underpins everyday navigation and social interaction, making it a vital capability for Multimodal Large Language Models (MLLMs) deployed alongside people. Existing visual benchmarks, however, target either deliberate expert-level analysis in academic and mathematical domains or low-level perception, leaving the intuitive reasoning that people perform largely untested. To bridge this gap, we introduce **Humanity's Sixth Sense (HSS)**, a benchmark for intuitive visual reasoning. HSS spans diverse image and video inputs, organizes items under a structured taxonomy, and pairs each with human-written prompts probing the implicit temporal, spatial, social, and abstract structure that people infer at a glance. State-of-the-art models fall well short of human performance on these tasks: participants reach 93.1% accuracy, while the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort. Despite excelling in many complex tasks that require advanced perception and knowledge, current models still struggle significantly on these visual tasks tasks intuitive for humans. We further explore agentic frameworks that apply dynamic visual manipulation to HSS, which narrows but does not close the gap. HSS establishes intuitive visual reasoning as a measurable axis of progress and directs attention to a capability that scaling on current benchmarks has so far left behind.

Primary area:
datasets and benchmarks
Keywords:
intuitive visual reasoning, multimodal

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TB9NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HSS is a benchmark for "intuitive visual reasoning": inferring what is implied but not depicted in an image or video—past causes, future trajectories, affordances, social roles and intent, occluded structure. The benchmark contains 522 human-authored tasks (288 image, 234 video) organized into 4 domains and 11 subdomains, produced by a highly lossy curation pipeline (3,466 authored → 522 released, 15.1% acceptance) with three independent review rounds requiring unanimous consensus. Answers are free-form and graded against atomic rubrics by an LLM judge, with an item counting as solved only if every rubric criterion is met. Across 24 frontier MLLMs, the best model (GPT-6-astra, max reasoning effort) reaches 53.6% vs. 93.1% human accuracy; a blind control drops the top model to 6.6%; social understanding is the weakest domain (23.9% avg.); failures concentrate in perception (53%) and latent inference (41%) rather than deliberate reasoning (5%); agentic harnesses (Claude Code, Codex) add +5 to +20 points but do not close the gap; and higher reasoning effort sometimes hurts (e.g., −14 points on retrodiction for GPT-6-astra).

Critical strengths
  1. A genuinely under-served evaluation axis, cleanly delimited. The explicit-perception (V*, BLINK) vs. implicit-inference distinction, and the positioning against MMMU-style "images as givens for academic problems," is accurate and useful. Complementing BabyVision ("perceive what is explicit") with "infer what is implicit" is a crisp conceptual contribution.
  2. Curation rigor is the strongest part of the paper. A 15.1% end-to-end acceptance rate with documented attrition at each of three independent unanimous-consensus gates, plus explicit exclusion principles (no pure perception, no specialized expertise), is well above typical benchmark practice and directly targets the subjectivity risk inherent to "intuition" tasks.
  3. The evaluation is unusually well-audited. Blind control (6.6% residual, with a concrete analysis of why—two-way questions + invented detail), three-vendor judge cross-grading (r ≥ 0.98, κ ≥ 0.92, >95% agreement, self-preference shift of only −0.44), a 20-person human baseline with confidence tracking, and bootstrap CIs on all main numbers. This is the standard other benchmark papers should meet.
  4. The failure analysis yields non-obvious, actionable findings. That 94% of failures are perceptual/latent rather than reasoning failures; that on multi-model-failed tasks a median of 80% fail for the same cause; that agentic re-inspection fixes missed cues but cannot fix 2D-overlap-read-as-3D errors (102→101); and that failed agent runs take more steps than successful ones—each of these is a concrete signal for where modeling effort should go.
Critical weaknesses
  1. Subdomain-level statistics are thin. With 522 tasks across 11 subdomains, many subdomains hold roughly 40–50 items; the subdomain-level claims (e.g., per-subdomain failure signatures like "extrapolation fails on facing direction 42%", reasoning-effort swings of −14 to +16 points) rest on dozens of items and a handful of models, yet are presented without per-subdomain uncertainty. The item-level ±3–4 bootstrap intervals on the main table are good practice, but the more interesting subdomain narratives need the same treatment.
  2. The human baseline may not be independent of the curation pool. The paper says twenty annotators answered tasks "under the same free-form conditions," but does not state whether these annotators had previously authored or reviewed any HSS tasks. Given the pipeline's unanimity filtering, anyone involved in review has seen the reference answers, which would inflate the 93.1%. This needs to be stated explicitly.
  3. Judge mediation of the human baseline is unclear. It is not stated whether human answers were also graded by Claude-Opus-5 under the same rubric protocol, or scored by human reviewers. If judge-graded, residual judge strictness affects both sides; if human-graded, the comparison mixes grading regimes. Either is workable, but the choice should be reported.
Questions for authors

See weaknesses above

1 comment
AuthorsOpenReview ↗

Response to Reviewer TB9N

We thank the reviewer for the detailed review. We address the reviewer's comments point by point below.

W1. Subdomain-level uncertainty.

We agree that the subdomain claims need the same treatment as the main table. The new Appendix H puts 95% intervals on every subdomain-level claim, and Secs. 3.2 and 3.4 now report them.

  • Failure signatures hold. All six signatures named in Sec. 3.2 still hold. Extrapolation fails on facing direction in 42% of failures [31, 55], against 17% [10, 26] for the next cause, and affordance fails on size/fit in 62% [49, 74], against 22% [12, 32]. In all six, the named cause stays on top in at least 99% of resamples, and its interval does not overlap the runner-up's (Table 19). For retrodiction, social role, patterns and alien viewpoint, the top two causes are close, so we claim no single signature there.
  • Reasoning effort: the gains are real, but the drops cannot be told apart from noise. Of the 33 model × subdomain changes from lowest to highest effort, 7 differ clearly from zero, and all 7 are gains (Table 20). The 14-point retrodiction drop from high to xhigh is significant on its own ([−26, −4]), but it is one comparison among many, so Sec. 3.4 now presents it as an example of non-monotonic behaviour rather than an established effect.
  • Subdomain scores. A typical subdomain score has an interval of about ±12 points (range ±5 to ±19), against ±3–4 overall (Table 21). We compare subdomains, or models within a subdomain, only where the intervals separate. The headline conclusions rest on the full-benchmark and domain-level results.

W2. Independence of the human baseline.

The human participants are independent of the curation pool. None of them authored or reviewed any HSS task, so none had seen a task or its reference answer. Sec. 3.1 and App. D now state this explicitly.

W3. How human answers were graded.

Human answers were graded by the same LLM judge (Claude-Opus-5) with the same rubrics and prompt as the models, so any residual judge strictness applies equally to both sides. To check the judge itself, human reviewers who saw the media audited 150 of its verdicts and agreed with all of them (App. J.1), and the reference answers pass all 522 tasks under the same grading (App. J.2). Sec. 3.1 and App. D now state the grading protocol.

We hope our responses address the reviewer's concerns and that the reviewer will reevaluate our work based on these responses and the updated manuscript.

Reviewer_Avv8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Humanity’s Sixth Sense (HSS), a benchmark of 522 human-authored image and video questions across four domains and eleven subdomains. Each question has a reference answer and an open-ended grading rubric, and the authors describe three independent review rounds for task admission. On the released benchmark, the reported human accuracy is 93.1%, while the strongest of 24 evaluated multimodal models reaches 53.6% (Table 1). The paper also studies blind answering, reasoning effort, video frame budgets, automated grading, and agentic tool use.

Critical strengths

1.The authors invest substantially in task curation: 522 of 3,466 authored tasks survive three review rounds (Section 2.4). The free-form format and blind control help reduce simple answer-guessing explanations. The evaluation covers many model families and reports useful checks, including alternative LLM judges, frame-budget sweeps, and paired agentic comparisons (Tables 1–2; Sections 3.4 and Appendix H).

Critical weaknesses

1.The paper defines intuition partly as rapid and automatic (Section 1), but does not report response-time measurements or a time-constrained human evaluation. Its own automated analysis attributes 53% of model failures to perception (Section 3.2). Thus, the present accuracy gap does not by itself show which portion reflects latent inference rather than difficulty finding or resolving visual evidence. 2.Several analyses require stronger independent validation. The protocol behind the 93.1% human baseline is underdescribed, including whether respondents were independent of task authors and how many people answered each item. Cross-judge agreement is measured among LLMs, while the failure-cause labeller neither sees the media nor has human-validated labels (Appendix D). In addition, models receive different video pathways and frame caps (Appendix C), and the frame-budget sweep shows that this choice can materially affect accuracy.

Questions for authors

1.How was the human baseline collected and graded? Please report annotator independence, answers per item, uncertainty intervals, and response times if available. 2.Can the authors validate a stratified sample of failure-cause labels with human reviewers who can see the media, and add a control that supplies the decisive visual cues to models? These checks would help separate perception from inference. How stable are the cross-model conclusions under a shared video frame budget or a clearly separated native-video comparison?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Avv8 Part 1

We thank the reviewer for the detailed review. We address the reviewer's comments point by point below.

W1. Speed of intuition, and perception vs. latent inference.

In a new experiment, we timed human answers on 50 random tasks (25 image, 25 video, across all 11 subdomains). The recorded time covers viewing the media, reading the question, and writing a full-sentence answer (median 25 words).

nMedian (s)≤1 min≤5 min
All4918020%86%
Image2512232%96%
Video241868%75%

On the same tasks, single-call models answer in a median of 24–63 s and agentic harnesses in 1.5–10 min, yet none passes more than 54%, compared to 90% for humans (App. D, Tables 10–11).

W2 / Q1. Human baseline.

The 20 participants did not author or review any HSS task. Each answered about 25 tasks, each task was answered once, and there was no time limit. Answers were graded using the same LLM judge and rubrics as the models. Participants also flagged uncertainty: they were unsure on 63 tasks (12.1%), scoring 94.1% on the rest and 85.7% on the unsure ones. The full protocol is detailed in Sec. 3.1 and App. D.

W2. Independent validation of the LLM-based analyses.

We added three additional human checks to evaluate alignment with the LLM annotators:

  • Failure labels (App. E.1): model achieves 92.7% primary-cause and 95.3% family-level agreement across 150 labels from three models.
  • Judge verdicts (App. J.1): 150/150 agreement on a balanced sample (77 pass, 73 fail).
  • Oracle (App. J.2): reference answers pass all 522 tasks and all 723 criteria.

Q2. A control that supplies the decisive visual cue.

We thank the reviewer for this insightful comment. For GPT-6-astra perception failures, we further generated text hints specifying where to look and what to check without revealing the answer (e.g., "Follow the right-hand man's right arm down and inspect the hand before deciding what it is doing") and re-ran inference (App. E.2, Table 16):

nNo hintWith hint
All550.054.5
Image / Video41 / 140.056.1 / 50.0
Misidentified object/person/role160.068.8
Misread orientation or facing80.062.5
Read 2D overlap as 3D100.060.0
Misjudged size, distance or fit210.038.1

Even when told which region and property decide the answer, 45% of these tasks remain wrong. Size, distance, and fit improve least (13 of 25 remaining failures; 17 together with 2D-overlap errors). Echoing our agentic findings, locating visual evidence resolves identification errors, while judging 3D size and depth from 2D views stays hard.

AuthorsOpenReview ↗

Response to Reviewer Avv8 Part 2

W2 / Q3. Video pathways and frame caps.

Our frame ablation study shows that under a uniform budget, the ranking is stable. The four frame-sampled models below keep the same order at 10, 50, 100, and 500 frames (Table 3, now in the main text):

Video pass@1 (%)50 frames500 frames
GPT-6-astra41.951.7
Claude-Opus-5.532.136.3
GPT-6-sol27.831.2
Claude-Opus-519.727.4

Native video vs. frames.

We ran a controlled comparison between native video and sampled frames. Specifically, we re-ran the two native-video models, Gemini-3.8-flash and Muse Spark 1.3, on all 234 video tasks using sampled frames instead of the video file: 2 fps, up to 500 frames for Gemini and up to Muse Spark's limit of 50. Endpoint, reasoning effort, prompt, and judge were unchanged, and we compared first attempts task by task (new App. I.1, Table 22).

ModelNative run sentTasksNativeFramesΔ [95% CI]
Gemini-3.8-flashvideo file16836.345.2+8.9 [+1.2, +16.1]
frames (file too large)5543.643.6+0.0 [−10.9, +9.1]
Muse Spark 1.3video file15230.934.9+3.9 [−3.9, +11.8]
frames (file too large)7628.930.3+1.3 [−6.6, +9.2]

On clips where the native run received the video file, frames did at least as well. Clips over the provider's upload limit fell back to frames in both runs, so they act as a control. There the difference is 0.0 and +1.3 points, which is run-to-run noise.

The likely reason is that the native path provides a more compressed input rather than a richer one. The median native request carries 3.6k input tokens for Gemini-3.8-flash and 9.3k for Muse Spark 1.3, compared to 103k and 15.6k for the same clips as frames. Gemini receives about 30× more visual input as frames and gains the most, whereas Muse Spark receives about 2× more and gains less. This matches the frame-budget sweep (Table 3): fewer visual tokens yield lower scores. Native video therefore gives neither model an advantage on HSS, and the video pathway does not explain the gap to human performance.

Ethics flag.

We note the ethics flag and summarize the safeguards described in our Ethics Statement:

Media licensing. All media come from Wikimedia Commons, Openverse, and the Internet Archive, which provide machine-readable license metadata for each item. We admit only Public Domain, CC0, CC-BY, and CC-BY-SA items, all of which permit redistribution. Any item with non-commercial, no-derivatives, or unverifiable rights is rejected. Creator attribution is released with each item, and audio tracks are stripped from all videos.

  • People in the media. Identifiable individuals appear only in public settings, and the media are used strictly for research evaluation. We also exclude videos that mainly feature minors.
  • Social inferences. Theory-of-mind labels can encode cultural assumptions. Our consensus requirement removes tasks where interpretations diverge, and we explicitly interpret these results as agreement with annotator judgment rather than ground truth about the people depicted.
  • Annotators. Annotators were recruited and paid through a professional data-annotation platform at its standard contracted rates. We are happy to address any specific concern the reviewer has in mind.

We hope our responses address the reviewer's concerns and that the reviewer will reevaluate our work based on these responses and the updated manuscript.

Thank you for clarifying the human baseline and adding the timing, human-audit, and visual-cue experiments. The fact that the 20 respondents were independent of task authoring and review strengthens the human–model accuracy comparison.

The timing results should, however, be interpreted more narrowly. The median human completion time was 180 seconds, and only 20% of completed tasks took no more than one minute. Because this includes viewing the media, reading the question, and writing an answer, it does not directly establish that the measured responses were rapid or automatic. The results support a substantial accuracy gap under the tested protocol; please distinguish that finding from the paper’s stronger claim about the speed of intuition. Please also explain why timing results are reported for 49 of the 50 sampled tasks, and provide an uncertainty interval for the 93.1% human baseline, accounting for task and participant variation.

The human audits and cue experiment are useful additions. For the audits, please specify how the 150 failure labels and 150 judge verdicts were sampled, whether human evaluators saw the original media, and whether they were blinded to the LLM decisions. For the cue experiment, the 0% no-hint result follows from selecting 55 tasks the model had already failed. The 54.5% recovery after a hint shows that directing attention to visual evidence can resolve some failures. The remaining failures should not yet be attributed to inference alone unless the hints are independently checked for adequacy, particularly on size, fit, and depth questions. Finally, please reconcile the rebuttal’s count of 723 rubric criteria with the 753 criteria reported in Table 3 of the submission.

Reviewer_JakqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HSS, a benchmark of 522 human-authored open-ended tasks (288 image tasks and 234 video tasks; 17.6 hours of video in total) that tests whether MLLMs can infer information that natural images and videos do not show directly. The tasks span 4 domains and 11 subdomains (Sec. 2.2, Tables 3/4).

  • Scoring: Each task has a reference answer and a set of atomic rubric criteria (753 in total). An LLM judge (Claude-Opus-5) grades each criterion without seeing the media, and a task counts as solved only if all criteria are met (Sec. 2.1, C.3).
  • Construction: 3,466 drafted tasks go through three review rounds, and 522 are kept, an acceptance rate of 15.1% (Fig. 8).
  • Main results: The best model, GPT-6-astra, scores 53.6%; humans score 93.1%; the blind control scores 6.6% (Table 1).
  • Additional analyses: failure-cause analysis (perception 53%, latent 41%, reasoning 5%; Table 6), agentic harnesses (gains of +4.9 to +20.5; Table 2), sweeps over reasoning effort and frame budget (App. G/H), and cross-judge agreement across three vendors (κ of 0.914–0.946; App. I).
Critical strengths
  1. A valuable, hard-to-saturate dataset that complements existing benchmarks, built with a rigorous process. It uses natural media and open-ended answers to test implicit inference (retrodiction, affordance, theory of mind, etc.), which genuinely complements perception benchmarks (V*, BLINK) and academic-problem benchmarks (MMMU). The 53.6 vs. 93.1 gap will not close soon. The construction involves three review rounds with a 15.1% acceptance rate, and attrition is reported for every round (Fig. 8). Media are restricted to PD/CC0/CC-BY/CC-BY-SA licenses, which allows legal redistribution. This matters because the core value of a benchmark paper is to give the community a usable measurement instrument, and it is the main reason I am not recommending a clear rejection.
  2. The controls are more careful than in comparable work. The paper provides bootstrap CIs (Table 1). The blind control is illustrated with worked examples of what passes without the media, and the authors acknowledge defects such as questions that carry their own answer (C.6). The paper also includes cross-grading by three judges and a self-preference analysis (App. I), and McNemar tests for the agentic experiments (Table 9).
Critical weaknesses
  1. The headline human–model gap (93.1% vs. 53.6%) depends on the validity of the human baseline and the gold labels, and neither is adequately documented or verified.

    • The human baseline protocol is minimally reported. Apart from the human row of Table 1, the paper's only description of how the human baseline was collected, appendices included, is one paragraph in Sec. 3.1: "Twenty annotators answered the tasks under the same free-form conditions as the models", plus statistics on self-reported confidence. It does not say whether these participants overlap with the task authors or reviewers, whether they could see the reference, how many people answered each task, how answers were aggregated, who graded them, or whether there was a time limit. If answering was independent, 93.1% is strong evidence that the gold labels are reliable overall, though it cannot rule out a small number of mislabeled tasks; App. E suggests such tasks may concentrate among those all models fail, which are the tasks that drive the human–model gap. Reporting human accuracy on these tasks would therefore be informative both about label quality and about how independent the human answers were.
    • The gold labels are not shown to be independently verified, and the pipeline does not systematically catch errors in them. Reviewers assess the complete task, including the reference and rubric ("every released rubric has been verified by at least three reviewers beyond its author", Sec. 2.4), and the review pipeline includes no step in which reviewers answer without seeing the reference. Review of this kind does filter some defects (Fig. 8 attributes part of the first-round attrition to a lack of "unambiguous consensus on the ground truth") and is common in dataset construction, but it does not support a claim as strong as "guarantee strict human consensus" (Contribution 2), and the App. E examples below show that errors slip through. Because the judge does not see the media and treats the reference as ground truth (C.7, rule 5), any such error is recorded as a model error. Sec. 5 states that an oracle condition is reported, and App. I describes a comparison against human graders that "we report alongside the human baseline", yet neither result appears in the paper. The oracle in particular bounds only the judge's error: if a reference is wrong, the oracle still marks it correct. Whether the human-grader comparison can detect gold errors depends on whether the graders see the media, which the paper does not specify.
    • The paper's own appendix examples expose label problems. App. E picks one example from each subdomain out of the 79 tasks that GPT-6-astra, Gemini-3.8-flash and Claude-Opus-5.5 all missed on every one of their three attempts (11 examples in total) to argue that these failures are "a property of the task rather than of a grader", and the caption of Fig. 9 states that "each card shows exactly the media the models were served". I checked each image and found at least 3 cases where the reference is likely wrong or the grading too strict:
      • Extrapolation (Fig. 11, reversing truck): The only person in the frame stands directly behind the truck and wears an orange-red high-visibility vest, yet the reference says "the man in green hi-vis", and no such person appears in the image. The quoted model answer, "A man in a red vest on the left" (App. E notes that quoted answers are the judge's condensed reading), also misplaces this person, who stands on the right of the frame behind the truck, so the failure may not be due to the color mismatch alone; but the reference itself is inconsistent with the image.
      • Social role, norm & power (Fig. 13, hangar): The question asks "Are the people around the aircraft visitors, or crew", but no identifiable person stands near the foreground aircraft, so the premise of the question does not hold. The "open access panels" cited by the reference are also hard to confirm in the image, and the scene is a display hangar with vintage aircraft, where "visitors" is equally plausible.
      • Retrodiction (Fig. 11, landscape painting): The path widens in the foreground and runs from the lower left to the bottom edge, so "lower-left side" and "bottom edge" refer to the same region. In addition, the painting contains two groups of dogs and a rider, so "the traveller with the two dogs" is ambiguous; and since this is a static painting, asking from which side the traveller entered the frame is a contrived question.
  2. The core narrative (that the capability is intuitive, that it resists scaling and agentic augmentation, and that 94% of errors are not reasoning errors) goes beyond the evidence and conflicts with the paper's own data.

    • "Intuitive" is not operationalized. Sec. 1 defines intuitive as fast and automatic, but the paper has neither timed experiments nor response-time data.
    • "Resistant" overstates the data. The gap does remain large: the best agentic configuration reaches 59.3% (Codex with GPT-6-astra, Table 2), and the abstract itself says that agentic frameworks narrow "but does not close the gap". The problem is the wording. Contribution 4 states that the capability gap "remains resistant to both agentic augmentation and standard compute scaling", and the Conclusion states that the gap "persists despite test-time compute scaling and integration into agentic harnesses" and that "current scaling paradigms have so far yielded minimal progress". The reported gains are not minimal: Claude Code improves Opus-5 by 20.5 points (Table 2); from the lowest to the highest reasoning effort, GPT-6-astra improves in all four domains (+8.2 temporal, +12.5 physical, +15.5 social, +7.8 abstract; App. G); and raising the frame budget from 10 to 500 frames improves GPT-6-astra's video score by 17.9 points (Table 11). The evidence offered that deliberation does not help is that "peak accuracy is frequently achieved at intermediate effort levels" (Sec. 3.4; App. G adds that only five of the twelve model × domain curves peak at the highest setting), illustrated by GPT-6-astra's 14-point drop on retrodiction from high to xhigh. But subdomain cells are small (27–70 tasks, Table 3). Using the standard error of a binomial proportion, √(p(1−p)/n), at p≈0.5 the 95% confidence interval of a single cell is roughly ±12 (n=70) to ±19 (n=27) points, so non-monotone curves with peaks at intermediate levels are expected even with no true effect. For the highlighted cell, n=50 and p≈0.57 give √(0.57×0.43/50)≈0.07, i.e., a 95% confidence interval of roughly ±14 points, the same size as the drop. Table 10 in App. G also shows that this cell's accuracy across the five effort levels is 50 → 48 → 64 → 50 → 56, which fluctuates with no trend of more thinking making it worse: the high (64) → xhigh (50) step is just one dip, followed by a rebound to 56 at max, for an overall gain of +6 from the lowest to the highest level. (Because both levels are evaluated on the same tasks, this is a paired comparison and the true error is smaller than ±14, but the drop is still within noise.)
    • The failure-cause analysis is unreliable and inconsistent with the appendix narrative. The main-text Sec. 3.2 (Fig. 3), Sec. 3.3 (Fig. 4), and the Conclusion all rely on this failure-cause analysis. An LLM assigns each failure a cause based on how the model's answer departs from the reference text, without seeing the media and without human verification (App. D). Treating the reference as ground truth this way is workable when the reference spells out the evidence. But when the reference itself is wrong (as in the truck example in W1, where the reference describes a person in green hi-vis who does not appear in the image), or when it does not state the relevant evidence, the distinction between perception and reasoning breaks down. The authors acknowledge in App. D that these shares should be read as "a consistent automated reading… rather than as adjudicated ground truth", yet the Conclusion uses "94% of model errors stem from visual perception and latent inference" directly to support its central claim. In addition, App. E characterizes HSS's difficulty as "how little of the difficulty is perceptual… the models describe that evidence correctly while drawing the wrong conclusion". By App. D's own definition, this describes a reasoning failure ("has the facts right and draws the wrong conclusion"), yet App. D itself reports that 53% of failures are perception failures (66% for image tasks) and only 5% are reasoning failures (Table 6).
Questions for authors
  1. Do the 20 human participants overlap with the task authors or reviewers? How many people answered each task, and how were answers aggregated? Who graded the human answers, and was it the same LLM judge? Was there a time limit?
  2. Can you provide the oracle result that Sec. 5 states is reported (the proportion of references the judge marks correct), and the agreement rate between human graders and the LLM judge on a stratified subset promised in App. I? Did the human graders see the media?
  3. What is human accuracy on the 79 tasks that all three models failed? After manually re-checking these 79 tasks and the 63 tasks on which humans were unsure, how many references are wrong or ambiguous, and what is the human–model gap after removing them?
  4. Can independent annotators answer a subset blind, without seeing the reference, and can you report their agreement?
  5. How do the video column and the overall ranking change under a uniform frame budget (e.g., 50 frames)?
  6. What does the blind control score with the same prompt as the main experiment (i.e., without the instruction allowing it to decline)? What fraction of tasks are yes/no or two-way questions, and what is the corresponding chance baseline? The blind control scores 11.7% on video but only 2.4% on images; does this indicate that video questions leak their answers?
  7. Can the failure-cause labels be spot-checked by humans who see the media (e.g., 300 cases), with agreement against the LLM labels reported?
  8. Can you report accuracy crossed by domain × media (especially social image vs. social video tasks), to separate the social-domain effect from the video effect?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Jakq Part 1

We thank the reviewer for the detailed review. We address the reviewer's comments point by point below.

W1(a) / Q1. Human baseline protocol.

The 20 participants do not overlap with the task authors or reviewers, so none had seen a task or its reference before. The 522 tasks were split among them (about 25 each), and each task was answered once. Participants saw the same media and prompt as the models, had no time limit, and flagged whether they were unsure. Their answers were graded by the same LLM judge and rubrics as the models, so the 93.1% is directly comparable with Table 1 (Sec. 3.1, App. D).

W1(b) / Q4. Independent verification of the gold labels.

The human baseline is the blind check the reviewer describes. Twenty people outside the curation pool answered every task without seeing the reference, and their answers agree with the reference on 93.1% of tasks (94.1% on the 459 tasks where they were sure). We agree that this shows the labels are reliable overall but cannot rule out a small number of bad items. To eliminate such cases, our review process ensures that tasks are admitted only with unanimous approval from three reviewers.

Q3. Human accuracy on the tasks all three models failed.

GPT-6-astra, Gemini-3.8-flash and Claude-Opus-5.5 fail 83 tasks on all three attempts and humans pass 84.3% of them (70 of 83) (App. D). These tasks are harder for people too, but people still answer most of them correctly, so they are not mainly broken or unanswerable items.

Q2. Oracle and human–judge agreement. Both results are now in the paper.

  • Oracle (App. J.2). We graded every reference answer (same judge, prompt and rubric). The references pass all 522 tasks and all 723 criteria.
  • Human alignment with LLM judge (App. J.1). Human annotators audited 150 LLM-judge verdicts across 50 tasks (28 image and 22 video tasks), with responses generated by GPT-6-astra, Gemini-3.8-flash, and Claude-Opus-5. Reviewers had full access to the media, alongside the prompt, rubric, model response, and per-criterion verdicts, achieving 100% agreement (150/150) with the judge. The audited set is balanced (77 pass, 73 fail) and includes 50 responses from the judge's own model. These results confirm that our rubrics are objective and atomic, yielding minimal LLM-judge error.

W1(c). Appendix examples.

The earlier appendix showed cropped media at low resolution and abridged prompts. Appendix K now shows the full image and the full prompt for every example.

  • Extrapolation (Fig. 16). The crop had cut off the man in green hi-vis on the far right, directly behind the truck. He is visible in the full image.
  • Social role, norm & power (Fig. 24). The full prompt asks about the people around the large aircraft in the background, not the foreground aircraft. At full resolution, the person bent over the open panel at the workstation (middle left) is working on the aircraft, not visiting it.
  • Retrodiction (Fig. 18). The full prompt names the traveller "on the wooden footbridge in the lower foreground", which resolves the ambiguity between the two groups of dogs. The reference gives a single answer, the bottom of the frame, which is where the path leads.

W2(a). Human response time.

We now report human response times (Sec. 3.1, App. D, Tables 10–11). On 50 random tasks the median is 3.0 minutes (122 s for images, 186 s for videos). This covers viewing the media, reading the question and writing a full-sentence answer, so it overstates the time needed to reach the answer. Models given the same or more time still fall well short: no single-call or agentic configuration exceeds 54% on these tasks, against 90% for humans.

W2(b). "Resistant" overstates the data.

We agree that the wording was too strong: gains of +21.2 points (Claude Code) and +17.9 points (10→500 frames) are not minimal. We have revised the claim throughout. Contribution 4 now says that performance improves with reasoning effort, frame budget and agentic tools, but the best configuration (61.1%) remains far below human performance. The Introduction says these strategies help, by up to 21 points with agentic tools, but none brings models close to humans. The Conclusion says current scaling strategies help but still leave models far below people. On reasoning effort, the reviewer's statistical point is correct. The new Appendix H shows that of the 33 lowest-to-highest effort changes, 7 clearly differ from zero, and all 7 are gains; no decrease is distinguishable from noise. We have revised the claims in Sec. 3.4 accordingly.

AuthorsOpenReview ↗

Response to Reviewer Jakq Part 2

W3 / Q7. Reliability of the failure-cause labels.

We conducted a new human study to evaluate this. Human reviewers who saw the media checked 150 labels: 50 each from GPT-6-astra, Claude-Opus-5, and Gemini-3.8-flash, covering 89 image and 61 video failures (App. E.1, Table 15).

LabelsSame primary causeSame family
All150139 (92.7%)143 (95.3%)
Labelled perception908386
Labelled latent514950
Labelled reasoning977

LLM annotators achieve a high alignment rate with humans (>92%). The 11 disagreements go both ways and mostly swap neighboring causes (e.g., size/fit vs. facing).

Q5. Uniform frame budget.

We ran four models on all 234 video tasks at caps of 10, 50, 100, 200 and 500 frames under the same setting. The sweep is now Table 3 in Sec. 3.4.

Video pass@1 (%)1050100200500
GPT-6-astra33.841.945.347.051.7
Claude-Opus-5.530.332.132.537.236.3
GPT-6-sol23.127.829.126.131.2
Claude-Opus-518.819.728.627.427.4

At a uniform 50 frames the four models keep their 500-frame order, and the same holds at 10 and 100. Combining each model's image score with its 50-frame video score gives overall scores of 49.7, 44.8, 30.8 and 28.5, which is also the Table 1 order.

Q6. Blind control without the decline instruction, two-way questions, and video leakage.

We ran a new blind experiment on GPT-6-astra without the system prompt (results are now reported in Appendix C.7, Table 8).

With blind promptNo system prompt
Pass@1, all6.918.8
Pass@1, image / video2.9 / 11.920.0 / 17.2
Abstains, image / video90% / 45%40% / 4%
Tasks passed on ≥1 attempt13.4%34.1%
Tasks passed on all 3 attempts0.8%5.9%

Without the prompt, the model stops declining and answers as if it had seen the scene. However, its passes are unstable: 34.1% of tasks pass at least once, but only 5.9% pass all three attempts.

  • Two-way questions. 38 tasks (7.3%) are yes/no and 42 (8.0%) are left/right or which-side, so 80 (15.3%) are explicitly two-way. Without the prompt, the blind model passes 26.3% of yes/no and 20.6% of which-side tasks, well below the 50% a coin flip gives, and 19.8% of open-ended ones. Blind passes are therefore not concentrated on two-way questions (Table 9).
  • Video vs. image. The gap reflects how often the model is willing to answer rather than leakage. With the prompt, it declines 90% of image questions but only 45% of video ones. Video questions often narrate the scene to pin down a moment ("At one point in the video, a woman in a green shirt descends a path…"), prompting the model to answer "based on your description" (16% of video answers vs. 6% of image answers). Without the prompt, image and video blind scores are similar (20.0% vs. 17.2%).

Q8. Domain × media.

Mean pass@1 over the 25 models of Table 1 (number of tasks in parentheses; App. C.5, Table 7):

TemporalPhysicalSocialAbstract
Image40.3 (81)35.7 (114)31.5 (28)35.8 (65)
Video33.5 (53)30.9 (62)22.8 (82)32.1 (37)

Social is the lowest domain for both images and videos, so the social effect is not only a video effect. Video does cost social tasks more than other domains (8.7 points, against 3.7–6.8 elsewhere), and 82 of the 110 social tasks are videos, so part of the social gap in Table 1 comes from that mix.

We hope our responses address the reviewer's concerns and that the reviewer will reevaluate our work based on these responses and the updated manuscript.

I thank the authors updating pdf and showing more detail about their experiment. These address my conceren about quantity of annotation, I have raised the score from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer Jakq

Thank you so much for reviewing the revised paper and for the positve assememnt of HSS. We are glad the additional details on the annotation and human baseline addressed your concern. Your comments have made the paper clearer and more rigorous.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their time and constructive feedback. We have carefully addressed each comment and revised the manuscript accordingly; all changes are highlighted in blue.

We appreciate that the reviewers recognize HSS as a benchmark for an under-served evaluation axis: inferring what is implicit in natural images and videos, which complements perception benchmarks such as V* and BLINK and academic-problem benchmarks such as MMMU (Jakq, TB9N). The reviewers also highlight the rigor of our curation, in which 522 of 3,466 authored tasks (15.1%) survive three independent review rounds, with attrition reported at every stage (all reviewers), and our use of only redistributable media (Jakq). They further note the care of our evaluation: bootstrap CIs, a blind control, three-vendor cross-judging with a self-preference check, which TB9N describes as "the standard other benchmark papers should meet." Finally, they note the large and durable gap between humans (93.1%) and the best model (53.6%) and the actionable failure analysis (TB9N). Below we summarize our responses to the concerns shared across reviews; detailed responses follow under each review.

Common concerns

Human baseline protocol (all reviewers).

The 20 participants did not author or review any HSS task. Each answered about 25 tasks, each task was answered once, there was no time limit, and answers were graded by the same LLM judge and rubrics as the models. The full protocol is now in Sec. 3.1 and the new Appendix D.

Response time (Jakq, Avv8).

We timed human answers on 50 random tasks of HSS. The median sovle time is 3.0 minutes, including watching the media and writing a full-sentence answer, and 86% finish within 5 minutes. On the same tasks, single-call models take 24–63 s and agentic harnesses 1.5–10 min, and none achieves more than 54% pass rate, against 90% for humans. Hence spending more time does not close the gap (Sec. 3.1, Appendix D, Tables 10–11).

Human validation of the LLM components (Jakq, Avv8).

  • Judge vs. humans (App. J.1): reviewers who saw the media agreed with all 150 audited verdicts (77 pass / 73 fail), including the 50 that the judge gave to its own model.
  • Oracle (App. J.2): the reference answers pass all 522 tasks and all 723 criteria.
  • Failure labels (App. E.1): on 150 labels, humans who saw the media agreed on the primary cause 92.7% of the time and on the family 95.3% of the time.

Perception vs. inference (Avv8, Jakq).

A new hint control (App. E.2) tells the model where to look on 55 perception failures that genuinely need the media. The hint fixes only 54.5% of them, and size/distance/fit errors recover least (38.1%). Locating the evidence is only part of the difficulty.

Subdomain-level uncertainty (Jakq, TB9N).

The new Appendix H puts 95% intervals on every subdomain claim. All six failure signatures survive. Of the 33 lowest-to-highest effort changes, 7 differ clearly from zero, and all 7 are gains. We have softened the reasoning-effort narrative accordingly.

Frame budget (Jakq, Avv8).

The sweep is now in the main text (Table 3). Under a uniform 10, 50, 100 or 500 frames, the four models keep the same order. A new same-model comparison (App. I.1) shows that native video input gives no advantage over sampled frames.

Below, we provide a detailed, point-by-point response to each reviewer's comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.