acceptodds
Withdrawn from ICLR 2027

An Exam for Active Observers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.6 Sol, solves at most 23.5% of items at any reasoning-effort tier, and even Claude Fable 5, despite its strong results on reasoning and coding leaderboards, solves just 5.9% at its highest tier, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code. Such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception–reasoning loop.

Primary area:
datasets and benchmarks
Keywords:
multimodal LLMs, evaluation benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UGgXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ActiveVision, a benchmark for evaluating active observation in multimodal large language models. It contains 17 tasks across distributed scanning, sequential traversal, and visual attribute transfer, designed to require repeated visual inspection rather than one-shot image understanding. Experiments show a large gap between frontier MLLMs and humans, while increased reasoning effort and vision-related tool use only partially improve performance.

Critical strengths
  1. Well-motivated problem formulation. The paper isolates active observation as a distinct capability that is not well captured by existing multimodal benchmarks.
  2. Carefully designed benchmark. The 17 tasks cover complementary abilities such as exhaustive scanning, sequential tracing, and fine-grained visual comparison, while reducing simple linguistic shortcuts.
  3. Strong diagnostic experiments. The paper evaluates both increased reasoning effort and tool-using agents, providing useful evidence that the observed failures are not easily resolved by longer reasoning or external vision code.
Critical weaknesses
  1. Limited attribution to active observation. The observed undercounting and early traversal errors do not uniquely indicate a lack of active observation, as they may also stem from lower-level perception, object individuation, or path-tracing failures. The current analysis does not disentangle these alternative causes. For example, Figure 6, in Bounded Face Counting, the model analyzes the relevant region but misidentifies nodes and edges, leading to an incorrect count (12 vs. 10), which is more naturally explained as a perceptual error than a failure to observe.
  2. Small evaluation set. ActiveVision contains only 85 evaluation items, with five examples per task. This relatively small scale may reduce the statistical robustness of the results and make per-task conclusions sensitive to individual examples.
  3. Synthetic image distribution. Although the images are photorealistically re-rendered, they are generated from procedural scaffolds rather than collected from real-world data. This raises questions about how well the findings generalize to natural visual environments.
Questions for authors

see above

1 comment

Thank you for the careful review, and for your positive view of the problem formulation, the task design and the diagnostic experiments.

W1. Perception error vs. active observation. As defined in the paper, active vision is a series of actions, perceptions and decisions, and these are all the substrate of this capability. The scope of this paper is to test this series in a verifiable test bed, so failures in perception and object individuation are all counted as failures of this behavior. The only true alternative is that the model transcribes the image into a pure text description once and reasons only over the text. We take steps to prevent this alternative and force iterative perception (Sec. 3.2).

W2. Data scale. We do not view the dataset size as a weakness. The 85 items already give clear separation among strong agents (24.7% to 68.2%) and informative differences across the three task families (Fig. 5b), while the per-item runtime and cost (Table 1) make much larger agentic evaluations nontrivial. This is a reasonable balance between discriminative power and evaluation cost, consistent with the scale of widely used agent benchmarks such as OSWorld 2.0 and Terminal-Bench 2.0. The gap to humans is also far beyond sampling noise: the best tool-free model solves 20/85 items, with a 95% Wilson interval of 15.8% to 33.6%, against 96.1% for humans.

W3. Image distribution. We agree, but this is a trade-off. In exchange, we get the exact ground truth, and we can scale the benchmark. Our re-rendering also goes beyond line drawings: it adds realistic texture, lighting and clutter while the answers stay known. This realism matters, because the agents' classical vision tools break on these textures. Finally, non-real images are common in this line of work: BlindTest is fully synthetic, and most BabyVision images are also not real photographs but puzzle-style graphics (e.g., mazes, line tracing, find-the-difference).

Reviewer_Lv4aOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark called ActiveVision. It has tasks that requires a model to look at multiple different places and aggregate the local evidence together to be able to answer the questions. There's a variety of tasks that are procedurally generated and then revised by a neural model (GPT-image-2) to be more photorealistic. The results show that frontier models struggle with the tasks.

Critical strengths
  • The results are interesting in that they identify a clear weakness of existing models.
  • The authors designed some interesting visual tasks.
Critical weaknesses
  • The paper does not cite or discuss multiple related works that reduce the novelty of the work: (a) VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs, Shmuel Berman, Jia Deng (b) VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context. Yunxin Li, Baotian Hu, Haoyuan Shi, Wei Wang, Longyue Wang, Min Zhang (c) VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following, Hyesoo Hong, Minsoo Kim, Wonje Jeung, Sangyeon Yoon, Dongjae Jeon, Albert No

These papers have proposed and evaluated similar tasks (for example, path following), but the paper makes no mention of them. Thus the novelty of the benchmark is not clear. The authors need a comprehensive discussion of similar tasks that have appeared in the literature.

  • For a benchmark paper, the results are thin. There's not much more than the main results that the models do poorly.

  • No open-weight models are included.

  • The photorealistic revision is by a neural model, but the validity of such revision is not guaranteed, and it is not clear whether this post-processing has corrupted any examples and make the ground truth incorrect.

Questions for authors

I'd like the authors to address the novelty of the benchmark, especially the novelty of each task introduced.

1 comment

Thank you for the review.

W1. Related work. Thank you for these references, we will cite and discuss them. We agree that single task types overlap with prior work: for example, path following appears in Trace Without Tracking and Tunnel Vision, and graph problems appear in VisionGraph. The difference is the focus: all 17 tasks share one condition (Sec. 3.2). The task-relevant state is too large for one description, so the solver must look again, guided by what it found. We test this condition under one protocol, with exact ground truth (Sec. 3.3), a human baseline (96.1%, Fig. 4), a full reasoning-effort sweep (Sec. 4.2) and five coding agents (Sec. 4.3).

W2. Thin results. We respectfully disagree that the results are thin. We show the performance of all models (Fig. 4), the effect of reasoning effort (Sec. 4.2), the models' failure modes (App. D), how agents solve the tasks (Fig. 5), and how agents fail (Fig. 6). Specifically, more reasoning closes little of the gap. The five coding agents are reported with accuracy, cost and time, and we find that code solves attribute transfer but not traversal.

W3. Open-weight models. We only test frontier models for now. The tasks are far from solved even by frontier models, so it is too early to test open-weight models. No item is solved by all eight frontier models, so open-weight models would add little information at this stage. Open-weight models can be added in the future, as they become more capable.

W4. Re-render fidelity. We manually checked every re-rendered image first. Three human participants then took the full test and scored 96.1%, all three participants solved 76 of the 85 items, and every item was solved by at least one participant.

Reviewer_wa5eOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ActiveVision, a benchmark containing 85 instances across 17 tasks organized into distributed scanning, sequential traversal, and visual attribute transfer. Procedurally generated scaffolds are re-rendered into photorealistic images using an image generation model. The authors evaluate existing multimodal models at different reasoning-effort levels, five coding-agent configurations, and three human participants. They report substantial human–model performance gaps and interpret these results as evidence that current models lack robust active visual observation.

Critical strengths

The procedural task construction and evaluation of both direct model responses and tool-using agents provide a potentially useful evaluation resource. Reporting agent costs and intermediate failure artifacts also gives practical context beyond aggregate accuracy. However, these strengths do not establish the benchmark’s claimed diagnostic novelty.

Critical weaknesses
  1. The research gap and the benchmark’s incremental contribution are insufficiently established. Existing benchmarks, including BlindTest and BabyVision, already investigate overlapping capabilities such as counting, path following, and fine-grained visual discrimination, and document substantial human–model gaps. The paper groups related tasks under “active observation,” but does not convincingly establish what previously unmeasured capability this grouping isolates. In particular, the characterization of earlier work as documenting gaps without identifying missing capabilities understates their task taxonomies and failure analyses. The concrete additions are a new collection of task instances, photorealistic re-rendering, and evaluations of existing models and agents. These could support a benchmark contribution, but the paper does not establish that they yield substantially different scientific conclusions or diagnostic value beyond existing evaluations. More difficult images and lower model scores alone do not establish a new research gap.
  2. The experimental design does not validate the central interpretation of “active observation.” The main evaluation presents a complete image and scores the final answer. It neither directly measures adaptive observation nor compares it against controlled alternatives. Consequently, failures cannot be attributed specifically to an inability to revisit visual evidence: insufficient visual detail, local recognition errors, state-tracking failures, and ineffective reasoning strategies remain plausible explanations. The argument in Section 3.2 also conflates the difficulty of losslessly describing an entire image with the necessity of repeated observation. A solver only needs task-relevant information, which could potentially be extracted once into a structured representation. Likewise, increased reasoning effort failing to close the gap does not identify its cause. Tool-agent failure examples illustrate possible mechanisms, but do not establish their prevalence or isolate active observation as the bottleneck. This concern is compounded by the missing scaffold-versus-re-rendered comparison and insufficiently documented verification that generative re-rendering preserves the answer-relevant geometry. Ground truth from a deterministic scaffold is not automatically ground truth for the final generated image. The strong human results support overall solvability, but do not replace an explicit fidelity audit. With only five instances per task, fine-grained conclusions about individual capabilities are also unstable. These issues affect the paper’s central contribution and would require additional validation experiments rather than presentation changes.
Questions for authors
  1. What specific capability does ActiveVision measure that existing counting, tracing, and fine-grained discrimination benchmarks do not? What evidence distinguishes this capability from general visual task difficulty?

  2. Is there a controlled comparison demonstrating the benefit or failure of hypothesis-dependent observation relative to fixed observation under comparable budgets? If not, what evidence supports attributing the reported failures specifically to active observation?

  3. Were the final re-rendered images independently checked for preservation of answer-relevant topology and geometry? Please clarify the verification procedure, rejection criteria, and whether paired scaffold/re-rendered evaluation results are available.

1 comment

Thank you for the review.

W1 and Q1. What ActiveVision measures. Active observation itself is the capability we isolate, as stated in the title. It is not a single primitive but a series of capabilities, such as visual perception, reasoning and grounding, that the model must combine: look, form a hypothesis, and look again (Sec. 1). Prior benchmarks mainly test these capabilities separately; no previous benchmark focuses on their combination. We make it measurable, with exact, verifiable ground truth (Sec. 3.3). The core new conclusion is that models struggle with it: the best tool-free run reaches 23.5% vs. 96.1% for humans (Fig. 4), and a question-only control solves only 2/85 (Sec. 4.1). More reasoning does not close the gap (Opus 5 stays at 7.1% from low to max effort; Sec. 4.2), and tools close it only in part (best agent: 68.2%; Sec. 4.3).

W2. Attribution to active observation. As defined in the paper (Sec. 1), active vision is a series of actions, perceptions and decisions; these are all the substrate of this capability, not alternatives. The scope of this paper is to test this series in a verifiable, end-to-end test bed, so failures in perception and object individuation are all counted as failures of this behavior. The only true alternative is that the model transcribes the image into a pure text description once and reasons only over the text. We take steps to prevent this alternative and force iterative perception (Sec. 3.2).

Q2. Fixed-observation comparison. Our experimental design deliberately prevents fixed perception from succeeding (Sec. 3.2). We study how well models perform active observation; we do not aim to justify why it is needed. That justification comes from the psychology literature (Sec. 2) and from real applications that rely on this capability (Sec. 1). A matched-budget control is also hard to define for closed APIs: no model-independent measure of observation exists, and crops or tool calls also change the reasoning load.

Q3. Render verification. All images generated by GPT-image-2 were validated by the authors, and then given to three human participants, who scored 96.1% (Fig. 4). During the check, we discarded re-renders that dropped or duplicated items or broke the topology. All three participants solved 76 of the 85 items, and every item was solved by at least one participant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.