- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ActiveVision, a benchmark for evaluating active observation in multimodal large language models. It contains 17 tasks across distributed scanning, sequential traversal, and visual attribute transfer, designed to require repeated visual inspection rather than one-shot image understanding. Experiments show a large gap between frontier MLLMs and humans, while increased reasoning effort and vision-related tool use only partially improve performance.
Critical strengths
- Well-motivated problem formulation. The paper isolates active observation as a distinct capability that is not well captured by existing multimodal benchmarks.
- Carefully designed benchmark. The 17 tasks cover complementary abilities such as exhaustive scanning, sequential tracing, and fine-grained visual comparison, while reducing simple linguistic shortcuts.
- Strong diagnostic experiments. The paper evaluates both increased reasoning effort and tool-using agents, providing useful evidence that the observed failures are not easily resolved by longer reasoning or external vision code.
Critical weaknesses
- Limited attribution to active observation. The observed undercounting and early traversal errors do not uniquely indicate a lack of active observation, as they may also stem from lower-level perception, object individuation, or path-tracing failures. The current analysis does not disentangle these alternative causes. For example, Figure 6, in Bounded Face Counting, the model analyzes the relevant region but misidentifies nodes and edges, leading to an incorrect count (12 vs. 10), which is more naturally explained as a perceptual error than a failure to observe.
- Small evaluation set. ActiveVision contains only 85 evaluation items, with five examples per task. This relatively small scale may reduce the statistical robustness of the results and make per-task conclusions sensitive to individual examples.
- Synthetic image distribution. Although the images are photorealistically re-rendered, they are generated from procedural scaffolds rather than collected from real-world data. This raises questions about how well the findings generalize to natural visual environments.
Questions for authors
see above
Thank you for the careful review, and for your positive view of the problem formulation, the task design and the diagnostic experiments.
W1. Perception error vs. active observation. As defined in the paper, active vision is a series of actions, perceptions and decisions, and these are all the substrate of this capability. The scope of this paper is to test this series in a verifiable test bed, so failures in perception and object individuation are all counted as failures of this behavior. The only true alternative is that the model transcribes the image into a pure text description once and reasons only over the text. We take steps to prevent this alternative and force iterative perception (Sec. 3.2).
W2. Data scale. We do not view the dataset size as a weakness. The 85 items already give clear separation among strong agents (24.7% to 68.2%) and informative differences across the three task families (Fig. 5b), while the per-item runtime and cost (Table 1) make much larger agentic evaluations nontrivial. This is a reasonable balance between discriminative power and evaluation cost, consistent with the scale of widely used agent benchmarks such as OSWorld 2.0 and Terminal-Bench 2.0. The gap to humans is also far beyond sampling noise: the best tool-free model solves 20/85 items, with a 95% Wilson interval of 15.8% to 33.6%, against 96.1% for humans.
W3. Image distribution. We agree, but this is a trade-off. In exchange, we get the exact ground truth, and we can scale the benchmark. Our re-rendering also goes beyond line drawings: it adds realistic texture, lighting and clutter while the answers stay known. This realism matters, because the agents' classical vision tools break on these textures. Finally, non-real images are common in this line of work: BlindTest is fully synthetic, and most BabyVision images are also not real photographs but puzzle-style graphics (e.g., mazes, line tracing, find-the-difference).