- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MNIST-PRO, a diagnostic benchmark that reformulates MNIST digit recognition as a partially observable sequential decision-making task. Agents must actively explore hidden images, maintain perceptual states from limited visual observations, and integrate collected evidence to recognize one or two digits. The authors evaluate ten multimodal models across different memory representations and agent harness configurations. Through controlled experiments, they identify three major bottlenecks: insufficient evidence collection, ineffective integration of partial observations, and errors in downstream visual interpretation. The results highlight substantial performance degradation under partial observability and demonstrate how external visual state reconstruction can alleviate some of these limitations.
Critical strengths
-
Clean and interesting research question. The paper uses a simple, controlled environment to study active perception and working memory, isolating these capabilities from the complexities of embodied control. This provides a clear and focused setting for investigating agentic perception.
-
Meaningful diagnostic findings. The paper identifies three bottlenecks in evidence collection, perceptual state construction, and downstream interpretation. The controlled experiments, particularly visual canvas reconstruction and predictor swapping, provide useful empirical insights into why multimodal agents struggle under partial observability.
Critical weaknesses
-
Limited novelty over prior work. Sequential visual exploration, partial observability, and memory-based state construction have been extensively studied in prior work, including Recurrent Models of Visual Attention [1] and ActiView [2]. While MNIST-PRO provides a controlled diagnostic setting, the paper does not sufficiently establish what fundamentally new capabilities or insights it reveals beyond existing active perception benchmarks.
-
Limited generalizability (major concern). All experiments are conducted on simplified, deterministic MNIST environments. It remains unclear whether the identified bottlenecks generalize to complex natural images or real-world settings. In particular, the substantial improvements from visual canvas reconstruction may rely on the simplicity of 2D image stitching, which does not reflect the challenges of spatial state construction in realistic environments. This significantly limits the broader implications of the findings.
[1] Mnih et al., Recurrent Models of Visual Attention, NeurIPS 2014.
[2] Wang et al., ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models, ACL 2025.
Questions for authors
-
What fundamentally distinguishes MNIST-PRO from existing work on active visual perception and sequential glimpse-based recognition [1, 2]? Could the authors clarify what new scientific insights this benchmark reveals that previous approaches have not already established?
-
Can the authors provide evidence that the three identified bottlenecks also arise in more complex natural images or real-world embodied environments?
[1] Mnih et al., Recurrent Models of Visual Attention, NeurIPS 2014.
[2] Wang et al., ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models, ACL 2025.