acceptodds
Withdrawn from ICLR 2027

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic perception requires agents in partially observable environments to coordinate active sensing with working memory to construct and maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this capability because they couple active vision with complex embodied navigation and control dynamics. To isolate this capability, we introduce MNIST-PRO, which converts digit recognition into a sequential search task with controlled access to past glimpses. Evaluating ten multimodal models across in-context memory representations reveals that accuracy drops from up to 99.0% on unmasked images to between 7.0% and 43.5% under partial observability. We identify three bottlenecks in the perception loop. First, agents often fail to search thoroughly, making premature guesses based on partial strokes instead of fully utilizing their exploration budget. Second, even when agents uncover most digit strokes, they struggle to combine separate glimpses in memory, a limitation that visual canvas reconstruction overcomes by recovering over 50 percentage points on two-digit tasks. Third, even with near-complete coverage on an assembled canvas, agents still make visual interpretation errors, showing that downstream recognition remains an independent hurdle. Additionally, equipping agents with persistent memory in a harness fails to improve accuracy, instead causing them to waste steps repeatedly re-checking strokes they have already seen.

Primary area:
datasets and benchmarks
Keywords:
Agentic Perception, Multimodal Agents, Working Memory, Partial Observability, Visual State Tracking, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BqBxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MNIST-PRO, a diagnostic benchmark that reformulates MNIST digit recognition as a partially observable sequential decision-making task. Agents must actively explore hidden images, maintain perceptual states from limited visual observations, and integrate collected evidence to recognize one or two digits. The authors evaluate ten multimodal models across different memory representations and agent harness configurations. Through controlled experiments, they identify three major bottlenecks: insufficient evidence collection, ineffective integration of partial observations, and errors in downstream visual interpretation. The results highlight substantial performance degradation under partial observability and demonstrate how external visual state reconstruction can alleviate some of these limitations.

Critical strengths
  1. Clean and interesting research question. The paper uses a simple, controlled environment to study active perception and working memory, isolating these capabilities from the complexities of embodied control. This provides a clear and focused setting for investigating agentic perception.

  2. Meaningful diagnostic findings. The paper identifies three bottlenecks in evidence collection, perceptual state construction, and downstream interpretation. The controlled experiments, particularly visual canvas reconstruction and predictor swapping, provide useful empirical insights into why multimodal agents struggle under partial observability.

Critical weaknesses
  1. Limited novelty over prior work. Sequential visual exploration, partial observability, and memory-based state construction have been extensively studied in prior work, including Recurrent Models of Visual Attention [1] and ActiView [2]. While MNIST-PRO provides a controlled diagnostic setting, the paper does not sufficiently establish what fundamentally new capabilities or insights it reveals beyond existing active perception benchmarks.

  2. Limited generalizability (major concern). All experiments are conducted on simplified, deterministic MNIST environments. It remains unclear whether the identified bottlenecks generalize to complex natural images or real-world settings. In particular, the substantial improvements from visual canvas reconstruction may rely on the simplicity of 2D image stitching, which does not reflect the challenges of spatial state construction in realistic environments. This significantly limits the broader implications of the findings.

[1] Mnih et al., Recurrent Models of Visual Attention, NeurIPS 2014.

[2] Wang et al., ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models, ACL 2025.

Questions for authors
  1. What fundamentally distinguishes MNIST-PRO from existing work on active visual perception and sequential glimpse-based recognition [1, 2]? Could the authors clarify what new scientific insights this benchmark reveals that previous approaches have not already established?

  2. Can the authors provide evidence that the three identified bottlenecks also arise in more complex natural images or real-world embodied environments?

[1] Mnih et al., Recurrent Models of Visual Attention, NeurIPS 2014.

[2] Wang et al., ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models, ACL 2025.

Reviewer_sPH5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper turns MNIST into an active search task. The agent only sees a 64×64 window of the canvas, moves it around, and has to name the digit (Level 1) or a pair of digits (Level 2) within a step budget. The authors vary how much visual history the agent keeps and compare several memory setups, from raw image history and text notes to environment-stitched canvases and a coding harness. Across ten models, accuracy drops from up to 99% on full images to 7.0–43.5% under the in-context memory setups in Table 1. They trace this drop to three things: agents stop too early, they can't combine glimpses, and they misread what they've seen. They also find that letting the agent keep notes across episodes in the harness doesn't improve accuracy and leads to more steps on Level 2.

Critical strengths
  1. The setup is simple and clean. Without navigation or physics, it's much easier to see where the search–memory–recognition loop breaks. The zero-lookback condition is also useful: as the paper notes, ActiView and similar benchmarks don't test it.
  2. The offline-canvas replay and the predictor swap are the right idea for pulling exploration apart from interpretation, though each still needs a control (see Weaknesses 2 and 3).
  3. It's broad and well documented: ten models, several memory setups, two ablations, and the full prompts.
  4. I found the Claude-5-Opus/Fable result striking. Under Textual State they see over 80% of the strokes yet score 13–18% on Level 2, and jump to 67–69% once the same glimpses are stitched.
Critical weaknesses
  1. The agent isn't told the rules that matter for early stopping. The prompts (A.3/A.4) never give the step budget, and the Level 2 prompt uses "58" as an example but never says there are exactly two digits, which undercuts the "fully transparent" claim in Sec. 1. So some of the early stopping and one-digit answers may come from the prompt, not from poor search. A run with the budget and digit count stated would settle this.
  2. The offline canvas changes more than the representation. If the canvas is given in a new query, the model also drops its long multi-turn context, including its earlier notes. That alone could help, since the paper's own audit finds that a wrong digit, once written down, stays in the final answer 88.2% of the time. Feeding the same unstitched glimpses in a new query would show how much of the canvas gain comes from stitching itself.
  3. The predictor swap may just reflect a recognition gap: Gemini-3.7-Flash already beats GPT-5.6-Sol(xhigh) on full images (98/97 vs 92/79). That's awkward for Sec. 2, which rules out recognition because models reach "up to 99.0%" on full images. Accuracy-vs-coverage curves on oracle canvases would show whether partial canvases are hard even for models near ceiling on full images.
  4. Some numbers don't match.
    • Figure 3b vs. Table 2: averaging Table 2 the way the caption describes reproduces six of the eight models, but gives 66.5 for Claude-5-Fable, not 73.5 (+37.0 rather than +44.0), and 54.5 for GPT-5.6-Sol(xhigh), not 56.5.
    • Appendix B says proprietary models get 92.0–98.0% on full images, but Table 3 shows 72.5% and 85.0% for GPT-5.6-Terra and GPT-5.6-Sol.
    • Table 10: the Level 1 native 66.5 seems to be the mean of 68.0 and 65.0 from Table 2, i.e. two conditions pooled without saying so, and over those 200 episodes 84.3% isn't a possible accuracy.
  5. There are no error bars. Each cell is 100 episodes with no repeated runs reported, so a single accuracy near 50% has a 95% CI of about ±10 points. Yet Sec. 3.2 draws conclusions from 3-point gaps (56 vs 53, 54 vs 57) without any test. Please add CIs or repeated runs.
  6. The persistent-memory result is thin: mainly one model (Gemini-3.7-Flash, 75.5 without memory vs 73.5 with), and no reported reruns over episode orders even though the notes build up across episodes. It's also unclear what persists: Sec. 3.3.4 says "notes and scripts," while E.2 says memory "retains instructions rather than executable files." If it's just NOTES.md, the claim should be scoped to that.
  7. There's no baseline for the search itself. A fixed raster scan, run at a few step budgets and read by the same predictor, would show how much adaptive search actually adds, and it's cheap to run here. Classic glimpse models on MNIST (Mnih et al., 2014; Ba et al., 2015) are another natural reference point, and a paper titled "MNIST is Back" should discuss them.
Questions for authors
  1. In the offline-canvas condition, is the canvas given in a new query or appended to the conversation?
  2. What exactly is in context for native multi-turn vs. Textual State with |Hv|=∞? Gemini-3.7-Flash gets 75/47 with the first and 64/23 with the second, though they seem to carry similar information.
  3. A.5 has the model update its coordinates by one unit per move, but A.3 silently clips moves at the border. How much of the grid-map drift comes from that?
  4. Which models go into Figure 4 and the 1,600-trajectory audit? Working back from Table 1, Figure 4 seems to leave out all GPT models.
  5. How was the audit in Sec. 3.3.1 done, and how do you decide a glimpse "directly contradicted" the stored guess (88.2%)?
  6. For the failed harness episodes, how often does the model get them right from a programmatic canvas of the same observed regions, and from the full image?
Reviewer_VhbLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the active perception capabilities of VLMs in partially observable visual environments, focusing on how models gather evidence through continuous local observations, maintain working memory, integrate spatial information, and ultimately complete visual recognition. The authors compare multiple visual state construction mechanisms and various models, using full-image recognition accuracy as a baseline for passive visual recognition capabilities. In my view, the experiments only demonstrate that externalized spatial representations and tool-assisted reconstruction can improve the performance of specific models on simplified visual tasks, and cannot be generalized to the perception capabilities of general-purpose agents.

Critical strengths

This paper primarily focuses on the capability gap between static visual recognition and active perception, and constructs the task on MNIST. This design facilitates the analysis of the specific behavior of the model under constrained observation conditions. In the experiments, the paper attempts to distinguish different sources of failure through interventional experiments and finds that explicit spatial memories have inconsistent impacts across different models, suggesting that the form, accuracy, and utilization of memory can all affect agent performance.

Critical weaknesses

I believe the primary weakness of this paper lies in the overgeneralization of its conclusions. Although MNIST-PRO can measure a certain degree of spatial tracking and visual information integration, significant discrepancies remain compared to the complex perceptual problems faced by agents, such as lighting variations and environmental noise. The initial observation window is anchored to the digit strokes, which reduces the difficulty of discovering targets from completely unknown areas. Furthermore, the different memory configurations are not fully matched. Performance variances might stem from spatial alignment mechanisms or tool usage capabilities rather than the memory module itself. The MNIST-PRO benchmark is built on binarized MNIST handwritten digit images and includes only two tasks: single-digit recognition and recognition of two horizontally arranged digits. The significant performance gains brought by the external visual canvas reconstruction in the paper may partly rely on task characteristics such as the regular structure of MNIST digits, clean backgrounds, and predictable spatial layouts. The contents of the paper fail to demonstrate whether the same state reconstruction mechanism remains effective in the presence of distractors or complex spatial relationships.

Questions for authors

Aside from the major concerns outlined in the weaknesses section, I have the following questions:

  1. Compared with existing benchmarks in active vision and partially observable decision-making, what capability dimensions does MNIST-PRO provide that could not be measured previously?
  2. Does the performance improvement brought by the external visual canvas rely on the special structure of MNIST images?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.