acceptodds
Under review as a conference paper at ICLR 2027

Amaze: Diagnosing Visual-Spatial Reasoning with Matched Maze Twins

Abstract

Vision-language models (VLMs) often fail on visual problems that require extended spatial computation, even when they can solve an equivalent symbolic problem. We introduce AMAZE , a diagnostic benchmark of procedurally generated mazes varying in scale, geometry, and visual texture. Its central design is a matched-twin protocol: each solvable maze is paired with a minimally modifiedunsolvable twin, and a model receives credit only if it classifies both correctly. This exposes a strong bias toward predicting that a path exists. To distinguish failures of visual access from failures of evidence integration, we equip agents with image-processing tools and compare monolithic execution with an orchestrated system of context-isolated specialists using local programmatic verification. Across 16 maze layouts and 1,600 matched pairs, the multi-step composition of low-level visual tools lifts paired accuracy from 12.9% to 31.6% for a monolithic agent. Orchestration with local verification further raises it to 67.6%. When a global reachability tool collapses connectivity reasoning to a single call, tools alone lift the monolithic agent to 75.8% and orchestration adds a smaller gain to 87.3%. The same pattern transfers to MentisOculi (1,000 puzzles), where accuracy rises from 58.0% without tools to 71.1% with a monolithic tool-using agent and 73.6% with orchestration. These results show that while visual tools yield large gains across visual reasoning tasks, tool access alone does not eliminate solvability bias: reliable integration of negative visual evidence is critical, and decentralization helps when that integration requires extended composition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Multimodal Reasoning, Spatial Reasoning, Visual Scratchpad, Test-Time Compute Scaling, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tShkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a visual maze solvability benchmark AMAZE consisting of paired solvable and unsolvable mazes with minimal variations. Results reveal significant performance degradation with increasing maze scale, and the paper shows that a multi-agent tool-use system with verifiers outperforms a single tool-use agent on this benchmark. The findings also transfer to another puzzle benchmark MentisOculi.

Critical strengths

(1) The idea of creating pairs of solvable and unsolvable mazes with minimal variations is insightful, and the result that models collapse with maze size is revealing: even if oracle ASCII is provided, model performance still degrades significantly with larger maps. The AMAZE benchmark thus contributes clearly to the visual spatial reasoning community and encourages more exploration of visual spatial reasoning fidelity in more complex scenes.

(2) The analyses and comparisons of different agentic approaches, along with their components, are thoroughly thought out and designed.

Critical weaknesses

(1) The fact that a multi-agent framework with verifiers outperforms a vanilla tool-use agent is an expected outcome. We know very well that these multi-agent components work better, as shown by the success of modern coding agents, and we naturally expect the case to be similar for visual maze solving. So it is unclear what exactly the comparison in the main results of Sec. 5 is trying to convey, and what it uniquely contributes.

(2) Judging from the prompt in Sec. B1, the performance may rely heavily on how well a VLM identifies where the green and red markers are in the maze image. As the maze scales up in size, these markers become less obvious and harder to spot, in which case the VLM’s performance degradation might simply be due to inaccurate start/end-point localization rather than any maze-related spatial logic.

Questions for authors

(1) Related to weakness (2), the authors should delve into the marker localization aspect and check how much it affects the outcome.

(2) The benchmark is designed only for yes/no answers, and it would help interpret the performance if the actual reasoning and tool-use chains behind the binary decision were presented. How often does the model or system answer correctly but with false reasoning or tool-use chains?

2 comments
AuthorsOpenReview ↗

Response to reviewer tShk

We thank the reviewer for their constructive feedback in response to their questions:

Contribution beyond existing multi-agent systems

We agree that delegation and verification are established techniques, particularly in coding agents. Our contribution is not a novel multi-agent architecture, but an investigation of whether and how these techniques address the specific difficulties of multi-step visual reasoning. Our experiments show that verification primarily improves the detection of unsolvable mazes, rather than uniformly improving both classes. Unlike code execution, visual tool use produces intermediate image artefacts that may distort important spatial information, accumulate as misleading evidence, or be difficult for the agent to validate. We study context-isolated execution and programmatic verification as interventions targeting these failure modes, and have clarified this motivation and positioning in the revised manuscript.

Marker localisation

We agree that identifying small start/end markers could contribute to performance degradation as maze size increases. Our oracle-ASCII experiments eliminate the need for visual marker localisation, but paired accuracy still declines substantially with maze size.

To investigate this further, we conducted an additional Gemini 3.5 Flash experiment on an alternate version of AMAZE v1 with fixed start/end positions. Paired accuracy increases from 52.3% to 78.2% at 5×5 and from 7.5% to 25.8% at 7×7, confirming that endpoint positioning contributes to difficulty. However, accuracy remains only 3.4% at 9×9 and near zero at larger scales, suggesting that localisation alone cannot explain the degradation.

We additionally investigated whether start/end markers can be recovered from the visual representations of Qwen3.8-27B. We rendered identical images with and without markers and compared their vision-encoder outputs. We trained a linear probe on 20 mazes to classify image tokens as start, end, or background. On 20 held-out mazes, the probe correctly localised the start/end markers in 90–100% of cases from 5×5 onward. This suggests that marker information can be recovered from the visual representations, although it does not establish whether the language model actually successfully uses this information to reason about solvability. Together, these findings suggest that marker localisation contributes to the difficulty, but does not fully account for the observed failure of visual spatial reasoning.

We thank the reviewer for their constructive feedback. As the discussion period draws to a close, we would appreciate knowing whether our author response has addressed your questions or if any concerns remain that we could clarify.

Reviewer_M13sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AMAZE, a test set that pairs solvable mazes with nearly identical unsolvable versions. It compares models working without tools, one agent using tools, and teams of agents with checks on their work. The team with checks performs much better on the maze pairs, but its gains are smaller and less consistent on other spatial puzzles.

Critical strengths

The maze pairs make a strong test. Adding a small barrier changes whether the maze can be solved while leaving most of the image unchanged. A model must answer both versions correctly to get credit. This exposes models that tend to say a path exists without checking carefully. Giving models a correct text version of the maze also helps separate errors in reading the image from errors in finding a path.

The paper separates tools that help inspect an image from tools that can directly check for a path. This mechanism avoids claiming that a model reasons better just because a tool solves the maze for it. With basic image tools, the team with checks scores much higher, compared against just one agent. Most of the gain comes from recognizing unsolvable mazes. Also, I like the tests with Flash-Lite and other spatial puzzles, which help show where this approach does and does not work.

Critical weaknesses

I lean toward acceptance because the maze test is useful and the gains are large. I am less convinced that splitting work across agents is the main reason for those gains. A single agent with the same checks and control over its history would make the comparison clearer.

The team with checks, Multi+Ver, can use a Python checker that the single agent cannot use. Appendix D explains why giving the single agent unrestricted Python would let it solve the maze directly. But the single agent could still call the same limited checker used by the team. Also, lowering its step limit does not test whether removing outdated images from its history would help. The results show that the full system works. However, it does not show whether separate specialist agents are needed.

The benefit beyond mazes is less clear. The tool-use tests cover Flash and Flash-Lite, not models from different families. On MentisOculi, adding checks raises the team score. Appendix F reports that this difference is not statistically significant. The team with checks also uses over a million input tokens in total per puzzle, compared with 500K for the single agent. The approach helps on some tasks, but the results do not show a general gain in efficiency.

Questions for authors

How does one agent perform when it can call the same separate checker, with the same inputs and limits? What if it can also remove or summarize failed attempts? This would show whether the gains require specialist teams or mainly come from checking work and keeping a cleaner history.

How do you prevent the checker from solving the maze itself? Can you provide execution records showing that checks in the basic-tool setting do not run a hidden path solver? The checker cannot submit the final answer, but it could still pass that answer through its feedback.

Does the gain with basic image tools hold for a strong non-Gemini model at the same compute cost? This would show whether the problem and proposed fix extend beyond the tested model family.

2 comments
AuthorsOpenReview ↗

Response to Reviewer M13s

We thank the reviewer for their constructive feedback. In response to their questions:

Necessity of specialisation

Replacing the five specialised roles with a single general-purpose specialist role with access to all visual tools improves paired accuracy from 67.6% to 77.0%. Several factors may explain this result. First, our specialists differ primarily in their assigned tools and instructions, rather than possessing independently learned expertise. Second, the fixed partitioning of tools may introduce unnecessary handoffs when tasks require composing operations across specialist boundaries. Finally, the unified-specialist configuration preserves delegated execution and separate working contexts, suggesting these properties may matter more than predefined specialist roles. Overall, our results suggest that flexible tool access within delegated, context-isolated execution may be preferable to rigid role specialisation in this setting.

Context management

We evaluated history truncation and explicit artefact removal as alternative single-agent context-management strategies. Neither improved upon the original baseline (31.6%), suggesting that simple history pruning is insufficient. We provide the experimental details in our response to Reviewer bnUh.

Computational efficiency

Our additional analyses reveal an accuracy–compute trade-off: Multi+Ver performs better under constrained visual-tool-call budgets, while Multi-noVer performs better under tight token budgets. We report the detailed comparisons in our response to Reviewer bnUh.

Generalization

We agree that the overall improvement on MentisOcul is not statistically significant. However, our task-level analysis in Appendix F reveals differences consistent with the nature of the tasks. Verification is beneficial in Rush Hour (75.5% for Multi-NoVer versus 79.3 for Multi+Ver), where invalid moves can compromise subsequent states but provides limited benefit on tasks such as Form Board and Hinge folding, where failures primarily arise from combinatorial search exhaustion. On PaperFold, Single achieves 96.1%, leaving little room for improvement. We will clarify that the observed benefits are task-dependent rather than universal.

We also conducted a preliminary evaluation with GPT-5.6 Sol on 25 maze pairs per layout, obtaining 28.0% without tools, 30.8% with Single, 35.2% with Multi-NoV, and 94.0% paired accuracy with Multi+Ver. While encouraging, this pilot is too small to establish generalisation.

Verifier restrictions

The verifier is tasked to check the work from the Specialist, and its context does not include any reference to solving the maze task itself. However, we agree that unrestricted Python execution could allow the verifier to perform connectivity computations, potentially confounding the architectural comparison. Therefore, we conducted additional experiments in which library routines for flood fill, connected-component and region labeling, watershed segmentation, and graph/shortest-path computation were blocked in the code execution sandbox. We provide the experimental details in our response to Reviewer bnUh. We observe that even with this constrained verifier, verification substantially benefits both architectures, but Multi+Ver retains a 15.0 percentage-point advantage over Single+Ver.

We thank the reviewer for their constructive feedback. As the discussion period draws to a close, we would appreciate knowing whether our author response has addressed your questions or if any concerns remain that we could clarify.

Reviewer_bnUhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AMAZE, a diagnostic benchmark of procedurally generated mazes varying in scale, geometry, and wall texture. Each solvable maze is paired with a nearly pixel-identical unsolvable twin, and a model receives credit only if it classifies both correctly. Zero-shot experiments on five VLMs show that paired accuracy collapses as mazes grow and that models strongly default to predicting that a path exists. The authors then compare a monolithic tool-using agent with a hierarchical multi-agent system of context-isolated specialists, with or without programmatic verification, given the same image-processing tools. Multi-agent execution with verification performs best, mainly by better detecting unsolvable mazes.

Critical strengths

The "matched-pair" maze design is highly ingenious. Maze tasks present a meaningful challenge, effectively testing a vision model's observational and reasoning capabilities. Furthermore, mazes have definitive, objective answers and can be easily generated at scale. Paired designs are effective for revealing biases in a model’s responses.

The empirical analysis is highly thorough. Through detailed experiments and data analysis, the study clarifies the performance boundaries of the model on visual tasks, providing a more precise assessment of its perceptual and reasoning capabilities. Furthermore, the methodological exploration is significant; analyses conducted at both the token and tool-use levels facilitate a deeper understanding of the model’s behavior—specifically its patterns in invoking tools.

Critical weaknesses

The introduction of a verifier increases computational cost, which somewhat compromises the fairness of the comparison.

Section 4.3 explicitly states that the verifier cannot invoke a "reachability solver," yet the mechanism for enforcing this restriction remains unclear. This is problematic because the verifier is capable of executing arbitrary Python code and comes with pre-loaded libraries; it could either utilize existing libraries or implement its own Breadth-First Search (BFS) algorithm and pass the results to the orchestrator.

Given the current configuration, a Python-based verifier agent could theoretically implement a BFS algorithm to solve the problems. Although the authors claim that the role-separation mechanism within the "Multi+Ver" (Multi-agent + Verifier) setup prevents this, the paper suggests that the mechanism only restricts the verifier from submitting the final answer—it does not prevent the verifier from computing the solution and passing it to the orchestrator. Since the inclusion of the verification stage yields the most significant performance gains, it is essential to audit the verifier’s code and provide experimental results for scenarios where these capabilities are disabled, in order to substantiate the paper’s core claims.

Minor comments:

The text and thumbnails in Figure 3 are too small and difficult to discern.

Subfigure (b) should not include the "flood-fill" agent, as it is not utilized in the primary experimental setup. Moreover, the figure fails to adequately illustrate the verifier’s feedback mechanism or the "matched-pair" maze configuration—both of which are central contributions of the paper.

The design of the single-agent baseline could be further strengthened. An ablation study (c = 50) indicates that reducing the context length alone yields a performance improvement. By introducing context management for single-agent setups and ensuring computational parity, the authors can better disentangle "context isolation" from "multi-agent task decomposition," thereby further substantiating the paper’s core conclusions.

The authors identify the "matched-pair" metric as a contribution; however, prior work on "paired evaluation"—its applications and efficacy—should be discussed. For instance, the design of NaturalBench is quite similar to the authors' proposed approach. Furthermore, that study notes a distinct bias in VLMs toward affirmative responses [1], which overlaps with the findings here. Similarly, the use of minor perturbations in maze tasks aligns with the "contrast sets" methodology, which also adheres to the principle of "minimal perturbation" [2]. The comparisons involving "image-text mazes" in SpatialEval are also relevant to the analysis concerning "oracle-ASCII" representations [3].

[1] NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples. NeurIPS 2024 [2] Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of EMNLP 2020 [3] Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models. NeurIPS 2024

Questions for authors

Can the verifier construct solutions in code and thereby circumvent the restriction against calling a reachability solver? Could the authors provide a detailed explanation of the underlying mechanism? How frequently are techniques such as "flood fill," "connected components," or "breadth-first search (BFS)" employed within the verifier's code? Furthermore, could the authors provide experimental results for the "Multi+Ver" approach with these specific functions disabled? This would help strengthen the authors' central claims.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bnUh

We thank the reviewer for their constructive feedback, in response to their questions:

Verifier restrictions

We agree that unrestricted Python execution could allow the verifier to perform connectivity computations, potentially confounding the architectural comparison. To investigate this, we conducted additional experiments in which library routines for flood fill, connected-component and region labeling, watershed segmentation, and graph/shortest-path computation were blocked. We report results on AMAZE V2 below (Gemini 3.5 Flash):

Tool suiteLayoutsNo toolsSinglenew Single+Ver(restricted)Multi-noVernew Multi+Ver(restricted)Multi+Ver
Atomic (main)Square (5)13.641.662.251.679.874.8
Hexagonal (5)8.420.850.046.667.458.2
Polar (3)15.025.755.038.767.068.7
Triangular (3)17.038.761.748.371.370.3
Overall (16)12.931.656.947.071.967.6

Under these restrictions, Multi+Ver achieves 71.9% paired accuracy, compared with 67.6% in the original setting. Performance is comparable or higher across all four maze geometries, indicating that access to these library routines does not explain the gains from verification. We also evaluated the previously omitted Single+Ver baseline under the same restrictions. It achieves 56.9%, compared with 31.6% for Single and 47.0% for Multi-noVer. Thus, verification substantially benefits both architectures, but Multi+Ver retains a 15.0 percentage-point advantage over Single+Ver.

Stronger single-agent baseline and context management

We evaluated two additional context-management strategies to determine whether improved handling of intermediate visual artefacts could strengthen Single. First, retaining only the most recent one or two intermediate images (while preserving the original input) reduces paired accuracy from 31.6% to 17.8% and 18.4%, respectively. Second, allowing the agent to explicitly remove obsolete artefacts using a remove_from_inventory tool yields 22.2%. We observed that the agent rarely chooses to remove unsuccessful intermediate outputs. These findings indicate that neither simple history pruning nor explicit artefact removal improves the single-agent baseline, although more sophisticated context-management strategies remain an open comparison.

Compute comparison

The original experiments allocate similar maximum interaction budgets, but Single voluntarily terminates before exhausting its budget. We therefore conducted additional experiments examining accuracy under constrained interaction budgets. With a limit of 20 visual tool calls, Single, Multi-noVer, and Multi+Ver achieve 15.6%, 25.0%, and 45.6% paired accuracy, respectively. However, with a 500K-token limit, the same configurations achieve 11.8%, 28.6%, and 6.8%. We additionally constrained Multi+Ver to three orchestrator turns and five turns per specialist (matching approximately the number of turns Single uses on average), obtaining 64.0% paired accuracy compared with 67.6% originally. These results show that Multi+Ver retains a considerable advantage under restricted visual-tool budgets and reduced interaction depth, but its token costs remain substantially higher. We will make this trade-off explicit in the revised paper.

Figure 3

We have revised the figure to improve legibility, remove the flood-fill specialist from the atomic-tool illustration, show the verification feedback loop, and reference Figure 1(a) for matched-pair construction.

Paired evaluation

We agree that paired evaluation and minimal contrastive perturbations are established techniques. We have expanded Related Work to discuss NaturalBench, contrast sets, and SpatialEval and revised our contribution statement to emphasise that AMAZE's application of procedurally generated solvable/unsolvable twins applies to multi-step visual reasoning.

Thank you for the additional experiments and detailed response. The experiments conducted by the authors have thoroughly addressed my questions, and the results clearly support the claims made in the paper.

Regarding the issue of the verifier going out of bounds, the additional experiments provided by the authors have largely alleviated my concerns. As stated in the paper, the design of the verifier is acceptable.

The results reported by the authors indicate that standard context management techniques do not necessarily help the model perform better on maze-solving tasks.

The comparison of computational resources highlights a drawback of the multi-agent+verifier approach: the high token consumption is an issue that may warrant further attention.

Overall, I believe the authors' constructive responses have addressed most of my concerns and I am inclined to improve my overall assessment of the paper.

We thank the reviewer for their continued engagement and for raising their score. We address the remaining points below.

Verifier restrictions

We agree that blocking library functions does not stop a verifier from writing its own reachability code. To check how often this happens, we searched the verifier's execution traces for graph-search code patterns; 16.3% of Python calls in the AMAZE V2 evaluation were flagged. Inspecting these showed that connectivity is computed for different purposes, such as checking an intermediate representation or a proposed path. We then traced the inputs of the flagged calls: 71.7% operate on artefacts produced by specialists, 21.1% only on the original maze or representations derived from it, and 7.2% could not be traced. Of the calls on specialist outputs, 84.8% inspect thresholded images. Since even computations on intermediate representations can reveal whether the maze is solvable, we cannot attribute the gains on the atomic suite solely to verification of intermediate artefacts.

In the flood-fill control setup, however, all architectures have access to connectivity tools, yet verification improves paired accuracy from 81.0% to 87.3%. On MentisOculi Rush Hour, verification improves accuracy from 75.5% to 79.3% (p = 0.027), where it can check intermediate vehicle movements and collisions rather than maze reachability. Other MentisOculi tasks show smaller or no gains.

We will clarify the verifier's computational capabilities and report these trace analyses and ablations in the revised paper.

Compute comparison

To clarify our previous response, the results under token caps were obtained through a post-hoc analysis of existing runs, rather than new executions with those caps enforced. We computed the proportion of successful episodes at each token cap, treating episodes exceeding the cap as truncated. Multi+Ver's accuracy therefore drops substantially under these retrospective constraints.

In our previous response, we also reported a new experiment with Multi+Ver under reduced interaction budgets (3 orchestrator turns and 5 turns per specialist). In this setting, where the constraints are known to the system during execution, Multi+Ver's accuracy decreases much less, from 67.6% to 64.0%.

For the new Single+Ver(restricted) and Multi+Ver(restricted) experiments, we additionally report token usage below:

Tool suiteArch.Orch. (K)Sub. (K)Verif. (K)Out (K)Vis. callsAcc. pair
AtomicSingle+Ver (restricted)607—19681656.9
Multi+Ver (restricted)42250476131071.9

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.