- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a visual maze solvability benchmark AMAZE consisting of paired solvable and unsolvable mazes with minimal variations. Results reveal significant performance degradation with increasing maze scale, and the paper shows that a multi-agent tool-use system with verifiers outperforms a single tool-use agent on this benchmark. The findings also transfer to another puzzle benchmark MentisOculi.
Critical strengths
(1) The idea of creating pairs of solvable and unsolvable mazes with minimal variations is insightful, and the result that models collapse with maze size is revealing: even if oracle ASCII is provided, model performance still degrades significantly with larger maps. The AMAZE benchmark thus contributes clearly to the visual spatial reasoning community and encourages more exploration of visual spatial reasoning fidelity in more complex scenes.
(2) The analyses and comparisons of different agentic approaches, along with their components, are thoroughly thought out and designed.
Critical weaknesses
(1) The fact that a multi-agent framework with verifiers outperforms a vanilla tool-use agent is an expected outcome. We know very well that these multi-agent components work better, as shown by the success of modern coding agents, and we naturally expect the case to be similar for visual maze solving. So it is unclear what exactly the comparison in the main results of Sec. 5 is trying to convey, and what it uniquely contributes.
(2) Judging from the prompt in Sec. B1, the performance may rely heavily on how well a VLM identifies where the green and red markers are in the maze image. As the maze scales up in size, these markers become less obvious and harder to spot, in which case the VLM’s performance degradation might simply be due to inaccurate start/end-point localization rather than any maze-related spatial logic.
Questions for authors
(1) Related to weakness (2), the authors should delve into the marker localization aspect and check how much it affects the outcome.
(2) The benchmark is designed only for yes/no answers, and it would help interpret the performance if the actual reasoning and tool-use chains behind the binary decision were presented. How often does the model or system answer correctly but with false reasoning or tool-use chains?
2 comments
We thank the reviewer for their constructive feedback. As the discussion period draws to a close, we would appreciate knowing whether our author response has addressed your questions or if any concerns remain that we could clarify.
Response to reviewer tShk
We thank the reviewer for their constructive feedback in response to their questions:
Contribution beyond existing multi-agent systems
We agree that delegation and verification are established techniques, particularly in coding agents. Our contribution is not a novel multi-agent architecture, but an investigation of whether and how these techniques address the specific difficulties of multi-step visual reasoning. Our experiments show that verification primarily improves the detection of unsolvable mazes, rather than uniformly improving both classes. Unlike code execution, visual tool use produces intermediate image artefacts that may distort important spatial information, accumulate as misleading evidence, or be difficult for the agent to validate. We study context-isolated execution and programmatic verification as interventions targeting these failure modes, and have clarified this motivation and positioning in the revised manuscript.
Marker localisation
We agree that identifying small start/end markers could contribute to performance degradation as maze size increases. Our oracle-ASCII experiments eliminate the need for visual marker localisation, but paired accuracy still declines substantially with maze size.
To investigate this further, we conducted an additional Gemini 3.5 Flash experiment on an alternate version of AMAZE v1 with fixed start/end positions. Paired accuracy increases from 52.3% to 78.2% at 5×5 and from 7.5% to 25.8% at 7×7, confirming that endpoint positioning contributes to difficulty. However, accuracy remains only 3.4% at 9×9 and near zero at larger scales, suggesting that localisation alone cannot explain the degradation.
We additionally investigated whether start/end markers can be recovered from the visual representations of Qwen3.8-27B. We rendered identical images with and without markers and compared their vision-encoder outputs. We trained a linear probe on 20 mazes to classify image tokens as start, end, or background. On 20 held-out mazes, the probe correctly localised the start/end markers in 90–100% of cases from 5×5 onward. This suggests that marker information can be recovered from the visual representations, although it does not establish whether the language model actually successfully uses this information to reason about solvability. Together, these findings suggest that marker localisation contributes to the difficulty, but does not fully account for the observed failure of visual spatial reasoning.