- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents PuzRian, a Gymnasium-based benchmark for testing visual-spatial reasoning in vision-language agents. It includes three types of puzzles with different levels of complexity and interaction. Nine models are evaluated, with results showing good performance on simple symbolic puzzles but significant difficulties with irregular shapes and visual fragments. The authors also examine how feedback, interaction limits, spatial transformations and error recovery affect performance.
Critical strengths
- Well-designed and flexible benchmark.
The main strength of PuzRian is its flexibility. The three puzzle types cover different levels of spatial complexity, while the configurable settings allow a closer look at how agents reason and interact. The process-level metrics are particularly useful for understanding failures beyond final success rates. The availability of the code is another positive aspect.
- Interesting experimental findings.
The evaluation of nine models highlights the gap between simple spatial tasks and more challenging geometric problems. The controlled experiments are also useful, particularly the finding that rotations and reflections cause more difficulty than simply increasing the number of available actions. The results on feedback and error recovery further show that agents often struggle to correct mistakes, even when given more opportunities to interact.
Critical weaknesses
- Handling of alternative valid solutions.
My main concern is how the benchmark handles puzzles with multiple valid solutions. As the authors acknowledge, Irregular Assembly may allow different correct arrangements, yet piece-level feedback appears to rely on a particular solution. This could incorrectly penalise valid placements and affect the feedback and recovery metrics. The authors should clarify how often this happens and how alternative solutions are handled.
- Limited discrimination at higher difficulty levels.
All nine models achieve zero success on Irregular-Hard, making it difficult to compare their capabilities. Additional intermediate difficulty levels would make the benchmark more informative. I also think the conclusions about the causes of failure could be more cautious, as perception and planning are difficult to separate. Finally, the ablation experiments use relatively few instances, and uncertainty estimates would help assess the reliability of the results.
Questions for authors
How does the benchmark handle multiple valid solutions in Irregular Assembly? Could these affect piece-level feedback and recovery metrics?
Could the authors include intermediate difficulty levels to better distinguish model performance, particularly for Irregular Assembly?
Could the authors report confidence intervals for the ablation results to show how consistent the observed differences are?
We thank the reviewer for their positive feedback and constructive suggestions.
We clarify that episode success accepts any feasible reconstruction, without requiring the model to match the sampled ground-truth solution. For piece-level feedback, although our environment supports comparing placements against the sampled ground truth, we intentionally use a less restrictive setting in the reported Irregular experiments. The feedback only checks whether each piece is entirely contained within the target shape, avoiding incorrectly penalizing potentially valid alternative placements. For process-level metrics such as Piece Accuracy and Recovery Rate, we allow swapping pieces with identical shapes rather than requiring exact piece-ID matching. We also analyzed solution multiplicity on the same evaluation instances. For Irregular-Medium, 86% of instances have a unique solution and 13% have two solutions. For Irregular-Hard, 58% have a unique solution and 18% have two solutions. We will clarify the feedback design and this limitation in the revised paper. We have clarified the success criterion and piece-level feedback design for Irregular Assembly in 3.2, and added a discussion of the limitations of reference-based process metrics in 3.1.
We thank the reviewer for this suggestion. We agree that 0 success rates on Irregular-Hard make it difficult to distinguish model capabilities. Although our process-level metrics provide some additional information, more fine-grained difficulty levels would be useful. Due to computational costs, we have not systematically evaluated all intermediate settings. We plan to include additional experiments with 30 instances per setting, covering 5×5 and 6×6 puzzles with and without required rotation, to better investigate performance across difficulty levels.
We thank the reviewer for this suggestion. We will include Standard Error of the Mean, which is n−1p^(1−p^) in the revised paper. Here we present in ablation Effect of Additional Actions as an example. As shown in the table, the performance of GPT-5.4 and Gemini-3.1 drops substantially when spatial transformations are required, consistent with our main observation.