- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MentalBlackboard, a benchmark for vision language models (VLMs), which is meant to assess spatial visualisation capabilities of VLMs. To this end, authors simulate a sheet of paper being folded (possibly many times) and (possibly) rotated afterwards. Then, hole-punching actions on the folded sheet are simulated. The task for the VLM is to, among others, predict locations and shapes of holes in the unfolded paper. Several VLMs are tested, including closed source and open source models. Different modalities are tested: video-based, image-based and text-based. The authors observe poor performance of all of the tested models on this task, with the models generally performing the best when their input is represented as text (with the best-performing model, OpenAI’s o3, achieving 25.07% success rate in such case). The authors claim that this low success rate is caused by deficiencies of spatial visualisation in VLMs and perform an ablation to justify this claim.
Critical strengths
The overall benchmark design is sound, with many possible tasks and modalities being tested. Furthermore, authors have conducted a “preliminary test” (described in Appendix A) meant to account for confounding factors, such as models having trivial issues with basic understanding of test instances (including testing their understanding of text, images and videos), which is a good addition to the benchmark. Human baseline is established (Appendix C) and compared to the performance of OpenAI o3 and Claude Opus 4.1, showing visible advantage of humans over VLMs in this benchmark, which strengthens the findings of this paper. The ablation meant to check for the capability of spatial information transfer is a good idea, although I am not sure about its methodology (see Question 1).
Critical weaknesses
While the general design of the study is sound, I believe that its findings are not particularly novel. For instance, the fact that VLMs such as GPT-4o struggle on simple tasks requiring a basic degree of spatial understanding is already demonstrated in [A], with [B, C] (cited by the authors) further demonstrating spatial reasoning issues in VLMs. Differences in VLM performance are also not particularly surprising, with [D] already noting that providing textual description in IQ tests can improve the model’s performance. The spatial information transfer ablation seems to be more novel, but even if such mechanism was never previously documented (which I am not aware of, but I have not performed a literature search for it) I don’t think it’s a particularly significant contribution. The benchmark idea is not particularly novel as well, as there are many other benchmarks concerned with spatial reasoning (cited by the authors); while the paper tries to differentiate itself by claiming to focus on symmetry and rotation (Table 1 and L128-L132), I don’t think it’s enough, especially given that (using authors’ own classification from Table 1) each of symmetry/memory/rotation categories is already represented by at least one of the other benchmarks.
As such, I have to unfortunately recommend rejection of this paper.
[A] Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024, December). Vision language models are blind. In Asian Conference on Computer Vision (pp. 293-309). Singapore: Springer Nature Singapore. [B] Wang, S., Pei, M., Sun, L., Deng, C., Li, Y., Shao, K., ... & Wang, J. (2026, April). Spatialviz-bench: A cognitively-grounded benchmark for diagnosing spatial visualization in mllms. In International Conference on Learning Representations (Vol. 2026, pp. 47210-47279). [C] Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024). Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14455-14465). [D] Zhang, Y., Bai, H., Zhang, R., Gu, J., Zhai, S., Susskind, J., & Jaitly, N. (2024). How far are we from intelligent visual deductive reasoning?. arXiv preprint arXiv:2403.04732.
Questions for authors
- In section 6.1, why is the generalization task limited to three configurations (one-fold, two-folds, and one-fold-one-rotation)? Intuitively, I think it would be optimal to do such a test on exactly the original benchmark fold settings so that the results can be directly compared to the main results reported in Table 2. Otherwise, I cannot say for certain how 25.07% of exact match for OpenAI o3 corresponds to 71% this model achieves during the ablation.
- Suggestion: I do appreciate the fact that the benchmark is not overcomplicated while not being trivial at the same time, with all tested VLMs failing. Perhaps it could be reframed to focus more on qualitative evaluation of VLMs, including expanding the spatial information transfer study?
10 comments
Response to Reviewer QKjd (part 2)
What weaknesses MentalBlackboard reveals that was missed by previous work?
MentalBlackboard provides multiple levels of quantitative and qualitative analyses that identify the main failures of current VLMs. Below, we provide the diagnostic results MentalBlackboard reveals were missed by previous spatial visualization works.
-
Transformation selection vs. execution: Models may identify the correct unfolding operation but fail to execute the corresponding hole transformation. MentalBlackboard separates these failures: o3 drops from 52% to 24% when applying the exact unfolding transformations in video prediction (Figure 2).
-
Fold-depth reasoning: Models often apply symmetry without considering folded-layer depth, producing holes where no underlying paper exists. Extra-hole error rates reach 12.78% for o3 and 16.89% for Claude Opus 4.1 (Appendix D.3).
-
Transformation-specific weaknesses: MentalBlackboard reveals model-specific deficits hidden by overall rankings. o3 exceeds 90% on one-step horizontal & vertical unfolds but achieves only 6.35% on diagonal unfolds, while Claude Opus 4.1 reaches 45% on diagonal but 0% on horizontal unfolds (Tables 14–15).
-
Rotation sensitivity: Models struggle to account for orientation changes introduced by rotation. For o3, unfolding accuracy drops from 90.91% to 28.57% on horizontal folds and from 100% to 37.50% on vertical folds when rotation is introduced (Table 14).
-
Field-wise spatial deficits: Shape and size are generally easiest to predict, while location is the main bottleneck, followed by direction (Table 2). GPT-5.4 performs best on shape, size, and direction, whereas o3 leads on location. Final-answer accuracy hides these attribute- and model-specific deficits.
-
Planning failure strategies: MentalBlackboard exposes model-specific planning errors, including too few folds, physically infeasible actions, incorrect layer reasoning, and incorrect initial-hole placement (Appendix G.2). For example, Qwen2.5VL uses one fold instead of two and compensates with four initial holes instead of two. Rendering the plans makes these failures directly observable.
-
Formatting vs. reasoning: Output-format analysis separates instruction-following errors from reasoning failures (Appendix D.7). Strong models, including o3, GPT-5.1, GPT-5.4, Claude 4, and Claude 4.6, achieve 100% schema compliance despite incorrect spatial reasoning.
-
Complexity-specific failures: Increasing transformation complexity changes not only accuracy but also the type of error. In Group 5 with rotation, Claude Opus 4.1 produces more extra holes, whereas o3 produces more missing holes (Appendix D.3). MentalBlackboard therefore reveals how failure mechanisms change with complexity.
-
Prerequisite capability: Preliminary tests distinguish basic input understanding from spatial-reasoning failures (Appendix A). In the image-sequence test, all three models achieve 100% on structure, mark count, shape, spatial relations, and action recognition, indicating that later failures arise beyond basic perception.
-
Modality sensitivity: Reasoning is weakest for continuous transformation representations. o3 achieves 25% on text, 19% on 2D images, and 15.8% on video, with spatial location easier to recover from text. MentalBlackboard therefore reveals how representation modality affects reasoning.
-
Backward transformation: MentalBlackboard isolates partially observed, view-dependent transformations through backward folding. o3 achieves only 11.11% exact match and Claude Opus 4.1 only 0.56%, revealing severe perspective-dependent difficulties in continuous transformations (Table 16).
-
Spatial knowledge vs. transformation: Models transfer spatial information much better than they execute transformations. o3 and Claude 4.6 solve about 71% of spatial-transfer problems, while the best prediction accuracy is only 25% and planning 14%. Thus, possessing spatial knowledge does not imply the ability to transform it correctly (Section 6.1).
-
Error propagation: Simulator-based reconstruction visually traces how errors emerge and propagate across successive transformations (Appendix G). Unlike multiple-choice evaluation, it shows how an incorrect intermediate transformation leads to subsequent and final errors.
The main finding of the paper is that models are weak at spatial visualization, and previous benchmarks already present evidence for this observation. The findings of MentalBlackboard decompose the previously observed general “spatial reasoning weakness” into specific reasoning deficits, providing information about what aspects of spatial reasoning require improvement that cannot be obtained from final-answer accuracy alone. This distinction is central to our contribution.
Response to Reviewer QKjd (part 3)
[D] already noting that providing textual description in IQ tests can improve the model’s performance.
We would like to clarify that the purposes of [D] and MentalBlackboard are different. While [D] studies Raven’s Progressive Matrices and uses textual descriptions to investigate the perceptual bottleneck in visual deductive reasoning, MentalBlackboard does not simply provide a language description of the visual problem; it re-encodes the same spatial transformation into a structured symbolic representation, where the paper state and hole locations are represented through grid-based symbols and spatial coordinates. This allows us to compare reasoning over the same underlying transformations across continuous video, static 2D images, and symbolic text representations. More importantly, our analysis goes beyond observing that text produces higher overall accuracy by examining which spatial computations change across representations. For example, calculation of spatial location is better with textual representations. Thus, our finding is not simply that “text helps,” but that the representation of spatial information changes which components of the transformation process models can successfully compute, helping distinguish representation-related limitations from transformation-related failures, including symmetry execution, rotation, sequential transformation, physical layer reasoning, perspective changes, and spatial knowledge transfer.
The spatial information transfer seems to be more novel, but even if such mechanism was never previously documented, I don’t think it’s a particularly significant contribution.
The spatial information transfer experiment provides a controlled evaluation to determine whether poor spatial-visualization performance originates from an inability to represent and transfer spatial information or from difficulty executing the required mental transformations. The contribution of this experiment does not lie in the visual analogy framework itself, but in the diagnostic distinction it enables. The results show that models can successfully transfer spatial information at substantially higher accuracy than they can perform transformation-based prediction, indicating that the major bottleneck lies not in possessing spatial information, but in manipulating that information through sequential symmetry and rotation transformations. It provides one component of MentalBlackboard's broader diagnostic framework, together with its process-level, transformation-specific, representation-specific, and model-specific analyses.
Why is the generalization task limited to three configurations? I think it would be optimal to do a test on exactly the original benchmark fold settings so that the results can be directly compared to the main results in Table 2.###
Thank you for pointing out this comparability issue. The generalization task was designed to isolate spatial-information transfer rather than transformation complexity. In this task, models do not need to execute the unfolding and symmetry transformations themselves but instead infer and transfer the spatial relationship from the first scenario to the second. Therefore, increasing the number of folds, as in other groups, will just increase the number of images for inference, as mental transformations are not included. We agree that comparing the generalization results with the overall prediction results in Table 2 does not fully control for transformation complexity. For a fair comparison, we compare generalization with prediction using the same Groups 1, 2, and 5. While o3 achieves 71% on spatial information transfer, its exact-match accuracy decreases to 40.56% for text, 21.69% for 2D images, and 16.66% for video prediction, demonstrating that transferring spatial information is substantially easier than executing the mental transformations even under matched fold configurations.
Suggestion: I do appreciate the fact that the benchmark is not overcomplicated while not being trivial at the same time, with all tested VLMs failing. Perhaps it could be reframed to focus more on qualitative evaluation of VLMs, including expanding the spatial information transfer study?
We thank the reviewer for this insightful suggestion. We agree that qualitative evaluation is particularly valuable for identifying how and where spatial transformation processes break down, and this is an important component of MentalBlackboard. The current paper already includes simulator-based qualitative analyses of both prediction and planning failures in Appendix G, where model outputs are reconstructed and executed to visualize failures such as incorrect symmetry transformations, physical-layer errors, rotation-related failures, and the propagation of errors across successive transformations. Rather than only labeling a response as incorrect, these analyses make the underlying failure mechanism directly observable.
Response to Reviewer QKjd (part 1)
We thank the reviewer for the insightful comments on our paper. We are pleased to see that our work, MentalBlackboard, is recognized as a solid benchmark design, with many tasks and modalities. We are also pleased that you recognize the value of our additional experiments, including the preliminary tests, human–model performance comparison, and generalization task.
“GPT-4o struggle on simple tasks requiring a basic degree of spatial understanding is already demonstrated in [A]”
We would like to clarify an important distinction between spatial understanding and spatial transformation.[A] primarily evaluates low-level visual perception of spatial information in static images, such as recognizing overlaps and intersections or counting visual elements, and does not evaluate the same capability targeted by MentalBlackboard. In contrast, MentalBlackboard evaluates spatial visualization through transformation, requiring models to mentally manipulate an object's spatial state through sequential symmetry and rotation operations and track how its spatial state changes through this sequence of operations. Thus, while [A] demonstrates limitations in perceiving and decoding spatial information already present in an image, our work investigates whether models can maintain and transform spatial information to derive a new, unobserved spatial state.
“[B, C] demonstrating spatial reasoning issues in VLMs.”
We agree with the reviewer that symmetry, rotation, and memory individually are not new components in spatial reasoning benchmarks. However, we respectfully disagree with the characterization that these components constitute the primary distinction of MentalBlackboard from previous benchmarks. Table 1 compares the benchmarks along several dimensions, including the evaluation methodology, dimension, data size, and transformation composition. Datasets in Table 1 present the spatial visualization tasks through static 2D images. The continuous physical transformations are not observable, and the model must infer these transformations to understand what happened between static 2D states. In contrast, MentalBlackboard represents the spatial visualization task in three forms: text, 2D image, and video, where the physical folding transformation becomes perceptually visible with continuous motions. This distinction allows MentalBlackboard to examine whether failures arise when transformations must be mentally inferred from static states versus when the transformation trajectory itself is visually available. By introducing a fully 3D simulation environment, we model real-world applications with spatial and physical features of objects and actions. This makes our benchmark better reflect complex spatial and physical interactions found in real-world tasks.
The most important difference between MentalBlackboard and other benchmarks in the paper is the evaluation method. MentalBlackboard uses open-ended structured outputs rather than relying on multiple-choice alternatives, unlike all other datasets. This is important because these benchmarks only provide whether answers are correct or not. This multiple-choice method limits models to fixed responses and lacks deeper reasoning analysis. Providing the correct answer does not mean that the model applies the correct reasoning steps to solve the problem. Most of these benchmarks employ a four-option multiple-choice format, establishing a 25% chance-level accuracy and allowing models to obtain correct responses without necessarily demonstrating the underlying spatial reasoning process. Besides, as we mentioned in the paper, a model can use different shortcut strategies to reach the correct answer without a reasoning process, such as eliminating the candidates, perceptually matching the most visually similar option, and testing each candidate backward to fit the question (Snow et al., 1980). However, by employing an open-ended format, MentalBlackboard offers diagnostic evaluation of which transformation processes fail, under which conditions they fail, and whether the failure arises from sequencing, symmetry execution, rotation, perspective, or reverse planning. While other datasets only enable aggregated model accuracy for spatial visualization tasks, MentalBlackboard localizes bottlenecks causing failures, characterizes the model behaviors for their weaknesses and strengths, and offers guidance for model improvement. Because existing benchmarks do not provide ground truth for intermediate reasoning steps, their evaluations are primarily limited to measuring overall model performance rather than diagnosing where and why failures occur within the reasoning process. Therefore, while existing benchmarks reveal whether models struggle with spatial reasoning, MentalBlackboard goes beyond performance measurement by localizing failures within the transformation process and identifying the specific conditions under which these failures emerge.