acceptodds
Under review as a conference paper at ICLR 2027

MentalBlackboard: Evaluating Spatial Visualization via Mathematical Transformations

Abstract

Spatial visualization is the mental ability to imagine, transform, and manipulate the spatial characteristics of objects and actions. This intelligence is a part of human cognition in which actions and perception are connected on a mental level. To explore whether state-of-the-art Vision-Language Models (VLMs) exhibit this ability, we develop MentalBlackboard, an open-ended spatial visualization benchmark for Paper Folding and Hole Punching tests within two core tasks: prediction and planning. Our prediction experiments reveal that models struggle to apply symmetrical transformations, even when they correctly predict the sequence of unfolding steps. Additionally, rotations pose a significant challenge to models' physical situational awareness. The planning task reveals limitations in models' ability to analyze symmetrical relationships and implement the multi-stage symmetry process, with Claude 4.6 achieving the highest planning score at an accuracy of 14%. The top-performing model, o3, attains a peak performance of 71.6% on the generalization task, which does not require spatial visualization but transfers spatial data; however, it achieves only 25% accuracy on text-based prediction tasks.

Primary area:
datasets and benchmarks
Keywords:
spatial visualization, spatial cognition, spatial reasoning, VLMs, MLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QKjdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MentalBlackboard, a benchmark for vision language models (VLMs), which is meant to assess spatial visualisation capabilities of VLMs. To this end, authors simulate a sheet of paper being folded (possibly many times) and (possibly) rotated afterwards. Then, hole-punching actions on the folded sheet are simulated. The task for the VLM is to, among others, predict locations and shapes of holes in the unfolded paper. Several VLMs are tested, including closed source and open source models. Different modalities are tested: video-based, image-based and text-based. The authors observe poor performance of all of the tested models on this task, with the models generally performing the best when their input is represented as text (with the best-performing model, OpenAI’s o3, achieving 25.07% success rate in such case). The authors claim that this low success rate is caused by deficiencies of spatial visualisation in VLMs and perform an ablation to justify this claim.

Critical strengths

The overall benchmark design is sound, with many possible tasks and modalities being tested. Furthermore, authors have conducted a “preliminary test” (described in Appendix A) meant to account for confounding factors, such as models having trivial issues with basic understanding of test instances (including testing their understanding of text, images and videos), which is a good addition to the benchmark. Human baseline is established (Appendix C) and compared to the performance of OpenAI o3 and Claude Opus 4.1, showing visible advantage of humans over VLMs in this benchmark, which strengthens the findings of this paper. The ablation meant to check for the capability of spatial information transfer is a good idea, although I am not sure about its methodology (see Question 1).

Critical weaknesses

While the general design of the study is sound, I believe that its findings are not particularly novel. For instance, the fact that VLMs such as GPT-4o struggle on simple tasks requiring a basic degree of spatial understanding is already demonstrated in [A], with [B, C] (cited by the authors) further demonstrating spatial reasoning issues in VLMs. Differences in VLM performance are also not particularly surprising, with [D] already noting that providing textual description in IQ tests can improve the model’s performance. The spatial information transfer ablation seems to be more novel, but even if such mechanism was never previously documented (which I am not aware of, but I have not performed a literature search for it) I don’t think it’s a particularly significant contribution. The benchmark idea is not particularly novel as well, as there are many other benchmarks concerned with spatial reasoning (cited by the authors); while the paper tries to differentiate itself by claiming to focus on symmetry and rotation (Table 1 and L128-L132), I don’t think it’s enough, especially given that (using authors’ own classification from Table 1) each of symmetry/memory/rotation categories is already represented by at least one of the other benchmarks.

As such, I have to unfortunately recommend rejection of this paper.

[A] Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024, December). Vision language models are blind. In Asian Conference on Computer Vision (pp. 293-309). Singapore: Springer Nature Singapore. [B] Wang, S., Pei, M., Sun, L., Deng, C., Li, Y., Shao, K., ... & Wang, J. (2026, April). Spatialviz-bench: A cognitively-grounded benchmark for diagnosing spatial visualization in mllms. In International Conference on Learning Representations (Vol. 2026, pp. 47210-47279). [C] Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024). Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14455-14465). [D] Zhang, Y., Bai, H., Zhang, R., Gu, J., Zhai, S., Susskind, J., & Jaitly, N. (2024). How far are we from intelligent visual deductive reasoning?. arXiv preprint arXiv:2403.04732.

Questions for authors
  1. In section 6.1, why is the generalization task limited to three configurations (one-fold, two-folds, and one-fold-one-rotation)? Intuitively, I think it would be optimal to do such a test on exactly the original benchmark fold settings so that the results can be directly compared to the main results reported in Table 2. Otherwise, I cannot say for certain how 25.07% of exact match for OpenAI o3 corresponds to 71% this model achieves during the ablation.
  2. Suggestion: I do appreciate the fact that the benchmark is not overcomplicated while not being trivial at the same time, with all tested VLMs failing. Perhaps it could be reframed to focus more on qualitative evaluation of VLMs, including expanding the spatial information transfer study?
10 comments
AuthorsOpenReview ↗

Response to Reviewer QKjd (part 1)

We thank the reviewer for the insightful comments on our paper. We are pleased to see that our work, MentalBlackboard, is recognized as a solid benchmark design, with many tasks and modalities. We are also pleased that you recognize the value of our additional experiments, including the preliminary tests, human–model performance comparison, and generalization task.

“GPT-4o struggle on simple tasks requiring a basic degree of spatial understanding is already demonstrated in [A]”

We would like to clarify an important distinction between spatial understanding and spatial transformation.[A] primarily evaluates low-level visual perception of spatial information in static images, such as recognizing overlaps and intersections or counting visual elements, and does not evaluate the same capability targeted by MentalBlackboard. In contrast, MentalBlackboard evaluates spatial visualization through transformation, requiring models to mentally manipulate an object's spatial state through sequential symmetry and rotation operations and track how its spatial state changes through this sequence of operations. Thus, while [A] demonstrates limitations in perceiving and decoding spatial information already present in an image, our work investigates whether models can maintain and transform spatial information to derive a new, unobserved spatial state.

“[B, C] demonstrating spatial reasoning issues in VLMs.”

We agree with the reviewer that symmetry, rotation, and memory individually are not new components in spatial reasoning benchmarks. However, we respectfully disagree with the characterization that these components constitute the primary distinction of MentalBlackboard from previous benchmarks. Table 1 compares the benchmarks along several dimensions, including the evaluation methodology, dimension, data size, and transformation composition. Datasets in Table 1 present the spatial visualization tasks through static 2D images. The continuous physical transformations are not observable, and the model must infer these transformations to understand what happened between static 2D states. In contrast, MentalBlackboard represents the spatial visualization task in three forms: text, 2D image, and video, where the physical folding transformation becomes perceptually visible with continuous motions. This distinction allows MentalBlackboard to examine whether failures arise when transformations must be mentally inferred from static states versus when the transformation trajectory itself is visually available. By introducing a fully 3D simulation environment, we model real-world applications with spatial and physical features of objects and actions. This makes our benchmark better reflect complex spatial and physical interactions found in real-world tasks.

The most important difference between MentalBlackboard and other benchmarks in the paper is the evaluation method. MentalBlackboard uses open-ended structured outputs rather than relying on multiple-choice alternatives, unlike all other datasets. This is important because these benchmarks only provide whether answers are correct or not. This multiple-choice method limits models to fixed responses and lacks deeper reasoning analysis. Providing the correct answer does not mean that the model applies the correct reasoning steps to solve the problem. Most of these benchmarks employ a four-option multiple-choice format, establishing a 25% chance-level accuracy and allowing models to obtain correct responses without necessarily demonstrating the underlying spatial reasoning process. Besides, as we mentioned in the paper, a model can use different shortcut strategies to reach the correct answer without a reasoning process, such as eliminating the candidates, perceptually matching the most visually similar option, and testing each candidate backward to fit the question (Snow et al., 1980). However, by employing an open-ended format, MentalBlackboard offers diagnostic evaluation of which transformation processes fail, under which conditions they fail, and whether the failure arises from sequencing, symmetry execution, rotation, perspective, or reverse planning. While other datasets only enable aggregated model accuracy for spatial visualization tasks, MentalBlackboard localizes bottlenecks causing failures, characterizes the model behaviors for their weaknesses and strengths, and offers guidance for model improvement. Because existing benchmarks do not provide ground truth for intermediate reasoning steps, their evaluations are primarily limited to measuring overall model performance rather than diagnosing where and why failures occur within the reasoning process. Therefore, while existing benchmarks reveal whether models struggle with spatial reasoning, MentalBlackboard goes beyond performance measurement by localizing failures within the transformation process and identifying the specific conditions under which these failures emerge.

AuthorsOpenReview ↗

Response to Reviewer QKjd (part 2)

What weaknesses MentalBlackboard reveals that was missed by previous work?

MentalBlackboard provides multiple levels of quantitative and qualitative analyses that identify the main failures of current VLMs. Below, we provide the diagnostic results MentalBlackboard reveals were missed by previous spatial visualization works.

  1. Transformation selection vs. execution: Models may identify the correct unfolding operation but fail to execute the corresponding hole transformation. MentalBlackboard separates these failures: o3 drops from 52% to 24% when applying the exact unfolding transformations in video prediction (Figure 2).

  2. Fold-depth reasoning: Models often apply symmetry without considering folded-layer depth, producing holes where no underlying paper exists. Extra-hole error rates reach 12.78% for o3 and 16.89% for Claude Opus 4.1 (Appendix D.3).

  3. Transformation-specific weaknesses: MentalBlackboard reveals model-specific deficits hidden by overall rankings. o3 exceeds 90% on one-step horizontal & vertical unfolds but achieves only 6.35% on diagonal unfolds, while Claude Opus 4.1 reaches 45% on diagonal but 0% on horizontal unfolds (Tables 14–15).

  4. Rotation sensitivity: Models struggle to account for orientation changes introduced by rotation. For o3, unfolding accuracy drops from 90.91% to 28.57% on horizontal folds and from 100% to 37.50% on vertical folds when rotation is introduced (Table 14).

  5. Field-wise spatial deficits: Shape and size are generally easiest to predict, while location is the main bottleneck, followed by direction (Table 2). GPT-5.4 performs best on shape, size, and direction, whereas o3 leads on location. Final-answer accuracy hides these attribute- and model-specific deficits.

  6. Planning failure strategies: MentalBlackboard exposes model-specific planning errors, including too few folds, physically infeasible actions, incorrect layer reasoning, and incorrect initial-hole placement (Appendix G.2). For example, Qwen2.5VL uses one fold instead of two and compensates with four initial holes instead of two. Rendering the plans makes these failures directly observable.

  7. Formatting vs. reasoning: Output-format analysis separates instruction-following errors from reasoning failures (Appendix D.7). Strong models, including o3, GPT-5.1, GPT-5.4, Claude 4, and Claude 4.6, achieve 100% schema compliance despite incorrect spatial reasoning.

  8. Complexity-specific failures: Increasing transformation complexity changes not only accuracy but also the type of error. In Group 5 with rotation, Claude Opus 4.1 produces more extra holes, whereas o3 produces more missing holes (Appendix D.3). MentalBlackboard therefore reveals how failure mechanisms change with complexity.

  9. Prerequisite capability: Preliminary tests distinguish basic input understanding from spatial-reasoning failures (Appendix A). In the image-sequence test, all three models achieve 100% on structure, mark count, shape, spatial relations, and action recognition, indicating that later failures arise beyond basic perception.

  10. Modality sensitivity: Reasoning is weakest for continuous transformation representations. o3 achieves 25% on text, 19% on 2D images, and 15.8% on video, with spatial location easier to recover from text. MentalBlackboard therefore reveals how representation modality affects reasoning.

  11. Backward transformation: MentalBlackboard isolates partially observed, view-dependent transformations through backward folding. o3 achieves only 11.11% exact match and Claude Opus 4.1 only 0.56%, revealing severe perspective-dependent difficulties in continuous transformations (Table 16).

  12. Spatial knowledge vs. transformation: Models transfer spatial information much better than they execute transformations. o3 and Claude 4.6 solve about 71% of spatial-transfer problems, while the best prediction accuracy is only 25% and planning 14%. Thus, possessing spatial knowledge does not imply the ability to transform it correctly (Section 6.1).

  13. Error propagation: Simulator-based reconstruction visually traces how errors emerge and propagate across successive transformations (Appendix G). Unlike multiple-choice evaluation, it shows how an incorrect intermediate transformation leads to subsequent and final errors.

The main finding of the paper is that models are weak at spatial visualization, and previous benchmarks already present evidence for this observation. The findings of MentalBlackboard decompose the previously observed general “spatial reasoning weakness” into specific reasoning deficits, providing information about what aspects of spatial reasoning require improvement that cannot be obtained from final-answer accuracy alone. This distinction is central to our contribution.

AuthorsOpenReview ↗

Response to Reviewer QKjd (part 3)

[D] already noting that providing textual description in IQ tests can improve the model’s performance.

We would like to clarify that the purposes of [D] and MentalBlackboard are different. While [D] studies Raven’s Progressive Matrices and uses textual descriptions to investigate the perceptual bottleneck in visual deductive reasoning, MentalBlackboard does not simply provide a language description of the visual problem; it re-encodes the same spatial transformation into a structured symbolic representation, where the paper state and hole locations are represented through grid-based symbols and spatial coordinates. This allows us to compare reasoning over the same underlying transformations across continuous video, static 2D images, and symbolic text representations. More importantly, our analysis goes beyond observing that text produces higher overall accuracy by examining which spatial computations change across representations. For example, calculation of spatial location is better with textual representations. Thus, our finding is not simply that “text helps,” but that the representation of spatial information changes which components of the transformation process models can successfully compute, helping distinguish representation-related limitations from transformation-related failures, including symmetry execution, rotation, sequential transformation, physical layer reasoning, perspective changes, and spatial knowledge transfer.

The spatial information transfer seems to be more novel, but even if such mechanism was never previously documented, I don’t think it’s a particularly significant contribution.

The spatial information transfer experiment provides a controlled evaluation to determine whether poor spatial-visualization performance originates from an inability to represent and transfer spatial information or from difficulty executing the required mental transformations. The contribution of this experiment does not lie in the visual analogy framework itself, but in the diagnostic distinction it enables. The results show that models can successfully transfer spatial information at substantially higher accuracy than they can perform transformation-based prediction, indicating that the major bottleneck lies not in possessing spatial information, but in manipulating that information through sequential symmetry and rotation transformations. It provides one component of MentalBlackboard's broader diagnostic framework, together with its process-level, transformation-specific, representation-specific, and model-specific analyses.

Why is the generalization task limited to three configurations? I think it would be optimal to do a test on exactly the original benchmark fold settings so that the results can be directly compared to the main results in Table 2.###

Thank you for pointing out this comparability issue. The generalization task was designed to isolate spatial-information transfer rather than transformation complexity. In this task, models do not need to execute the unfolding and symmetry transformations themselves but instead infer and transfer the spatial relationship from the first scenario to the second. Therefore, increasing the number of folds, as in other groups, will just increase the number of images for inference, as mental transformations are not included. We agree that comparing the generalization results with the overall prediction results in Table 2 does not fully control for transformation complexity. For a fair comparison, we compare generalization with prediction using the same Groups 1, 2, and 5. While o3 achieves 71% on spatial information transfer, its exact-match accuracy decreases to 40.56% for text, 21.69% for 2D images, and 16.66% for video prediction, demonstrating that transferring spatial information is substantially easier than executing the mental transformations even under matched fold configurations.

Suggestion: I do appreciate the fact that the benchmark is not overcomplicated while not being trivial at the same time, with all tested VLMs failing. Perhaps it could be reframed to focus more on qualitative evaluation of VLMs, including expanding the spatial information transfer study?

We thank the reviewer for this insightful suggestion. We agree that qualitative evaluation is particularly valuable for identifying how and where spatial transformation processes break down, and this is an important component of MentalBlackboard. The current paper already includes simulator-based qualitative analyses of both prediction and planning failures in Appendix G, where model outputs are reconstructed and executed to visualize failures such as incorrect symmetry transformations, physical-layer errors, rotation-related failures, and the propagation of errors across successive transformations. Rather than only labeling a response as incorrect, these analyses make the underlying failure mechanism directly observable.

Reviewer_TmzgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MentalBlackboard is an open-ended benchmark for spatial visualization based on the Paper Folding and Hole Punching Test, combining symmetry, rotation, and action-tracking across prediction and planning tasks in video, 2D-image, and text modalities. Evaluated models show low exact-match accuracy (≤25% prediction, ≤14% planning), with failures traced to symmetry misapplication and rotation-induced disorientation. A generalization (analogy) task and backward-folding ablation are also provided.

Critical strengths

1.Open-ended evaluation avoids the shortcut strategies of multiple-choice benchmarks (Sec. 1), and the Partial Accuracy metric with over-prediction penalty (Sec. 4) enables fine-grained failure analysis.

2.The planning task's execution-based scoring (Sec. 5.2) — re-simulating model-proposed folds and comparing final hole patterns — correctly handles multiple valid solutions.

Critical weaknesses

1.Human baseline (App. D.1) covers only Groups 1/2/5 (≤2 steps), with 16 answers per category and no participant count, agreement statistics, or IRB mention for the MTurk study. The uniform 75% human score is anecdotal; extend to harder groups and report n and agreement. 2.No decoding parameters, seeds, or repeated runs are reported (Sec. 4); with exact-match accuracies of 1–8% on 315–900 items, single-run differences may not be meaningful. Report confidence intervals. 3.The ">1M questions / 12K configurations" claim (Sec. 3) contrasts with only 9 fold configurations (App. B) and ~1.8K evaluated problems; diversity comes mostly from hole attributes. Clarify what is released vs. evaluated. 4.No symbolic/programmatic baseline is included, although the simulator defines ground truth — a solver upper bound would validate solvability and calibrate difficulty.

Questions for authors

How many human participants were evaluated, what quality controls were applied, and was IRB approval or exemption obtained for the MTurk study? What decoding settings (temperature, max tokens) were used per model, and were results averaged over multiple seeds? Which models were run on which task/modality? Table 2's coverage across video/image/text appears uneven (e.g., Claude 4.6 and o3 in text-based prediction). Does planning evaluation enforce the required number of folding steps, or can a different step count reproducing the target succeed? Will the release include the VPython generation code and evaluation scripts, or only the rendered dataset?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Tmzg (part 1)

We appreciate your comprehensive evaluation of our paper. We are pleased that you acknowledge MentalBlackboard’s contribution in offering fine-grained failure analysis with open-ended evaluation by avoiding shortcut problem-solving strategies. We are pleased that you recognize the importance of simulation-based evaluation for planning tasks. Below, we provide detailed responses to your specific inquiries:

Human participants, quality controls and IRB

Thank you for raising these concerns regarding the human baseline. We agree that the current human evaluation is limited in scale and was intentionally restricted to simpler configurations of Groups 1, 2, and 5 due to the substantial time required for manual annotation of more complex multi-step folding problems. However, it covered four tasks: video prediction, image prediction, text prediction, and planning, involving 16 participants with 64 distinct questions. The purpose of this experiment was to provide a controlled human reference on a representative subset, including both folding-only and folding-with-rotation cases. Humans and models were evaluated on the same subset using identical evaluation metrics to ensure a fair comparison. We also acknowledge that extending the human evaluation to Groups 3–9 would provide a more comprehensive analysis of how the human–model gap changes with increasing transformation complexity and identify this as a limitation.

Each participant was allowed to complete the HIT only once, preventing repeated participation, and recruitment was restricted to participants with Master’s or PhD-level education. All participants provided informed consent prior to participation. No response-time restriction was imposed. In addition, responses were collected through a structured interface to reduce invalid or ambiguous answers. For each prediction problem, participants first specified the number of resulting holes, after which the corresponding fields for shape, size, direction, and location were displayed. Each attribute was restricted to the valid values defined by the benchmark (e.g., directions of 0°, 90°, 180°, or 270° and locations from 1–32), not a text input. For planning, participants similarly specified the required folds in order and the corresponding hole attributes. Responses were therefore constrained to the same output space provided to the models through the structured JSON instructions. After collection, human responses were compared directly with the deterministic ground truth using the same evaluation metrics applied to model outputs. These controls standardized the response format and prevented invalid or ambiguous outputs from affecting the human–model comparison. For IRB, the study underwent institutional screening and was determined not to involve human-subjects research.

Decoding settings, multiple seeds and confidence intervals

We agree that repeated runs can provide additional information about inference variability. MentalBlackboard evaluates a large number of proprietary models across computationally expensive multimodal inputs, including videos and sequences of images, with hundreds of test instances per evaluation condition. Because of substantially increasing inference costs, we use a single run per model.

For proprietary models, we used the default decoding settings provided by their respective APIs without explicitly specifying temperature or maximum output tokens. Qwen-3.5-9B was served locally using vLLM with a maximum model context length of 262,144 tokens and thinking mode disabled.

For confidence interval calculation, we used a 95% Wilson score for exact-match accuracy, treating each video prediction problem as a binary correct & incorrect outcome. The results show that for video-based prediction, most of the models’ intervals overlap, but o3 is separated from many of the lower-performing models.

ModelExact Match (%)95% CI (%)
Qwen3.5-9B1.781.10–2.87
Claude Opus 4.11.220.68–2.18
Claude 41.560.93–2.59
Claude 4.63.672.62–5.10
LLaVA-OneVision0.000.00–0.43
GPT-4o0.330.11–0.98
GPT-5.11.000.53–1.89
GPT-5.44.673.47–6.25
o38.006.40–9.96

No symbolic/programmatic baseline is included, although the simulator defines ground truth

MentalBlackboard uses a deterministic, rule-based simulator in which folding, rotation, and unfolding operations are explicitly implemented through geometric reflection and rotation transformations in a Python-based simulation environment. The simulator applies these transformation rules to generate the ground-truth hole configurations, ensuring that every generated problem is physically valid and has a programmatically computable solution. Therefore, the simulation-based execution system already serves as a programmatic oracle, both generating the problems and computing their ground-truth solutions.

AuthorsOpenReview ↗

Response to Reviewer Tmzg (part 2)

The ">1M questions / 12K configurations" claim (Sec. 3) contrasts with only 9 fold configurations (App. B) and ~1.8K evaluated problems; diversity comes mostly from hole attributes. Clarify what is released vs. evaluated.”

Thank you for pointing out this ambiguity. We clarify that 9 configurations in Appendix B refer to folding-sequence structures (Groups 1–9) rather than nine individual folding configurations. In total, we have 8 folding configurations, and with backward direction, it becomes 16 different folding cases. Each group in Appendix B generates many valid combinations of specific folding types and directions; for example, for one sequence of actions, there are 16 distinct cases, which is equal to the number of folds. Across these groups, the action types show the configuration settings, where F represents paper folding, and R represents rotation. When we combine 16 folds, which are Groups 1, 2, 3, and 4, in total, the generation pipeline produces more than 12K valid folding configurations without rotation included. When we add rotation (Groups 1 to 9), the total number of distinct configurations is 69600. These configurations are further combined with different hole attributes (shape, size, direction, and location), resulting in a potential problem space of more than 1M questions. The >1M figure therefore describes the generation capacity of MentalBlackboard, not the number of problems evaluated in our experiments. We evaluated ~1.8K problems, which has 100 question per group in the prediction task. These configurations are randomly selected and combined with random hole attributes.

Which models were run on which task/modality? Table 2's coverage across video/image/text appears uneven (e.g., Claude 4.6 and o3 in text-based prediction).

Table 2 evaluates different model sets because model capabilities differ across input modalities. Models o3, GPT-5.1, GPT-5.4, GPT-4o, Qwen3.5-9B, Claude 4, Claude Opus 4.1, and Claude 4.6, were evaluated across all representations: video, 2D-image, and text prediction. As DeepSeeks-R1, Nemotron-Nano-9B, and Qwen3-8B are text-only models, we evaluated them only for the text prediction task. As LLaVa-OneVision is a vision model, we evaluated it on image and video tasks. As Gemma-3-12B accepts image format but not video, we evaluated it only for the image prediction task.

Does planning evaluation enforce the required number of folding steps, or can a different step count reproducing the target succeed

The planning task enforces the required number of folding steps for planning the solution. It is provided as part of the input specifically to prevent simple solutions using fewer folds. However, we do not require the predicted folding sequence to exactly match the ground-truth sequence, since multiple valid sequences can produce the same target output. The predicted plan is executed in our simulation environment, and it is considered successful if it uses the required number of folding steps and produces the target hole configuration. Although instructed, some open-source models produce a plan with a one-step solution by avoiding the required number of folds.

Will the release include the VPython generation code and evaluation scripts, or only the rendered dataset?

The VPython-based simulation code for generating the problems, together with the evaluation scripts and test set, will be publicly released. Also, the scripts for generating the video, 2D-image, and textual representations will be released. Researchers can customize folding and rotation configurations to generate new problems tailored to their specific objectives. Thus, MentalBlackboard provides a flexible dataset-generation environment rather than only a fixed benchmark dataset.

Reviewer_ckxYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MentalBlackboard, which is based on the paper-folding test, where you fold, rotate, and punch holes in a sheet of paper, and then ask the model about the results. They have two modes: prediction, which involves predicting the outcome of the actions, and planning, which involves finding the correct actions to reach a certain outcome. The authors specify a setting for the tests that allows them to automatically render examples and generate questions based on the selected actions. Finally, they evaluate a variety of models on this benchmark and show their shortcomings on these tasks, followed by additional studies involving in-context examples and investigating the effect of different input representations.

Critical strengths
  1. The paper introduces a large, open-ended spatial reasoning dataset in multiple formats, with high-complexity problems that can be generated synthetically. The planning mode is also interesting, as it adds another layer of complexity. Its evaluation accepts different valid solutions by executing the proposed plans and checking their outcomes. The authors have done a good job of evaluating a wide range of models under different settings.

  2. The presentation is clear and easy to follow. The details for the dataset generation and the experiments are included, which makes it easier to reproduce the results.

Critical weaknesses
  1. The main issue of this work, in my opinion, is that the new lessons learned from this paper are not clear. Many papers, such as “Mind the Gap,” “OrigamiSpace,” and “Paper Folding Puzzles” (the last two are not cited in the paper, but they should be), focus on closely related problems and show that spatial reasoning is a major weakness. I do understand that MentalBlackboard has open-ended questions and includes a large number of samples, but the paper does not demonstrate an actual benefit of these features that is not present in previous work. I believe the paper requires a clear distinction from previous work and a discussion of its purpose and benefits for this field. To be clear, by distinction, I mean a particular weakness that MentalBlackboard reveals that was missed by previous work, or a particular advantage that MentalBlackboard offers for improving spatial reasoning.

  2. The discussions at the end seem disjointed, without a clear message. Three separate studies are discussed in Section 6, but after reading them, it is not clear what their purpose or conclusions are. The paper requires a better and more thorough discussion. I do understand that the page constraint can limit the freedom to add discussion, but the paper needs to make room for and prioritize this, as the discussion currently seems a bit disconnected and disorganized.

Questions for authors
  1. In the opinion of the authors, what is the main advantage of MentalBlackboard for evaluating spatial reasoning compared with existing benchmarks?

  2. Could the authors please describe the main goal and lessons learned from each ablation study provided in Section 6, and explain how these findings connect to the paper’s overall contribution?

10 comments
AuthorsOpenReview ↗

Response to Reviewer ckxY (part 1)

Thank you for your insightful review of our paper, MentalBlackboard. We are pleased that you recognized our work as interesting, including high-complexity problems, and a comprehensive evaluation of a wide range of models under different settings and multiple representation formats. We are delighted that you recognize that the reproducibility of MentalBlackboard results is easy by following the details of the dataset generation pipeline and experiments. Below, we provide detailed responses to your specific inquiries:

In the opinion of the authors, what is the main advantage of MentalBlackboard for evaluating spatial reasoning compared with existing benchmarks?

Thank you for your excellent question, which reveals the significance of MentalBlackboard. The benchmarks in Table 1 and OrigamiSpace and Paper Folding Puzzles focus on spatial visualization with different tasks and configurations. What makes MentalBlackboard different than all these benchmarks is how it represents the spatial transformations in the task and how it evaluates the task.

Like all other datasets in Table 1, Mind the Gap, OrigamiSpace, and Paper Folding Puzzles datasets present the spatial visualization tasks through static 2D images. The continuous physical transformations are not observable, and the model must infer these transformations to understand what happened between static 2D states. In contrast, MentalBlackboard represents the spatial visualization task in three forms: text, 2D image, and video, where the physical folding transformation becomes perceptually visible with continuous motions. This distinction allows MentalBlackboard to examine whether failures arise when transformations must be mentally inferred from static states versus when the transformation trajectory itself is visually available. By introducing a fully 3D simulation environment, we model real-world applications with spatial and physical features of objects and actions. This makes our benchmark better reflect complex spatial and physical interactions found in real-world tasks.

The most important difference between MentalBlackboard and all other benchmarks mentioned above is the evaluation method. MentalBlackboard uses open-ended structured outputs rather than relying on multiple-choice alternatives, unlike all other datasets. This is important because these benchmarks only provide whether answers are correct or not. This multiple-choice method limits models to fixed responses and lacks deeper reasoning analysis. Providing the correct answer does not mean that the model applies the correct reasoning steps to solve the problem. Most of these benchmarks employ a four-option multiple-choice format, establishing a 25% chance-level accuracy and allowing models to obtain correct responses without necessarily demonstrating the underlying spatial reasoning process. Besides, as we mentioned in the paper, a model can use different shortcut strategies to reach the correct answer without a reasoning process, such as eliminating the candidates, perceptually matching the most visually similar option, and testing each candidate backward to fit the question (Snow et al., 1980). However, by employing an open-ended format, MentalBlackboard offers diagnostic evaluation of which transformation processes fail, under which conditions they fail, and whether the failure arises from sequencing, symmetry execution, rotation, perspective (visibility), or reverse planning. While other datasets only enable aggregated model accuracy for spatial visualization tasks, MentalBlackboard localizes bottlenecks causing failures, characterizes the model behaviors for their weaknesses and strengths, and offers guidance for model improvement. Because existing benchmarks do not provide ground truth for intermediate reasoning steps, their evaluations are primarily limited to measuring overall model performance rather than diagnosing where and why failures occur within the reasoning process. Therefore, while existing benchmarks reveal whether models struggle with spatial reasoning, MentalBlackboard goes beyond performance measurement by localizing failures within the transformation process and identifying the specific conditions under which these failures emerge.

OrigamiSpace and Paper Folding Puzzles will be included in the paper and in Table 1 as related works.

AuthorsOpenReview ↗

Response to Reviewer ckxY (part 2)

Purpose and benefits of MentalBlackboard for the field?

The purpose of the MentalBlackboard benchmark is to provide a diagnostic evaluation of spatial visualization tasks in VLMs with changing evaluation settings and various tasks and representations. Rather than only determining whether a model can solve a spatial visualization problem including different Math transformations, MentalBlackboard identifies where failures occur within a sequence of spatial transformations and under which conditions they emerge. Through its 3D simulation-based dataset generation pipeline, MentalBlackboard provides ground truth for intermediate transformation steps, enabling the model’s reasoning process to be evaluated at each stage. For the prediction task, this allows assessment of whether the model identifies the correct unfolding sequence and accurately predicts the resulting hole configuration. For the planning task, it enables evaluation of whether the model can determine the sequence of folding actions and initial hole attributes required to produce a specified final configuration. By systematically changing the evaluation conditions like folding direction, rotation, number of transformations, and input representation, MentalBlackboard distinguishes different sources of spatial reasoning failure. While multiple-choice evaluation can reveal that a model has a spatial reasoning weakness, it provides limited information about the source of that weakness. However, MentalBlackboard provides diagnostic evidence that enables researchers to identify the specific reasoning operations that require improvement and to develop more targeted approaches for strengthening spatial visualization capabilities.

Advantages MentalBlackboard offer for improving the spatial reasoning?

MentalBlackboard offers several improvements of models for spatial reasoning by providing diagnostic evaluation results rather than only final-answer correctness. MentalBlackboard reveals that these improvements in MLLMs are necessary for successful spatial visualization ability:

1- Improvement of applying symmetry transformations in a 3D environment,

2- Improvement of rotation awareness for understanding the object orientation in 3D.

3- Improvement of visuospatial working memory to update the spatial states across multi-step transformations

4- Improvement of spatial information integration to combine correct attributes for a coherent final configuration

5- Improvement of temporal transformation reasoning for better performance on continuous motion representations.

6- Improvements on target-based, model-specific cases such as diagonal symmetry for o3 and horizontal symmetry for Claude Opus 4.1.

MentalBlackboard can support future improvements in spatial reasoning in several ways. First, its automated generation pipeline allows researchers to create their own training or evaluation data by changing folding, rotation, hole, and complexity configurations, enabling targeted data generation for specific model weaknesses with more than 1 million questions. Second, this pipeline provides ground truth for intermediate unfolding steps, individual spatial attributes, transformation types, and increasing levels of transformation complexity. This information can be used to construct targeted training or fine-tuning data for specific weaknesses identified by the benchmark, such as symmetry transformations, rotation-aware reasoning, spatial location tracking, and multi-step transformations. Third, the simulation environment enables models to generate plans that can be executed and validated based on their resulting spatial configurations, providing an environment for testing and potentially refining spatial reasoning strategies. Fourth, the framework automatically creates three representations of the same underlying transformations, which can support multimodal or cross-representation training. Finally, the generalization task provides a separate setting for training and evaluating whether spatial information learned from one configuration can be transferred to another. Together, these capabilities allow MentalBlackboard to move from diagnosing spatial reasoning failures toward a flexible framework for generating targeted data and validation mechanisms for spatial plans and studying how spatial reasoning can be improved across transformations and representations.

AuthorsOpenReview ↗

Response to Reviewer ckxY (part 3)

What weaknesses MentalBlackboard reveals that was missed by previous work?

Although MentalBlackboard focuses on spatial visualization like other benchmarks in Table 1, the dataset creation pipeline and evaluation method differ from other benchmarks. To determine where a model fails or exhibits weaknesses, the task requires a relevant level of ground truth rather than a final answer. With increasing the granularity of ground truths in intermediate steps, we provide a more detailed and diagnostic evaluation of the model. Beyond reporting overall benchmark accuracy, MentalBlackboard provides multiple levels of quantitative and qualitative analyses that identify the main failures of current VLMs. Below, we provide the diagnostic results that MentalBlackboard reveals were missed by previous works.

  1. Transformation selection vs. execution: Models may identify the correct unfolding operation but fail to execute the corresponding hole transformation. MentalBlackboard separates these failures: o3 drops from 52% to 24% when applying the exact unfolding transformations in video prediction (Figure 2).

  2. Fold-depth reasoning: Models often apply symmetry without considering folded-layer depth, producing holes where no underlying paper exists. Extra-hole error rates reach 12.78% for o3 and 16.89% for Claude Opus 4.1 (Appendix D.3).

  3. Transformation-specific weaknesses: MentalBlackboard reveals model-specific deficits hidden by overall rankings. o3 exceeds 90% on one-step horizontal & vertical unfolds but achieves only 6.35% on diagonal unfolds, while Claude Opus 4.1 reaches 45% on diagonal but 0% on horizontal unfolds (Tables 14–15).

  4. Rotation sensitivity: Models struggle to account for orientation changes introduced by rotation. For o3, unfolding accuracy drops from 90.91% to 28.57% on horizontal folds and from 100% to 37.50% on vertical folds when rotation is introduced (Table 14).

  5. Field-wise spatial deficits: Shape and size are generally easiest to predict, while location is the main bottleneck, followed by direction (Table 2). GPT-5.4 performs best on shape, size, and direction, whereas o3 leads on location. Final-answer accuracy hides these attribute- and model-specific deficits.

  6. Planning failure strategies: MentalBlackboard exposes model-specific planning errors, including too few folds, physically infeasible actions, incorrect layer reasoning, and incorrect initial-hole placement (Appendix G.2). For example, Qwen2.5VL uses one fold instead of two and compensates with four initial holes instead of two. Rendering the plans makes these failures directly observable.

  7. Formatting vs. reasoning: Output-format analysis separates instruction-following errors from reasoning failures (Appendix D.7). Strong models, including o3, GPT-5.1, GPT-5.4, Claude 4, and Claude 4.6, achieve 100% schema compliance despite incorrect spatial reasoning.

  8. Complexity-specific failures: Increasing transformation complexity changes not only accuracy but also the type of error. In Group 5 with rotation, Claude Opus 4.1 produces more extra holes, whereas o3 produces more missing holes (Appendix D.3). MentalBlackboard therefore reveals how failure mechanisms change with complexity.

  9. Prerequisite capability: Preliminary tests distinguish basic input understanding from spatial-reasoning failures (Appendix A). In the image-sequence test, all three models achieve 100% on structure, mark count, shape, spatial relations, and action recognition, indicating that later failures arise beyond basic perception.

  10. Modality sensitivity: Reasoning is weakest for continuous transformation representations. o3 achieves 25% on text, 19% on 2D images, and 15.8% on video, with spatial location easier to recover from text. MentalBlackboard therefore reveals how representation modality affects reasoning.

  11. Backward transformation: MentalBlackboard isolates partially observed, view-dependent transformations through backward folding. o3 achieves only 11.11% exact match and Claude Opus 4.1 only 0.56%, revealing severe perspective-dependent difficulties in continuous transformations (Table 16).

  12. Spatial knowledge vs. transformation: Models transfer spatial information much better than they execute transformations. o3 and Claude 4.6 solve about 71% of spatial-transfer problems, while the best prediction accuracy is only 25% and planning 14%. Thus, possessing spatial knowledge does not imply the ability to transform it correctly (Section 6.1).

  13. Error propagation: Simulator-based reconstruction visually traces how errors emerge and propagate across successive transformations (Appendix G). Unlike multiple-choice evaluation, it shows how an incorrect intermediate transformation leads to subsequent and final errors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to Reviewers

We greatly appreciate the reviewers’ thoughtful and constructive feedback. We are pleased to note their unanimous positive assessments, which highlight the strengths of our work across multiple dimensions.

• Reviewer ckxY emphasizes the planning task as an interesting and challenging problem, and Reviewers ckxY and Tmzg acknowledge that the simulation-based evaluation of the planning task handles multiple valid solutions.

• Reviewer Tmzg acknowledges that the open-ended evaluation of MentalBlackboard avoids shortcut strategies, and the generated metrics enable fine-grained failure analysis.

• Reviewer QKjd acknowledges that the benchmark design is sound, and Reviewer ckxY emphasizes the reproducibility of the results with included details in dataset generation and experiments.

• Reviewer QKjd highlights that the conducted preliminary tests for evaluating the fundamental capabilities of the model and generalization tasks for assessing the capability of spatial information transfer are good additional studies of the benchmark.

• Reviewer QKjd highlights that the performance comparison of the human baseline and models shows the visible advantage of humans over VLMs in spatial visualization tasks.

• Reviewer ckxY and QKjd highlight that many tasks and modalities are evaluated on a wide range of models under different settings.

• Additionally, we are pleased that all reviewers described the presentation of our paper as clear and easy to follow.

We have addressed each reviewer’s comments in detail. Below, we summarize our responses to the key points and offer additional clarifications.

Clarification of MentalBlackboard’s Contributions

MentalBlackboard offers diagnostic evaluation of which transformation processes fail, under which conditions they fail, and whether the failure arises from sequencing, symmetry execution, rotation, perspective, or reverse planning. While other datasets only enable aggregated model accuracy for spatial visualization tasks, MentalBlackboard localizes bottlenecks causing failures, characterizes the model behaviors for their weaknesses and strengths, and offers guidance for model improvement. MentalBlackboard demonstrates the value of diagnostic evaluation by revealing failure mechanisms that aggregate accuracy cannot capture:

  1. Transformation selection vs. execution: Models can identify the correct unfolding operation but fail to execute it (o3: 52% → 24%).
  2. Fold-depth reasoning: Models apply symmetry without accounting for physical layer depth, producing extra holes (o3: 12.78%; Claude Opus 4.1: 16.89%).
  3. Transformation-specific deficits: Model weaknesses differ substantially by transformation; e.g., o3 performs >90% on horizontal & vertical but only 6.35% on diagonal unfolds.
  4. Rotation sensitivity: Orientation changes strongly degrade reasoning; o3 drops from 90.91% → 28.57% (horizontal) and 100% → 37.50% (vertical).
  5. Field-wise spatial deficits: Shape and size are generally easiest to predict, while location is the main bottleneck, with distinct field-wise strengths across models.
  6. Planning failures: Models exhibit distinct errors such as insufficient folds, physically infeasible actions, incorrect layer reasoning, and incorrect initial hole placement.
  7. Reasoning vs. formatting: Strong models achieve 100% schema compliance despite incorrect spatial predictions, separating reasoning failures from formatting errors.
  8. Complexity-specific failures: Increasing complexity changes failure types, with Claude producing more extra holes and o3 more missing holes under rotation.
  9. Prerequisite capability: Models can achieve 100% on basic perception tests including mark count, spatial relations, shape and action recognition while failing subsequent transformations, localizing errors to spatial reasoning.
  10. Modality sensitivity: The same task becomes harder across text, image, and continuous video representations (o3: 25%, 19%, 15.8%).
  11. Backward transformation: Partially observed backward transformations expose severe perspective-dependent difficulties (o3: 11.11%; Claude Opus 4.1: 0.56%).
  12. Spatial knowledge vs. transformation: Models transfer spatial information (~71%) far better than they execute prediction (≤41%) or planning (≤14%).
  13. Error propagation: Simulator reconstruction reveals how intermediate transformation errors propagate into subsequent steps and the final prediction.

Together, these findings demonstrate that MentalBlackboard contributes more than a new paper-folding benchmark: it provides a diagnostic evaluation that distinguishes what spatial information models have, which transformation processes fail, how failures vary across conditions and models, and how intermediate errors propagate to the final prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.