- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark called MENTISOCULI-3D, which aims to assess the spatial reasoning and prediction capabilities of current open-source and commercial multimodal LLMs. Specifically, the benchmark constructs a validation dataset based on verified pairs of questions designed to evaluate whether models can correctly track and reason about state changes in 3D environments. The experimental results show that GPT-6 Astra nearly solves the core benchmark under high reasoning effort, while seven evaluated open-source model configurations achieve substantially lower performance.
Critical strengths
- Evaluating the spatial reasoning and prediction capabilities of current open-source and commercial multimodal LLMs is an interesting and relevant research direction.
- The paired-question design is effective and provides a useful mechanism for assessing whether models genuinely rely on the relevant visual and spatial information.
- The evaluation covers a sufficiently broad range of open-source models, providing a useful comparison of their capabilities on the proposed benchmark.
Critical weaknesses
-
The motivation is not clearly presented. The Introduction is difficult to follow and does not clearly explain the research background, the limitations of existing benchmarks, or why a dedicated 3D benchmark is needed.
-
The novelty over the original MentisOculi benchmark is limited. MENTISOCULI-3D appears to be a relatively straightforward extension of MentisOculi to 3D environments. The main evaluation objectives—multi-step state updating, answerability from the provided information, and reliance on visual evidence—are largely similar to those of the original benchmark. The paper should better clarify what fundamentally new research questions arise in the 3D setting that cannot be evaluated by the original framework.
-
Figure 1 does not clearly communicate the main idea of the paper. Figure 1 mainly presents examples of the five task categories, but it does not clearly illustrate how these tasks relate to the motivation, benchmark design, or main contribution. A reader should ideally be able to understand the key idea of the benchmark from this figure without relying heavily on the main text.
-
The presentation and information hierarchy of Figure 1 are unclear. There are lots of overlaps in the fig.
-
The dataset construction pipeline lacks a clear high-level explanation. The Dataset Construction section contains many details about thresholds, randomization, filtering, perturbations, and acceptance criteria, but the overall construction logic is difficult to follow. The paper would benefit from a clear end-to-end pipeline showing how scenes are sampled, actions are executed, paired questions are constructed, visual evidence is verified, and final samples are retained.
-
The low acceptance rate may introduce substantial selection bias. Only 11.96% of the core candidate samples are retained. The resulting benchmark is therefore a highly filtered subset conditioned on several criteria, including visibility, physical stability, answer boundaries, and twin constructability. The paper does not sufficiently analyze whether this filtering changes the scene distribution, task difficulty, or answer distribution.
-
Current manual inspection covers only approximately 10% of the dataset, stronger human validation or perceptual-readability analysis would be helpful.
-
The evaluation of commercial models is incomplete. While the evaluation of open-source models is relatively comprehensive, the commercial-model evaluation is limited. In particular, representative multimodal models from other major providers, such as Google and Anthropic, are not included. This makes it difficult to draw broad conclusions about the capabilities of current commercial or frontier multimodal models.
-
The conclusion is very straightforward and do not have enough insights.
Questions for authors
pls see the weakness part.
Response to Reviewer RNRL Part 1/4
Thank you for recognizing the construction and verification and for asking for a clearer account of what the benchmark lets researchers learn. We agree that verified labels alone do not establish broad embodied competence. Our claim is a controlled evaluation of action-conditioned state prediction, with explicit tests of scene dependence, information use and output validity. We have expanded the model comparisons and failure analysis to make that contribution concrete.
Q1. What practical problem motivates these tasks? A system that evaluates candidate actions needs to predict their consequences from its current observation. We isolate that forward-prediction problem by supplying the actions. This removes policy selection and exploration from the score and lets us compare models on an identical transition/query problem.
We selected these mechanisms to test coordinate-frame composition, persistence of hidden identity, a restricted physical transition, and independent action branches while retaining executable labels and controlled scene changes. These connections motivate the tasks; transfer to deployment remains unmeasured. This diagnostic resource can complement embodied evaluations, where uncertainty in observation, action selection and outcome labels is harder to separate.
What is new relative to STARE and MindCube? We agree that neither 3D transformations nor intermediate visualizations are new on their own. STARE already studies both, including perception probes; MindCube includes partial-view reasoning and hypothetical viewpoint changes. STARE, MindCube.
The additional contribution is an executable, paired protocol: each scene has a different-answer twin with identical text; both variants undergo evidence and outcome checks; and exact-state and slot-matched image conditions reuse the same questions. Its payoff is the following set of comparisons:
Thus, a proposed method can be evaluated for whether it improves paired correctness, benefits from updated observations beyond repeated images, or mainly produces more parseable answers. These controls are also usable independently of the five task families when designing subsequent evaluations.