acceptodds
Under review as a conference paper at ICLR 2027

MENTIS-OCULI-3D: A VERIFIED BENCHMARK FOR 3D STATE TRACKING UNDER PRESCRIBED ACTIONS

Abstract

Evaluating spatial prediction requires both a correct terminal label and evidence that the supplied observations determine it. We introduce MENTISOCULI-3D, a procedural benchmark for tracking 3D state under prescribed actions. Five task families cover articulated markers, hidden token identities, support release, counterfactual branches, and combined root and carousel rotations. Each sampled root yields two questions with identical text and different verified answers. The generator checks initial-view evidence, marker visibility, independent kinematic execution, and numerical stability of physical outcomes. Matched inputs vary access to intermediate images, exact states, and terminal coordinates. The artifact also contains 4,800 distribution-shift and long-horizon questions, diagnostic banks, and an interactive human-review interface that records visual assistance. The core contains 4,000 questions from 2,000 paired roots. Across 6 full-core open-model evaluations, accuracy ranges from 22.35% to 33.40% under a shared 2,048-token response budget. In two matched 800-question panels, terminal-coordinate readout scores 95.38% to 98.75%. Neither feedback-minus-repeated-image accuracy interval excludes zero. Further two-model experiments compare state trajectories, frame rewrites, output budgets, voting, and model-estimated-state tools. Separate guided-JSON probes and first-valid voting controls distinguish response completeness from geometric correctness. These results support reporting information controls and response validity alongside accuracy. They characterize behavior on the generated tasks, without establishing an internal simulation mechanism.

Primary area:
datasets and benchmarks
Keywords:
multimodal evaluation, 3D spatial reasoning, mental imagery, state tracking, procedural benchmarks, counterfactual reasoning, vision- language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j6CKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark called MENTISOCULI-3D, which aims to assess the spatial reasoning and prediction capabilities of current open-source and commercial multimodal LLMs. Specifically, the benchmark constructs a validation dataset based on verified pairs of questions designed to evaluate whether models can correctly track and reason about state changes in 3D environments. The experimental results show that GPT-6 Astra nearly solves the core benchmark under high reasoning effort, while seven evaluated open-source model configurations achieve substantially lower performance.

Critical strengths
  1. Evaluating the spatial reasoning and prediction capabilities of current open-source and commercial multimodal LLMs is an interesting and relevant research direction.
  2. The paired-question design is effective and provides a useful mechanism for assessing whether models genuinely rely on the relevant visual and spatial information.
  3. The evaluation covers a sufficiently broad range of open-source models, providing a useful comparison of their capabilities on the proposed benchmark.
Critical weaknesses
  1. The motivation is not clearly presented. The Introduction is difficult to follow and does not clearly explain the research background, the limitations of existing benchmarks, or why a dedicated 3D benchmark is needed.

  2. The novelty over the original MentisOculi benchmark is limited. MENTISOCULI-3D appears to be a relatively straightforward extension of MentisOculi to 3D environments. The main evaluation objectives—multi-step state updating, answerability from the provided information, and reliance on visual evidence—are largely similar to those of the original benchmark. The paper should better clarify what fundamentally new research questions arise in the 3D setting that cannot be evaluated by the original framework.

  3. Figure 1 does not clearly communicate the main idea of the paper. Figure 1 mainly presents examples of the five task categories, but it does not clearly illustrate how these tasks relate to the motivation, benchmark design, or main contribution. A reader should ideally be able to understand the key idea of the benchmark from this figure without relying heavily on the main text.

  4. The presentation and information hierarchy of Figure 1 are unclear. There are lots of overlaps in the fig.

  5. The dataset construction pipeline lacks a clear high-level explanation. The Dataset Construction section contains many details about thresholds, randomization, filtering, perturbations, and acceptance criteria, but the overall construction logic is difficult to follow. The paper would benefit from a clear end-to-end pipeline showing how scenes are sampled, actions are executed, paired questions are constructed, visual evidence is verified, and final samples are retained.

  6. The low acceptance rate may introduce substantial selection bias. Only 11.96% of the core candidate samples are retained. The resulting benchmark is therefore a highly filtered subset conditioned on several criteria, including visibility, physical stability, answer boundaries, and twin constructability. The paper does not sufficiently analyze whether this filtering changes the scene distribution, task difficulty, or answer distribution.

  7. Current manual inspection covers only approximately 10% of the dataset, stronger human validation or perceptual-readability analysis would be helpful.

  8. The evaluation of commercial models is incomplete. While the evaluation of open-source models is relatively comprehensive, the commercial-model evaluation is limited. In particular, representative multimodal models from other major providers, such as Google and Anthropic, are not included. This makes it difficult to draw broad conclusions about the capabilities of current commercial or frontier multimodal models.

  9. The conclusion is very straightforward and do not have enough insights.

Questions for authors

pls see the weakness part.

Reviewer_RNRLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MENTIS OCULI-3D, a procedural benchmark for predicting 3D states from initial images and prescribed actions. It covers five task families and constructs paired questions with identical text but different initial scenes and correct answers. Automated verification and matched information conditions support the evaluation. The tested open models achieve relatively low accuracy, while Astra approaches perfect performance under a separate high-reasoning configuration.

Critical strengths

Careful benchmark construction and verification. The benchmark pairs instances with identical textual instructions but different visual states and answers, and combines this design with observation checks and independent verification of the generated outcomes. These measures help control textual shortcuts and reduce ambiguity or errors in the evaluation data. Although these principles are not individually new, their systematic implementation provides a credible experimental resource within the defined task setting.

Critical weaknesses
  1. The incremental contribution and broader significance remain insufficiently established. STARE already evaluates instruction-based 3D transformations and intermediate visual states, while MindCube studies spatial mental modeling and hypothetical motion. Although this paper adds paired scene construction and careful verification, it does not convincingly demonstrate what substantially better assessment or new insight this combination provides beyond existing evaluations. Moreover, the motivation for selecting these particular synthetic mechanisms is not sufficiently connected to practical state-tracking problems. Label verification does not establish that performance on these tasks reliably reflects broader capabilities relevant to world modeling or embodied reasoning. The reported distribution shifts remain within the procedural task families and therefore provide limited evidence of broader applicability.

  2. The empirical evaluation has limited model coverage and diagnostic depth. Although eight models are evaluated, several share closely related backbones, and the detailed diagnostic and generalization analyses cover only two models. The strongest model also uses a different inference configuration, complicating the interpretation of performance differences. Consequently, the evaluation does not yet provide a sufficiently representative account of model capabilities. While the paper includes several diagnostic conditions, the analysis does not sufficiently disentangle initial-state perception, action interpretation, state updating, and response-format failures. The observed performance gaps therefore offer limited explanation of why models fail or what specific capability the benchmark reliably measures.

Additional presentation concern: Several visualizations have limited information density. More compact layouts and clearer presentation of informative error patterns would improve readability and analytical value.

Questions for authors
  1. Research motivation and practical relevance What practical research problem is this benchmark intended to address? How do these synthetic tasks relate to capabilities needed in world models or embodied settings, and what motivates selecting these tasks?

  2. Evaluation validity and failure analysis What theoretical or empirical evidence supports interpreting performance on these simplified instructions and synthetic scenes as a reliable measure of model capability? Can the authors explain more clearly why models perform poorly and how applicable and robust these findings are in more complex settings?

  3. Model coverage and representativeness Is the evaluated model set sufficiently representative? Is there broader comparative evidence showing that the benchmark meaningfully distinguishes model capabilities beyond the behavior of a few models under particular configurations?

1 comment

Response to Reviewer RNRL Part 1/4

Thank you for recognizing the construction and verification and for asking for a clearer account of what the benchmark lets researchers learn. We agree that verified labels alone do not establish broad embodied competence. Our claim is a controlled evaluation of action-conditioned state prediction, with explicit tests of scene dependence, information use and output validity. We have expanded the model comparisons and failure analysis to make that contribution concrete.

Q1. What practical problem motivates these tasks? A system that evaluates candidate actions needs to predict their consequences from its current observation. We isolate that forward-prediction problem by supplying the actions. This removes policy selection and exploration from the score and lets us compare models on an identical transition/query problem.

TaskCapability testedCorresponding practical subproblem
A: articulated markerCompose root and joint transformations in local/world framesPredict a tool endpoint after base-frame versus tool-frame commands
B: covered tokensPreserve previously observed identities under rigid motionTrack objects inside a moved container
E: carousel on a moving rootCombine parent and child motion while preserving identityTrack parts on a moving platform or turntable
C: support releasePredict a disclosed support/contact outcomeEvaluate a restricted release or support-change action
D: counterfactual branchesEvaluate alternatives from the same prefix stateCompare candidate next actions without confusing their histories

We selected these mechanisms to test coordinate-frame composition, persistence of hidden identity, a restricted physical transition, and independent action branches while retaining executable labels and controlled scene changes. These connections motivate the tasks; transfer to deployment remains unmeasured. This diagnostic resource can complement embodied evaluations, where uncertainty in observation, action selection and outcome labels is harder to separate.

What is new relative to STARE and MindCube? We agree that neither 3D transformations nor intermediate visualizations are new on their own. STARE already studies both, including perception probes; MindCube includes partial-view reasoning and hypothetical viewpoint changes. STARE, MindCube.

The additional contribution is an executable, paired protocol: each scene has a different-answer twin with identical text; both variants undergo evidence and outcome checks; and exact-state and slot-matched image conditions reuse the same questions. Its payoff is the following set of comparisons:

  1. Scene sensitivity: pair accuracy and same-answer rates detect a failure that aggregate accuracy hides. For example, InternVL and Qwen3.5-4B have almost identical ordinary accuracy (26.1% and 26.2%), yet their pair accuracy is 4.95% and 9.80%. SmolVLM2 gives identical valid answers to 82.6% of twin pairs, InternVL to 51.6%, and the original Astra run to 1.2%.
  2. Information use: true intermediate images compared with repeated initial images isolate the value of updated state information from the effect of additional image slots. For Qwen3.5-9B and InternVL, FB minus REP is −2.9 [−6.0, 0.3] and −1.0 [−4.3, 2.3] percentage points on the matched 800. These intervals show no detected average benefit, not proof that feedback is never used.
  3. Failure localization: initial-state, terminal-state and response-format controls reveal different limitations, detailed below.
  4. Reproducible hypotheses: because the actions are executable, a proposed frame/order error can be tested against the answer implied by a corrupted program.

Thus, a proposed method can be evaluated for whether it improves paired correctness, benefits from updated observations beyond repeated images, or mainly produces more parseable answers. These controls are also usable independently of the five task families when designing subsequent evaluations.

Reviewer_KkutOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MENTIS OCULI-3D benchmark for tracking 3D states from initial images and a sequence of actions. It covers five task families involving rotation, hidden identities, support release, and counterfactual actions. Paired questions use the same text and answer options but have different correct answers, which helps test whether models use the images.

Open models perform poorly under the tested output limit, while GPT-6 Astra nearly solves the core under a separate high-reasoning setting. Two open models perform well when given exact final coordinates, but intermediate images provide no clear average gain over repeated initial images.

Critical strengths

The paired questions help check whether models use the images. Both questions have the same text and answer options but different correct answers. The paper reports how often models answer both questions correctly, as well as accuracy on individual questions. The tests help distinguish understanding the final state from predicting how it changes.

Critical weaknesses

Astra answers almost all core questions correctly. The tasks use a small set of rules, fixed question templates, and simple objects with at most one joint connecting moving parts.

Questions for authors

Are there common types of errors made by the open models that Astra avoids? What do these differences suggest about Astra’s strong performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.