acceptodds
Withdrawn from ICLR 2027

SimVerse:Benchmarking Inverse Mental Simulation via Interactive Games

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inverse mental simulation, the process of reasoning backward from observed physical and spatial outcomes to their causal antecedents, requires compositional inference over continuous physical dynamics and discrete spatial topology. Existing benchmarks predominantly target forward prediction, and none is dedicated to the inverse, effect-to-cause direction in multimodal large language models. We introduce **SimVerse**, a benchmark for inverse mental simulation in multimodal large language models. **SimVerse** comprises 2,486 simulator-verified instances across four interactive game environments, organized into a taxonomy by operation order-sensitivity (order-invariant or order-sensitive) and state-transition type (discrete or continuous). Every model output is executed in a domain-specific physics or geometry simulator, so any valid antecedent earns credit and partial credit reflects physical or geometric progress.We evaluate seven models against an IQ-screened human baseline (). The strongest model (Gemini-3.1-Pro) achieves an overall score of 52.7%, falling 22.8 percentage points (95% CI: 19.3–26.4) below the human reference (75.5%), with marked variation across task families. On dice face inference, models identify up to 92.3% of symbols, yet none exceeds 32% once orientation is also required, exposing a bottleneck in cumulative state tracking. A concise direct-answer prompt and removal of the rendered image each lower success for all seven models and raise it on no task, while per-task model rankings remain largely stable. For both humans and models, order-sensitive discrete tasks are markedly harder than order-invariant continuous ones, identifying operation order-sensitivity as a central obstacle for inverse reasoning.

Primary area:
datasets and benchmarks
Keywords:
inverse mental simulation, multimodal large language models, benchmark, physical reasoning, simulation-based evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mbRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SimVerse, a benchmark for evaluating inverse mental simulation in multimodal language models.

It consists of 2,486 simulator-verified instances across four game environments, organized along two dimensions: operation order-sensitivity and discrete versus continuous state transitions. The authors evaluate seven frontier models against a human baseline and find a substantial performance gap, particularly in tasks requiring cumulative state tracking. The paper also includes prompt and modality ablations and a detailed failure analysis.

Critical strengths

The benchmark is generally well constructed. Simulator-based evaluation is a sensible choice for inverse problems, where multiple valid solutions may exist, and the use of partial-credit metrics provides useful diagnostic information beyond binary success rates.

The experimental analysis is reasonably thorough, including human comparisons, uncertainty estimates, ablations, and failure decomposition. In particular, the gap between pattern recognition and orientation tracking in the dice tasks is an interesting observation that identifies a concrete weakness in current models.

Critical weaknesses

My main concern is the significance and broader research implications of the contribution, rather than its technical correctness.

First, the benchmark feels rather toyish and limited in scope. It consists of only four highly specialized 2D game environments, some of which are closer to constrained combinatorial reasoning than realistic physical reasoning. While the articulated lamp and rope-cutting tasks do involve continuous geometry and dynamics, the overall setting remains quite far from the perceptual uncertainty, physical complexity, and interaction requirements of real-world world models or embodied agents. It is unclear what broader capabilities success on these tasks would establish, or how the findings would inform research beyond this particular benchmark.

Second, the contribution feels somewhat preliminary for a general benchmark paper. Although the dataset contains 2,486 instances, the diversity of underlying tasks and environments is quite limited. The proposed taxonomy is intuitively reasonable, but each category is represented by only one game, with different evaluation metrics. This makes it difficult to separate genuine structural difficulty from task-specific effects.

Most basically, the paper does not establish why inverse mental simulation, as operationalized here, constitutes a particularly important or underexplored bottleneck for general world-model research. The lack of matched forward-simulation controls further limits the conclusions that can be drawn about the inverse nature of the tasks.

Overall, this is a technically competent benchmark with reasonable empirical findings, but I find its conceptual contribution and broader significance insufficiently compelling. The work feels closer to a well-executed pilot study than a benchmark that opens up a substantial new research direction.

Questions for authors

Given that each cell of the proposed taxonomy is represented by a different environment and evaluated using different metrics, how confident can we be that the observed difficulty differences are attributable to operation order-sensitivity or state-transition type, rather than task-specific characteristics?

5 comments
AuthorsOpenReview ↗

Clarifying broader significance, taxonomy controls, and inverse specificity (W1)

W1: Controlled 2D scope and relevance to real-world world models

In realistic 3D embodied settings, action-planning failures are heavily confounded with low-level visual perception errors (depth ambiguity, occlusion, segmentation noise). Following the controlled paradigm of PHYRE and CLEVRER, SIMVERSE pairs clean renderings with exact symbolic states to isolate internal state propagation and constraint inversion from perceptual noise. Even in noise-free 2D environments, frontier MLLMs struggle severely (Gemini-3.1-Pro: 52.69%52.69\%, GPT-5.4: 27.37%27.37\% vs. Human 75.50%75.50\%; 5/7 models score 0.0%0.0\% strict pass on Dice Face), exposing two fundamental bottlenecks required by any general world model: (i) non-Abelian cumulative state tracking (O2O2) across multi-step 3D rotations (where 7-model mean POA decays monotonically across step depth t∈{1,2,3,4,≥5}t \in \{1, 2, 3, 4, \ge 5\} from 59.7%→30.1%→16.2%→8.8%→4.6%59.7\% \to 30.1\% \to 16.2\% \to 8.8\% \to 4.6\%), and (ii) continuous constraint inversion (A2A2) over collision-free manifolds (Lamp) and timed rigid-body dynamics (Rope).


AuthorsOpenReview ↗

Clarifying broader significance, taxonomy controls, and inverse specificity (W2 & Q1)

W2 & Q1: Separating taxonomy difficulty from task-specific effects

We isolated the O×AO \times A taxonomy dimensions from game- and metric-specific effects via two within-environment O1O1 vs. O2O2 controls (n=100n=100 stratified instances each, holding environment, rendering, and metric 100%100\% fixed) and cross-game metric harmonization:

  1. Within-Environment Dice Control (Table R6-A): Single-axis rolls ({N,S}\{N, S\} or {E,W}\{E, W\}) form a commutative subgroup C4C_4 (O1O1), whereas orthogonal rolls {N,S,E,W}\{N, S, E, W\} generate the 24-element non-Abelian group (O2O2). Holding roll length NN, face coverage, and POA metric fixed, changing O1→O2O1 \to O2 drops 4-model mean POA by −28.0 pp-28.0\text{ pp} on Dice Path (76.3%→48.3%76.3\% \to 48.3\%) and −32.9 pp-32.9\text{ pp} on Dice Face (55.0%→22.1%55.0\% \to 22.1\%).
  2. Within-Environment VOI Control (Table R6-B): Holding canvases, shapes, and metrics fixed, replacing commutative XOR (O1O1) with order-sensitive opaque layer occlusion (O2O2) drops 4-model mean Match Rate by −13.2 pp-13.2\text{ pp} (28.0%→14.8%28.0\% \to 14.8\%) and IoU by −0.157-0.157 (0.690→0.5330.690 \to 0.533).
  3. Cross-Game Metric Alignment (Table R1): Replacing task-specific binary thresholds with unified continuous partial credit across all four games (Appendix Tables 7 & 14) eliminates metric-strictness asymmetry and restores the exact predicted ordering: O1+A1 (58.8% IoU / 70.0% Dice)>O1+A2 (42.5%)>O2+A2 (38.6%)>O2+A1 (27.7%)O1+A1\text{ (58.8\% IoU / 70.0\% Dice)} > O1+A2\text{ (42.5\%)} > O2+A2\text{ (38.6\%)} > O2+A1\text{ (27.7\%)}. Under strict all-or-nothing pass rates, VOI (19.3%19.3\%) exceeds Dice Face Strict Pass (1.31%1.31\%, Table 15) by 14.7×14.7\times.

Table R6-A: Within-Environment Dice Control (O1O1 1D Commutative Subgroup vs. O2O2 2D Non-Abelian Group, n=100n=100, %)

ModelDice Path O1O1 (Commutative) PA / POADice Path O2O2 (Non-Abelian) PA / POAPath OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)Dice Face O1O1 (Commutative) PA / POADice Face O2O2 (Non-Abelian) PA / POAFace OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)
Gemini-3.1-Pro100.0 / 89.0100.0 / 65.0+24.078.0 / 56.062.5 / 19.0+37.0
Qwen3.5-397B99.0 / 85.097.0 / 59.0+26.091.0 / 64.084.0 / 30.0+34.0
Qwen3.6-Plus98.0 / 79.094.0 / 49.0+30.096.0 / 66.092.0 / 32.0+34.0
GPT-5.484.0 / 52.060.0 / 20.0+32.058.0 / 34.033.5 / 7.5+26.5
4-Model Mean95.3 / 76.387.8 / 48.3+28.080.8 / 55.068.0 / 22.1+32.9

Table R6-B: Within-Environment VOI Control (O1O1 Commutative VOI-XOR vs. O2O2 Order-Sensitive VOI-Occlusion, n=100n=100)

ModelVOI-XOR (O1O1: Order-Invariant) Match Rate (%) / IoUVOI-Occlusion (O2O2: Order-Sensitive) Match Rate (%) / IoUWithin-Env OO-Axis Cost ΔO1−O2\Delta_{O1-O2} Match Rate (pp) / Δ\Delta IoU
Gemini-3.1-Pro30.0 / 0.65516.0 / 0.512+14.0 / +0.143
Qwen3.5-397B23.0 / 0.69012.0 / 0.535+11.0 / +0.155
Qwen3.6-Plus44.0 / 0.78525.0 / 0.620+19.0 / +0.165
GPT-5.415.0 / 0.6306.0 / 0.465+9.0 / +0.165
4-Model Mean28.0 / 0.69014.8 / 0.533+13.2 / +0.157

Table R1: Cross-Family Comparison under Harmonized Continuous Partial Credit and Strict/Quasi-Match Thresholds (%)

SubjectVOI (O1+A1O1+A1) Pattern IoU [Tab. 7/14]VOI (O1+A1O1+A1) Dice Score [Tab. 7]Lamp (O1+A2O1+A2) Cont. Score [Tab. 14]Dice Face (O2+A1O2+A1) Face Acc. [Tab. 14]Rope (O2+A2O2+A2) Stars/3 [Tab. 14]VOI (O1+A1O1+A1) Quasi-Match (IoU≥0.8\text{IoU} \ge 0.8)Dice Face (O2+A1O2+A1) Strict Pass [Tab. 15]
Human (n=50n=50)94.597.189.260.068.388.018.00
Gemini-3.1-Pro65.575.371.626.366.552.53.19
Qwen3.5-397B69.178.577.045.727.654.00.00
Qwen3.6-Plus78.686.239.444.341.768.55.98
GPT-5.463.274.633.916.041.444.00.00
GLM-5V-Turbo54.567.725.923.237.336.50.00
Qwen3-VL-Plus50.364.636.925.335.431.00.00
Step-1O-Turbo-Vision30.343.312.612.920.211.50.00
7-Model Mean58.870.042.527.738.642.61.31

AuthorsOpenReview ↗

Clarifying broader significance, taxonomy controls, and inverse specificity (W3)

W3: Bottleneck of inverse mental simulation and matched forward controls

We evaluated four frontier models on n=100n=100 paired forward vs. inverse instances (20 per tier) in Dice (A1A1) and Lamp (A2A2) under identical scenes and simulators (Tables R5-A & R5-B). On Dice, Forward Roll reaches 73.3%73.3\% mean POA, whereas Inverse Path drops by −25.0 pp-25.0\text{ pp} (48.3%48.3\%) and Inverse Face drops by −51.2 pp-51.2\text{ pp} (22.1%22.1\%). On Lamp, Forward Kinematics (FK) reaches 85.8%85.8\% mean SR (15.2 px15.2\text{ px} error), whereas Inverse Kinematics (IK) drops SR by −25.8 pp-25.8\text{ pp} (60.0%60.0\%) and nearly triples error (42.0 px42.0\text{ px}), demonstrating that inverting physical and spatial transitions is a distinct, severe bottleneck beyond forward prediction.

Table R5-A: Matched Forward vs. Inverse Control on Non-Standard Dice (n=100n=100, %)

ModelForward Roll (PA / POA)Inverse Path (PA / POA)Inverse Face (PA / POA)Forward-to-Path Drop ΔFwd−Path\Delta_{\text{Fwd}-\text{Path}} POA (pp)Forward-to-Face Drop ΔFwd−Face\Delta_{\text{Fwd}-\text{Face}} POA (pp)
Gemini-3.1-Pro100.0 / 86.0100.0 / 65.062.5 / 19.0+21.0+67.0
Qwen3.5-397B98.0 / 81.097.0 / 59.084.0 / 30.0+22.0+51.0
Qwen3.6-Plus97.0 / 78.094.0 / 49.092.0 / 32.0+29.0+46.0
GPT-5.482.0 / 48.060.0 / 20.033.5 / 7.5+28.0+40.5
4-Model Mean94.3 / 73.387.8 / 48.368.0 / 22.1+25.0+51.2

Table R5-B: Matched Forward Kinematics (FK) vs. Inverse Kinematics (IK) on Articulated Lamp (n=100n=100)

ModelForward (FK) SR (%) / CD (px)Inverse (IK) SR (%) / CD (px)SR Drop ΔFK−IK\Delta_{\text{FK}-\text{IK}} (pp)Error Increase ΔIK−FK\Delta_{\text{IK}-\text{FK}} CD (px)
Gemini-3.1-Pro96.0 / 6.887.0 / 21.5+9.0+14.7
Qwen3.5-397B93.0 / 9.280.0 / 23.8+13.0+14.6
Qwen3.6-Plus78.0 / 21.437.0 / 60.2+41.0+38.8
GPT-5.476.0 / 23.536.0 / 62.5+40.0+39.0
4-Model Mean85.8 / 15.260.0 / 42.0+25.8+26.8

Reviewer_2CBbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper presents a benchmark called SimVerse for "inverse mental simulation" in multimodal LLMs focusing on inferring antecedents from observed or desired outcomes. It introcues 2.4K simulator-verified instances in four 2D game environments arranged in a 2 by 2 taxonomy. One axis is order-sensitivity the other is state-transition type. The games are VOI, Non-Standard Dice, Articulated Lamp, and Cut the Rope. The authors evaluate the outputs by running them in the simulator, so any valid antecedent gets credit. They judge seven frontier models and compare with a human reference: 50 graduate STEM participants with high IQ scores.

Critical strengths
  • Sound evaluation: All outputs are run in a domain simulator, so any valid solution gets credit and partial credit reflects physical or geometric progress. This avoids the reference-matching problems of many reasoning benchmarks and is a real methodological advantage for evaluating inverse problems, which often have many solutions.
  • Somewhat substantive empirical and diagnostic results: The evaluation covers seven models, a human reference, uncertainty estimates, prompt and modality ablations, and a decomposition of failure types.
  • Rigorous results reporting: The paper reports confidence intervals throughout and presents a human baseline (scored by 50 high IQ graduates students), and their prompt and modality ablations that keep rankings largely stable (Spearman ρ 0.88–0.99), and a failure decomposition showing most failures are executable near misses rather than formatting errors.
Critical weaknesses
  • Although the paper motivates inverse reasoning as a separate evaluation target, it does not compare matched forward and inverse formulations of the same tasks. It remains unclear how much of the observed difficulty reflects inverse reasoning specifically, versus difficulties in spatial transformation, exact state tracking, combinatorial search, or action planning that could also occur in forward tasks. Since novelty is framed around the inverse direction (Fig. 1a, Table 1), a matched forward variant of at least some tasks (e.g., Dice: predict the final face from a roll sequence) seems necessary to support the framing. This limits the strength of the conclusions that can be drawn from the benchmark.
  • The proposed difficulty taxonomy is not validated. Each of the four order-sensitivity x transition-type combinations corresponds to a different game, so differences attributed to either axis are confounded with game design, underlying difficulty, answer representations, and scoring metrics. For example, VOI uses an exact-match score, whereas the Dice family aggregates partial face-orientation accuracy.
Questions for authors
  • Each human completed 50 of 2,486 instances. How was the stratified sample constructed, and is the human–model comparison on the same instance distribution, given that it is unpaired?
  • Can the authors provide any controlled comparison in which operation order-sensitivity varies while the task environment, scoring metric, and other major difficulty factors are held approximately fixed?
  • Can you add matched forward versions of one or more tasks (e.g., forward dice rolling, forward kinematics for Lamp) to test whether the inverse direction itself is harder for models?
5 comments
AuthorsOpenReview ↗

Adding forward baselines, taxonomy validations, and paired human analyses (W1 & Q3)

W1 & Q3: Matched forward vs. inverse task controls

We evaluated four frontier models on n=100n=100 paired forward vs. inverse instances (20 per tier) in Dice (A1A1) and Lamp (A2A2) under identical scene geometries, step distributions, and simulators (Tables R5-A & R5-B):

  1. Forward vs. Inverse Dice (Table R5-A): Given the 6-face map and a roll sequence of length NN, Forward Roll (predicting the terminal top symbol and orientation) achieves 73.3%73.3\% 4-model mean POA (94.3%94.3\% PA). On the exact same instances, Inverse Path drops mean POA by −25.0 pp-25.0\text{ pp} (48.3%48.3\%) and Inverse Face drops by −51.2 pp-51.2\text{ pp} (22.1%22.1\%), while widening the PA−POA\text{PA}-\text{POA} gap from 21.0 pp21.0\text{ pp} (Forward) to 39.5 pp39.5\text{ pp} (Path) and 45.9 pp45.9\text{ pp} (Face).
  2. Forward vs. Inverse Lamp (Table R5-B): Given joint angles θ\theta, Forward Kinematics (FK) achieves 85.8%85.8\% mean Success Rate (15.2 px15.2\text{ px} Center Distance error), whereas Inverse Kinematics (IK) drops SR by −25.8 pp-25.8\text{ pp} (60.0%60.0\%) and nearly triples error (42.0 px42.0\text{ px}), confirming that inverse constraint satisfaction is substantially harder than forward prediction.

Table R5-A: Matched Forward vs. Inverse Control on Non-Standard Dice (n=100n=100, %)

ModelForward Roll (PA / POA)Inverse Path (PA / POA)Inverse Face (PA / POA)Forward-to-Path Drop ΔFwd−Path\Delta_{\text{Fwd}-\text{Path}} POA (pp)Forward-to-Face Drop ΔFwd−Face\Delta_{\text{Fwd}-\text{Face}} POA (pp)
Gemini-3.1-Pro100.0 / 86.0100.0 / 65.062.5 / 19.0+21.0+67.0
Qwen3.5-397B98.0 / 81.097.0 / 59.084.0 / 30.0+22.0+51.0
Qwen3.6-Plus97.0 / 78.094.0 / 49.092.0 / 32.0+29.0+46.0
GPT-5.482.0 / 48.060.0 / 20.033.5 / 7.5+28.0+40.5
4-Model Mean94.3 / 73.387.8 / 48.368.0 / 22.1+25.0+51.2

Table R5-B: Matched Forward Kinematics (FK) vs. Inverse Kinematics (IK) on Articulated Lamp (n=100n=100)

ModelForward (FK) SR (%) / CD (px)Inverse (IK) SR (%) / CD (px)SR Drop ΔFK−IK\Delta_{\text{FK}-\text{IK}} (pp)Error Increase ΔIK−FK\Delta_{\text{IK}-\text{FK}} CD (px)
Gemini-3.1-Pro96.0 / 6.887.0 / 21.5+9.0+14.7
Qwen3.5-397B93.0 / 9.280.0 / 23.8+13.0+14.6
Qwen3.6-Plus78.0 / 21.437.0 / 60.2+41.0+38.8
GPT-5.476.0 / 23.536.0 / 62.5+40.0+39.0
4-Model Mean85.8 / 15.260.0 / 42.0+25.8+26.8

AuthorsOpenReview ↗

Adding forward baselines, taxonomy validations, and paired human analyses (W2 & Q2)

W2 & Q2: Taxonomy validation and within-environment order-sensitivity

We validated the O×AO \times A taxonomy by testing order-sensitivity (O1O1 vs. O2O2) within the exact same environment and metric (n=100n=100), and by harmonizing metrics across all four games:

  1. Within-Environment Dice Control (Table R6-A): Restricting rolls to a single axis ({N,S}\{N, S\} or {E,W}\{E, W\}) forms a commutative subgroup C4C_4 (O1O1), whereas orthogonal 2D rolls {N,S,E,W}\{N, S, E, W\} generate the non-Abelian group (O2O2). With roll length NN, face coverage, and POA metric held fixed, switching O1→O2O1 \to O2 drops 4-model mean POA by −28.0 pp-28.0\text{ pp} on Dice Path (76.3%→48.3%76.3\% \to 48.3\%) and −32.9 pp-32.9\text{ pp} on Dice Face (55.0%→22.1%55.0\% \to 22.1\%).
  2. Within-Environment VOI Control (Table R6-B): Holding canvases, shapes, and metrics fixed, replacing commutative XOR (O1O1) with order-sensitive opaque layer occlusion (O2O2) drops 4-model mean Match Rate by −13.2 pp-13.2\text{ pp} (28.0%→14.8%28.0\% \to 14.8\%) and IoU by −0.157-0.157 (0.690→0.5330.690 \to 0.533).
  3. Cross-Game Metric Alignment (Table R1): In Table 2, VOI (19.3%19.3\%) appeared harder than Dice (26.3%26.3\%) solely because VOI Match Rate requires a 100%100\% pixel-exact match whereas Dice POA credits partial faces. Aligning all games via continuous partial credit (Appendix Tables 7 & 14) restores the exact predicted ordering: O1+A1 (58.8% IoU / 70.0% Dice)>O1+A2 (42.5%)>O2+A2 (38.6%)>O2+A1 (27.7%)O1+A1\text{ (58.8\% IoU / 70.0\% Dice)} > O1+A2\text{ (42.5\%)} > O2+A2\text{ (38.6\%)} > O2+A1\text{ (27.7\%)}. Under strict all-or-nothing pass rates, VOI (19.3%19.3\%) is 14.7×14.7\times higher than Dice Face Strict Pass (1.31%1.31\%, Table 15).

Table R6-A: Within-Environment Dice Control (O1O1 1D Commutative Subgroup vs. O2O2 2D Non-Abelian Group, n=100n=100, %)

ModelDice Path O1O1 (Commutative) PA / POADice Path O2O2 (Non-Abelian) PA / POAPath OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)Dice Face O1O1 (Commutative) PA / POADice Face O2O2 (Non-Abelian) PA / POAFace OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)
Gemini-3.1-Pro100.0 / 89.0100.0 / 65.0+24.078.0 / 56.062.5 / 19.0+37.0
Qwen3.5-397B99.0 / 85.097.0 / 59.0+26.091.0 / 64.084.0 / 30.0+34.0
Qwen3.6-Plus98.0 / 79.094.0 / 49.0+30.096.0 / 66.092.0 / 32.0+34.0
GPT-5.484.0 / 52.060.0 / 20.0+32.058.0 / 34.033.5 / 7.5+26.5
4-Model Mean95.3 / 76.387.8 / 48.3+28.080.8 / 55.068.0 / 22.1+32.9

Table R6-B: Within-Environment VOI Control (O1O1 Commutative VOI-XOR vs. O2O2 Order-Sensitive VOI-Occlusion, n=100n=100)

ModelVOI-XOR (O1O1: Order-Invariant) Match Rate (%) / IoUVOI-Occlusion (O2O2: Order-Sensitive) Match Rate (%) / IoUWithin-Env OO-Axis Cost ΔO1−O2\Delta_{O1-O2} Match Rate (pp) / Δ\Delta IoU
Gemini-3.1-Pro30.0 / 0.65516.0 / 0.512+14.0 / +0.143
Qwen3.5-397B23.0 / 0.69012.0 / 0.535+11.0 / +0.155
Qwen3.6-Plus44.0 / 0.78525.0 / 0.620+19.0 / +0.165
GPT-5.415.0 / 0.6306.0 / 0.465+9.0 / +0.165
4-Model Mean28.0 / 0.69014.8 / 0.533+13.2 / +0.157

Table R1: Cross-Family Comparison under Harmonized Continuous Partial Credit and Strict/Quasi-Match Thresholds (%)

SubjectVOI (O1+A1O1+A1) Pattern IoU [Tab. 7/14]VOI (O1+A1O1+A1) Dice Score [Tab. 7]Lamp (O1+A2O1+A2) Cont. Score [Tab. 14]Dice Face (O2+A1O2+A1) Face Acc. [Tab. 14]Rope (O2+A2O2+A2) Stars/3 [Tab. 14]VOI (O1+A1O1+A1) Quasi-Match (IoU≥0.8\text{IoU} \ge 0.8)Dice Face (O2+A1O2+A1) Strict Pass [Tab. 15]
Human (n=50n=50)94.597.189.260.068.388.018.00
Gemini-3.1-Pro65.575.371.626.366.552.53.19
Qwen3.5-397B69.178.577.045.727.654.00.00
Qwen3.6-Plus78.686.239.444.341.768.55.98
GPT-5.463.274.633.916.041.444.00.00
GLM-5V-Turbo54.567.725.923.237.336.50.00
Qwen3-VL-Plus50.364.636.925.335.431.00.00
Step-1O-Turbo-Vision30.343.312.612.920.211.50.00
7-Model Mean58.870.042.527.738.642.61.31

AuthorsOpenReview ↗

Adding forward baselines, taxonomy validations, and paired human analyses (Q1)

Q1: Stratified human sampling and paired-subset evaluation

Each of the n=50n=50 participants completed 50 instances—10 per task variant across the 5 variants (500500 trials/task, 2,5002,500 total trials covering 1,5561,556 unique instances)—sampled proportionally across difficulty tiers (e.g., 2 instances per tier across the 5 tiers of Dice and Lamp) to prevent fatigue. Evaluating models strictly on the exact 2,500 human-tested trials (Paired Subset, Table R4) yields scores within ≤0.40 pp\le 0.40\text{ pp} of the full 2,486-instance benchmark on every task (Gemini Overall: 52.83%52.83\% paired vs. 52.69%52.69\% full; paired human–model gap +22.7 pp [19.2,26.2]+22.7\text{ pp } [19.2, 26.2]), confirming zero instance-distribution bias.

Table R4: Stratified Human Sampling Coverage and Paired-Subset vs. Full-Benchmark Model Comparison

Task VariantFull Benchmark (NN)Trials per Human (mkm_k)Total Human TrialsUnique Human InstancesHuman Score (%)Gemini Full (%)Gemini Paired Subset (%)Qwen3.5 Full (%)Qwen3.5 Paired Subset (%)Gemini Paired Human–Model Gap [95% CI]
VOI (Match Rate)6001050034282.0030.0030.4023.3323.60+51.6 [+46.2, +57.0]
Dice Face (POA)5021050031860.0018.5218.8030.0829.85+41.2 [+36.1, +46.3]
Dice Path (POA)5021050031676.0064.5464.2059.1659.40+11.8 [+6.8, +16.8]
Lamp (Success Rate)6101050034887.0087.3887.2080.4980.80-0.2 [-4.3, +3.9]
Rope (Pass Rate)2721050023265.0051.8452.2021.6921.40+12.8 [+6.5, +19.1]
Overall (Family Macro)2,486502,5001,55675.5052.6952.8342.5342.61+22.7 [+19.2, +26.2]

Reviewer_MX1TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SimVerse evaluates inverse spatial and physical reasoning on 2,486 instances across four game environments. It uses simulator-based scoring and compares seven multimodal models with a screened human reference. The benchmark is useful. However, some explanations of model failures go beyond the evidence.

Critical strengths
  1. Simulator-based scoring accepts valid alternative solutions.
  2. The tasks cover several forms of spatial and physical reasoning.
  3. Human comparisons, partial-credit metrics, and ablations provide useful diagnostics.
Critical weaknesses
  1. Task difficulty is confounded with game design. Each taxonomy cell contains one game, with different rules and metrics. Comparing Lamp with Dice cannot establish that order sensitivity matters more than continuous-state reasoning.
  2. The proposed failure mechanism is not established. The PA–POA gap shows difficulty with orientation. It does not distinguish cumulative tracking errors from single-step rotation errors or misunderstood coordinate conventions.
  3. The value specific to inverse reasoning remains unclear. There is no matched forward control. The failures may reflect general spatial reasoning limitations. Relative to benchmarks such as PHYRE, the novelty lies mainly in task coverage and evaluation design.
Questions for authors
  1. Can the authors test order sensitivity within the same environment, while keeping other factors fixed?
  2. How does orientation accuracy change with roll length after controlling for observable-face coverage? Are errors present from the first step or accumulated later?
  3. How are participant-clustered bootstrap and assumed ICC values combined to compute human confidence intervals?
  4. What model versions, reasoning budgets, token limits, and retry limits were used?
5 comments
AuthorsOpenReview ↗

Adding forward baselines, taxonomy controls, cumulative drift analyses, and setup details (W1 & Q1)

W1 & Q1: Taxonomy confounding and within-environment order-sensitivity

We decoupled operation order-sensitivity (OO) from game rules and metric design via two within-environment O1O1 vs. O2O2 controls (n=100n=100 stratified instances each, holding environment, rendering, and metric strictly fixed) and cross-game metric harmonization:

  1. Within-Environment Dice Control (Table R6-A): Single-axis rolls ({N,S}\{N, S\} or {E,W}\{E, W\}) form a commutative cyclic subgroup C4C_4 (O1O1), whereas orthogonal rolls {N,S,E,W}\{N, S, E, W\} generate the 24-element non-Abelian group (O2O2). Holding roll length NN, face coverage, and POA metric fixed, changing O1→O2O1 \to O2 drops 4-model mean POA by −28.0 pp-28.0\text{ pp} on Dice Path (76.3%→48.3%76.3\% \to 48.3\%) and −32.9 pp-32.9\text{ pp} on Dice Face (55.0%→22.1%55.0\% \to 22.1\%).
  2. Within-Environment VOI Control (Table R6-B): Holding canvases, shapes, and metrics fixed, replacing commutative XOR (O1O1) with order-sensitive opaque layer occlusion (O2O2, where later shapes overwrite earlier ones with a 1-px clear border) drops 4-model mean Match Rate by −13.2 pp-13.2\text{ pp} (28.0%→14.8%28.0\% \to 14.8\%) and IoU by −0.157-0.157 (0.690→0.5330.690 \to 0.533).
  3. Cross-Game Metric Alignment (Table R1): In Table 2, VOI (19.3%19.3\%) scored below Dice (26.3%26.3\%) only because VOI Match Rate requires a 100%100\% pixel-exact match while Dice POA credits partial faces. Aligning all games under continuous partial credit (Appendix Tables 7 & 14) restores the exact Eq. (1) ordering: O1+A1 (58.8% IoU / 70.0% Dice)>O1+A2 (42.5%)>O2+A2 (38.6%)>O2+A1 (27.7%)O1+A1\text{ (58.8\% IoU / 70.0\% Dice)} > O1+A2\text{ (42.5\%)} > O2+A2\text{ (38.6\%)} > O2+A1\text{ (27.7\%)}. Under strict all-or-nothing grading, VOI (19.3%19.3\%) exceeds Dice Face Strict Pass (1.31%1.31\%, Table 15) by 14.7×14.7\times.

Table R6-A: Within-Environment Dice Control (O1O1 1D Commutative Subgroup vs. O2O2 2D Non-Abelian Group, n=100n=100, %)

ModelDice Path O1O1 (Commutative) PA / POADice Path O2O2 (Non-Abelian) PA / POAPath OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)Dice Face O1O1 (Commutative) PA / POADice Face O2O2 (Non-Abelian) PA / POAFace OO-Axis Cost ΔO1−O2\Delta_{O1-O2} POA (pp)
Gemini-3.1-Pro100.0 / 89.0100.0 / 65.0+24.078.0 / 56.062.5 / 19.0+37.0
Qwen3.5-397B99.0 / 85.097.0 / 59.0+26.091.0 / 64.084.0 / 30.0+34.0
Qwen3.6-Plus98.0 / 79.094.0 / 49.0+30.096.0 / 66.092.0 / 32.0+34.0
GPT-5.484.0 / 52.060.0 / 20.0+32.058.0 / 34.033.5 / 7.5+26.5
4-Model Mean95.3 / 76.387.8 / 48.3+28.080.8 / 55.068.0 / 22.1+32.9

Table R6-B: Within-Environment VOI Control (O1O1 Commutative VOI-XOR vs. O2O2 Order-Sensitive VOI-Occlusion, n=100n=100)

ModelVOI-XOR (O1O1: Order-Invariant) Match Rate (%) / IoUVOI-Occlusion (O2O2: Order-Sensitive) Match Rate (%) / IoUWithin-Env OO-Axis Cost ΔO1−O2\Delta_{O1-O2} Match Rate (pp) / Δ\Delta IoU
Gemini-3.1-Pro30.0 / 0.65516.0 / 0.512+14.0 / +0.143
Qwen3.5-397B23.0 / 0.69012.0 / 0.535+11.0 / +0.155
Qwen3.6-Plus44.0 / 0.78525.0 / 0.620+19.0 / +0.165
GPT-5.415.0 / 0.6306.0 / 0.465+9.0 / +0.165
4-Model Mean28.0 / 0.69014.8 / 0.533+13.2 / +0.157

Table R1: Cross-Family Comparison under Harmonized Continuous Partial Credit and Strict/Quasi-Match Thresholds (%)

SubjectVOI (O1+A1O1+A1) Pattern IoU [Tab. 7/14]VOI (O1+A1O1+A1) Dice Score [Tab. 7]Lamp (O1+A2O1+A2) Cont. Score [Tab. 14]Dice Face (O2+A1O2+A1) Face Acc. [Tab. 14]Rope (O2+A2O2+A2) Stars/3 [Tab. 14]VOI (O1+A1O1+A1) Quasi-Match (IoU≥0.8\text{IoU} \ge 0.8)Dice Face (O2+A1O2+A1) Strict Pass [Tab. 15]
Human (n=50n=50)94.597.189.260.068.388.018.00
Gemini-3.1-Pro65.575.371.626.366.552.53.19
Qwen3.5-397B69.178.577.045.727.654.00.00
Qwen3.6-Plus78.686.239.444.341.768.55.98
GPT-5.463.274.633.916.041.444.00.00
GLM-5V-Turbo54.567.725.923.237.336.50.00
Qwen3-VL-Plus50.364.636.925.335.431.00.00
Step-1O-Turbo-Vision30.343.312.612.920.211.50.00
7-Model Mean58.870.042.527.738.642.61.31

AuthorsOpenReview ↗

Adding forward baselines, taxonomy controls, cumulative drift analyses, and setup details (W2 & Q2)

W2 & Q2: Cumulative state tracking vs. single-step or coordinate errors

Full-benchmark stratification (N=502N=502) and step-depth survival analysis confirm progressive multi-step error accumulation rather than first-step blindness or misunderstood coordinate conventions:

  1. Roll-Length Stratification & Fixed Face Coverage (Tables R2-A & R2-B): Across Tiers 1–5 (N∈[2,10]N \in [2, 10]), 7-model mean POA on Dice Face drops 6.4×6.4\times from 34.20%34.20\% (Tier 1, 2–3 steps) to 5.36%5.36\% (Tier 5, 10 steps). Holding observable-face coverage strictly fixed at 4 faces on Dice Face, lengthening rolls from N≤5N \le 5 (n=85n=85) to N≥6N \ge 6 (n=110n=110) still drops mean POA from 22.80%22.80\% to 8.60%8.60\% (Qwen3.6-Plus: 41.00%→18.50%41.00\% \to 18.50\%). On Dice Path (all 6 faces known upfront), Gemini-3.1-Pro's POA drops from 86.00%86.00\% (Tier 1) to 45.54%45.54\% (Tier 5) while PA stays 100.00%100.00\%.
  2. Step-Depth Survival Analysis (Table R3): Evaluating Dice Face predictions by the step depth t∈{1,2,3,4,≥5}t \in \{1, 2, 3, 4, \ge 5\} at which each face is first observed (requiring tt inverse rotations R1−1⋯Rt−1R_1^{-1}\cdots R_t^{-1}) shows that at Step 1 (t=1t=1), top models achieve 74.2%–84.0%74.2\%\text{--}84.0\% POA (7-model mean 59.7%59.7\% vs. 25%25\% chance), ruling out coordinate misunderstandings or single-step rotation failure. Mean POA then decays monotonically with step depth: 59.7%→30.1%→16.2%→8.8%→4.6%59.7\% \to 30.1\% \to 16.2\% \to 8.8\% \to 4.6\% (mean first orientation error at step 2.142.14).

Table R2-A: Dice Face Inference by Roll-Length Tier and Fixed Observable-Face Control (PA / POA / Gap, %)

Stratum & ID RangeRoll LengthObserved FacesnnGemini-3.1-Pro (PA / POA / Gap)Qwen3.6-Plus (PA / POA / Gap)Qwen3.5-397B (PA / POA / Gap)7-Model Mean (PA / POA / Gap)
Tier 12–3 stepsAll10074.00 / 41.00 / 33.0097.00 / 58.00 / 39.0092.00 / 55.00 / 37.0066.50 / 34.20 / 32.30
Tier 24–5 stepsAll10067.50 / 24.50 / 43.0095.00 / 39.50 / 55.5088.50 / 38.00 / 50.5061.20 / 21.40 / 39.80
Tier 36–7 stepsAll10061.50 / 14.50 / 47.0092.50 / 28.00 / 64.5084.50 / 26.50 / 58.0056.40 / 13.60 / 42.80
Tier 48–9 stepsAll10156.50 / 8.00 / 48.5090.00 / 19.00 / 71.0081.00 / 18.00 / 63.0051.20 / 8.10 / 43.10
Tier 510 stepsAll10151.42 / 4.86 / 46.5687.28 / 13.68 / 73.6076.64 / 13.19 / 63.4546.63 / 5.36 / 41.27
Control: Short (N≤5N \le 5)4–5 stepsFixed = 4 faces8566.80 / 26.20 / 40.6094.80 / 41.00 / 53.8088.00 / 39.20 / 48.8060.50 / 22.80 / 37.70
Control: Long (N≥6N \ge 6)6–10 stepsFixed = 4 faces11055.20 / 9.40 / 45.8089.60 / 18.50 / 71.1080.40 / 17.50 / 62.9050.80 / 8.60 / 42.20

Table R2-B: Dice Path Inference by Roll-Length Tier (All 6 Faces Known Upfront, PA / POA / Gap, %)

Stratum & ID RangeSource Roll LengthKnown FacesnnGemini-3.1-Pro (PA / POA / Gap)Qwen3.6-Plus (PA / POA / Gap)Qwen3.5-397B (PA / POA / Gap)7-Model Mean (PA / POA / Gap)
Tier 12–3 stepsAll 6 faces100100.00 / 86.00 / 14.0098.00 / 75.00 / 23.0099.00 / 82.00 / 17.0081.50 / 58.20 / 23.30
Tier 24–5 stepsAll 6 faces100100.00 / 74.00 / 26.0096.00 / 59.00 / 37.0098.00 / 69.00 / 29.0076.80 / 44.50 / 32.30
Tier 36–7 stepsAll 6 faces100100.00 / 64.00 / 36.0094.00 / 47.00 / 47.0097.00 / 58.00 / 39.0072.40 / 34.80 / 37.60
Tier 48–9 stepsAll 6 faces101100.00 / 53.47 / 46.5392.08 / 35.64 / 56.4497.03 / 47.52 / 49.5167.50 / 24.60 / 42.90
Tier 510 stepsAll 6 faces101100.00 / 45.54 / 54.4689.11 / 28.71 / 60.4096.04 / 39.60 / 56.4462.68 / 19.04 / 43.64

Table R3: Step-Wise State-Tracking Accuracy on Dice Face (N=502N=502) by Step Depth t∈{1,2,3,4,≥5}t \in \{1, 2, 3, 4, \ge 5\}

ModelNNStep 1 (PA / POA %)Step 2 (PA / POA %)Step 3 (PA / POA %)Step 4 (PA / POA %)Step ≥5\ge 5 (PA / POA %)Mean First-Error Step
Gemini-3.1-Pro50286.5 / 74.274.0 / 38.564.2 / 19.856.8 / 10.449.5 / 5.22.35
Qwen3.5-397B50294.2 / 81.589.5 / 52.085.0 / 31.581.2 / 18.477.5 / 10.22.78
Qwen3.6-Plus50297.5 / 84.095.2 / 54.593.0 / 33.290.8 / 19.588.0 / 11.02.86
GPT-5.450262.0 / 42.544.5 / 14.233.0 / 5.526.5 / 2.421.0 / 1.01.68
GLM-5V-Turbo50274.5 / 53.058.0 / 22.546.5 / 10.839.0 / 5.233.5 / 2.51.95
Qwen3-VL-Plus50278.0 / 51.562.5 / 21.050.2 / 9.842.5 / 4.836.8 / 2.21.91
Step-1O-Turbo-Vision50252.5 / 31.035.0 / 7.824.8 / 2.519.2 / 1.015.0 / 0.41.42

AuthorsOpenReview ↗

Adding forward baselines, taxonomy controls, cumulative drift analyses, and setup details (W3)

W3: Matched forward-simulation controls

We evaluated four models on n=100n=100 paired forward vs. inverse instances (20 per tier) in Dice (A1A1) and Lamp (A2A2) under identical scenes and simulators (Tables R5-A & R5-B). On Dice, Forward Roll reaches 73.3%73.3\% mean POA, whereas Inverse Path drops by −25.0 pp-25.0\text{ pp} (48.3%48.3\%) and Inverse Face drops by −51.2 pp-51.2\text{ pp} (22.1%22.1\%). On Lamp, Forward Kinematics (FK) reaches 85.8%85.8\% mean SR (15.2 px15.2\text{ px} error), whereas Inverse Kinematics (IK) drops SR by −25.8 pp-25.8\text{ pp} (60.0%60.0\%) and nearly triples error (42.0 px42.0\text{ px}), proving that constraint inversion imposes substantial difficulty beyond forward spatial transformations.

Table R5-A: Matched Forward vs. Inverse Control on Non-Standard Dice (n=100n=100, %)

ModelForward Roll (PA / POA)Inverse Path (PA / POA)Inverse Face (PA / POA)Forward-to-Path Drop ΔFwd−Path\Delta_{\text{Fwd}-\text{Path}} POA (pp)Forward-to-Face Drop ΔFwd−Face\Delta_{\text{Fwd}-\text{Face}} POA (pp)
Gemini-3.1-Pro100.0 / 86.0100.0 / 65.062.5 / 19.0+21.0+67.0
Qwen3.5-397B98.0 / 81.097.0 / 59.084.0 / 30.0+22.0+51.0
Qwen3.6-Plus97.0 / 78.094.0 / 49.092.0 / 32.0+29.0+46.0
GPT-5.482.0 / 48.060.0 / 20.033.5 / 7.5+28.0+40.5
4-Model Mean94.3 / 73.387.8 / 48.368.0 / 22.1+25.0+51.2

Table R5-B: Matched Forward Kinematics (FK) vs. Inverse Kinematics (IK) on Articulated Lamp (n=100n=100)

ModelForward (FK) SR (%) / CD (px)Inverse (IK) SR (%) / CD (px)SR Drop ΔFK−IK\Delta_{\text{FK}-\text{IK}} (pp)Error Increase ΔIK−FK\Delta_{\text{IK}-\text{FK}} CD (px)
Gemini-3.1-Pro96.0 / 6.887.0 / 21.5+9.0+14.7
Qwen3.5-397B93.0 / 9.280.0 / 23.8+13.0+14.6
Qwen3.6-Plus78.0 / 21.437.0 / 60.2+41.0+38.8
GPT-5.476.0 / 23.536.0 / 62.5+40.0+39.0
4-Model Mean85.8 / 15.260.0 / 42.0+25.8+26.8

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.