- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SimVerse, a benchmark for evaluating inverse mental simulation in multimodal language models.
It consists of 2,486 simulator-verified instances across four game environments, organized along two dimensions: operation order-sensitivity and discrete versus continuous state transitions. The authors evaluate seven frontier models against a human baseline and find a substantial performance gap, particularly in tasks requiring cumulative state tracking. The paper also includes prompt and modality ablations and a detailed failure analysis.
Critical strengths
The benchmark is generally well constructed. Simulator-based evaluation is a sensible choice for inverse problems, where multiple valid solutions may exist, and the use of partial-credit metrics provides useful diagnostic information beyond binary success rates.
The experimental analysis is reasonably thorough, including human comparisons, uncertainty estimates, ablations, and failure decomposition. In particular, the gap between pattern recognition and orientation tracking in the dice tasks is an interesting observation that identifies a concrete weakness in current models.
Critical weaknesses
My main concern is the significance and broader research implications of the contribution, rather than its technical correctness.
First, the benchmark feels rather toyish and limited in scope. It consists of only four highly specialized 2D game environments, some of which are closer to constrained combinatorial reasoning than realistic physical reasoning. While the articulated lamp and rope-cutting tasks do involve continuous geometry and dynamics, the overall setting remains quite far from the perceptual uncertainty, physical complexity, and interaction requirements of real-world world models or embodied agents. It is unclear what broader capabilities success on these tasks would establish, or how the findings would inform research beyond this particular benchmark.
Second, the contribution feels somewhat preliminary for a general benchmark paper. Although the dataset contains 2,486 instances, the diversity of underlying tasks and environments is quite limited. The proposed taxonomy is intuitively reasonable, but each category is represented by only one game, with different evaluation metrics. This makes it difficult to separate genuine structural difficulty from task-specific effects.
Most basically, the paper does not establish why inverse mental simulation, as operationalized here, constitutes a particularly important or underexplored bottleneck for general world-model research. The lack of matched forward-simulation controls further limits the conclusions that can be drawn about the inverse nature of the tasks.
Overall, this is a technically competent benchmark with reasonable empirical findings, but I find its conceptual contribution and broader significance insufficiently compelling. The work feels closer to a well-executed pilot study than a benchmark that opens up a substantial new research direction.
Questions for authors
Given that each cell of the proposed taxonomy is represented by a different environment and evaluated using different metrics, how confident can we be that the observed difficulty differences are attributable to operation order-sensitivity or state-transition type, rather than task-specific characteristics?
5 comments
Clarifying broader significance, taxonomy controls, and inverse specificity (W2 & Q1)
W2 & Q1: Separating taxonomy difficulty from task-specific effects
We isolated the taxonomy dimensions from game- and metric-specific effects via two within-environment vs. controls ( stratified instances each, holding environment, rendering, and metric fixed) and cross-game metric harmonization:
- Within-Environment
DiceControl (Table R6-A): Single-axis rolls ( or ) form a commutative subgroup (), whereas orthogonal rolls generate the 24-element non-Abelian group (). Holding roll length , face coverage, and POA metric fixed, changing drops 4-model mean POA by on Dice Path () and on Dice Face (). - Within-Environment
VOIControl (Table R6-B): Holding canvases, shapes, and metrics fixed, replacing commutative XOR () with order-sensitive opaque layer occlusion () drops 4-model mean Match Rate by () and IoU by (). - Cross-Game Metric Alignment (Table R1): Replacing task-specific binary thresholds with unified continuous partial credit across all four games (Appendix Tables 7 & 14) eliminates metric-strictness asymmetry and restores the exact predicted ordering: . Under strict all-or-nothing pass rates, VOI () exceeds Dice Face Strict Pass (, Table 15) by .
Table R6-A: Within-Environment Dice Control ( 1D Commutative Subgroup vs. 2D Non-Abelian Group, , %)
| Model | Dice Path (Commutative) PA / POA | Dice Path (Non-Abelian) PA / POA | Path -Axis Cost POA (pp) | Dice Face (Commutative) PA / POA | Dice Face (Non-Abelian) PA / POA | Face -Axis Cost POA (pp) |
|---|---|---|---|---|---|---|
| Gemini-3.1-Pro | 100.0 / 89.0 | 100.0 / 65.0 | +24.0 | 78.0 / 56.0 | 62.5 / 19.0 | +37.0 |
| Qwen3.5-397B | 99.0 / 85.0 | 97.0 / 59.0 | +26.0 | 91.0 / 64.0 | 84.0 / 30.0 | +34.0 |
| Qwen3.6-Plus | 98.0 / 79.0 | 94.0 / 49.0 | +30.0 | 96.0 / 66.0 | 92.0 / 32.0 | +34.0 |
| GPT-5.4 | 84.0 / 52.0 | 60.0 / 20.0 | +32.0 | 58.0 / 34.0 | 33.5 / 7.5 | +26.5 |
| 4-Model Mean | 95.3 / 76.3 | 87.8 / 48.3 | +28.0 | 80.8 / 55.0 | 68.0 / 22.1 | +32.9 |
Table R6-B: Within-Environment VOI Control ( Commutative VOI-XOR vs. Order-Sensitive VOI-Occlusion, )
| Model | VOI-XOR (: Order-Invariant) Match Rate (%) / IoU | VOI-Occlusion (: Order-Sensitive) Match Rate (%) / IoU | Within-Env -Axis Cost Match Rate (pp) / IoU |
|---|---|---|---|
| Gemini-3.1-Pro | 30.0 / 0.655 | 16.0 / 0.512 | +14.0 / +0.143 |
| Qwen3.5-397B | 23.0 / 0.690 | 12.0 / 0.535 | +11.0 / +0.155 |
| Qwen3.6-Plus | 44.0 / 0.785 | 25.0 / 0.620 | +19.0 / +0.165 |
| GPT-5.4 | 15.0 / 0.630 | 6.0 / 0.465 | +9.0 / +0.165 |
| 4-Model Mean | 28.0 / 0.690 | 14.8 / 0.533 | +13.2 / +0.157 |
Table R1: Cross-Family Comparison under Harmonized Continuous Partial Credit and Strict/Quasi-Match Thresholds (%)
| Subject | VOI () Pattern IoU [Tab. 7/14] | VOI () Dice Score [Tab. 7] | Lamp () Cont. Score [Tab. 14] | Dice Face () Face Acc. [Tab. 14] | Rope () Stars/3 [Tab. 14] | VOI () Quasi-Match () | Dice Face () Strict Pass [Tab. 15] |
|---|---|---|---|---|---|---|---|
| Human () | 94.5 | 97.1 | 89.2 | 60.0 | 68.3 | 88.0 | 18.00 |
| Gemini-3.1-Pro | 65.5 | 75.3 | 71.6 | 26.3 | 66.5 | 52.5 | 3.19 |
| Qwen3.5-397B | 69.1 | 78.5 | 77.0 | 45.7 | 27.6 | 54.0 | 0.00 |
| Qwen3.6-Plus | 78.6 | 86.2 | 39.4 | 44.3 | 41.7 | 68.5 | 5.98 |
| GPT-5.4 | 63.2 | 74.6 | 33.9 | 16.0 | 41.4 | 44.0 | 0.00 |
| GLM-5V-Turbo | 54.5 | 67.7 | 25.9 | 23.2 | 37.3 | 36.5 | 0.00 |
| Qwen3-VL-Plus | 50.3 | 64.6 | 36.9 | 25.3 | 35.4 | 31.0 | 0.00 |
| Step-1O-Turbo-Vision | 30.3 | 43.3 | 12.6 | 12.9 | 20.2 | 11.5 | 0.00 |
| 7-Model Mean | 58.8 | 70.0 | 42.5 | 27.7 | 38.6 | 42.6 | 1.31 |
Clarifying broader significance, taxonomy controls, and inverse specificity (W3)
W3: Bottleneck of inverse mental simulation and matched forward controls
We evaluated four frontier models on paired forward vs. inverse instances (20 per tier) in Dice () and Lamp () under identical scenes and simulators (Tables R5-A & R5-B). On Dice, Forward Roll reaches mean POA, whereas Inverse Path drops by () and Inverse Face drops by (). On Lamp, Forward Kinematics (FK) reaches mean SR ( error), whereas Inverse Kinematics (IK) drops SR by () and nearly triples error (), demonstrating that inverting physical and spatial transitions is a distinct, severe bottleneck beyond forward prediction.
Table R5-A: Matched Forward vs. Inverse Control on Non-Standard Dice (, %)
| Model | Forward Roll (PA / POA) | Inverse Path (PA / POA) | Inverse Face (PA / POA) | Forward-to-Path Drop POA (pp) | Forward-to-Face Drop POA (pp) |
|---|---|---|---|---|---|
| Gemini-3.1-Pro | 100.0 / 86.0 | 100.0 / 65.0 | 62.5 / 19.0 | +21.0 | +67.0 |
| Qwen3.5-397B | 98.0 / 81.0 | 97.0 / 59.0 | 84.0 / 30.0 | +22.0 | +51.0 |
| Qwen3.6-Plus | 97.0 / 78.0 | 94.0 / 49.0 | 92.0 / 32.0 | +29.0 | +46.0 |
| GPT-5.4 | 82.0 / 48.0 | 60.0 / 20.0 | 33.5 / 7.5 | +28.0 | +40.5 |
| 4-Model Mean | 94.3 / 73.3 | 87.8 / 48.3 | 68.0 / 22.1 | +25.0 | +51.2 |
Table R5-B: Matched Forward Kinematics (FK) vs. Inverse Kinematics (IK) on Articulated Lamp ()
| Model | Forward (FK) SR (%) / CD (px) | Inverse (IK) SR (%) / CD (px) | SR Drop (pp) | Error Increase CD (px) |
|---|---|---|---|---|
| Gemini-3.1-Pro | 96.0 / 6.8 | 87.0 / 21.5 | +9.0 | +14.7 |
| Qwen3.5-397B | 93.0 / 9.2 | 80.0 / 23.8 | +13.0 | +14.6 |
| Qwen3.6-Plus | 78.0 / 21.4 | 37.0 / 60.2 | +41.0 | +38.8 |
| GPT-5.4 | 76.0 / 23.5 | 36.0 / 62.5 | +40.0 | +39.0 |
| 4-Model Mean | 85.8 / 15.2 | 60.0 / 42.0 | +25.8 | +26.8 |
Clarifying broader significance, taxonomy controls, and inverse specificity (W1)
W1: Controlled 2D scope and relevance to real-world world models
In realistic 3D embodied settings, action-planning failures are heavily confounded with low-level visual perception errors (depth ambiguity, occlusion, segmentation noise). Following the controlled paradigm of PHYRE and CLEVRER, SIMVERSE pairs clean renderings with exact symbolic states to isolate internal state propagation and constraint inversion from perceptual noise. Even in noise-free 2D environments, frontier MLLMs struggle severely (Gemini-3.1-Pro: 52.69%, GPT-5.4: 27.37% vs. Human 75.50%; 5/7 models score 0.0% strict pass on
Dice Face), exposing two fundamental bottlenecks required by any general world model: (i) non-Abelian cumulative state tracking (O2) across multi-step 3D rotations (where 7-model mean POA decays monotonically across step depth t∈{1,2,3,4,≥5} from 59.7%→30.1%→16.2%→8.8%→4.6%), and (ii) continuous constraint inversion (A2) over collision-free manifolds (Lamp) and timed rigid-body dynamics (Rope).