acceptodds
Under review as a conference paper at ICLR 2027

Mental Simulation for Vision-Language Models on Intuitive Physics

Abstract

Vision-language models (VLMs) excel at interpreting what is visible, but can often struggle with future-dependent questions whose answers hinge on events that have not yet occurred. Such questions require anticipating how a scene will unfold—something humans do naturally through mental simulation. We propose a test-time approach that gives VLMs an analogous capability. To complement a VLM’s internal text-based reasoning, we use a generative video model to produce multiple plausible future rollouts conditioned on the observed video. These “mental simulations” are fed back to the VLM as additional visual context before it answers. We evaluate this simulate-then-reason framework on three intuitive physics benchmarks: Physion, CLEVRER, and real-world Physics-IQ videos. Across a diverse set of open-source and proprietary VLMs, augmenting models with imagined futures can improve prediction accuracy over both observed-only baselines and strong test-time language reasoning baselines, suggesting that allocating test-time computation to explicit video-based mental simulation can enhance multi-modal reasoning about the physical world. Video examples are available on our project page. (https://projectpagexxx.github.io/mental-simulation/)

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language model, generative video model, intuitive physics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_brj1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the physical reasoning in Vision-Language Models (VLMs). It proposes a test-time method that uses imagined futures generated by external generative models as supplementary input. Experiments on three physics benchmarks show the effectiveness of the proposed mental simulation idea.

Critical strengths
  1. This paper is well written and easy to follow.
  2. The experiments are extensive, with comparisons covering both open-source and advanced commercial models.
Critical weaknesses
  1. Using predicted visual observations to improve reasoning has been extensively validated across many scenarios [1][2][3], and the performance gains from additional input information are hardly surprising. This paper is more of an incremental verification of existing conclusions, lacking innovative insights.
  2. The proposed method directly feeds predicted images as input to VLMs. Although intuitive, this incurs excessive computational overhead. In fact, multiple future predictions inevitably contain redundant information, e.g., background regions. It is suggested to further compress the input information, e.g., keyframes and textual descriptions, which would greatly enhance the practicality of the method.
  3. Lack of analytical experiments on computational overhead. Compared with the baselines, the authors should report how much additional time and GPU hours are incurred by using LVP, as well as the inference overhead introduced by the additional visual input.

[1] Video Language Planning. ICLR 2024.

[2] Genex: Generating an explorable world. ICLR 2025.

[3] Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm. CVPR 2026.

Questions for authors

Please refer to the issues raised in the Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer brj1

We thank the reviewer for the thoughtful suggestions and recognizing the clarity and breadth of our evaluation.

1. Contribution and novelty.

Our work investigates how well VLMs can reason about future physical events, and whether explicit video simulation can improve this ability beyond language-based reasoning. Prior works address related but different problems: GenEx focuses on spatial exploration, Video Language Planning on robotic planning, and Thinking with Video on the reasoning capabilities of video generation models themselves. In contrast, we study how generated future videos can help VLMs reason about physical and dynamic events beyond the observed video.

Beyond introducing a simulate-then-reason framework, we systematically evaluate this approach across diverse VLMs and physical scenarios. Our experiments demonstrate the value of explicit future visual evidence beyond language reasoning, while revealing how generation quality and the VLM's ability to interpret and integrate multiple futures affect performance.

2. Temporal redundancy and alternative representations.

We evaluate alternative approaches to future reasoning, including written future predictions, point tracks, and motion summaries (Table 2). The following results are evaluated on Physion and CLEVRER subsets:

Dataset / VLMWritten futureMoving tracksMoving summaryVisual IndependentVisual Joint
Physion / Qwen3-VL-235B61%54%52%71%66%
Physion / GPT-5.654%53%48%72%68%
CLEVRER / Qwen3-VL-235B72%61%60%87%82%
CLEVRER / GPT-5.678%79%80%91%95%

These results suggest that generated future videos provide useful information beyond the tested language- and motion-based reasoning approaches. The written predictions and motion representations are derived from the observed video, rather than compressed versions of the generated futures.

Our method already reduces visual input through temporal subsampling, rather than feeding all generated frames to the VLM. We agree that further compression of generated futures through selected keyframes or textual descriptions could be valuable for improving efficiency.

3. Compute cost and language-reasoning controls

To evaluate whether additional language-space reasoning can improve physical prediction, we include Structured prompting, which asks VLMs to reason about objects, motions, and interactions, and Future prompting, which asks them to anticipate how the scene will evolve. We also evaluate their vote@3 variants, matching the three downstream VLM calls used by Visual Independent (K=3). For example, on CLEVRER:

VLMObserved vote@3Structured vote@3Future vote@3Visual Independent
Qwen3-VL-235B68.6%68.4%68.3%88.2%
Kimi-K3 (max)90.1%89.8%89.3%94.8%
GPT-5.685.5%85.3%85.9%93.4%

These results show that explicitly simulating future dynamics in visual space provides benefits beyond prompting VLMs to reason about the future in language space, even with repeated inference and across capable VLMs.

Regarding computational cost, LVP-14B takes approximately 210 seconds of sampling per 49-frame rollout on a single H100 GPU. Generated videos are cached and reused across VLM evaluations. While video generation introduces additional latency, its computational cost is not directly comparable to VLM inference, which has different model sizes, hardware requirements, and inference characteristics. While generation adds computational cost, our study focuses on understanding the value of explicit future simulation for physical reasoning.

Thank you for the detailed response. Compared to prior work that uses predicted visual observations for reasoning, the main novelty lies in the new application rather than in the methodology. I will therefore maintain my score.

Reviewer_J53QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a test-time “simulate-then-reason” framework that enhances vision-language models (VLMs) with visual mental simulation, the ability to generate and reason over multiple plausible future video rollouts before answering future-dependent questions. The method uses a diffusion-based generative video model to sample K continuations from the observed video segment, which are then provided as additional visual context to the VLM. The VLM can either reason independently over each extended video and aggregate via majority vote (Independent), or jointly reason over all rollouts and the observed clip (Joint). The approach is evaluated on three intuitive physics benchmarks: Physion, CLEVRER, and a repurposed subset of Physics-IQ, across a diverse set of open-source and proprietary VLMs (e.g., Qwen3-VL, Gemini, GPT-5.6/6). Results show consistent gains on CLEVRER, more mixed outcomes on Physion, and exploratory improvements on real-world Physics-IQ.

Critical strengths
  • Simple and grounded idea: The proposal to externalize mental simulation via video generation is elegant and well-motivated by human cognition. Unlike prior work that simulates in language space (e.g., CoT), this approach leverages the visual nature of physical dynamics, offering a complementary path for test-time reasoning.
  • Extensive and rigorous evaluation: The paper evaluates across 11 VLM configurations, including multiple sizes and families, and three benchmarks spanning synthetic (Physion, CLEVRER) and real-world (Physics-IQ) domains. This diversity strengthens the generality claim and avoids overfitting to a single model or dataset.
  • Useful ablations: The use of ground-truth futures (Table 3) and human-rated rollout plausibility (Tables 4–5) provides strong evidence that the quality of the imagined future directly impacts downstream reasoning. The large gap to GT futures (e.g., +25–35 pp on Physion) highlights room for improvement and validates the direction.
  • Clear pipeline in Alg. 1: The simulate-then-reason framework is cleanly formalized and reproducible in principle. The distinction between independent and joint integration strategies is well-articulated and empirically meaningful.
Critical weaknesses
  • Computation cost is not reported: The paper frames the method as allocating test-time compute, but never reports the compute it uses. Generating 3 future videos per question with a large diffusion video model (the default setting) is likely much more expensive than the vanilla VLM or the language baselines. Without compute numbers or a compute-matched baseline (e.g., language reasoning with more samples or a higher reasoning budget), it is unclear whether this is a fair comparison.
  • The gap between Independent and Joint is not explained thoroughly. For example, Qwen3-VL-8B does much better with Independent than Joint on CLEVRER (74.2% vs 57.5%), while stronger models do better with Joint. The two settings also differ in input format: Joint gives the VLM four videos in one prompt. Table 6 suggests the format itself hurts smaller models: Qwen3-VL-8B’s Joint control (no generated futures) scores 51.3% on CLEVRER, compared with 63.2% for Observed Single. This is not investigated.
  • Gains are mixed and shrink for stronger models. On Physion, some strong models get worse with visual simulation, e.g., Gemini 3.7-Flash drops from 66.1% to 58.6% with Independent, and Kimi-K3 max from 67.2% to 61.6%. Physics-IQ has only 66 questions, with wide confidence intervals, so those results are not very reliable.
  • Heavy reliance on one generator, and the generator comparison is not controlled. The alternative generators (Cosmos, LongCat) give much smaller or even negative gains. They also receive different numbers of context frames, frame rates and prompts (Appendix A.2), so it is unclear whether LVP’s advantage comes from the model itself or from the setup. Important ablations are missing. The number of rollouts is always 3; what about 1, or more such as 5? How far into the future the generator predicts is also never varied.
Questions for authors
  • What is the compute cost (e.g., time and GPU cost per question) of generating the rollouts in your default setting, compared with the vanilla VLM and the language baselines?
  • How does the method compare with language reasoning given a similar compute budget?
  • What is the effect of the number of rollouts (e.g., K = 1 or 5) and of the rollout horizon?
  • Can you compare the generators under matched conditions (same context frames, frame rate and prompts)?
  • Can you separate the effect of the input format (one vs. four videos) from the benefit of the generated futures when comparing Independent and Joint?
1 comment
AuthorsOpenReview ↗

Response to Reviewer J53Q

We thank the reviewer for the constructive feedback and for recognizing the intuitive motivation, broad evaluation across VLMs, and useful ablations.

1. Compute cost and language-reasoning controls

To evaluate whether additional language-space reasoning can improve physical prediction, we include Structured prompting, which asks VLMs to reason about objects, motions, and interactions, and Future prompting, which asks them to anticipate how the scene will evolve. We also evaluate their vote@3 variants, matching the three downstream VLM calls used by Visual Independent (K=3). For example, on CLEVRER:

VLMObserved vote@3Structured vote@3Future vote@3Visual Independent
Qwen3-VL-235B68.6%68.4%68.3%88.2%
Kimi-K3 (max)90.1%89.8%89.3%94.8%
GPT-5.685.5%85.3%85.9%93.4%

These results show that explicitly simulating future dynamics in visual space provides benefits beyond prompting VLMs to reason about the future in language space, even with repeated inference and across capable VLMs.

Regarding computational cost, LVP-14B takes approximately 210 seconds of sampling per 49-frame rollout on a single H100 GPU. Generated videos are cached and reused across VLM evaluations. While video generation introduces additional latency, its computational cost is not directly comparable to VLM inference, which has different model sizes, hardware requirements, and inference characteristics. Our goal is to study the value of explicit future visual evidence, rather than claim greater compute efficiency over language-only reasoning.

2. Joint vs. Independent Reasoning

Our results suggest that stronger VLMs can benefit from comparing multiple imagined futures jointly, while smaller VLMs may perform better when reasoning over each future independently.

We agree that the input format may contribute to this difference. However, on CLEVRER, both strategies improve over their respective observed-only controls across all 11 VLM configurations (Table 6). Even for Qwen3-VL-8B, Visual Joint improves by +6.2 pp, while Visual Independent improves by +10.8 pp over their corresponding controls.

Our additional disagreement analysis on CLEVRER further supports this observation:

VLM# Disagreement questionsIndependent vote (acc)Joint reasoning (acc)
GPT-5.614969.8%85.2%
Qwen3-VL-8B18059.4%47.8%

For GPT-5.6, Joint corrects 32 Independent errors while introducing only 9 new errors. In contrast, Qwen3-VL-8B performs better with Independent voting, suggesting that processing multiple videos together may be more challenging for smaller VLMs.

3. Benefits of Future Visual Evidence for Strong VLMs

Our results show that even strong VLMs can benefit substantially from informative future visual evidence. While the gains from generated futures vary across models on Physion, our ground-truth future experiments help explain this variation. On the Physion subset (Table 3), GPT-6 achieves 63% with the observed Joint control, 80% with LVP-generated futures, and 99% with ground-truth futures. This substantial gap suggests that improving video generation quality could further unlock the benefits of temporal visual simulation, even for capable VLMs.

4. Number of rollouts

To study the benefit of multiple imagined futures, we compare reasoning with a single rollout (K=1) against majority voting over three rollouts (K=3). For a fair comparison, we use the same generated videos, VLMs, and evaluation questions. The K=1 accuracy is averaged across the three individual rollouts.

CLEVRER VLMK=1 (avg.)K=3 (vote)Improvement (95% CI)
GPT-5.691.4%93.4%+2.0 pp [1.2, 2.8]
Gemini-2.5-Pro85.3%89.1%+3.8 pp [2.7, 4.9]

These results show that aggregating multiple imagined futures improves prediction over relying on a single continuation, supporting our motivation to consider multiple possible futures.

5. Video generators

We use LVP as our main generator because it produces temporally consistent video continuations, providing a useful setting for studying how VLMs benefit from imagined futures. We also evaluate Cosmos and LongCat on the same questions using the same downstream Joint reasoning procedure (Table 3). For GPT-5.6 on CLEVRER, accuracy improves from 81% with the observed control to 86% with Cosmos, 89% with LongCat, and 95% with LVP. These results show that the benefits of generated futures are not limited to a single video model.

All generators use the same observed video segment, with conditioning frames sampled according to their respective input requirements. Our main LVP experiments use fixed, benchmark-specific prediction horizons. We agree that exploring different horizons could provide further insight into their effects on physical reasoning.

Reviewer_JtMyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method to enhance VLMs’ physical reasoning ability at test time by supplementing their input with imagined future videos generated by a video generative model. A video generation model produces multiple possible continuations conditioned on the observed video. The framework combines these imagined futures through either independent reasoning followed by majority voting or joint reasoning over multiple rollouts. Experiments on Physion, predictive questions from CLEVRER, and a repurposed Physics-IQ subset report improvements over observed-only and language-based reasoning baselines, with gains varying across models and datasets.

Critical strengths
  1. The idea is presented clearly and intuitively. The motivation is easy to follow, and the figures help explain how generated future videos can support VLM reasoning.

  2. The method is a training-free framework applied at test time. It can be used with existing VLMs, including proprietary models, without updating their parameters, making it broadly applicable.

  3. The design for combining multiple rollouts is simple and clever. Independent reasoning followed by majority voting and joint reasoning provide two ways to incorporate multiple possible futures. The finding that different VLMs benefit from different integration strategies is also informative.

Critical weaknesses
  1. Limited methodological novelty. The core idea is closely related to MindJourney (Yang et al., 2025), which also uses generated visual evidence to support VLM reasoning at test time, although it focuses on spatial reasoning. The main additions here appear to be video-conditioned continuation and two rollout integration strategies. The paper should explain more clearly what these additions contribute beyond prior work.

  2. Limited evaluation coverage. The evaluation mainly relies on two simulated benchmarks, Physion and CLEVRER, while the real-world Physics-IQ evaluation contains only 66 questions. This provides limited evidence of generalization across physical settings. Focusing on CLEVRER’s predictive questions is reasonable, but the selection of only 1,000 questions needs justification and a clear sampling protocol. Evaluation on Physion++ and the temporal prediction tasks in ContPhy would help test whether the method handles latent physical properties and more complex dynamics.

  3. Missing comparisons with existing physical reasoning methods. The experiments compare against several prompting, voting, and tracking baselines, but do not establish how the framework compares with existing methods for augmenting physical reasoning at inference time. PhysMind (Yang et al., 2026), which also evaluates future prediction on CLEVRER, is a particularly relevant comparison. PhysAgent (Chow et al., 2025) offers another useful reference. Comparisons should use matched observations and VLM backbones where possible, and report inference costs.

  4. Insufficient handling of unreliable generated futures. The generator ablations show that the benefit depends strongly on the generation configuration and can become negative. Incorrect simulated events may therefore propagate into the VLM’s final answer. The human-rated quality analysis is useful, but remains retrospective and does not establish a scalable way to assess rollout reliability at inference time. Moreover, physical plausibility does not necessarily imply a correct prediction of the queried event. The paper should analyze whether the VLM can identify and discount misleading rollouts, and when generated futures introduce errors.

  5. Inconsistent gains across settings. Improvements are most consistent on CLEVRER, while some models perform worse on Physion. For example, Gemini-3.7-Flash drops from 66.2% with observed vote@3 to 58.6% with Visual–Independent. On Physics-IQ, many paired gains have confidence intervals that include zero. Although the paper acknowledges this variability, it provides limited guidance on when simulation is beneficial and which integration strategy should be used.

Questions for authors
  1. Clarification of the contribution. What is the main contribution over prior world-model-assisted VLM reasoning methods such as MindJourney? Which methodological component or empirical finding do the authors consider the central advance?

  2. Clarification of the evaluation subsets. How were the 1,000 CLEVRER questions and 66 Physics-IQ examples selected? Were these subsets defined before generating rollouts, and what inclusion or exclusion criteria were applied? This would help assess how representative the reported results are.

  3. Additional analysis of existing results. How often does adding generated rollouts correct a baseline error, and how often does it turn a correct baseline answer into an incorrect one? For examples where rollout-level answers disagree, does joint reasoning improve on majority voting? These results would help assess whether the integration strategies reliably handle conflicting future hypotheses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer JtMy (1/2)

We thank the reviewer for the thoughtful feedback and for recognizing the motivation and broad applicability of our training-free approach.

1. Contribution and novelty

MindJourney uses generative world models primarily to explore different spatial viewpoints of a scene. We investigate a complementary problem of predicting how a physical scene may evolve beyond what’s observed. Answering questions about future motion requires anticipating temporal dynamics, not only imagining unseen viewpoints. We condition video generation on the observed video segment, sample alternative future continuations, and study how VLMs reason from these hypotheses.

Our contribution is therefore a systematic investigation of test-time temporal mental simulation for future-dependent physical reasoning. In addition to demonstrating its potential, we compare it against observed-only and language-based reasoning, examine how VLMs integrate multiple possible futures, and study how the quality of generated futures and the VLM's ability to interpret them affect reasoning performance. This establishes and characterizes a complementary visual-dynamic reasoning pathway; we will make the distinction from related visual-imagination work clearer.

2. Error correction analysis

We conducted the requested paired, per-question analysis on the final predictions for all 11 VLM configurations. For each question, we count corrected (observed baseline wrong, visual answer correct) and harmed (baseline correct, visual answer wrong). Visual Independent is paired with Observed vote@3; Visual Joint is paired with its Observed Joint control. The results are computed from the predictions underlying Table 1 across all 11 VLM configurations.

BenchmarkIntegrationCorrected (%)Harmed (%)Net gain (pp)Configurations improved
CLEVRERIndependent16.04.5+11.511/11
CLEVRERJoint15.03.4+11.611/11
PhysionIndependent21.720.1+1.57/11
PhysionJoint20.916.5+4.37/11
Physics-IQIndependent22.915.6+7.39/11
Physics-IQJoint21.210.9+10.311/11

On CLEVRER, corrections consistently outweigh harms across all 11 VLM configurations and both integration strategies. Results on Physion are more mixed, suggesting that the benefit depends on the quality of generated futures and how VLMs interpret them.

To better understand this, we compare generated and ground-truth futures on Physion subset (main paper Table 3). For GPT-6, accuracy improves from 63% with observed Joint control to 80% with generated futures and 99% with ground-truth futures. This substantial gap suggests that more accurate future generation could further improve reasoning performance.

3. Joint vs. Independent reasoning

Our results suggest that the best way to integrate multiple imagined futures depends on the VLM. Stronger VLMs can benefit from jointly considering futures, while smaller models may perform better when evaluating each future independently.

We further analyze questions where the three independently evaluated rollouts produce different answers in CLEVRER:

VLM#Disagreement questionsIndependent vote (acc)Joint reasoning (acc)
GPT-5.614969.8%85.2%
Qwen3-VL-8B18059.4%47.8%

For GPT-5.6, Joint reasoning corrects 32 Independent errors while introducing only 9 new errors. In contrast, Qwen3-VL-8B performs better with Independent voting, suggesting that reasoning over multiple videos simultaneously may be more challenging for smaller VLMs. In such cases, evaluating each future separately can be more effective than combining them in a single prompt.

AuthorsOpenReview ↗

Response to Reviewer JtMy (2/2)

4. Clarification of the evaluation subsets

For CLEVRER, we select 1,000 questions from over 3,000 eligible predictive questions due to the computational cost of generating multiple future rollouts and evaluating many VLMs. We use a fixed, answer-balanced random sampling procedure, independent of model performance. All methods are evaluated on the same 1,000 questions to ensure fair comparisons.

For Physics-IQ, we use all 66 scenes, generating question/answer options from the video descriptions with GPT and manually verifying the questions and answers. We acknowledge that the relatively small number of scenes leads to greater statistical uncertainty, and view these results as preliminary evidence on real-world physical interactions.

5. Additional benchmarks and related methods

We thank the reviewer for these suggestions. Our current experiments already cover diverse physical interactions in Physion and CLEVRER, as well as real-world examples from Physics-IQ. We would be happy to include additional evaluations on Physion++ and ContPhy in the camera-ready version.

We also appreciate the references to PhysAgent and PhysMind, which explore complementary ways to enhance VLM physical reasoning. PhysAgent incorporates specialized perception models and physical knowledge, while PhysMind uses executable 3D scenes and physics simulation. In contrast, we study how generated future videos support physical reasoning across different frozen VLMs. Although PhysMind also evaluates predictive questions on CLEVRER, differences in question formats, evaluation protocols, and backbones prevent direct comparison of the reported numbers. We will clarify these connections in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.