acceptodds
Under review as a conference paper at ICLR 2027

Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing

Abstract

Physical reasoning from video requires inferring latent physical properties and dynamics beyond direct observation. Direct VLM inference remains unreliable on complex physical tasks without explicit modeling and validation, while predefined tool pipelines rely on task- and domain-specific priors that limit generalization across materials, dynamics, and reasoning tasks. We introduce Asking the World (ATW), a generalist agent that constructs and interrogates task-relevant executable worlds through two adaptive stages: World Modeling calibrates a world from video, while World Probing queries, simulates, and intervenes on it to obtain question-relevant evidence. Rather than prescribing the operations in either stage, ATW determines how to model and probe according to the scene and question. We develop PolyWorld Engine, a lightweight and highly programmable Warp-based multiphysics simulator for constructing and probing worlds with rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions. CEM-based system identification recovers task-relevant dynamics during World Modeling. The resulting world becomes an active workspace for question-directed physical experiments rather than a predetermined downstream tool. We evaluate ATW on CLEVRER, ContPhy, and three real-world scenarios. Using Gemini-3-Flash as its base VLM, ATW achieves 80.82% overall per-question accuracy on CLEVRER, improving direct Gemini-3-Flash by 46.50 points, GPT-5.5 by 13.58 points, and PhysMind by 8.27 points. On ContPhy, it reaches 70.56% overall accuracy, surpassing Gemini-3-Flash by 28.10 points and GPT-5.5 by 3.53 points. Across the three real-world scenarios, ATW achieves 71.67% accuracy, 28.33 points above GPT-5.5. These results establish agentic world modeling and probing as an effective, execution-grounded approach to generalist physical reasoning.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
generalist physical reasoning, agentic reasoning, world modeling, inverse simulation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_cnw4OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Asking the World (ATW) to address weak physical reasoning in VLMs and the limited flexibility of fixed tool pipelines. ATW first builds and calibrates an executable physical world from a video and its questions, then selects queries, simulations, and interventions to gather evidence and answer the questions. The authors develop the PolyWorld engine to support rigid bodies, soft bodies, cloth, ropes, and fluids, and use CEM to fit physical parameters to video observations. Experiments show clear gains on several physical reasoning tasks without fine-tuning on the benchmarks, while ablation studies show the benefits of physical simulation and adaptive feedback.

Critical strengths
  1. The paper proposes a well-designed framework that combines world modeling with active experiments and supports its answers with simulation evidence that can be checked.
  2. The framework supports several material types and question types and works with two VLM backbones.
  3. The experiments examine the roles of perception, physical simulation, parameter fitting, feedback-based revision, and object identity matching.
Critical weaknesses
  1. Unequal comparison settings: Some VLM baselines receive only eight frames, while ATW uses extra perception tools and multiple simulation rounds. Input information and compute budgets are therefore not fully matched.
  2. Limited real-world evaluation: The evaluation includes only 60 videos. The counterfactual answers rely on human judgment and are not verified through real interventions.
  3. Unclear contributions of individual adaptive steps: World-model revision and additional probing are introduced together. Their separate benefits, and the gains from extra computation, remain unclear.
  4. The paper does not fully report the physics solver details, CEM settings, runtime, or compute costs.
  5. The method mainly relies on a single fitted world and does not explicitly account for uncertainty in its physical parameters. A good fit to the observed video does not always lead to correct predictions after an intervention.
Questions for authors
  1. How does the system decide that the current world is accurate enough to start probing? Does it use a fitting error threshold, the VLM’s judgment, or a maximum number of iterations? (Section 3.1)

  2. The appendix mentions video-fallback and unresolved, but does not fully explain how they work. Does fallback let the VLM answer directly from the video? How often is it used, and how are unresolved cases counted in the final accuracy? (Appendix D)

  3. Which physical parameters are estimated by the vision modules, set by the agent, optimized with CEM, or kept fixed? Are the parameter ranges and fitting losses set separately for each dataset? (Section 3.3 and Appendix A.2)

  4. Goal-driven accuracy on ContPhy drops from about 73.2% with Single-pass to about 61.0% with Full ATW. Is this mainly due to errors in the world model, the probing strategy, or random variation across runs? (Figure 6)

7 comments
AuthorsOpenReview ↗

Response to Reviewer cnw4 (Part 1/5)

Thanks for your careful review and constructive feedback. We are pleased that you appreciate ATW’s combination of world modeling and active probing, its use of simulation evidence that can be checked, and its support for different materials and VLM backbones.

Thank you also for your suggestions on the experimental comparisons, implementation details, and limitations. Below, we address each of your comments and questions.

W1: Unequal comparison settings: Some VLM baselines receive only eight frames, while ATW uses extra perception tools and multiple simulation rounds. Input information and compute budgets are therefore not fully matched.

A: Thanks for pointing out the differences in comparison settings. We added 16/32-frame ablations for GPT-5.5 and Qwen3-VL on the CLEVRER and ContPhy ablation subsets and conducted the same comparison for GPT-5.5 on the real-world evaluation set. The table below reports overall per-question accuracy (%).

MethodInput framesCLEVRERContPhyReal-world
GPT-5.5869.4366.4643.33
GPT-5.51668.9767.3840.00
GPT-5.53271.7266.1550.00
Qwen3-VL838.8545.23—
Qwen3-VL1634.2542.46—
Qwen3-VL3236.5541.54—
ATW—82.3068.0071.67

We find that increasing the number of input frames improves some baseline results but does not yield consistent accuracy gains. Across the tested 8/16/32-frame settings, providing the baselines with richer temporal input does not eliminate ATW’s overall advantage or change the paper’s conclusions.

W2: Limited real-world evaluation: The evaluation includes only 60 videos. The counterfactual answers rely on human judgment and are not verified through real interventions.

A: Thank you for highlighting this issue.

We have not yet found a suitable real-world physical reasoning benchmark with high-quality observations and rich physical interactions. Constructing such a benchmark is challenging: it requires recording complete dynamic interactions and establishing reliable reference answers for predictive and counterfactual questions. Our real-world evaluation set was therefore designed to evaluate the effectiveness of ATW in real-world settings, rather than to serve as a comprehensive benchmark. Its scale remains limited by personnel and hardware constraints, and we will expand the diversity and scale of the evaluation in future work to further assess ATW’s generality.

Regarding counterfactual validation, hardware limitations make it difficult to reproduce identical initial conditions in physical experiments apart from the intended intervention. We therefore did not validate these counterfactual outcomes through actual physical interventions. Nevertheless, we took the following steps to make the reference answers as reliable as possible:

  • Human annotators inspected synchronized multi-view recordings to resolve occlusions and spatial ambiguities in individual views.

  • Human annotators viewed complete recordings extending beyond the model’s input—for example, until all balls had stopped—to establish the trajectories and collision relationships used to assess the specified counterfactual.

  • We retained only questions whose answers could be determined with high confidence and excluded cases with unresolved ambiguity.

  • Three annotators participated. Each question was assessed separately by at least two annotators, with a third designated to adjudicate disagreements. In practice, the two annotators agreed on every question, and no disagreement required adjudication by the third annotator.

AuthorsOpenReview ↗

Response to Reviewer cnw4 (Part 2/5)

W3: Unclear contributions of individual adaptive steps: World-model revision and additional probing are introduced together. Their separate benefits, and the gains from extra computation, remain unclear.

A: Thanks for raising this question. To distinguish the contributions of world-model revision and additional probing, we added ATW w/o Adaptive Probing as an intermediate variant between the existing Single-pass World Model and Full ATW configurations in the paper.

The new variant reuses Full ATW’s world-modeling assets while keeping the simulation backend and tools unchanged. It generates the complete probing plan in a single planning step before execution and then answers based on the execution evidence; intermediate feedback cannot trigger additional probes or changes to the plan.

Single-pass World Model and the new variant use the same probing and answer-generation procedures and differ only in world modeling, so their comparison assesses the benefit of world-model revision. The new variant and Full ATW share the same world assets, allowing their comparison to assess the benefit of autonomous probing that adapts to feedback.

The table below combines the new results with the existing results from Fig. 6, reporting overall per-question accuracy (%) on the same ablation subsets.

ConfigurationWorld-model revisionFeedback-driven multi-step probingCLEVRERContPhy
Single-pass World ModelDisabledDisabled75.4059.08
ATW w/o Adaptive ProbingRetainedDisabled77.7064.62
Full ATWRetainedRetained82.3068.00

Retaining world-model revision improves overall accuracy on CLEVRER and ContPhy by 2.30 and 5.54 percentage points, respectively. Further enabling feedback-driven multi-step probing with the same world assets yields additional gains of 4.60 and 3.38 percentage points, respectively. These results indicate that both adaptive steps contribute to overall performance: world-model revision provides a more reliable physical basis, while feedback-driven probing uses intermediate evidence to adjust subsequent queries and experiments.

W4: The paper does not fully report the physics solver details, CEM settings, runtime, or compute costs.

A: Thank you for pointing out the incomplete reporting of these implementation details and computational costs.

1. Physics solvers and CEM settings.

We have added Physical solvers and CEM configurations to Appendix A.2 of the revised manuscript. Specifically, rigid-body dynamics use event-driven time-of-impact detection and contact impulses; cloth uses position-based structural and bending constraints; rope systems use constrained dynamics for ropes and connected objects; fluids use a position-based fluid solver; and soft-body dynamics couple overall motion with deformation and contact response.

We also provide the CEM population size, elite fraction, number of generations at each fitting stage, and distribution update coefficient for each physical representation. For example, the rigid-body configuration uses a population size of 1024, an elite fraction of 0.08, 64 generations per stage, and a weight of 0.2 on the previous distribution. The appendix also describes the parameter bounds, retention of the incumbent candidate, and representation-specific initialization and local refinement.

AuthorsOpenReview ↗

Response to Reviewer cnw4 (Part 3/5)

W4 (continued)

2. Runtime and parallel execution.

On a single NVIDIA A100 GPU with 80GB of memory, the average runtime per scene for each benchmark is reported below. This includes world modeling, probing and answering, as well as time spent waiting for VLM API responses:

BenchmarkAverage runtime per scene
CLEVRER9.34 minutes
ContPhy11.11 minutes
Real-world6.11 minutes

This wall-clock time does not represent continuous GPU computation: while waiting for VLM API responses, the local GPU can process other scenes, and computation for a single scene may not fully utilize the GPU. Execution can therefore be organized by stage, processing multiple scenes in parallel on the same GPU to improve resource utilization.

With stage-wise parallel execution across scenes, we measured the actual elapsed time for sampled batches and divided each batch’s total runtime by its number of scenes to calculate the average processing time per scene. The resulting times for CLEVRER, ContPhy, and the real-world evaluation are 6.23, 7.11, and 4.89 minutes, respectively.

Furthermore, world modeling accounts for most of the runtime, and the constructed worlds are reusable. For the same scene, additional questions within the scope supported by the existing world do not require rebuilding it; they incur only the corresponding probing and answering costs. The world-modeling cost can therefore be amortized over more questions. The physical reasoning tasks evaluated in this paper generally do not require real-time responses.

W5: The method mainly relies on a single fitted world and does not explicitly account for uncertainty in its physical parameters. A good fit to the observed video does not always lead to correct predictions after an intervention.

A: Thanks for raising this issue. A single-view video is generally insufficient to uniquely determine all physical parameters, as different parameter combinations can produce similar visible motion. ATW aims to construct an executable world from observations that supports physical reasoning, rather than to guarantee recovery of a unique ground-truth parameter set. To examine the practical effect of parameter uncertainty on downstream answers, we conducted an additional experiment on 10 selected CLEVRER scenes.

1. Selection of different fitted worlds.

We fixed the perception and reconstruction assets from the existing experiments and performed parameter fitting with multiple random seeds under the original optimization configuration. From the candidate states retained during optimization and the final fitting results, we selected two complete parameter sets, denoted A and B. Both sets met the observational fit-quality requirements while differing substantially in their parameter values.

This selection ensures that we compare physical worlds with substantially different parameters that both explain the observations well. The scenes and parameters were fixed before question answering and scoring, and were not selected based on answer accuracy.

2. Downstream answering results.

After fixing the scenes and parameters, we loaded worlds A and B separately and reran probing and answering, comparing their results with the saved answers from the original experiment on the same selected subset. The table below reports overall per-question accuracy.

ConfigurationOverall accuracy
Worlds selected in the original experiment90.70%
World A88.37%
World B86.05%

Despite substantial differences in physical parameters, both alternative worlds maintain high accuracy. This indicates that, in these targeted cases, the inability to uniquely determine the parameters does not necessarily cause physical reasoning to fail; fitted worlds with different parameters that are consistent with the observations can still support effective reasoning.

Reviewer_Df67OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Asking the World (ATW), a training-free agent for physical reasoning from video. ATW first builds an executable world from the video. It reconstructs the relevant objects, chooses a physical representation for each one, and fits their physical parameters with CEM in PolyWorld Engine, a Warp-based simulator that covers rigid bodies, soft bodies, cloth, ropes and fluids. It then probes this world through rollouts, interventions and event queries, and answers each question from the resulting evidence. With Gemini-3-Flash as the backbone, ATW outperforms strong VLMs including GPT-5.5, as well as the training-free method PhysMind, on CLEVRER, ContPhy and three real-world scenarios.

Critical strengths

The paper presents a clear and well-motivated formulation: it separates building a world from questioning it, and builds only what the questions require. PolyWorld Engine brings several material types and their interactions into one environment. Its agent-facing interface supports branching rollouts and records the source of every result, which makes it potentially useful beyond this work. The results are strong, with especially large gains on counterfactual questions in CLEVRER. The experiments are solid. The comparison covers a wide range of baselines, from foundation VLMs and training-based video reasoners to training-free agents, and includes a blind baseline to check for language bias. The ablations are well designed. The step-by-step variants show the separate value of structured perception, physical execution and feedback-guided revision. The paired study of system identification and physics backends convincingly links lower fitting error to higher reasoning accuracy. Additional experiments on a second backbone and on alternative perception modules further support the design choices. Finally, the real-world evaluation uses multi-view recordings to obtain ground truth, and the paper is clearly written with helpful figures.

Critical weaknesses

W1. The sources of errors are not analyzed. ATW is a long pipeline with six stages: segmentation and identity binding, tracking and geometry, the choice of physical representation, parameter fitting, probing, and answer generation. The last stage is done by Gemini-3-Flash from compact textual evidence. The paper reports only final accuracy, so it is unclear which stages limit performance and where the remaining errors come from. The cross-backbone experiment in Fig. 8 replaces the model in all stages at once. It therefore cannot show how much the final answer depends on the language model’s reasoning rather than on the evidence itself.

W2. The frame budget of some baselines may limit their performance. According to Sec. 4.1, GPT-5.5, Qwen3-VL and GLM-4.6V receive only eight uniformly sampled frames. In contrast, the Gemini models receive the full video, and ATW processes the full video through its perception modules. Collisions in CLEVRER are brief, so eight frames may miss the key moments needed for explanatory and counterfactual questions. The same concern applies to the predictive questions in ContPhy and in the real-world set, where the motion just before the observation ends is important. Qwen3-VL also supports video input natively. Since GPT-5.5 is the strongest baseline and the comparison with it is a central claim, this setting may overstate the gap.

Questions for authors

Q1. On error analysis. I would appreciate a breakdown of errors by stage on subset. For example, the authors could sample incorrect answers on each benchmark and identify the first stage at which each one fails.

To isolate the answer stage, the authors could keep the world modeling and probing stages fixed, cache the evidence, and re-run only the answer step with other models such as GPT-5.5 or Qwen3-VL. Asking human annotators to answer a subset of questions from the same evidence would further show whether the evidence itself is sufficient.

Q2. On the number of input frames. Could the authors explain why eight frames were chosen? I would also suggest an ablation over the number of frames, for example 8, 16, 32 and 64, or the full video where the model supports it. Running this for GPT-5.5 and Qwen3-VL on the CLEVRER and ContPhy subsets from Sec. 4.3 and on the real-world set would show whether the gap to ATW remains when the baselines see the same temporal information.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Df67 (Part 1/3)

We sincerely thank you for your careful review, positive comments, and valuable suggestions. We are delighted that you appreciate ATW’s framework design, PolyWorld’s unified support for multiple material types and their interactions, and the design of our experiments and ablation studies. Your recognition is very encouraging to us.

Your suggestions are very helpful for improving our work. Following your suggestions, we have carried out further analyses and experiments. Below, we address each of your suggestions and questions.

W1a: The sources of errors are not analyzed. ATW is a long pipeline with six stages: segmentation and identity binding, tracking and geometry, the choice of physical representation, parameter fitting, probing, and answer generation. The last stage is done by Gemini-3-Flash from compact textual evidence. The paper reports only final accuracy, so it is unclear which stages limit performance and where the remaining errors come from.

Q1a: On error analysis. I would appreciate a breakdown of errors by stage on subset. For example, the authors could sample incorrect answers on each benchmark and identify the first stage at which each one fails.

A: Thank you very much for your valuable suggestion. Following your suggestion, we conducted a stage-wise error analysis on the CLEVRER and ContPhy ablation subsets and added §4.4 Error Analysis and Fig. 9 to the revised manuscript, presenting the distribution of error sources and representative failure cases.

Each incorrect answer is attributed to the earliest stage whose error affects the final answer. The percentages below are calculated over all incorrect answers in the corresponding benchmark.

BenchmarkSegmentationTracking & GeometryDynamics FittingWorld ProbingAnswer Generation
CLEVRER0.00%32.47%32.47%19.48%15.58%
ContPhy6.73%21.15%41.35%14.42%16.35%

Feedback-driven revisions in adaptive world modeling check and adjust the choice of physical representation, leaving few remaining errors of this type in our analysis. We group these errors under Tracking & Geometry. CLEVRER scenes are relatively simple, and among the incorrect answers analyzed, none were attributed to segmentation or identity binding during world modeling as the earliest source of error.

On CLEVRER, Tracking & Geometry and Dynamics Fitting are the main error sources. The former includes missed collisions and inaccurate pose tracking due to limited boundary-mask observations; the latter involves insufficient fitting accuracy in some scenes, resulting in incorrect simulation rollouts. World Probing errors include omitted collision queries that leave the evidence incomplete, while Answer Generation errors include VLM misinterpretation of execution evidence.

On ContPhy, Dynamics Fitting is the largest error source, followed by Tracking & Geometry. Representative cases include inaccurate parameter fitting that produces incorrect rollout results and depth inaccuracies that cause cloth penetration. Other errors include incomplete rope-system segmentation masks, identity misbinding or incorrect intervention targets during probing, and mismatches between the final answer and execution evidence.

Dynamics Fitting errors mainly occur in more complex scenes: insufficient fitting accuracy can cause simulation queries or rollouts to produce results that deviate from the actual motion, affecting the answer. This issue is more pronounced in scenes involving coupled interactions between different materials.

We also examined errors in the real-world evaluation. The observed error types fall within the categories above and primarily concern tracking and geometry, as well as dynamics-fitting accuracy. Given the limited number of incorrect answers, we include these observations as a qualitative supplement rather than reporting a separate stage-wise distribution.

AuthorsOpenReview ↗

Response to Reviewer Df67 (Part 2/3)

W1b: The cross-backbone experiment in Fig. 8 replaces the model in all stages at once. It therefore cannot show how much the final answer depends on the language model’s reasoning rather than on the evidence itself.

Q1b: To isolate the answer stage, the authors could keep the world modeling and probing stages fixed, cache the evidence, and re-run only the answer step with other models such as GPT-5.5 or Qwen3-VL. Asking human annotators to answer a subset of questions from the same evidence would further show whether the evidence itself is sufficient.

A: Thanks for this suggestion. The full-pipeline backbone replacement experiment in Fig. 8 primarily evaluates the ATW framework’s applicability across different VLMs. To further isolate the effect of answer generation and assess the effectiveness of the execution evidence, we conducted the following two additional experiments. Both tables report per-question accuracy.

1. Keeping the evidence fixed and replacing only the answer-generation model.

On the CLEVRER and ContPhy ablation subsets, we kept world modeling and world probing fixed, cached the resulting evidence, and replaced only the final answer-generation model, from Gemini-3-Flash to Qwen3-VL. For reference, the table also includes the existing results from Fig. 8, where Qwen3-VL is used throughout ATW.

ConfigurationCLEVRERContPhy
ATW (Gemini-3-Flash)82.30%68.00%
Same cached evidence, with Qwen3-VL used only for answer generation83.68%67.08%
ATW (Qwen3-VL, Fig. 8)76.32%62.46%

With the evidence held fixed, replacing the answer-generation model yields similar overall accuracy on both CLEVRER and ContPhy. This indicates that the evidence produced by ATW can be effectively used by different answer-generation models, without strong dependence on Gemini-3-Flash specifically.

2. Human answers based on the same evidence.

We further randomly selected scenes from the ablation subsets of both benchmarks and asked human participants to answer questions using the same evidence provided to the VLM at the answer-generation stage. Three participants took part, with each question answered separately by at least two participants; disagreements were adjudicated by the third. The Gemini-3-Flash results below are from the same selected scenes.

AnswererCLEVRERContPhy
Gemini-3-Flash83.18%68.75%
Human86.82%73.13%

Using the same evidence, human accuracy exceeds that of Gemini-3-Flash by 3.64 and 4.38 percentage points on the two benchmarks, respectively. This indicates that reasoning over the evidence still has a modest effect on the final results, while the execution evidence remains effective overall.

W2: The frame budget of some baselines may limit their performance. According to Sec. 4.1, GPT-5.5, Qwen3-VL and GLM-4.6V receive only eight uniformly sampled frames. In contrast, the Gemini models receive the full video, and ATW processes the full video through its perception modules. Collisions in CLEVRER are brief, so eight frames may miss the key moments needed for explanatory and counterfactual questions. The same concern applies to the predictive questions in ContPhy and in the real-world set, where the motion just before the observation ends is important. Qwen3-VL also supports video input natively. Since GPT-5.5 is the strongest baseline and the comparison with it is a central claim, this setting may overstate the gap.

Q2: On the number of input frames. Could the authors explain why eight frames were chosen?I would also suggest an ablation over the number of frames, for example 8, 16, 32 and 64, or the full video where the model supports it. Running this for GPT-5.5 and Qwen3-VL on the CLEVRER and ContPhy subsets from Sec. 4.3 and on the real-world set would show whether the gap to ATW remains when the baselines see the same temporal information.

A: Thanks for pointing out this issue and suggesting specific experiments. We added input-frame ablations on the CLEVRER and ContPhy ablation subsets used in the paper and on the real-world evaluation set to examine whether denser temporal observations improve baseline performance and whether the gap to ATW remains.

AuthorsOpenReview ↗

Response to Reviewer Df67 (Part 3/3)

W2/Q2 (continued)

1. Eight-frame setting and video-input interface.

The eight-frame setting in the paper was primarily chosen to keep evaluation costs within our API budget. Since input token usage mainly increases with the number of input images, increasing the input from 8 frames to 16/32 frames requires approximately 2×/4× as many input tokens, respectively.

Qwen3-VL natively supports video input. However, our evaluations consistently accessed it through OpenRouter, where the model’s API endpoint does not support direct video input. In addition, the corresponding model’s official hosted API service was recently discontinued, preventing us from conducting additional native-video evaluations through that official endpoint.

2. Results with different numbers of input frames.

We added 16/32-frame experiments for GPT-5.5 and Qwen3-VL on the CLEVRER and ContPhy ablation subsets and conducted the same comparison for GPT-5.5 on the real-world evaluation set. All results below report per-question accuracy (%).

CLEVRER

MethodInput framesExplanatoryPredictiveCounterfactualOverall
GPT-5.5878.3388.5753.5169.43
GPT-5.51679.4485.7152.4368.97
GPT-5.53283.8980.0056.7671.72
Qwen3-VL854.4447.1420.5438.85
Qwen3-VL1650.5634.2918.3834.25
Qwen3-VL3253.8922.8624.8636.55
ATW—85.0094.2975.1482.30

ContPhy

MethodInput framesPropertyPredictiveCounterfactualGoal-drivenOverall
GPT-5.5871.2051.1672.6073.1766.46
GPT-5.51673.6058.1469.8663.4167.38
GPT-5.53271.2053.4971.2368.2966.15
Qwen3-VL864.0046.5123.2924.3945.23
Qwen3-VL1659.2037.2128.7726.8342.46
Qwen3-VL3256.8038.3724.6631.7141.54
ATW—80.8059.3060.2760.9868.00

Real-world evaluation

MethodInput framesBilliards counterfactualBouncing-ball predictionToy-car predictionOverall
GPT-5.5835.0045.0050.0043.33
GPT-5.51640.0035.0045.0040.00
GPT-5.53240.0050.0060.0050.00
ATW—85.0070.0060.0071.67

On CLEVRER, GPT-5.5 improves on some question types, but the overall gain is limited, while Qwen3-VL’s overall performance does not improve with more frames. On ContPhy, increasing the frame count likewise does not produce consistent accuracy gains. In the real-world evaluation, GPT-5.5’s accuracy decreases with 16 frames and increases with 32 frames. Given the limited size of the real-world evaluation set, variability due to stochastic VLM responses may be more pronounced.

These additional results suggest that increasing the number of input frames has a limited effect on performance, with fluctuations potentially also influenced by stochastic VLM responses. Denser temporal observations do not eliminate ATW’s overall advantage across the three evaluations.

We hope our replies and supplementary results help resolve your questions. We welcome feedback on any remaining issues and will include the relevant experiments, analyses, and clarifications in the revised manuscript. Thank you again for your thoughtful review and suggestions, which have greatly helped us improve the paper.

Reviewer_rSv6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ATW, a training-free framework that builds executable physical worlds from videos and uses an agent to probe them through simulation and counterfactual interventions. It combines adaptive world modeling with the PolyWorld physics engine and evaluates the approach on CLEVRER, ContPhy, and three real-world scenarios.

Critical strengths
  1. The paper reports strong results on physical reasoning benchmarks. ATW achieves 80.82% accuracy on CLEVRER, compared with 72.55% for PhysMind, and 70.56% on ContPhy. These are meaningful improvements to existing methods.

  2. The ablations are useful. On CLEVRER, accuracy improves from 35.40% with video-only reasoning to 75.40% with a single-pass world model and 82.30% with full ATW. The separation between world construction and question-directed probing is also a sensible system design.

Critical weaknesses
  1. The main concern is what actually explains the improvement over PhysMind. PhysMind already reconstructs executable worlds and uses them for physical reasoning. ATW adds adaptive modeling and probing, but also uses stronger perception tools, system identification, and a different simulation backend. The current ablations do not clearly separate these contributions. Inference costs are also missing, making it difficult to assess whether the gains come from the proposed agentic mechanism or additional computation.

  2. The generalist claim is not fully supported by the evaluation. The real-world experiments contain only 60 examples across three relatively simple scenarios. Although PolyWorld supports multiple physical representations, the paper does not systematically test coupled interactions between different materials. Real-world counterfactual answers are also based on human judgments rather than executed interventions.

Questions for authors
  1. Can the authors clarify which components account for the improvement over PhysMind? In particular, can adaptive world modeling and multi-step probing be evaluated separately under the same perception and simulation setup? Reporting VLM calls, simulation rollouts, and inference time would also help assess the computational cost of these improvements.
  2. Can the authors provide evidence of reasoning over coupled physical systems, such as rigid-fluid or rigid-cloth interactions? For the real-world counterfactual evaluation, how were the human annotations validated, and were any predictions checked through actual physical interventions?
6 comments
AuthorsOpenReview ↗

Response to Reviewer rSv6 (Part 1/4)

We deeply appreciate your detailed and constructive feedback. We are greatly encouraged by your recognition of ATW’s effectiveness, informative ablation studies, and sensible design separating world construction from question-directed probing.

We believe our responses have addressed most of your concerns. Below, we provide detailed responses to all your comments.

W1a: The main concern is what actually explains the improvement over PhysMind. PhysMind already reconstructs executable worlds and uses them for physical reasoning. ATW adds adaptive modeling and probing, but also uses stronger perception tools, system identification, and a different simulation backend. The current ablations do not clearly separate these contributions.

Q1a: Can the authors clarify which components account for the improvement over PhysMind? In particular, can adaptive world modeling and multi-step probing be evaluated separately under the same perception and simulation setup?

A: Thanks for the questions! PhysMind also reconstructs executable worlds for physical reasoning, with a VLM selecting scene-dependent constraints and tools during world construction. ATW emphasizes feedback-driven revisions within world modeling and multi-step probing across diverse physical systems. To clarify the contributions of individual components, we supplemented the existing CLEVRER ablations with new experiments. We discuss perception and tracking tools first, followed by system identification and the simulation backend, and then adaptive world modeling and multi-step probing.

1. Perception and tracking tools

We chose our perception and tracking tools to support a wider range of physical systems. PhysMind uses FoundationPose to track object poses, which requires an input object mesh. ATW uses Track4World to track 3D points, allowing it to handle systems with evolving shapes, such as cloth and soft bodies. In the tracking-method ablation within our perception pipeline (Appendix C.2 / Fig. 10), the FoundationPose variant achieves 80.46% overall per-question accuracy on CLEVRER, compared with 82.30% for the default configuration—a difference of 1.84 percentage points. Our main reason for choosing Track4World is therefore its broader applicability across physical systems, rather than a specific advantage on CLEVRER.

2. System identification and the simulation backend

To examine system identification and the simulation backend, we added PhysMind model + Adam, which follows the physical modeling and parameter-fitting approach used in the original PhysMind method: its physical model fitted using Adam. Compared with PhysMind model + CEM, this configuration keeps the physical model unchanged and replaces CEM with Adam. The table below combines this new result with PhysMind model + CEM and Warp + CEM from Fig. 7 / Appendix C.3, together with fitting times and candidate evaluation counts. All three configurations are evaluated on the same CLEVRER ablation subset.

ConfigurationPer-question accuracy (%)Per-option accuracy (%)Total fitting time (s)Forward time (s)Non-forward time (s)RMSE (m)Candidate evaluations
PhysMind model + CEM53.5676.58447.99442.555.440.05913898,305
PhysMind model + Adam76.3289.72209.54102.59106.950.01410011,280
Warp + CEM82.7693.1942.0519.5722.480.0076555,163,704

We report mean times and median trajectory RMSE. Candidate evaluations are the median cumulative count per scene. Forward time is spent on simulation rollouts; non-forward time covers the remaining fitting work, mainly loss evaluation and optimization. For Adam, this also includes gradient backpropagation, which takes 84.39 s on average.

With PolyWorld Engine, our physics engine built on Warp, CEM can evaluate many more candidate parameter sets in less total fitting time. At each iteration, CEM samples a population of candidates, selects those with the lowest fitting losses as an elite set, and updates the sampling distribution using their mean and standard deviation. The search thus moves toward low-loss regions while retaining exploration. Among the evaluated configurations, Warp + CEM achieves the lowest trajectory RMSE and the highest downstream accuracy, supporting the combined contribution of our simulation backend and parameter-fitting strategy.

These results show that the simulation backend and system identification are important contributors to the quality of the executable world and, in turn, to physical reasoning accuracy on CLEVRER.

AuthorsOpenReview ↗

Response to Reviewer rSv6 (Part 2/4)

W1a/Q1a (continued)

3. Adaptive world modeling and multi-step probing

Adaptive world modeling helps ATW handle different physical scenes and build more accurate worlds. It uses observations and execution feedback to revise object segmentation, physical representations, and other modeling decisions, giving subsequent probing a more reliable basis. To separate its contribution from that of multi-step probing, we introduced ATW w/o Adaptive Probing alongside the Single-pass World Model and Full ATW configurations in the paper.

This new variant reuses Full ATW’s perception outputs, reconstructed geometry, and fitted world, keeping the simulation backend and tools unchanged. It generates a complete probing plan in a single planning step before execution and answers from the resulting execution evidence. Intermediate feedback cannot trigger additional probes or changes to the plan. Single-pass World Model and ATW w/o Adaptive Probing use the same probing and answer-generation procedure and differ only in world modeling. Comparing these two variants evaluates the contribution of adaptive world modeling, while comparing ATW w/o Adaptive Probing with Full ATW evaluates feedback-driven multi-step probing with the constructed world held fixed.

The table below combines the Single-pass World Model and Full ATW results from Fig. 6 with the new variant evaluated on the same ablation subset. All values are CLEVRER per-question accuracy (%). These agentic world-reasoning ablations were run independently of the physical-grounding ablations above.

ConfigurationExplanatoryPredictiveCounterfactualOverall
Single-pass World Model77.2282.8670.8175.40
ATW w/o Adaptive Probing79.4488.5771.8977.70
Full ATW85.0094.2975.1482.30

On CLEVRER, retaining adaptive world modeling while disabling feedback-driven multi-step probing improves overall accuracy from 75.40% to 77.70% over Single-pass World Model. Allowing feedback-driven multi-step probing on the same world assets raises it further to 82.30%. Both steps improve all three question categories, supporting the value of both a better constructed world and probes that adapt to execution feedback.

The same experiment on the ContPhy ablation subset shows a similar overall trend: the three configurations achieve 59.08%, 64.62%, and 68.00% accuracy, respectively. Adaptive world modeling adds 5.54 percentage points, and feedback-driven multi-step probing adds another 3.38 percentage points.

W1b: Inference costs are also missing, making it difficult to assess whether the gains come from the proposed agentic mechanism or additional computation.

Q1b: Reporting VLM calls, simulation rollouts, and inference time would also help assess the computational cost of these improvements.

A: Thanks for the suggestion. The two methods differ in the scope of world modeling and the responsibilities assigned to the VLM. ATW’s adaptive world modeling places substantial emphasis on accommodating diverse physical systems through scene-dependent modeling decisions and feedback-driven revisions. We therefore focus the cost comparison on their common task of probing and answering questions given an already constructed world. Under the same VLM configuration, we report VLM calls, API costs, simulation rollout time, and inference time for this stage on the CLEVRER ablation subset. Parameter-fitting times are reported in our response to W1a/Q1a.

MethodMean VLM calls per questionMean API cost per question (USD)Mean probing and answering time per question (s)Mean time per simulation rollout (ms)
PhysMind5.110.0163814.289.09
ATW8.290.0130411.141.94

Unlike parameter fitting, world probing does not require large batches of parallel simulation rollouts. Even without exploiting this batch-parallel advantage, ATW takes only 1.94 ms per rollout on average, compared with 9.09 ms for PhysMind. Both methods use a single-digit average number of rollouts per question, so simulation execution contributes negligibly to the total probing and answering time.

Although ATW makes more VLM calls on average, it achieves higher answer accuracy with a 20.39% lower API cost and a 21.99% shorter probing and answering time per question. Its performance gains therefore do not come at the expense of higher costs or longer inference time within this stage. These cost results complement the ablation in W1a/Q1a, which shows the accuracy benefit of feedback-driven multi-step probing.

AuthorsOpenReview ↗

Response to Reviewer rSv6 (Part 3/4)

W2a: The generalist claim is not fully supported by the evaluation. The real-world experiments contain only 60 examples across three relatively simple scenarios.

Real-world counterfactual answers are also based on human judgments rather than executed interventions.

Q2b: For the real-world counterfactual evaluation, how were the human annotations validated, and were any predictions checked through actual physical interventions?

A: Thanks for pointing out these issues.

1. Real-world evaluation.

Constructing real-world dynamic physical reasoning tasks requires complete recordings of motion and interactions, together with reliable reference answers for predictive and counterfactual questions. We have not yet found a real-world physical reasoning benchmark suitable for this study that combines high-quality observations with rich physical interactions. We therefore constructed this evaluation set to examine whether ATW can extend from simulated benchmarks to real-world physical settings, providing complementary evidence of its generality, rather than to establish a comprehensive real-world benchmark. Personnel and hardware constraints have limited the scale and scenario coverage of the current evaluation. Nevertheless, across the three evaluated scenarios—billiards counterfactual reasoning, bouncing-ball prediction, and toy-car prediction—ATW achieves accuracies of 85.00%, 70.00%, and 60.00%, respectively, outperforming both GPT-5.5 and Gemini-3-Flash in every category, with an overall accuracy of 71.67%. Our evaluation assesses generality across different physical systems and reasoning tasks in CLEVRER and ContPhy. The real-world results provide complementary evidence that the same framework extends to the evaluated real-world settings. We have carefully considered your suggestion and will expand the diversity and scale of the real-world evaluation to further assess ATW’s applicability across a wider range of physical interactions.

2. Validation of counterfactual answers.

For real-world counterfactual questions, hardware limitations make it difficult to ensure that all other initial conditions remain identical across repeated physical experiments apart from the intended intervention. We therefore did not validate counterfactual outcomes through actual physical interventions. However, we used complete multi-view observations, question selection, and annotation by multiple annotators to make the reference answers as reliable as possible. As described in Appendix B.3, annotators can inspect synchronized multi-view recordings to resolve occlusions and spatial ambiguities in individual views. Annotators also view the complete recordings, which extend beyond the model’s input—for example, until all balls have stopped. This helps establish the observed trajectories and collision relationships, which are then used to assess the specified counterfactual. We retain only questions whose answers can be determined with high confidence from this evidence and exclude cases with unresolved ambiguity.

Three annotators participated. Each question was assessed separately by at least two annotators, with a third designated to adjudicate disagreements. In practice, the two annotators agreed on every question, and no disagreement required adjudication by the third annotator.

3. Illustrative example.

In a billiards scene where only the white ball is initially moving, the first white–purple collision redirects its motion and triggers subsequent white–brown and purple–brown collisions. Based on the scene geometry and initial motion observed from multiple viewpoints, annotators can clearly determine that, if the purple ball were removed, the white ball would follow a path that avoids the remaining balls, so these subsequent collisions would not occur. Annotators can therefore assess this counterfactual from the observed motion and spatial relationships.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their careful reviews and constructive suggestions. Below, we briefly summarize our core contributions and the additional evidence and clarifications addressing the main concerns. For ease of reading, revisions in the PDF are marked in blue.

Core Contributions of Our Work

  1. Agentic world modeling and probing. ATW constructs executable worlds from observations and feedback, obtains verifiable evidence through queries, simulations, and interventions, and uses this evidence to answer physical reasoning questions.
  2. Unified support for diverse physical systems. Our Warp-based PolyWorld Engine supports rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions.
  3. Effectiveness across scenarios. ATW achieves overall per-question accuracies of 80.82%, 70.56%, and 71.67% on CLEVRER, ContPhy, and the real-world evaluation, respectively, outperforming PhysMind by 8.27 percentage points on CLEVRER.

Additional Experimental Evidence

  • Component contributions. Additional ablations separately examine world-model revision and feedback-driven multi-step probing, both of which improve overall accuracy. Together with optimizer, simulation-backend, and tracking ablations, these results further clarify the sources of improvement over PhysMind.
  • Input frame count. Additional 8/16/32-frame comparisons show that increasing the number of frames does not yield consistent accuracy gains or eliminate ATW’s overall advantage on the evaluated benchmarks.
  • Evidence effectiveness. Keeping the evidence fixed and replacing only the answer-generation model yields accuracy close to the original configuration. Human participants can also answer effectively using the same evidence, supporting the usability of the execution evidence.
  • Coupled interactions and parameter ambiguity. We provide additional statistics by scenario type on ContPhy and visual reasoning examples, and examine the effect of parameter ambiguity on downstream answers by comparing worlds fitted with different parameter sets.

Analyses and Clarifications

  • We add stage-wise error analysis in §4.4 / Fig. 9 and representative reasoning processes in Appendix E / Fig. 11.
  • We supplement Appendix A.2 with physics solver details and CEM settings, and report runtime and computational costs in our responses.
  • We clarify the scope and annotation procedures of the real-world evaluation, the conditions for completing world modeling, and the handling and scoring of video-fallback and unresolved cases.

We hope these responses address the reviewers’ main questions. The relevant experimental results, analyses, and clarifications will be incorporated into the revised manuscript based on the outcomes of the discussion. We welcome further feedback and thank all reviewers again for their time and valuable suggestions.

Dear Reviewers,

Thank you again for your time and thoughtful feedback. We have provided point-by-point responses, additional experimental results, and a revised manuscript to address your concerns.

As the discussion period draws to a close, we would welcome the opportunity to discuss our responses and findings with you. If you have any remaining concerns or would like further clarification, please let us know. We would be happy to respond promptly and continue the discussion.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.