- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Asking the World (ATW) to address weak physical reasoning in VLMs and the limited flexibility of fixed tool pipelines. ATW first builds and calibrates an executable physical world from a video and its questions, then selects queries, simulations, and interventions to gather evidence and answer the questions. The authors develop the PolyWorld engine to support rigid bodies, soft bodies, cloth, ropes, and fluids, and use CEM to fit physical parameters to video observations. Experiments show clear gains on several physical reasoning tasks without fine-tuning on the benchmarks, while ablation studies show the benefits of physical simulation and adaptive feedback.
Critical strengths
- The paper proposes a well-designed framework that combines world modeling with active experiments and supports its answers with simulation evidence that can be checked.
- The framework supports several material types and question types and works with two VLM backbones.
- The experiments examine the roles of perception, physical simulation, parameter fitting, feedback-based revision, and object identity matching.
Critical weaknesses
- Unequal comparison settings: Some VLM baselines receive only eight frames, while ATW uses extra perception tools and multiple simulation rounds. Input information and compute budgets are therefore not fully matched.
- Limited real-world evaluation: The evaluation includes only 60 videos. The counterfactual answers rely on human judgment and are not verified through real interventions.
- Unclear contributions of individual adaptive steps: World-model revision and additional probing are introduced together. Their separate benefits, and the gains from extra computation, remain unclear.
- The paper does not fully report the physics solver details, CEM settings, runtime, or compute costs.
- The method mainly relies on a single fitted world and does not explicitly account for uncertainty in its physical parameters. A good fit to the observed video does not always lead to correct predictions after an intervention.
Questions for authors
-
How does the system decide that the current world is accurate enough to start probing? Does it use a fitting error threshold, the VLM’s judgment, or a maximum number of iterations? (Section 3.1)
-
The appendix mentions video-fallback and unresolved, but does not fully explain how they work. Does fallback let the VLM answer directly from the video? How often is it used, and how are unresolved cases counted in the final accuracy? (Appendix D)
-
Which physical parameters are estimated by the vision modules, set by the agent, optimized with CEM, or kept fixed? Are the parameter ranges and fitting losses set separately for each dataset? (Section 3.3 and Appendix A.2)
-
Goal-driven accuracy on ContPhy drops from about 73.2% with Single-pass to about 61.0% with Full ATW. Is this mainly due to errors in the world model, the probing strategy, or random variation across runs? (Figure 6)
7 comments
Response to Reviewer cnw4 (Part 2/5)
W3: Unclear contributions of individual adaptive steps: World-model revision and additional probing are introduced together. Their separate benefits, and the gains from extra computation, remain unclear.
A: Thanks for raising this question. To distinguish the contributions of world-model revision and additional probing, we added ATW w/o Adaptive Probing as an intermediate variant between the existing Single-pass World Model and Full ATW configurations in the paper.
The new variant reuses Full ATW’s world-modeling assets while keeping the simulation backend and tools unchanged. It generates the complete probing plan in a single planning step before execution and then answers based on the execution evidence; intermediate feedback cannot trigger additional probes or changes to the plan.
Single-pass World Model and the new variant use the same probing and answer-generation procedures and differ only in world modeling, so their comparison assesses the benefit of world-model revision. The new variant and Full ATW share the same world assets, allowing their comparison to assess the benefit of autonomous probing that adapts to feedback.
The table below combines the new results with the existing results from Fig. 6, reporting overall per-question accuracy (%) on the same ablation subsets.
| Configuration | World-model revision | Feedback-driven multi-step probing | CLEVRER | ContPhy |
|---|---|---|---|---|
| Single-pass World Model | Disabled | Disabled | 75.40 | 59.08 |
| ATW w/o Adaptive Probing | Retained | Disabled | 77.70 | 64.62 |
| Full ATW | Retained | Retained | 82.30 | 68.00 |
Retaining world-model revision improves overall accuracy on CLEVRER and ContPhy by 2.30 and 5.54 percentage points, respectively. Further enabling feedback-driven multi-step probing with the same world assets yields additional gains of 4.60 and 3.38 percentage points, respectively. These results indicate that both adaptive steps contribute to overall performance: world-model revision provides a more reliable physical basis, while feedback-driven probing uses intermediate evidence to adjust subsequent queries and experiments.
W4: The paper does not fully report the physics solver details, CEM settings, runtime, or compute costs.
A: Thank you for pointing out the incomplete reporting of these implementation details and computational costs.
1. Physics solvers and CEM settings.
We have added Physical solvers and CEM configurations to Appendix A.2 of the revised manuscript. Specifically, rigid-body dynamics use event-driven time-of-impact detection and contact impulses; cloth uses position-based structural and bending constraints; rope systems use constrained dynamics for ropes and connected objects; fluids use a position-based fluid solver; and soft-body dynamics couple overall motion with deformation and contact response.
We also provide the CEM population size, elite fraction, number of generations at each fitting stage, and distribution update coefficient for each physical representation. For example, the rigid-body configuration uses a population size of 1024, an elite fraction of 0.08, 64 generations per stage, and a weight of 0.2 on the previous distribution. The appendix also describes the parameter bounds, retention of the incumbent candidate, and representation-specific initialization and local refinement.
Response to Reviewer cnw4 (Part 3/5)
W4 (continued)
2. Runtime and parallel execution.
On a single NVIDIA A100 GPU with 80GB of memory, the average runtime per scene for each benchmark is reported below. This includes world modeling, probing and answering, as well as time spent waiting for VLM API responses:
| Benchmark | Average runtime per scene |
|---|---|
| CLEVRER | 9.34 minutes |
| ContPhy | 11.11 minutes |
| Real-world | 6.11 minutes |
This wall-clock time does not represent continuous GPU computation: while waiting for VLM API responses, the local GPU can process other scenes, and computation for a single scene may not fully utilize the GPU. Execution can therefore be organized by stage, processing multiple scenes in parallel on the same GPU to improve resource utilization.
With stage-wise parallel execution across scenes, we measured the actual elapsed time for sampled batches and divided each batch’s total runtime by its number of scenes to calculate the average processing time per scene. The resulting times for CLEVRER, ContPhy, and the real-world evaluation are 6.23, 7.11, and 4.89 minutes, respectively.
Furthermore, world modeling accounts for most of the runtime, and the constructed worlds are reusable. For the same scene, additional questions within the scope supported by the existing world do not require rebuilding it; they incur only the corresponding probing and answering costs. The world-modeling cost can therefore be amortized over more questions. The physical reasoning tasks evaluated in this paper generally do not require real-time responses.
W5: The method mainly relies on a single fitted world and does not explicitly account for uncertainty in its physical parameters. A good fit to the observed video does not always lead to correct predictions after an intervention.
A: Thanks for raising this issue. A single-view video is generally insufficient to uniquely determine all physical parameters, as different parameter combinations can produce similar visible motion. ATW aims to construct an executable world from observations that supports physical reasoning, rather than to guarantee recovery of a unique ground-truth parameter set. To examine the practical effect of parameter uncertainty on downstream answers, we conducted an additional experiment on 10 selected CLEVRER scenes.
1. Selection of different fitted worlds.
We fixed the perception and reconstruction assets from the existing experiments and performed parameter fitting with multiple random seeds under the original optimization configuration. From the candidate states retained during optimization and the final fitting results, we selected two complete parameter sets, denoted A and B. Both sets met the observational fit-quality requirements while differing substantially in their parameter values.
This selection ensures that we compare physical worlds with substantially different parameters that both explain the observations well. The scenes and parameters were fixed before question answering and scoring, and were not selected based on answer accuracy.
2. Downstream answering results.
After fixing the scenes and parameters, we loaded worlds A and B separately and reran probing and answering, comparing their results with the saved answers from the original experiment on the same selected subset. The table below reports overall per-question accuracy.
| Configuration | Overall accuracy |
|---|---|
| Worlds selected in the original experiment | 90.70% |
| World A | 88.37% |
| World B | 86.05% |
Despite substantial differences in physical parameters, both alternative worlds maintain high accuracy. This indicates that, in these targeted cases, the inability to uniquely determine the parameters does not necessarily cause physical reasoning to fail; fitted worlds with different parameters that are consistent with the observations can still support effective reasoning.
Response to Reviewer cnw4 (Part 1/5)
Thanks for your careful review and constructive feedback. We are pleased that you appreciate ATW’s combination of world modeling and active probing, its use of simulation evidence that can be checked, and its support for different materials and VLM backbones.
Thank you also for your suggestions on the experimental comparisons, implementation details, and limitations. Below, we address each of your comments and questions.
W1: Unequal comparison settings: Some VLM baselines receive only eight frames, while ATW uses extra perception tools and multiple simulation rounds. Input information and compute budgets are therefore not fully matched.
A: Thanks for pointing out the differences in comparison settings. We added 16/32-frame ablations for GPT-5.5 and Qwen3-VL on the CLEVRER and ContPhy ablation subsets and conducted the same comparison for GPT-5.5 on the real-world evaluation set. The table below reports overall per-question accuracy (%).
We find that increasing the number of input frames improves some baseline results but does not yield consistent accuracy gains. Across the tested 8/16/32-frame settings, providing the baselines with richer temporal input does not eliminate ATW’s overall advantage or change the paper’s conclusions.
W2: Limited real-world evaluation: The evaluation includes only 60 videos. The counterfactual answers rely on human judgment and are not verified through real interventions.
A: Thank you for highlighting this issue.
We have not yet found a suitable real-world physical reasoning benchmark with high-quality observations and rich physical interactions. Constructing such a benchmark is challenging: it requires recording complete dynamic interactions and establishing reliable reference answers for predictive and counterfactual questions. Our real-world evaluation set was therefore designed to evaluate the effectiveness of ATW in real-world settings, rather than to serve as a comprehensive benchmark. Its scale remains limited by personnel and hardware constraints, and we will expand the diversity and scale of the evaluation in future work to further assess ATW’s generality.
Regarding counterfactual validation, hardware limitations make it difficult to reproduce identical initial conditions in physical experiments apart from the intended intervention. We therefore did not validate these counterfactual outcomes through actual physical interventions. Nevertheless, we took the following steps to make the reference answers as reliable as possible:
Human annotators inspected synchronized multi-view recordings to resolve occlusions and spatial ambiguities in individual views.
Human annotators viewed complete recordings extending beyond the model’s input—for example, until all balls had stopped—to establish the trajectories and collision relationships used to assess the specified counterfactual.
We retained only questions whose answers could be determined with high confidence and excluded cases with unresolved ambiguity.
Three annotators participated. Each question was assessed separately by at least two annotators, with a third designated to adjudicate disagreements. In practice, the two annotators agreed on every question, and no disagreement required adjudication by the third annotator.