acceptodds
Under review as a conference paper at ICLR 2027

PhysiCalWorld: In-Context Physics Calibration for World Models

Abstract

Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that govern transition dynamics, such as an object’s mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene’s actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, robotics, planning, in-context learning, physics calibration, video models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ysz4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhysiCalWorld to help world models generalize to new environments by trying a few actions and seeing what happens. For example, a robot can learn that an object slides more easily on one surface than another. Instead of retraining the whole model, it uses a context encoder to turn observations and actions into a compact physics descriptor to make better predictions. This allows the model to make better predictions and plan actions without retraining or needing explicit physics labels.

Critical strengths
  1. This paper tackles physics ambiguity that can cause substantial errors in world-model prediction and planning, it's one of the hardest problem, and the paper proposed to use simple yet convincing technic by leveraging a compact descriptor without analytical physics models, explicit parameter labels, or test-time optimization.

  2. Several result show strong improvement, e.g., planning success improves from 53.3% to 62.7% in Push-T and from 26.7% to 56.0% in FetchSlide over DINO-WM. Exhaustive experimental results are shown in four simulated environments and two world-model backbones, including out-of-distribution experiments.

  3. Also strong computational efficiency where the descriptor reuse provides inference efficiency close to the original backbone while avoiding expensive raw-context conditioning.

Critical weaknesses
  1. Several results are worse than baseline, e.g., context-prefix conditioning achieves 82.7% success in BeerPong vs. 80.0% in the proposed method (2.7% worse).

  2. Some context-prefix baselines receive fewer training epochs, which may lead to unfair comparison.

  3. While the paper shows that the descriptor tokens for nearby center-of-mass configurations exhibit high cosine similarity, while some distant configurations have negative similarity. This is not a direct evidence of the interpretability claim (it only suggested that there is some latent structures).

  4. All experiments seem to be evaluated on simulated data (not real-world data).

Questions for authors
  1. Does the number of training epoch impact the result (my concern is: if yes, some experiments are unfair)? Or perhaps a better ask is: could the author provide the training curve to see if the training is converged?

  2. How is the performance on real-world data?

  3. Would the experimental seed impacts that result (running the same experiments several times change the number)? My concern is: this could impact some results with smaller gains, e.g., in BeerPong.

6 comments
AuthorsOpenReview ↗

Author Response (1/4)

We sincerely thank reviewer Ysz4 for their thoughtful review and constructive feedback. We have run additional experiments and revised our manuscript accordingly. Please see our response to each concern below.

W1. Several results are worse than baseline, e.g., context-prefix conditioning achieves 82.7% success in BeerPong vs. 80.0% in the proposed method (2.7% worse).

W2. Some context-prefix baselines receive fewer training epochs, which may lead to unfair comparison.

Thank you for raising these concerns. We provide our responses along with additional experiments results for Weakness 1 and Weakness 2 together in the four tables below.

Weakness 1. In terms of planning success rate in BeerPong environment, we do notice that our method is slightly behind (2.7% worse) in our original manuscript. However, our method is significantly more compute efficient, in both training and inference. For example, in BeerPong environment, the planning time of our method is only 9 seconds while Context Prefix needs 76 seconds. For training, the per epoch time of our method is around 9 hours while Context Prefix needs more than 18 hours. In addition, with the matched training budget, our method is able to achieve better visual accuracy (e.g., IoU) over the context prefix in all environments, for example, ours (65.1%) vs context-prefix (61.4%) in Push-T. Please see the detailed training and inference speed comparison in the provided tables below.

Weakness 2. (regarding unfair comparison), we would like to note that both ours and the context-prefix baseline use the same amount of training budget, measured by total GPU hours. Due to the slow training speed of the context-prefix baseline, using similar GPU hours results in much less training epochs compared with ours. For example, in Push-T, as shown in the table below, context-prefix baseline is only trained for 5 epochs, but it requires 52 hours of training using 4xL40S GPUs, while our method only needs 50 hours to reach 10 epochs with the same 4xL40S GPU setup.

For a complete study, we provide two comparisons in the table belows, "matched training cost" (measured by similar GPU hours, compare row 1 and row 3 of each table) and "matched training epoch" (compare row 2 and row 3 of each table). Note that for "matched training epoch", Context Prefix uses significantly more GPU hours than Ours. For the planning results, we report the mean and standard deviation over 3 random seeds. Note that for Reacher and BeerPong, the results are slightly different with the ones reported in the initial manuscript, since we re-train the models using the same 4xL40S setup as our method for an apple-to-apple comparison. Total GPU-hours are four times these values because all models below are trained on four L40S GPUs.

Push-T:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)PSNR↑SSIM↑LPIPS↓IoU(%)↑
Context Prefix552 h59.1±3.4323.09 s22.980.9610.029561.4
Context Prefix10110 h59.6±2.8323.09 s23.130.9620.030061.5
Ours1050 h61.8±4.132.67 s23.830.9640.026765.1

Reacher:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)PSNR↑SSIM↑LPIPS↓IoU(%)↑
Context Prefix1018 h61.8±2.0321.30 s20.210.9700.033039.6
Context Prefix2036 h69.3±1.3321.28 s20.870.9720.027344.3
Ours2020 h58.7±2.731.68 s21.580.9750.022251.0

BeerPong:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)PSNR↑SSIM↑LPIPS↓IoU(%)↑
Context Prefix355 h77.8±5.476.47 s28.980.9940.008441.1
Context Prefix592 h81.3±1.376.46 s29.510.9940.007644.9
Ours545 h80.9±1.59.01 s29.490.9950.006746.3
AuthorsOpenReview ↗

Author Response (2/4)

FetchSlide:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)PSNR↑SSIM↑LPIPS↓IoU(%)↑
Context Prefix327 h55.6±0.837.21 s26.780.9900.005470.1
Context Prefix545 h58.2±3.437.21 s27.140.9910.004871.5
Ours526 h54.7±4.03.91 s27.870.9910.004773.8

The tables above show that our method yields predictions of better visual quality (PSNR, IoU, etc.) across all four simulation environments. However, in some environments, it achieves lower planning success rates than the context-prefix baseline. One possible reason is the model's sensitivity to noisy action sequences, as CEM initializes candidate actions by sampling from a Gaussian distribution. Nevertheless, our method enables substantially more efficient planning, requiring approximately one-tenth of the time of the context-prefix baseline, for example, 32 seconds versus 323 seconds per task in Push-T. We have also included these results in Appendix D.3 of our updated manuscript.

W3. While the paper shows that the descriptor tokens for nearby center-of-mass configurations exhibit high cosine similarity, while some distant configurations have negative similarity. This is not a direct evidence of the interpretability claim (it only suggested that there is some latent structures).

Thank you for the comment. We have updated paper and replace the 'interpretability' claim with 'latent structures'. Please see Page 9 of the updated manuscript.

To further evaluate the information encoded in the physics descriptor, we conduct an additional linear probing experiment in Push-T. Specifically, we freeze the descriptor encoder and train a single linear layer to predict the T-block’s normalized center-of-mass (CoM) coordinates from zphy\mathbf{z}_{\text{phy}}. As a reference, we train a separate Physics Estimator with approximately 20 million parameters from scratch to predict the same target.

Probe inputTypeModel ParamTrainable ParamCoM error
Physics EstimatorFull Model Train from Scratch19.7M19.7M0.092
Ours zphy\mathbf{z}_{\text{phy}}Linear Probe19.7M0.00013M0.129

The linear probe achieves a CoM mean square error of 0.129 while training only 130 parameters (the context encoder that generate the descriptors is frozen), compared with 0.092 for the fully trained Physics Estimator as reference. Although the dedicated estimator achieves slightly lower error, this result shows that CoM information is accessible through a simple linear probing of the learned descriptors. This also indicates that the physics descriptors form some effective physics-related latent structure. We have also included these results in Appendix D.5 of our updated manuscript.

W4. All experiments seem to be evaluated on simulated data (not real-world data).

Thank you for raising this concern. We are currently setting up and conducting a real robot version of our FetchSlide environment, where we adjust the surface friction and the mass of the puck. However, due to time constraints, we probably cannot present the real robot results within the rebuttal period. But we promise to include real robot experiment results in our camera-ready version.

AuthorsOpenReview ↗

Author Response (3/4)

Q1. Does the number of training epoch impact the result (my concern is: if yes, some experiments are unfair)? Or perhaps a better ask is: could the author provide the training curve to see if the training is converged?

Thank you for the question. For the experiments shown in our original manuscript, we explicitly align the number of total training GPU hours between context-prefix baseline and ours. Because the per epoch train time of context-prefix baseline is much longer, it hence receive less training epochs.

In the response to W1 and W2 (please see the four tables above), we provide side-by-side comparison of 'matched epoch' and 'matched training hours' between context-prefix baseline and our method, where we observe that our method consistently outperforms context-prefix baseline in visual metrics, highlighting the prediction accuracy of our method. We also notice that in some environments, our method falls behind context-prefix baseline on planning success rate. This may be due to the model's sensitivity to noisy action inputs (the initial action of CEM planning are randomly sampled Gaussian distribution). However, planning with our method is much more efficient than the context-prefix baseline, which typically requires 10 times more time than ours, e.g., in Push-T ours(32 seconds) vs context-prefix(323 seconds).

We also provide the training curve in the paper, shown in Figure A2 on Page 29. From the training curve, we observe that, for example, the models in Push-T may not completely converged. Hence, we further study the model performance with more training steps. Due to time and compute limit, we train these two models until 16 epochs, and compare their performance under various training epoch: 6,8,10,12,14,16. (Each training epoch of the +Context Prefix model requires approximately 11 hours on 4xL40S GPU and each planning run of +Context Prefix needs 24 hours on a single L40S).

In the table below, we present the evaluation performance of the context-prefix baseline and our method over different numbers of training epoch: from epoch 6 to epoch 16, in Push-T environment. Both + Context Prefix and + PhysiCal (Ours) are based on DINO-WM. We have also included these results in Appendix D.4 of our updated manuscript.

MethodEpochPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
Context Prefix622.800.9610.030460.359.1±6.6
PhysiCal (Ours)623.170.9620.029062.263.6±0.8
Context Prefix823.110.9610.030860.861.3±4.0
PhysiCal (Ours)823.230.9620.029562.258.2±5.4
Context Prefix1023.130.9620.030061.559.6±2.8
PhysiCal (Ours)1023.830.9640.026765.161.8±4.1
Context Prefix1223.090.9610.031660.361.3±2.7
PhysiCal (Ours)1223.150.9610.032259.865.8±4.7
Context Prefix1422.880.9600.033758.562.2±4.7
PhysiCal (Ours)1423.690.9630.029563.262.7±2.7
Context Prefix1623.330.9620.031760.664.9±4.1
PhysiCal (Ours)1623.600.9630.029662.368.9±4.7

From the table, we can see that the evaluation performance fluctuate after epoch 10, even though the training loss can still decrease. In addition, we can see that our method generally outperform the context-prefix baseline in both visual metrics and planning success rate.

Q2. How is the performance on real-world data?

Thank you for the question. We are currently setting up and conducting a real robot version of our FetchSlide environment, where we adjust the surface friction and the mass of the puck. However, due to time constraints, we probably cannot present the real robot results within the rebuttal period. But we promise to include real robot experiment results in our camera-ready version.

Reviewer_zKCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysiCalWorld calibrates an action-conditioned world model to the current scene's physics from a short interaction history: a small context encoder compresses a 12-frame clip into one reusable physics descriptor read by the predictor, computed once per scene. Training uses no physics labels but needs trajectories grouped by shared physical configuration. Across four environments and two backbones, calibration improves prediction and planning, on Push-T nears the true-parameter bound, and planning runs ~10× faster than full-context conditioning.

Critical strengths
  1. The reuse is the point. One 2.3 ms encoding, per-step latency near the backbone, planning about 10× faster than the full-context baseline at equal or better quality. For a planner that calls the world model thousands of times per task, that is the kind of speedup that actually changes what is feasible, and none of it needs test-time fine-tuning.
  2. The gains show up across four kinds of physical variation (three coupled factors in FetchSlide) and two backbones, in both open-loop prediction and planning, and the descriptor space tracks physical proximity rather than just clustering prettily.
Critical weaknesses
  1. The orderings are not statistically supported. 75 planning tasks per environment, single runs, no seeds, no error bars. Gaps of 2.7 to 4.0 points (Push-T 62.7 vs 58.7; BeerPong 80.0 vs 82.7, with the method losing), and at most 5.4 points, sit well inside binomial noise (roughly ±11 points at 95% CI). Meanwhile the full-context baseline, the very one the efficiency argument targets, trained only 3-5 epochs in some environments where others trained 5-20.
  2. The grouping assumption is demanding: 135 same-configuration groups per environment, and knowing which interactions share physics is itself a form of supervision. The "label-free" framing should say this up front.
  3. Everything is clean-rendered simulation. No real robot, no sim-to-real, descriptor capacity tested up to three coupled factors, and since the paper itself notes that physical cues are temporally sparse, a weak prompt could cap the whole method.
Questions for authors
  1. How sensitive is the descriptor to the prompt collection policy, the length TcT_c, and state-action coverage? Can a bad prompt make calibration worse than none?
  2. Full-context baseline accuracy at matched epochs, please, and separately at matched compute, since per-epoch costs differ.
  3. What concretely separates this from SyncWorld in assumptions, training and applicability? A direct comparison would help.
6 comments
AuthorsOpenReview ↗

Author Response (1/5)

We sincerely thank reviewer zKCf for their thoughtful review and constructive feedback. We have run additional experiments and revised our manuscript accordingly. Please see our response to each concern below.

W1. The orderings are not statistically supported. 75 planning tasks per environment, single runs, no seeds, no error bars. Gaps of 2.7 to 4.0 points (Push-T 62.7 vs 58.7; BeerPong 80.0 vs 82.7, with the method losing), and at most 5.4 points, sit well inside binomial noise (roughly ±11 points at 95% CI). Meanwhile the full-context baseline, the very one the efficiency argument targets, trained only 3-5 epochs in some environments where others trained 5-20.

Thank you for highlighting these concerns. We have run the planning evaluations with three random seeds across all four simulation environments and report the mean and standard deviation of the success rates below.

Regarding the difference in the number of train epochs between the full-context (context-prefix) baseline and our method, we would like to note that we actually align the total training GPU time (GPU hours) between context-prefix and ours. But since the training speed (i.e., iterations per second) of context-prefix is much slower (due to the excessive additional tokens), 3-epoch training time of the context-prefix baseline is already longer than the 5-epoch training time of our method, as presented in Table A5 and A6 of our original submission. As requested, we provide a side-by-side comparison between context-prefix and ours at 'matched train time' (see row 1 and row 3 of each table) and 'matched epoch' (see row 2 and row 3 of each table) with 3 random seed in the table below. All models are trained using 4 × L40S GPUs.

Push-T:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)
Context Prefix552 h59.1±3.4323.09 s
Context Prefix10110 h59.6±2.8323.09 s
Ours1050 h61.8±4.132.67 s

Reacher:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)
Context Prefix1018 h61.8±2.0321.30 s
Context Prefix2036 h69.3±1.3321.28 s
Ours2020 h58.7±2.731.68 s

BeerPong:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)
Context Prefix355 h77.8±5.476.47 s
Context Prefix592 h81.3±1.376.46 s
Ours545 h80.9±1.59.01 s

FetchSlide:

MethodTrain # of EpochsModel Train TimeSuccess RatePlanning Time (per task)
Context Prefix327 h55.6±0.837.21 s
Context Prefix545 h58.2 ± 3.437.21 s
Ours526 h54.7 ± 4.03.91 s

At matched training epochs, the two methods achieve numerically close mean success rates in Push-T and BeerPong, while Context Prefix achieves higher mean success rates in Reacher and FetchSlide. While these results do not establish a consistent success-rate advantage for our method over context-prefix, we would like to note that our method requires much less training time and achieves approximately 8–10x faster planning across all four environments.

AuthorsOpenReview ↗

Author Response (2/5)

W2. The grouping assumption is demanding: 135 same-configuration groups per environment, and knowing which interactions share physics is itself a form of supervision. The "label-free" framing should say this up front.

Thank you for the suggestion. We have updated our manuscript and mention that our method does not require exact physics parameters/values, but it needs physics grouping as supervision. Specifically, we detailed this in our abstract and contribution statement. In addition, we revised our original statement of 'without physical labels' to 'without physics parameters' throughout the updated manuscript. Please see the blue text in the updated manuscript.

We would also like to clarify that this data grouping is used for constructing data pairs used for training: the pair of an exploratory prompt C\mathcal{C} and the prediction target ot:t+Ho_{t:t+H}. Grouping is indeed required when each video episode is short, because in that case we cannot construct diverse pairs per scene that facilitate learning a meaningful physics descriptor. However, if the collected episode is long enough (contains diverse interactions), the training data pairs (i.e., the prompt and the target sequence) can be directly sampled from the same episode. In this case, our method can potentially be trained without groupings.

For a complete study, we run additional experiments on how the number of physics configuration groups will affect the performance of our method in the Push-T environment, shown in the table below.

Specifically, in each experiment, per physics group contains identical number of episodes (we use 101 episodes as the experiment in Table 1 of the manuscript). Hence, a dataset that contains more groups will have a larger number of total episodes. We align the number of train epoch for each experiment: all models in the table below are trained from scratch for 20 epoch.

# of groupsPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
2522.420.9590.031859.150.2±2.8
5023.670.9640.025965.558.7±2.3
7523.840.9640.028164.061.8±2.0
10024.100.9650.027265.464.4±0.8
135 (all)23.840.9640.029163.469.3±3.5

Note that the results of 135(all) is slightly different with the one in Table 1, because in our initial manuscript, we train the model for 10 epoch (while here 20 epoch). For the planning success rates, we report the mean and standard deviation of 3 seeds. We observe that adding more number of groups can generally enhance performance, but the gain tends to marginalize after 100 groups. We have also included this experiment in Appendix D.6.

W3. Everything is clean-rendered simulation. No real robot, no sim-to-real, descriptor capacity tested up to three coupled factors, and since the paper itself notes that physical cues are temporally sparse, a weak prompt could cap the whole method.

Thank you for the comment. Our work focuses on calibrating a world model such that its predictions can align with the actual test-scene dynamics. We agree that calibration depends on the information available in the exploratory prompt. If the given prompt is "weak" (e.g., contains insufficient interaction to reveal the relevant physical properties), the model cannot reliably infer those properties and the calibration may consequently fail.

More informative (stronger) prompts could potentially be collected through exploration strategies that maximize information gain or policies trained with rewards that encourage informative interactions. Developing such prompt collection strategies is beyond the scope of this paper.

To examine the dependence on the collection strategy, we conduct additional experiments using prompts collected by an RL policy, a diffusion policy trained on human demonstrations, a predefined heuristic, and random pushes. The results show that our method remains effective across these strategies, including simple heuristics and random pushes. Please see our response to Q1 below for the experimental details and results. We have also included the results in Appendix D.8 of our updated manuscript.

AuthorsOpenReview ↗

Author Response (3/5)

Q1. How sensitive is the descriptor to the prompt collection policy, the length, and state-action coverage? Can a bad prompt make calibration worse than none?

Thank you for the question. We run additional experiments in Push-T environment, including: (a) the prompts are generated by different methods; (b) various prompt lengths (from 3 seconds to 8 seconds); (c) different amount of object contact within the prompt; to study how these factors can affect the calibration performance.

(a) Prompt collection strategy. we conduct ablation studies on different prompt collection methods in our Push-T environment. Specifically, for the Push-T results shown in Table 1 of our original manuscript, the exploratry prompt is collected via a RL policy. During the rebuttal period, we train a model with explortary prompts collected by four different methods, including (1) heuristc method: the agent pushes the T-block from some certain hard-coded directions; (2) diffusion policy: we train a diffusion policy using human collected trajectories. This behavior cloning based collection method is to mimic how human may interact with the T-block; (3) random push: the agent pushes the T-block from randomly sampled directions, and the resulting interaction sequence serves as the exploratory prompt. (4) RL policy: this is the original prompt collection method used in Table 1 of the manuscript.

For training efficiency, we finetune the model with all four types of prompts starting from our existing 5-epoch checkpoint trained only on RL-policy generated prompts. Note that we train one model using all four prompt collection methods. We report the evaluation performance using each type of prompt below. All prompt methods are evaluated on the same set of validation data. We have also included these results in Appendix D.8 of our updated manuscript.

Prompt Collection MethodPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
No prompt (DINO-WM)21.870.9550.040251.551.1±3.1
RL Policy (Current)23.750.9640.028963.961.3±6.7
Heuristic23.750.9640.027764.667.6±3.1
Diffusion Policy23.220.9620.030961.362.2±3.4
Random Push23.040.9610.032559.663.1±4.3

The experiments above shows our calibration method is effective across different prompt collection strategies where all prompt collection methods outperform the no-prompt DINO-WM baseline by a significant margin.

(b) various prompt length. We train a model that can consume prompt length from 3 seconds to 8 seconds and evaluate the model under different prompt lengths, as shown in the table below. (Note that the control frequency in Push-T is 10Hz and we sample one frame every 5 timestep, hence 3-8 seconds correspond to 6–16 input context frames.)

Prompt Length (sec)PSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
No prompt (DINO-WM)21.870.9550.040251.551.1±3.1
3s23.010.9600.031559.859.6±6.2
4s22.970.9600.031759.460.4±4.7
5s23.370.9620.028862.563.1±0.8
6s23.320.9620.029362.360.0±3.5
7s23.240.9610.030061.560.4±4.1
8s23.180.9610.031160.660.9±2.0

From the results above, we can see that our calibration method can significantly enhance the world model's prediction accuracy and planning success rate regardless of the prompt length. We also observe that extending the prompt length beyond 5s cannot further enhance the performance, probably because that in Push-T, 5 seconds of interaction is enough to reveal the needed physical dynamics. We have also included these results in Appendix D.9 of our updated manuscript.

Reviewer_kGGKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PhysiCalWorld introduces an in-context physics calibration framework for action-conditioned visual world models, resolving dynamics drift caused by unobservable physical properties (e.g., mass, friction, damping). It compresses a short exploratory interaction clip from the test scene into a compact 64-dimensional descriptor token to condition the world model. Trained end-to-end via grouped self-supervised latent prediction—without physics labels, analytical equations, or test-time fine-tuning—the token is extracted once and cached, enabling physically consistent rollouts and planning with negligible compute overhead.

Three Contributions:

  1. In-Context Calibration Paradigm: Enables zero-shot dynamics adaptation without manual analytical modeling (system identification) or test-time fine-tuning.
  2. Compact Physics Descriptor Token: Compresses temporal dynamics into a single token, eliminating the quadratic attention overhead of raw-frame context prefixing.
  3. Label-Free Self-Supervised Training: Exploits environmental grouping to train the model end-to-end using purely visual prediction loss, requiring no ground-truth physics supervision.
Critical strengths
  1. Propose a novel self-supervised paradigm for zero-shot physics calibration: By exploiting the data-grouping property that trajectories within the same physical environment share identical transition dynamics, the model learns to extract dynamic context end-to-end using solely future latent prediction loss. This design removes the reliance on explicit physics labels and physics simulators, providing a highly scalable pathway for embodied agents to adapt their dynamics in unknown, complex physical environments.

  2. A compact single-token descriptor yielding superior downstream planning efficiency: By compressing the entire physical interaction into a single 64-dimensional descriptor token, the method achieves "one-pass extraction and global reuse." This design speeds up downstream visual planning by nearly 10× with improved accuracy, making context-driven world models genuinely viable for real-time robotic control.

  3. Experimental design is systematic and comprehensive: the evaluated scenarios cover various dynamic tasks involving varying centers of mass and damping coefficients; it conducts thorough comparisons against context-prefixing, explicit physics estimation, and privileged oracle baselines; it balances both visual generation quality and downstream planning success rates; it achieves consistent performance gains across two distinct backbones (DINO-WM and LeWM); and it convincingly demonstrates the representational continuity and out-of-distribution (OOD) robustness through latent space geometric analysis and extrapolation experiments.

Critical weaknesses

The training set's strong reliance on the prior assumption of "environment grouping (data grouping)" limits its scalability to large-scale real-world video pretraining. While grouping by parameter configurations is trivial in simulators, object mass and surface friction often vary across video clips in unstructured, in-the-wild real-world datasets, making it difficult to automatically obtain such "physics-invariant" groupings. This assumption hinders the method's direct applicability to truly large-scale, generalist video world model pretraining.

Questions for authors
  1. A deeper investigation into the collection strategy of the test-time exploratory prompt is recommended The model heavily relies on a brief interaction video clip provided prior to testing. How were these exploratory trajectories generated across all evaluation experiments (e.g., via random actions, expert policies, or pre-scripted exploratory behaviors)? Furthermore, how does the model perform if the exploration fails to make effective contact or collision with objects (e.g., the robotic arm waving in free space without contact)?

  2. Ruling out "static visual shortcuts" in self-supervised grouped training In simulation environments, do trajectories within the same physical group share subtle static rendering artifacts, textures, or initial pose correlations? In other words, is it possible that the model identifies the environment ID via static visual shortcuts rather than genuinely extracting dynamics from temporal state transitions? How can this form of overfitting be rigorously ruled out?

3 comments
AuthorsOpenReview ↗

Author Response (1/3)

We sincerely thank reviewer kGGK for their thoughtful review and positive evaluation. We have conducted additional experiments and revised our manuscript accordingly. Please see our response to each concern below.

W1. The training set's strong reliance on the prior assumption of "environment grouping (data grouping)" limits its scalability to large-scale real-world video pretraining. While grouping by parameter configurations is trivial in simulators, object mass and surface friction often vary across video clips in unstructured, in-the-wild real-world datasets, making it difficult to automatically obtain such "physics-invariant" groupings. This assumption hinders the method's direct applicability to truly large-scale, generalist video world model pretraining.

Thank you for the comment. We agree that the internet-scale video datasets usually do not provide such "physics-invariant" grouping and we also mention this limitation in Section 5 (the limitations paragraph) in our initial submission.

Many existing work has shown that video generative models finetuned on sim data can generalize to real world scenarios, such as Force Prompting [1] and PhyCo[2]. Hence, one solution is to construct a large-scale diverse physics-grouped video pretraining datasets in simulations or Blender and (pre)train a video model on these sim datasets. Such models can potentially directly generalize to some real-world scenes.

To further enhance performance on specific downstream tasks, we can use a small set of in-domain real-world data to finetune the world models pretrained on sim physics data. We deem that large-scale sim video data pretraining can enhance physics understanding of the resulting world models and have better data efficiency, as the learned physics prior can be shared between sim and real.

In addition, we would like to clarify that this data grouping is used for constructing the data pairs used for training: the pair consists of an exploratory prompt C\mathcal{C} and the prediction target ot:t+Ho_{t:t+H}. Grouping is indeed required when each video episode is short, because in that case we cannot construct diverse pairs per scene that facilitate learning a meaningful physics descriptor. However, if the collected episode / video is long enough (contains diverse interaction), the training data pair (i.e., the prompt and the target sequence) can be directly sampled from the same episode. In this case, our method can be trained without groupings.

For a complete study, we also conduct additional experiments in Push-T on how will the model's performance vary when using different number of groups for training. We align the number of train epoch for each experiment where all models in the table below are trained from scratch for 20 epoch. As shown in the table below, we observe that the model performance generally scale with more training groups.

# of groupsPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
2522.420.9590.031859.150.2±2.8
5023.670.9640.025965.558.7±2.3
7523.840.9640.028164.061.8±2.0
10024.100.9650.027265.464.4±0.8
135 (all)23.840.9640.029163.469.3±3.5

[1] Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals
[2] PhyCo: Learning Controllable Physical Priors for Generative Motion

AuthorsOpenReview ↗

Author Response (2/3)

Q1. A deeper investigation into the collection strategy of the test-time exploratory prompt is recommended The model heavily relies on a brief interaction video clip provided prior to testing. How were these exploratory trajectories generated across all evaluation experiments (e.g., via random actions, expert policies, or pre-scripted exploratory behaviors)? Furthermore, how does the model perform if the exploration fails to make effective contact or collision with objects (e.g., the robotic arm waving in free space without contact)?

Thank you for the suggestion. We present how the exploratory trajectories are generated in each environment in the table below.

EnvExploratory Trajectories Generated Method
Push-TA pretrained PPO policy (center-of-mass agnostic)
ReacherRandom joint torques of up to ±0.05 N·m at every step
BeerPongRandom throw release velocity of 2.5–3.0 m/s forward and 6–8 m/s upward
FetchSlideRandom strike of 1.75–4.25 cm forward and up to 4 cm sideways per step for 2 steps

We then study how will the model perform given exploration prompts of different amount of contact between the target objects (e.g., including scenarios with little effective contacts). In the table below, we show additional experiment results of exploratory prompts of different contact ratio (the contact between the T-block and the agent). For example, 'contact < 5' means that less than 5% of the whole time interval of the prompt contains contact with the T-block, which constructs the scenario that the prompts contain almost no effective contact with objects in the environments.

Contact RatioPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
DINO-WM21.870.9550.040251.551.1±3.1
contact < 5%21.260.9530.043847.751.1±3.4
contact < 10%21.820.9550.039551.651.6±6.3
contact < 30%23.220.9620.030161.462.2±2.0
contact < 50%22.870.9600.032958.263.6±0.8
contact < 70%23.420.9630.029162.468.4±0.8
contact ≤ 100% (reported in paper)23.830.9640.026765.161.8±4.1

As shown in the table, when the contact ratio is below 10%, our calibration method performs similarly to the uncalibrated DINO-WM baseline. This is consistent with our hypothesis that calibration becomes less effective when the exploratory prompt contains little contact, because such prompts provide limited information for inferring the test scene’s physical properties. We also observe that the full-range condition ("contact ≤ 100%") achieves the best prediction metrics, including IoU, but a lower mean success rate than the "contact < 70%" condition. This suggests that improvements in average prediction quality do not necessarily translate into higher downstream planning success. Differences in the actual sampled prompts and the randomness of the CEM planning process may contribute to this discrepancy.

We also add ablation studies on different prompt collection methods in our Push-T environment. Specifically, for the Push-T results shown in Table 1 of our original manuscript, the exploratory prompt is collected via a RL policy. During the rebuttal period, we train a model with exploratory prompts collected by four different methods, including (1) heuristic method: the agent will always push the T-block from some certain hard-coded direction.
(2) diffusion policy: we train a diffusion policy using human collected trajectories. This behavior cloning based collection method is to mimic how human might interact with the T-block. (3) random push: push the T-block from some randomly sampled directions and use the resulting interaction sequence as prompt. (4) RL policy: this is the original prompt collection method used in Table 1 of the manuscript.

For training efficiency, we finetune the model with all four types of prompts starting from our existing 5-epoch checkpoint trained only on RL-policy generated prompts. Note that we train one model using all four prompt collection methods. We report the evaluation performance using each type of prompt below.

Prompt Collection MethodPSNR↑SSIM↑LPIPS↓IoU(%)↑SR(%)↑
No prompt (DINO-WM)21.870.9550.040251.551.1±3.1
RL Policy (Current)23.750.9640.028963.961.3±6.7
Heuristic23.750.9640.027764.667.6±3.1
Diffusion Policy23.220.9620.030961.362.2±3.4
Random Push23.040.9610.032559.663.1±4.3

The experiments above shows our calibration method is effective across different prompt collection strategies, where all prompt collection methods outperform the no-prompt DINO-WM baseline by a significant margin. We have also included these results in Appendix D.8 of our updated manuscript.

AuthorsOpenReview ↗

Author Response (3/3)

Q2. Ruling out "static visual shortcuts" in self-supervised grouped training In simulation environments, do trajectories within the same physical group share subtle static rendering artifacts, textures, or initial pose correlations? In other words, is it possible that the model identifies the environment ID via static visual shortcuts rather than genuinely extracting dynamics from temporal state transitions? How can this form of overfitting be rigorously ruled out?

Thank you for the question. We explicitly prevent such "visual shortcuts" when we design our environments. Hence, in principle, it is not possible that the model can identify the environment ID via static visual shortcuts. For example, for the push-t environment, we only modify the center of mass of the T-Block in the simulator and this will only affect how the T-Block respond to the agent's action and will not affect how the T-Block is rendered. We add the purple diamond to the Push-T images in Figure 3 of our manuscript only for visualization purpose. For actual model input, we do not add this purple diamond. In FetchSlide, we alter puck mass, surface friction, and aerodynamic drag. Similarly, we directly change these physics parameters in the simulator, and this will not change how the image rendering result.

Please see Appendix E.6 on Page 48 of our updated manuscript for the rendering comparison. In this figure, we show two different physics settings separated in two columns for each environment. The two images are rendered with identical agent and object states but different physics configurations (the T-block's centers of mass are different). The images are visually indistinguishable, and we have verified that they are numerically identical. Thus, these physical configurations cannot be distinguished through static visual shortcuts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and reviewers,

We sincerely thank the Area Chair and all reviewers (kGGK, zKCf, Ysz4) for their time, effort, and constructive feedback in reviewing our manuscript.

We are glad that the reviewers recognized the importance of the problem we address and our core contributions, such as (1) our in-context physics calibration formulation, (2) our group-based training objective that requires no explicit physics parameters as supervision, (3) the efficiency and the emerged physics-correlated latent structure of our compact single-token physics descriptor (enabling ~8–10× faster downstream planning), and (4) our systematic evaluation that shows substantial gains over uncalibrated baselines across four environments and two world-model backbones.

We have posted detailed individual responses to each reviewer and uploaded a revised manuscript with all updates highlighted in blue. Below, we summarize how we addressed the common questions across reviews and list all added experiments.

1.Multi-Seed Evaluation, Compute Fairness, & Convergence (zKCf, Ysz4)

3-seed & matched-budget comparisons (Appendix D.3): We re-ran CEM planning over 3 random seeds across all four environments and compared against Context Prefix under both matched GPU hours and matched epochs. PhysiCalWorld consistently achieves better or comparable visual prediction quality (PSNR, SSIM, LPIPS, IoU) with ~2× faster training per epoch and ~8–10× faster planning, matching Context Prefix in planning success rate (SR) at matched compute (while relatively lower in Reacher and FetchSlide at matched epochs).

Training convergence (Figure A2; Appendix D.4): We added training loss curves and extended Push-T evaluations up to 16 epochs, where PhysiCalWorld consistently outperforms Context Prefix.

2.Physics Grouping Assumption & Framing (kGGK, zKCf)

Clarified framing: We revised "without physical labels" to "without physics parameters" and explicitly stated our reliance on physics grouping supervision up front. We also noted that sufficiently long episodes can avoid cross-episode grouping and sim-to-real pretraining offers a practical scaling path.

No-grouping experiment in Reacher: We conduct additional experiments in Reacher, where we directly collect a single long episode per physics configuration. Specifically, we collected 1 episode of 2,000 steps across 150 physics configurations (matching the total number of steps in our original grouped dataset) and sampled prompt–target pairs directly from the same episode. Hence, we do not assume any episode groupings in this training dataset. Below are the open-loop evaluation results comparing the ungrouped dataset (first row) against our original grouped dataset for reference (second row):

Dataset CompositionPSNR↑SSIM↑LPIPS↓IoU(%)↑
150 ×\times 1 (episodes) ×\times 2000 (steps)21.690.9270.037856.7
150 ×\times 20 (episodes) ×\times 10021.580.9750.022251.0

Even without cross-episode grouping, our method achieves comparable open-loop prediction quality. This shows that PhysiCalWorld can effectively learn physics calibration directly from single long trajectories without requiring episode groupings.

  1. Sensitivity to Exploratory Prompts (kGGK, zKCf)

We documented prompt collection methods for all environments and ran three ablations in Push-T:

(1) Prompt collection strategies (Appendix D.8): RL policy, heuristic, diffusion policy, and random push all substantially outperform uncalibrated DINO-WM.

(2) Contact ratio / weak prompts (Appendix D.7): Low-contact prompts (<10%) fall back to roughly uncalibrated DINO-WM performance rather than degrading catastrophically, with clear gains once contact increases.

(3) Prompt length TcT_c (Appendix D.9): Varying TcT_c from 3s to 8s yields consistent improvements over DINO-WM; performance peaks at 5s.

  1. Additional Experiments & Revisions

Latent structure & linear probing (Ysz4, Appendix D.5): We reframed "interpretability" to "latent structure" (on Page 9) and added a linear probe on frozen zphyz_{\text{phy}} (130 trainable parameters) as additional experiments to demonstrate the latent structure.

Ruling out visual shortcuts (kGGK, Appendix E.6): We added side-by-side renderings showing identical states under different physics are visually identical and we have also verified the renderings are numerically identical.

  1. Real-World Experiments (zKCf, Ysz4)

We are currently setting up and conducting a real-robot version of our FetchSlide environment, varying surface friction and puck mass. Due to time constraints, we are unable to present these hardware results within the rebuttal period, but we promise to include them in the camera-ready version.

We hope that our responses, revisions, and additional experiments address the reviewers' concerns, and we are happy to answer any further questions.

Best,
Paper Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.