acceptodds
Under review as a conference paper at ICLR 2027

What Can Latent World Models Know? Physical Information in Multimodal Predictive Representations

Abstract

A central premise of latent world models is that predicting the future encourages representations to internalize the physics of their environment. We ask which physical quantities are accessible in learned latent states, how this depends on training, and how those quantities relate to the model’s predictions. We present PokeWorld, a simulated environment in which a robot finger pushes objects whose mass, drag, and contact stiffness vary across episodes while remaining visually identical. We first measure parameter recovery from raw observation sequences, then train matched action-conditioned world models. Prediction targets can strongly shape latent content. For example, contact stiffness becomes decodable when touch is predicted, while providing touch only as an input does not. Longer-horizon prediction improves estimation of position and velocity from learned representations relative to single-step prediction. Drag is recoverable from raw observations, but has weak linear readout from the learned latent states. Yet the models’ glide forecasts depend systematically on drag. Experiments on RH20T reproduce the same input–target dependence on real multimodal robot data. Together, these results show that observations, prediction targets, and prediction horizons shape both which physical quantities are accessible in latent states and how they affect future predictions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
latent world models, physical representation learning, multimodal prediction, physical parameter identifiability, robot learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SpmMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces POKEWORLD, a 2D pushing simulator in which a robot finger pushes objects that remain visually identical across episodes despite variations in mass, drag, and contact stiffness. The paper first assess physical-parameter recovery from observation sequences, then train matched action-conditioned world models with different input streams, prediction targets, and prediction horizons. They evaluate whether physical parameters can be decoded from the frozen models' latent states using linear probes and whether the models' forecasts depend on those parameters.

Critical strengths
  1. The paper introduces POKEWORLD, a 2D pushing environment in which objects remain visually identical while their mass, drag, and contact stiffness vary across episodes.

  2. The paper presents extensive ablation studies examining input modalities, prediction targets, and prediction horizons.

Critical weaknesses
  1. Comparisons with established world-model approaches are missing. The paper uses a relatively simple architecture with a small number of parameters, but does not compare it with established alternatives. This makes it difficult to assess whether the chosen architecture is appropriate for the study.

  2. The simulation experiments rely on a simple, purpose-built environment, without further validation in more general physics simulation settings. The paper also does not demonstrate the advantages of this custom 2D pushing environment over established simulation platforms such as MuJoCo and Isaac Gym. These limitations weaken the persuasiveness of the experimental evidence.

  3. There is no real-world execution using the estimated physical parameters, reducing the credibility.

Questions for authors
  1. What motivated the choice of a custom 2D pushing environment and a compact world-model architecture? How do these choices affect the scope and generalizability of the conclusions?

  2. The simulation experiments evaluate hidden parameters such as mass, drag, and stiffness, whereas the real-data experiments evaluate force readout and prediction. Given that these measure different capabilities, which simulation findings do the RH20T results directly validate?

  3. Can the physical parameters decoded from the learned representations be used to guide robot actions? For example, in a goal-directed pushing task, how does a controller using these estimates compare with the same controller using fixed nominal parameters or ground-truth parameters?

3 comments

We thank the reviewer for the constructive feedback. We have expanded the experiments to address the concerns about model generalizability, simulation settings, and downstream utility.

Q3: can the decoded parameters guide actions? Yes. We ran the experiment you describe (general response, part 2; App. I). In a goal-directed pushing task, the controller that plans with parameters decoded from the frozen world model lies between the nominal and the ground-truth controller. The peak-force error is 19.8% with nominal parameters, 19.1% with estimates from VF, 15.8% with estimates from VFX, 13.1% for VFX with four times more data, and 1.3% with the true parameters. VF and VFX receive the same inputs; VFX additionally predicts proprioception and touch.

Q2: which simulation findings does RH20T validate? One of them, the input-target dependence. In simulation, a sensor stream that is only an input does not stay readable after training, and it does when it is also predicted. RH20T tests this on a quantity that the robot measures. Force stays readable only when it is also a prediction target (0.88 for VFX against −0.16 for VF on held-out tasks). The dataset has no parameter labels, so the findings on hidden parameters, horizons, and forecasts are tested in simulation only. The revised Sec. 6 says this.

Q1: why a custom 2D environment and a compact model? The measurements require four properties at once: physical parameters that do not directly change the rendering at a fixed state, exact replay of a state under another parameter or action, touch resolved within a control step, and a closed-form relation between each parameter and an observable for the reference estimators. The analytic simulator allows us to isolate these factors and compare training configurations under matched conditions. We acknowledge that the simplified environment and compact model limit the scope of our conclusions. The revision extends the evaluation to larger models, other model families, and MuJoCo versions of the task.

W1: established world-model approaches. We added a recurrent state-space model that follows DreamerV3, predictors on frozen DINOv2 and V-JEPA 2 features in the manner of DINO-WM and V-JEPA 2-AC, larger versions of our model up to 56M parameters, and a LeWorldModel-style configuration. The effect of the touch target appears in all of them, and the horizon effect wherever we tested it (general response, part 1).

W2: general physics simulators. We rebuilt the task in MuJoCo (App. H). The planar version reproduces both findings. A 3D version with a free block that topples, seen by a perspective camera, reproduces the horizon effect. Stiffness readout stays low there even with a touch target (0.08 at most, with the smallest model).

W3: real-world execution. The pushing task runs in simulation, and Sec. 7 lists this as a limitation. RH20T is logged data, so actions cannot be executed on it.

Dear Reviewer SpmM,

Thank you again for your constructive feedback. Our revision now includes comparisons with additional world-model architectures, MuJoCo experiments, and the goal-directed pushing evaluation you specifically suggested.

As the discussion period is nearing its end, we would appreciate any further thoughts on these additions, particularly whether they address your original concerns.

Thank you for your time and consideration.

Thank you for the response and the new experiments, which resolve most of my concerns. On whether the paper's claims hold outside the custom simulator, the MuJoCo results show that predicting further ahead still helps decode position and velocity. Since the gain appears in both the planar and 3D versions, I think this claim is now solid and consider raising my score.

However, predicting touch helps decode contact stiffness less in the planar version and very little in 3D. My reservation about the missing real-robot tests also remains. A real-robot demo in this setting could be difficult, but without it the practical value of the paper's claims is unclear. On balance, I am raising my rating from 2 to 3.

Reviewer_snnHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

Summary

This paper investigates which physical quantities are accessible in the latent states of predictive world models and how this depends on model inputs, prediction targets, and prediction horizons. The authors introduce PokeWorld, a controlled 2D pushing environment in which object mass, drag, and contact stiffness vary across episodes while remaining visually indistinguishable. They use this environment to compare physical-parameter readout from learned representations against empirical recovery references from the observation streams, and separately measure whether model forecasts depend on the underlying physical parameters. Experiments on RH20T provide additional evidence for the dependence of representation readout on input and target modalities.

The paper's main findings are that prediction targets influence which physical quantities are linearly decodable, longer prediction horizons improve motion-state readout, and representation readout does not necessarily imply that a model uses the corresponding information correctly in its forecasts.

Assessment:

I find the paper's central question interesting, and PokeWorld provides a useful controlled environment for investigating physical information in predictive representations. The distinction between decodability and actual use of physical information is also valuable.

However, I believe the current study remains limited in scale and scope, and it does not sufficiently establish the practical implications of its findings for downstream world-model capabilities. In particular, the relationship between improved physical-property readout and actual task performance remains unclear.

I therefore lean toward rejection in the current form. I would be open to revising this assessment if the authors could provide a downstream evaluation demonstrating the utility of the proposed training choices, together with scaling experiments showing that the main findings persist beyond the current small-model setting.

Critical strengths

Controlled experimental environment. PokeWorld is a useful contribution for studying physical information in learned representations. By varying physical parameters independently of visual appearance, the authors enable controlled experiments that are difficult to perform using conventional visual datasets.

Careful experimental design. The paper systematically varies model inputs, prediction targets, and prediction horizons, helping isolate their individual effects. The distinction between linear readout and forecast dependence is particularly useful.

Critical weaknesses

1. Lack of downstream evaluation.
Although the paper studies physical-property readout and forecast dependence, it remains unclear whether the proposed training choices improve downstream capabilities such as planning, control, or action-conditioned forecasting. Improved decodability does not necessarily translate into a better world model.

2. Limited scale and experimental scope.
PokeWorld provides a useful controlled environment, but it is restricted to a simple 2D setup with a single object and three hidden physical parameters. The models are also relatively small (5M parameters) and trained on only 4,000 trajectories. Scaling experiments and more complex environments would help establish whether the findings generalize.

3. Limited real-world validation.
The RH20T experiments support the observed input–target dependence for force readout, but do not establish whether the findings extend to recovering hidden physical parameters or improving real-world downstream tasks.

4. Related work and positioning.
The related-work section mostly summarizes existing papers, with limited discussion of how they relate to the proposed approach and what distinguishes this work from prior studies on physical information in latent representations.

Questions for authors
  1. Do the observed relationships between prediction targets, prediction horizons, and physical-property readout persist at substantially larger model scales?

  2. Do the proposed training choices improve downstream planning, control, or action-conditioned forecasting, and are any such improvements correlated with the observed gains in physical-property readout?

  3. How well do the conclusions generalize to more complex environments and to world-model architectures beyond the particular JEPA framework studied here?

2 comments

Thank you for stating clearly what would change your assessment. The revision contains both the downstream evaluation and the scaling experiments.

W1 / Q2: downstream evaluation. We built a goal-directed pushing task (general response, part 2). When a controller plans with parameters decoded from the frozen world model, the training choices matter. Estimates from VFX lower the peak-force error from 19.1% to 15.8% relative to VF, which receives the same inputs (paired difference 3.3 points, 95% interval [2.8, 3.8]), and to 13.1% with four times more data. The gains track the readout: across the 117 models we evaluated, the error falls as the readout rises (rank correlation −0.98; Fig. 5c).

Multi-horizon training also helps action-conditioned forecasting directly. On held-out episodes, the 16-step position error is 0.113 for the multi-horizon model and 0.282 when a single-step model is rolled out (Table 19).

The baseline and the coordinate-target model of Table 3 give a matched comparison of the two uses of the representation. Between them, mass readout on the task contexts rises from 0.18 to 0.59, and the displacement error falls from 16.2% to 11.9% when the controller plans with decoded parameters and the simulator. With the models' own forecasts, the errors are 27.2% and 27.0% (paired difference 0.2 points, 95% interval [−0.4, 0.9]). The candidate pushes are rare under the behavior policy of the training data. A higher readout thus helps parameter-based control without a comparable gain in action selection with the model's own forecasts.

W2 / Q1: scale. Both findings persist at 16M and 56M parameters, in a LeWorldModel-style configuration, and with four times more data (general response, part 1). When touch is only an input, stiffness readout stays at chance in every setting. When it is predicted, VFX reads it at 0.50 to 0.53, and at 0.66 with more data.

Q3: other architectures and environments. The effect of the touch target recurs in a recurrent state-space model that follows DreamerV3 and in predictors on frozen DINOv2 and V-JEPA 2 features. With MuJoCo physics, both findings hold in the planar task. In a 3D version with a toppling block and a perspective camera, the horizon effect holds and the touch target raises mass readout, but stiffness readout stays low (0.08 at most, with the smallest model).

W3: real-world validation. We agree with your reading of the RH20T experiment, and the revised Sec. 6 states it. RH20T has no labels of mass, friction, or stiffness, so it can test only the input-target dependence, on a quantity that the robot measures. Force stays readable only when it is also predicted (0.88 for VFX against −0.16 for VF on held-out tasks). Hidden-parameter recovery and control on a real robot need data that this dataset does not provide.

W4: related work. Each paragraph of Sec. 2 now ends with how our study differs. Work that adds targets to JEPA world models proposes a model or a target; we vary inputs, targets, and horizons within one architecture and test whether the forecasts depend on the hidden parameters. A probe alone tells neither what the observations make recoverable nor whether the forecasts use the quantity, so we pair each readout with both. Multimodal models usually use a modality as input and target at once, and our factorial separates the two roles.

On soundness. We have added experiments addressing the concerns about scale, experimental scope, and downstream evaluation. We would be glad to address any remaining methodological concerns during the discussion.

Dear Reviewer snnH,

Thank you again for your constructive review. We have added the scaling experiments and downstream evaluation you identified as important for reconsidering your assessment.

As the discussion period is nearing its end, we would appreciate your feedback on whether these additions address your main concerns, or if any important questions remain unresolved.

Thank you for your time and consideration.

Reviewer_27izOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper starts off with the premise that latent world models should learn the underlying physics of their environment. The authors ask which physical quantities become accessible in the latent space, how this depends on training choices, and whether the model uses them for their predictive capabilities.

To study this question, the authors introduce POKEWORLD, a simulated environment in which robot fingers push visually identical objects that differ in mass, drag, stiffness. Additionally, the authors provide extensive controlled experiments with action-conditioned JEPA variants on POKEWORLD show that the choice of prediction targets, the prediction horizon, and input-target dependence affects how well the hidden parameters can be recovered.

The authors also evaluate on real robot data from RH20T to emphasize that these trends from the simulated environment POKEWORLD also hold true on more realistic robotic environments.

Critical strengths

The controlled analysis varies one factor at a time and tests various JEPA-based world model variants making it a clean comparison across the board. Additionally, studying the latent readouts helps future researchers understand what JEPA-based world models utilize for their prediction and what their learned latent space captures. Extending the analysis to RH20T data was a nice addition and validates the claim beyond their simulation benchmark.

Critical weaknesses

W1. Only one model family at a small scale is evaluated on POKEWORLD. Every comparison is a variant of the authors' own JEPA-based world model implementation which tends to be smaller than most other JEPA implementations. To add on, POKEWORLD is a relatively small scale benchmark with simple objects, three hidden parameters, and grayscale frames. While the authors acknowledge these limitations and also mention that larger visual models may yield higher readouts in the paper already, the research question the paper seeks to answer is whether latent world models can learn the underlying physics of their environment. In my opinion, this question cannot sufficiently be answered by this benchmark since it is too small scale as well as only one small implementation of a JEPA-based world model. These sort of benchmarks are usually used to motivate a new JEPA architecture which was not the score of this paper. Since POKEWORLD and its analysis are measured as their contributions, I would expect various different world model implementations such as perhaps V-JEPA, DINO-WM, LeWM, or Dreamer models to be tested and analyzed to emphasize whether these findings hold true for latent world models generally or just this particular implementation and scale.

W2. The takeaways/insights from this paper to build better world models are not exactly clear. Section 7 recommends including the physical quantities of interests as prediction targets, predict at several horizons, and tests on the effects of unseen actions. While predicting at several horizons can be an interesting point, the other two recommendation are less convincing. Testing on unseen actions can be a reasonable evaluation practice, yet finding these actions in practice is the hard part and goes more if world models have generalization capabilities. Additionally, the readout gains come from predicting a stream that directly carries the quantity which requires that input at training time. The strongest results on forecast dependence additionally rely on targets that are not available in most real world settings. It is therefore unclear to me how these findings point towards better latent world model design as opposed to adding more supervised learning signals.

Questions for authors

Q1. Could the same readout and forecast analysis be applied to latents of widely implemented world models like V-JEPA 2AC, DINOWM, or Dreamer on POKEWORLD to validate the trend across latent world models generally?

Q2. The limitations note that larger models may yield higher readouts by default. Can you test to see if the JEPA model is scaled to the typical parameter size changes the main findings at all or not? From my understanding the model implementation is based off of LeWM which should be around 15 million parameters instead of the implemented 5 million parameters. My concern is that the reduction of the encoder and predictor may have resulted in worse latent modeling as analysis on V-JEPA had shown emergent understanding of intuitive physics [1].

[1] Garrido, Quentin, et al. "Intuitive physics understanding emerges from self-supervised pretraining on natural videos." arXiv preprint arXiv:2502.11831 (2025).

4 comments

Thank you for the careful review, and for naming the model families and the scale you wanted to see.

W1 / Q1: other world models. We added the families you named (general response, part 1): a recurrent state-space model that follows DreamerV3, and predictors on frozen DINOv2 and V-JEPA 2 features in the manner of DINO-WM and V-JEPA 2-AC. All of them show the effect of the touch target. Stiffness readout rises from 0.38 to 0.87, from 0.13 to 0.49, and from 0.00 to 0.29. Our frozen-encoder predictors are a reduced form of the two methods, with pooled features and our own predictor.

Q2: scale. Your understanding is correct: our model follows LeWorldModel at a smaller size. We now test 16M and 56M versions and a LeWorldModel-style configuration with its ViT-Tiny encoder at 16.6M parameters. The findings do not change (table in the general response). On the concern about worse latent modeling, increasing capacity alone did not improve the held-out prediction loss under the same training setting, while four times more data did (Table 19). You are also right that large pretrained video models read more physics from the frames. Frozen V-JEPA 2 encoders, probed directly, read all three parameters better than our vision-only model (stiffness 0.31 and 0.42 against −0.03), in line with Garrido et al.

W1: size of the benchmark. PokeWorld is small by design. The measurements need to replay the same state under another parameter or action, and to know what each observation stream makes recoverable. To check that the findings are not tied to our simulator, we repeated them with MuJoCo physics. The planar version reproduces both findings. In a 3D version with a toppling block, the horizon effect holds and stiffness readout stays low even with a touch target.

W2: takeaways. We agree with much of this point and rewrote Sec. 7.

"The readout gains come from predicting a stream that directly carries the quantity, which requires that input at training time." Yes, the sensor is needed during training. What we find useful is that a stream used only as a target is not needed at inference: VX_t sees vision alone at test time and still reads stiffness (0.43). It is also not obvious that giving the model the sensor is insufficient. VF receives the full touch stream and reads stiffness at chance at every scale we tried. The targets are raw sensor values, so no label of a physical parameter is involved.

"The strongest results on forecast dependence rely on targets that are not available in most real-world settings." We agree. The coordinate targets and action-effect heads use simulator state. They serve to locate where the limitation lies, and they are no longer among the recommendations.

"Finding these actions in practice is the hard part." We agree, and we replaced this recommendation. The new pushing experiment (general response, part 2) gives a more practical form of the same message. A higher readout helps a controller that plans with decoded parameters, where the peak-force error is 19.1% for VF and 15.8% for VFX. The models' own forecasts, in contrast, select pushes worse than a simulator with nominal parameters, whatever their readout.

Sec. 7 now makes three recommendations: predict the sensor streams that carry the quantity of interest, predict at several horizons, and do not choose a world model for control by readout alone.

Thank you for the thorough and detailed rebuttal. The additional scaling experiments as well as comparisons across model families were insightful and great to see. While my questions for the paper were answered, I am still not fully convinced the weakness of the insights and results of this paper are enough to raise my score for the following reasons.

Regarding W1 and Q1. I would like to thank the efforts to provide additional experiments to alleviate my concerns on scale of the model tested and lack of other world model implementations. I believe my concerns have been thoroughly answered here. However, the results on MuJuCo and 3D settings are not convincing and lead me to think that my concerns regarding the size and scale of the benchmark still hold true. While some of the benefits outlined on the original POKEWORLD setting are shown on the 2 new settings, I remain uncertain if the takeaways and insights are supported given the lack of benefits overall across the more realistic settings. The benefits of VX_t are nonexistent in 3D and slightly noticeable on stiffness for the MuJuCo planar setting. Additionally, the RH20T results highlight that VFX is necessary for readout gains. I still believe the results and significance of the paper is limited towards practical recommendations that can be applied towards training.

I appreciate the addition of the downstream control experiments. However, I am not convinced that these experiments provide evidence of practical improvements to world modeling. The improvements in control performance rely on the simulator-based controller, rather than the world model’s own dynamics predictions. The argument do not choose a world model for control by readout alone since even when the physical readout can improve, the model’s own predictions perform poorly when used for action selection can easily be explained by the inherent limitations of latent planning with error accumulation. The peak force error in the simulator-based control task is above 13% in all cases which is enough error to accumulate in latent planning to fail. Rather than this specifically being related to physical information being accessible but underused in world models, I believe that this is just an issue of latent planning with enough error.

I remain uncertain that the additional experiments sufficiently support strong implications for improving the capabilities of world models. My remaining concerns are about whether the conclusions and practical implications of the paper are sufficiently significant enough and therefore I will maintain my original score.

Thank you for your thoughtful follow-up. We appreciate your acknowledgment that our additional experiments have thoroughly addressed the original concerns about model scale and alternative world-model implementations. We would like to clarify the remaining points regarding generality and practical significance.

Generality across environments. We agree that predicting touch does not consistently improve stiffness readout across environments. In planar MuJoCo, stiffness readout increases from −0.03 (VF) to 0.29 (VFX), while the improvement is much smaller in PokeWorld-3D. In 3D, predicting touch instead improves mass readout from 0.06 (V) to 0.15 (VXt), and multi-horizon prediction improves position readout from 0.37 to 0.95. On RH20T, predicting force preserves its readout when force is also provided as an input. These results distinguish the environment-dependent effect of sensor prediction targets from the more consistent benefit of multi-horizon prediction for motion readout.

Downstream control. The simulator-based controller isolates the value of physical parameters decoded from learned representations. With identical sensor inputs, changing the training targets from VF to VFX reduces peak-force error from 19.1% to 15.8%, and further to 13.1% with more training data. This demonstrates that differences in physical-parameter readout can affect downstream decisions when the decoded parameters are used for planning.

The controller using learned dynamics selects actions through a direct 16-step prediction head, without iterative latent rollout. Its poor performance therefore cannot be attributed specifically to errors accumulating over repeated latent rollouts. Long-horizon prediction error and limited coverage of candidate actions in the training data may still contribute. We also clarify that the reported 13.1% peak-force error is a normalized task outcome error of the simulator-based controller, not a per-step dynamics prediction error.

Readout and forecast dependence. Table 3 shows that adding coordinate targets increases mass readout from 0.10 to 0.48, while action-effect prediction error does not improve (0.68 versus 0.72). Figure 4 provides a further controlled comparison: with the same frozen representation and intervention inputs, changing the prediction-head objective from future-latent prediction to action-effect prediction reduces error from 0.34 to 0.19. Although this diagnostic uses simulator-derived supervision, it demonstrates that the prediction objective affects action-effect accuracy even when the representation is held fixed.

Scientific significance and practical implications. Our experiments establish two findings relevant to world-model research. First, prediction objectives affect which physical quantities remain accessible in learned representations. In PokeWorld, stiffness readout increases from −0.01 (VF) to 0.50 (VFX). Moreover, VXt achieves a stiffness readout of 0.43 using vision alone at inference, demonstrating that touch can improve learned representations through training targets without being required as an inference input. Second, stronger physical-parameter readout does not guarantee accurate action-conditioned predictions. Our results therefore support evaluating both representation readout and forecast dependence when assessing learned world models.

More fundamentally, our contribution is a controlled empirical investigation of how physical information becomes accessible in learned world-model representations and whether that information is reflected in their predictions. This is an important scientific question in its own right, with direct implications for training-objective design and model evaluation. Its value can be assessed through the empirical evidence without requiring the same study to introduce a new control algorithm. The controlled setting makes it possible to compare physical parameters and counterfactual actions under matched conditions, while the additional model families, MuJoCo experiments, and RH20T evaluation extend the scope of our analysis.

We sincerely appreciate the time and care you have devoted to reviewing our work, especially your consideration of the substantial additional experiments. We are grateful that you found the original concerns about model scale and architecture coverage thoroughly addressed. We fully respect your remaining reservations, and respectfully hope that the new evidence and the revisions made in direct response to your feedback will be given due weight in your overall assessment.

Thank you for your response and clarifications. I appreciate the authors efforts and polite comments and will respond below.

While the multi-horizon results appear more consistent across environments, the benefits of the prediction targets are substantially weaker in the complex settings as discussed. I remain unconvinced that while there are some areas where performance improved, there is not overall sufficient evidence beyond the original POKEWORLD benchmark that these targets provide meaningful benefits without requiring the corresponding sensors during inference and in some cases, even when the sensors are available.

In 3D, VXt provides no improvement in stiffness readout and limited improvement in mass readout. On RH20T VX achieves poor force readouts and needs VFX to perform better. Similarly, the downstream control task sees the most benefit from VF to VFX. These results limit the broad applicability of the proposed training recommendations other than the multi-horizon prediction result which had been roughly explored in prior related work [1], [2].

Regarding the downstream control, a large motivation of world models is to enable accurate, efficient prediction and planning without having to interact with the underlying simulator during inference. The results show that the predicted parameters can improve planning when planning with the simulator, but is not enough without the simulator. I see that the point is to show that readout quality does not necessarily correlate to better world model performance/control. This is a valid distinction, but I remain unconvinced that these results by themselves show strong implications for improving world model capabilities and would need see something more (e.g. what physical quantities could improve control, what hurts control performance when ablated out).

Overall, I appreciate the authors responses to my concerns but still hold my reservations on the strength and significance of the empirical study. However, I will remain open minded and will also take in consideration the responses of the other reviewers given that they have higher confidence scores in this field.

[1] Hafner, Danijar, et al. "Learning latent dynamics for planning from pixels." International conference on machine learning. PMLR, 2019.

[2] Schwarzer, Max, et al. "Data-efficient reinforcement learning with self-predictive representations." arXiv preprint arXiv:2007.05929 (2020).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

Part 1: scale, other world models, and other physics

We thank the reviewers for their careful and constructive reviews. The reviews share three requests: evidence beyond one small model and one model family, a downstream evaluation, and a clearer statement of scope. The revision adds experiments for each of them. Changes are blue in the revised PDF, and the main text stays within ten pages.

Scale (Fig. 5a,b; App. G). We repeated the two main comparisons with 16M and 56M models, with a LeWorldModel-style configuration built on a ViT encoder, and with four times more training data. The findings do not change. At every scale, receiving touch as an input does not make stiffness readable, and predicting it does.

stiffness readout4.2M16M56MLeWM-style4.2M, 4× data
VF: proprioception and touch as inputs−0.010.010.030.050.00
VFX: both also predicted0.500.520.530.500.66

Predicting at several horizons also keeps its advantage for position and velocity readout at every size. Velocity readout is 0.78 against 0.40 with a single next-step head at 4.2M, and 0.83 against 0.52 at 56M. The gap is smaller with the ViT encoder.

Other world models (App. H). We added a recurrent state-space model that follows DreamerV3, and predictors trained on frozen DINOv2 and V-JEPA 2 features, which is the design of DINO-WM and V-JEPA 2-AC in a reduced form. In all three, predicting touch raises stiffness readout: from 0.38 to 0.87 in the RSSM, from 0.13 to 0.49 with DINOv2 features, and from 0.00 to 0.29 with V-JEPA 2 features.

These models also showed us something that our own models had not. The RSSM and the DINOv2 predictors read part of the stiffness from vision alone. A new reference estimator confirms that the frames contain it: once the finger is tracked in the image, stiffness is recovered from the frames at 0.80. The revised paper therefore says that stiffness is visible in the interaction, and that our JEPA models keep little of it unless touch is predicted.

Other physics (App. H; Fig. 7). We rebuilt the task in MuJoCo. In the planar version both findings hold, with a lower stiffness readout (0.29 for VFX against −0.03 for VF). We also built a 3D version in which a block slides, yaws, and topples in front of a perspective camera (animated in the supplementary material). There the horizon effect is as strong as before (position readout 0.95 against 0.37 with a single next-step head) and the touch target raises mass readout, but stiffness readout stays low (0.08 at most). This variant was run only with the smallest model.

Part 2: does readout matter for control?

Reviewers snnH and SpmM asked whether a better readout helps a downstream task. We built a goal-directed pushing task to test this (Fig. 5c,d; App. I). A controller picks one of 39 push strengths so that the object moves by a target distance, or so that the peak contact force reaches a target value.

We used each world model in two ways. In the first, the controller plans with the simulator, and the frozen world model supplies the hidden parameters through a linear probe fitted on separate contexts. The better the readout, the better the push.

parameters fromstiffness readoutpeak-force error
nominal values19.8%
VF0.0419.1%
VFX0.4515.8%
VFX, 4× data0.7113.1%
true parameters1.001.3%

VF and VFX receive the same inputs; VFX additionally predicts proprioception and touch. On the same test contexts, the difference between them is 3.3 points (95% interval [2.8, 3.8]).

In the second, the controller selects the push with the model's own 16-step forecast, without the simulator. This works poorly for all 32 tested JEPA configurations. The displacement error is 27 to 36%, worse than planning with nominal parameters (20.5%). Raising the readout does not help either: a model whose mass readout is 0.59 selects pushes no better than one at 0.18.

A good readout therefore helps a controller that is handed the estimate, and it does not tell us whether the model's own forecasts will choose good actions. The paper reported this gap between readout and use for forecasts, and the pushing task shows it for decisions. We changed the third recommendation of Sec. 7 accordingly.

Text changes. The abstract states the scope of each claim. Sec. 2 says how the study differs from prior work, Sec. 3 states the design of the environment, and Sec. 6 says which finding the RH20T experiment tests. To make room for the new results, the RH20T trace figure (Fig. 5 of the submitted version) is now Fig. 6 in App. F.

Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss.

Thanks, AC for #586

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.