- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces POKEWORLD, a 2D pushing simulator in which a robot finger pushes objects that remain visually identical across episodes despite variations in mass, drag, and contact stiffness. The paper first assess physical-parameter recovery from observation sequences, then train matched action-conditioned world models with different input streams, prediction targets, and prediction horizons. They evaluate whether physical parameters can be decoded from the frozen models' latent states using linear probes and whether the models' forecasts depend on those parameters.
Critical strengths
-
The paper introduces POKEWORLD, a 2D pushing environment in which objects remain visually identical while their mass, drag, and contact stiffness vary across episodes.
-
The paper presents extensive ablation studies examining input modalities, prediction targets, and prediction horizons.
Critical weaknesses
-
Comparisons with established world-model approaches are missing. The paper uses a relatively simple architecture with a small number of parameters, but does not compare it with established alternatives. This makes it difficult to assess whether the chosen architecture is appropriate for the study.
-
The simulation experiments rely on a simple, purpose-built environment, without further validation in more general physics simulation settings. The paper also does not demonstrate the advantages of this custom 2D pushing environment over established simulation platforms such as MuJoCo and Isaac Gym. These limitations weaken the persuasiveness of the experimental evidence.
-
There is no real-world execution using the estimated physical parameters, reducing the credibility.
Questions for authors
-
What motivated the choice of a custom 2D pushing environment and a compact world-model architecture? How do these choices affect the scope and generalizability of the conclusions?
-
The simulation experiments evaluate hidden parameters such as mass, drag, and stiffness, whereas the real-data experiments evaluate force readout and prediction. Given that these measure different capabilities, which simulation findings do the RH20T results directly validate?
-
Can the physical parameters decoded from the learned representations be used to guide robot actions? For example, in a goal-directed pushing task, how does a controller using these estimates compare with the same controller using fixed nominal parameters or ground-truth parameters?
3 comments
Dear Reviewer SpmM,
Thank you again for your constructive feedback. Our revision now includes comparisons with additional world-model architectures, MuJoCo experiments, and the goal-directed pushing evaluation you specifically suggested.
As the discussion period is nearing its end, we would appreciate any further thoughts on these additions, particularly whether they address your original concerns.
Thank you for your time and consideration.
Thank you for the response and the new experiments, which resolve most of my concerns. On whether the paper's claims hold outside the custom simulator, the MuJoCo results show that predicting further ahead still helps decode position and velocity. Since the gain appears in both the planar and 3D versions, I think this claim is now solid and consider raising my score.
However, predicting touch helps decode contact stiffness less in the planar version and very little in 3D. My reservation about the missing real-robot tests also remains. A real-robot demo in this setting could be difficult, but without it the practical value of the paper's claims is unclear. On balance, I am raising my rating from 2 to 3.
We thank the reviewer for the constructive feedback. We have expanded the experiments to address the concerns about model generalizability, simulation settings, and downstream utility.
Q3: can the decoded parameters guide actions? Yes. We ran the experiment you describe (general response, part 2; App. I). In a goal-directed pushing task, the controller that plans with parameters decoded from the frozen world model lies between the nominal and the ground-truth controller. The peak-force error is 19.8% with nominal parameters, 19.1% with estimates from VF, 15.8% with estimates from VFX, 13.1% for VFX with four times more data, and 1.3% with the true parameters. VF and VFX receive the same inputs; VFX additionally predicts proprioception and touch.
Q2: which simulation findings does RH20T validate? One of them, the input-target dependence. In simulation, a sensor stream that is only an input does not stay readable after training, and it does when it is also predicted. RH20T tests this on a quantity that the robot measures. Force stays readable only when it is also a prediction target (0.88 for VFX against −0.16 for VF on held-out tasks). The dataset has no parameter labels, so the findings on hidden parameters, horizons, and forecasts are tested in simulation only. The revised Sec. 6 says this.
Q1: why a custom 2D environment and a compact model? The measurements require four properties at once: physical parameters that do not directly change the rendering at a fixed state, exact replay of a state under another parameter or action, touch resolved within a control step, and a closed-form relation between each parameter and an observable for the reference estimators. The analytic simulator allows us to isolate these factors and compare training configurations under matched conditions. We acknowledge that the simplified environment and compact model limit the scope of our conclusions. The revision extends the evaluation to larger models, other model families, and MuJoCo versions of the task.
W1: established world-model approaches. We added a recurrent state-space model that follows DreamerV3, predictors on frozen DINOv2 and V-JEPA 2 features in the manner of DINO-WM and V-JEPA 2-AC, larger versions of our model up to 56M parameters, and a LeWorldModel-style configuration. The effect of the touch target appears in all of them, and the horizon effect wherever we tested it (general response, part 1).
W2: general physics simulators. We rebuilt the task in MuJoCo (App. H). The planar version reproduces both findings. A 3D version with a free block that topples, seen by a perspective camera, reproduces the horizon effect. Stiffness readout stays low there even with a touch target (0.08 at most, with the smallest model).
W3: real-world execution. The pushing task runs in simulation, and Sec. 7 lists this as a limitation. RH20T is logged data, so actions cannot be executed on it.