- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to tackle long-horizon planning in JEPA type world models. The main contributions are:
- A longer-horizon training objective, instead of predicting next state, predict 10 latent states (50 frames, 5 skip) and actions into the future
- A modification of evaluation setting where instead of random start, always start at the beginning
The authors perform simple simulation evaluations on tasks like pick and place, stacking, cyclic rearrangement for up to 4 objects, and show that their method is more robust to increased object number and in the controlled-start setting which requires long-horizon planning. While their method does not uniformly surpass baselines, it wins more often.
While it is interesting to see the effect of longer-horizon training objective, I have questions about whether the training objective makes sense, since there are many possibilities of future states and actions conditioned on current state and action. I wonder whether this training objective just overfits to the task distribution better - the small perturbations in object numbers etc. does not really test generalization. I feel the method would require more justification and thorough evaluation if the claim is that this training recipe yields better general representation.
Critical strengths
- The long horizon angle is interesting - it is an important aspect in world modeling and action planning
- The authors caught the effect of an evaluation setting difference, i.e. random frame start vs. controlled start
- Though small-scale, the ablations are quite comprehensive
Critical weaknesses
- The method design does not seem well-justified - could the authors explain why the model should predict a specific future state and action sequence in long horizon only given the current state and action?
- The evaluation is only on 3 tasks in sim environments. While increasing object numbers does perturb environments slightly, it does not really test generalization or how good the learned representation is in general.
Questions for authors
- Could the authors explain why the model should predict a specific future state and action sequence in long horizon only given the current state and action?
- Could the authors comment on why the proposed method could be more robust to object number change, whereas LeWM is better at 2 objects in both controlled start and random?
- For all methods, controlled start/multiple object performance is quite low. Could you explain this or shed light on what might improve the performance? Since the authors present the controlled-start and multi-object as important findings, it would be good to provide rationale on what these settings are testing.
Thank you all for your thoughtful reviews and the time spent on them.
W1 / Q1 (why predict a future state and action sequence). The predictor is a training signal for the encoder, not a planning model: it is discarded after training and the controller uses only the frozen latent. Regressing 10 future latents (50 steps) and the actions between them from the current state and first action forces the latent to carry what determines how the scene will evolve under the expert, including object positions far from the gripper; with a deterministic loss it learns the expected expert continuation rather than one sampled future. The ablations (§7, Fig. 6) support this: a one-step horizon returns LH-JEPA to LeWM's level and collapses position decodability, the inverse-dynamics term improves control without changing decodability, and the same supervision applied conventionally (to observed frame pairs, on LeWM) lowers control from 14.4 to 2.0 on
pnp4, so predicting actions jointly with future latents is what helps.W2 (scope of the evaluation). We added LIBERO-Object (§6.3, Figs. 2–3): ten tasks with household objects in a cluttered scene, goal offsets up to 100 steps, a controlled clutter-free variant (
LO-Simple), a pick-3 task (LO-Pick3) and search baselines, all with three training seeds. With GC-IDM, LH-JEPA leads LeWM at every offset in every scene (e.g. 95.0±1.7 vs. 72.6±1.5 at 50 steps). The distractor study (App. E) evaluates encoders trained on the clean task in scenes with added distractor objects of varying colour and size. We agree that real-robot data is the next step (§8).Q2 (robustness to object count; LeWM at two objects). The decodability analysis (§6.4) shows why: from two to four objects LeWM's held-out position R2 falls on every task family (and its MLP error grows 54–87×), while LH-JEPA's stays above 0.8, so LeWM's latent stops encoding the cubes the controller must act on. With two cubes this capacity limit does not bind and LeWM is ahead; §8 states that no representation leads on every task.
Q3 (low start-controlled performance). From the true start the controller must approach, grasp and move a cube within the budget, whereas random starts often begin mid-manipulation or already at the goal: a policy that outputs zero actions succeeds on 25.3% of random-start
pnp1episodes. Start-controlled success therefore measures the full manipulation, which is why it is lower. On LIBERO the same protocol separates the representations sharply once grasping and carrying are required (69.4 vs. 19.2 at 75 steps). The LIBERO-Object experiments also exposed a second cause of low performance, the training data distribution: with the number of training frames held fixed, moving fromLO-Fullto the longerLO-Pick3episodes collapses LeWM from 72.6% to 11.9% at 50 steps and from 5.9% to 0.0% at 100 steps, while LH-JEPA falls only from 95.0% to 89.7% and from 48.3% to 35.0%.Here we summarise the key revisions to the paper: