- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes H-JEPA, which introduces end-to-end training and planning methods for hierarchical JEPA. Higher-level features encode abstract information closer to goal, and lower-level features encode detailed information which changes quickly. Higher-level predictions become a subgoal for low-level planners. This paper provides a useful recipe for training hierarchical JEPA, and several helpful analyses which explains what features in different levels actually encodes, why and where the proposed framework works well. Also, this paper evaluates preliminary results for real-world planning with DROID dataset, demonstrating that planned end-effector trajectories using H-JEPA are closer to expert trajectories with a lower planner budget.
Critical strengths
-
This work shows end-to-end hierarchical learning of JEPA with different temporal resolution levels, and this approach improves planning performances, showing that features at each level encode different information. Low-level features encode detailed information that changes quickly, while high-level ones encode abstract information that changes slowly.
-
This work provides several interesting analyses, such as the relationship between temporal frequency gaps and abstraction, and the temporal decomposition of planning costs. These provide a deeper understanding of why and where the proposed framework works well.
Critical weaknesses
-
It seems most of the simulation tasks in this work (FourRoom, Cube, Push-T) are somewhat simple, and it leaves questions for more complex, long-horizon tasks. I’m also not sure this framework is generally helpful for manipulation or other tasks, where temporal frequency gap is not that large and faster information is crucial for solving such as dexterous manipulation tasks.
-
Real-world results seem to be limited. There are no online, closed-loop results for real world robotics. DROID is only scored in the 16 clips, which seems too few for a reliable conclusion. Also it is not scored with success rate, but it is scored with the fidelity of planned end-effector path and the expert’s, only with xyz coordinates. Furthermore, closer trajectories to the expert’s doesn’t guarantee success in real execution, and there might be successful trajectories which are totally different from expert’s.
Questions for authors
-
What is the difference between the LeWM (flat) features and level-1 features of H-JEPA? Do they encode different information?
-
It seems IDM is essential for real-world datasets, but why does the base objective of H-JEPA not include it? Is there any circumstance that IDM hurts training H-JEPA?
-
Can the authors show closed-loop success rates of real-world robots? This can make the paper more powerful.
-
In manipulation tasks that temporal frequency gap is not that large, what exactly makes gains of H-JEPA compared to HWM which also has hierarchical planning but not hierarchical features?
-
I wonder that for tasks where faster and detailed information is important such as dexterous tasks with robot hands, would the proposed method work well? I am curious about the authors’ thoughts on this.
4 comments
rebuttal ihK9 Part 2
Q3, W2 - Real-world closed-loop robot success evaluation.
There was not enough time during the rebuttal period to run closed-loop experiments on a real robot. App. J.2 relates our offline metric to closed-loop success on OGBench Cube, a grasp-centred pick-and-place task close to our DROID clips with binary success: fidelity is a necessary condition for success (Fig. 15) and understates the hierarchy's larger success advantage. DROID results now use planner settings tuned on 48 separate validation clips and held-out test clips, 3 training × 3 planner seeds (see oMBF-Q3).
Q4. H-JEPA vs. HWM gains in manipulation tasks with weak timescale separation.
HWM and H-JEPA both plan hierarchically; what differs is that H-JEPA learns a separate representation at each level. On DROID, which has no large frequency gap, this is what makes the difference: training the level-2 representation end to end reshapes level 1. H-JEPA's level 1 gives a planning cost that tracks progress toward the goal more consistently and is easier to optimize than both LeWM+IDM's and HWM's level 1, whose shared embedding behaves like a flat model's (table below), and that alone retains H-JEPA's full advantage when planning with level 1 only.
With the planner's goal cost and the expert plan:
| Planning space | Monotonicity ↑ | Bumpiness ↓ | GD success ↑ |
|---|---|---|---|
| LeWM+IDM, level 1 | 0.51 ± 0.03 | 15 ± 1% | 83 ± 4% |
| HWM, level 1 | 0.51 ± 0.01 | 13 ± 2% | 81 ± 1% |
| H-JEPA, level 1 | 0.63 ± 0.01 | 6 ± 1% | 87 ± 4% |
Monotonicity (App. Table 10): ; 1 = always decreasing. Bumpiness: , with the total variation (sum of absolute changes along ) and the cost from a random plan , rescaled to ; 0 = steady descent. GD success: share of 16 gradient-descent runs with (final, median initial and best cost).
Q5, W1. Applicability to dexterous manipulation
We believe dexterous manipulation is a promising application for H-JEPA because it requires multiple levels of abstraction.
Higher-level representations could abstract away detailed finger poses while retaining task-relevant quantities such as arm position and grasp state (e.g., open/closed), with actions encoding arm displacement and grasp/release. Lower levels would preserve fine-grained finger poses and dynamics for precise control. This would allow high-level planning to guide task progress while low-level planning handles precise manipulation.
Moreover, as in AntMaze, higher-level representations could provide more informative planning objectives: task success often depends on object position or grasp state rather than exact finger configurations. Abstracting away such goal-irrelevant details could therefore improve high-level planning.
If we addressed the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.
Thank you for the detailed rebuttal. It partially resolved my concerns. I am still curious why the HWM and H-JEPA level 1 features are different. Overall, I will keep my score.
Thank you for the follow-up. We provide a more detailed explanation of the differences between HWM and H-JEPA representations, along with additional probing results, from our response to Zho1–Q1.
Why the representations differ. In HWM, all predictors share a single latent representation, jointly optimized for both short- and long-horizon prediction. These objectives exert competing pressures: short-horizon prediction (with SigReg) benefits from retaining fast-varying details, while long-horizon prediction favors discarding such details and preserving slower, more predictable features. Consequently, HWM's shared representation may settle at an intermediate degree of abstraction. In contrast, H-JEPA learns separate representations at each level, allowing them to specialize for their respective timescales.
Our AntMaze probes support this interpretation:
| AntMaze body-state NMSE (3-level; lower is better) | L1 | L3 |
|---|---|---|
| LeWM | 0.085 | – |
| H-JEPA | 0.065 | 0.697 |
| HWM | 0.182 | 0.182 |
All models retain agent position (NMSE ≤ 2e-4). HWM's shared representation lies between H-JEPA's detail-rich level 1 and abstract level 3 in body-state recoverability.
Implications for planning. This creates a potential tradeoff for HWM: its shared representation retains less body-state information than H-JEPA's level 1, yet preserves more goal-irrelevant detail than H-JEPA's level 3. It may therefore be less suitable for fine-grained locomotion control while also providing a less informative objective for high-level goal matching. H-JEPA can avoid this compromise by allowing each level to retain the information most relevant to its planning timescale.
We hypothesize that this specialization helps explain H-JEPA's substantial improvement over HWM on AntMaze.
rebuttal ihK9 Part 1
We thank the reviewer for the detailed feedback. We address your questions and concerns in two comments below.
Q1. Representation differences between flat LeWM and H-JEPA level 1.
Yes: probing the level-1 latents of the Figure 5 checkpoints shows that LeWM and H-JEPA both encode the agent position needed for planning. H-JEPA's level 1 also encodes dynamics more accurately (AntMaze body state above; Cube arm joint velocities 0.92 → 0.47) and retains less about task-irrelevant distractors. The exception is Push-T, where state variables become harder to decode as depth grows; this is correlated with there being less training data windows available at each depth (86/58/14% of LeWM's clips per epoch, App. C.1), the same confound behind the 3- and 4-level planning drop. Figure R1 reports all targets for all four environments.
Q2. IDM necessity, omission from the base objective, and potential adverse effects.
IDM is not part of the base objective because the simulated environments do not need it: their scenes are fixed across episodes, so the slow-feature collapse that IDM prevents on DROID does not occur. We tested this directly by retraining every headline simulated model from the released code with IDM added at level 1 (on raw actions) and level 2 (on latent actions), with weights 0, 0.1, 1 and 10, the paper's epoch budgets and 3 seeds. Success (%), hierarchical planning from the top level, final epoch:
Weight 0, a retraining of the paper's models with the released code, is within noise of the paper's results (largest gap: Visual AntMaze, 1.4 standard errors). Small and moderate weights help on Visual AntMaze only, so IDM is unnecessary in these settings, and large weights hurt in three of the four environments. Per-epoch curves are in Figure R4. IDM is therefore unnecessary, and sometimes harmful when weighted heavily, on fixed simulated scenes. We left it out of the base objective to keep the method simple, and add it only where it is clearly necessary: visually complex real-world data whose scene changes across episodes, such as DROID. The revised appendix includes these ablations (App. K.3, Table R2, Figure R4).