acceptodds
Under review as a conference paper at ICLR 2027

H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning

143 views

Abstract

Long-horizon planning with latent world models requires reasoning across timescales and levels of abstraction. Existing task-agnostic JEPA world models predict and plan at a single timescale or with multiple horizons in one shared latent space. We introduce H-JEPA, an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space. Planning proceeds top-down: the top level optimizes progress toward the goal, and each level's predictions become subgoals for the planner below it. When factors in the data evolve at separated timescales, higher levels discard fast, unpredictable detail and retain slower task-relevant state. Across four simulated navigation and manipulation environments, hierarchical planning improves over a flat JEPA; on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% at matched compute. Ablations attribute these gains to both temporal decomposition and higher-level goal representations. With inverse-dynamics supervision, the approach extends to diverse real-robot videos from DROID, where hierarchy improves offline planning fidelity at lower planner compute.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Model, JEPA, Planning, SSL, Hierarchical world models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_ihK9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes H-JEPA, which introduces end-to-end training and planning methods for hierarchical JEPA. Higher-level features encode abstract information closer to goal, and lower-level features encode detailed information which changes quickly. Higher-level predictions become a subgoal for low-level planners. This paper provides a useful recipe for training hierarchical JEPA, and several helpful analyses which explains what features in different levels actually encodes, why and where the proposed framework works well. Also, this paper evaluates preliminary results for real-world planning with DROID dataset, demonstrating that planned end-effector trajectories using H-JEPA are closer to expert trajectories with a lower planner budget.

Critical strengths
  • This work shows end-to-end hierarchical learning of JEPA with different temporal resolution levels, and this approach improves planning performances, showing that features at each level encode different information. Low-level features encode detailed information that changes quickly, while high-level ones encode abstract information that changes slowly.

  • This work provides several interesting analyses, such as the relationship between temporal frequency gaps and abstraction, and the temporal decomposition of planning costs. These provide a deeper understanding of why and where the proposed framework works well.

Critical weaknesses
  • It seems most of the simulation tasks in this work (FourRoom, Cube, Push-T) are somewhat simple, and it leaves questions for more complex, long-horizon tasks. I’m also not sure this framework is generally helpful for manipulation or other tasks, where temporal frequency gap is not that large and faster information is crucial for solving such as dexterous manipulation tasks.

  • Real-world results seem to be limited. There are no online, closed-loop results for real world robotics. DROID is only scored in the 16 clips, which seems too few for a reliable conclusion. Also it is not scored with success rate, but it is scored with the fidelity of planned end-effector path and the expert’s, only with xyz coordinates. Furthermore, closer trajectories to the expert’s doesn’t guarantee success in real execution, and there might be successful trajectories which are totally different from expert’s.

Questions for authors
  • What is the difference between the LeWM (flat) features and level-1 features of H-JEPA? Do they encode different information?

  • It seems IDM is essential for real-world datasets, but why does the base objective of H-JEPA not include it? Is there any circumstance that IDM hurts training H-JEPA?

  • Can the authors show closed-loop success rates of real-world robots? This can make the paper more powerful.

  • In manipulation tasks that temporal frequency gap is not that large, what exactly makes gains of H-JEPA compared to HWM which also has hierarchical planning but not hierarchical features?

  • I wonder that for tasks where faster and detailed information is important such as dexterous tasks with robot hands, would the proposed method work well? I am curious about the authors’ thoughts on this.

4 comments
AuthorsOpenReview ↗

rebuttal ihK9 Part 1

We thank the reviewer for the detailed feedback. We address your questions and concerns in two comments below.

Q1. Representation differences between flat LeWM and H-JEPA level 1.

Level-1 probe NMSE (lower is better)LeWMH-JEPA 2 levelsH-JEPA 3 levelsH-JEPA 4 levels
AntMaze: agent xy2e-53e-54e-55e-5
AntMaze: body state0.0850.0730.0650.061
FourRoom: agent xy9e-55e-58e-51e-4
FourRoom: distractor xy0.0090.2870.0980.054

Yes: probing the level-1 latents of the Figure 5 checkpoints shows that LeWM and H-JEPA both encode the agent position needed for planning. H-JEPA's level 1 also encodes dynamics more accurately (AntMaze body state above; Cube arm joint velocities 0.92 → 0.47) and retains less about task-irrelevant distractors. The exception is Push-T, where state variables become harder to decode as depth grows; this is correlated with there being less training data windows available at each depth (86/58/14% of LeWM's clips per epoch, App. C.1), the same confound behind the 3- and 4-level planning drop. Figure R1 reports all targets for all four environments.

Q2. IDM necessity, omission from the base objective, and potential adverse effects.

IDM is not part of the base objective because the simulated environments do not need it: their scenes are fixed across episodes, so the slow-feature collapse that IDM prevents on DROID does not occur. We tested this directly by retraining every headline simulated model from the released code with IDM added at level 1 (on raw actions) and level 2 (on latent actions), with weights 0, 0.1, 1 and 10, the paper's epoch budgets and 3 seeds. Success (%), hierarchical planning from the top level, final epoch:

IDM weightPaper (no IDM)00.1110
FourRoom (3 levels)96.0 ± 1.294.0 ± 1.294.7 ± 2.993.3 ± 0.781.3 ± 3.5
Visual AntMaze (3 levels)73.3 ± 3.564.0 ± 5.870.7 ± 1.376.0 ± 1.241.3 ± 1.8
OGBench Cube (4 levels)62.7 ± 2.458.0 ± 3.150.7 ± 6.752.7 ± 4.462.0 ± 3.1
Push-T (2 levels)45.3 ± 1.341.3 ± 4.846.7 ± 2.417.3 ± 9.823.3 ± 4.7

Weight 0, a retraining of the paper's models with the released code, is within noise of the paper's results (largest gap: Visual AntMaze, 1.4 standard errors). Small and moderate weights help on Visual AntMaze only, so IDM is unnecessary in these settings, and large weights hurt in three of the four environments. Per-epoch curves are in Figure R4. IDM is therefore unnecessary, and sometimes harmful when weighted heavily, on fixed simulated scenes. We left it out of the base objective to keep the method simple, and add it only where it is clearly necessary: visually complex real-world data whose scene changes across episodes, such as DROID. The revised appendix includes these ablations (App. K.3, Table R2, Figure R4).

AuthorsOpenReview ↗

rebuttal ihK9 Part 2

Q3, W2 - Real-world closed-loop robot success evaluation.

There was not enough time during the rebuttal period to run closed-loop experiments on a real robot. App. J.2 relates our offline metric to closed-loop success on OGBench Cube, a grasp-centred pick-and-place task close to our DROID clips with binary success: fidelity is a necessary condition for success (Fig. 15) and understates the hierarchy's larger success advantage. DROID results now use planner settings tuned on 48 separate validation clips and held-out test clips, 3 training × 3 planner seeds (see oMBF-Q3).

Q4. H-JEPA vs. HWM gains in manipulation tasks with weak timescale separation.

HWM and H-JEPA both plan hierarchically; what differs is that H-JEPA learns a separate representation at each level. On DROID, which has no large frequency gap, this is what makes the difference: training the level-2 representation end to end reshapes level 1. H-JEPA's level 1 gives a planning cost that tracks progress toward the goal more consistently and is easier to optimize than both LeWM+IDM's and HWM's level 1, whose shared embedding behaves like a flat model's (table below), and that alone retains H-JEPA's full advantage when planning with level 1 only.

With C(a)=∥z^H(a)−zg∥22C(a) = \|\hat z_H(a) - z_g\|_2^2 the planner's goal cost and a⋆a^\star the expert plan:

Planning spaceMonotonicity ↑Bumpiness ↓GD success ↑
LeWM+IDM, level 10.51 ± 0.0315 ± 1%83 ± 4%
HWM, level 10.51 ± 0.0113 ± 2%81 ± 1%
H-JEPA, level 10.63 ± 0.016 ± 1%87 ± 4%

Monotonicity (App. Table 10): −Spearman(t, ∥z^t(a⋆)−zg∥22)-\mathrm{Spearman}\big(t,\ \|\hat z_t(a^\star) - z_g\|_2^2\big); 1 = always decreasing. Bumpiness: TV(c~)−1\mathrm{TV}(\tilde c) - 1, with TV\mathrm{TV} the total variation (sum of absolute changes along α\alpha) and c~(α)\tilde c(\alpha) the cost C((1−α)ar+αa⋆)C\big((1-\alpha)a_r + \alpha a^\star\big) from a random plan ara_r, rescaled to [0,1][0,1]; 0 = steady descent. GD success: share of 16 gradient-descent runs with (cn−c⋆)/(c0−c⋆)≤1%(c_n - c^\star)/(c_0 - c^\star) \le 1\% (final, median initial and best cost).

Q5, W1. Applicability to dexterous manipulation

We believe dexterous manipulation is a promising application for H-JEPA because it requires multiple levels of abstraction.

Higher-level representations could abstract away detailed finger poses while retaining task-relevant quantities such as arm position and grasp state (e.g., open/closed), with actions encoding arm displacement and grasp/release. Lower levels would preserve fine-grained finger poses and dynamics for precise control. This would allow high-level planning to guide task progress while low-level planning handles precise manipulation.

Moreover, as in AntMaze, higher-level representations could provide more informative planning objectives: task success often depends on object position or grasp state rather than exact finger configurations. Abstracting away such goal-irrelevant details could therefore improve high-level planning.

If we addressed the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.

Thank you for the detailed rebuttal. It partially resolved my concerns. I am still curious why the HWM and H-JEPA level 1 features are different. Overall, I will keep my score.

Thank you for the follow-up. We provide a more detailed explanation of the differences between HWM and H-JEPA representations, along with additional probing results, from our response to Zho1–Q1.

Why the representations differ. In HWM, all predictors share a single latent representation, jointly optimized for both short- and long-horizon prediction. These objectives exert competing pressures: short-horizon prediction (with SigReg) benefits from retaining fast-varying details, while long-horizon prediction favors discarding such details and preserving slower, more predictable features. Consequently, HWM's shared representation may settle at an intermediate degree of abstraction. In contrast, H-JEPA learns separate representations at each level, allowing them to specialize for their respective timescales.

Our AntMaze probes support this interpretation:

AntMaze body-state NMSE (3-level; lower is better)L1L3
LeWM0.085–
H-JEPA0.0650.697
HWM0.1820.182

All models retain agent position (NMSE ≤ 2e-4). HWM's shared representation lies between H-JEPA's detail-rich level 1 and abstract level 3 in body-state recoverability.

Implications for planning. This creates a potential tradeoff for HWM: its shared representation retains less body-state information than H-JEPA's level 1, yet preserves more goal-irrelevant detail than H-JEPA's level 3. It may therefore be less suitable for fine-grained locomotion control while also providing a less informative objective for high-level goal matching. H-JEPA can avoid this compromise by allowing each level to retain the information most relevant to its planning timescale.

We hypothesize that this specialization helps explain H-JEPA's substantial improvement over HWM on AntMaze.

Reviewer_oMBFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recent JEPA world models (DINO-WM, PLDM, LeWM) predict the future embedding of an image rather than the image itself. They plan by searching for actions whose predicted embedding matches the embedding of a goal image. They use a single embedding and predict only a few frames ahead. Reaching a far-away goal means chaining many small predictions, so errors build up. And since the embedding keeps every detail, its distance to the goal says little about real progress, for example whether the agent is getting closer to the right room.

The paper builds H-JEPA, proposed by LeCun in 2022, as a stack of JEPAs starting from LeWM at the bottom level. Each level above compresses the embeddings of the level below into its own embedding and predicts further into the future, given a learned summary of the actions in between. All levels are trained together from scratch, without rewards or image reconstruction. To plan, the top level picks a rough route to the goal and its predictions become intermediate goals for the level below, all the way down to the bottom level, which outputs actions.

The authors posit that this helps in two ways:

  1. Splitting a long task into short hops makes each planning problem easier.
  2. Each higher level should learn to keep only what it can predict over its longer horizon, keeping only low-frequency information, such as position, making the distance to the goal better reflect progress.

The authors test two navigation tasks (a four-room maze with a moving distractor, and a visual ant maze), two manipulation tasks (Push-T and cube pick-and-place), and real robot video (DROID), evaluated offline. They compare against the single-level LeWM and against a baseline hierarchy where each level shares the same embedding. On most tasks the hierarchy plans better than the single-level model while using less compute. The gain is clearest on the ant maze, where it also clearly beats the shared-embedding hierarchy. Probes show that higher levels drop fast details, like the ant's legs or the distractor, and keep position. This happens only when parts of the data change at clearly different speeds, as in navigation but not manipulation. On the navigation tasks, just measuring goal distance in a higher level's embedding already improves planning. On DROID the background changes every episode, so the model needs an extra inverse-dynamics loss to keep track of the robot.

Critical strengths
  1. The ant-maze result is significant and well controlled. H-JEPA succeeds much more often than the single-level model, even when that model gets more planning compute. It also clearly beats the shared-embedding hierarchy, which uses the same code, data and planner.
  2. The paper tests its explanation directly and reports where it fails. Keeping the bottom-level model fixed and only measuring goal distance in a higher level's embedding already helps on navigation. The distance maps show that in higher levels, distance shrinks smoothly along the maze corridors. The authors also report the cases where the idea doesn't help (manipulation, deeper models on Push-T) and the limits of their DROID metric.
Critical weaknesses
  1. The baselines are not run at their best, so it is unclear how much of the gain comes from the hierarchy.
  • The single-level LeWM trains via gradient descent on a much longer horizon than in its own paper, which uses a sampling-based planner (CEM) over a short horizon. Gradient descent over long horizons is known to work poorly, so part of the "better with less compute" result may come from this choice.
  • The shared-embedding baseline is not the published HWM, which trains higher levels on a frozen encoder. Here everything is trained together, which may explain why it gets much worse with more levels. Its planner tuning isn't reported.
  • On cube pick-and-place, levels 3 and 4 are used only during training, not during planning. Their one-step horizon makes the planner skip them (Table 7), so the gains from adding them come from training.
  1. Outside the ant maze, the evidence for the two claimed mechanisms is weak.
  • DROID is evaluated offline on a small set of hand-picked clips that also seem to have been used to tune the method and the authors show the score of the arm path can be high when the task fails.
  • The DROID gain doesn't fit the explanation. The shared-embedding hierarchy doesn't help there, so splitting the task in time isn't what helps, and DROID shows no clear fast/slow separation.
  • The link between speed and dropped details is only a correlation. It comes from comparing about six environments (Fig. 4), not from changing how fast something moves within one environment.
  • The benefit of splitting a task into short hops is shown on recorded expert trajectories, not on actual plans.
Questions for authors
  1. Does the hierarchy still beat the single-level LeWM when LeWM uses its own planner (CEM over a short horizon) at the same compute?
  2. How does the hierarchy compare with the published HWM setup (higher levels trained on a frozen encoder) on the ant maze? How was your shared-embedding baseline's planner tuned?
  3. Were the DROID settings chosen on the evaluation clips? If so, can you report results on clips that weren't used for tuning?
  4. Are levels 3 and 4 used during planning on cube pick-and-place? Table 7 suggests they are skipped, which would make the gains from depth a training effect.
  5. Why do the four-level results in Fig. 5 differ from Tables 1 and 9? They use the same tasks.
  6. Does the planning compute include every level's encoders and backward passes? How do model size and training compute compare with LeWM?

Smaller fixes:

  • Table 2 says two levels throughout, but the experiments use up to four.
  • The text says Fig. 8 shows a window size of 2, but the figure shows 1.
  • The Fig. 14 caption says stride 3, while the simulated models use 2.
  • Fig. 13 and Table 1 report different two-level ant-maze results.
  • Fig. 19's boxed "best" cell isn't the highest-scoring one.
  • The Humanoid data and model behind Figs. 4 and 12 aren't described.
3 comments
AuthorsOpenReview ↗

oMBF Rebuttal Part 1

We thank the reviewer for the detailed feedback. We provide the ablations and clarifications you asked in two comments below.

Q1, W1a - Compute-matched comparison against LeWM with its native CEM planner

Success %LeWM + GD (Fig. 5)LeWM + CEM, best ≤ 100 TF
Visual AntMaze18.0 ± 3.50.7 ± 0.7
FourRoom40.7 ± 6.80.0
OGBench Cube32.0 ± 6.161.3 ± 2.4
Push-T40.0 ± 2.38.0 ± 2.0

We ran LeWM's CEM planner with its published hyperparameters and increased the sample count up to our 100 TFLOPs/episode budget, at both LeWM's short horizon and ours. It does worse than gradient-based planning in every environment except Cube.

Success %LeWM + CEM (TFLOPs)H-JEPA 2 levels + CEM (TFLOPs)H-JEPA 3 levels + CEM (TFLOPs)H-JEPA 4 levels + CEM (TFLOPs)
OGBench Cube61.3 ± 2.4 (87)70.7 ± 2.4 (1.5)75.3 ± 3.7 (1.5)78.7 ± 4.8 (1.5)

For Cube, when H-JEPA also plans with CEM at every level, it beats LeWM + CEM at about 1/60 of the compute, so the hierarchy's advantage does not depend on the planner. Full protocol and per-environment results: App. E.1 of the revised PDF.

Q2, W1b - Comparison against original HWM on AntMaze and shared-embedding baseline tuning.

**Two-stage HWM. ** Below are AntMaze results for HWM trained in two stages, as in the original HWM paper: the level-1 world model is pretrained and frozen, and the higher levels are trained on top of it.

AntMaze success rate (%)2 levels3 levels4 levels
HWM, two-stage12.7 ± 2.736.0 ± 3.128.7 ± 1.3
HWM, joint (paper)27.3 ± 1.827.3 ± 3.313.3 ± 3.3
H-JEPA38.7 ± 5.364.0 ± 1.250.7 ± 4.1

H-JEPA still outperforms this HWM variant at every depth (App. E.2).

HWM planner tuning. In the paper, HWM's planner was not tuned separately. It used the planner parameters tuned for H-JEPA. To make the comparison fair, we re-tuned the HWM planner's per-level sample counts on all four environments, under the same 100 TFLOPs/episode budget:

Success rate (%), 2 / 3 / 4 levelsHWM (paper)HWM (tuned)H-JEPA (paper)
AntMaze34.0 / 38.7 / 14.032.0 / 38.0 / 23.339.3 / 73.3 / 67.3
FourRoom75.3 / 99.3 / 92.771.3 / 98.7 / 95.381.3 / 96.0 / 88.0
Cube46.7 / 53.3 / 34.053.3 / 60.0 / 38.047.3 / 60.0 / 62.7
Push-T42.0 / 12.0 / 0.054.0 / 15.3 / 0.745.3 / 17.3 / 0.7

Mean over 3 seeds. Overall, tuning HWM's planner does not change the trend of Figure 5.

We will update the HWM results in Figure 5 with the tuned planners, and describe in the appendix how each model's planner was tuned.

Q3. DROID evaluation independence from hyperparameter tuning. W2a

Yes: in the submission, the GD planner settings were selected on the 16 reported clips. We now follow the protocol used for our simulated environments (App. E): the trained models are fixed, the planner is tuned on a separate set, and the evaluation set is scored once. Tuning uses 48 new grasp-centred clips (3–4 per lab, built like the test clips) from training episodes, disjoint from the 16 test episodes, which are held out from both training and tuning. Each model gets the same grid of learning rates and restart counts, and we keep its best setting on the validation clips. The 16 test clips are then scored once with these frozen settings, 3 training × 3 planner seeds. Fréchet fidelity (%):

LeWM+IDMHWMH-JEPA
Submission (Fig. 7)34.1 ± 1.335.0 ± 0.340.0 ± 2.9
Tuned on validation clips32.9 ± 1.232.1 ± 2.337.2 ± 0.4

The submission row used one planner seed per model; the new row averages three planner seeds per training seed and uses the released code and checkpoints (App. J.2). Part of the drop is therefore planner-seed variance rather than tuning, and H-JEPA remains clearly ahead of both baselines. [Figure R6 (App. J.1)] shows every grid setting against its planner compute, on the validation and test clips: the selected H-JEPA setting costs 2.9 TFLOPs per episode, between LeWM+IDM's (1.9) and HWM's (8.2), and from H-JEPA's cheapest setting (about 1 TFLOP) upward, its best fidelity matches or exceeds both baselines' at every budget, on both sets. The hierarchical models cannot plan with less compute in this offline protocol: with no environment to replan in, level 1 plans the whole 36-step clip at once, so its planning horizon cannot be shortened.

On the second point raised by the reviewer, our Fréchet fidelity offline metric is a proxy for success. There was not enough time during the rebuttal period to run closed-loop experiments on a real robot. App. J.2 relates our offline metric to closed-loop success on OGBench Cube, a grasp-centred pick-and-place task close to our DROID clips with binary success: fidelity is a necessary condition for success (Fig. 15) and understates the hierarchy's larger success advantage.

AuthorsOpenReview ↗

oMBF Rebuttal Part 2

Q4, W1c - Whether Cube uses levels 3–4 during planning or benefits only from deeper training.

Correct. On Cube, levels 3–4 are not used at test time, either as planners or as cost spaces. Cube tasks have start–goal pairs separated by 20 environment steps, which a single level-3 prediction already spans (level 3 spans 20). Their planning horizons are therefore one, preventing meaningful temporal decomposition, so these levels are skipped (§E, Table 7). Level 2 instead plans directly toward the goal and supplies subgoals to level 1.

Thus, the improvement from two to three levels (47.3% → 60.0%) cannot be attributed to deeper test-time planning. Rather, it is consistent with an additional benefit of end-to-end hierarchical training: losses at higher levels propagate into the lower-level representations and predictors, potentially improving their planning effectiveness.

Table 9 provides supporting evidence: using level-1 dynamics with a level-2 projected goal cost, success improves from 13.3% (two-level model) to 28.0% (three-level model). This suggests that training additional levels can improve the lower-level models even when those additional levels are not used during planning. App. H.1 (Fig. R5) plans the same checkpoints with fewer levels.

Q5. Discrepancies in four-level results between Fig. 5 and Tables 1/9.

Thank you for catching this. It is a synchronization error, and Fig. 5 is the source of truth. We have corrected the table cells in Table 9 to match those of Fig. 5 in the revision.

Q6 Planning compute accounting (encoders and backward passes), model size, and training compute vs. LeWM.

Yes. As stated in the caption of Fig. 5, our reported planning FLOPs include both forward and backward passes. To clarify, the measurement accounts for computations across all active planning levels, including state and action encoders, latent predictors, and the gradients propagated through upper-level encoders when evaluating subgoal costs (Eq. 6). We will make this accounting explicit in the paper.

Model size. World-model parameters (encoders, predictors and action encoders), in millions:

LeWM2 levels3 levels4 levels
Visual AntMaze12.323.634.846.0
FourRoom11.316.521.726.9
OGBench Cube, Push-T11.323.235.247.1
DROID (LeWM+IDM)53.886.4––

Each added level contributes one predictor (about 5–12M parameters in simulation, 32M on DROID) and a small encoder (0.15 - 0.6M). HWM on DROID has 85.8M (Table R1, App. C.1).

Training compute. Total training FLOPs (FLOPs per optimizer step × optimizer steps of each paper run), relative to LeWM:

2 levels3 levels4 levels
Visual AntMaze×1.08×1.38×1.52
FourRoom×0.92×1.17×1.28
OGBench Cube×0.80×0.91×0.68
Push-T×0.87×0.81×0.27
DROID (LeWM+IDM)×1.002––

On DROID, all models train for 100 epochs (29,200 steps of batch 256) and the level-1 image encoder dominates compute, so level 2 adds only 0.2% (HWM: ×1.001).

W2b. DROID improvements without benefits from temporal decomposition or clear timescale separation.

We agree that DROID lacks a clear timescale separation, like our simulated manipulation tasks Cube and Push-T (single frequency band, App. D). Our new analyses of the trained DROID models (see our answer to Zho1-Q3) show where H-JEPA's improvement on DROID comes from: the separately learned level-2 representation reshapes level 1, giving a planning cost that tracks progress toward the goal more consistently and is easier to optimize than both LeWM+IDM's and HWM's level 1, and that alone retains H-JEPA's full advantage when planning with level 1 only. On DROID, the benefit therefore comes through the representation rather than through temporal decomposition.

If we addressed the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.

Thank you for response and additional experiments. The CEM comparison and frozen-encoder HWM results addresses my main concerns around baselines. I am raising my rating from 2 to 3.

My remaining concern is how much separate representations adds beyond shared hierarchy. AntMaze gives clearest evidence, on Cube gains from levels 3 and 4 come from training, and DROID still has the gain when planning with level 1 only.

Could you also clarify if DROID loss weights was chosen on separate clips? The new split seems to cover planner settings but less clear for loss weights. Regardless I now find the contribution sufficient for weak acceptance.

Reviewer_PFv1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

summary: H-JEPA jointly learns a hierarchy of action-conditioned latent world models, with each level predicting over a longer temporal interval in its own representation space. Planning proceeds top-down: higher-level predicted states become subgoals for lower-level planners, which ultimately produce executable actions. The proposed benefit is a division of labor between representations supporting detailed local dynamics and representations supporting longer-range prediction and goal matching. Evaluation covers four simulated environments and offline planning on DROID with additional inverse-dynamics supervision. The strongest result is the increase from approximately 18% to 73% success on Visual AntMaze, alongside an improved success–planning-compute tradeoff.

Critical strengths

1.The architectural motivation is clear and addresses an important planning problem. A representation suitable for detailed local dynamics need not provide a useful distance to a distant goal. Learning separate representations across temporal scales is therefore a meaningful extension of shared-space hierarchical world models. The resulting architecture provides a coherent way to combine coarse planning with fine control, without pixel reconstruction, task-specific rewards, or learned hierarchical control policies.

2.The navigation experiments establish a substantial practical benefit for the overall system. The AntMaze improvement is large, and the paper reports results across three seeds and measured planning FLOPs. The fixed-dynamics cost-projection ablation is particularly informative: changing the goal-cost representation alone can improve control, demonstrating that the learned hierarchy can contribute through the planning objective as well as through temporal decomposition. These results make the proposed approach worth investigating even though its explanatory claims require stronger validation.

Critical weaknesses

1.The central claim of a useful temporal division of labor is only partially validated. Increasing the prediction stride establishes different training timescales, but does not itself establish that higher-level representations are better suited to long-horizon action-conditioned prediction while lower levels retain the information needed for precise short-horizon control. Figures 3–4 provide relevant evidence: selected fast-varying variables become less recoverable while slower variables remain recoverable. However, fixed-capacity probe errors measure accessibility to those probes, not information loss, and a variable’s temporal frequency does not determine whether it is necessary for predicting future action effects.

2.Since the upper encoders are pointwise mappings of lower-level representations in the reported experiments, the key issue is whether they reorganize or select information in a way that improves predictive efficiency—not whether they acquire long-term information absent from lower levels. The paper does not directly evaluate this functional specialization across prediction horizons with matched predictor capacity, available history, and common targets. Table 1 supports the usefulness of higher-level goal costs, but does not establish the accuracy or necessity of the higher-level dynamics predictors. Consequently, the results support an effective hierarchical planning system more strongly than they establish the proposed mechanism of temporal abstraction.

3.The reliability of the high-to-low predictive interface, including its behavior at greater depth, remains insufficiently characterized. Higher-level predictors are trained on encoded action blocks, whereas planning optimizes latent macro-actions and asks lower-level planners to realize the resulting predicted states. Accurate prediction on recorded transitions does not automatically establish that the optimizer-generated macro-actions induce reliable predictions or that their subgoals are achievable. The paper does not directly report, at aligned physical horizons, the relationship between higher-level predicted subgoals, projected lower-level predictions, and states actually reached through execution. The collapse analysis in Appendix L addresses a different failure mode and does not resolve this issue.

4.This missing diagnosis is consequential for interpreting the depth results. On Push-T, three- and four-level models retain only 58% and 14% of the flat model’s training clips per epoch; epoch matching therefore changes training exposure and the distribution of eligible episodes (Appendix C.1, Table 3). Data scarcity is plausible, but the existing evidence does not distinguish it from loss of predictive state information, inaccurate coarse dynamics, or increasing subgoal-tracking errors. Moreover, Cube’s horizon-one upper solvers are skipped (Table 7), so trained depth and active planning depth are not consistently the same variable. Without separating these effects, the paper leaves unclear when additional levels improve planning and when the predictive hierarchy becomes unreliable. A universal theoretical guarantee is unnecessary, but targeted empirical validation of this interface is important to the central contribution.

Questions for authors

1.Additional validation—temporal specialization: Can frozen representations from different levels be evaluated on common short-, medium-, and long-horizon state-prediction targets, using matched predictor capacity, training budget, and available physical history? At the same physical prediction horizon and in a common target space, how does direct high-level prediction compare with projected low-level rollouts and a persistence baseline? This would help distinguish useful temporal specialization from easier prediction targets or metric reparameterization.

2.Clarification and additional validation—subgoal realization: What constrains optimized macro-actions relative to the encoded training-action distribution? For subgoals generated during actual planning, can you compare the higher-level prediction, the projected lower-level predicted endpoint, and the endpoint actually reached after execution? Reporting both latent tracking error and physical task progress would help identify whether failures originate in high-level prediction, lower-level optimization, or execution.

3.Additional validation—depth and failure mechanisms: Can the Push-T depth comparison control the eligible training-episode pool and optimizer-update budget? Separately, can one fixed deep checkpoint be evaluated with different numbers of active planning levels at comparable planning compute? Please explicitly identify the active levels for Cube. These checks would materially clarify whether the observed deterioration reflects insufficient training, the learned abstractions, or the composition of planning levels.

2 comments
AuthorsOpenReview ↗

PFv1 Rebuttal

Q1, W1–W2 — Temporal specialization. We agree pointwise upper encoders cannot add information; the question is whether they select it usefully. We ran both requested tests (Figs. R2–R3: https://anonymous.4open.science/r/hjepa_iclr_rebuttal-712B/).

(1) Frozen features, common targets. For each level of four-level H-JEPA, we train an identical MLP (matched budget/anchors) to predict simulator state at t+ht+h from frozen features at t−10,t−5,tt-10,t-5,t and actions through t+ht+h. NMSE (3-seed mean; full curves and LeWM/persistence baselines in R2):

TargetL1L2L3L4
AntMaze body, now / +40.028/.287.396/.282.667/.280.745/.278
FourRoom agent, +40.009.007.007.007
FourRoom distractor, now.033.087.226.376

Higher levels make fast body state and uncontrolled distractors less recoverable while preserving long-horizon predictability of task-relevant state. Identical targets, histories, actions and probes rule out easier targets or metric changes (though probe error measures accessibility, not information loss).

(2) Direct vs. projected rollouts. Using recorded actions, we unroll trained three-level predictors for 200 steps, project L1/L2 rollouts to L3, and evaluate with one L3 probe (R3). All beat persistence on controlled position (200-step NMSE: ≤.054 vs .23, AntMaze; ≤.056 vs 1.29, FourRoom). Direct L3 is not more accurate than projected L1 (FourRoom .056 vs .020; similar on AntMaze), as L1 receives full fine-grained actions rather than one 4-D macro-action per L3 step. Hierarchical training also improves L1 over flat LeWM (.046 vs .058 AntMaze; .020 vs .052 FourRoom).

Interpretation. Upper levels preserve slow, controllable features and provide coarse-step models and abstract goal costs for shorter searches; they do not outperform fine-grained rollouts with known future actions. This supports the planning mechanisms in §4.2, not intrinsically better long-range forecasts. We will revise our claims about compounding error/“long-range prediction” and add R2–R3 to the appendix.

Q2, W3 — Subgoal realizability. Macro-actions: Optimization starts at the mean training macro-action (App. E); candidates are clipped to training 2nd–98th percentiles (to be documented). Replacing optimized macro-actions with encodings of executed actions shifts L2 predictions by only 2–21% of actual motion.

Interface test: We logged every replanning call in Fig. 5 two-level evaluations (4 tasks, 3 seeds). L2→L1 subgoals and execution intervals span 10 steps. In L2 space, let z^\hat z be the predicted subgoal, pp the projected L1 planned endpoint, and rr the reached endpoint. Normalized latent errors and physical progress (success/failure episodes):

Taskz^\hat z–pppp–rrz^\hat z–rrProgress
AntMaze.03/.06.55/.81.55/.8183/54%
FourRoom.03/.31.10/.36.12/.6690/55%
Cube.02/.02.32/.29.33/.30—
Push-T.04/.07.12/.26.15/.3152/51%

Errors are normalized by latent motion over the interval. z^\hat z–pp measures L1 optimization against its model; pp–rr measures model–execution mismatch (planned actions executed unchanged); z^\hat z–rr measures realized subgoal error. Progress is the fraction of 10-step segments ending physically closer to the goal (simulator state; omitted for Cube, whose lift goal is reached only at the end). Most error arises from pp–rr, not failure to match subgoals in the L1 model, identifying L1 predictive accuracy on optimized actions as the dominant limitation. Failed AntMaze/FourRoom episodes also stall physically; Push-T progress does not separate outcomes, as the agent often repositions around the block without moving it.

Q3, W4 — Depth and failures. Matched Push-T training: Success (%) over 3 seeds:

SetupLeWM2L3L4L
Paper40.745.317.30.67
Same episode pool, clips, updates5.34.00.00.0

In the paper, 3L/4L retain only 58%/14% of LeWM's eligible clips per epoch. Matching episode pool, clips and updates makes even LeWM collapse (40.7→5.3%). This supports the under-training claim regarding Push-T 3/4 Levels.

Fixed checkpoint, variable planning depth: Evaluating the same Push-T 4L checkpoint with its lowest k=1,2,3,4k=1,2,3,4 levels yields 0.7/0/0/0% success (3 seeds). Failure persists at k=1k=1, without hierarchical subgoal composition, so it cannot be attributed solely to planning across levels.

Cube active levels: All models plan with L1–L2 only. L3–L4 generate neither subgoals nor costs: 20 steps equal one L3 prediction, giving horizon one; L2 plans directly to the goal (§E, Table 7). Thus Cube's deeper-model gains arise from training, not additional planning levels.

If we addressed the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.

Thanks for your reply. Your supplementary explanation addresses the previously mentioned issues: 1. the high-level error is approximately 2.8 times that of the low-level one, and the advantage of long-term prediction remains unsupported; 2. furthermore, the Push-T matching experiment reveals severe training confounding, which is still insufficient to establish a causal explanation for insufficient training. I highly recognize the work in other parts, so I am willing to retain my original score.

Reviewer_Zho1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H-JEPA, an end-to-end framework for learning hierarchical JEPA world models. H-JEPA learns a separate latent representation and dynamics predictor at each level through action-conditioned latent prediction, without requiring reward supervision. During planning, higher levels generate subgoals that are progressively refined into low-level actions. The authors analyze the learned representations across different levels and evaluate planning performance across four simulated navigation and manipulation environments, comparing H-JEPA with the flat LeWM and hierarchical HWM baselines. They also introduce an inverse-dynamics loss and evaluate the method on real-world robot videos from DROID using an offline trajectory-fidelity metric.

Critical strengths
  1. The paper presents a well-motivated extension of hierarchical world models by learning separate latent representations at different levels. The improvement over HWM on Visual AntMaze is substantial, suggesting that level-specific representations can provide benefits beyond temporal decomposition.
  2. The experimental analysis is relatively thorough. In addition to comparing planning performance across different environments and compute budgets, the authors include representation probing and planning-cost ablations to investigate why the method works.
Critical weaknesses
  1. The improvement over HWM is not consistent across different tasks. H-JEPA shows a substantial improvement on Visual AntMaze, but the gains are less clear on FourRoom and Push-T and vary with hierarchy depth on Cube. Although the paper provides some analysis of the learned representations, it is still unclear when and why separate latent spaces lead to better planning performance. In particular, the method may be more effective for tasks with clearly separated temporal scales, but this possibility is not sufficiently investigated.
  2. The real-world evaluation is relatively limited compared to HWM. While HWM has demonstrated closed-loop planning on a physical robot, H-JEPA is evaluated on only 16 selected DROID clips using an offline trajectory-fidelity metric. Since this metric does not directly reflect task success, it remains unclear whether the proposed hierarchical representations provide additional benefits in real-world execution. A more comparable real-robot evaluation would help establish the practical effectiveness of the proposed approach.
Questions for authors

Q1. Figures 3 and 4 show that selective abstraction is more evident in AntMaze and FourRoom, while it is less apparent in the manipulation tasks. However, Figure 5 shows that the improvement over HWM is much more pronounced on AntMaze than on FourRoom, while Cube still benefits from H-JEPA at higher hierarchy depths. Could the authors further explain how selective abstraction is related to the actual planning improvements, and under what task conditions the learned hierarchical representations are most beneficial? Q2. Section 4.2 attributes the planning improvements to both temporal decomposition and more informative high-level representations. However, Table 1 shows that, for the three-level AntMaze model, using the level 3 projected cost improves flat planning success from 16.7% to only 22.0%, while the full hierarchical planner achieves 73.3%. Could the authors further clarify how these two mechanisms interact, and whether the additional gains come from better high-level subgoal predictions rather than the cost representation alone?

Q3. Section 4.3 suggests that both temporal decomposition and hierarchical representations contribute to the improvements on DROID. However, Figure 7(b) shows only a small improvement from LeWM+IDM (34%) to HWM (35%), compared with the improvement from HWM to H-JEPA (40%). Could the authors provide further evidence that temporal decomposition contributes meaningfully in this setting, or clarify how the two effects are distinguished?

2 comments
AuthorsOpenReview ↗

Zho1 Rebuttal Part 1

We thank the reviewer for the detailed feedback. We address your questions and concerns in two comments.

Q1, W1 — Selective abstraction and planning gains across tasks

We hypothesize that distinct representations help when planning levels require different information.

In end-to-end HWM, all predictors share an encoder, potentially suppressing fast-varying details. On FourRoom, the distractor is irrelevant to both navigation and low-level control (App. B), so all levels benefit from the same abstraction. Frozen-latent probes confirm that HWM largely discards the distractor, LeWM retains it, and H-JEPA progressively discards it at higher levels. All retain agent position (NMSE ≤ 2e-4).

FourRoom distractor NMSE (3-level; 1 = mean baseline)L1L3
LeWM0.009–
H-JEPA0.0980.460
HWM0.7740.772

HWM's shared representation is thus already as abstract as H-JEPA's upper levels without sacrificing low-level control, consistent with their similar FourRoom performance (Fig. 5).

On AntMaze, however, body pose is largely irrelevant to high-level navigation but essential for low-level locomotion. HWM must accommodate both needs in one space, whereas H-JEPA preserves body pose at lower levels while abstracting it at higher levels. The same probes are consistent with this: HWM's shared space sits between the two needs, retaining less body state than LeWM or H-JEPA's level 1, yet far more than H-JEPA's level 3. All retain agent position (NMSE ≤ 2e-4).

AntMaze body-state NMSE (3-level; 1 = mean baseline)L1L3
LeWM0.085–
H-JEPA0.0650.697
HWM0.1820.182

Therefore, in the case of antmaze, HWM's shared encoder ends up learning a representation that is neither optimal for high level goal matching, nor optimal for low level control. And this is the exact scenario where having a hierarchy of representations enables H-JEPA to outperform HWM the most. Thus, separate latent spaces should help most when abstraction requirements differ across planning levels.

On Cube, where selective abstraction is weak, we lack evidence that the modest gains over HWM stem from representational specialization.

Q2, W1 - Interaction between temporal decomposition, high-level cost representations, and subgoal predictions.

Table 1 isolates the benefit of higher-level cost representations without temporal decomposition. On three-level AntMaze, projecting the flat planner's cost into the level-3 latent space improves success from 16.7% to 22.0%. However, projected flat planning without temporal decomposition still requires optimizing long-horizon level-1 rollouts (23 prediction steps), even with a more informative cost. Hierarchical planning instead decomposes the task: the high-level planner optimizes a coarse trajectory toward the goal, and each lower-level planner only needs to reach the next predicted subgoal, requiring short rollouts (2–5 prediction steps across levels). Thus, H-JEPA combines informative abstract goal representations with substantially easier, shorter-horizon optimization problems, explaining why the full hierarchical planning approach outperforms projected planning alone.

AuthorsOpenReview ↗

Zho1 Rebuttal Part 2

Q3. Temporal decomposition vs. representation on DROID.

Like Cube and Push-T (App. D), DROID shows no clear separation of timescales: end-effector position, orientation and gripper vary at similar frequencies. Two analyses indicate that H-JEPA's gain comes mainly through its level-1 representation rather than temporal decomposition.

(1) Planning with level 1 only. Fréchet fidelity (%), 16 test clips:

LeWM+IDMHWMH-JEPA
Level 1 only, flat planner34.2 ± 1.333.4 ± 1.738.9 ± 0.7
Full planner, tuned (oMBF-Q3)32.9 ± 1.232.1 ± 2.337.2 ± 0.4

Level 2 adds little in this offline setup: with no environment to replan in, level 1 plans the whole 36-step clip at once, and level-2 subgoals only add cost terms. Our simulated results, including Cube, rely on closed-loop replanning where level 1 plans only 2 steps toward the next subgoal; there, hierarchy doubles Cube success over level-1 planning (24.0% → 47.3%, App. Table 9).

(2) Geometry of the planning cost. With C(a)=∥z^H(a)−zg∥22C(a) = \|\hat z_H(a) - z_g\|_2^2 the planner's goal cost and a⋆a^\star the expert plan:

Planning spaceMonotonicity ↑Bumpiness ↓GD success ↑
LeWM+IDM, level 10.51 ± 0.0315 ± 1%83 ± 4%
HWM, level 10.51 ± 0.0113 ± 2%81 ± 1%
H-JEPA, level 10.63 ± 0.016 ± 1%87 ± 4%

Monotonicity (App. Table 10): −Spearman(t, ∥z^t(a⋆)−zg∥22)-\mathrm{Spearman}\big(t,\ \|\hat z_t(a^\star) - z_g\|_2^2\big); 1 = always decreasing. Bumpiness: TV(c~)−1\mathrm{TV}(\tilde c) - 1, with TV\mathrm{TV} the total variation (sum of absolute changes along α\alpha) and c~(α)\tilde c(\alpha) the cost C((1−α)ar+αa⋆)C\big((1-\alpha)a_r + \alpha a^\star\big) from a random plan ara_r, rescaled to [0,1][0,1]; 0 = steady descent. GD success: share of 16 gradient-descent runs with (cn−c⋆)/(c0−c⋆)≤1%(c_n - c^\star)/(c_0 - c^\star) \le 1\% (final, median initial and best cost).

H-JEPA's level-1 cost best tracks progress and is easiest to optimize, and its frozen level-1 features decode the end-effector position better (RMSE 13.0 vs 16.7/16.8 cm). This points to H-JEPA's level-2 objective (own encoder, SIGReg, inverse dynamics), which shapes level 1 through end-to-end training.

W2. Limited real-world evaluation: offline trajectory fidelity vs. closed-loop robot success.

We agree fidelity is a proxy. There was not enough time during the rebuttal period to run closed-loop experiments on a real robot, so App. J.2 relates the metric to closed-loop success in simulation, on OGBench Cube: a grasp-centred pick-and-place task close to our DROID clips, with observable binary success. There, fidelity is a necessary condition for success (no episode succeeds below a fidelity floor, Fig. 15), and the hierarchy's fidelity advantage understates its larger advantage in success. Cube thus complements DROID with a closed-loop success measurement on a very similar task. DROID planner settings are now tuned on separate validation clips (see oMBF-Q3).

If we addressed the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General rebuttal comment

We thank all reviewers for their feedback. We have uploaded a revised PDF; changes are in blue, and new figures and tables are numbered R1–R6 so the original numbering is unchanged. The anonymized code and all rebuttal figures are at https://anonymous.4open.science/r/hjepa_iclr_rebuttal-712B/README.md.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.