acceptodds
Withdrawn from ICLR 2027

PlanJEPA: Learning Control-Structured Representations for Latent Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning representations that support planning is essential for latent world models. However, existing representation-learning objectives focus on latent-state prediction or action recoverability and do not constrain the complexity of the control relation between actions and latent transitions, so an accurate predictor can still operate over a latent space with highly nonlinear control relations. Consequently, accurate latent prediction does not necessarily lead to successful planning, since goal costs, candidate-trajectory ranking, and gradient-based action optimization all depend on the control structure exposed by the latent space. We introduce PlanJEPA, which learns control-structured representations with a multi-horizon linear action objective. By requiring actions to be predicted through a shared linear map of current and future latent states across multiple temporal horizons, PlanJEPA encourages the encoder to organize control-relevant variations directly in the representation. We prove that the imposed linear action relation forces nontrivial control errors to induce nontrivial latent goal errors and improves the local conditioning of goal-directed planning objectives. Empirically, PlanJEPA better organizes control-relevant state changes in latent space and consistently outperforms strong latent world model baselines across closed-form control, learned policies, cross-entropy method search, and gradient-based planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Latent World Model, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qgqmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PlanJEPA, a latent world model that augments LeWorldMode (LeWM) with two auxiliary heads during training, something that is said to make the latent space easier to plan in. Alongside a standard nonlinear inverse dynamics head, PlanJEPA uses a multi-horizon linear action objective to train a linear “Plan head” that recovers the first action taken after a current latent state, given that latent state and a randomly sampled future timestep. The key idea behind it is “requiring actions to be predicted through a shared linear map of current and future latent states across multiple temporal horizons” to encourage “the encoder to organize control-relevant variations directly in the representation”.

Critical strengths
  • I like how clear the paper's main idea is, and how it presents different experiments to try to support the claim that learning a representation that can be linearly combined with a set of weights to predict an action can lead to more effective solution methods. The paper includes results for ablated methods throughout is a plus.
  • Some of the visualizations are quite nice and didactic.
  • The method uses no additional test-time computation. During pretraining, the method uses a single inverse dynamics head and a linear PLAN head.
Critical weaknesses
  • The work contextualizes itself mostly within the JEPA literature and glosses over what seem to me to be important ideas from the reinforcement learning literature. The successor representation (Dayan, 1993) and more modern variants, such as the successor measure (Touati and Ollivier, 2011), are explicitly designed to be a representation that can be linearly combined with a weight vector to recover a value function. The Laplacian representation has similar properties (Mahadevan, 2005; Gomez et al., 2024). For example, Farebrother et al. (2026) and Shehmar et al. (2026) have explicitly used these methods for planning. I would have expected to see a discussion around these ideas, including the notion of putting together in representation space two states that are close in time, while ensuring that they don’t collapse—these ideas are way older than the JEPA line of work. Ideally, some of these methods would also serve as baselines. The Forward-Backward representation line of work is also likely relevant here (Touati et al., 2023).
  • It is unclear to me how meaningful the theoretical results in Section 4 are. Assumption 1 assumes that an agent is able to perfectly predict the action taken, and this is not even in expectation. In large environments, this is bound to be infeasible, and I suspect the same is true for stochastic environments.
  • Most of the results do not seem statistically significant, undermining many of the paper's claims. When looking at Tables B.1-B.4, PlanJEPA w/o plan head performs quite similarly to PlanJEPA in most environments (the mean + std. deviation overlap). I imagine that statistical tests across four samples, while correcting for multiple comparisons, would find the results indistinguishable. Other baselines also perform quite competitively in many environments. For example, is Delta-JEPA really worse than PlanJEPA in Tables B.3 and B.4? Even in Table B.2, Delta-JEPA seems to perform worse in very few environments.
  • The discussion of the different planners and their assumptions is concise and assumes a lot of background knowledge. While this is not grounds for rejecting a paper, as it is natural to expect some familiarity with the topic, it does not necessarily make the paper easy to read. There’s a middle ground where a brief overview is provided in the paper.
  • It is unclear whether the paper reports training the frozen encoder once per environment and that the error bars come only from evaluation noise, or if they come from the representation learning process. Captions such as “mean ± std over four training seeds, each evaluated with three evaluation seeds” make it even more confusing, as it suggests the paper is aggregating different sources of variance.

References

Peter Dayan: Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Comput. 5(4): 613-624 (1993).

Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Marc G. Bellemare, Alessandro Lazaric, Ahmed Touati: Compositional Planning with Jumpy World Models. ICML 2026.

Diego Gomez, Michael Bowling, Marlos C. Machado: Proper Laplacian Representation Learning. ICLR 2024

Sridhar Mahadevan: Proto-value functions: developmental reinforcement learning. ICML 2005.

Dikshant Shehmar, Matthew Schlegel, Matthew E. Taylor, Marlos C. Machado: Laplacian Representations for Decision-Time Planning. ICML 2026.

Ahmed Touati, Yann Ollivier: Learning One Representation to Optimize All Rewards. NeurIPS 2021.

Ahmed Touati, Jérémy Rapin, Yann Ollivier: Does Zero-Shot Reinforcement Learning Exist? ICLR 2023.

Questions for authors
  • Did you tune the hyperparameters of PlanJEPA w/o plan head the same way you tuned the hyperparameters of PlanJEPA? Or did you use the best parameters found for PlanJEPA in PlanJEPA w/o plan head?
  • Do your theoretical results assume that the same U and V can satisfy the prediction across all horizons? While you have UkU_k and VkV_k in Eq. (5), later you equate them simply to UU and VV.
  • Did you measure how much Assumption 1 (Locally exact one-step inverse relation) holds in practice? Is it fair to make strong claims based on results with a linear controller when only PlanJEPA was designed to lead to a representation that can be used with a linear relationship? Isn’t it a self-fulfilling prophecy?
Reviewer_EjbaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors insist the latent representation of world model should have “control-structure”: an action happened between adjacent latent representations needs to be throughly recovered from the latents, and this relation should not be overly complex. This goal is realized by 1. IDM loss (like EB-JEPA (Terver et al.) and 2. Plan loss, where the latter is the author’s main technical novelty that makes the latent representation control-relevant: Table 2 and Fig. 4 (right) show it creating much bigger delta than IDM loss.

Critical strengths
  • The empirical performance gain looks strong, and the way plan head smoothes the latent space align with geometric space seems convincing.
  • The motivation of latent space respecting the control space is indeed valid.
Critical weaknesses
  • The main motivation comes from the observation Fig.1(a), with the explanation: “a learned goal-conditioned MLP controller clearly outperforms cross-entropy method (CEM) search through the latent dynamics, and adding an inverse-dynamics head to LeWM (LeWM+IDM) raises the MLP controller further while leaving CEM essentially unchanged”. This explanation seems very odd and wrong; CEM varies as much as MLP does, and MLP drops two out of four cases. This already breaks the author’s hypothesis of having latent space with “control-sturucture” should be beneficial, and makes the rest of the paper unconvincing.
  • Although there are theorems supporting LplanL_{\textrm{plan}} to be a regularizer to make the latent space not complex (direct quote: “limit the complexity of the control relation itself”), the theorems clearly does not show why UkU_k and VkV_k can be reduced to a single UU and VV. If there is a place that explains why this reduction is a valid choice, please point it out.
  • Minor, but Table 1 is very misleading. “w/o plan head” should come after the PlanJEPA. This makes the reader confuse that the values are from “LeWM w/o plan head”, which is very odd.
Questions for authors
  • Fig.4 means the latent structure resembles the actual physical geometry, and this should not be the one indicating the latent space respecting “task geometry”. For example, the coordinates that have a wall between them will not differ much in terms of (per se) “x”-difference, but they do differ a lot in terms of task geometry as one needs to detour all the way around the wall to reach at the other side. Is this really what PlanJEPA should induce? This actually contradicts with what it has been insisting in the paper; if PlanJEPA makes the latent space respect the task geometry, the so-called “folding” is a natural consequence since task geometry often differs from the physical geometry.
  • LplanL_{\textrm{plan}} enforces at=Uzt+Vzt+ka_t = Uz_t + Vz_{t+k} for all kk. If this holds for all kk, the ideal case that the model heads to is the case where Vzt+kVz_{t+k} is identical across all kk. This may result in either VV or zz collapsing to 0 or constant, making the theorem 1 meaningless. This concern primarily roots from the authors’ choice to use uniform UU and VV across all kk.
Reviewer_B1UhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PlanJEPA, a method for learning representations for latent planning using nonlinear inverse dynamics supervision and a shared linear objective for action prediction across multiple horizons. The objective encourages control information to be organized so that the first action can be recovered linearly from current and future representations. The paper provides conditional theoretical guarantees for action separation and local optimization conditioning, and reports improvements across several control and planning methods.

Critical strengths
  • The paper clearly distinguishes action recoverability from a representation’s suitability for planning. The proposed linear objective offers a simple way to encourage accessible control structure while retaining nonlinear forward dynamics.

  • The evaluation covers multiple controller and planner families, with particularly strong improvements in linear control. Candidate ranking analyses and predictor retraining experiments provide additional evidence supporting the contribution of the learned representation.

Critical weaknesses
  • The ablations do not consistently establish the additional benefits of multiple horizons and head sharing, while uniform horizon weighting remains insufficiently examined. It is unclear whether downweighting distant endpoints would improve prediction of the first action on trajectories involving turns or detours.

  • The evaluation does not establish performance on tasks requiring the stitching of separately recorded trajectory segments, such as OGBench’s stitch tasks. It remains unclear whether supervision from state pairs within individual trajectories generalizes to goals requiring such composition.

Questions for authors
  • Equation (3) gives equal weight to the first-action prediction losses across horizons. Consider a trajectory that must first move right to avoid an obstacle and then repeatedly move upward. The target action remains rightward, but longer-horizon endpoints increasingly reflect the subsequent upward motion. Could uniform weighting compromise first-action prediction in such cases? Would exponentially decreasing the weights with the horizon help, and have the authors compared these alternatives?

  • How would PlanJEPA handle datasets containing short trajectory segments that must be stitched together to reach a goal, such as OGBench’s stitch datasets? The multi-horizon objective uses current and future states from the same recorded trajectory, whereas evaluation may require reaching a goal through multiple segments. Can the learned representation support this composition when no recorded trajectory connects the start and goal, or would changes to training or planning be necessary?

  • The ablations support the linear plan objective, but the additional benefits of multiple horizons and head sharing appear less consistent. In Table B.6, single-horizon and shared multi-horizon variants achieve identical CEM success rates on Reacher and Cube-PG, while unshared heads achieve higher mean success for linear control on Push-T. Could the authors clarify when multiple horizons and head sharing are beneficial, and distinguish their contributions from that of the linear action-prediction constraint itself?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.