acceptodds
Under review as a conference paper at ICLR 2027

Latent Projection for Latent Planning

Abstract

Latent world models let an agent plan in latent space. The planner searches for an action sequence that minimizes a cost, and that cost is often the distance between the predicted terminal latent and the goal latent. The latent, however, is partly noise: the regularizer that prevents collapse during world model training promotes variance in the directions carrying no signal. We show that the same regularizer also supplies the structure to prune that noise in the latent. SIGReg drives the latent marginal to a standard Gaussian at every step, and assuming further that the latent sequence is a Gauss–Markov process, the transition is linear with its covariance pinned by the map itself, . Fitting the state dynamics therefore yields the covariance at no additional cost, and in its eigenbasis each latent direction carries its own uncertainty. Furthermore, constraining to be a normal matrix fixes that basis across timesteps, so one eigenbasis measures the uncertainty over an arbitrary number of steps and serves as a stable basis to compute the planning cost on. We then restrict the cost to the -dimensional subspace the state dynamics carry forward with the least uncertainty. The correction is post hoc: the encoder, the predictor, the rollout, and the planner are all left untouched, and only a single constrained matrix, , is fit. We evaluate across six environments and show large gains on the 2D navigation tasks, TwoRoom and PointMaze, raising success rate over the plain cost by up to points under a harder planning setting.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, world model, planning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_tc4eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies goal-conditioned planning with a latent world model (LeWM), where a planner ranks candidate action sequences by the l2 distance between the predicted terminal latent and the goal latent. The authors argue that this distance is polluted by noise, partly because the SIGReg regularizer, which drives the latent marginal toward N(0, I) to prevent collapse, also promotes variance in directions that carry no signal. Their key observation is that the same regularizer supplies the structure needed to remove that noise. Under a Gauss-Markov model with stationary standard Gaussian marginals and a normal state dynamics matrix A, the one-step transition is pinned to z'|z ~ N(Az, I - AA^T), so fitting the mean dynamics A yields the noise covariance for free. Normality of A further guarantees A^t(A^t)^T = (AA^T)^t, which means the eigenbasis of the covariance is shared across all planning horizons. The method is purely post hoc. The encoder, predictor, rollout, and planner stay frozen, and only a single constrained linear matrix A is fit on trajectory data by multi-step least squares. The fit is action-free, that is, Eqs. (7) and (8) take no action as input, and A is parameterized under the structural constraints of Appendix B, an orthogonal matrix together with blocks of scaled lines and scaled rotation planes. The planning cost is then restricted to the k eigenvectors of I - AA^T with the smallest eigenvalues, with k selected on a validation seed. On six environments, the method gives large gains on the 2D navigation tasks under a harder planning setting (up to +54.7 points over the plain l2 cost, e.g. PointMaze-UMaze 42.0 to 96.7), clearly beating a random projection control, and outperforming a PCA baseline by a wide margin in the harder setting, although the average margin over PCA in the default setting is small, and matching a TD-JEPA quasimetric cost that is trained only at the head level. An ablation over the anti-collapse regularizer (SIGReg, VISReg, RDMReg, stop-grad) shows the gain decays monotonically as the latent deviates from the Gaussian target, and an ablation over the structural constraints shows normality contributes the most.

Critical strengths
  1. The theory is self-consistent and the method mounts onto an existing world model at essentially zero cost. The derivation that SIGReg both injects noise and provides the structure to prune it, z'|z ~ N(Az, I - AA^T), together with the result that normality fixes one eigenbasis for every horizon, makes the method a purely post hoc patch. The encoder, predictor, rollout, and planner are untouched and only one constrained linear matrix is fit. For the latent planning community this is a practical and elegant contribution, and any LeJEPA-style world model can plug it in directly.
  2. The ablations target the mechanistic claims directly. The regularizer ablation in Section 5.4 (SIGReg, VISReg, RDMReg, stop-grad) shows the fraction of the gap closed decaying monotonically with the Epps-Pulley Gaussianity statistic (92%, 98%, 87%, 0%), which supports the causal claim that the method relies on the Gaussian structure the regularizer imposes. The structural ablation in Section 5.5 removes the PSD, normality, motion, and multi-step constraints one at a time, and dropping normality costs the most (96.7 to 88.0 on TwoRoom and 96.7 to 90.7 on PointMaze-UMaze), exactly matching the theoretical motivation that a non-normal A has an eigenbasis that drifts with the horizon. These two experiments answer why the method works more convincingly than is typical in this line of work.
Critical weaknesses
  1. The action-free fit is misaligned with what planning needs, and this sets the real boundary of the method rather than being a technicality. Equations (7) and (8) marginalize over actions, so the fitted A measures which directions are predictable under the behavior policy, not which directions are action-sensitive and reachable along planned trajectories. Assumption (i) of the model is then generally false, since z alone does not contain the policy state. The paper acknowledges the fit is action-free in one sentence in Section 4.1 and never examines the issue directly, beyond a general remark in Section 5.2 and the conclusion that planning can also fail in the representation and the rollout. The consequences are visible in the results. The method shows no gain over the random control on the tasks where actions dominate the transition (PushT needs the full basis, and Reacher and OGBench-Cube show no comparable gain over the random control), and all large gains are concentrated on 2D navigation. The actual contribution is therefore much narrower than the title and abstract suggest. It is not a cost fix for latent planning in general, but a cost fix for the regime where the directions predictable under the behavior policy happen to coincide with the directions planning cares about. The authors need at least one diagnostic experiment with an action-conditioned variant to show where this boundary lies, instead of burying it inside a remark that planning can fail in three places.
  2. The comparison against TD-JEPA is not on equal footing, and several of the reported orderings sit inside the evaluation noise. In the original TD-JEPA the encoder, predictor, and quasimetric head are trained jointly, and the representation itself is shaped by the cost. Here only the head is trained on the frozen LeWM, and the paper then concludes the unsupervised cost is comparable to the supervised one (-0.9 averaged over tasks in the default setting and -1.7 in the harder setting). What is weakened is the baseline, so this comparison cannot support the selling point that a supervision-free cost matches a trained distance. The evaluation itself is also thin. Each task is scored on 50 episodes at a single test seed, with k* chosen on a separate set of 50 episodes. With standard deviations of one to five points, the +3.2 average gain over PCA in the default setting and the single-digit margins against TD-JEPA are not statistically meaningful, yet they are used to rank methods. The headline +54.7 is far above the noise, but much of the surrounding narrative about competitiveness with other costs is built from differences that are not.
  3. The core modeling assumptions receive no empirical test. Section 5.4 checks only the marginal Gaussianity through the Epps-Pulley statistic, while the model also requires joint Gaussianity of the pair (z, z'), Markovianity of the latent sequence, and normality of A. None of these is diagnosed. Table 1 has four variants on a single task, which is a very small sample on which to rest a monotonicity claim. The default all-planar layout assumes the true dynamics are well described by scaled rotation planes, and the only evidence offered is an existence ablation, with no motivation for why real contact-rich dynamics should have this form. The authors themselves say the assumptions may not hold under contact-rich dynamics, but the paper provides no quantification, such as fit residuals or block-structure diagnostics, so there is no way to predict when the method will fail.
  4. The framing around the regularizer overstates the result. The abstract and introduction headline the +54.7 point gain, but that number comes from three of the six tasks and only under a deliberately hardened setting with planning horizon 10 and goal offset 50. In the default setting the authors themselves report that a small number of ranks already matches the full-basis cost, and that projection onto a random basis behaves similarly on most tasks. The real increment of the method is the margin by which it pulls away from random projection on the navigation tasks in the harder setting, not a general denoising story. The current presentation hides this.
Questions for authors
  1. If the action, or an embedding of it, is included as a condition when fitting A, or equivalently if the action dependence of the residual is modeled, do the certain directions change on TwoRoom and PointMaze? Would such an action-conditioned fit recover the zero gains on PushT and OGBench-Cube? This directly tests whether Weakness 1 explains the boundary of applicability, and the answer would materially change my assessment. This goes well beyond a clarification request.
  2. What are the numbers for the TD-JEPA baseline under its original joint training setup, at the same evaluation scale? If the fully trained baseline is substantially stronger, does the claim that a supervision-free cost is comparable to it still stand?
  3. How stable is k* across fitting seeds and across validation seeds? Given that k* is selected on 50 validation episodes, how large is the selection variance, and does the +3.2 average gain over PCA in the default setting survive once that variance is taken into account?
5 comments
AuthorsOpenReview ↗

Author's Response (1/4)

We thank the reviewer for the constructive review and for recognizing the method as "a practical and elegant contribution" that runs "at essentially zero cost." We address each concern below.

"How stable is k∗k^{\ast} across fitting seeds and across validation seeds?"

To investigate the robustness of rank selection and of the reported results, we repeated the evaluation in the harder setting with more fits and seeds.

  • 5 independent fits of AA and 5 validation seeds (0, 10, 20, 30, 40).
  • For each of the (5 fits ×\times 5 validation seeds) pairs, kk is selected following Section 5.1.
  • Each selected kk is evaluated on 5 different test seeds (1, 11, 21, 31, 41); mean ±\pm std over all runs reported.

Table 1. Number of (fit, validation seed) pairs that picked each kk.

Task248163264128192
TwoRoom⋅\cdot⋅\cdot9115⋅\cdot⋅\cdot⋅\cdot
UMaze10114⋅\cdot⋅\cdot⋅\cdot⋅\cdot⋅\cdot
Medium⋅\cdot54412⋅\cdot⋅\cdot⋅\cdot
Cube⋅\cdot585331⋅\cdot
Reacher⋅\cdot1845421
PushT⋅\cdot⋅\cdot⋅\cdot143116

Table 2. Planning success rate (%), harder setting, on 5 test seeds. ±\pm is the std over all runs: test seeds ×\times fits ×\times validation seeds for Ours, test seeds ×\times validation seeds for PCA, test seeds ×\times fits for TD-JEPA, and test seeds for ℓ2\ell_2.

Taskℓ2\ell_2PCATD-JEPA (frozen)Ours
TwoRoom56.4 ±\pm 4.871.4 ±\pm 8.298.3 ±\pm 1.992.8 ±\pm 3.5
UMaze46.8 ±\pm 4.663.3 ±\pm 6.996.0 ±\pm 2.496.4 ±\pm 1.7
Medium60.4 ±\pm 10.972.1 ±\pm 6.890.9 ±\pm 3.088.2 ±\pm 4.8
Cube50.8 ±\pm 7.646.6 ±\pm 5.748.6 ±\pm 8.949.1 ±\pm 6.7
Reacher78.8 ±\pm 3.079.4 ±\pm 7.777.3 ±\pm 5.477.5 ±\pm 5.1
PushT33.2 ±\pm 6.431.0 ±\pm 6.333.7 ±\pm 6.730.7 ±\pm 6.5

The large gains on the 2D navigation tasks hold, but the exact improvement varies across seeds. On these tasks, k∗k^{\ast} falls within three to four adjacent values of the sweep (8-32 on TwoRoom, 2-8 on UMaze, 4-32 on Medium).

The average margin over PCA across the six tasks is +13.1→+11.8+13.1 \to +11.8 points in the harder setting (paper →\to this protocol). We ran the same protocol in the default setting, where it is +3.2→+2.9+3.2 \to +2.9, so the gain over PCA holds in both settings.

AuthorsOpenReview ↗

Author's Response (2/4)

"much narrower than the title and abstract suggest" and "headline the +54.7 point gain"

We agree. We will retitle the paper "Latent Projection for Latent Planning under a Gaussian World Model Approximation" to make the approximation explicit. In the revised abstract and introduction, we will drop the +54.7, since the size of the gain varies across seeds (see above), and instead state a significant improvement on 2D navigation tasks.

"If the action, or an embedding of it, is included as a condition when fitting AA"

Thank you for suggesting this experiment to further investigate the boundary of the action-free fit.

We fit and evaluated the requested action-conditioned model, z′=Az+Ba+εz' = Az + Ba + \varepsilon, in two versions. In both, the cost projects onto the kk-dimensional least-noise subspace of the noise covariance Σε=Cov(ε)\Sigma_\varepsilon = \mathrm{Cov}(\varepsilon), as ours does.

  • Free AA: AA and BB fit jointly by least squares.
  • Fixed AA: we fit only the action term BaBa, with AA fixed to our fitted AA, to isolate the effect of the action.

Table 3. Planning success rate (%), harder setting, on 5 test seeds. ±\pm is the std over all runs: test seeds ×\times fits ×\times validation seeds for Ours and fixed AA, test seeds ×\times validation seeds for free AA, and test seeds for ℓ2\ell_2.

Taskℓ2\ell_2OursAction-cond. (free AA)Action-cond. (fixed AA)
TwoRoom56.4 ±\pm 4.892.8 ±\pm 3.591.2 ±\pm 1.091.0 ±\pm 3.4
UMaze46.8 ±\pm 4.696.4 ±\pm 1.795.2 ±\pm 1.688.2 ±\pm 4.5
Medium60.4 ±\pm 10.988.2 ±\pm 4.888.0 ±\pm 1.889.1 ±\pm 2.5
Cube50.8 ±\pm 7.649.1 ±\pm 6.749.0 ±\pm 9.350.1 ±\pm 6.8
Reacher78.8 ±\pm 3.077.5 ±\pm 5.180.1 ±\pm 7.976.1 ±\pm 5.4
PushT33.2 ±\pm 6.430.7 ±\pm 6.534.7 ±\pm 8.337.1 ±\pm 8.2
  • Does it recover PushT and OGBench-Cube? No. Conditioning on the action may help modestly on some manipulation and control tasks, but it gives no substantial gain.

This is consistent with what the cost needs. A good cost for MPC ranks a terminal state by how reachable the goal is from the terminal state [1, 2]. Reachability is a property of the state transition under a policy. Under the standard MDP formulation, the action is marginalized under the policy: p(z′∣z)=∫p(z′∣z,a) π(a∣z) dap(z' \mid z) = \int p(z' \mid z, a)\, \pi(a \mid z)\, da. Thus, shaping the cost should not require an action-conditioned transition.

The code for this analysis is at https://anonymous.4open.science/r/lp2-anon-0019/.

AuthorsOpenReview ↗

Author's Response (3/4)

"TD-JEPA baseline under its original joint training setup"

We did not intend to weaken the TD-JEPA baseline. We trained only its head on the same frozen LeWM, so that the comparison between the two costs is not confounded by a change in representation. We will make this intention clearer in the revision.

To further investigate the gain from training the quasimetric head and the world model jointly, we also trained TD-JEPA as in the original (encoder, predictor and head; upstream code, unmodified) and evaluated it under the same protocol as above.

Table 4. Default setting. Planning success rate (%) on 5 test seeds. Ours: mean ±\pm std over 5 fits ×\times 5 validation seeds ×\times 5 test seeds. Frozen: TD-JEPA head on the frozen LeWM, mean ±\pm std over 5 fits ×\times 5 test seeds. ℓ2\ell_2 and Joint: mean ±\pm std over the 5 test seeds. α∗\alpha^{\ast} is the best α∈{0,0.1,1}\alpha \in \lbrace 0, 0.1, 1 \rbrace per task.

Taskℓ2\ell_2OursFrozen α∗\alpha^{\ast}Joint α∗\alpha^{\ast}
TwoRoom86.8 ±\pm 3.699.4 ±\pm 1.599.8 ±\pm 0.698.4 ±\pm 1.7
UMaze82.0 ±\pm 3.299.9 ±\pm 0.4100.0 ±\pm 0.0100.0 ±\pm 0.0
Medium97.2 ±\pm 1.198.3 ±\pm 1.999.5 ±\pm 0.999.2 ±\pm 1.1
Cube64.8 ±\pm 9.365.3 ±\pm 8.671.4 ±\pm 7.674.0 ±\pm 5.8
Reacher85.2 ±\pm 5.683.3 ±\pm 5.785.2 ±\pm 5.685.2 ±\pm 3.6
PushT85.2 ±\pm 5.084.4 ±\pm 4.386.6 ±\pm 4.386.4 ±\pm 6.1

Table 5. Harder setting, otherwise as Table 4.

Taskℓ2\ell_2OursFrozen α∗\alpha^{\ast}Joint α∗\alpha^{\ast}
TwoRoom56.4 ±\pm 4.892.8 ±\pm 3.598.3 ±\pm 1.992.0 ±\pm 4.5
UMaze46.8 ±\pm 4.696.4 ±\pm 1.796.0 ±\pm 2.492.4 ±\pm 5.0
Medium60.4 ±\pm 10.988.2 ±\pm 4.890.9 ±\pm 3.090.0 ±\pm 1.4
Cube50.8 ±\pm 7.649.1 ±\pm 6.750.8 ±\pm 7.649.6 ±\pm 6.2
Reacher78.8 ±\pm 3.077.5 ±\pm 5.179.1 ±\pm 5.180.4 ±\pm 9.0
PushT33.2 ±\pm 6.430.7 ±\pm 6.536.9 ±\pm 8.528.8 ±\pm 3.0

We do not observe a substantial improvement from joint training. Taking the best cost on each task, the frozen head performs on par with the joint model. The code for both setups is at https://anonymous.4open.science/r/lp2-anon-0019/.

Reviewer_iqw7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a method for improving planning in latent world models by projecting representations onto dimensions with predictable dynamics. Under some assumptions, i.e. Gaussian and normality, the authors derive a transition covariance and use its eigenvectors to identify a low-dimensional planning subspace. The approach produces sizable improvements on several benchmarks.

Critical strengths

I think this is a well-motivated approach that requires no retraining of the underlying world model. The paper was detailed yet well-written. I could follow along fairly nicely. The improvements in planning success on TwoRoom and PointMaze are nice. The experiment were well-designed, including comparisons against PCA, random projections, and TD-JEPA, as well as several ablations. The related work section was detailed enough as far as I can tell.

Critical weaknesses

The central idea is cool but also related to other, known things, like slow feature analysis, PCA (as acknowledged in the paper), and related spectral methods. These connections deserve further discussion. It is not always clear how much the proposed constrained dynamics model contributes beyond these simpler alternatives. The main improvements are concentrated in the more simplistic navigation environments, while n manipulation and other control tasks the results are a little less convincing. This could at least be discussed a bit more. The theoretical derivation relies on Gaussianity, stationarity, and normality of the latent dynamics. These assumptions are imposed rather than established, so it'd be good to check if they are fulfilled (maybe I've msised this?).

Questions for authors

How does the method compare with slow feature analysis or other dimensionality reduction methods? Have you measured how closely the learned representations satisfy your assumptions?

2 comments
AuthorsOpenReview ↗

Author's Response (1/2)

We thank the reviewer for the constructive feedback, and in particular for bringing slow feature analysis (SFA) to our attention.

"compare with slow feature analysis or other dimensionality reduction methods?"

Our method projects the latent onto a subspace with low predictive uncertainty. SFA is closely related, as linear SFA finds the same features as predictive coding when the dynamics are Gaussian and time-reversible [1], which in our model means AA is symmetric. Predictable motion makes AA non-symmetric, and SFA penalizes it while our method does not.

To compare the two empirically, we fit linear SFA following Wiskott and Sejnowski [2] and report its planning success rate on all six tasks, with kk chosen by the same protocol as ours (Sec. 5.1). In our component ablation (Table 2 in the paper), the gain from modelling predictable motion is small (+1.4 on TwoRoom and +0.7 on UMaze), which explains why SFA performs comparably to our cost.

Table 1. Planning success rate (%), harder setting, under the same protocol as Section 5.1. Ours shows mean ±\pm std over 3 fits of AA.

MethodTwoRoomUMazeMediumCubeReacherPushT
ℓ2\ell_260.042.074.046.082.022.0
PCA54.072.076.046.084.022.0
ICA20.052.046.046.064.012.0
SFA98.096.092.052.080.022.0
Ours96.7 ±\pm 2.396.7 ±\pm 1.289.3 ±\pm 1.242.7 ±\pm 3.180.0 ±\pm 5.327.3 ±\pm 4.6
AuthorsOpenReview ↗

Author's Response (2/2)

"how closely the learned representations satisfy your assumptions?"

We recap the assumptions of the Gaussian world model and justify each below. In the revision, we tone these assumptions down and present Eq. 3, z′∣z∼N(Az,I−AA⊤)z' \mid z \sim \mathcal{N}(Az, I - AA^\top), as an approximation.

(i) Markov. We follow the common MDP formulation, where the state transition is Markovian: p(z′∣z)=∫p(z′∣z,a) π(a∣z) dap(z' \mid z) = \int p(z' \mid z, a)\, \pi(a \mid z)\, da.

(ii) Joint Gaussianity. SIGReg acts on the marginals, so joint Gaussianity of (z,z′)(z, z') does not follow directly. A necessary condition is that the residual r=z′−Azr = z' - Az is Gaussian. To test this, we score the whitened rr with SIGReg's Epps-Pulley statistic (lower is more Gaussian) at batch 128 on held-out pairs of all six tasks. We compare with a lower and an upper reference, in which z′z' is constructed differently.

  • Ours: the real z′z'.
  • Lower: z′=Az+ηz' = Az + \eta, η∼N(0,I−AA⊤)\eta \sim \mathcal{N}(0, I - AA^\top), following Eq. 3.
  • Upper: we extend the classic 1D example of a non-joint Gaussian pair under Gaussian marginals, z′=±zz' = \pm z with a random sign flip, to high dimension by replacing each singular value σi\sigma_i in z′=Azz' = Az with a random sign. For each pair, we draw one u∼U(0,1)u \sim \mathcal{U}(0, 1) and set the ii-th sign to −1-1 when u>(1+σi)/2u > (1 + \sigma_i)/2, i.e. with probability (1−σi)/2(1 - \sigma_i)/2, so that z′z' is AzAz on average, as in Lower.

Table 2. Mean ±\pm std over 3 seeds.

TwoRoomUMazeMediumCubeReacherPushT
Lower1.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.02
Ours1.05 ±\pm 0.001.23 ±\pm 0.001.39 ±\pm 0.001.49 ±\pm 0.011.17 ±\pm 0.011.28 ±\pm 0.00
Upper1.60 ±\pm 0.024.06 ±\pm 0.065.44 ±\pm 0.276.85 ±\pm 0.104.96 ±\pm 0.1116.65 ±\pm 0.26

On all six tasks, our residual stays close to the Gaussian reference. We therefore treat (ii) as an approximation.

(iii) Stationarity. SIGReg acts on the latent at every step, driving zz and z′z' toward the same marginal N(0,I)\mathcal{N}(0, I).

(iv) Normality. Our assumption should be looser, and instead posit that the state dynamics consist of approximately independent factors. Normality is the model class we use to capture the factors invariant across time horizons. The structural ablation (Table 2 in the paper) supports this choice, as removing normality costs the most (96.7→88.096.7 \to 88.0 on TwoRoom, 96.7→90.796.7 \to 90.7 on UMaze). We will revise the paper accordingly.

"manipulation and other control tasks"

We agree that our gains concentrate on 2D navigation; outside navigation, our cost is on par with ℓ2\ell_2. Other reviewers suggested our action-free fit of AA as a cause. We further tried an action-conditioned variant, which also shows no clear gain outside navigation. TD-JEPA (Sec. 5.3), a learned cost on the same world model, likewise gains only on navigation. This evidence steers us toward a bottleneck outside navigation that is not specific to our cost. We will add this discussion to the revision.

We thank the reviewer again and will incorporate these clarifications and results into the revision.

References

[1] F. Creutzig, H. Sprekeler. Predictive coding and the slowness principle: an information-theoretic approach. Neural Computation 20(4), 2008.

[2] L. Wiskott, T. J. Sejnowski. Slow feature analysis: unsupervised learning of invariances. Neural Computation 14(4), 2002.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author's Response (1/3)

We thank the reviewer for the extensive review. We address the main concerns below.

"compare with horizon-specific bases, ranks, or reachability-weighted task-aware projections?" ([C6], Q3)

This is an interesting suggestion, but a horizon-specific cost is out of the scope of this paper. In model predictive control (MPC) with latent world models, the common practice is a terminal cost, where candidate action sequences are ranked by the distance between the predicted terminal state and the goal state [1-5], as discussed in our related work (Sec. 6). Likewise, our work focuses on modifying the cost incurred at the terminal predicted state. Moreover, the suggested variant is not well defined. It needs a horizon tt for each candidate, but the remaining gap from the predicted terminal latent to the goal is neither given nor fixed. We discuss this in detail in App. A.

"improved terminal-cost ranking versus improved latent prediction or state recovery?" ([C9], Q5)

To clarify, the encoder and the predictor stay frozen, and only the planning cost changes, so latent prediction is identical with and without our method. Nor does the subspace improve state recovery, since the full latent recovers the state at least as well as the subspace does (Appendix F of the paper). The gain therefore comes from how the terminal cost ranks candidates.

Robustness of rank selection and evaluation ([C8], Q4)

To investigate the robustness of rank selection and of the reported results, we repeated the evaluation in the harder setting with more fits and seeds.

  • 5 independent fits of AA and 5 validation seeds (0, 10, 20, 30, 40).
  • For each of the (5 fits ×\times 5 validation seeds) pairs, kk is selected following Section 5.1.
  • Each selected kk is evaluated on 5 different test seeds (1, 11, 21, 31, 41); mean ±\pm std over all runs reported.

Table 1. Number of (fit, validation seed) pairs that picked each kk.

Task248163264128192
TwoRoom⋅\cdot⋅\cdot9115⋅\cdot⋅\cdot⋅\cdot
UMaze10114⋅\cdot⋅\cdot⋅\cdot⋅\cdot⋅\cdot
Medium⋅\cdot54412⋅\cdot⋅\cdot⋅\cdot
Cube⋅\cdot585331⋅\cdot
Reacher⋅\cdot1845421
PushT⋅\cdot⋅\cdot⋅\cdot143116

Table 2. Planning success rate (%) on 5 test seeds. ±\pm is the std over all runs: test seeds ×\times fits ×\times validation seeds for Ours, test seeds ×\times validation seeds for PCA, test seeds ×\times fits for TD-JEPA, and test seeds for ℓ2\ell_2.

Taskℓ2\ell_2PCATD-JEPA (frozen)Ours
TwoRoom56.4 ±\pm 4.871.4 ±\pm 8.298.3 ±\pm 1.992.8 ±\pm 3.5
UMaze46.8 ±\pm 4.663.3 ±\pm 6.996.0 ±\pm 2.496.4 ±\pm 1.7
Medium60.4 ±\pm 10.972.1 ±\pm 6.890.9 ±\pm 3.088.2 ±\pm 4.8
Cube50.8 ±\pm 7.646.6 ±\pm 5.748.6 ±\pm 8.949.1 ±\pm 6.7
Reacher78.8 ±\pm 3.079.4 ±\pm 7.777.3 ±\pm 5.477.5 ±\pm 5.1
PushT33.2 ±\pm 6.431.0 ±\pm 6.333.7 ±\pm 6.730.7 ±\pm 6.5

On the 2D navigation tasks, k∗k^* falls within three to four adjacent values of the sweep (8-32 on TwoRoom, 2-8 on UMaze, 4-32 on Medium), and the gains hold under this protocol: +27.8 to +49.6 points over ℓ2\ell_2.

AuthorsOpenReview ↗

Author's Response (2/3)

"fit and evaluate an action-conditioned transition model" ([C1], Q1)

A good cost for MPC ranks a terminal state by how reachable the goal is from the terminal state [6, 7]. Reachability is a property of the state transition under a policy. Under the standard MDP formulation, the action is marginalized under the policy: p(z′∣z)=∫p(z′∣z,a) π(a∣z) dap(z' \mid z) = \int p(z' \mid z, a)\, \pi(a \mid z)\, da. Thus, shaping the cost does not require an action-conditioned transition, even though the predictor itself is action-conditioned.

We nevertheless fit and evaluated the requested action-conditioned model, z′=Az+Ba+εz' = Az + Ba + \varepsilon, in two versions. In both, the cost projects onto the kk-dimensional least-noise subspace of the noise covariance Σε=Cov(ε)\Sigma_\varepsilon = \mathrm{Cov}(\varepsilon), as ours does.

  • Free AA: AA and BB fit jointly by least squares.
  • Fixed AA: we fit only the action term BaBa, with AA fixed to our fitted AA, to isolate the effect of the action.

We do not observe a clear gain from conditioning on the action.

Table 3. Planning success rate (%) on 5 test seeds. ±\pm is the std over all runs: test seeds ×\times fits ×\times validation seeds for Ours and fixed AA, test seeds ×\times validation seeds for free AA, and test seeds for ℓ2\ell_2.

Taskℓ2\ell_2OursAction-cond. (free AA)Action-cond. (fixed AA)
TwoRoom56.4 ±\pm 4.892.8 ±\pm 3.591.2 ±\pm 1.091.0 ±\pm 3.4
UMaze46.8 ±\pm 4.696.4 ±\pm 1.795.2 ±\pm 1.688.2 ±\pm 4.5
Medium60.4 ±\pm 10.988.2 ±\pm 4.888.0 ±\pm 1.889.1 ±\pm 2.5
Cube50.8 ±\pm 7.649.1 ±\pm 6.749.0 ±\pm 9.350.1 ±\pm 6.8
Reacher78.8 ±\pm 3.077.5 ±\pm 5.180.1 ±\pm 7.976.1 ±\pm 5.4
PushT33.2 ±\pm 6.430.7 ±\pm 6.534.7 ±\pm 8.337.1 ±\pm 8.2

TD-JEPA under joint training ([C7])

We study a method that shapes the terminal cost under a fixed world model. The original implementation of TD-JEPA, however, retrains the world model, and the supervision from its temporal-distance head also shapes the representation. To isolate the gain that comes from the cost shaping, we chose to train the distance head on the same frozen world model.

Nevertheless, we trained TD-JEPA jointly as in the original (encoder, predictor and head; upstream code, unmodified) and evaluated it in the harder setting under the same protocol as above. We do not observe a clear gain from joint training.

Table 4. Planning success rate (%) on 5 test seeds. Frozen: mean ±\pm std over 5 fits ×\times 5 test seeds. Joint: mean ±\pm std over the 5 test seeds. α∗\alpha^{\ast} is the best α∈{0,0.1,1}\alpha \in \lbrace 0, 0.1, 1 \rbrace (Sec. 5.3) per task.

TaskTD-JEPA frozen (α∗\alpha^{\ast})TD-JEPA joint (α∗\alpha^{\ast})
TwoRoom98.3 ±\pm 1.992.0 ±\pm 4.5
UMaze96.0 ±\pm 2.492.4 ±\pm 5.0
Medium90.9 ±\pm 3.090.0 ±\pm 1.4
Cube50.8 ±\pm 7.649.6 ±\pm 6.2
Reacher79.1 ±\pm 5.180.4 ±\pm 9.0
PushT36.9 ±\pm 8.528.8 ±\pm 3.0
AuthorsOpenReview ↗

Author's Response (3/3)

"Gaussianity diagnostics across all tasks" ([C2], [C4], [C12], Q2)

We first report the marginal. Following Table 1 of the paper, we score the raw zz with SIGReg's Epps-Pulley statistic (lower is more Gaussian) at batch 128 on held-out frames of all six tasks. A N(0,I)\mathcal{N}(0, I) sample scores 1.06±0.011.06 \pm 0.01 and a random-init encoder 56.1±0.256.1 \pm 0.2 on every task. For reference, the regularizers in Table 1 of the paper that do not enforce Gaussianity score 23.80 (RDMReg) and 42.63 (Stop-grad) on TwoRoom.

Table 5. Mean ±\pm std over 3 seeds.

TwoRoomUMazeMediumCubeReacherPushT
Ours2.81 ±\pm 0.112.05 ±\pm 0.031.66 ±\pm 0.021.34 ±\pm 0.011.66 ±\pm 0.011.44 ±\pm 0.01

On all six tasks, the marginal is far closer to the Gaussian reference than to the random-init encoder or to these regularizers.

SIGReg acts on the marginals, so joint Gaussianity of (z,z′)(z, z') does not follow directly. A necessary condition is that the residual r=z′−Azr = z' - Az is Gaussian. To test this, we score the whitened rr with SIGReg's Epps-Pulley statistic (lower is more Gaussian) at batch 128 on held-out pairs of all six tasks. We compare with a lower and an upper reference, in which z′z' is constructed differently.

  • Ours: the real z′z'.
  • Lower: z′=Az+ηz' = Az + \eta, η∼N(0,I−AA⊤)\eta \sim \mathcal{N}(0, I - AA^\top), following Eq. 3.
  • Upper: we extend the classic 1D example of a non-joint Gaussian pair under Gaussian marginals, z′=±zz' = \pm z with a random sign flip, to high dimension by replacing each singular value σi\sigma_i in z′=Azz' = Az with a random sign. For each pair, we draw one u∼U(0,1)u \sim \mathcal{U}(0, 1) and set the ii-th sign to −1-1 when u>(1+σi)/2u > (1 + \sigma_i)/2, i.e. with probability (1−σi)/2(1 - \sigma_i)/2, so that z′z' is AzAz on average, as in Lower.

Table 6. Mean ±\pm std over 3 seeds.

TwoRoomUMazeMediumCubeReacherPushT
Lower1.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.021.04 ±\pm 0.02
Ours1.05 ±\pm 0.001.23 ±\pm 0.001.39 ±\pm 0.001.49 ±\pm 0.011.17 ±\pm 0.011.28 ±\pm 0.00
Upper1.60 ±\pm 0.024.06 ±\pm 0.065.44 ±\pm 0.276.85 ±\pm 0.104.96 ±\pm 0.1116.65 ±\pm 0.26

On all six tasks, our residual stays close to the Gaussian reference.

Normality of AA. Our assumption should be looser, and instead posit that the dynamics consist of approximately independent factors. Normality is the model class we use to capture the factors shared across horizons. The structural ablation supports this choice, as removing normality costs the most (Table 2 in the paper).

In the revision, we present the Gaussian world model as an approximation rather than an assumption.

References

[1] G. Zhou et al. DINO-WM: World models on pre-trained visual features enable zero-shot planning. ICML 2025.

[2] M. Assran et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv 2025.

[3] A. Bar et al. Navigation world models. CVPR 2025.

[4] B. Terver et al. What drives success in physical planning with joint-embedding predictive world models? TMLR 2026.

[5] L. Maes et al. LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv 2026.

[6] M. Destrade et al. Value-guided action planning with JEPA world models. World Modeling Workshop, 2026.

[7] W. Li et al. Predictive but not plannable: RC-aux for latent world models. arXiv 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.