- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies goal-conditioned planning with a latent world model (LeWM), where a planner ranks candidate action sequences by the l2 distance between the predicted terminal latent and the goal latent. The authors argue that this distance is polluted by noise, partly because the SIGReg regularizer, which drives the latent marginal toward N(0, I) to prevent collapse, also promotes variance in directions that carry no signal. Their key observation is that the same regularizer supplies the structure needed to remove that noise. Under a Gauss-Markov model with stationary standard Gaussian marginals and a normal state dynamics matrix A, the one-step transition is pinned to z'|z ~ N(Az, I - AA^T), so fitting the mean dynamics A yields the noise covariance for free. Normality of A further guarantees A^t(A^t)^T = (AA^T)^t, which means the eigenbasis of the covariance is shared across all planning horizons. The method is purely post hoc. The encoder, predictor, rollout, and planner stay frozen, and only a single constrained linear matrix A is fit on trajectory data by multi-step least squares. The fit is action-free, that is, Eqs. (7) and (8) take no action as input, and A is parameterized under the structural constraints of Appendix B, an orthogonal matrix together with blocks of scaled lines and scaled rotation planes. The planning cost is then restricted to the k eigenvectors of I - AA^T with the smallest eigenvalues, with k selected on a validation seed. On six environments, the method gives large gains on the 2D navigation tasks under a harder planning setting (up to +54.7 points over the plain l2 cost, e.g. PointMaze-UMaze 42.0 to 96.7), clearly beating a random projection control, and outperforming a PCA baseline by a wide margin in the harder setting, although the average margin over PCA in the default setting is small, and matching a TD-JEPA quasimetric cost that is trained only at the head level. An ablation over the anti-collapse regularizer (SIGReg, VISReg, RDMReg, stop-grad) shows the gain decays monotonically as the latent deviates from the Gaussian target, and an ablation over the structural constraints shows normality contributes the most.
Critical strengths
- The theory is self-consistent and the method mounts onto an existing world model at essentially zero cost. The derivation that SIGReg both injects noise and provides the structure to prune it, z'|z ~ N(Az, I - AA^T), together with the result that normality fixes one eigenbasis for every horizon, makes the method a purely post hoc patch. The encoder, predictor, rollout, and planner are untouched and only one constrained linear matrix is fit. For the latent planning community this is a practical and elegant contribution, and any LeJEPA-style world model can plug it in directly.
- The ablations target the mechanistic claims directly. The regularizer ablation in Section 5.4 (SIGReg, VISReg, RDMReg, stop-grad) shows the fraction of the gap closed decaying monotonically with the Epps-Pulley Gaussianity statistic (92%, 98%, 87%, 0%), which supports the causal claim that the method relies on the Gaussian structure the regularizer imposes. The structural ablation in Section 5.5 removes the PSD, normality, motion, and multi-step constraints one at a time, and dropping normality costs the most (96.7 to 88.0 on TwoRoom and 96.7 to 90.7 on PointMaze-UMaze), exactly matching the theoretical motivation that a non-normal A has an eigenbasis that drifts with the horizon. These two experiments answer why the method works more convincingly than is typical in this line of work.
Critical weaknesses
- The action-free fit is misaligned with what planning needs, and this sets the real boundary of the method rather than being a technicality. Equations (7) and (8) marginalize over actions, so the fitted A measures which directions are predictable under the behavior policy, not which directions are action-sensitive and reachable along planned trajectories. Assumption (i) of the model is then generally false, since z alone does not contain the policy state. The paper acknowledges the fit is action-free in one sentence in Section 4.1 and never examines the issue directly, beyond a general remark in Section 5.2 and the conclusion that planning can also fail in the representation and the rollout. The consequences are visible in the results. The method shows no gain over the random control on the tasks where actions dominate the transition (PushT needs the full basis, and Reacher and OGBench-Cube show no comparable gain over the random control), and all large gains are concentrated on 2D navigation. The actual contribution is therefore much narrower than the title and abstract suggest. It is not a cost fix for latent planning in general, but a cost fix for the regime where the directions predictable under the behavior policy happen to coincide with the directions planning cares about. The authors need at least one diagnostic experiment with an action-conditioned variant to show where this boundary lies, instead of burying it inside a remark that planning can fail in three places.
- The comparison against TD-JEPA is not on equal footing, and several of the reported orderings sit inside the evaluation noise. In the original TD-JEPA the encoder, predictor, and quasimetric head are trained jointly, and the representation itself is shaped by the cost. Here only the head is trained on the frozen LeWM, and the paper then concludes the unsupervised cost is comparable to the supervised one (-0.9 averaged over tasks in the default setting and -1.7 in the harder setting). What is weakened is the baseline, so this comparison cannot support the selling point that a supervision-free cost matches a trained distance. The evaluation itself is also thin. Each task is scored on 50 episodes at a single test seed, with k* chosen on a separate set of 50 episodes. With standard deviations of one to five points, the +3.2 average gain over PCA in the default setting and the single-digit margins against TD-JEPA are not statistically meaningful, yet they are used to rank methods. The headline +54.7 is far above the noise, but much of the surrounding narrative about competitiveness with other costs is built from differences that are not.
- The core modeling assumptions receive no empirical test. Section 5.4 checks only the marginal Gaussianity through the Epps-Pulley statistic, while the model also requires joint Gaussianity of the pair (z, z'), Markovianity of the latent sequence, and normality of A. None of these is diagnosed. Table 1 has four variants on a single task, which is a very small sample on which to rest a monotonicity claim. The default all-planar layout assumes the true dynamics are well described by scaled rotation planes, and the only evidence offered is an existence ablation, with no motivation for why real contact-rich dynamics should have this form. The authors themselves say the assumptions may not hold under contact-rich dynamics, but the paper provides no quantification, such as fit residuals or block-structure diagnostics, so there is no way to predict when the method will fail.
- The framing around the regularizer overstates the result. The abstract and introduction headline the +54.7 point gain, but that number comes from three of the six tasks and only under a deliberately hardened setting with planning horizon 10 and goal offset 50. In the default setting the authors themselves report that a small number of ranks already matches the full-basis cost, and that projection onto a random basis behaves similarly on most tasks. The real increment of the method is the margin by which it pulls away from random projection on the navigation tasks in the harder setting, not a general denoising story. The current presentation hides this.
Questions for authors
- If the action, or an embedding of it, is included as a condition when fitting A, or equivalently if the action dependence of the residual is modeled, do the certain directions change on TwoRoom and PointMaze? Would such an action-conditioned fit recover the zero gains on PushT and OGBench-Cube? This directly tests whether Weakness 1 explains the boundary of applicability, and the answer would materially change my assessment. This goes well beyond a clarification request.
- What are the numbers for the TD-JEPA baseline under its original joint training setup, at the same evaluation scale? If the fully trained baseline is substantially stronger, does the claim that a supervision-free cost is comparable to it still stand?
- How stable is k* across fitting seeds and across validation seeds? Given that k* is selected on 50 validation episodes, how large is the selection variance, and does the +3.2 average gain over PCA in the default setting survive once that variance is taken into account?
5 comments
Author's Response (2/4)
"much narrower than the title and abstract suggest" and "headline the +54.7 point gain"
We agree. We will retitle the paper "Latent Projection for Latent Planning under a Gaussian World Model Approximation" to make the approximation explicit. In the revised abstract and introduction, we will drop the +54.7, since the size of the gain varies across seeds (see above), and instead state a significant improvement on 2D navigation tasks.
"If the action, or an embedding of it, is included as a condition when fitting "
Thank you for suggesting this experiment to further investigate the boundary of the action-free fit.
We fit and evaluated the requested action-conditioned model, , in two versions. In both, the cost projects onto the -dimensional least-noise subspace of the noise covariance , as ours does.
- Free : and fit jointly by least squares.
- Fixed : we fit only the action term , with fixed to our fitted , to isolate the effect of the action.
Table 3. Planning success rate (%), harder setting, on 5 test seeds. is the std over all runs: test seeds fits validation seeds for Ours and fixed , test seeds validation seeds for free , and test seeds for .
| Task | Ours | Action-cond. (free ) | Action-cond. (fixed ) | |
|---|---|---|---|---|
| TwoRoom | 56.4 4.8 | 92.8 3.5 | 91.2 1.0 | 91.0 3.4 |
| UMaze | 46.8 4.6 | 96.4 1.7 | 95.2 1.6 | 88.2 4.5 |
| Medium | 60.4 10.9 | 88.2 4.8 | 88.0 1.8 | 89.1 2.5 |
| Cube | 50.8 7.6 | 49.1 6.7 | 49.0 9.3 | 50.1 6.8 |
| Reacher | 78.8 3.0 | 77.5 5.1 | 80.1 7.9 | 76.1 5.4 |
| PushT | 33.2 6.4 | 30.7 6.5 | 34.7 8.3 | 37.1 8.2 |
- Does it recover PushT and OGBench-Cube? No. Conditioning on the action may help modestly on some manipulation and control tasks, but it gives no substantial gain.
This is consistent with what the cost needs. A good cost for MPC ranks a terminal state by how reachable the goal is from the terminal state [1, 2]. Reachability is a property of the state transition under a policy. Under the standard MDP formulation, the action is marginalized under the policy: . Thus, shaping the cost should not require an action-conditioned transition.
The code for this analysis is at https://anonymous.4open.science/r/lp2-anon-0019/.
Author's Response (3/4)
"TD-JEPA baseline under its original joint training setup"
We did not intend to weaken the TD-JEPA baseline. We trained only its head on the same frozen LeWM, so that the comparison between the two costs is not confounded by a change in representation. We will make this intention clearer in the revision.
To further investigate the gain from training the quasimetric head and the world model jointly, we also trained TD-JEPA as in the original (encoder, predictor and head; upstream code, unmodified) and evaluated it under the same protocol as above.
Table 4. Default setting. Planning success rate (%) on 5 test seeds. Ours: mean std over 5 fits 5 validation seeds 5 test seeds. Frozen: TD-JEPA head on the frozen LeWM, mean std over 5 fits 5 test seeds. and Joint: mean std over the 5 test seeds. is the best per task.
| Task | Ours | Frozen | Joint | |
|---|---|---|---|---|
| TwoRoom | 86.8 3.6 | 99.4 1.5 | 99.8 0.6 | 98.4 1.7 |
| UMaze | 82.0 3.2 | 99.9 0.4 | 100.0 0.0 | 100.0 0.0 |
| Medium | 97.2 1.1 | 98.3 1.9 | 99.5 0.9 | 99.2 1.1 |
| Cube | 64.8 9.3 | 65.3 8.6 | 71.4 7.6 | 74.0 5.8 |
| Reacher | 85.2 5.6 | 83.3 5.7 | 85.2 5.6 | 85.2 3.6 |
| PushT | 85.2 5.0 | 84.4 4.3 | 86.6 4.3 | 86.4 6.1 |
Table 5. Harder setting, otherwise as Table 4.
| Task | Ours | Frozen | Joint | |
|---|---|---|---|---|
| TwoRoom | 56.4 4.8 | 92.8 3.5 | 98.3 1.9 | 92.0 4.5 |
| UMaze | 46.8 4.6 | 96.4 1.7 | 96.0 2.4 | 92.4 5.0 |
| Medium | 60.4 10.9 | 88.2 4.8 | 90.9 3.0 | 90.0 1.4 |
| Cube | 50.8 7.6 | 49.1 6.7 | 50.8 7.6 | 49.6 6.2 |
| Reacher | 78.8 3.0 | 77.5 5.1 | 79.1 5.1 | 80.4 9.0 |
| PushT | 33.2 6.4 | 30.7 6.5 | 36.9 8.5 | 28.8 3.0 |
We do not observe a substantial improvement from joint training. Taking the best cost on each task, the frozen head performs on par with the joint model. The code for both setups is at https://anonymous.4open.science/r/lp2-anon-0019/.
Author's Response (1/4)
We thank the reviewer for the constructive review and for recognizing the method as "a practical and elegant contribution" that runs "at essentially zero cost." We address each concern below.
"How stable is k∗ across fitting seeds and across validation seeds?"
To investigate the robustness of rank selection and of the reported results, we repeated the evaluation in the harder setting with more fits and seeds.
Table 1. Number of (fit, validation seed) pairs that picked each k.
Table 2. Planning success rate (%), harder setting, on 5 test seeds. ± is the std over all runs: test seeds × fits × validation seeds for Ours, test seeds × validation seeds for PCA, test seeds × fits for TD-JEPA, and test seeds for ℓ2.
The large gains on the 2D navigation tasks hold, but the exact improvement varies across seeds. On these tasks, k∗ falls within three to four adjacent values of the sweep (8-32 on TwoRoom, 2-8 on UMaze, 4-32 on Medium).
The average margin over PCA across the six tasks is +13.1→+11.8 points in the harder setting (paper → this protocol). We ran the same protocol in the default setting, where it is +3.2→+2.9, so the gain over PCA holds in both settings.