- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper argues that, on its own, the criterion optimized by JEPA approaches to world modeling may not lead to informative data representations for downstream tasks such as robotic planning. The authors thus propose to add a term to the loss function used during learning that favors representations for which Euclidean distance in feature space highly correlates to distance in "standardized task-relevant state space" (which from Appendix D.1 is some form of configuration space). They also add a second "Aux" term penalizing the decoding error from learned features to the corresponding configurations. This is reasonable and the proposed approach improves on LeWM on several tasks, but even the distance in configuration space is not always a good guide for planning, in the presence of obstacles for example.
Critical strengths
It is clearly true (and I would say well known) that the usual JEPA criterion, irrespective of the regularization used to prevent collapse (Sigreg, Vicreg, etc.), does not impose any constraint related to downstream tasks on the learned representation, with no a priori reason why the Euclidean distance used in most JEPA-based approaches to planning would be informative for the target tasks. It is thus reasonable to add constraints linked to these tasks. In practice, the proposed approach improves on LeWM on several tasks.
Critical weaknesses
The novelty of the proposed approach is limited, see for example Park et al., Foundation Policies with Hilbert Representations (2023) for a method attempting to have the metric in the learned feature space match an optimal value function.
The Euclidean distance in configuration space (or task-relevant state space) by itself is, however, not a strong indicator of the true path distance to the goal in a planning task: two configurations may indeed be very close to each other and yet separated by an obstacle that would prevent linking them without a long detour. In the version of planning used in the paper (trajectory optimization using sampling methods without explicit exploration), it may lead to dead ends.
Using some kind of geodesic distance or optimal value function in configuration space would be more informative, but would require solving a potentially difficult planning problem. Also, although it is possible to have such a distance correlate positively with the Euclidean distance in sensorymotor space as proposed by the authors, it is known that the two cannot in general coincide (see again Park et al.).
It is unclear whether the addition of the AUX term really improves performance. The one case in Table 2 where it appears to do so shows improvements that are within the margin of error given the corresponding error bars.
Questions for authors
Can you please elaborate on the relevance of the proposition and the theorem proven in the paper bring to the table? For example, most trajectory-optimization and planning methods I know of do not work by ranking a finite set of given trajectories, but I may be missing something.
3 comments
Effectiveness of Auxiliary State Regression
[W4] Effectiveness of Auxiliary State Regression
Thank you for raising this concern. We would like to clarify that **Aux is designed as a mechanism control. Its purpose is to investigate whether making task-relevant state information more decodable is sufficient to improve planning. To further examine its effectiveness, we conducted paired significance tests using the original training runs, each evaluated on 600 held-out episodes across five planning budgets.
Table: Planning success rates (%) averaged over five planning budgets. Δ = Aux − LeWM in percentage points. Confidence intervals and p-values are computed from paired episode-level evaluations.
| Task | Planner | LeWM | Aux | Δ | 95% CI | p-value |
|---|---|---|---|---|---|---|
| Push-T | CEM | 70.7 | 74.3 | +3.6 | [+2.4, +4.7] | <0.05 |
| Push-T | iCEM | 63.9 | 69.1 | +5.2 | [+3.9, +6.4] | <0.05 |
| Reacher | CEM | 71.4 | 71.4 | +0.0 | [-1.2, +1.2] | 1.000 |
| Reacher | iCEM | 78.8 | 80.0 | +1.2 | [+0.0, +2.2] | <0.05 |
| Cube | CEM | 60.9 | 63.3 | +2.4 | [+1.2, +3.5] | <0.05 |
| Cube | iCEM | 64.3 | 68.4 | +4.1 | [+3.0, +5.2] | <0.05 |
| Two-Room | CEM | 77.5 | 77.7 | +0.2 | [-0.7, +1.3] | 0.626 |
| Two-Room | iCEM | 82.5 | 83.3 | +0.8 | [-0.1, +1.8] | 0.089 |
| PointMaze | CEM | 83.5 | 81.9 | -1.6 | [-3.0, -0.2] | <0.05 |
| PointMaze | iCEM | 80.9 | 78.7 | -2.2 | [-3.8, -0.6] | <0.05 |
The results confirm that Aux does not consistently improve planning: 5/10 comparisons show significant improvements. Importantly, our probing results show that Aux makes task-relevant state highly decodable, while its latent–state distance alignment remains close to LeWM (Appendix D.2). This supports our central observation that state decodability alone does not guarantee useful planning geometry, motivating SCALE's explicit distance calibration.
Proposition 4.1 and Theorem 5.1
[Q1] Relevance of Proposition 4.1 and Theorem 5.1
Thank you for this question. We appreciate the opportunity to clarify the motivation and scope of our theoretical results.
Proposition 4.1 establishes a fundamental distinction between state decodability and the geometry used for planning. Even when task-relevant state can be perfectly recovered from the latent representation, it may contribute arbitrarily little to the Euclidean distance used by the planner. This motivates explicitly calibrating latent geometry beyond improving state recoverability.
Theorem 5.1 bounds how much worse the candidate selected by latent-space planning can be, in task-state distance to the goal, than the best candidate in the same set. The bound separates two error sources: imperfect correlation between latent and task-state distances, and rollout prediction error. Its main implication is that accurate prediction alone is not sufficient. Even with exact rollouts, poorly calibrated latent distances can lead the planner to select a suboptimal candidate. This motivates SCALE's distance-alignment objective, which directly targets the correlation term. When both terms vanish, latent planning selects the candidate with the minimum task-state cost.
Regarding the finite candidate set, we apologize that this was not clear in the paper. The reviewer is right that many trajectory-optimization methods do not rank a fixed set of trajectories. However, sampling-based planners such as CEM, iCEM, and MPPI, which are the planners used in our experiments and in prior latent world-model work, do operate this way at each iteration: they sample a finite set of action sequences, score them with the latent cost, and select the best (or elite) candidates to update the sampling distribution. Theorem 5.1 applies to each of these selection steps, without assuming the candidates are fixed in advance. We note that it does not directly cover gradient-based trajectory optimization, although the same calibration issue affects the cost landscape such methods optimize.
We thank the reviewer for raising this point. In the revision, we will state this scope more clearly, describing Theorem 5.1 as a decision-consistency result for candidate selection rather than a general optimality guarantee for trajectory optimization.
response
We thank the reviewer for the careful and constructive feedback, and for recognizing the motivation and practical benefits of introducing task-relevant constraints into JEPA representations. We have conducted additional statistical analyses and provide further explanations to address these points below.
[W1] Related Work
We sincerely thank the reviewer for pointing out this relevant work. We agree that HILP and SCALE share the high-level motivation of learning meaningful representation geometry for decision-making. However, their approaches and settings differ. HILP learns representations aligned with temporal distances through value learning, followed by policy training, whereas SCALE focuses on improving the geometry of existing visual world models for latent-space planning. In particular, SCALE can be incorporated as a lightweight training-time regularizer without changing the underlying world-model architecture or planning pipeline, potentially allowing it to be applied across different world models. We appreciate this insightful connection and will clarify the distinction and discuss HILP in our revised paper.
[W2/W3] Euclidean Geometry vs. Geodesic Distance
Thank you for this insightful comment. We fully agree that Euclidean distance in latent space does not necessarily reflect geodesic distance or the underlying environmental topology, especially when obstacles separate states that are spatially close. We appreciate the reviewer pointing this out, and we realize that we should have discussed this limitation more clearly in the paper.
We would like to briefly share our motivation for focusing on Euclidean geometry. Existing JEPA planners measure progress toward the goal using Euclidean distance in the latent space, and in many robotic control and manipulation settings, spatial distances have been shown to provide useful signals for goal-directed behavior [1, 2, 3]. With SCALE, our aim is to help this Euclidean latent geometry better reflect task-relevant physical differences, so that existing planners can benefit from it. We do not intend to suggest that Euclidean distance fully captures all task difficulty, and we agree that it has clear limitations in settings like the one the reviewer describes.
Following the reviewer's suggestion, we will revise the paper to (1) clearly state this scope and its limitations in the method and discussion sections, and (2) discuss topology-aware and reachability-based metrics as an important direction for future work. We are grateful for this suggestion, which helps make the positioning of our work clearer and more accurate.
References
[1] O. Khatib. Real-Time Obstacle Avoidance for Manipulators and Mobile Robots. The International Journal of Robotics Research, 5(1):90–98, 1986.
[2] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, S. Levine. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv:1812.00568, 2018.
[3] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine. Time-Contrastive Networks: Self-Supervised Learning from Video. ICRA, 2018.
、