acceptodds
Under review as a conference paper at ICLR 2027

Representation Geometry Matters for Planning with JEPA World Models

Abstract

Joint-embedding predictive world models support planning through latent predictions, but unconstrained joint training can collapse distinct observations to identical embeddings. Two prominent strategies for avoiding collapse are to inherit pretrained features, as in DINO-WM, or to learn representations end-to-end with anti-collapse regularization, as in LeWorldModel (LeWM). Yet avoiding collapse does not ensure that latent distances distinguish outcomes in ways that matter for the task. In object manipulation, for example, success depends on the object's position and orientation relative to the goal. Such task-relevant state information can remain accurately decodable while barely influencing latent distance. The resulting planning cost may fail to reflect how close a predicted outcome is to the task goal. In this paper, we propose SCALE (State-CAlibrated Latent Embeddings), a method that correlates sampled pairwise latent distances with distances in task-relevant state space. Added to LeWM's existing objective, SCALE preserves its architecture, requires privileged state only during training, and adds no planning-time computation. We show that SCALE improves planning success over LeWM across manipulation and navigation tasks with multiple solvers and provide a comprehensive analysis of how SCALE reshapes representation geometry to support planning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world model, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oiAAOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that, on its own, the criterion optimized by JEPA approaches to world modeling may not lead to informative data representations for downstream tasks such as robotic planning. The authors thus propose to add a term to the loss function used during learning that favors representations for which Euclidean distance in feature space highly correlates to distance in "standardized task-relevant state space" (which from Appendix D.1 is some form of configuration space). They also add a second "Aux" term penalizing the decoding error from learned features to the corresponding configurations. This is reasonable and the proposed approach improves on LeWM on several tasks, but even the distance in configuration space is not always a good guide for planning, in the presence of obstacles for example.

Critical strengths

It is clearly true (and I would say well known) that the usual JEPA criterion, irrespective of the regularization used to prevent collapse (Sigreg, Vicreg, etc.), does not impose any constraint related to downstream tasks on the learned representation, with no a priori reason why the Euclidean distance used in most JEPA-based approaches to planning would be informative for the target tasks. It is thus reasonable to add constraints linked to these tasks. In practice, the proposed approach improves on LeWM on several tasks.

Critical weaknesses

The novelty of the proposed approach is limited, see for example Park et al., Foundation Policies with Hilbert Representations (2023) for a method attempting to have the metric in the learned feature space match an optimal value function.

The Euclidean distance in configuration space (or task-relevant state space) by itself is, however, not a strong indicator of the true path distance to the goal in a planning task: two configurations may indeed be very close to each other and yet separated by an obstacle that would prevent linking them without a long detour. In the version of planning used in the paper (trajectory optimization using sampling methods without explicit exploration), it may lead to dead ends.

Using some kind of geodesic distance or optimal value function in configuration space would be more informative, but would require solving a potentially difficult planning problem. Also, although it is possible to have such a distance correlate positively with the Euclidean distance in sensorymotor space as proposed by the authors, it is known that the two cannot in general coincide (see again Park et al.).

It is unclear whether the addition of the AUX term really improves performance. The one case in Table 2 where it appears to do so shows improvements that are within the margin of error given the corresponding error bars.

Questions for authors

Can you please elaborate on the relevance of the proposition and the theorem proven in the paper bring to the table? For example, most trajectory-optimization and planning methods I know of do not work by ranking a finite set of given trajectories, but I may be missing something.

3 comments

We thank the reviewer for the careful and constructive feedback, and for recognizing the motivation and practical benefits of introducing task-relevant constraints into JEPA representations. We have conducted additional statistical analyses and provide further explanations to address these points below.

[W1] Related Work

We sincerely thank the reviewer for pointing out this relevant work. We agree that HILP and SCALE share the high-level motivation of learning meaningful representation geometry for decision-making. However, their approaches and settings differ. HILP learns representations aligned with temporal distances through value learning, followed by policy training, whereas SCALE focuses on improving the geometry of existing visual world models for latent-space planning. In particular, SCALE can be incorporated as a lightweight training-time regularizer without changing the underlying world-model architecture or planning pipeline, potentially allowing it to be applied across different world models. We appreciate this insightful connection and will clarify the distinction and discuss HILP in our revised paper.

[W2/W3] Euclidean Geometry vs. Geodesic Distance

Thank you for this insightful comment. We fully agree that Euclidean distance in latent space does not necessarily reflect geodesic distance or the underlying environmental topology, especially when obstacles separate states that are spatially close. We appreciate the reviewer pointing this out, and we realize that we should have discussed this limitation more clearly in the paper.

We would like to briefly share our motivation for focusing on Euclidean geometry. Existing JEPA planners measure progress toward the goal using Euclidean distance in the latent space, and in many robotic control and manipulation settings, spatial distances have been shown to provide useful signals for goal-directed behavior [1, 2, 3]. With SCALE, our aim is to help this Euclidean latent geometry better reflect task-relevant physical differences, so that existing planners can benefit from it. We do not intend to suggest that Euclidean distance fully captures all task difficulty, and we agree that it has clear limitations in settings like the one the reviewer describes.

Following the reviewer's suggestion, we will revise the paper to (1) clearly state this scope and its limitations in the method and discussion sections, and (2) discuss topology-aware and reachability-based metrics as an important direction for future work. We are grateful for this suggestion, which helps make the positioning of our work clearer and more accurate.

References

[1] O. Khatib. Real-Time Obstacle Avoidance for Manipulators and Mobile Robots. The International Journal of Robotics Research, 5(1):90–98, 1986.

[2] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, S. Levine. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv:1812.00568, 2018.

[3] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine. Time-Contrastive Networks: Self-Supervised Learning from Video. ICRA, 2018.

、

AuthorsOpenReview ↗

Effectiveness of Auxiliary State Regression

[W4] Effectiveness of Auxiliary State Regression

Thank you for raising this concern. We would like to clarify that **Aux is designed as a mechanism control. Its purpose is to investigate whether making task-relevant state information more decodable is sufficient to improve planning. To further examine its effectiveness, we conducted paired significance tests using the original training runs, each evaluated on 600 held-out episodes across five planning budgets.

Table: Planning success rates (%) averaged over five planning budgets. Δ = Aux − LeWM in percentage points. Confidence intervals and p-values are computed from paired episode-level evaluations.

TaskPlannerLeWMAuxΔ95% CIp-value
Push-TCEM70.774.3+3.6[+2.4, +4.7]<0.05
Push-TiCEM63.969.1+5.2[+3.9, +6.4]<0.05
ReacherCEM71.471.4+0.0[-1.2, +1.2]1.000
ReacheriCEM78.880.0+1.2[+0.0, +2.2]<0.05
CubeCEM60.963.3+2.4[+1.2, +3.5]<0.05
CubeiCEM64.368.4+4.1[+3.0, +5.2]<0.05
Two-RoomCEM77.577.7+0.2[-0.7, +1.3]0.626
Two-RoomiCEM82.583.3+0.8[-0.1, +1.8]0.089
PointMazeCEM83.581.9-1.6[-3.0, -0.2]<0.05
PointMazeiCEM80.978.7-2.2[-3.8, -0.6]<0.05

The results confirm that Aux does not consistently improve planning: 5/10 comparisons show significant improvements. Importantly, our probing results show that Aux makes task-relevant state highly decodable, while its latent–state distance alignment remains close to LeWM (Appendix D.2). This supports our central observation that state decodability alone does not guarantee useful planning geometry, motivating SCALE's explicit distance calibration.

AuthorsOpenReview ↗

Proposition 4.1 and Theorem 5.1

[Q1] Relevance of Proposition 4.1 and Theorem 5.1

Thank you for this question. We appreciate the opportunity to clarify the motivation and scope of our theoretical results.

Proposition 4.1 establishes a fundamental distinction between state decodability and the geometry used for planning. Even when task-relevant state can be perfectly recovered from the latent representation, it may contribute arbitrarily little to the Euclidean distance used by the planner. This motivates explicitly calibrating latent geometry beyond improving state recoverability.

Theorem 5.1 bounds how much worse the candidate selected by latent-space planning can be, in task-state distance to the goal, than the best candidate in the same set. The bound separates two error sources: imperfect correlation between latent and task-state distances, and rollout prediction error. Its main implication is that accurate prediction alone is not sufficient. Even with exact rollouts, poorly calibrated latent distances can lead the planner to select a suboptimal candidate. This motivates SCALE's distance-alignment objective, which directly targets the correlation term. When both terms vanish, latent planning selects the candidate with the minimum task-state cost.

Regarding the finite candidate set, we apologize that this was not clear in the paper. The reviewer is right that many trajectory-optimization methods do not rank a fixed set of trajectories. However, sampling-based planners such as CEM, iCEM, and MPPI, which are the planners used in our experiments and in prior latent world-model work, do operate this way at each iteration: they sample a finite set of action sequences, score them with the latent cost, and select the best (or elite) candidates to update the sampling distribution. Theorem 5.1 applies to each of these selection steps, without assuming the candidates are fixed in advance. We note that it does not directly cover gradient-based trajectory optimization, although the same calibration issue affects the cost landscape such methods optimize.

We thank the reviewer for raising this point. In the revision, we will state this scope more clearly, describing Theorem 5.1 as a decision-consistency result for candidate selection rather than a general optimality guarantee for trajectory optimization.

Reviewer_9CWaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEPA world models encode observations into latent representations and predict how these representations evolve under candidate action sequences. For goal-conditioned planning, candidates are ranked by the squared Euclidean distance between predicted terminal and goal embeddings. The paper argues that this requires latent distances to reflect task-relevant state differences, whereas accurate state decodability alone is insufficient. Empirically, LeWM exhibits weaker distance alignment than DINO-WM on most tasks despite accurate state recovery. Proposition 4.1 formalizes this distinction through a counterexample.

The proposed method SCALE addresses this issue by adding a regularizer to LeWM that maximizes the correlation between squared latent and task-state distances over sampled frame pairs. Half the pairs are sampled within sub-trajectories and half across episodes. Privileged simulator state is used only during training, while the architecture and planning procedure remain unchanged. Theorem 5.1 relates candidate-selection error to distance alignment and rollout prediction error.

Experiments across five manipulation and navigation tasks show improved planning performance over LeWM. An auxiliary state-regression control (Aux) demonstrates that strong state decodability does not necessarily improve distance alignment. Further analysis indicates that SCALE makes task-relevant information more influential in leading principal directions, improves latent–state distance alignment, and increases the fidelity of predicted latent candidate rankings.

Critical strengths
  1. The core distinction is real and underappreciated in the community. Separating decodability from planning utility is a useful reframing of the collapse discussion for JEPA world models, and the contrast between LeWM's low-distance alignment in Table 1 and the fact that most task-relevant state remains accurately decodable from it makes the point. The Aux control supports it further: state supervision through a learned head raises decodability while leaving alignment near LeWM's level. This holds up regardless of whether L_corr is the right remedy.

  2. The method is clear, cheap, and honestly scoped. Sec. 5 specifies the objective precisely enough to follow. Keeping LeWM's architecture fixed avoids capacity confounds, using privileged state only in training keeps the planner image-only, and the method adds no planning-time computation. The reported comparison of the compact hand-chosen state against the full simulator state is the right robustness check for a method that depends on choosing q.

Critical weaknesses
  1. Insufficient comparison with alternative state-supervision objectives. The paper's methodological contribution is one particular objective, a pairwise correlation between latent and task-state distances, yet it is compared against only one alternative that uses the same privileged state supervision. Aux is a reasonable control for whether decodability alone improves planning, but it does not aim at the same goal: as the authors note, its decoder can compensate for weak state-carrying directions by amplifying them, so nothing in that objective encourages the state to influence the latent metric. Untested is the class of alternatives that do pursue that goal by simpler means, such as regression through a constrained decoder, swept over regularization strength, or a loss under which rescaling is unproductive. The present comparison therefore supports the distinction between state decodability and distance alignment, but not the choice of the proposed objective over simpler metric-aware ones.
  1. Limited evidence for robustness. Although SCALE improves all ten reported task-solver averages, several improvements are modest, and the evaluation uses only two independent training runs per method. The reported standard deviations pool evaluation-set measurements across both runs, without separating evaluation variability from training-seed variability. Consequently, it is difficult to assess how consistently the smaller improvements would reproduce across independently trained models. Additional training seeds and uncertainty estimates for the paired performance differences would strengthen the empirical claims.
Questions for authors
  1. What happens if the state supervision is made metric-aware without the pairwise objective, for instance, by regularizing or constraining the regression head, or by reformulating the loss so that rescaling weak directions does not help?

  2. Could you ablate supervision on state components that LeWM already recovers accurately, separately from components with poor baseline recoverability? This would clarify whether SCALE's improvements are mainly associated with better recovery of previously poorly represented variables or with better use of information that was already present.

  3. Could you train additional seeds and report seed-level results, along with uncertainty estimates for the paired performance differences?

6 comments

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of the distinction between state decodability and planning utility, as well as the simplicity and efficiency of SCALE. We have conducted additional experiments and analyses to address the concerns and questions raised, as detailed below.

Response to W1/Q1: Alternative State-Supervision Objectives

We sincerely thank the reviewer for this constructive suggestion. We additionally evaluated Aux + L2, which regularizes the regression head to discourage amplification of weak state-carrying directions, with {0.01,0.1,1.0} under both CEM and iCEM.

Table 1. CEM planning success rate (%), averaged over five planning budgets.

TaskLeWMAuxL2 (.01)L2 (.1)L2 (1.0)SCALE
Push-T70.774.375.374.072.073.1
Reacher71.471.472.071.871.171.7
PointMaze83.581.984.185.881.184.9
Two-Room77.577.780.183.185.889.2
Cube60.963.363.663.761.264.1

Table 2. iCEM planning success rate (%), averaged over five planning budgets.

TaskLeWMAuxL2 (.01)L2 (.1)L2 (1.0)SCALE
Push-T63.969.169.668.766.168.3
Reacher78.880.080.080.479.681.4
PointMaze80.978.782.582.478.883.4
Two-Room82.583.385.087.389.692.7
Cube64.368.468.869.764.770.8

SCALE outperforms all tested Aux + L2 variants in 6/10 task–planner comparisons. We acknowledge that Aux + L2 remains competitive on Push-T, Reacher, and PointMaze. To better understand these task-dependent differences, we additionally evaluated a state-based JEPA, replacing the visual encoder with an MLP operating on complete simulator states.

Table 3. State-based JEPA versus image-based models. Planning success rate (%).

TaskPlannerState-basedLeWMSCALE
Push-TCEM69.470.773.1
Push-TiCEM63.163.968.3
ReacherCEM83.571.471.7
ReacheriCEM90.578.881.4
PointMazeCEM89.683.584.9
PointMazeiCEM90.480.983.4
Two-RoomCEM82.677.589.2
Two-RoomiCEM88.182.592.7
CubeCEM50.060.964.1
CubeiCEM51.664.370.8

Interestingly, the strong performance of state-based JEPA on Reacher and PointMaze suggests that planning in these relatively simple tasks is closely tied to explicit state coordinates. This may also explain why Aux + L2 can be competitive, as improving state recoverability already provides useful planning information. However, for more complex manipulation and spatially structured tasks such as Push-T, Cube, and Two-Room, directly using state representations becomes less effective, while visual representations offer additional benefits. In these settings, the key challenge is not simply recovering state information, but effectively incorporating task-relevant state information into visual latent geometry for planning. This is precisely the problem SCALE aims to address. To further examine this observation, we extended our evaluation to the more challenging Cube-Quadruple manipulation task, which involves more complex object interactions. Table 4. Additional Cube-Quadruple planning success rate (%).

PlannerLeWMAuxL2 (.01)L2 (.1)L2 (1.0)SCALE
CEM28.629.228.827.928.129.9
iCEM30.331.030.930.630.332.1

SCALE again achieves the highest success rates among the evaluated methods under both planners, further supporting the benefit of explicit state–visual geometry alignment in more challenging manipulation settings.We will clarify and discuss more on this in our revision.

AuthorsOpenReview ↗

Q2: Ablation by State Recoverability

Q2: Ablation by State Recoverability

Thank you for the suggestion. Our existing linear-probe results show that most task-state components are already highly recoverable from LeWM. We therefore selected Push-T and Cube, where certain components exhibit noticeably lower baseline recoverability, to investigate whether SCALE's gains arise from improving poorly represented information or better utilizing information already encoded.

We split the state components based on LeWM's held-out linear-probe R²:

  • Push-T: High: block position (x, y); Low: agent position (x, y) and block orientation (cos θ, sin θ).
  • Cube: High: end-effector and block positions (x, y, z); Low: end-effector yaw (cos 2ψ, sin 2ψ) and gripper opening.

We trained three SCALE variants using all components, high-recoverability components only, or low-recoverability components only, with **two independent training seeds per variant (All, High-only, and Low-only) **. Each model was evaluated using CEM and iCEM across five planning budgets (T1–T5), with 600 held-out test episodes (6 sets × 100) per seed. Table: Planning success rate (%) averaged across five planning budgets and two training seeds. Δ denotes the improvement over LeWM in percentage points.

TaskPlannerSupervisionLeWMSCALEΔp-value
Push-TCEMAll71.074.1+3.1<0.05
Push-TCEMHigh only71.071.6+0.70.275
Push-TCEMLow only71.073.5+2.6<0.05
Push-TiCEMAll65.268.0+2.7<0.05
Push-TiCEMHigh only65.266.8+1.5<0.05
Push-TiCEMLow only65.267.7+2.4<0.05
CubeCEMAll61.364.6+3.3<0.05
CubeCEMHigh only61.364.4+3.1<0.05
CubeCEMLow only61.363.0+1.7<0.05
CubeiCEMAll64.970.6+5.7<0.05
CubeiCEMHigh only64.969.7+4.8<0.05
CubeiCEMLow only64.968.5+3.6<0.05

Results. Both high- and low-recoverability supervision improve planning, with 11 of 12 comparisons statistically significant (p < 0.05). On Push-T, low-recoverability supervision yields larger gains, whereas on Cube, high-recoverability supervision produces stronger improvements. Notably, supervising only already well-recovered components improves Cube by 3.1 points with CEM and 4.8 points with iCEM. These results suggest that SCALE's improvements are not solely driven by recovering previously poorly represented state information. Better reflecting already recoverable task-relevant information in the latent geometry can also substantially improve planning performance.

AuthorsOpenReview ↗

Additional Training Seeds and Statistical Significance

Thank you for the suggestion. We added one independent training seed for both LeWM and SCALE, bringing the total to three seeds (S1, S2, S3) per method. We report seed-level results with paired 95% confidence intervals and p-values.

Table 1. Planning success rates (%) averaged over five planning budgets (T1–T5), with 600 test episodes per seed. Δ = SCALE − LeWM.

TaskPlannerSeedLeWMSCALEΔ95% CIp-value
Push-TCEMS170.573.9+3.4[+1.8, +5.0]<0.05
Push-TCEMS271.072.4+1.4[-0.2, +3.0]0.089
Push-TCEMS371.074.0+3.0[+1.4, +4.6]<0.05
Push-TiCEMS163.968.2+4.3[+2.7, +5.9]<0.05
Push-TiCEMS263.968.4+4.5[+3.0, +6.0]<0.05
Push-TiCEMS364.667.9+3.3[+1.7, +4.9]<0.05
ReacherCEMS170.671.5+0.9[-0.7, +2.8]0.269
ReacherCEMS272.371.8-0.5[-2.2, +1.3]0.651
ReacherCEMS371.271.6+0.4[-1.4, +2.2]0.697
ReacheriCEMS178.081.1+3.1[+1.6, +4.6]<0.05
ReacheriCEMS279.781.7+2.0[+0.6, +3.4]<0.05
ReacheriCEMS380.078.3-1.7[-3.2, -0.1]<0.05
CubeCEMS161.364.1+2.8[+1.3, +4.3]<0.05
CubeCEMS260.564.0+3.5[+2.0, +5.0]<0.05
CubeCEMS361.464.4+3.0[+1.6, +4.5]<0.05
CubeiCEMS165.370.5+5.2[+3.8, +6.5]<0.05
CubeiCEMS263.371.2+7.9[+6.3, +9.5]<0.05
CubeiCEMS365.771.0+5.3[+3.9, +6.8]<0.05
Two-RoomCEMS177.989.2+11.3[+9.2, +13.4]<0.05
Two-RoomCEMS277.089.2+12.2[+10.2, +14.3]<0.05
Two-RoomCEMS377.590.1+12.6[+10.5, +14.7]<0.05
Two-RoomiCEMS182.992.9+10.0[+8.1, +11.8]<0.05
Two-RoomiCEMS282.092.6+10.6[+8.8, +12.5]<0.05
Two-RoomiCEMS382.692.9+10.3[+8.4, +12.2]<0.05
PointMazeCEMS182.186.2+4.1[+2.2, +5.9]<0.05
PointMazeCEMS284.883.7-1.1[-3.2, +0.8]0.269
PointMazeCEMS381.283.4+2.2[+0.4, +4.1]<0.05
PointMazeiCEMS180.283.4+3.2[+1.2, +5.1]<0.05
PointMazeiCEMS281.683.4+1.8[-0.2, +3.8]0.084
PointMazeiCEMS376.981.6+4.7[+2.8, +6.7]<0.05

Results across all budgets. Averaged over three training seeds, SCALE improves planning success in all 10 task–planner combinations, with 9/10 improvements statistically significant(p < 0.05). At the seed level, 7/10 combinations improve consistently across all three runs.

Reviewer_3TukOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SCALE, a method built on top of LeWM. It adds a loss term that encourages pairwise distances between latent states to correlate with distances between the corresponding task-relevant ground-truth states. The paper compares with LeWM, DINO-WM, and an ablation baseline Aux which only regularizes the latents themselves to contain ground truth state information. SCALE shows improved performance compared to LeWM, and on most environments than Aux, but still falls short with DINO-WM for two of the five environments.

Critical strengths
  • The paper is well-written.
  • The problem of shaping the geometry of the latent space of JEPA world models is quite relevant.
Critical weaknesses
  • The method assumes access to ground truth state information, which JEPA world models mostly don’t assume. This makes it hard for the method to scale to natural videos as the object states, etc are usually not available, while they can be task-relevant.
  • The method also requires identifying what states are “task-relevant”. E.g. for a pick and place task, are both the robot’s proprioception and the object positions task-relevant? If so, this could mislead the latent representations since a state matching the object location but not the arm location is better than a state matching the arm location but not the object location in terms of task progress.
  • Even so, the task-relevant state distance may not be the ideal metric to align the latents on. E.g. for maze environments, the states are coordinates of the agents. However, the distance between the state vectors doesn’t correspond to the actual geodesic distance between the two states (e.g. when there’s a wall in between, the states might be close but the task distance is not)
  • The performance of the Aux ablation is comparable to that of SCALE. In Figure 2, the eigenvalue rank plot of Aux pretty much overlaps perfectly with SCALE. It raises concerns on whether the performance gain of SCALE simply comes from having access to ground truth state information, and it raises concerns on why the results in Figure 3 would be drastically different between SCALE and Aux.
  • In Figure 2, the LeWM curve is missing for most plots?
  • Since the method assumes access to ground truth state information, why not include the original states directly in the world model as inputs and outputs? How would such a state-based, or jointly state and vision-based world models perform compared to SCALE?
Questions for authors

See weaknesses above. Can you clarify why regularizing to ground truth state dist is desired while the state dist itself might not correlate well with task difficulty? Also, what would you expect if you apply SCALE on DINO-WM? Would there be any performance gains? The method itself isn’t only specific to LeWM so would be interesting to see how it interacts with other types of world models.

6 comments

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the relevance of shaping latent geometry in JEPA world models. We appreciate the concerns regarding state supervision, task relevance, and the distinction between SCALE and auxiliary regression. To address these questions, we have conducted additional experiments on state weighting, comparisons with state-based world models, and generalization to another JEPA architecture. We have also performed statistical analyses and provide further clarification of our method and its limitations below.

[W1] Availability of Ground-Truth State Information

We thank the reviewer for highlighting this important limitation. We agree that obtaining ground-truth states from natural videos can be challenging, and appreciate the opportunity to clarify the scope of our approach. In particular, 3D simulation provides a scalable setting where state information is readily available, especially for embodied AI and robotics research. We also believe that recent advances in 3D reconstruction, Gaussian splatting, and vision-language models offer promising possibilities for obtaining approximate state information from real-world videos. We appreciate this valuable suggestion and will further discuss the current limitations and potential extensions in our revision.

[W2] Identifying and Weighting Task-Relevant States

We thank the reviewer for raising this important point. We agree that not all state components are equally relevant to planning. As the reviewer pointed out, in manipulation tasks, object position and robot configuration may have different levels of importance, and treating them equally could lead to suboptimal latent geometry. To investigate state relevance, we selected more challenging manipulation tasks with higher-dimensional state information and learned state weights using a gated inverse dynamics model, which predicts actions from current states and weighted future-state changes. The learned weights then define a weighted state distance for SCALE's correlation objective. For example, on Cube, end-effector height and block height receive weights of 0.81 and 0.65, while the sine of the third robot arm joint angle receives only 0.09, suggesting that the model can distinguish behaviorally informative state variables.

We further evaluated this approach on more complex manipulation tasks: Table: CEM planning success (%)

TaskLeWMSCALESCALE (Learned Weights)
Cube67.673.975.8
Cube-Triple30.331.432.9
Scene30.734.039.3

Although these are exploratory results, they suggest that state relevance can be learned from behavior rather than relying entirely on manually specified priors. We appreciate the reviewer for raising this important question and will further discuss this direction and its potential extensions in the Discussion and Future Work sections of our revised paper.

[W3/Q1] Euclidean Geometry vs. Geodesic Distance

Thank you for this insightful comment. We fully agree that Euclidean distance in latent space does not necessarily reflect geodesic distance or the underlying environmental topology, especially when obstacles separate states that are spatially close. We appreciate the reviewer pointing this out, and we realize that we should have discussed this limitation more clearly in the paper.

We would like to briefly share our motivation for focusing on Euclidean geometry. Existing JEPA planners measure progress toward the goal using Euclidean distance in the latent space, and in many robotic control and manipulation settings, spatial distances have been shown to provide useful signals for goal-directed behavior [1, 2, 3]. With SCALE, our aim is to help this Euclidean latent geometry better reflect task-relevant physical differences, so that existing planners can benefit from it. We do not intend to suggest that Euclidean distance fully captures all task difficulty, and we agree that it has clear limitations in settings like the one the reviewer describes.

Following the reviewer's suggestion, we will revise the paper to (1) clearly state this scope and its limitations in the method and discussion sections, and (2) discuss topology-aware and reachability-based metrics as an important direction for future work. We are grateful for this suggestion, which helps make the positioning of our work clearer and more accurate.

References

[1] O. Khatib. Real-Time Obstacle Avoidance for Manipulators and Mobile Robots. The International Journal of Robotics Research, 5(1):90–98, 1986.

[2] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, S. Levine. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv:1812.00568, 2018.

[3] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine. Time-Contrastive Networks: Self-Supervised Learning from Video. ICRA, 2018.

AuthorsOpenReview ↗

SCALE vs. Aux

[W4] SCALE vs. Aux: Planning Performance and Latent Geometry

Thank you for this observation. We agree that the similar spectra of SCALE and Aux in Figure 2 deserve clarification. The spectrum shows how much each direction contributes to Euclidean distance, but not which information those directions encode. The key difference lies in the latter: Aux makes the state decodable without constraining where it is stored, so task-relevant information may reside in low-variance directions that contribute little to planning distances. In contrast, SCALE's state-distance alignment explicitly redistributes task-relevant information toward the leading directions that dominate Euclidean distance. Because this redistribution changes what the leading directions encode rather than the overall shape of the spectrum, the two methods appear similar in Figure 2, and the difference emerges only when examining the leading PCA directions. Figure 3 confirms this: although Aux decodes the state well from the full embedding (Appendix D), its leading components are less predictive of the state than SCALE's on four of five tasks, and its latent–state distance alignment remains near LeWM (Appendix D.2). To compare planning performance directly, we did significance tests between SCALE and Aux.

Planning success rates (%), averaged over five planning budgets. Δ = SCALE − Aux.

TaskPlannerAuxSCALEΔ95% CIp-value
Push-TCEM74.373.1−1.2[−2.3, −0.1]< 0.05
Push-TiCEM69.168.3−0.8[−1.8, +0.3]0.169
ReacherCEM71.471.7+0.3[−0.9, +1.5]0.652
ReacheriCEM80.081.4+1.4[+0.4, +2.5]< 0.05
CubeCEM63.364.1+0.8[−0.2, +1.7]0.118
CubeiCEM68.470.8+2.4[+1.6, +3.3]< 0.05
Two-RoomCEM77.789.2+11.5[+9.6, +13.5]< 0.05
Two-RoomiCEM83.392.7+9.4[+7.8, +11.2]< 0.05
PointMazeCEM81.984.9+3.0[+1.5, +4.6]< 0.05
PointMazeiCEM78.783.4+4.7[+3.0, +6.4]< 0.05

SCALE significantly outperforms Aux in 6 of 10 settings, while Aux is slightly better on Push-T. Since both methods use identical state supervision, this suggests SCALE's gains come not only from state information but from how it shapes the planning geometry. We will add this comparison and clarify the discussion of Figure 2 in the revision.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful comments and constructive suggestions. Their feedback has helped us strengthen the experimental evidence and clarify the contribution of SCALE. Our main additions are:

  1. Stronger comparisons and mechanism analysis. We added baselines and targeted ablations to examine whether the benefits come from improved state recoverability or better representation geometry. These results provide further support for the value of explicitly calibrating latent distances for planning.

  2. More comprehensive reliability assessment. We conducted additional independent training runs and expanded the statistical analysis. These experiments strengthen the evidence for SCALE’s improvements while making variability across runs and tasks clearer.

  3. Generalization across backbones and tasks. We extended the experiments to another world-model backbone and a more challenging manipulation task. The improvements in these additional settings support SCALE’s generalizability beyond the original backbone and tasks.

We appreciate Reviewer 9CWa’s follow-up confirming that all of their main concerns have been addressed. We thank all reviewers for helping us improve the paper and will incorporate these additions into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.