acceptodds
Under review as a conference paper at ICLR 2027

Latent Metrics for Planning with World Models

Abstract

Joint-Embedding Predictive Architectures (JEPAs) enable visual planning by predicting the future outcomes of candidate actions directly in latent space. However, existing JEPA planners typically evaluate candidate actions using Euclidean distance between the predicted outcome and the goal. This treats all latent directions equally, despite differences in their sensitivity to actions and in the model's prediction accuracy along them. We introduce Latent Metrics for Planning with World Models (LENS). LENS identifies the action-response fidelity of different latent directions from the action-conditioned dynamics already learned by a pretrained world model, while keeping both the encoder and predictor frozen. Specifically, LENS derives action-response fidelity from the consistency of action-conditioned variation between real and predicted futures together with the accuracy of the corresponding future predictions, and uses this fidelity to determine the relative influence of different latent directions in planning. Directions with higher action-response fidelity receive greater relative weight in the planning metric. The resulting metric is used both to evaluate predicted futures and to guide sampling-based search, without retraining the world model. Across four benchmarks, LENS improves average success rate from 85.8% to 95.8%. With candidate actions and world-model predictions fixed, it increases the average Spearman correlation between model rankings and real outcomes from 0.540 to 0.696. These results show that the action-conditioned dynamics already learned by a world model can provide not only a basis for predicting the future, but also a basis for determining the relative role of different latent directions in planning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Joint-Embedding Predictive Architectures (JEPA), Robot Learning, Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bZS3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the prediction and planning Gap in JEPA-based world models to improve goal-conditioned planning with pretrained world models. The proposed model earns a weighted distance metric based on how accurately the world model predicts action-induced changes in different latent directions in stead of using Euclidean distance between predicted future states and goals. The experiments are conducted on 4 visual control benchmarks, and LENS improves average planning success from 85.8% to 95.8% and produces better action rankings.

Critical strengths

The author makes a great point of identifying the limitation of existing world-model planning methods, and claims the foundamental differences in prediction reliability and action sensitivity.

The proposed method, LENS, improves planning without retraining the model, which makes it easy to apply to pretrained models. Also, it shows good empirical results across different world-model architectures.

Critical weaknesses

I think the action-response fidelity formulation includes some heuristic choices, and the author could provide more theoretical justification for why this particular formulation is optimal for planning.

The experiment part covers only four relatively simple visual control environments. More challenging tasks, distribution shifts, and larger scale world models would help establishing the generalizability.

As I understand, the assumption of this paper is that more predictable and action-sensitive latent directions should receive higher weights in planning. However, these directions are not necessarily more important for achieving the goal. Since the learned metric is fixed across tasks, it may overemphasize predictable but task-irrelevant features. The paper lacks experiments investigating this potential mismatch.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer bZS3 — Weakness 1

We thank the reviewer for the thoughtful feedback. We have uploaded a revised PDF. An overview is available in the Official Comment for the full submission, “Summary of Major Manuscript Revisions.” The locations below refer to the revised PDF.

Weakness 1. Theoretical Basis of Action-Response Fidelity

LENS combines two complementary principles: Action-Response Coupling emphasizes latent directions whose action associations agree between real transitions and World Model predictions. Prediction Accuracy calibrates the contribution of directions according to their prediction errors.

(1) Action-Response Coupling. From offline transitions, we estimate the real and predicted response statistics and their symmetric coupling matrix:

S=12(RrealRpred⊤+RpredRreal⊤).S=\frac12(R_{\mathrm{real}}R_{\mathrm{pred}}^{\top}+R_{\mathrm{pred}}R_{\mathrm{real}}^{\top}).

For any unit latent direction uu, the coupling score satisfies

u⊤Su=⟨Rreal⊤u,Rpred⊤u⟩u^{\top}Su=\langle R_{\mathrm{real}}^{\top}u,R_{\mathrm{pred}}^{\top}u\rangle

u⊤Su=12(∥Rreal⊤u∥22+∥Rpred⊤u∥22−∥(Rreal−Rpred)⊤u∥22)u^{\top}Su=\frac{1}{2}\left(\left\lVert R_{\mathrm{real}}^{\top}u\right\rVert_2^2+\left\lVert R_{\mathrm{pred}}^{\top}u\right\rVert_2^2-\left\lVert (R_{\mathrm{real}}-R_{\mathrm{pred}})^{\top}u\right\rVert_2^2\right)

The first two terms reflect the strengths of real and predicted action associations, while the last captures their mismatch. Thus, the score measures response strength together with real–predicted agreement.

With residualized, whitened actions satisfying E[a~a~⊤]=I\mathbb E[\tilde a\tilde a^{\top}]=I, the same score equals

u⊤Su=Ea~[(u⊤Rreala~)(u⊤Rpreda~)].u^{\top}Su=\mathbb E_{\tilde a}[(u^{\top}R_{\mathrm{real}}\tilde a)(u^{\top}R_{\mathrm{pred}}\tilde a)].

It is therefore the average product of real and predicted linear responses to the same standardized action variation. This matters for planning because candidate actions are ranked by predicted consequences. Positively aligned response directions provide a stronger link between modeled action effects and observed dynamics (Section 4, page 6 and Appendix B, pages 15–17).

(2) Positive Spectral Weighting. Let Sui=λiuiSu_i=\lambda_i u_i. Then

λi=⟨Rreal⊤ui,Rpred⊤ui⟩.\lambda_i=\langle R_{\mathrm{real}}^{\top}u_i,R_{\mathrm{pred}}^{\top}u_i\rangle.

LENS uses wi=1+βlog⁡(1+max⁡(λi,0))w_i=1+\beta\log(1+\max(\lambda_i,0)). Positive eigenvalues identify aligned responses. The unit offset retains all directions, and logarithmic compression limits domination by large modes.

We added spectral-function comparisons and a sweep of β\beta (Appendix D.2, page 24, Tables 10–11 and Appendix D.8, page 29, Table 20).

(3) Minimax Prediction-Error Calibration. Define prediction error e=zt+H−z^t+He=z_{t+H}-\hat z_{t+H} and non-centered second moment Σe=E[ee⊤]≻0\Sigma_e=\mathbb E[ee^{\top}]\succ0. We prove that

min⁡X⪰0,tr⁡(X)=dλmax⁡(Σe1/2XΣe1/2)\min_{X\succeq0,\\ \operatorname{tr}(X)=d}\lambda_{\max}(\Sigma_e^{1/2}X\Sigma_e^{1/2})

has the unique minimizer

X⋆=dΣe−1tr⁡(Σe−1)=N(Σe−1),N(X)=dXtr⁡(X).X^\star=\frac{d\Sigma_e^{-1}}{\operatorname{tr}(\Sigma_e^{-1})}=\mathcal N(\Sigma_e^{-1}),\quad \mathcal N(X)=\frac{dX}{\operatorname{tr}(X)}.

Under a fixed trace budget, trace-normalized inverse error weighting minimizes the worst-case weighted error over standardized directions. Larger-error directions receive lower relative weight. The normalization is invariant to a common positive rescaling of Σe\Sigma_e.

Theorem 1 is in Section 4 (page 5). The full proof is in Appendix A (page 14).

Together, the two components calibrate prediction reliability and emphasize action-response alignment. The added theorem justifies the existing Prediction Accuracy component. It does not alter the deployed LENS method.

AuthorsOpenReview ↗

Response to Reviewer bZS3 — Weakness 2

Weakness 2. Generalization Beyond the Original Benchmarks

We added evaluations on V-JEPA-2-AC under DROID-to-RoboCasa zero-shot transfer, OGBench Visual Scene multi-object manipulation, and Navier–Stokes fluid control. These experiments examine generalization across model scales, task complexities, and dynamical systems.

(1) Large World Model: V-JEPA-2-AC. We use the released vjepa2_ac_droid.pth.tar checkpoint with a ViT-G/16 visual encoder and a 24-layer action-conditioned predictor. Single-view 256×256 RGB images produce 256 patches of 1,408 channels; actions encode seven-dimensional relative pose/gripper commands. We freeze the encoder and predictor and fit a shared 1,408×1,408 channel metric across patches, averaging goal costs over spatial positions.

From 2,000 deduplicated DROID trajectories, 1,800 are used for fitting and 200 held out. Each window includes three actions and aligned real/predicted futures. We use five-fold trajectory-grouped cross-fitting and 100 trajectory-level bootstrap replicates.

We evaluate RoboCasa PnPCounterTop Place on 32 paired episodes (seeds 2–33). Both methods use the same goals, initial states, and CEM initialization; only the Planning Metric changes. Neither Retrieval Initialization nor Elite Reuse is used, and no RoboCasa data are used to fit the metric or train the model.

SettingConfiguration
CEM candidates / elites / iterations300 / 10 / 15
Prediction horizon3 actions
Execution / replanningFirst action for 5 environment steps
Episode budget60 steps / 12 replans
SuccessObject–target distance below 0.15 simulator units
MethodZero-shot Success
V-JEPA-2-AC + L231.25%
V-JEPA-2-AC + LENS-metric40.63%

Metric-only LENS improves zero-shot success by 9.38 percentage points, without updating the pretrained backbone or using target-domain fitting (Section 5.4, page 9; Appendix D.5, pages 26–28, Tables 16–17).

(2) OGBench Visual Scene. We also test image-goal multi-object manipulation involving a block, drawer, window, and buttons. The goal is the image from 25 steps later in the same validation trajectory; the maximum execution budget is 50 steps. Both methods share the frozen World Model, initial states, goals, and CEM settings (300 candidates / 30 elites / 30 iterations, horizon 25).

Success requires joint positional tolerance 0.10 (10 cm) and exact agreement of both button states. All conditions must hold after executing actions; time-zero successes are excluded. We use three seeds (42–44), 50 paired tasks per seed.

MethodFull-task Success
LeWM60.67%
Full LENS69.33%

Full LENS improves success by 8.67 percentage points, with gains in all three seeds. This evaluation extends the task setting to interacting objects and combined continuous/discrete goals. These Scene results are reported here as additional rebuttal evidence.

(3) Navier–Stokes Fluid Control. We test a two-dimensional boundary-control task: choosing the upper boundary's horizontal velocity to follow a target flow field. We evaluate 50 regular and 30 nonperiodic reference trajectories across three seeds, yielding 240 paired trials per method. Both methods use identical frozen dynamics models and reference trajectories. CEM uses 300 candidates / 30 elites / 30 iterations, horizon five macro-actions (two primitive transitions each), with replanning after each executed macro-action.

RMSE (lower is better); return (higher is better):

MetricLeWMFull LENS
Late Full-field RMSE0.0463400.040119
Final Full-field RMSE0.0238050.012722
Mean Full-field RMSE0.0517310.044475
Late Interior RMSE0.0047030.003988
Final Interior RMSE0.0021250.001854
Average Cumulative Return−9.6966−8.5480

LENS improves all six measures. Final Full-field RMSE falls by approximately 46.6% (0.023805 to 0.012722). The three principal metrics appear in Section 5.4 (page 10, Table 3); the additional measurements are reported above.

These experiments show improvements with a larger pretrained World Model and cross-domain manipulation, multi-object visual control, and nonlinear fluid dynamics, without retraining the respective World Models.

AuthorsOpenReview ↗

Response to Reviewer bZS3 — Weakness 3

Weakness 3. Task Relevance of Learned Directions

We added three held-out linear-probe experiments to test whether the directions emphasized by Action-Response Coupling and the final Planning Metric contain task-relevant physical information, rather than only predictable features. Full protocols are in Appendix D.4 (pages 25–26, Tables 13–15).

(1) Action Information. We project real future latent changes into four 32-dimensional subspaces: Action-Response Top-32, Bottom-32, PCA Top-32, and Random-32. Linear regression predicts the associated 25-step actions, using 256 training and 256 independent test trajectories per environment.

Held-out action decoding R² (↑):

EnvironmentResponse Top-32Bottom-32PCA Top-32Random-32
TwoRoom0.02563−0.03934−0.01784−0.01861
PushT0.20044−0.015310.121510.07606
Reacher−0.02868−0.03032−0.02815−0.03258
Cube0.13757−0.031240.098120.04462

These results indicate that the Response Top-32 subspace retains more action-related information (Table 13, page 25).

(2) Real Physical-state Changes. Using the same subspaces, we predict realized physical-state differences: agent XY displacement (TwoRoom), agent/block XY and block orientation changes (PushT), two joint-angle changes (Reacher), and object XYZ displacement (Cube).

These variables relate to task completion: TwoRoom and Cube require reaching target positions, PushT requires moving and orienting the block, and Reacher's end-effector position is determined by its joints.

Held-out physical-change decoding R² (↑):

EnvironmentResponse Top-32Bottom-32PCA Top-32Random-32
TwoRoom0.59478−0.052650.078960.13598
PushT0.289660.016790.229840.19649
Reacher0.11137−0.03099−0.02135−0.00482
Cube0.63289−0.023200.312550.18558

These results show that the selected directions retain more information about actual action-induced physical changes. Crucially, these changes are directly related to task progress, including navigation, object manipulation, and reaching. Thus, Action-Response Coupling emphasizes latent directions that reflect task-relevant physical dynamics, rather than merely predictable variations in latent representations (Table 14, page 26).

(3) Task-related Information in the Final Metric. We next analyze eigendirections of the complete LENS Planning Metric. From 20,000 real frames per environment, we predict same-frame task-state variables: agent XY (TwoRoom), agent/block pose (PushT), joint angles (Reacher), and object XYZ (Cube).

We split data by trajectory into approximately 70% train / 15% validation / 15% test, over three split seeds. Normalization and PCA use training data, while validation selects the ridge penalty. Random-subspace results average ten draws per split.

Held-out task-state decoding R² (↑):

EnvironmentLENS Top-32Bottom-32Random-32PCA Top-32
TwoRoom0.85290.04890.33520.5567
PushT0.45450.06180.30510.4072
Reacher0.76830.05480.43450.7392
Cube0.9117−0.00850.25950.2075
Avg.0.74690.03920.33360.4776

LENS Top-32 achieves the highest R² across all four environments, with an average of 0.7469, compared with 0.4776 for PCA, 0.3336 for Random, and 0.0392 for Bottom-32. Importantly, the decoded variables describe the physical states directly involved in achieving the task goals, including agent positions, object poses, and joint configurations. These findings suggest that the learned Planning Metric emphasizes latent directions containing task-relevant physical-state information. Together with the action-induced change analysis, this connects the metric's high-weight directions to both the underlying action-conditioned dynamics and the physical states relevant to goal achievement (Table 15, page 26).

(4) Fixed-candidate Decision Ordering. We also fix the action candidates and all World Model predictions and compare rankings with real outcomes. Average Spearman correlation increases from 0.540 (Euclidean) to 0.696 (LENS), +0.156, providing direct decision-level evidence (Section 5.3, page 9, Figure 3(d)). These results jointly connect the emphasized directions to task-state information and improved action ranking.

Reviewer_3E8COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the use of an augmented metric in the planning stage using World Models. The authors argue that due to the action-agnostic training of the world model, it is not guaranteed that the resulting embedding space is isotropic and thus, it stands to reason that a non-uniform metric would be better suited to order candidate actions during planning rather than something like squared error.

The experiments display promising improvements over existing baselines and the authors also provide solid ablations to validate their claims.

Critical strengths

The writing in the paper is generally clean and reader-friendly. I understand the motivation of studying the problem and the method that the authors present seems reasonable to address the issues they identify. I don't believe the observations in the paper to be unreasonable by any means, but have some doubts that I list in the weaknesses below.

Critical weaknesses

My major concern is with the lack of evidence that this work provides in its claims --- specifically that standard metrics are not well suited to planning and that embedding spaces of world models are not usually isotropic. While I could believe these claims, I find it hard to accept a paper that is based on this premise and doesn't provide any concrete evidence (empirically or theoretically) of these phenomenon. I must note that the empirical results provided in the paper seem to suggest that alternative metrics might help, I find it attribute it solely to LENS: Table 1 does not present uniform improvements under all environments. In fact, LENS is often not the best performing model.

My less major, yet important concern is the opacity of Section 4. Notation is generally abused and is sometimes never introduced or re-used (eg: What is A~\tilde{A}? It is not even clear that AA needs to be a numerical vector on which a Cov(.)Cov(.) operation can be applied. Is it okay if A is categorical, for instance?) It also assumes a fairly advanced level of linear algebra knowledge to be able to parse this section. No other section in the paper is this heavy in notation or mathematical rigor and this section sticks out awkwardly in my opinion. The accompanying text does not provide intuition for the choices made and the authors never mention why they chose this particular method of augmentation of the metric among any others.

I am also not pleased by the way in which the numerical results are presented. The authors chose to present the success rate of LENS in bold, presumably to highlight that it is their method. However. it is quite standard for bold numerals to indicate the best performing method in such tables. LENS, as I note above, is often not the best. I believe this to be misleading. Further, in the ablations that are performed in Table 3 (which I appreciate the authors for carrying out), w/o Reuse under PushT seems suspiciously high. Why this could be the case should be discussed. Elites don't seem to be as effective under the other 3 presented environments, so this outlier looks odd.

Finally, the authors point out multiple times that the encoder and dynamics predictor need not be retrained to use LENS and I definitely see this as a positive, although it would be nice to elucidate why this is a nice-to-have. However, I am curious as to the extent of improvements to be had if the world model itself was trained to be isotropic by using this metric as a potential regularizer. Why this isn't feasible/why this does not fit the scope of this paper is something I would like to see justifications for.

Questions for authors

Most of my questions/comments are around the weaknesses. I list them specifically below:

  1. Is there an easy experiment that can be had to visualize the non-isotropic nature of existing world models? Is there anything that can be used to strengthen the assumption that isotropic metrics are not well suited for planning (other than the improvements presented by your current experiments)?
  2. Section 4 could use a major re-write to shed light on the specific choices you make and any alternatives that you considered that didn't work.
  3. All the tables should be re-formatted to fit standard practices in the community of highlighting best performing methods. Any clarification/hypothesis on the PushT over-performance would be welcome.
  4. Finally, text around any real-world use cases for doing this metric augmentation post-hoc rather than finding ways to incorporate it during training would also help strengthen the case for LENS.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3E8C — Q1–Q2

We thank the reviewer for the feedback. We have revised the notation and added anisotropy analyses. The revised PDF and our Official Comment for the full submission, “Summary of Major Manuscript Revisions,” describe these changes.

Q1. Direct Evidence of Latent Anisotropy

SIGReg encourages marginal isotropy of encoder outputs, which does not imply isotropic action-conditioned responses or prediction errors. LENS addresses these latter properties.

(1) Spectral Analysis. We analyze the action-association matrix and prediction-error second moment:

Qa=RrealRreal⊤,Qe=E[ee⊤].Q_a=R_{\mathrm{real}}R_{\mathrm{real}}^{\top},\quad Q_e=\mathbb{E}[ee^{\top}].

We quantify nonuniformity using Normalized Spectral Entropy:

Hnorm=−∑ipilog⁡pilog⁡r,pi=λi∑jλj,H_{\mathrm{norm}}=-\frac{\sum_i p_i\log p_i}{\log r},\quad p_i=\frac{\lambda_i}{\sum_j\lambda_j},

and Coefficient of Variation (CV):

CV=Std⁡(λ)Mean⁡(λ).\mathrm{CV}=\frac{\operatorname{Std}(\lambda)}{\operatorname{Mean}(\lambda)}.

Lower entropy and higher CV indicate stronger spectral nonuniformity. Excluding structural zero modes gives 50 action-associated directions for TwoRoom, PushT, and Reacher, and 125 for Cube. Prediction error uses all 192 directions.

EnvironmentAction EntropyAction CVError EntropyError CV
TwoRoom0.88131.17630.73592.7955
PushT0.87071.14950.79711.7274
Reacher0.90400.89750.75432.3084
Cube0.83451.48430.78712.0046

Both spectra show clear directional nonuniformity. Figure 2 and Table 2 (Section 5.2, page 8) present the visualizations and measurements. Each dot represents an eigendirection, ordered by eigenvalue rank rather than latent-space location. Color encodes:

ci=log⁡2(λi/λˉenv).c_i=\log_2(\lambda_i/\bar\lambda_{\mathrm{env}}).

Orange denotes above-average spectral energy; blue denotes below-average energy.

(2) Held-out Prediction Errors. Using 256 analysis-training and 256 independent test trajectories per environment, we compare prediction MSE in the highest- and lowest-weight 32 directions of the fitted Prediction Accuracy precision matrix.

EnvironmentTop-32 MSE ↓Bottom-32 MSE ↓
TwoRoom0.0007450.605080
PushT0.0005770.255458
Reacher0.0001140.079772
Cube0.0002330.131553

High-weight directions have substantially lower held-out errors (Appendix D.9, page 30, Table 21). With candidate actions and World Model predictions fixed, LENS also improves average Spearman correlation with real outcomes from 0.540 to 0.696 (+0.156) (Section 5.3, page 9, Figure 3(d)).

Q2. Method Clarity, Notation, and Design Choices

We revised Sections 3–4 (pages 4–7) to define quantities consistently and organize the Method as Prediction Accuracy → Action-Response Coupling → Planning Metric.

(1) Notation and Categorical Actions. The action sequence and predicted future are:

a=(at,…,at+H−1),z^t+H=predictor⁡(zt,a).a=(a_t,\ldots,a_{t+H-1}),\quad \hat z_{t+H}=\operatorname{predictor}(z_t,a).

Here ztz_t and zgz_g denote current and goal latent representations. We use a~\tilde a for the context-residualized and whitened action vector, and rrealr_{\mathrm{real}}, rpredr_{\mathrm{pred}} for conditional latent-change residuals. The Action-Response statistics are:

Rreal=E[rreala~⊤],Rpred=E[rpreda~⊤].R_{\mathrm{real}}=\mathbb{E}[r_{\mathrm{real}}\tilde a^{\top}],\quad R_{\mathrm{pred}}=\mathbb{E}[r_{\mathrm{pred}}\tilde a^{\top}].

For categorical actions, one-hot encoding or another feature map provides numerical action features. Our experiments use continuous control.

(2) Mathematical Construction. Prediction Accuracy uses the non-centered error second moment:

Σe=E[ee⊤].\Sigma_e=\mathbb{E}[ee^{\top}].

The Minimax Calibration theorem justifies inverse-error weighting (Theorem 1, page 5; proof in Appendix A, page 14).

Action-Response Coupling combines real and predicted statistics:

S=12(RrealRpred⊤+RpredRreal⊤).S=\frac12(R_{\mathrm{real}}R_{\mathrm{pred}}^{\top}+R_{\mathrm{pred}}R_{\mathrm{real}}^{\top}).

For any unit latent direction uu:

u⊤Su=⟨Rreal⊤u,Rpred⊤u⟩.u^{\top}Su=\langle R_{\mathrm{real}}^{\top}u,R_{\mathrm{pred}}^{\top}u\rangle.

Positive scores indicate aligned real and predicted action responses. The final metric is:

M=N(Σ^e−1/2WΣ^e−1/2),N(X)=dXtr⁡(X).M=\mathcal{N}(\hat\Sigma_e^{-1/2}W\hat\Sigma_e^{-1/2}),\quad \mathcal{N}(X)=\frac{dX}{\operatorname{tr}(X)}.

LENS calibrates latent discrepancies using prediction reliability, then weights them by Action-Response agreement.

(3) Supporting Comparisons. Alternative covariance metrics and spectral functions are evaluated in Appendix D.1–D.2 (pages 23–24; Reviewer TpZu — Q1 and Q4).

AuthorsOpenReview ↗

Response to Reviewer 3E8C — Q3–Q4

Q3. Table Formatting and PushT Ablation

We have revised the tables to bold the best result in each column, including ties. The PushT results are:

ConfigurationSuccess Rate
Full LENS147/150 (98.00%)
w/o Elite Reuse148/150 (98.67%)

The difference is one episode out of 150 (0.67 percentage points). Both settings are near the success ceiling. A possible explanation is the exploration–exploitation trade-off in CEM. Elite Reuse preserves promising earlier candidates, leaving fewer slots for new samples; removing it allows broader exploration. Since PushT admits multiple successful pushing trajectories, this can occasionally help on a fixed evaluation set.

Elite Reuse affects search, whereas the Planning Metric determines how candidate outcomes are scored. The ablation is shown in Section 5.3 (page 9) and Figure 4(a) (page 10).

Q4. Practical Value of Post-hoc Planning Adaptation

LENS fits the Planning Metric after World Model training, with the encoder and predictor frozen. This supports model reuse, deployment-specific adaptation, and independent development of prediction and planning.

(1) Reusing Pretrained World Models

Retraining a large encoder or predictor for each downstream task incurs considerable computational and maintenance costs. LENS instead estimates a metric from offline transitions without changing the pretrained model.

For example, a metric fitted on DROID real-robot trajectories increases frozen V-JEPA-2-AC's zero-shot RoboCasa Place success from 31.25% to 40.63%. See Section 5.4 (page 9) and Appendix D.5 (pages 26–28).

(2) Adapting to Deployment Conditions

Different environments may have different action distributions, dynamics, and prediction-error structures. LENS allows Action-Response Coupling and Prediction Accuracy statistics to be re-estimated from offline transitions without changing the shared World Model. Thus, one predictive backbone can support different deployment-specific Planning Metrics.

(3) Training-time Planning-aware Regularization

Incorporating planning-aware signals during World Model training is a worthwhile research direction. Joint learning could produce representations better suited to action selection, but it raises three challenges.

First, prediction and planning have different objectives. Predictive losses target the accuracy of future latent states, while planning depends on the relative quality and ranking of candidate actions. A joint objective needs suitable planning supervision and a balance that preserves predictive quality. Moreover, isotropic marginal features do not imply isotropic Action-Response structure or prediction errors; these depend on action-conditioned dynamics.

Second, latent geometry changes during training. Encoder updates alter representation coordinates; predictor updates change action responses and errors. A jointly trained metric must remain aligned with both, complicating estimation, update schedules, and stability. Post-hoc fitting operates in a fixed latent space.

Third, offline trajectories provide incomplete planning supervision. They reveal outcomes of executed actions, generally not counterfactual results of alternatives from the same state. Learning candidate rankings may require additional interactions, counterfactual data, or surrogate objectives. Differentiating through CEM also poses challenges due to sampling, ranking, and elite selection.

We see incorporating Action-Response Coupling and Prediction Accuracy into World Model training as a promising extension. For now, LENS separates predictive learning from metric estimation, allowing each to be developed, evaluated, and updated independently. This preserves the pretrained model's predictive capabilities while improving planning through lightweight offline calibration, and permits deployment-specific metrics without retraining the shared backbone.

Reviewer_3E8COpenReview ↗

Response to rebuttal

I thank the authors for their efforts. Most of my concerns have been adequately addressed. I have improved my score to a 3.

Reviewer_TpZuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • studies the prediction–planning gap in JEPA world models: accurate future predictions do not necessarily yield good action selection
  • proposes LENS, a latent metric that weights directions by action-response agreement and prediction reliability
  • improves planning success from 85.8% to 95.8% (!!!) across four environments, without retraining the world model
Critical strengths
  • pretty clear motivation: Euclidean distance need not be optimal for planning in learned latent spaces
  • quite strong results across four environments, with gains on multiple frozen world-model backbones
  • controlled experiment: improves action ranking (Spearman Improvement) with identical predictions and candidates
  • detailed ablations separating metric learning, retrieval, and elite reuse
Critical weaknesses
  • the main idea of replacing Euclidean distance with a learned Mahalanobis metric is not particularly novel
  • the method combines several estimation steps and heuristics: ridge residualization, covariance shrinkage, spectral weighting, and bootstrap aggregation. Hard to isolate what matters or not which is annoying...
  • prediction-error weighting alone already achieves 93.5% versus 95.8% for full LENS. The added value of action-response fidelity is modest...
  • gains depend partly on retrieval initialization and elite reuse, rather than the proposed metric alone (compound with above point)
  • evaluation remains limited to four simulated environments, with no robustness analysis under distribution shift or limited offline data.... but that's not the biggest issue IMO
Questions for authors

Some simple questions below, thank you:

  • how does LENS compare against simple covariance whitening, inverse prediction-error covariance, or regularized Mahalanobis metrics?
  • can you isolate the metric's effect on closed-loop success without retrieval initialization or elite reuse?
  • how sensitive is the metric to offline dataset size, action coverage, and distribution shift? (very important IMO)
  • why choose the specific spectral weighting 1+Blog⁡(1+λ)1+B\log(1+\lambda)? How sensitive are results to BB and alternative weighting functions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer TpZu — Q1–Q2

We thank the reviewer for the thoughtful comments. We have uploaded a revised PDF. The changes are summarized in the Official Comment for the full submission, “Summary of Major Manuscript Revisions.” References below point to the revised PDF.

Q1. Comparison with Alternative Metrics

We compare five metrics with matched frozen models, tasks, and planning budgets. Retrieval Initialization and Elite Reuse are disabled. Each environment uses three seeds and 50 tasks per seed.

Closed-loop success rate (%, ↑):

MethodTwoRoomPushTReacherCubeAvg.
Euclidean87.3395.3377.3366.0081.50
Latent Covariance Whitening71.389.370.072.075.67
Regularized Mahalanobis86.093.378.772.082.50
Inverse Prediction-Error Covariance86.092.6780.068.6781.83
LENS-metric100.093.384.776.088.50

LENS-metric achieves the highest average success rate (88.50%).

An important distinction exists between Inverse Prediction-Error Covariance and the Prediction Accuracy component of LENS. The former directly inverts the centered sample covariance of prediction errors, whereas LENS uses the non-centered prediction-error second moment:

E[ee⊤]=Cov⁡(e)+E[e]E[e]⊤\mathbb{E}[ee^{\top}]=\operatorname{Cov}(e)+\mathbb{E}[e]\mathbb{E}[e]^{\top}

Centered covariance captures fluctuations of prediction errors around their mean but does not account for persistent prediction bias. The non-centered second moment incorporates both, providing a more comprehensive measure of prediction reliability across latent directions.

Prior work has used uncertainty to adjust learning objectives or latent distances. For example, Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (CVPR 2018) uses uncertainty to balance training losses, while Visual Reinforcement Learning with Imagined Goals (NeurIPS 2018) leverages uncertainty in VAE-based latent representations for goal-conditioned control.

LENS differs by estimating direction-dependent prediction reliability from the actual errors between real future states and World Model predictions, together with the agreement between real and predicted Action-Responses. The entire procedure is performed after World Model training, with the encoder and predictor kept frozen. See Appendix D.1 (pages 23–24, Table 9).

Q2. Isolating the Planning Metric

We evaluate metric-only closed-loop planning in five environments. Both methods share the frozen model, paired tasks, and planning budget. Retrieval Initialization and Elite Reuse are disabled.

Closed-loop success rate (%, ↑):

MethodTwoRoomPushTReacherCubeWallAvg.
LeWM (matched)87.3395.3377.3366.0022.0069.60
LENS metric-only100.093.384.776.056.0082.00

Metric-only LENS improves four of five environments, increasing average success from 69.60% to 82.00% (+12.40 percentage points).

For TwoRoom, PushT, Reacher, and Cube, LeWM (matched) follows LENS's evaluation protocol (three planner seeds, 50 tasks each). These local baselines are reported in Appendix C.10 (page 22, Table 7) and differ from the original LeWM paper's results because its evaluation setup was different.

We add Wall, an image-goal navigation task in which the agent must pass through an opening in a fixed wall. Both methods use the same LeWM checkpoint trained for 10 epochs and 50 paired tasks. Success requires Euclidean agent–goal distance strictly below 4.5 environment units. Closed-loop CEM/MPC uses 600 candidates / 60 elites / 20 iterations, horizon 40, replanning every 20 steps, a 300-step episode budget, identical cross-replanning warm starts, and a terminal-only goal cost. Replacing only the Planning Metric raises Wall success from 22.00% to 56.00% (+34.00 percentage points). Wall's planning budget is matched within the task, and its episodes are not pooled with those of the other four environments (Appendix D.3, pages 24–25, Table 12).

Additionally, we evaluate metric-only planning on a larger frozen World Model, V-JEPA-2-AC, pretrained on DROID. We use 2,000 deduplicated DROID trajectories, with 1,800 for metric fitting and 200 held out, then evaluate zero-shot on RoboCasa Place without any target-domain fitting. L2 achieves 31.25% success versus 40.63% for LENS-metric (Appendix D.5, pages 26–28, Tables 16–17).

In a separate metric-only ranking experiment on the original four benchmarks, we hold candidate actions and World Model predictions fixed, so only the scoring metric affects the rankings. Average Spearman correlation with real outcomes rises from 0.540 to 0.696 (+0.156) (Section 5.3, page 9, Figure 3(d)). This isolates the metric's effect on decisions from search enhancements.

AuthorsOpenReview ↗

Response to Reviewer TpZu — Q3–Q4

Q3. Offline Data, Action Coverage, and Distribution Shift

(1) Offline Data Size. We vary offline trajectories from 10% to 100%, refitting the metric and Retrieval Bank using three trajectory-level subset realizations.

Closed-loop success rate (%, ↑):

DataTwoRoomPushTReacherCubeAvg.
10%100.095.385.393.393.5
20%100.096.487.896.295.1
30%99.695.886.795.194.3
40%100.096.482.794.293.3
50%99.696.786.496.094.7
60%100.096.484.996.294.4
70%99.695.685.395.694.0
80%99.697.387.195.894.9
90%98.796.485.695.694.1
100%100.098.088.097.395.8

LENS achieves strong results across different offline-data fractions, reaching 95.8% average success with the full dataset (Appendix D.6, page 28, Table 18).

(2) Action Coverage. We cluster standardized 25-step actions into 20 modes (MiniBatch K-means), varying coverage at fixed fitting and Retrieval Bank sample counts.

Closed-loop success rate (%, ↑):

CoverageTwoRoomPushTReacherCubeAvg.
10%96.990.086.981.188.7
20%99.192.483.886.490.4
30%99.395.383.689.191.8
40%99.692.488.091.692.9
50%99.693.183.693.192.3
60%99.693.885.391.892.6
70%99.694.788.792.293.8
80%99.396.984.994.994.0
90%99.895.185.392.293.1
100%100.098.088.097.395.8

Average success generally increases with action-mode coverage, peaking at full coverage. This suggests that diverse offline actions and broader Action-Response coverage benefit LENS (Appendix D.7, pages 28–29, Table 19).

(3) Distribution Shift. We fit a metric on DROID real-robot trajectories with frozen V-JEPA-2-AC and evaluate zero-shot on RoboCasa PnPCounterTop Place, without RoboCasa fitting or search enhancements and with matched CEM budgets.

MethodSuccess (%, ↑, 32 paired episodes)
V-JEPA-2-AC + L231.25
V-JEPA-2-AC + LENS-metric40.63

The metric improves zero-shot success by 9.38 percentage points under distribution shift (Appendix D.5, pages 26–28, Tables 16–17).

Q4. Spectral Weighting and Sensitivity to β

LENS uses the logarithmic spectral weighting rule to emphasize positive real–predicted Action-Response agreement:

wi=1+βlog⁡(1+max⁡(λi,0))w_i=1+\beta\log(1+\max(\lambda_i,0))

(1) Spectral Functions. We compare Logarithmic, Linear, Square Root, and Saturating functions using β = 100, with Retrieval Initialization and Elite Reuse disabled.

Closed-loop success rate (%, ↑):

FunctionTwoRoomPushTReacherCubeAvg.
Logarithmic100.093.384.776.088.50
Linear98.792.084.077.388.00
Square Root100.095.378.778.788.17
Saturating99.396.082.075.388.17

The four functions attain similar averages (88.00%–88.50%), with Logarithmic highest. This indicates that the benefit does not depend on one exact weighting function. The logarithm preserves baseline weights and emphasizes positive response agreement. Its sublinear growth compresses large eigenvalues, limiting the dominance of a few high-response directions in candidate scoring. See Appendix D.2 (page 24, Tables 10–11).

(2) Sensitivity to β. We sweep β = 0–200 using full LENS, including Retrieval Initialization and Elite Reuse, with other settings fixed.

Closed-loop success rate (%, ↑):

βTwoRoomPushTReacherCubeAvg.
098.096.082.795.393.0
2099.395.387.395.394.3
40100.096.088.096.795.2
60100.098.084.794.794.3
8099.396.085.397.394.5
100100.098.088.097.395.8
12099.398.086.097.395.2
14099.398.782.096.794.2
160100.098.781.394.793.7
18099.397.384.796.094.3
20099.396.080.794.792.7

β = 100 gives the highest average success (95.8%). For β = 40–120, average success remains 94.3%–95.8%, suggesting that LENS achieves favorable results across a relatively broad range of weighting strengths (Appendix D.8, page 29, Table 20).

Comments on the paper

2 comments

2. Additional Theoretical and Experimental Evidence

We have added the following analyses and experiments in response to the reviewers' questions.

Theoretical justification. We introduce Theorem 1 (Minimax Relative Error Calibration), which provides an optimization-based justification for inverse prediction-error weighting under a fixed total metric scale. The theorem appears in Section 4 (page 5), and its complete proof is given in the new Appendix A (page 14). This result provides a theoretical interpretation of an existing component of LENS; it does not change the deployed metric.

Directional anisotropy. We add spectral visualizations of Action-associated and Prediction-error directions, together with normalized spectral entropy and coefficient-of-variation measurements (Section 5.2, page 8, Figure 2 and Table 2). We also include held-out prediction-MSE measurements comparing high- and low-weight directions (Appendix D.9, page 30, Table 21).

Planning Metric comparisons. We compare LENS against Euclidean Distance, Latent Covariance Whitening, Regularized Mahalanobis, and Inverse Prediction-Error Covariance (Appendix D.1, pages 23–24, Table 9). We additionally conduct matched metric-only closed-loop evaluations across five environments, including Wall, with Retrieval Initialization and Elite Reuse disabled (Appendix D.3, pages 24–25, Table 12).

Spectral weighting and data sensitivity. We compare alternative spectral weighting functions (Appendix D.2, page 24, Tables 10–11) and study sensitivity to offline data size (Appendix D.6, page 28, Table 18), action-mode coverage (Appendix D.7, pages 28–29, Table 19), and the weighting coefficient β\beta (Appendix D.8, page 29, Table 20).

Task relevance of learned directions. We add three subspace analyses examining action information, realized physical-state changes, and task-related physical-state information in the final LENS metric (Appendix D.4, pages 25–26, Tables 13–15).

Generalization. We extend the evaluation to frozen V-JEPA-2-AC under DROID-to-RoboCasa zero-shot transfer and to continuous Navier–Stokes fluid boundary control (Section 5.4, pages 9–10; Appendix D.5, pages 26–28, Tables 16–17).

3. Appendix Reorganization and Reproducibility

We have expanded and reorganized the appendices to provide clearer mathematical derivations, implementation details, and experimental protocols.

  • Appendix A (page 14) provides the complete proof of the Minimax Calibration theorem.
  • Appendix B (pages 14–18) gives the finite-sample metric construction, including conditional residualization, ridge regression, cross-fitting, action whitening, prediction-error estimation, bootstrap aggregation, and the directional decomposition of the deployed metric.
  • Appendix C (pages 18–23) documents the planning algorithm, frozen checkpoints, evaluation protocols, hyperparameters, matched baselines, and component-ablation definitions.
  • Appendix D (pages 23–30) presents the additional metric comparisons, sensitivity analyses, task-relevance probes, zero-shot transfer experiments, and prediction-error diagnostics.

These revisions provide more complete mathematical and experimental specifications while preserving the original method and its principal findings.

4. Highlighting of Revisions

Because the mathematical notation and presentation have been reorganized throughout the manuscript, highlighting every local symbol replacement or editorial adjustment would introduce extensive visual markings and make the revised paper harder to read.

We therefore highlight the principal substantive additions, including the Minimax theorem and its proof, new experimental analyses, and the corresponding tables and figures, rather than marking every notation or wording change. The locations of the major revisions are listed above, and OpenReview's automatically generated PDF diff provides an additional way to inspect the changes between versions.

We hope this summary helps the reviewers and Area Chair navigate the revised manuscript. Our detailed responses to individual concerns are provided in the corresponding review discussion threads.

AuthorsOpenReview ↗

Summary of Major Manuscript Revisions

We thank the reviewers and Area Chair for their careful evaluation and constructive feedback. We have uploaded a revised manuscript to clarify the mathematical presentation, strengthen the theoretical justification, and provide additional experimental evidence. We summarize the main revisions in two comments.

1. Mathematical Clarification and Unified Notation

The original submission contained some overlapping notation, insufficiently explicit definitions, and mathematical expressions that could be presented more clearly. We have therefore revised the Problem and Method (Sections 3–4, pages 4–7), introducing consistent notation and reorganizing the exposition around Prediction Accuracy, Action-Response Coupling, and the final Planning Metric.

The notation in the revised manuscript is summarized as follows.

World-model prediction. The current and goal observations are denoted by oto_t and ogo_g, with latent representations zt=encoder⁡(ot)z_t=\operatorname{encoder}(o_t) and zg=encoder⁡(og)z_g=\operatorname{encoder}(o_g). A candidate action sequence is a=(at,…,at+H−1)a=(a_t,\ldots,a_{t+H-1}), where HH denotes the prediction horizon. The real future is zt+Hz_{t+H}, and its World Model prediction is

z^t+H=predictor⁡(zt,a).\hat z_{t+H}=\operatorname{predictor}(z_t,a).

Prediction Accuracy. We denote the prediction error by e=zt+H−z^t+He=z_{t+H}-\hat z_{t+H}, its non-centered second-moment matrix by Σe=E[ee⊤]\Sigma_e=\mathbb E[ee^\top], and its regularized finite-sample estimate by Σ^e\hat\Sigma_e.

Action-Response Statistics. The real and predicted latent changes are Δz=zt+H−zt\Delta z=z_{t+H}-z_t and Δz^=z^t+H−zt\Delta\hat z=\hat z_{t+H}-z_t. The pre-action context is denoted by ctc_t. The residualized and whitened action vector is a~\tilde a, while rrealr_{\mathrm{real}} and rpredr_{\mathrm{pred}} denote the context-conditioned residuals of real and predicted latent changes. The corresponding statistics are

\qquad R_{\mathrm{pred}}=\mathbb E[r_{\mathrm{pred}}\tilde a^\top].$$ **Action-Response Coupling.** We use $S$ for the symmetric coupling matrix: $$S=\frac12\left( R_{\mathrm{real}}R_{\mathrm{pred}}^\top+ R_{\mathrm{pred}}R_{\mathrm{real}}^\top \right).$$ Its eigenpairs are denoted by $(\lambda_i,u_i)$. The spectral weighting function is $g(\lambda)=1+\beta\log(1+\max(\lambda,0))$, where $\beta$ controls the strength of positive Action-Response weighting. The weighting matrix for bootstrap replicate $b$ is $W_b$, and $W$ denotes the aggregated positive-definite Action-Response weighting matrix. **Planning Metric.** The final deployed Mahalanobis metric is denoted by $M$, with latent dimension $d$ and trace-normalization operator $\mathcal N(X)=dX/\operatorname{tr}(X)$. Its construction is $$M=\mathcal N\left( \hat\Sigma_e^{-1/2}W\hat\Sigma_e^{-1/2} \right),$$ and the planning cost of a candidate action sequence is $$J(a)=(\hat z_{t+H}-z_g)^\top M(\hat z_{t+H}-z_g).$$ We have also revised the mathematical expressions and their explanations to make the construction easier to follow. The detailed finite-sample estimation procedure is now presented in **Appendix B (pages 14–18)**. **These changes do not alter the underlying LENS method. The revised Planning Metric construction is mathematically equivalent to the original formulation.** In particular, the previous directional weighting and projector-based representation yields the same trace-normalized full-matrix metric. Conditional residualization, action whitening, bootstrap aggregation, and the deployed planning procedure remain unchanged. The revisions improve the precision and readability of the mathematical presentation without introducing a different planning algorithm. The additional theoretical and experimental evidence, appendix revisions, and highlighting conventions are summarized in next comment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.