- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the prediction and planning Gap in JEPA-based world models to improve goal-conditioned planning with pretrained world models. The proposed model earns a weighted distance metric based on how accurately the world model predicts action-induced changes in different latent directions in stead of using Euclidean distance between predicted future states and goals. The experiments are conducted on 4 visual control benchmarks, and LENS improves average planning success from 85.8% to 95.8% and produces better action rankings.
Critical strengths
The author makes a great point of identifying the limitation of existing world-model planning methods, and claims the foundamental differences in prediction reliability and action sensitivity.
The proposed method, LENS, improves planning without retraining the model, which makes it easy to apply to pretrained models. Also, it shows good empirical results across different world-model architectures.
Critical weaknesses
I think the action-response fidelity formulation includes some heuristic choices, and the author could provide more theoretical justification for why this particular formulation is optimal for planning.
The experiment part covers only four relatively simple visual control environments. More challenging tasks, distribution shifts, and larger scale world models would help establishing the generalizability.
As I understand, the assumption of this paper is that more predictable and action-sensitive latent directions should receive higher weights in planning. However, these directions are not necessarily more important for achieving the goal. Since the learned metric is fixed across tasks, it may overemphasize predictable but task-irrelevant features. The paper lacks experiments investigating this potential mismatch.
Questions for authors
See weakness
3 comments
Response to Reviewer bZS3 — Weakness 2
Weakness 2. Generalization Beyond the Original Benchmarks
We added evaluations on V-JEPA-2-AC under DROID-to-RoboCasa zero-shot transfer, OGBench Visual Scene multi-object manipulation, and Navier–Stokes fluid control. These experiments examine generalization across model scales, task complexities, and dynamical systems.
(1) Large World Model: V-JEPA-2-AC. We use the released vjepa2_ac_droid.pth.tar checkpoint with a ViT-G/16 visual encoder and a 24-layer action-conditioned predictor. Single-view 256×256 RGB images produce 256 patches of 1,408 channels; actions encode seven-dimensional relative pose/gripper commands. We freeze the encoder and predictor and fit a shared 1,408×1,408 channel metric across patches, averaging goal costs over spatial positions.
From 2,000 deduplicated DROID trajectories, 1,800 are used for fitting and 200 held out. Each window includes three actions and aligned real/predicted futures. We use five-fold trajectory-grouped cross-fitting and 100 trajectory-level bootstrap replicates.
We evaluate RoboCasa PnPCounterTop Place on 32 paired episodes (seeds 2–33). Both methods use the same goals, initial states, and CEM initialization; only the Planning Metric changes. Neither Retrieval Initialization nor Elite Reuse is used, and no RoboCasa data are used to fit the metric or train the model.
| Setting | Configuration |
|---|---|
| CEM candidates / elites / iterations | 300 / 10 / 15 |
| Prediction horizon | 3 actions |
| Execution / replanning | First action for 5 environment steps |
| Episode budget | 60 steps / 12 replans |
| Success | Object–target distance below 0.15 simulator units |
| Method | Zero-shot Success |
|---|---|
| V-JEPA-2-AC + L2 | 31.25% |
| V-JEPA-2-AC + LENS-metric | 40.63% |
Metric-only LENS improves zero-shot success by 9.38 percentage points, without updating the pretrained backbone or using target-domain fitting (Section 5.4, page 9; Appendix D.5, pages 26–28, Tables 16–17).
(2) OGBench Visual Scene. We also test image-goal multi-object manipulation involving a block, drawer, window, and buttons. The goal is the image from 25 steps later in the same validation trajectory; the maximum execution budget is 50 steps. Both methods share the frozen World Model, initial states, goals, and CEM settings (300 candidates / 30 elites / 30 iterations, horizon 25).
Success requires joint positional tolerance 0.10 (10 cm) and exact agreement of both button states. All conditions must hold after executing actions; time-zero successes are excluded. We use three seeds (42–44), 50 paired tasks per seed.
| Method | Full-task Success |
|---|---|
| LeWM | 60.67% |
| Full LENS | 69.33% |
Full LENS improves success by 8.67 percentage points, with gains in all three seeds. This evaluation extends the task setting to interacting objects and combined continuous/discrete goals. These Scene results are reported here as additional rebuttal evidence.
(3) Navier–Stokes Fluid Control. We test a two-dimensional boundary-control task: choosing the upper boundary's horizontal velocity to follow a target flow field. We evaluate 50 regular and 30 nonperiodic reference trajectories across three seeds, yielding 240 paired trials per method. Both methods use identical frozen dynamics models and reference trajectories. CEM uses 300 candidates / 30 elites / 30 iterations, horizon five macro-actions (two primitive transitions each), with replanning after each executed macro-action.
RMSE (lower is better); return (higher is better):
| Metric | LeWM | Full LENS |
|---|---|---|
| Late Full-field RMSE | 0.046340 | 0.040119 |
| Final Full-field RMSE | 0.023805 | 0.012722 |
| Mean Full-field RMSE | 0.051731 | 0.044475 |
| Late Interior RMSE | 0.004703 | 0.003988 |
| Final Interior RMSE | 0.002125 | 0.001854 |
| Average Cumulative Return | −9.6966 | −8.5480 |
LENS improves all six measures. Final Full-field RMSE falls by approximately 46.6% (0.023805 to 0.012722). The three principal metrics appear in Section 5.4 (page 10, Table 3); the additional measurements are reported above.
These experiments show improvements with a larger pretrained World Model and cross-domain manipulation, multi-object visual control, and nonlinear fluid dynamics, without retraining the respective World Models.
Response to Reviewer bZS3 — Weakness 3
Weakness 3. Task Relevance of Learned Directions
We added three held-out linear-probe experiments to test whether the directions emphasized by Action-Response Coupling and the final Planning Metric contain task-relevant physical information, rather than only predictable features. Full protocols are in Appendix D.4 (pages 25–26, Tables 13–15).
(1) Action Information. We project real future latent changes into four 32-dimensional subspaces: Action-Response Top-32, Bottom-32, PCA Top-32, and Random-32. Linear regression predicts the associated 25-step actions, using 256 training and 256 independent test trajectories per environment.
Held-out action decoding R² (↑):
| Environment | Response Top-32 | Bottom-32 | PCA Top-32 | Random-32 |
|---|---|---|---|---|
| TwoRoom | 0.02563 | −0.03934 | −0.01784 | −0.01861 |
| PushT | 0.20044 | −0.01531 | 0.12151 | 0.07606 |
| Reacher | −0.02868 | −0.03032 | −0.02815 | −0.03258 |
| Cube | 0.13757 | −0.03124 | 0.09812 | 0.04462 |
These results indicate that the Response Top-32 subspace retains more action-related information (Table 13, page 25).
(2) Real Physical-state Changes. Using the same subspaces, we predict realized physical-state differences: agent XY displacement (TwoRoom), agent/block XY and block orientation changes (PushT), two joint-angle changes (Reacher), and object XYZ displacement (Cube).
These variables relate to task completion: TwoRoom and Cube require reaching target positions, PushT requires moving and orienting the block, and Reacher's end-effector position is determined by its joints.
Held-out physical-change decoding R² (↑):
| Environment | Response Top-32 | Bottom-32 | PCA Top-32 | Random-32 |
|---|---|---|---|---|
| TwoRoom | 0.59478 | −0.05265 | 0.07896 | 0.13598 |
| PushT | 0.28966 | 0.01679 | 0.22984 | 0.19649 |
| Reacher | 0.11137 | −0.03099 | −0.02135 | −0.00482 |
| Cube | 0.63289 | −0.02320 | 0.31255 | 0.18558 |
These results show that the selected directions retain more information about actual action-induced physical changes. Crucially, these changes are directly related to task progress, including navigation, object manipulation, and reaching. Thus, Action-Response Coupling emphasizes latent directions that reflect task-relevant physical dynamics, rather than merely predictable variations in latent representations (Table 14, page 26).
(3) Task-related Information in the Final Metric. We next analyze eigendirections of the complete LENS Planning Metric. From 20,000 real frames per environment, we predict same-frame task-state variables: agent XY (TwoRoom), agent/block pose (PushT), joint angles (Reacher), and object XYZ (Cube).
We split data by trajectory into approximately 70% train / 15% validation / 15% test, over three split seeds. Normalization and PCA use training data, while validation selects the ridge penalty. Random-subspace results average ten draws per split.
Held-out task-state decoding R² (↑):
| Environment | LENS Top-32 | Bottom-32 | Random-32 | PCA Top-32 |
|---|---|---|---|---|
| TwoRoom | 0.8529 | 0.0489 | 0.3352 | 0.5567 |
| PushT | 0.4545 | 0.0618 | 0.3051 | 0.4072 |
| Reacher | 0.7683 | 0.0548 | 0.4345 | 0.7392 |
| Cube | 0.9117 | −0.0085 | 0.2595 | 0.2075 |
| Avg. | 0.7469 | 0.0392 | 0.3336 | 0.4776 |
LENS Top-32 achieves the highest R² across all four environments, with an average of 0.7469, compared with 0.4776 for PCA, 0.3336 for Random, and 0.0392 for Bottom-32. Importantly, the decoded variables describe the physical states directly involved in achieving the task goals, including agent positions, object poses, and joint configurations. These findings suggest that the learned Planning Metric emphasizes latent directions containing task-relevant physical-state information. Together with the action-induced change analysis, this connects the metric's high-weight directions to both the underlying action-conditioned dynamics and the physical states relevant to goal achievement (Table 15, page 26).
(4) Fixed-candidate Decision Ordering. We also fix the action candidates and all World Model predictions and compare rankings with real outcomes. Average Spearman correlation increases from 0.540 (Euclidean) to 0.696 (LENS), +0.156, providing direct decision-level evidence (Section 5.3, page 9, Figure 3(d)). These results jointly connect the emphasized directions to task-state information and improved action ranking.
Response to Reviewer bZS3 — Weakness 1
We thank the reviewer for the thoughtful feedback. We have uploaded a revised PDF. An overview is available in the Official Comment for the full submission, “Summary of Major Manuscript Revisions.” The locations below refer to the revised PDF.
Weakness 1. Theoretical Basis of Action-Response Fidelity
LENS combines two complementary principles: Action-Response Coupling emphasizes latent directions whose action associations agree between real transitions and World Model predictions. Prediction Accuracy calibrates the contribution of directions according to their prediction errors.
(1) Action-Response Coupling. From offline transitions, we estimate the real and predicted response statistics and their symmetric coupling matrix:
S=21(RrealRpred⊤+RpredRreal⊤).
For any unit latent direction u, the coupling score satisfies
u⊤Su=⟨Rreal⊤u,Rpred⊤u⟩
u⊤Su=21(Rreal⊤u22+Rpred⊤u22−(Rreal−Rpred)⊤u22)
The first two terms reflect the strengths of real and predicted action associations, while the last captures their mismatch. Thus, the score measures response strength together with real–predicted agreement.
With residualized, whitened actions satisfying E[a~a~⊤]=I, the same score equals
u⊤Su=Ea~[(u⊤Rreala~)(u⊤Rpreda~)].
It is therefore the average product of real and predicted linear responses to the same standardized action variation. This matters for planning because candidate actions are ranked by predicted consequences. Positively aligned response directions provide a stronger link between modeled action effects and observed dynamics (Section 4, page 6 and Appendix B, pages 15–17).
(2) Positive Spectral Weighting. Let Sui=λiui. Then
λi=⟨Rreal⊤ui,Rpred⊤ui⟩.
LENS uses wi=1+βlog(1+max(λi,0)). Positive eigenvalues identify aligned responses. The unit offset retains all directions, and logarithmic compression limits domination by large modes.
We added spectral-function comparisons and a sweep of β (Appendix D.2, page 24, Tables 10–11 and Appendix D.8, page 29, Table 20).
(3) Minimax Prediction-Error Calibration. Define prediction error e=zt+H−z^t+H and non-centered second moment Σe=E[ee⊤]≻0. We prove that
minX⪰0,tr(X)=dλmax(Σe1/2XΣe1/2)
has the unique minimizer
X⋆=tr(Σe−1)dΣe−1=N(Σe−1),N(X)=tr(X)dX.
Under a fixed trace budget, trace-normalized inverse error weighting minimizes the worst-case weighted error over standardized directions. Larger-error directions receive lower relative weight. The normalization is invariant to a common positive rescaling of Σe.
Theorem 1 is in Section 4 (page 5). The full proof is in Appendix A (page 14).
Together, the two components calibrate prediction reliability and emphasize action-response alignment. The added theorem justifies the existing Prediction Accuracy component. It does not alter the deployed LENS method.