acceptodds
Under review as a conference paper at ICLR 2027

LeWAM: End-to-End World and Action Modeling with JEPAs

29 views

Abstract

Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction. However, existing JEPAs typically learn representations and dynamics independently of downstream control objectives, complicating the pipeline and leaving unexplored how world- and action- learning benefit each other. We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics. Compared with prior JEPAs in visual planning, LeWAM improves success rate from 26.8% to 85.5% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3x faster. On behavior cloning tasks in simulation and physical robots, LeWAM is also competitive with established methods. Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Visual Control, World Action Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2QiSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LeWAM jointly learns JEPA-based latent dynamics and a flow-matching action policy, supporting both direct control and visual planning, achieving a significant increase in the success rate.

Critical strengths
  1. Joint modeling of the world and actions based on the JEPA framework is clearly an emerging trend.
  2. This paper is very clearly written; apart from the experiments, which are not convincing, everything else is quite good.
Critical weaknesses
  1. The core innovation of LEWM lies in its low computational requirements; however, it does not represent the highest-performing variant within the JEPA architecture. Consequently, using LEWM as the primary benchmark fails to demonstrate whether the proposed method offers any performance improvement, as the research objectives are entirely different.
  2. Numerous existing implementations model the world and action jointly within the JEPA framework, such as ACT-JEPA and WA-JEPA, yet the paper provides no comparison against them.
Questions for authors
  1. How does the JEPA architecture achieve out-of-distribution (OOD) generalization?
  2. How can fairness across different environments be ensured?
  3. Given that tasks vary by environment, does this imply that a separate JEPA model must be trained for each individual environment?
5 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for recognizing joint world-and-action modeling within the JEPA framework as an emerging direction and for finding the paper clearly written. Please see our point-by-point responses below.

A0: General Response:

Regarding the research objective, we want to respectfully clarify that:

  1. LeWM [4] and LeWAM are in the same research direction. Both aim to (1) learn a strong action-conditioned world model that captures the environment dynamics, (2) plan and reason at test time using the world model, and (3) explore end-to-end training without additional techniques.

  2. LeWAM is different from ACT-JEPA/WA-JEPA/JEPA-WAM. ACT-JEPA [3] and WA-JEPA [2] are JEPA-enhanced policy models; LeWAM is a world model with a coupled policy. Their world-model components provide only auxiliary training supervision, and their latent spaces support policy learning while retaining minimal information about the environment's dynamics. Instead, LeWAM's model-based planning and control rely on the learned dynamics, while the stochastic policy only provides a prior distribution.

  3. LeWAM benefits from two design choices. Joint prediction of latent states and actions shapes the representation and dynamics. The learned policy distribution provides a prior that improves sampling efficiency and narrows the planning search space.

W1: LeWM is an efficiency-oriented JEPA rather than the highest-performing variant, so using it as the primary benchmark does not demonstrate a performance improvement, and the research objectives differ.

A1: While we agree that LeWM is not the highest-performing JEPA variant, we also do not attempt to create the highest-performing JEPA architecture. Our research goal is to study joint world and action modeling in a JEPA that is simple by design (requiring no extensive hyperparameter tuning), is trained end-to-end with no pretrained components, and works across a variety of tasks. In other words, we want to push toward a model that, like LeWM, is small and efficient yet supports both direct control and planning. Note that LeWM has approximately 16M parameters and LeWAM approximately 17M. Both models are trained end-to-end with SIGReg. Therefore, we respectfully disagree that the research objectives are entirely different and believe they are actually quite aligned.

W2: Existing methods that model the world and actions jointly within the JEPA framework, such as ACT-JEPA and WA-JEPA, are not compared against.

A2: While WA-JEPA [2] and ACT-JEPA [3] are indeed WAMs that predict in latent space, they do not model and leverage forward dynamics. They mirror works like DreamZero [1] in jointly predicting the next state and action. Modeling forward dynamics is a crucial part of our work, as it enables test-time planning beyond direct policy execution. We have added both methods to the Related Work section (Section 2).

Q1: How does the JEPA architecture achieve out-of-distribution (OOD) generalization?

A3: Thanks for the insightful question. As a simple OOD test, we analyze the robustness of LeWAM and LeWM to visual changes in the environment. Specifically, for each of the three evaluation seeds, we sample 50 random episode starts and set the goal 25 environment steps ahead. In every episode, we independently and uniformly redraw the colors of the agent, the block, and/or the background across RGB. The draws come from a fixed per-episode seed, so every method sees exactly the same colors on exactly the same episodes. The results have been added to Appendix C.9 (Table 14), and we summarize them below.

ModelIn-distributionAgent onlyBlock onlyBackground onlyAll 3 colours
LeWAM, reactive86.0 ± 4.054.0 ± 4.056.0 ± 3.522.0 ± 5.38.7 ± 3.1
LeWAM, gradient planning98.0 ± 2.058.0 ± 4.056.7 ± 13.024.0 ± 2.012.0 ± 2.0
LeWAM (w_dyn=0), reactive82.0 ± 5.346.0 ± 4.040.0 ± 2.019.3 ± 5.010.0 ± 2.0
LeWM, CEM87.3 ± 5.028.0 ± 10.627.3 ± 9.22.7 ± 2.32.7 ± 3.1
Where LeWAM is much more robust to single-variable perturbations than LeWM, especially with respect to agent or block changes. Extending LeWAM to better OOD generalization is a direction for future work, as is the case for world models/WAMs generally.

Q2: How can fairness across different environments be ensured?

A4: Whenever possible, we use public checkpoints for benchmark scores. Otherwise, we use the configs specified in the respective papers or repositories if available. Lastly, all models in all environments see the same number of episodes (50), drawn from the same eval seeds (42, 0, 1), and thus the exact same starting and goal frames. All models receive the same execution budget: 2x the horizon length. Beyond this, it would be great if the reviewer could clarify what “fairness” means in this question so we may answer it concretely.

Q3: Given that tasks vary by environment, must a separate JEPA model be trained for each environment?

A5: Yes, we discuss this in the paper (Conclusion and Limitations, “Single-Task and Single-Embodiment Learning”). We identify a version of the model that supports multiple tasks and environments as a direction for future work, since the current work has all the environment training share the same model architecture and hyperparameter setting.

References

[1] Ye, Seonghyeon, et al. "World action models are zero-shot policies." arXiv preprint arXiv:2602.15922 (2026).

[2] Wang, Xinlin, et al. "WA-JEPA: Rethinking the video JEPA paradigm for world-action modeling in autonomous driving." arXiv preprint arXiv:2608.20974 (2026).

[3] Vujinovic, Aleksandar, and Aleksandar Kovacevic. "ACT-JEPA: Novel joint-embedding predictive architecture for efficient policy representation learning." arXiv preprint arXiv:2501.14622 (2025).

[4] Maes, Lucas, et al. "LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv preprint arXiv:2603.19312 (2026).

Reviewer_2QiSOpenReview ↗

The author did not resolve my issue.

Thank you for your response, but it has not addressed my concerns. Comparing only against LeWAM is insufficient, and the issue of OOD (out-of-distribution) generalization has not been considered at all. It is difficult for me to determine whether this approach possesses general applicability or holds any significant research value.

Reviewer_xpZQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeWAM is a lightweight JEPA trained from pixels that learns three aspects: a visual encoder, action-conditioned latent dynamics, and a flow-matching action policy. A MoT predictor carries both the state branch and the action branch with an attention mask that keeps the clean actions from leaking into the policy. SIGReg keeps the latents Gaussian. At test time the model either runs the policy directly or plans: it samples N action sequences from the policy, rolls them through the latent dynamics, and refines each with a few Adam steps on the latent distance to the goal. On 10 simulated tasks LeWAM averages 91.9% success versus 46.9% for LeWM with CEM, holds up better as the goal horizon grows, and plans about 32x faster than CEM.

Critical strengths
  • The proposal framework is simple yet effective. Two losses and one regularizer, 17M parameters, trains in hours. That is rare in this space and makes the method easy to reproduce and build on.

  • The planning story makes sense. Sampling plans from a learned policy and then doing a few gradient steps through the dynamics is a reasonable fix for CEM blowing up on contact-rich tasks.

  • The evaluation is sufficient for a JEPA paper. 10 sim tasks with 3 seeds and std, a horizon sweep, goal-free BC, and real-robot trials with 30 episodes per method.

Critical weaknesses
  • The headline gap is mostly "has a policy" vs "doesn't." LeWM (CEM) searches from a Gaussian prior; LeWAM starts from a learned policy. Against LeWM (IDM) the gaps shrink a lot (Push-T 98.7 vs 85.5, Cube 100 vs 97.3), and GCBC with no world model beats LeWAM on DrawerCleanup (77.3 vs 64.0). Table 10 shows planning adds about 5 points over the direct policy (86.7 to 91.5). The abstract should reflect that.

  • The "first JEPA" is a bit of an overclaim. Qantara (arXiv 2607.04978, July 2026) is an end-to-end JEPA with action flow matching plus latent prediction, one checkpoint for planning and BC, evaluated on the same LeWM suite. Not cited. Needs a comparison or at least a discussion, and the "first" wording should go.

  • The 32x speedup isn't compute-matched. The appendix admits this ("matched-compute comparisons remain pending"). 300x30 CEM vs five gradient steps. Run CEM on LeWAM's sampled plans, or match the rollout budget.

  • The LeWM baseline is a reimplementation that didn't reach the published numbers (App. B.1). Since that's the main comparison, show the published scores next to the reproduced ones.

  • Table 11: better proprio/action probes, but lower object R2 than LeWM on Push-T (0.90 vs 0.95) and Puzzle (0.37 vs 0.60), and half the effective rank. That's a trade-off, not a clean win.

  • Real robot: 33.3% vs 38.9% for Diffusion Policy at equal FLOPs. Fine as a feasibility check, but "competitive" is generous. The FLOP comparison with pi0.5 is about model size, not method.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for recognizing the simplicity of the framework, the reasoning behind the planning procedure, and the breadth of the evaluation. Please see our point-by-point responses below.

W1 (1): The headline gap is mostly "has a policy" versus "doesn't": against LeWM (IDM) the gaps shrink (Push-T 98.7 vs 85.5, Cube 100 vs 97.3), and GCBC without a world model beats LeWAM on DrawerCleanup (77.3 vs 64.0).

A1 (1): The learned policy proposes candidate plans, and the world model scores and refines them. Figure 6 includes LeWM (IDM), the baseline with its own goal-conditioned policy. Appendix C.4 (Table 10) also isolates LeWAM’s learned dynamics. With its policy removed and CEM as the solver, LeWAM reaches 86.0% on Cube, compared with 63.8% for LeWM (CEM). GCBC beats LeWAM on the drawer task because the drawer data is not playable but very expert, so a pure policy can easily memorize the successful trajectory without planning.

W1 (2): Table 10 shows planning adds about 5 points over the direct policy (86.7 to 91.5).

A1 (2): We agree that the direct policy accounts for most of the success on these four tasks. At test time, the policy proposes candidates and the learned dynamics guide refinement. Appendix C.2 (Table 8) reports the direct and planned results. Appendix C.8 (Table 13) isolates the effect of the dynamics loss during policy training.

  1. Without the dynamics loss, the direct policy averages 86.2% success across Push-T, Cube, Scene, and Puzzle, compared with 87.7% for the full LeWAM direct policy in the same evaluation (Appendix C.8, Table 13).
  2. Direct success averages 86.7% and rises to 94.2% with planning in the revised Table 8 (+7.5 pp).
  3. Even without a policy, in our Appendix C (Table 10), on Push-T and Cube, the pure world model CEM planning can get 84.0 ± 1.3% and 86.0 ± 3.2% success, respectively, which is competitive with LeWorldModel.
  4. Better linear probing results (average R² gains of +0.21 for proprioception, +0.20 for action, and +0.19 for inverse dynamics) show the benefits of joint training for interpretability of visual representation.

W1 (3): The abstract should reflect this.

A1 (3): Thanks for pointing it out. We have reflected it in the abstract and highlighted it.

W2 (1): Qantara [1] is an end-to-end JEPA with action flow matching plus latent prediction, one checkpoint for planning and BC, on the same LeWM suite, and is not cited; a comparison or discussion is needed.

A2 (1): We were not aware of Qantara [1] when working on LeWAM, but upon your insightful review, we agree that it should be mentioned. Qantara uses bridge-flow training over latent states and actions, whereas LeWAM conditions its next-latent predictor on clean actions and learns a policy via action-flow matching in the same forward pass.

Key differenceLeWAMQantara [1]
Joint predictionOne masked forward pass uses clean actions for next-latent regression and noisy actions for policy flow matching.A shared predictor uses Brownian-bridge latent regression and action flow matching, with five corruption modes per training sample.
Goal-conditioned planningThe learned policy proposes action sequences via joint rollouts, and gradients over the dynamics refine the candidates.Gaussian CEM proposes and refits action sequences. Its reported planner uses the dynamics head, with four predictor calls per latent step by default.
Direct controlThe policy can condition on a goal image or run goal-free, depending on training.The reported BC and video–inverse paths receive no goal image.

In the respective papers, LeWAM reports higher planning success on Push-T (98.7% vs. 90.1%), Cube (100.0% vs. 93.7%), and Reacher (100.0% vs. 80.9%), and ties on Two-Room (100.0%). That is because Qantara's fast goal-blind paths depend on the behavior in the training data. On random-policy Reacher data, its BC and video–inverse paths achieve 6.1% and 4.8% success, respectively, while goal-conditioned CEM reaches 80.9%. In its own timing study, that CEM path takes roughly 15–65× longer per environment step than its two direct-control paths [1, Table 2 and Appendix I].

W2 (2): The "first JEPA" wording is an overclaim and should go.

A2 (2): We would like to clarify the importance of training the representation, action-conditioned world model, and policy together in a single end-to-end stage. To the best of our knowledge, LeWAM is the first JEPA to do so.

  1. As shown in A2 (1), Qantara [1] separates the model into 5 different training modes, which do not constitute a single end-to-end training stage. LeWAM uses a single MoT predictor with a causal mask to jointly learn the representation, action conditioning, and policy, enabling the several latent branches to make more efficient use of each other’s gradients within a single training sample.

  2. Another relevant work is ACT-JEPA [3], which is not trained end-to-end in this sense. It uses a pretrained ResNet-18 backbone and an EMA target encoder with stop-gradient. ACT-JEPA’s action-conditioned dynamics are never used during inference, leaving the role of the world model branch unclear.

We have revised the manuscript and added a discussion of related work to strengthen our claim.

W3: The 32× speedup is not compute-matched (300×30 CEM vs five gradient steps); CEM should be run on LeWAM's sampled plans, or the rollout budget matched.

A3: Thanks for pointing this out. We added an H200 latency-matched comparison in Appendix C.7 (Table 12). Both planners use the same LeWAM checkpoint per task and the same evaluation episodes. Gradient planning refines 32 policy proposals for five Adam steps. CEM starts from the lowest-cost policy proposal and uses 32 candidates per round for ten rounds. At comparable latency, gradient planning is ahead by 2.0 pp on Push-T and 0.7 pp on Cube.

TaskGradient success (%)Gradient (ms/env)Policy-init CEM success (%)CEM (ms/env)
Push-T98.7 ± 1.215.796.7 ± 2.315.5
Cube100.0 ± 0.015.399.3 ± 1.215.8

W4: The LeWM baseline is a reimplementation that did not match the published numbers; the published scores should be shown alongside the reproduced ones.

A4: We reran LeWM with its official implementation [2] and trained a checkpoint for each task. The four published scores from LeWM Figure 6 and our corresponding scores are below. Our Cube success criterion also checks end-effector position, as detailed in Appendix B.2.

TaskLeWM published success (%) [2]LeWM rerun success (%)
Two-Room8790.0
Reacher8684.7
Push-T9692.4
Cube7463.8

W5: Table 11 shows better proprio and action probes but lower object R² than LeWM on Push-T (0.90 vs 0.95) and Puzzle (0.37 vs 0.60), and half the effective rank, a trade-off rather than a clean win.

A5: We agree this is a trade-off. LeWAM has lower object-state R² on Push-T and Puzzle and lower effective rank across the four tasks, alongside higher average proprioception, action, and inverse-dynamics R² (Section 4.3 and Appendix C.3). SIGReg constrains the latent distribution. The choice of training objective can affect how much information is retained that is planning- or control-centric [4]. The probes measure decodability of specific quantities, and the planning experiments evaluate control directly.

W6: On the real robot, 33.3% vs 38.9% for Diffusion Policy at equal FLOPs makes "competitive" generous, and the FLOP comparison with π0.5 reflects model size rather than method.

A6: Thanks for pointing it out.

(1) We have replaced "competitive" with "comparable" in the abstract and Section 4.4. We do not claim that LeWAM outperforms Diffusion Policy as a direct policy. The gap is 30/90 vs 35/90 successes, which is within evaluation noise at our trial count (two-proportion test, p = 0.44). The experiment is a feasibility check; it tests whether a JEPA world-action model whose encoder is fine-tuned end-to-end with a joint dynamics objective can still be deployed as a policy on physical manipulation tasks, which most JEPA world-model papers do not evaluate. The result indicates that the joint objective does not measurably degrade imitation performance compared to a policy-only method at equal compute cost. It does not show an advantage, and the revised phrasing makes this clear.

(2) We added Diffusion Policy to Figure 1(d). Both methods yield 3.70 action chunks per TFLOP at the displayed precision, consistent with 270.2 GFLOPs per chunk for LeWAM and 270.0 GFLOPs for Diffusion Policy in Table 2.

References

[1] Rakhimov, Ruslan, et al. "Qantara: Bridge-flow training for multi-paradigm JEPA inference." arXiv preprint arXiv:2607.04978 (2026).

[2] Maes, Lucas, et al. "LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv preprint arXiv:2603.19312 (2026).

[3] Vujinovic, Aleksandar, and Aleksandar Kovacevic. "ACT-JEPA: Novel joint-embedding predictive architecture for efficient policy representation learning." arXiv preprint arXiv:2501.14622 (2025).

[4] Fu, Minghao, et al. "Back to parsimonious latents: Learning task-centric world models from visual foundations." arXiv preprint arXiv:2605.25620 (2026).

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer,

Thank you for your thoughtful review and constructive suggestions. In our response, we clarified how LeWAM differs from Qantara and added a comparison with policy-initialized CEM at matched latency. If you have a chance to revisit our response, we would appreciate hearing whether it addresses your questions.

Best regards, Authors of Submission 20167

Reviewer_JfhiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LeWAM trains an image encoder, an action-conditioned latent dynamics model and a flow-matching policy together, in one stage and from scratch, with a latent prediction loss, an action flow-matching loss and SIGReg. The simulation model has about 16.9M parameters. A masked attention keeps the clean actions used by the dynamics branch from reaching the policy. At test time the policy can be run directly, or its samples can be refined by a few gradient steps through the learned dynamics toward a goal embedding. Section 4.2 reports that over ten simulated environments, planning success averages 91.9% against 46.9% for LeWM with CEM. The paper also reports a 32.3×32.3\times planning speedup on four tasks, and three real-robot tasks where LeWAM gets 33.3%, Diffusion Policy 38.9% and π0.5 28.9%.

Critical strengths
  1. In simulation, planning is much better than with the closest JEPA world-model baseline: 91.9% vs 46.9% averaged over ten environments and, in Table 7 and Fig. 7, 46.2% vs 10.9% on four tasks with 100-step goal offsets, with a small model trained from scratch. A gap this large across ten environments is the main reason I take the approach seriously.

  2. Learning the encoder, dynamics and policy in one stage keeps the system small and simple to train, and the attention mask that blocks clean-action leakage is a reasonable way to share one network between the two roles.

Critical weaknesses
  1. Several headline claims rest on evidence the paper itself describes as incomplete. The 32.3×32.3\times speedup comes from four tasks, although Section 1 states it "across ten heterogeneous simulated environments", and Appendix C.4 says the hardware and timing boundaries "remain to be documented". For Push-T at the same five refinement steps, Figure 12 shows about 0.2 s per plan while Table 8 gives 20.8 ms; at the Figure 12 rate the speedup over CEM would be only about 1.71.7–2×2\times. There is no compute-matched CEM comparison. Simulation results come from a single training seed, 42; the three seeds in Figs. 6 and 7 appear to be evaluation seeds. The same configuration also has different Scene and Puzzle numbers in Figure 6 and Table 9, 97.3 and 80.7, than in Table 10, 93.3 and 74.0.

  2. The paper does not show that learning the dynamics helps the policy, which is its core claim. On the four tasks of Table 10, running the policy directly already gives 86.7%, and planning raises this to 91.5%, so most of the result is the policy. There is no ablation without the dynamics loss, and Figure 6 reports no direct-policy results; direct success appears only for four tasks in Tables 9 and 10. Some baselines are weaker than they could be: GCBC is trained for 20 instead of 100 epochs, the simulated Diffusion Policy runs without EMA, and on the robot π0.5 is fine-tuned with its vision and language backbone frozen. The physical-robot experiment does not test the planning side at all. In Section 4.4, LeWAM is trained without goal context for this experiment and run as a direct policy, "bypassing the test-time planning". The robot models are still trained with the state-prediction loss, but the learned dynamics are not used at test time, and the only robot-side evidence for them is the qualitative rollout probe in Fig. 15 of App. E.4. In Table 2, with 30 trials per task, the end-to-end model reaches 33.3% against 38.9% for Diffusion Policy, at almost the same compute: 270.2 against 270.0 GFLOPs per action chunk.

Questions for authors

Questions that could change my score:

  1. What hardware, timing boundaries and number of candidates were used for Table 8, and why does Figure 12 show about 0.2 s per plan for Push-T at the same five refinement steps? Can you add a compute-matched CEM baseline, and CEM initialized from your policy samples?
  2. Can you add a policy-only model trained without the dynamics loss, and report direct-policy success on all ten tasks?
  3. Can you report at least three training seeds? App. D.1 states that Table 9 uses epoch-30 checkpoints, and App. B's simulation recipe keeps the lowest-validation-loss checkpoint, which presumably applies to Figure 6, yet both report Scene 97.3 and Puzzle 80.7. Which was used, and which checkpoint does Table 10 use?
  4. Which tasks are "contact-rich" in the abstract and Figure 1(a)? The 85.5% / 26.8% numbers match four tasks, but Figure 5 marks six.
  5. On the real robot, LeWAM is run as a direct policy without goal conditioning or planning. Can you report goal-conditioned planning on at least one real task?

If the policy-only ablation shows that the dynamics loss helps the policy, and the timing in Table 8 is documented, I would raise my score.

Not affecting my score: Figure 1(d) compares compute only with π0.5. Diffusion Policy uses almost the same compute as LeWAM, 270.0 against 270.2 GFLOPs, and could be shown as well.

7 comments
AuthorsOpenReview ↗

Response to Reviewer

We would like to thank the reviewer for their detailed and direct feedback. Thank you for recognizing the consistent planning gains across ten environments with a small model trained from scratch, and for finding the single-stage design and the clean-action attention mask reasonable. Please see our point-by-point responses below.

W1 (1): The 32.3× speedup is measured on four tasks, while Section 1 attributes it to ten environments.

A1 (1): Thanks for pointing it out. This is a typo; the 32.3× speedup is measured across 4 representative environments, not 10, and the 10-environment comparison is of success rate. We have corrected it in the revised manuscript.

W1 (2): Appendix C.4 leaves the hardware and timing boundaries of Table 8 undocumented.

A1 (2): Thanks for pointing this out. We have documented this information in our revised manuscript. Here is the exact table below: Table 6 (NVIDIA H100)

TaskLeWM (CEM), ms/planLeWAM, ms/plan
Scene70518.8
Puzzle73116.3
Cube77121.1
Push-T41420.8
W1 (3): Figure 12 shows about 0.2 s per plan on Push-T at five refinement steps, while Table 8 reports 20.8 ms; at the Figure 12 rate, the speedup over CEM would be about 2×.
A1 (3): The 20.8 ms in Table 6 is the H100 planner-call measurement used for the headline timing comparison. Figure 12 reports timing from a separate Push-T sensitivity sweep (about 0.23 s with 32 candidates and five updates). We now distinguish these measurements in Appendix B.3. The reported 32.3× is computed from Table 6.
W1 (4): There is no compute-matched CEM comparison.
A1 (4): We would like to reiterate that the CEM and gradient-based methods are distinct planning methods, with the gradient-based method being more efficient. So, if we compare them in a “compute-matched” setting, i.e., set CEM to use the same computation budget as gradient-based planning, as shown in the table below, it cannot yield useful plans and achieves a low success rate.
TaskLeWAM gradient planning success (%)Compute-matched CEM success (%)
Scene97.3 ± 1.288.67 ± 4.99
Puzzle80.7 ± 11.063.33 ± 2.49
Cube100.0 ± 0.089.3 ± 1.2
Push-T98.7 ± 1.292.7 ± 2.3

W1 (5): Simulation results use a single training seed (42); the three seeds in Figs. 6 and 7 are evaluation seeds.

A1 (5): In light of your comment, we added training-stability experiments to Appendix C.10 (Table 15) of the revised manuscript. The table below reports Push-T planning success for training seeds 42, 0, and 1 using the same evaluation protocol as in the paper.

Training seedPush-T planning success (%)
4298.7 ± 1.2
0100 ± 0
197.33 ± 2.49

We report the mean and standard deviation across evaluation seeds 42, 0, and 1, with 50 episodes per evaluation seed. On the Push-T, the planning success remained high across all three training seeds.

W1 (6): Scene and Puzzle are 97.3 and 80.7 in Figure 6 and Table 9 but 93.3 and 74.0 in Table 10 for the same configuration.

A1 (6): Thanks for pointing out. They are from different seed measurements and devices, as shown in the undocumented results. We have used the results in Table 9 under the consistent evaluation setup in our revised manuscript.

W2 (1): The direct policy already reaches 86.7% on the Table 10 tasks, and planning adds about 5 points, so most of the result comes from the policy.

A2 (1): We agree that the policy's success is fundamental to the method's overall success. However, the world-model component is significant.

  1. Without the dynamics loss, the direct policy averages 86.2% success across Push-T, Cube, Scene, and Puzzle, compared with 87.7% for the full LeWAM direct policy in the same evaluation (Appendix C.8, Table 13).
  2. The SR improves from test-time world model planning, compared with the direct policy.
  3. Even without a policy, in our Appendix C (Table 10), on Push-T and Cube, the pure world model CEM planning can get 84.0 ± 1.3% and 86.0 ± 3.2% success, respectively, which is competitive with LeWorldModel.
  4. Better linear probing results (average R² gains of +0.21 for proprioception, +0.20 for action, and +0.19 for inverse dynamics) show the benefits of joint training for interpretability.

W2 (2): There is no ablation without the dynamics loss, so the paper does not show that learning the dynamics helps the policy.

A2 (2): We added the dynamics-loss ablation for all ten tasks in Appendix C.8 (Table 13). Both variants execute the goal-conditioned policy directly. We found that

  1. Removing the dynamics loss lowers direct policy success on five of ten tasks. For example, success drops from 76.7% to 70.0% on Puzzle and from 100.0% to 86.7% on Reacher.

  2. Model-based planning through the learned dynamics improves success on play datasets such as Push-T (84.7% to 98.7%), Scene (90.0% to 97.3%), and Puzzle (76.7% to 80.7%). This is because diverse trajectories help the world model learn more informative dynamics. The expert datasets, such as DrawerCleanup and ToolHang, contain a narrower range of trajectories; hence, their direct policy success is similar with and w/o the dynamics part, since their dynamics part is less informative.

TaskLeWAM direct success (%)Without dynamics loss (%)LeWAM gradient planning (%)
Push-T84.7 ± 6.184.7 ± 5.098.7 ± 1.2
Cube99.3 ± 1.298.0 ± 2.0100.0 ± 0.0
Scene90.0 ± 5.392.0 ± 2.097.3 ± 1.2
Puzzle76.7 ± 8.170.0 ± 5.380.7 ± 11.0
Reacher100.0 ± 0.086.7 ± 7.0100.0 ± 0.0
PointMaze-Large100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Two-Room99.3 ± 1.298.0 ± 3.5100.0 ± 0.0
ToolHang92.7 ± 3.192.7 ± 4.294.7 ± 1.2
DrawerCleanup64.7 ± 5.868.7 ± 7.680.7 ± 4.2
Transport85.3 ± 4.284.0 ± 2.083.3 ± 6.1

W2 (3): Figure 6 reports no direct-policy results; direct success appears only for four tasks in Tables 9 and 10.

A2 (3): We add the direct-policy results for all 10 tasks in Appendix C.8 (Table 13) in our revised manuscript and the table in A2 (3).

The results show that LeWAM learns a policy prior that effectively guides the model-based test-time planning.

W3 (1): GCBC is trained for 20 instead of 100 epochs. A3 (1): We follow the setting of LeWM [1] and Stable-World-Model [2] for training GCBC, where 20 epochs are sufficient for convergence. W3 (2): The simulated Diffusion Policy is evaluated without EMA. A3 (2): While the original DP paper uses EMA, evaluating without EMA weights is also common (e.g., the LeRobot implementation until recently). We observed in our initial training that EMA tended to perform the same or slightly worse than without, and opted to use the best validation checkpoint instead. Retraining with EMA on toolhang, we achieve a success rate of 83.3 ± 1.2%, compared with 84.7 ± 3.1% for the checkpoint evaluated without EMA. W3 (3): On the robot, π0.5 is fine-tuned with its vision and language backbone frozen. A3 (3): We fine-tune pi0.5 with its VLM frozen, training only the action expert and its state/action projections. pi0.5 itself is trained with knowledge insulation, which blocks gradients from the continuous action expert from reaching the VLM. Since public fine-tuning code supports only the flow-matching head, we freeze the VLM to preserve its pretrained representations in the same spirit. Training only the action expert is a common VLA fine-tuning configuration (for example, SmolVLA [3]), and we expect it to be the more suitable regime at our data scale of 100 to 200 demonstrations per task. Even with the backbone frozen, pi0.5 has more trainable parameters than both LeWAM and our Diffusion Policy baseline, and it benefits from large-scale robot pretraining that neither of the other methods has. W4: The physical-robot experiment does not test planning: LeWAM runs as a goal-free direct policy, the learned dynamics are unused at test time, and the only robot-side evidence for them is the qualitative rollout probe in Fig. 15. A4: Thanks for pointing it out. We have stated this explicitly in the limitations section of the revised manuscript. The real-robot experiments evaluate LeWAM only in its direct-policy mode, testing whether a jointly trained JEPA world-action model remains deployable as a policy on physical tasks.

We did not run planning on the robot because preliminary offline planning experiments showed noticeable model exploitation, where the planner finds action sequences that score well under the learned dynamics but are not physically plausible. We hypothesize that this is due to our low-data regime of 100-200 demonstrations (13-34 minutes) per task, which yields a narrow distribution of demonstrations for the dynamics model. For reference, V-JEPA 2-AC, which builds on the same V-JEPA encoder family, trains a 300M-parameter action-conditioned predictor on 62 hours of robot data (23k trajectories) to plan on a real arm. We therefore regard this as a limitation of our data regime rather than of the architecture, and leave real-world planning to future work.

W5: On the robot, LeWAM reaches 33.3% against 38.9% for Diffusion Policy at almost the same compute (270.2 vs 270.0 GFLOPs), with 30 trials per task.

A5: We agree that LeWAM does not outperform Diffusion Policy as a direct policy, and we do not claim that it does. With 30 trials per task (90 per method), the gap of 30/90 vs 35/90 successes is within evaluation noise (two-proportion test, p = 0.44), so our results do not separate the two methods at matched compute. Our intention is not to present LeWAM as a state-of-the-art behavior cloning method, but to show that it can be applied in real environments. The experiment is a feasibility check: it tests whether a JEPA world-action model whose encoder is fine-tuned end-to-end with a joint dynamics objective can still be deployed as a policy on physical contact-rich manipulation tasks, which most JEPA world-model papers do not evaluate. The result indicates that the joint objective does not measurably degrade imitation performance relative to a policy-only method at equal inference cost (270.2 vs 270.0 GFLOPs). To avoid overclaiming, we have replaced "competitive" with "comparable" in the abstract and Section 4.4.

Q1 (1): What hardware, timing boundaries, and number of candidates were used for Table 8, and why does Figure 12 show about 0.2 s per plan at the same five refinement steps?

A6 (1): Table 6 reports H100 latency per planner call for LeWM (CEM, 300 candidates × 30 rounds) and LeWAM (five gradient steps). Figure 12 uses 32 candidates in its separate Push-T refinement-step sweep. See A1 (2), A1 (3), and Appendix B.3 for the timing scopes.

Q1 (2): Can a compute-matched CEM baseline be added?

A6 (2): This question repeats W1 (4); please see A1 (4).

Q1 (3): Can CEM initialized from LeWAM's policy samples be added?

A6 (3): Yes, definitely. We chose gradient-based planning because it is GPU-scalable and more efficient, but CEM can also benefit from our method. In a separate initialization ablation, we used the same checkpoint for both CEM variants within each task. At the same CEM iteration budget, initializing the search from the best policy proposal improved success on all six tasks below. We report the mean success across three evaluation seeds, each with 50 episodes.

TaskZero-init CEM success (%)Policy-init CEM success (%)Gradient planning (paper, %)
Push-T80.086.098.7
Two-Room98.099.3100.0
PointMaze-Large76.0100.0100.0
Cube74.699.3100.0
Transport4.062.083.3
DrawerCleanup6.716.080.7

Compared with policy-initialized CEM, gradient-based planning achieves higher reported success on these tasks.

Q2: Can a policy-only model trained without the dynamics loss be added, with direct-policy success reported on all ten tasks?

A7: This question repeats W2 (2) and W2 (3); please see A2 (2) and A2 (3).

Q3 (1): Can at least three training seeds be reported?

A8 (1): This question repeats W1 (5); please see A1 (5).

Q3 (2): App. D.1 says Table 9 uses epoch-30 checkpoints, while App. B keeps the lowest validation-loss checkpoint, yet both Figure 6 and Table 9 report Scene 97.3 and Puzzle 80.7. Which checkpoint was used, and which does Table 10 use?

A8 (2): This question repeats W1 (6); please see W1(6).

Q4: Which tasks are "contact-rich" in the abstract and Figure 1(a)? The 85.5% / 26.8% numbers correspond to four tasks, while Figure 5 shows six.

A9: The “contact-rich” includes 6 tasks, including cube, scene, puzzle, toolhang, transport, and drawer. We have corrected them in the manuscript.

Q5: On the real robot, LeWAM runs as a direct policy without goal conditioning or planning. Can goal-conditioned planning be reported on at least one real task?

A10: This question relates to W4; please see A4. Our preliminary offline planning experiments indicate that our current real-robot data scale is insufficient for reliable planning, and we were not able to collect the additional data required within the rebuttal period. We will state this in the limitations section and leave real-world planning to future work.

Minor (not affecting the score): Figure 1(d) compares compute only with π0.5; the Diffusion Policy (270.0 vs 270.2 GFLOPs) could also be shown.

A11: We added Diffusion Policy to Figure 1(d). Diffusion Policy and LeWAM both yield 3.70 action chunks per TFLOP at the displayed precision. Table 2 reports 270.0 and 270.2 GFLOPs per chunk, respectively.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of Rebuttal and Reviewer Discussion

Dear Area Chair,

During the rebuttal we addressed all concerns raised in the reviews:

  • Planning efficiency. We documented the timing hardware and boundaries (Reviewer Jfhi, A1(2)–(3) and A6(1); App. B.3, Table 6). We added three CEM comparisons: compute-matched CEM (Jfhi, A1(4)), policy-initialized CEM on six tasks (Jfhi, A6(3)), and CEM at matched latency (Reviewer xpZQ, A3; App. C.7, Table 12).
  • Training seeds. We report three training seeds on Push-T (Jfhi, A1(5); App. C.10, Table 15).
  • Role of the dynamics. We added a no-dynamics ablation with direct-policy results on all ten tasks (Jfhi, A2(2)–(3); App. C.8, Table 13).
  • Baselines.
    • LeWM retrained with its official implementation and shown next to its published scores (xpZQ, A4).
    • Diffusion Policy retrained with EMA on ToolHang (Jfhi, A3(2)).
    • The π0.5 fine-tuning setup explained (Jfhi, A3(3)).
    • Qantara and a goal-conditioned ACT-JEPA compared on all ten environments (Reviewer 2QiS, follow-up A1; xpZQ, A2(1)).
  • OOD robustness. We added a color-shift evaluation (2QiS, A3; App. C.9, Table 14).
  • Scope of claims.
    • Speedup scope corrected (Jfhi, A1(1)).
    • "Contact-rich" defined (Jfhi, A9).
    • "Competitive" changed to "comparable" for the robot results (Jfhi, A5).
    • The abstract revised to reflect the policy's role (xpZQ, A1(3)).
    • The single-stage claim is now distinguished from Qantara and ACT-JEPA (xpZQ, A2(2)).
    • Robot planning stated as a limitation (Jfhi, A4 and A10).

In the follow-up discussion:

  • Reviewer Jfhi acknowledged that "the overall technique is sound" and the improvement LeWAM brings in simulation, and kept a rating of 2 because the dynamics loss gives only a modest direct-policy gain. We agreed the gain is modest, and explained that the dynamics' main role is test-time planning (+7.5 pp over the same checkpoint's policy). With planning, LeWAM outperforms the no-dynamics policy on eight of ten tasks (93.5% vs. 87.5%, Figure 6 and Table 13). We also explained the Reacher correction.
  • Reviewer 2QiS's follow-up came before our comparison with Qantara and ACT-JEPA. We then posted that comparison and pointed to the OOD evaluation already in our response. No further comment was posted by the reviewer.
  • Reviewer xpZQ (weak accept) has not commented since our response.

We have incorporated the requested changes into the revised manuscript and will add the Qantara and ACT-JEPA comparison to the final version.

We believe the revised manuscript and the responses above address the substantive concerns raised in the reviews. We thank the AC and the reviewers for their efforts in improving the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.