acceptodds
Under review as a conference paper at ICLR 2027

LS-JEPA: Learning Long Short-Term Dynamics for Fast Latent Planning

Abstract

Joint-Embedding Predictive Architecture (JEPA) offers a promising route to model-based control by predicting future latent dynamics. Current approaches mainly adopt one-step prediction objectives, which fail to capture the long-term latent transitions for action recovery. In this work, we introduce LS-JEPA, an end-to-end latent world model that jointly learns forward and inverse dynamics across Long Short-term temporal scales. Specifically, LS-JEPA combines one-step prediction with fixed-horizon autoregressive rollout, while an endpoint-conditioned action decoder reconstructs both individual actions and multi-step action sequences from latent displacements. These complementary objectives encourage predictive, action-relevant representations and directly connect latent transitions to control. Furthermore, the learned one-step and multi-step inverse mappings can be naturally used for fast latent planning during the inference time. We propose a Test-Time Replanning (TTR) strategy in which a goal-directed action sequence is decoded and then a one-step correction from a fresh observation is applied to refine the action. Extensive experiments are conducted to validate our proposed approach. The results show that LS-JEPA achieves the state-of-the-art results with 96.00%, 100.00%, 99.67%, and 100.00% on Push-T, OGBench-Cube, Reacher, and Two-Room, respectively, yielding a macro-average of 98.92% with only 11 ms of planning time per decision. These results suggest that jointly learning Long Short-Term forward and inverse dynamics with our proposed TTR strategy enjoys both the accuracy and efficiency for latent planning. Code is available at https://anonymous.4open.science/r/LS-JEPA-E034/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Model, JEPA, long short-term

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_926sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The core of this article is optimizing a model that generates latent representations using two modules. Specifically, a JEPA-style model predicts future observations at the latent level, and then supervises the generation of latent representations through two pathways, FD and ID, ensuring that the latent representations not only have the ability to predict the future but also learn changes that require actions, similar to the recently popular WAM. A loss is provided for both single-step actions and action chunks for compatibility, allowing the model to balance long-term and short-term understanding and predictive abilities. During inference, the TTR method is used, leveraging the trained FD and ID models by correcting actions only at the final step, balancing effectiveness and speed. In the experimental section, four simulation scenarios Push-T, Reacher, Cube, and Two-Room were tested, and a series of ablation experiments successfully validated the differences between long-time and short-time predictions as well as the advantages of the TTR method.

Critical strengths

Firstly, the idea of jointly optimizing the latent representation model through FD and ID is innovative. The combination of FD and ID allows more comprehensive supervision of the model, enabling it to possess both the policy's action generation capability and the WM's future prediction ability. This section also integrates long-term and short-term dynamics, allowing the model to simultaneously learn to manage short-term states and long-term objectives. Secondly, the TTR method used during testing performed very well, achieving a success rate of 98.92%, significantly higher than other methods. This indicates that TTR can form a closed loop during inference, providing immediate feedback to the model and promptly adjusting the policy. Moreover, this part includes extensive comparative experiments with other methods, highlighting the significant advantages of TTR. Finally, the overall article is logically clear, and no errors or signs of falsification were found in the experimental section.

Critical weaknesses

The experimental section relies on a single platform, namely LeWorldModel, and only includes four tasks. It lacks evaluation results for policies on popular benchmarks such as Libero, Robotwin, and Robocasa, as well as metrics for WM capabilities like video clarity and future prediction. Moreover, the difficulty of the evaluation tasks appears low, failing to highlight the model's long-term and complex reasoning abilities. In addition, there are no real-world experiments, and simulation results cannot demonstrate good S2R performance in actual environments. Another minor issue is the absence of ablation studies removing FD while keeping only ID or removing ID while keeping only FD, leaving unclear whether these two modules are complementary. Therefore, the impact of each component within the entire framework is not fully clear. Overall, I consider the experimental section insufficiently solid.

Questions for authors
  1. What is the relationship between the FD and ID modules? Are there any additional experiments that can clarify this issue?
  2. Are there experimental results for other simulation benchmarks? For example, more common ones like Libero, Robotwin, or Robocasa.
  3. Could you provide some real-world experiments, even if they are simple PnP tasks? At the very least, it should demonstrate that the method proposed in the paper works on real hardware, rather than only being applicable in simulation. It would be even better if experiments could be conducted for more complex long-term tasks or those related to instruction understanding/memory.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 926s (Part 1/2)

Dear Reviewer 926s,

Thank you for appreciating our work and your valuable comments. We answer the raised questions below. We have included all discussions and results in our revised manuscript (blue content).

1. Evaluation on more complex tasks

Thank you for your valuable comment. To further validate the proposed model and learning principles under more challenging conditions, we evaluated LS-JEPA on the LIBERO benchmark. Since our original framework was designed for goal-conditioned tasks, we made minor architecture adaptations for the LIBERO setting, where goal images are not available. Specifically, we incorporated a future-action predictor that enables the model to generate future action sequences conditioned on either historical observations or task instructions, while preserving the core components and learning principles of LS-JEPA. The experimental results are summarized in Table 1. Results for the other methods are taken from TC-LeWM.

Table 1. Comparison with other methods on LIBERO benchmark.

MethodModel SizeSpatialObjectGoalLongMean
Diffusion Policy (scratch)-78.392.568.350.572.4
Octo (fine-tuned)93M78.985.784.651.175.1
OpenVLA (fine-tuned)7.54B84.788.479.253.776.5
Raw LeWM103M61.6 ± 4.346.4 ± 12.874.8 ± 6.229.9 ± 8.353.2
TC-LeWM103M68.6 ± 1.887.8 ± 3.286.0 ± 1.151.8 ± 9.273.6
LS-JEPA (ours)31M97.8 ± 2.3998.2 ± 1.7598.4 ± 1.2682.2 ± 12.9194.15

It can be seen from Table 1 that our method significantly outperform other methods on the LIBERO benchmark, with around 20% improvment compared to TC-LeWM. This is because our method can jointly learn multi-scale forward and inverse dynamics, thereby improving the quality of the generated future action sequences and making it more suitable for handling complex, long-horizon tasks. We have include the results in Table 2 and the discussion in the manuscript (blue content).

To further assess whether our proposed inverse dynamics (ID) objective transfers to other world-model frameworks, we evaluate a LeWAM-based variant, using LeWAM [a] as the baseline and adding our ID objective to it. The results are summarized below:

Table 2. Resuls of LeWAM and our proposed ID.

Transfer experimentSpatialObjectGoalLongMean
LeWAM baseline78.20%81.80%88.00%73.60%80.40%
LeWAM + ID84.50% (+6.30)84.10% (+2.30)91.00% (+3.00)81.40% (+7.80)85.25% (+4.85)

We have include the reaults in the appendix (blue content).

AuthorsOpenReview ↗

Response to Reviewer 926s (Part 2/2)

2. Visual fidelity and future-prediction quality

Thank you for your valuable comment. To evaluate LS-JEPA's video reconstruction quality and future prediction capability, we freeze the encoder and train an image decoder to reconstruct images from its CLS representations. We decode the predicted latent state at each horizon and compute PSNR against the ground-truth frame at 5, 25, or 50 environment steps ahead. PSNR is measured in decibels (dB), and higher values indicate closer agreement with the ground truth. The results are shown in Table 3.

Table 3. Resuls of generated video fidelity.

TaskModelRollout PSNR @5 ↑Rollout PSNR @25 ↑Rollout PSNR @50 ↑Gap 50 ↓
TwoRoomLeWM23.0921.6320.5226.01
TwoRoomLS-JEPA41.01 (+17.92)40.60 (+18.97)39.07 (+18.55)5.51 (-20.50)
PushTLeWM31.0127.8825.237.16
PushTLS-JEPA29.86 (-1.15)27.95 (+0.07)25.72 (+0.49)5.51 (-1.65)
CubeLeWM28.2827.6826.421.76
CubeLS-JEPA29.29 (+1.01)29.24 (+1.56)28.76 (+2.34)0.28 (-1.48)
ReacherLeWM38.7234.3432.149.39
ReacherLS-JEPA37.27 (-1.45)36.03 (+1.69)34.67 (+2.53)5.94 (-3.45)

It can be seen from the table that our model demonstrates better performance, which is particularly evident from the prediction results on the TwoRoom task, where our method achieves consistent PSNR improvements of approximately 18 dB over LeWM across all prediction horizons.

We have included the results and discussion in Table 16 in the appendix(blue content). We have also included the future prediction visual results in the appendix (Figures 18–21).

3. Relative roles of forward dynamics and inverse dynamics

Thank you for the question. Forward Dynamics (FD) and Inverse Dynamics (ID) serve different but complementary purposes. FD learns to predict the future latent state from the current latent state and an action, thereby modeling how actions change the environment. ID infers action information from latent transitions; this both constrains the representation to encode control-relevant information and provides the action-recovery mechanism used during deployment. We conducted ablations that retain only one of these objectives to evaluate the effectiveness of each component as shown in Table 4.

Table 4. Effectivenss of FD and ID

Training objectivePushTTwoRoomCubeReacherMacro
FD only (without ID)3.00 ± 0.8232.00 ± 3.2747.00 ± 2.4511.00 ± 0.8223.25 ± 1.06
ID only (without FD)92.33 ± 2.1695.33 ± 0.47100.00 ± 0.0097.33 ± 0.4796.25 ± 0.56
FD + ID96.00 ± 3.00100.00 ± 0.00100.00 ± 0.0099.67 ± 0.5898.92 ± 0.76

With FD only (i.e., removing ID), success rates fall sharply, which is consistent with a failure to learn sufficiently informative control representations in the absence of additional anti-collapse regularization. With ID only (i.e., removing FD), performance remains relatively strong but is lower than with joint learning, indicating that forward prediction contributes useful structure for predictability and reachability.

We have included the results in Table 5 and the discussion in the manuscript (blue content).

4. Real-robot evaluation

Thank you for the valuable comment. Following prior works such as LeWM, TC-LeWM, and INTACT, which evaluate their methods only in simulation, we likewise conduct experiments in simulation. We plan to evaluate our approach on more complex real-world manipulation tasks in the future. Indeed, we have designed several real-world-oriented components to guide the model in completing more complex, long-horizon tasks. For example, since ground-truth goal images are generally unavailable in real-world settings, we plan to introduce an additional future-action-sequence predictor, trained in parallel with the original framework, to enable long-horizon action generation. Preliminary experiments on LIBERO (Answer 1) suggest that this design is promising, but we have not yet deployed it in real-world settings; we leave real-world validation to future work.

Reference

[a] Fu, Minghao, et al. "LeWAM: End-to-End World and Action Modeling with JEPAs." arXiv preprint arXiv:2610.12818 (2026).

Reviewer_Ci72OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LS-JEPA to address three challenges: 1. learning representations that retain action information; 2. limiting errors over repeated predictions; 3. reducing planning time. It jointly trains a world model to predict future states and an action decoder to recover the actions connecting two states, using both short and longer sequences. At test time, the decoder generates a fixed-length action sequence directly. After executing most of the sequence, the agent takes a fresh observation and replaces the final action block with a correction toward the goal. This allows control without searching through candidate action sequences.

Critical strengths
  1. The method gets good performance with little computation during execution. The comparison with Decoder-Guided CEM is also interesting: both receive feedback before the final block, but TTR achieves higher average success with lower reported planning time.

  2. The ablations are also comprehensive. They demonstrate that adding either long-term forward or inverse supervision improves performance, and using both raises average success as well.

Critical weaknesses
  1. Learning actions from a starting state and a future goal has been studied before. I am therefore unclear about what new insight comes from using the difference between endpoint representations to generate an action sequence. The paper would benefit from explaining more directly how this design differs from previous approaches and why that difference helps action generation. Or is proposed mechanism specifically targeted at certain weakness of specific architecture?

  2. The predictor is trained by feeding its own predicted states back into subsequent predictions and comparing them with the corresponding observed states. This gives it practice handling its own prediction errors, but the general idea is familiar from multi-step rollout training.

  3. Observing the actual state and updating the remaining action plan follows a familiar feedback-control principle. TTR provides a lightweight way to do this through the learned decoder, and the results show that the correction helps. However, I am still unclear about the reason for regenerating the plan only before the last block. What advantage does this timing offer over correcting earlier or replanning more frequently? Can authors give a design motivation, or is it primarily motivated by empirical findings.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ci72 (Part 1/2)

Dear Reviewer Ci72,

Thank you for your valuable comments. We answer the raised questions below. We have included all discussions and results in our revised manuscript (blue content).

1. Recover actions from endpoint representation differences

Thank you for the question. Our choice to use the difference between endpoint representations to generate an action sequence is primarily motivated by our empirical findings. We have conducted an ablation study on the decoder input design and reported the results for Push-T in Table 8 of the Appendix. As suggested, we have further conducted a more detailed ablation study across four tasks under different deployment strategies, as shown in Table 1 below. These results have been included in Table 7 of the revised manuscript, together with additional discussion (blue content).

Table 1. Effectiveness of different decoder input design

DeploymentRepresentationPush-TTwo-RoomCubeReacherMacro
LS-JEPA + Pure CEM 300×30[zt,zg][z_t, z_g]91.33 ± 3.5198.33 ± 0.4778.33 ± 0.4784.67 ± 0.9488.17
Δz\Delta z94.00 ± 2.00 (+2.67)100.00 ± 0.00 (+1.67)80.67 ± 3.06 (+2.34)86.00 ± 6.00 (+1.33)90.17 (+2.00)
LS-JEPA + TTR[zt,zg][z_t, z_g]94.00 ± 0.8299.67 ± 0.47100.00 ± 0.0096.67 ± 0.4797.59
Δz\Delta z96.00 ± 3.00 (+2.00)100.00 ± 0.00 (+0.33)100.00 ± 0.00 (+0.00)99.67 ± 0.58 (+3.00)98.92 (+1.33)

As shown in Table 1, using endpoint representation differences consistently yields better or comparable performance across all four tasks under both CEM and TTR deployment strategies. Specifically, Δz\Delta z improves the macro-average success rate by 2.00% and 1.33% under CEM and TTR, respectively. These results further demonstrate the effectiveness of endpoint representation differences for action sequence generation, and we therefore adopt this design in our method.

In addition, this choice is also motivated by modeling considerations. The difference between endpoint representations explicitly encodes the change over a given transition interval, providing a more focused signal for inferring the actions that produced that change. Conversely, requiring action recovery from these differences encourages the latent representations to preserve action-relevant transition information. This creates a beneficial coupling between inverse-dynamics learning and representation learning.

2. Multi-step rollout training

Thank you for your comment. Multi-step rollout strategies have been studied in reinforcement learning and model-based prediction, but they have received relatively little attention within JEPA-style world models. The most closely related work is V-JEPA 2 [a], which considers a two-step rollout objective in practical settings. In this work, we adopt a multi-step rollout objective with T=5, validate its effectiveness across a range of tasks, and observe consistent improvements.  In addition, we build on this training objective and introduce the TTR training strategy to further improve performance. We view these two design choices as the main differences between LS-JEPA and prior methods. We have included the dicussion in related work in the manuscript (blue content).

AuthorsOpenReview ↗

Response to Reviewer Ci72 (Part 2/2)

3. Replanning frequency

Thank you for your question. In the following, we first explain why we choose to regenerate the plan only before the final chunk. We then report an experiment with a higher replanning frequency in Table 2.

Why replan only the final chunk? In the original training configuration, the action decoder was trained with action chunk sizes H = 1 and H = 5, allowing it to recover either one or five chunks at inference time. Executing all five predicted chunks without feedback may accumulate control errors. TTR therefore adopts a lightweight compromise: it executes the first four chunks and replans before executing the final chunk, reducing open-loop error without replanning after every action chunk.

Comparison with more frequent replanning. We evaluated a setting that supports action recovery for H = 1, 2, 3, 4, and 5, using a one-chunk-at-a-time execution strategy that replans after each chunk. We also tested decoder-guided CEM optimization on top of this strategy. The corresponding success rates and execution latencies are summarized in Table 2.

Table 2. Effect of replanning frequency on success rate and latency.

Execution / planning strategyPushTReacherCubeTwoRoomMacroLatency (s)
One-Step Refinement(replan after each chunk)95.0099.67100.00100.0098.6731
TTR (final-chunk replanning)96.0099.67100.00100.0098.9211
Decoder-Guided One-Step Refinement(replan after each chunk)99.00100.00100.00100.0099.75139

Table 2 shows that frequent one-chunk replanning incurs additional planning overhead and is slightly less effective than TTR in our experiments. Decoder-guided One-Step Refinement further improves success rates but also introduces search costs. Considering both control performance and computational efficiency, we adopt TTR as a practical trade-off between success rate and computational cost.

We have included these results in Table 1 (performance) and in Table 3 (latency) in the manuscript (blue content). We also revise the descriptionof these different planning strategies in Section 3.3 in the manuscipt (blue content).

Reference

[a] Assran, Mahmoud, et al. "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning." arXiv preprint arXiv:2506.09985 (2025).

Thank you for the detailed response and for the additional experiments. W3 is now resolved.

I have a remaining concern about the scope of the method. The paper's main contribution is to improve JEPA-style world models for costly test-time search and for rollout errors. This is well motivated for this architecture, but it would be stronger to also explain why this is important beyond JEPA-style world models, e.g., for JEPA world models to solve challenging tasks that other architectures can't solve. The LIBERO table is already a good step in this direction.

Ideally, a future version of the paper would test LS-JEPA on a task that remains difficult for recent methods from other architectures but can be solved by LS-JEPA. I hope these comments are helpful.

AuthorsOpenReview ↗

Response to Reviewer Ci72

Dear Reviewer Ci72,

Thank you for your follow-up comments and for acknowledging that W3 has been resolved.

Regarding your question, we summarize our rationale below:

  1. Why JEPA? We chose JEPA as our primary world model due to its potential to be efficiently applied in real-world robot manipulation tasks.

  2. Generalizability to other architectures. As you noted, our method can be applied to architectures beyond just JEPA-based ones. The LIBERO experiment already serves as a good step in this direction. Our main contribution is investigating how learning short- and long-term dynamics can better connect predictive representations with action generation. A JEPA-based architecture provides a solid starting point due to its aforementioned potential. However, we argue that our method is broadly applicable. For instance, Fast-WAM [a] combines a pretrained video diffusion backbone with flow-matching-based action generation. Our proposed multi-scale inverse dynamics supervision could potentially complement such frameworks by encouraging action-relevant latent representations across different temporal scales. By integrating short- and long-horizon inverse dynamics objectives with existing flow-matching (FM) losses, representation learning and action generation can be jointly optimized without altering the original action-generation mechanism.

We will include these discussions in the revised manuscript upon acceptance.

Thank you again for your constructive feedback.

Reference

[a] Tianyuan Yuan, et al. "Fast-WAM: Do World Action Models Need Test-time Future Imagination?" arXiv preprint arXiv:2603.16666 (2026).

Reviewer_4FLhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes learning long- and short-term dynamics for latent planning. The proposed mechanism is simple and reasonable, and the broad experimental comparisons provide meaningful support for it. However, the paper lacks careful per-component grounding in the related literature and does not provide long-horizon comparisons where the method could be particularly useful.

Critical strengths
  1. The paper explores an important problem in planning with JEPA-based models: extensive action search. It proposes a way to make learned representations useful for control. The method is simple and easy to understand, and its overall design is reasonable.

  2. The reported results are strong and provide evidence to proposed method benefits, both in terms of quality and speed.

Critical weaknesses
  1. In its current form, the method appears incremental. Prior work explores similar long-term representations for planning in both flat models, such as [1] (similar approach, similar class of problem), and hierarchical models, such as [2] (they use different interface but solve similar class of problem). A deeper discussion on each detail and idea proposed by method would substantially strengthen the paper.

  2. The evidence for long-horizon planning is limited. Table 4 includes 50-step goals, but this comparison uses CEM and a single seed. The main TTR evaluation uses 25-step goals, matching the training horizon. Longer-horizon TTR comparisons would help establish the method’s advantages.

[1] - Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

[2] - Hierarchical Planning with Latent World Models

Questions for authors

My main questions concern weaknesses W1 and W2. I am willing to raise my score if the distinctions from prior work are clarified and the evidence for long-horizon TTR is strengthened.

It would also be useful to see an ablation removing both short- and long-horizon forward supervision while keeping the encoder, action decoder, training data, and TTR procedure fixed

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4FLh (Part 1/2)

Dear Reviewer 4FLh,

Thank you for your valuable comments. We answer the raised questions below. We have included all discussions and results in our revised manuscript (blue content).

1. Contributions recap

We provide a detailed comparison between prior methods (i.e., Delta-JEPA [a] and Hierarchical Planning with Latent World Models (HWM) [b]) and ours in related work in the revised manuscript (blue content). The relationship between ours and these methods is summarized as follows:

Similarity

We agree that our work shares some similarities with prior flat and hierarchical methods (i.e., Delta-JEPA [a] and HWM [b]). Specifically, Delta-JEPA [a] and our LS-JEPA both employ endpoint differences to recover actions. HWM [b] and our method both aim to capture longer-horizon task information.

Differences

  • We learn dynamics-aware representations differently. Previous flat models, such as Delta-JEPA [a], mainly employ inverse dynamics between adjacent latent representations (i.e., H = 1) to preserve local action-related information. Unlike these methods, our LS-JEPA jointly learns multi-scale forward and inverse dynamics, where the action decoder recovers both short-term action chunks between adjacent latent states and long-term action sequences between distant latent states. This enables the learned representations to capture both local dynamics and long-term temporal dependencies, rather than focusing only on adjacent state transitions.

  • We model long-term dependencies differently. Previous hierarchical models, such as HWM [b], employ a coarse-to-fine planning strategy, where a high-level planner generates intermediate latent subgoals to guide low-level planning. Unlike these methods, our LS-JEPA directly recovers the intervening action sequences between distant latent representations, encouraging the encoder to preserve long-term dynamics information. During execution, the learned action decoder directly generates action sequences conditioned on the current and goal representations, without requiring explicit hierarchical subgoal generation.

  • We generate actions for goal-reaching tasks differently. Previous methods, such as Delta-JEPA [a] and HWM [b], rely on CEM for action planning. In contrast, our LS-JEPA jointly trains the encoder, forward predictor, and shared action decoder in a unified framework. The same action decoder is directly employed to generate future action sequences during execution, while TTR further refines the final action chunk based on updated observations. Therefore, our method integrates representation learning and policy learning into a single-stage training framework, without requiring additional policy fitting or online CEM search.

AuthorsOpenReview ↗

Response to Reviewer 4FLh (Part 2/2)

2. Long-horizon planning and robustness

Thank you for your valuable comments. To further validate the advantages of our method in long-horizon planning, and following your suggestions, we have conducted experiments comparing LeWM with CEM and our method on 50-step goals under three settings.

Specifically, We use the same CEM planning framework for LeWM and LS-JEPA to isolate the effect of the learned world-model representation and dynamics, rather than confounding the comparison with different planners. We further evaluate LS-JEPA with TTR under the same 50-step goal-reaching setting to investigate its performance on longer-horizon tasks. To examine whether TTR can effectively scale to longer horizons, we additionally train a TTR model with an extended training horizon (T = 10 instead of T = 5) and evaluate it on 50-step goals. All experiments are performed with three different random seeds. We summarized the results in Table 1 and have included the results in Table 6 (blue content) in the manuscript.

Table 1. Goal-reaching success rates on 50-step goals

Evaluation settingTraining Horizon (T)PushTTwoRoomCubeReacherMacro
LeWM + CEM (50-step goals)136.00 ± 5.5763.33 ± 8.3348.33 ± 3.5189.67 ± 1.5359.33 ± 2.66
LS-JEPA + CEM (50-step goals)552.00 ± 2.00100.00 ± 0.0058.67 ± 3.0692.67 ± 1.1575.84 ± 0.96
LS-JEPA + TTR (50-step goals)521.33 ± 1.1596.00 ± 0.0065.33 ± 1.1599.67 ± 0.5870.58 ± 0.43
LS-JEPA + TTR (50-step goals)1078.00 ± 1.73100.00 ± 0.00100.00 ± 0.0095.67 ± 1.5393.42 ± 0.58

On 50-step goals, LS-JEPA outperforms LeWM on all four tasks under the same Pure CEM planner (300×30), with macro success rates of 75.83% and 59.33%, respectively. With TTR (T = 5), LS-JEPA achieves a macro success rate of 70.58% on 50-step goals; retraining with T = 10 raises this to 93.42%, with the largest gains on Push-T and Cube.

3. Contribution of forward dynamics

Thank you for your valuable comment. Following your suggestions, we conducted an ablation study on removing the forward dynamics (FD) objective and retaining only inverse dynamics (ID) with TTR. These results are summarized in Table 2 and we have included the results in Table 5 (blue content) in the manuscript.

Table 2. Effectiveness of ID without FD

Training objectivePushTTwoRoomCubeReacherMacro
ID92.33 ± 2.1695.33 ± 0.47100.00 ± 0.0097.33 ± 0.4796.25 ± 0.56
ID+FD96.00 ± 3.00100.00 ± 0.00100.00 ± 0.0099.67 ± 0.5898.92 ± 0.76

Across multiple seeds, removing both short- and long-horizon forward supervision reduces the success rates to 92.33% on Push-T, 95.33% on TwoRoom, 100.00% on Cube, and 97.33% on Reacher, compared with the full model. These results indicate that inverse action recovery alone does not fully replace forward prediction. The FD objective requires the model to predict how actions change the world, encouraging representations to retain dynamics-relevant information and to remain predictable over time. Joint FD–ID learning therefore provides complementary training signals for representation quality and control performance. We have added this discussion in the manuscript (blue content).

Reference

[a] Zhenghao Zhang, et al. "Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding." arXiv preprint arXiv:2606.31232 (2026).

[b] Wancong Zhang, et al. "Hierarchical Planning with Latent World Models." arXiv preprint arXiv:2604.03208 (2026).

Thank you for additional experiments and clarifications. I will raise my score.

AuthorsOpenReview ↗

Response to Reviewer 4FLh

Dear Reviewer 4FLh,

Thank you very much for your positive feedback and for reconsidering your assessment of our work. We sincerely appreciate your constructive suggestions, which have helped us significantly strengthen the paper.

Thank you again for your time and thoughtful evaluation.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our rebuttal and discussion

Dear Reviewers and ACs,

We sincerely appreciate all reviewers for their time and efforts in reviewing our paper. We are glad to find that reviewers recognized the following contributions of our work:

  • An important problem for the JEPA-based planning community. The paper explores an important problem in planning with JEPA-based models: extensive action search. It proposes a way to make learned representations useful for control. [4FLh, 926s]
  • An efficient and effective method supported by extensive comparisons and ablations with impressive results. The reported results are strong and provide evidence to proposed method benefits, both in terms of quality and speed. The method gets good performance with little computation during execution. The ablations are also comprehensive. The idea of jointly optimizing the latent representation model through FD and ID is innovative. The TTR method used during testing performed very well, achieving a success rate of 98.92%, significantly higher than other methods. [4FLh, Ci72, 926s]
  • Well-written paper and clearly-presented methodology: The method is simple and easy to understand, and its overall design is reasonable. The overall article is logically clear, and no errors or signs of falsification were found in the experimental section. [4FLh, 926s]

We also thank all reviewers for their insightful and constructive suggestions, which help further improve our paper. In addition to the pointwise responses below, we summarize the major revision in the rebuttal according to the reviewers’ suggestions.

  • Extended experiments:  ([4FLh, Ci72, 926s]) We further validate our methods advantages in long-horizon planning by conducting experiments to compare 50-step goals LeWM with CEM to our methods under three different settings. We compare forward-only, inverse-only, and joint training. We compare two decoder inputs across four tasks. We also compare different replaning frequency. We further demostrate the effectiveness of our method on the more complex LIBERO benchmark.
  • Manuscript update: We add more experimental results and discussions in the revised manuscript. We clarify how LS-JEPA differs from Delta-JEPA and from hierarchical planning with latent world models.We add visual fidelity and future prediction quality in the appendix. The updated contents are included in our revision of the paper (content in blue).

We really thank all reviewers’ and ACs’ time and efforts again.

Best wishes,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.