- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The core of this article is optimizing a model that generates latent representations using two modules. Specifically, a JEPA-style model predicts future observations at the latent level, and then supervises the generation of latent representations through two pathways, FD and ID, ensuring that the latent representations not only have the ability to predict the future but also learn changes that require actions, similar to the recently popular WAM. A loss is provided for both single-step actions and action chunks for compatibility, allowing the model to balance long-term and short-term understanding and predictive abilities. During inference, the TTR method is used, leveraging the trained FD and ID models by correcting actions only at the final step, balancing effectiveness and speed. In the experimental section, four simulation scenarios Push-T, Reacher, Cube, and Two-Room were tested, and a series of ablation experiments successfully validated the differences between long-time and short-time predictions as well as the advantages of the TTR method.
Critical strengths
Firstly, the idea of jointly optimizing the latent representation model through FD and ID is innovative. The combination of FD and ID allows more comprehensive supervision of the model, enabling it to possess both the policy's action generation capability and the WM's future prediction ability. This section also integrates long-term and short-term dynamics, allowing the model to simultaneously learn to manage short-term states and long-term objectives. Secondly, the TTR method used during testing performed very well, achieving a success rate of 98.92%, significantly higher than other methods. This indicates that TTR can form a closed loop during inference, providing immediate feedback to the model and promptly adjusting the policy. Moreover, this part includes extensive comparative experiments with other methods, highlighting the significant advantages of TTR. Finally, the overall article is logically clear, and no errors or signs of falsification were found in the experimental section.
Critical weaknesses
The experimental section relies on a single platform, namely LeWorldModel, and only includes four tasks. It lacks evaluation results for policies on popular benchmarks such as Libero, Robotwin, and Robocasa, as well as metrics for WM capabilities like video clarity and future prediction. Moreover, the difficulty of the evaluation tasks appears low, failing to highlight the model's long-term and complex reasoning abilities. In addition, there are no real-world experiments, and simulation results cannot demonstrate good S2R performance in actual environments. Another minor issue is the absence of ablation studies removing FD while keeping only ID or removing ID while keeping only FD, leaving unclear whether these two modules are complementary. Therefore, the impact of each component within the entire framework is not fully clear. Overall, I consider the experimental section insufficiently solid.
Questions for authors
- What is the relationship between the FD and ID modules? Are there any additional experiments that can clarify this issue?
- Are there experimental results for other simulation benchmarks? For example, more common ones like Libero, Robotwin, or Robocasa.
- Could you provide some real-world experiments, even if they are simple PnP tasks? At the very least, it should demonstrate that the method proposed in the paper works on real hardware, rather than only being applicable in simulation. It would be even better if experiments could be conducted for more complex long-term tasks or those related to instruction understanding/memory.
2 comments
Response to Reviewer 926s (Part 2/2)
2. Visual fidelity and future-prediction quality
Thank you for your valuable comment. To evaluate LS-JEPA's video reconstruction quality and future prediction capability, we freeze the encoder and train an image decoder to reconstruct images from its CLS representations. We decode the predicted latent state at each horizon and compute PSNR against the ground-truth frame at 5, 25, or 50 environment steps ahead. PSNR is measured in decibels (dB), and higher values indicate closer agreement with the ground truth. The results are shown in Table 3.
Table 3. Resuls of generated video fidelity.
| Task | Model | Rollout PSNR @5 ↑ | Rollout PSNR @25 ↑ | Rollout PSNR @50 ↑ | Gap 50 ↓ |
|---|---|---|---|---|---|
| TwoRoom | LeWM | 23.09 | 21.63 | 20.52 | 26.01 |
| TwoRoom | LS-JEPA | 41.01 (+17.92) | 40.60 (+18.97) | 39.07 (+18.55) | 5.51 (-20.50) |
| PushT | LeWM | 31.01 | 27.88 | 25.23 | 7.16 |
| PushT | LS-JEPA | 29.86 (-1.15) | 27.95 (+0.07) | 25.72 (+0.49) | 5.51 (-1.65) |
| Cube | LeWM | 28.28 | 27.68 | 26.42 | 1.76 |
| Cube | LS-JEPA | 29.29 (+1.01) | 29.24 (+1.56) | 28.76 (+2.34) | 0.28 (-1.48) |
| Reacher | LeWM | 38.72 | 34.34 | 32.14 | 9.39 |
| Reacher | LS-JEPA | 37.27 (-1.45) | 36.03 (+1.69) | 34.67 (+2.53) | 5.94 (-3.45) |
It can be seen from the table that our model demonstrates better performance, which is particularly evident from the prediction results on the TwoRoom task, where our method achieves consistent PSNR improvements of approximately 18 dB over LeWM across all prediction horizons.
We have included the results and discussion in Table 16 in the appendix(blue content). We have also included the future prediction visual results in the appendix (Figures 18–21).
3. Relative roles of forward dynamics and inverse dynamics
Thank you for the question. Forward Dynamics (FD) and Inverse Dynamics (ID) serve different but complementary purposes. FD learns to predict the future latent state from the current latent state and an action, thereby modeling how actions change the environment. ID infers action information from latent transitions; this both constrains the representation to encode control-relevant information and provides the action-recovery mechanism used during deployment. We conducted ablations that retain only one of these objectives to evaluate the effectiveness of each component as shown in Table 4.
Table 4. Effectivenss of FD and ID
| Training objective | PushT | TwoRoom | Cube | Reacher | Macro |
|---|---|---|---|---|---|
| FD only (without ID) | 3.00 ± 0.82 | 32.00 ± 3.27 | 47.00 ± 2.45 | 11.00 ± 0.82 | 23.25 ± 1.06 |
| ID only (without FD) | 92.33 ± 2.16 | 95.33 ± 0.47 | 100.00 ± 0.00 | 97.33 ± 0.47 | 96.25 ± 0.56 |
| FD + ID | 96.00 ± 3.00 | 100.00 ± 0.00 | 100.00 ± 0.00 | 99.67 ± 0.58 | 98.92 ± 0.76 |
With FD only (i.e., removing ID), success rates fall sharply, which is consistent with a failure to learn sufficiently informative control representations in the absence of additional anti-collapse regularization. With ID only (i.e., removing FD), performance remains relatively strong but is lower than with joint learning, indicating that forward prediction contributes useful structure for predictability and reachability.
We have included the results in Table 5 and the discussion in the manuscript (blue content).
4. Real-robot evaluation
Thank you for the valuable comment. Following prior works such as LeWM, TC-LeWM, and INTACT, which evaluate their methods only in simulation, we likewise conduct experiments in simulation. We plan to evaluate our approach on more complex real-world manipulation tasks in the future. Indeed, we have designed several real-world-oriented components to guide the model in completing more complex, long-horizon tasks. For example, since ground-truth goal images are generally unavailable in real-world settings, we plan to introduce an additional future-action-sequence predictor, trained in parallel with the original framework, to enable long-horizon action generation. Preliminary experiments on LIBERO (Answer 1) suggest that this design is promising, but we have not yet deployed it in real-world settings; we leave real-world validation to future work.
Reference
[a] Fu, Minghao, et al. "LeWAM: End-to-End World and Action Modeling with JEPAs." arXiv preprint arXiv:2610.12818 (2026).
Response to Reviewer 926s (Part 1/2)
Dear Reviewer 926s,
Thank you for appreciating our work and your valuable comments. We answer the raised questions below. We have included all discussions and results in our revised manuscript (blue content).
1. Evaluation on more complex tasks
Thank you for your valuable comment. To further validate the proposed model and learning principles under more challenging conditions, we evaluated LS-JEPA on the LIBERO benchmark. Since our original framework was designed for goal-conditioned tasks, we made minor architecture adaptations for the LIBERO setting, where goal images are not available. Specifically, we incorporated a future-action predictor that enables the model to generate future action sequences conditioned on either historical observations or task instructions, while preserving the core components and learning principles of LS-JEPA. The experimental results are summarized in Table 1. Results for the other methods are taken from TC-LeWM.
Table 1. Comparison with other methods on LIBERO benchmark.
It can be seen from Table 1 that our method significantly outperform other methods on the LIBERO benchmark, with around 20% improvment compared to TC-LeWM. This is because our method can jointly learn multi-scale forward and inverse dynamics, thereby improving the quality of the generated future action sequences and making it more suitable for handling complex, long-horizon tasks. We have include the results in Table 2 and the discussion in the manuscript (blue content).
To further assess whether our proposed inverse dynamics (ID) objective transfers to other world-model frameworks, we evaluate a LeWAM-based variant, using LeWAM [a] as the baseline and adding our ID objective to it. The results are summarized below:
Table 2. Resuls of LeWAM and our proposed ID.
We have include the reaults in the appendix (blue content).