- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReWorldModel, adpating the Mixture-of-Depths attention to a JEPA encoder, to aggreate multi-level representation information. Promising results are reported across five navigation and manipulation environments.
Critical strengths
- The problem is well motivated: Instead of using the final-depth information, aggregating multi-level feature is a reasonable direction, where different layers have distinct functions.
- Comprehensive ablations: The component ablations in Table 2 are informative, where the proposed components are well ablated.
Critical weaknesses
- Lack of baselines: the paper only compared against LeWM, more baselines should be included, for example, PLDM [1], DINO-WM [2]
- I acknowledge the application and demonstrated effect of cross-depth attention, but the main contribution is to incorporate MoDA into LeWM.
[1] Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models, https://arxiv.org/abs/2502.14819v4 [2] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, https://arxiv.org/abs/2411.04983
Questions for authors
- What are the runtime and memory costs? Can the authors report the analysis?
2 comments
Response to Reviewer aHEu (2/2)
Concern 3: What are the runtime and memory costs? Can the authors report the analysis?
Response 3: Thank you for this practical question. We added a dedicated runtime, memory, and matched-recipe update efficiency analysis in Section 7 (Limitations) and Appendix R.
In a fixed-shape steady-state compute benchmark on one RTX 5090 (BF16, 64 four-frame sequences per update; fused AdamW and CUDA Graphs for both models; one full compute update covering encoder forward, predictor, loss, backward, and optimizer step with all 12 layers, historical K/V cache retained, and FlashAttention for joint attention, excluding data loading and initial compilation):
| Metric | Optimized ReWM | ViT (LeWM) |
|---|---|---|
| Full compute update | 59.00 ms | 56.48 ms |
| Peak reserved memory | 19.38 GiB | 15.71 GiB |
Table R12. Steady-state per-update compute and memory benchmark on PushT (single RTX 5090, BF16).
Cross-depth access adds +4.5% per-update compute time (59.00 ms vs. 56.48 ms) and +3.67 GiB of peak reserved memory (19.38 GiB vs. 15.71 GiB for caching earlier-layer K/V), fitting comfortably on a single consumer GPU. Separately, in a matched-recipe paired PushT run (LR , cosine schedule, seed 3072), ReWM reaches ≥95% GC-IDM success in 2,048 updates versus 9,216 for ViT (4.5× fewer updates).
Thank you again for your constructive comments, which have greatly strengthened our paper. We hope our responses have thoroughly addressed your concerns, and we would be sincerely grateful if you would consider raising your score! Please let us know if any further clarification would be helpful.
Response to Reviewer aHEu (1/2)
We sincerely thank the reviewer for the thoughtful evaluation, and for recognizing that our problem is "well motivated" and that our ablations are "comprehensive" and "informative". We have incorporated your valuable suggestions into the revised manuscript.
Concern 1: Lack of baselines: the paper only compared against LeWM, more baselines should be included, for example, PLDM [1], DINO-WM [2]
Response 1: Thank you for this important suggestion. We evaluated both PLDM [1] and DINO-WM [2] on PushT (300 shared evaluation goals) from two complementary angles:
Table R10. PushT GC-IDM control comparison (%) against PLDM and frozen DINOv2-S representations.
Table R11. PushT CEM planning success (%) comparing ReWM with PLDM and DINO-WM across training budgets.
On policy readout, ReWM substantially outperforms LeWM and frozen DINOv2-S features while matching PLDM. When planning through each model's learned dynamics, ReWM leads across budgets—exceeding PLDM by +17.0, +12.0, and +4.3 points, matching DINO-WM at epoch 2, and surpassing it by +7.7 and +18.7 points at epochs 5 and 10—using a compact 5.43M encoder trained from scratch rather than the frozen ~21M DINOv2-S backbone in DINO-WM. We have included these comparisons and discussed their complementarity in Sections 2, 5.4, 6 (Discussion), and Appendix Q.
Concern 2: I acknowledge the application and demonstrated effect of cross-depth attention, but the main contribution is to incorporate MoDA into LeWM.
Response 2: We appreciate your recognition of cross-depth attention's demonstrated effect. Rather than treating MoDA as a standalone operator novelty, our contribution follows a focused representational question in latent world models:
Table R1; Section 5.4, Appendix O) and trail ReWM (82.4%) in epoch-2 CEM planning.Table R8; Section 5.4, Appendix N).Table R5).Tables R2–R3, R11), the results consistently support full ReWM under the tested conditions.