- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes an algorithm to train a latent world-model to synthesise imaginary trajectories that are used to accelerate the learning process.
Critical strengths
The paper demonstrates that JEPA-style latent world model can be implemented upon a VLM backbone with LoRA adapters.
Critical weaknesses
-
Neither the proposed algorithm nor the empirical results are mind-blowing enough to make me lean towards clear acceptance. There have been multiple prior offline RL works that adopt JEPA-style world models (Zhou et al., 2024; Sobal et al., 2026). The learning algorithm is also similar to the classical model-based dataset augmentation method Dyna-Q. The usage of LoRA adapters to share a common backbone between the model and the policy is clever but not so surprising.
-
The ablation of seems meaningless. If I've understood correctly, the is the only loss function related to , which means dropping will leave , which is what is used during the "planning phase", untouched.
References
Sobal, U., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G., & LeCun, Y. (2026). Learning from reward-free offline data: A case for planning with latent dynamics models. Advances in Neural Information Processing Systems, 38, 43905-43941.
Zhou, G., Pan, H., LeCun, Y., & Pinto, L. (2024). Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983.
Questions for authors
-
It is unclear how the networks are implemented. I know that LoRA has been used, but what are the inputs and outputs? What's the dimension of the latent states? How do you feed the latent state, which is neither language nor image, to the VLM? So many key details are missing from the paper.
-
Is there a reason why AdaWorld results are not included in Table 1?
-
Adding these ablation studies might be interesting:
- Using , with linear projection to align the dimensions if needed, instead of . This experiment is necessary in order to justify the adoption of predicting posterior states.
- Always initializing posteriors with 0 posterior steps.
- Full fine-tuning instead of LoRA.
- Full fine-tuning two different VLMs (one for policy, the other for model)
- Sensitivity to the latent state dimension.
- Planning with GRPO instead of PPO.
-
Is there a reason for not using the real samples for training?
-
Consider adding a brief explanation on how DWELL differs from LeWorldModel and AdaWorld.
Miscellaneous comments
- First sentence of Introduction: Team et al., 2026 → Kimi Team et al., 2026
- First sentence of the Interaction Setting paragraph of Method: (POMDP) Åström(1965);Kaelblingetal.(1998) → (POMDP; Åström, 1965; Kaelbling et al., 1998)
- The rationale for why prior and posterior state prediction are both required comes after defining them. Consider placing it before the definitions so that readers wouldn't be wondering why we need while reading Eq. (2).
- The expressions prior and posterior latent states are unconventional and confusing.
- The if used in the first sentence of the second paragraph of Recurrent Latent Model is awkward. Consider rewriting the sentence.
- Section 3.1 Baselines paragraph: LeWorldModel → LeWorldModel (Maes et al., 2026)
5 comments
Response to the Official Review by Reviewer TgEB (2)
Q2 Why was AdaWorld omitted from Table 1?
We originally included AdaWorld in Section 3.3 to compare learning curves and computation (Figures 2–3). We agree that readers should also be able to compare final success rates in the main table.
Table T2. Final real-environment success rate. AdaWorld uses 3 seeds on Sokoban&FrozenLake and one seed on each of the other three environments.
| Method | Sokoban | FrozenLake | Navigation | PrimitiveSkill | EB-ALFRED |
|---|---|---|---|---|---|
| AdaWorld [7] | 0.37 ± 0.050 | 0.35 ± 0.105 | 0.33 | 0.50 | 0.233 |
| DWELL (paper Table 1, corrected) | 0.53 ± 0.012 | 0.51 ± 0.021 | 0.39 | 0.21 | 0.18 ± 0.048 |
Results: DWELL get better performance on each grid task in the three-seed comparison. AdaWorld’s single-seed score is lower on Navigation and higher on PrimitiveSkill and EB-ALFRED than DWELL’s reported averages. Once the remaining three-seed evaluations are complete, we will add the five-environment comparison to Table 1 and update Section 3.2 accordingly.
Q3(b–f) How sensitive is DWELL to rollout initialization, fine-tuning, latent width, and the policy objective?
We tested the five variants you suggested. In (b), setting in Algorithm 1 starts every imagined rollout at episode reset. In (e), linear projections map each recurrent slot from 2048 to 1408 dimensions for storage and back to 2048 when Qwen reads it. In (f), GRPO compares the imagined returns of action sequences from the same starting state, using the same imagined-transition and actor-update budgets as PPO.
Table T3. Real-environment success rate for the additional ablations. Within the rebuttal period, we completed 1 seed per task for (b)–(f).
| Variant | Sokoban | FrozenLake |
|---|---|---|
| DWELL, native 2048-dimensional slots | 0.53 ± 0.012 | 0.51 ± 0.021 |
| (b) Start all imagined rollouts at episode reset | 0.57 | 0.44 |
| (c) Fully fine-tune one shared VLM | 0.43 | 0.48 |
| (d) Fully fine-tune two VLMs, one per role | 0.25 | 0.14 |
| (e) Projected recurrent slots, | 0.38 | 0.44 |
| (f) Train the actor with GRPO on imagined rollouts | 0.56 | 0.68 |
Results: We get some interesting preliminary findings from the ablations: We find that the best starting state for an imagined rollout may vary by task. Under full fine-tuning, both tasks favored a shared backbone. Because the actor reads the dynamics model’s predicted slots directly, sharing weights may help the actor track changes in how the model encodes a state as training proceeds. Both tasks also favored native-width slots, although the narrower variant changes both the width and the projections. Imagined GRPO scored highest on FrozenLake. When two of its eight continuations from the same tile reach different outcomes, their relative returns can tell the actor which route to favor from that tile. Appendix A.3 reports all five variants, with one training seed per variant and task.
Q4 Why are real samples not used for training?
Real samples are used throughout DWELL, with a clear division between off-policy replay and on-policy optimization:
- Where real samples are used: Real trajectories in the replay buffer train and continually refresh the world and reward models, supply -step return targets for critic warm-up, supervise EB-ALFRED behavior cloning, and provide the real posterior starting states for imagination.
- Why PPO updates on imagined rollouts: Because PPO is an on-policy algorithm, directly reusing historical trajectories in quickly becomes off-policy as the actor updates. At the start of each PPO iteration, the current policy generates a fresh batch of imagined rollouts from posterior states inferred from real trajectories in . PPO then reuses this batch across multiple update epochs without additional environment calls. This lets us study the benefit of latent imagination under a fixed real-interaction budget, though mixing fresh real transitions with imagined rollouts is also compatible with our framework. We have clarified the data source for each update in Section 2.3 and Appendix A.2.
Response to the Official Review by Reviewer TgEB (3)
Q5 How does DWELL differ from LeWorldModel and AdaWorld?
The three methods differ in what the world model predicts and how the VLM policy consumes that prediction:
- AdaWorld [7] (pixel generation): Predicts future RGB observations that the VLM must re-encode through its vision tower at every imagined step, incurring substantially higher compute (38–58% less GPU-hours for Dwell; Figure 3).
- LeWorldModel [6] (standalone latent dynamics): Learns a separate visual encoder and compact latent dynamics outside the VLM. Even with a learned projection to bridge its states to the Qwen actor–critic, decoupling the prediction and decision spaces degrades policy learning (consistent with our bridged-interface ablation in Appendix A.3, Table 2).
- DWELL (native VLM latent dynamics): Predicts recurrent states directly in the VLM’s own token embedding space, allowing the actor and dynamics model to read predicted states immediately without pixel synthesis, re-encoding, or cross-space projection.
We have added this comparison to Section 3.1 and Appendix A.2.
Miscellaneous comments
Thank you for identifying the unclear wording and citations. We have addressed all six comments in the revised manuscript. In detail, we corrected the Kimi Team attribution (Sec 1) and POMDP citation (Sec 2), added Maes et al. in Section 3.1, and rewrote the conditional sentence in “Recurrent Latent Model.” Before Eq. (2) (Sec 2.1), we explain which information is available during real interaction and imagination, then clarify that posterior denotes the state updated with the current observation and prior denotes the state predicted without that observation. Both are deterministic latent representations.
References
[1] Zhou et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. ICML, 2025.
[2] Sobal et al. Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models. NeurIPS, 2025.
[3] Sutton. Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming. ICML, 1990.
[4] Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR, 2020.
[5] Zhang et al. DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Driving. ICML, 2026.
[6] Maes et al. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. 2026.
[7] Gao et al. AdaWorld: Learning Adaptable World Models with Latent Actions. ICML, 2025.
Response to the Official Review by Reviewer TgEB
Thank you for suggesting a direct test of the posterior prediction target. Your comparison with a frozen visual target helps us explain what DWELL adds to earlier latent world models.
1. Relation to prior work. Prior work has established latent dynamics prediction [1, 2, 5] and policy optimization from imagined experience [3, 4]. DWELL addresses a distinct question: how a pretrained VLM can continually improve its policy online using the internal states it forms during real interaction. Training VLMs online can lead to representation drift and high computational cost.
In DWELL, after a real action, the VLM updates its recurrent state with the new observation via lightweight LoRA adapters. This observation-conditioned posterior zt+1 serves both as the state the actor reads for its next decision and as the prediction target for the dynamics model before the observation becomes available (Eq. 3). Consequently, imagined PPO trains the actor on estimates of the exact states it will encounter when returning to the real environment, achieving stable representations and higher computational efficiency than both pixel-reconstruction world models (e.g., AdaWorld [7]) and model-free online RL (PPO/GRPO).
2. Why predict the posterior? Consider two distinct interaction histories that end in the same visual image. The frozen visual target et+1=E(ot+1) is identical for both, whereas their posteriors zt+1 can differ because each also depends on the preceding belief zt and action at. Furthermore, et+1 lives in the external encoder's feature space rather than the VLM's decision space. DWELL therefore separates the two roles: it retains the frozen visual feature et+1 as a visual anchor on a projection of the posterior to prevent representation drift (Eq. 4), while training the dynamics predictor to match the history-conditioned posterior zt+1 that the VLM actor directly reads (Eq. 3).
We agree that removing Ldyn only serves as a sanity check that dynamics supervision matters. Following your suggestion, we kept the exact same action-conditioned VLM predictor and its 2048-dimensional recurrent slots, and changed only the target used to train the predictor from zt+1 to the frozen V-JEPA 2 feature et+1=E(ot+1).
Table T1. Real-environment success rate by prediction target (mean ± std, 3 seeds).
Results: Both tasks favor the posterior target. Alignment to et+1 rewards fidelity to the next visual observation but gives the VLM little guidance about which features matter for its next decision. The posterior target may make the VLM’s pretrained knowledge more useful during imagination because it expresses the action’s outcome in the representation the policy already uses to choose its next move.
We also thank Reviewer DBhY for suggesting a standalone predictor. With the Qwen policy initialization and training data held fixed, replacing the VLM transition with V-JEPA 2 and a four-layer Transformer lowered success on both tasks. We have added your suggested prediction-target comparison to our ablation study in Appendix A.3.