- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LaLa-World trains an LLM-based latent dynamics predictor with a shared language interface for heterogeneous control actions and image/video changes. A frozen visual encoder supplies context and target features, while a Qwen-based predictor regresses future features in one forward pass. The model is used in CEM-based planning by verbalizing candidate numerical actions. The paper also fine-tunes the pretrained predictor into LaLa-Policy with a diffusion action head. Experiments cover several simulated planning domains, offline DROID action prediction, zero-shot RoboCasa planning, and policy learning on LIBERO and RoboCasa. Comparisons with an adapted RAEv2 baseline examine planning performance and computational cost.
Critical strengths
-
Using language to pool heterogeneous control transitions and broader visual-transition data is a useful direction for latent world models. The paper goes beyond a conceptual proposal: it evaluates multiple action spaces, reports zero-shot RoboCasa results, and studies initialization, joint training, visual encoders, action phrasing, and sensitivity to direction/magnitude words. The latter diagnostics are particularly helpful because low latent prediction error alone would not demonstrate that a model uses the meaning of its action condition. The joint-training results and the dual-encoder single-predictor result in Table D provide encouraging evidence for a shared dynamics predictor.
-
The paper investigates both planning and policy initialization, with a concrete efficiency advantage for direct latent prediction. The RAEv2 comparison matches data, language conditions, and the visual-encoder family, and includes both one- and five-step sampling and multiple predictor depths. LaLa-Policy also improves over random and Qwen-only initialization on the evaluated policy benchmarks. Taken together, these experiments make a useful case that predicting perceptual features can be practically valuable without reconstructing pixels during control; the supplementary visualizations help illustrate what is retained and what is lost.
Critical weaknesses
-
The main planning comparison does not isolate the proposed action interface from a substantial change in the optimization problem. Appendix D.4.2 states that the baselines concatenate the five substep actions, whereas LaLa-World sums each component and optimizes in that lower-dimensional space, executing a constant action across the five substeps. With a fixed CEM budget, reducing the search dimensionality can itself improve planning, even though aggregation also loses information. Thus, using the same horizon and CEM settings is insufficient to attribute the gains to language-conditioned world modeling. The initialization/joint-training ablations do not replace a jointly trained numerical-action baseline with the same aggregation, model capacity, data, and planning budget. This is the main gap in establishing which part of the proposed approach produces the gains.
-
Some headline comparisons need a narrower and more explicit interpretation. Appendix C.5 explains that Table 1 selects different visual encoders across benchmarks, so its 61.3 average is not the performance of one fixed encoder-predictor system. Table D does provide a compelling fixed dual-encoder result of 63.0, which should be foregrounded with its additional cost rather than leaving this distinction to the appendix. Similarly, the 13.9x planning speedup is relative to a particular five-step RAEv2 configuration with a separate text encoder; the one-step comparison is 2.9x. These are useful system-level results, but the predictor architectures, initialization sources, and conditioning implementations differ, so they do not establish an isolated effect of the prediction objective. The downstream gains over Qwen initialization are also modest on RoboCasa (57.3 versus 56.8), without training-seed uncertainty, and the exact world-model checkpoint used for the SigLIP2-based policy needs clarification.
Questions for authors
-
How do DINO-WM/JEPA-WMs and a jointly trained numerical-action predictor perform when given exactly the same aggregated action space and execution rule as LaLa-World? Please keep the CEM budget, visual encoder, predictor capacity, and available training data matched where possible. This control would materially strengthen the causal interpretation of the main comparison.
-
Please identify the exact encoder-predictor checkpoints used for each Table 1 result and report the fixed single-system comparison prominently. How much latency and memory does the dual-encoder model in Table D add? For the RAEv2 timing, does the reported total include text tokenization/encoding and all planning overhead, and are both models using the same DINOv3 feature layers/aggregation?
-
Which pretrained LaLa-World checkpoint initializes the SigLIP2 policy in Appendix D.6? Please clarify whether the LIBERO/RoboCasa future-prediction data described in D.7 are used only during policy fine-tuning or also in a preceding world-model adaptation stage, and provide uncertainty over independent training runs for the small policy-initialization gains. These clarifications are needed to interpret the reported transfer benefit.
6 comments
Response to Reviewer oPHe (2/4)
W1-b. Table C (initialization/joint-training ablations) does not replace a jointly trained baseline with numeric action conditioning, which should be added.
We appreciate the reviewer's valuable suggestion. In addition to Table C, we conduct experiments with a jointly trained numerical-action baseline and observe that it performs below language actions under matched conditions, so the gains do not come from joint training alone. We have added these results to Table C.
Experimental setup:
The baseline conditions the same Qwen predictor on numerical actions through a linear adapter for each domain to handle the different action dimensions across domains. For a fair comparison, we match the aggregation, model capacity, data, and planning budget to LaLa-World, so only the action interface differs.
Results:
Table III. Planning success rates (%) by action condition
| Training | Action condition | Wall | PointMaze | Push-T | DROID | RoboCasa-R | RoboCasa-P | Average |
|---|---|---|---|---|---|---|---|---|
| Single-domain (control) | Language | 97.9 (1.5) | 66.7 (4.8) | 70.8 (4.6) | 38.3 (2.3) | 21.9 (7.3) | 31.2 (8.2) | 54.5 (2.2) |
| Joint (control) | Numerical actions | 89.6 (3.1) | 56.2 (5.1) | 82.3 (3.9) | 39.3 (2.3) | 18.8 (6.9) | 31.2 (8.2) | 52.9 (2.2) |
| Joint (control) | Language | 96.9 (1.8) | 64.6 (4.9) | 81.2 (4.0) | 41.5 (2.3) | 25.0 (7.7) | 34.4 (8.4) | 57.3 (2.2) |
| Joint (control + image/video) | Language | 95.8 (2.0) | 56.2 (5.1) | 81.2 (4.0) | 40.2 (2.6) | 40.6 (8.7) | 46.9 (8.8) | 60.2 (2.4) |
- Language actions outperform numerical actions on average. Joint training with numerical actions even falls below single-domain training.
- We conjecture that language gives the meaning of each value along with the value itself. With raw values such as
[−0.5, 0.6], the predictor must learn for each domain the physical meaning of each value, whereas0.5 up and 0.6 leftstates it directly, so the predictor may focus on learning dynamics. - We would also like to highlight that the numerical baseline relies on per-domain adapters, so each new action space would require a new adapter, which limits its real-world applicability.
W2-a. The compelling fixed dual-encoder result in Table D should be presented with its additional cost.
Q2-a. How much latency and memory does the dual-encoder model in Table D add?
We are glad that the reviewer finds the dual-encoder result compelling. We measure its additional cost and report it with the planning results in Table R1 of the revised manuscript, shown below.
Table IV. Performance and cost of each vision encoder
| Vision encoder | LaLa-World (Encoder + Predictor) | ||||
|---|---|---|---|---|---|
| Encoder | Tokens / frame | Encoding (s / ep) | MPC Planning Perf. (%) | Planning (s / ep) | Mem (GB) |
| DINOv2-S/14 | 256 | 0.050 | 47.7 | 567.6 (1.27×) | 44.3 (1.31×) |
| DINOv3-L/16 | 196 | 0.158 | 53.7 | 445.2 (1.00×) | 33.7 (1.00×) |
| SigLIP2-So400m/14 | 256 | 0.124 | 37.3 | 567.8 (1.28×) | 48.2 (1.43×) |
| DINOv2 + DINOv3 | 452 | 0.209 | 63.0 | 1011.7 (2.27×) | 58.0 (1.72×) |
- The dual-encoder model (DINOv2 + DINOv3) achieves the best planning performance (Line 1329), but requires more computation and memory (2.27× the planning time and 1.72× the GPU memory of DINOv3-L/16).
- The planning time and memory depend on the number of tokens that the encoder produces. For example, DINOv2-S/14, although smaller, produces more tokens per frame than DINOv3-L/16 (256 vs. 196) and takes 1.27× the planning time and 1.31× the GPU memory.
W2-b. The explanation in Appendix C.5 and the result in Table D should be foregrounded.
Q2-b. Please identify the exact encoder-predictor checkpoints in Table 1 and report the fixed single-system comparison prominently.
We agree that this content is better suited for the main paper, and we have revised the manuscript as follows:
- We have added Table IV to the main paper as Table R1, which reports the success rate, planning time, and GPU memory for each vision encoder, and discuss it in a new paragraph in Section 4.1: "Complementary vision encoders improve LaLa-World planning at additional cost."
- We have moved the explanation in Appendix C.5 to Section 4 (Line 301) and specified the vision encoder used for each benchmark: "The comparison in Table 1 allows the vision encoder to vary across benchmarks, as does JEPA-WMs (Terver et al., 2025)" and "For LaLa-World, we use DINOv2-S/14 for Wall, Push-T, Granular, RoboCasa-R, and DROID, and DINOv3-L/16 for the other benchmarks."
Response to Reviewer oPHe (3/4)
W2-c. The speed comparison with RAEv2 gives useful system-level results, but predictor architectures and initialization sources differ, so they do not establish an isolated effect of the prediction objective.
We are glad that the reviewer recognizes these as useful system-level results. We would like to clarify that the comparison is between two approaches, latent and generative world models, and is intended as a system-level comparison rather than an isolated study of the prediction objective. Section 4.2 now states what each approach keeps from its original design and why, and Table V summarizes what is matched and what differs. The details are as follows:
- Since modifying RAEv2's tuned configuration would weaken the baseline, we keep each approach's original predictor architecture, initialization, and language conditioning (lower block of Table V).
- Previous works compare latent world models (e.g., V-JEPA 2 [1], V-JEPA 2.1 [2]) with video generation models (e.g., Cosmos [3], NWM [4]) that differ in action interface, encoder, predictor size, and training data (Line 978).
- In contrast, we match these factors: the same language interface, DINOv3-L/16 encoder, training data, and predictor size (upper block of Table V).
Table V. Setup of the latent vs. generative comparison
| LaLa-World | RAEv2-based world model | |
|---|---|---|
| Matched | ||
| Vision encoder | Frozen DINOv3-L/16 | Same |
| Action interface | Language | Same |
| Training data | Control + image/video transitions (Table G) | Same |
| Predictor size | ≈0.8B | Same |
| Kept from each approach's original design | ||
| Prediction objective | Latent regression | Flow matching + REPA |
| Predictor architecture | Language model (Qwen3.5-0.8B) | DiT (RAEv2) |
| Initialization | Pretrained Qwen3.5-0.8B | Pretrained DiT from the official RAEv2 release |
| Language conditioning | Read as tokens by the predictor itself | Frozen Qwen3-0.6B text encoder |
| DINOv3-L/16 features | Final layer | Features aggregated from seven layers |
[1] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv 2025
[2] V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv 2026
[3] Cosmos: Cosmos World Foundation Model Platform for Physical AI. arXiv 2025
[4] NWM: Navigation World Models. CVPR 2025
Q2-c. For the RAEv2 timing, does the reported total include text tokenization/encoding and all planning overhead?
Yes. For both models, the reported total includes text tokenization/encoding and all planning overhead. We additionally measured the time of each stage (Table VI) and added it to the paper.
Table VI. Breakdown of the average planning time per trajectory (min)
| Stage | LaLa-World | RAEv2 (1 step) | RAEv2 (5 steps) |
|---|---|---|---|
| Observation and goal encoding (DINOv3-L/16) | <0.1 | <0.1 | <0.1 |
| Text encoding (Qwen3-0.6B) | – | 1.0 | 1.0 |
| Predictor forward passes | 7.2 | 19.9 | 101.8 |
| CEM sampling and update | 0.1 | 0.1 | 0.1 |
| Total (Figure 5(b)) | 7.4 (1.0×) | 21.1 (2.9×) | 102.9 (13.9×) |
| Total without text encoding | 7.4 (1.0×) | 20.1 (2.7×) | 101.9 (13.8×) |
- Text encoding takes a small part of RAEv2's planning time, since the text embeddings are reused across denoising steps.
- Most of the planning time comes from the predictor forward passes, which grow with the number of denoising steps for RAEv2.
Q2-d. Are both models (LaLa-World and RAEv2) using the same DINOv3 feature layers/aggregation?
Both models use the same frozen DINOv3-L/16 weights, but not the same feature layers. LaLa-World uses the final-layer patch features. As noted in Appendix D.5, RAEv2 aggregates features from seven layers. We keep RAEv2's setting for two reasons: (1) the official DiT weights are pretrained in this aggregated space, and (2) changing it would weaken the baseline.
W2-d. Headline comparisons need a more explicit interpretation: the 13.9x speedup is relative to five-step RAEv2; the one-step comparison is 2.9x.
We thank the reviewer for this careful observation, and we agree that the configuration behind the 13.9× speedup should be stated. We have revised the abstract and introduction accordingly. We now write "up to 13.9× faster planning" in the abstract and state the configuration in the introduction: "With the same language conditions, training data, and visual encoder, LaLa-World outperforms this RAEv2 baseline in planning with 6× less training cost and 13.9× faster inference, relative to RAEv2 with five denoising steps." The one-step speedup of 2.9× is reported with it in Section 4.2 (Line 416).
Response to Reviewer oPHe (1/4)
We thank Reviewer oPHe for their time and valuable feedback. We are pleased that the reviewer recognizes (1) the usefulness of pooling heterogeneous data with language, (2) the concrete efficiency advantage of latent prediction, and (3) the encouraging joint-training results. We have carefully considered the concerns raised and provide our responses below.
We thank the reviewer for pointing out an important detail. To address this, we conduct additional experiments with substep aggregation for DINO-WM and JEPA-WMs and substep concatenation for LaLa-World. The results show that LaLa-World's gains do not come from the simpler search space, and that the baselines do not benefit from substep aggregation.
Experimental setup:
Table I. Action formats, with examples from Push-T (5 substeps per step, 2-D action)
Results:
Table II. Planning success rates (%)