acceptodds
Under review as a conference paper at ICLR 2027

LaLa-World: Scaling Latent World Models with Natural Language Actions

41 views

Abstract

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce **LaLa-World**, a **La**nguage-conditioned **La**tent **World** model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with less training cost and faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to faster inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent World Models, Language-Conditioned World Models, Model-Based Planning, Policy Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oPHeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LaLa-World trains an LLM-based latent dynamics predictor with a shared language interface for heterogeneous control actions and image/video changes. A frozen visual encoder supplies context and target features, while a Qwen-based predictor regresses future features in one forward pass. The model is used in CEM-based planning by verbalizing candidate numerical actions. The paper also fine-tunes the pretrained predictor into LaLa-Policy with a diffusion action head. Experiments cover several simulated planning domains, offline DROID action prediction, zero-shot RoboCasa planning, and policy learning on LIBERO and RoboCasa. Comparisons with an adapted RAEv2 baseline examine planning performance and computational cost.

Critical strengths
  1. Using language to pool heterogeneous control transitions and broader visual-transition data is a useful direction for latent world models. The paper goes beyond a conceptual proposal: it evaluates multiple action spaces, reports zero-shot RoboCasa results, and studies initialization, joint training, visual encoders, action phrasing, and sensitivity to direction/magnitude words. The latter diagnostics are particularly helpful because low latent prediction error alone would not demonstrate that a model uses the meaning of its action condition. The joint-training results and the dual-encoder single-predictor result in Table D provide encouraging evidence for a shared dynamics predictor.

  2. The paper investigates both planning and policy initialization, with a concrete efficiency advantage for direct latent prediction. The RAEv2 comparison matches data, language conditions, and the visual-encoder family, and includes both one- and five-step sampling and multiple predictor depths. LaLa-Policy also improves over random and Qwen-only initialization on the evaluated policy benchmarks. Taken together, these experiments make a useful case that predicting perceptual features can be practically valuable without reconstructing pixels during control; the supplementary visualizations help illustrate what is retained and what is lost.

Critical weaknesses
  1. The main planning comparison does not isolate the proposed action interface from a substantial change in the optimization problem. Appendix D.4.2 states that the baselines concatenate the five substep actions, whereas LaLa-World sums each component and optimizes in that lower-dimensional space, executing a constant action across the five substeps. With a fixed CEM budget, reducing the search dimensionality can itself improve planning, even though aggregation also loses information. Thus, using the same horizon and CEM settings is insufficient to attribute the gains to language-conditioned world modeling. The initialization/joint-training ablations do not replace a jointly trained numerical-action baseline with the same aggregation, model capacity, data, and planning budget. This is the main gap in establishing which part of the proposed approach produces the gains.

  2. Some headline comparisons need a narrower and more explicit interpretation. Appendix C.5 explains that Table 1 selects different visual encoders across benchmarks, so its 61.3 average is not the performance of one fixed encoder-predictor system. Table D does provide a compelling fixed dual-encoder result of 63.0, which should be foregrounded with its additional cost rather than leaving this distinction to the appendix. Similarly, the 13.9x planning speedup is relative to a particular five-step RAEv2 configuration with a separate text encoder; the one-step comparison is 2.9x. These are useful system-level results, but the predictor architectures, initialization sources, and conditioning implementations differ, so they do not establish an isolated effect of the prediction objective. The downstream gains over Qwen initialization are also modest on RoboCasa (57.3 versus 56.8), without training-seed uncertainty, and the exact world-model checkpoint used for the SigLIP2-based policy needs clarification.

Questions for authors
  1. How do DINO-WM/JEPA-WMs and a jointly trained numerical-action predictor perform when given exactly the same aggregated action space and execution rule as LaLa-World? Please keep the CEM budget, visual encoder, predictor capacity, and available training data matched where possible. This control would materially strengthen the causal interpretation of the main comparison.

  2. Please identify the exact encoder-predictor checkpoints used for each Table 1 result and report the fixed single-system comparison prominently. How much latency and memory does the dual-encoder model in Table D add? For the RAEv2 timing, does the reported total include text tokenization/encoding and all planning overhead, and are both models using the same DINOv3 feature layers/aggregation?

  3. Which pretrained LaLa-World checkpoint initializes the SigLIP2 policy in Appendix D.6? Please clarify whether the LIBERO/RoboCasa future-prediction data described in D.7 are used only during policy fine-tuning or also in a preceding world-model adaptation stage, and provide uncertainty over independent training runs for the small policy-initialization gains. These clarifications are needed to interpret the reported transfer benefit.

6 comments
AuthorsOpenReview ↗

Response to Reviewer oPHe (1/4)

We thank Reviewer oPHe for their time and valuable feedback. We are pleased that the reviewer recognizes (1) the usefulness of pooling heterogeneous data with language, (2) the concrete efficiency advantage of latent prediction, and (3) the encouraging joint-training results. We have carefully considered the concerns raised and provide our responses below.

W1-a. According to Appendix D.4.2, the baselines concatenate the five substep actions, whereas LaLa-World sums each component. Do the planning gains come from the lower-dimensional, simpler search space?

Q1. How do the baselines perform with the same aggregated action space (format B in Table I)?

We thank the reviewer for pointing out an important detail. To address this, we conduct additional experiments with substep aggregation for DINO-WM and JEPA-WMs and substep concatenation for LaLa-World. The results show that LaLa-World's gains do not come from the simpler search space, and that the baselines do not benefit from substep aggregation.

Experimental setup:

  • We retrain DINO-WM and JEPA-WMs with aggregated actions (format B in Table I), so they use the same aggregated action space and execution rule as LaLa-World, while the CEM budget, visual encoder, predictor capacity, and available training data are kept matched.
  • We also train LaLa-World with concatenated language actions (format C).

Table I. Action formats, with examples from Push-T (5 substeps per step, 2-D action)

IndexAction formatExampleTraining pairUsed by
AConcatenated substep actions (numerical)[0.15, −0.27, 0.06, −0.09, −0.03, 0.04, −0.11, 0.05, −0.54, −0.04](xt,A,xt+1)(x_t, A, x_{t+1})DINO-WM, JEPA-WMs (Table 1)
BAggregated substep actions (numerical)[−0.47, −0.32](xt,B,xt+1)(x_t, B, x_{t+1})DINO-WM, JEPA-WMs (new)
CConcatenated substep actions (language)move the pusher 0.15 right and 0.27 up, then move the pusher 0.06 right and 0.09 up, then move the pusher 0.03 left and 0.04 down, then move the pusher 0.11 left and 0.05 down, then move the pusher 0.54 left and 0.04 up(xt,C,xt+1)(x_t, C, x_{t+1})LaLa-World (new)
DAggregated substep actions (language)move the pusher 0.47 left and 0.32 up(xt,D,xt+1)(x_t, D, x_{t+1})LaLa-World (Table 1)

Results:

Table II. Planning success rates (%)

ModelAction formatWallPointMazePush-TMetaWorld-RMetaWorld-RWAverage
DINO-WMA (Table 1)90.6 (3.0)59.4 (5.0)65.6 (4.8)52.1 (7.2)43.8 (7.2)62.3 (2.5)
DINO-WMB89.6 (3.1)57.3 (5.0)68.8 (4.7)43.8 (7.2)33.3 (6.8)58.5 (2.5)
JEPA-WMsA (Table 1)79.2 (4.1)60.4 (5.0)68.8 (4.7)66.7 (6.8)47.9 (7.2)64.6 (2.6)
JEPA-WMsB89.6 (3.1)43.8 (5.1)65.6 (4.8)68.8 (6.7)37.5 (7.0)61.0 (2.5)
LaLa-WorldD (Table 1)95.8 (2.0)56.2 (5.1)81.2 (4.0)75.0 (6.2)52.1 (7.2)72.1 (2.3)
LaLa-WorldC93.8 (2.5)60.4 (5.0)78.1 (4.2)70.8 (6.6)52.1 (7.2)71.0 (2.4)
  • With aggregated actions (format B), DINO-WM and JEPA-WMs drop by 3.8 and 3.6 points on average. A lower-dimensional search space does not help them. As the reviewer noted and as mentioned in Appendix D.4.2 Line 1610, aggregation discards variation within a step, which likely causes this drop.
  • Compared with the baselines, LaLa-World is robust to the action format, achieving 71.0 and 72.1 on average with formats C and D, respectively.
  • These results indicate that the gains in Table 1 do not come from the simpler search space. We have added both tables and this discussion to Appendix D.4.2.
AuthorsOpenReview ↗

Response to Reviewer oPHe (2/4)

W1-b. Table C (initialization/joint-training ablations) does not replace a jointly trained baseline with numeric action conditioning, which should be added.

We appreciate the reviewer's valuable suggestion. In addition to Table C, we conduct experiments with a jointly trained numerical-action baseline and observe that it performs below language actions under matched conditions, so the gains do not come from joint training alone. We have added these results to Table C.

Experimental setup:

The baseline conditions the same Qwen predictor on numerical actions through a linear adapter for each domain to handle the different action dimensions across domains. For a fair comparison, we match the aggregation, model capacity, data, and planning budget to LaLa-World, so only the action interface differs.

Results:

Table III. Planning success rates (%) by action condition

TrainingAction conditionWallPointMazePush-TDROIDRoboCasa-RRoboCasa-PAverage
Single-domain (control)Language97.9 (1.5)66.7 (4.8)70.8 (4.6)38.3 (2.3)21.9 (7.3)31.2 (8.2)54.5 (2.2)
Joint (control)Numerical actions89.6 (3.1)56.2 (5.1)82.3 (3.9)39.3 (2.3)18.8 (6.9)31.2 (8.2)52.9 (2.2)
Joint (control)Language96.9 (1.8)64.6 (4.9)81.2 (4.0)41.5 (2.3)25.0 (7.7)34.4 (8.4)57.3 (2.2)
Joint (control + image/video)Language95.8 (2.0)56.2 (5.1)81.2 (4.0)40.2 (2.6)40.6 (8.7)46.9 (8.8)60.2 (2.4)
  • Language actions outperform numerical actions on average. Joint training with numerical actions even falls below single-domain training.
  • We conjecture that language gives the meaning of each value along with the value itself. With raw values such as [−0.5, 0.6], the predictor must learn for each domain the physical meaning of each value, whereas 0.5 up and 0.6 left states it directly, so the predictor may focus on learning dynamics.
  • We would also like to highlight that the numerical baseline relies on per-domain adapters, so each new action space would require a new adapter, which limits its real-world applicability.

W2-a. The compelling fixed dual-encoder result in Table D should be presented with its additional cost.

Q2-a. How much latency and memory does the dual-encoder model in Table D add?

We are glad that the reviewer finds the dual-encoder result compelling. We measure its additional cost and report it with the planning results in Table R1 of the revised manuscript, shown below.

Table IV. Performance and cost of each vision encoder

Vision encoderLaLa-World (Encoder + Predictor)
EncoderTokens / frameEncoding (s / ep)MPC Planning Perf. (%)Planning (s / ep)Mem (GB)
DINOv2-S/142560.05047.7567.6 (1.27×)44.3 (1.31×)
DINOv3-L/161960.15853.7445.2 (1.00×)33.7 (1.00×)
SigLIP2-So400m/142560.12437.3567.8 (1.28×)48.2 (1.43×)
DINOv2 + DINOv34520.20963.01011.7 (2.27×)58.0 (1.72×)
  • The dual-encoder model (DINOv2 + DINOv3) achieves the best planning performance (Line 1329), but requires more computation and memory (2.27× the planning time and 1.72× the GPU memory of DINOv3-L/16).
  • The planning time and memory depend on the number of tokens that the encoder produces. For example, DINOv2-S/14, although smaller, produces more tokens per frame than DINOv3-L/16 (256 vs. 196) and takes 1.27× the planning time and 1.31× the GPU memory.

W2-b. The explanation in Appendix C.5 and the result in Table D should be foregrounded.

Q2-b. Please identify the exact encoder-predictor checkpoints in Table 1 and report the fixed single-system comparison prominently.

We agree that this content is better suited for the main paper, and we have revised the manuscript as follows:

  1. We have added Table IV to the main paper as Table R1, which reports the success rate, planning time, and GPU memory for each vision encoder, and discuss it in a new paragraph in Section 4.1: "Complementary vision encoders improve LaLa-World planning at additional cost."
  2. We have moved the explanation in Appendix C.5 to Section 4 (Line 301) and specified the vision encoder used for each benchmark: "The comparison in Table 1 allows the vision encoder to vary across benchmarks, as does JEPA-WMs (Terver et al., 2025)" and "For LaLa-World, we use DINOv2-S/14 for Wall, Push-T, Granular, RoboCasa-R, and DROID, and DINOv3-L/16 for the other benchmarks."
AuthorsOpenReview ↗

Response to Reviewer oPHe (3/4)

W2-c. The speed comparison with RAEv2 gives useful system-level results, but predictor architectures and initialization sources differ, so they do not establish an isolated effect of the prediction objective.

We are glad that the reviewer recognizes these as useful system-level results. We would like to clarify that the comparison is between two approaches, latent and generative world models, and is intended as a system-level comparison rather than an isolated study of the prediction objective. Section 4.2 now states what each approach keeps from its original design and why, and Table V summarizes what is matched and what differs. The details are as follows:

  1. Since modifying RAEv2's tuned configuration would weaken the baseline, we keep each approach's original predictor architecture, initialization, and language conditioning (lower block of Table V).
  2. Previous works compare latent world models (e.g., V-JEPA 2 [1], V-JEPA 2.1 [2]) with video generation models (e.g., Cosmos [3], NWM [4]) that differ in action interface, encoder, predictor size, and training data (Line 978).
  3. In contrast, we match these factors: the same language interface, DINOv3-L/16 encoder, training data, and predictor size (upper block of Table V).

Table V. Setup of the latent vs. generative comparison

LaLa-WorldRAEv2-based world model
Matched
Vision encoderFrozen DINOv3-L/16Same
Action interfaceLanguageSame
Training dataControl + image/video transitions (Table G)Same
Predictor size≈0.8BSame
Kept from each approach's original design
Prediction objectiveLatent regressionFlow matching + REPA
Predictor architectureLanguage model (Qwen3.5-0.8B)DiT (RAEv2)
InitializationPretrained Qwen3.5-0.8BPretrained DiT from the official RAEv2 release
Language conditioningRead as tokens by the predictor itselfFrozen Qwen3-0.6B text encoder
DINOv3-L/16 featuresFinal layerFeatures aggregated from seven layers

[1] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv 2025
[2] V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv 2026
[3] Cosmos: Cosmos World Foundation Model Platform for Physical AI. arXiv 2025
[4] NWM: Navigation World Models. CVPR 2025

Q2-c. For the RAEv2 timing, does the reported total include text tokenization/encoding and all planning overhead?

Yes. For both models, the reported total includes text tokenization/encoding and all planning overhead. We additionally measured the time of each stage (Table VI) and added it to the paper.

Table VI. Breakdown of the average planning time per trajectory (min)

StageLaLa-WorldRAEv2 (1 step)RAEv2 (5 steps)
Observation and goal encoding (DINOv3-L/16)<0.1<0.1<0.1
Text encoding (Qwen3-0.6B)–1.01.0
Predictor forward passes7.219.9101.8
CEM sampling and update0.10.10.1
Total (Figure 5(b))7.4 (1.0×)21.1 (2.9×)102.9 (13.9×)
Total without text encoding7.4 (1.0×)20.1 (2.7×)101.9 (13.8×)
  • Text encoding takes a small part of RAEv2's planning time, since the text embeddings are reused across denoising steps.
  • Most of the planning time comes from the predictor forward passes, which grow with the number of denoising steps for RAEv2.

Q2-d. Are both models (LaLa-World and RAEv2) using the same DINOv3 feature layers/aggregation?

Both models use the same frozen DINOv3-L/16 weights, but not the same feature layers. LaLa-World uses the final-layer patch features. As noted in Appendix D.5, RAEv2 aggregates features from seven layers. We keep RAEv2's setting for two reasons: (1) the official DiT weights are pretrained in this aggregated space, and (2) changing it would weaken the baseline.

W2-d. Headline comparisons need a more explicit interpretation: the 13.9x speedup is relative to five-step RAEv2; the one-step comparison is 2.9x.

We thank the reviewer for this careful observation, and we agree that the configuration behind the 13.9× speedup should be stated. We have revised the abstract and introduction accordingly. We now write "up to 13.9× faster planning" in the abstract and state the configuration in the introduction: "With the same language conditions, training data, and visual encoder, LaLa-World outperforms this RAEv2 baseline in planning with 6× less training cost and 13.9× faster inference, relative to RAEv2 with five denoising steps." The one-step speedup of 2.9× is reported with it in Section 4.2 (Line 416).

Reviewer_BNhwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains one JEPA-style latent world model across several control domains. It rewrites numeric actions as text templates and adds captioned video and image-editing data. The predictor is a small pretrained LM on top of frozen DINO features. In MPC planning, the jointly trained model gets the best average success over per-domain latent world models. Against an RAEv2 diffusion world model with the same encoder and data, it plans better at lower training and planning cost. The pretrained model also fine-tunes into a policy that is comparable to Cosmos Policy and about 5x faster.

Critical strengths
  • The matched comparison with RAEv2 is an interesting part of the paper. With the same encoder, data and conditions, LaLa-World reaches higher planning success (59.4 vs 48.0) in a fraction of the training time and plans 3-14x faster. Controlled latent vs. generative comparisons like this show the strength of the method.
  • The recipe is simple. One predictor handles different action spaces as well as action-free video, and the evaluation is broad.
Critical weaknesses
  • The role of language is not isolated, and the idea itself is not new. A shared, largely text-based action interface for training one world model across heterogeneous datasets has already been explored for video world models, e.g. UniSim (Yang et al., ICLR 2024), which the paper does not discuss. The new part here is doing it for latent world models. Table C shows that, with the same predictor, joint training is what brings the gain (54.5 to 60.2). But it does not show that language is needed for joint training: the missing control is the same LM predictor trained jointly with numeric actions (e.g. a per-domain action embedding). Without it, the core claim that language is what makes joint training work is not shown.
  • The "scaling" claim is not supported. The title and abstract frame the work as scaling latent world models, but the paper has no real scaling study. Fig. 5(b) varies predictor depth, but mainly to compare cost with RAEv2, and Appendix B says the scaling behavior is not established yet. Adding image/video data improves average planning by only about 2 points. I would either add a scaling experiment or tone down the claim.
Questions for authors
  1. Can you add a joint-training baseline with the same Qwen predictor but numeric action conditioning? This would change my view.
  2. Are 1-5 denoising steps and a 12-hour budget fair to RAEv2? How do the results change with more steps on a few tasks?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BNhw (1/2)

We thank Reviewer BNhw for their time and thorough feedback. We appreciate that the reviewer values (1) the controlled latent vs. generative comparison with RAEv2, (2) the simple recipe handling diverse action spaces as well as video, and (3) the broad evaluation. We have carefully considered the concerns raised and provide our responses below.

W1-a. A shared text-based action interface has been explored for video world models (e.g., UniSim).

We thank the reviewer for suggesting UniSim. UniSim was only cited in Appendix A.2 without discussion; we have now added a discussion there. We would like to highlight that, although a language interface is a simple addition, our work is the first to show what it enables for latent world models: joint training across multiple domains, learning from action-free videos, matched comparison with video generation models, and policy learning. The details are as follows:

  1. Joint training across multiple domains: A single predictor achieves higher average MPC planning performance than existing latent world models trained separately for each domain or task (Table 1).
  2. Learning from action-free videos: LaLa-World learns from image and video data without action labels, a data source unexplored for latent world models.
  3. Matched comparison with video generation models: Against an RAEv2-based video generation model with the same encoder and data, LaLa-World reaches comparable planning performance.
  4. Policy learning: LaLa-World serves as an initialization for LaLa-Policy, which performs competitively with Cosmos Policy with up to 5.5× faster inference.

We have made this distinction explicit in Appendix A.2: UniSim shows that "the resulting video generation model can serve as a simulator," whereas LaLa-World shows what a language interface "enables beyond simulation."

W1-b. Table C is missing a jointly trained baseline with numeric actions to show whether language is needed for joint training.

Q1. Can you add a joint-training baseline with numeric action conditioning? This would change my view.

The reviewer provides an insightful suggestion. In addition to Table C, we conduct experiments with the same predictor jointly trained with numerical actions and observe that it performs below language actions under matched conditions, showing that language is needed for joint training. We have added these results to Table C.

Experimental setup:

The baseline conditions the same Qwen predictor on numerical actions through a linear adapter for each domain to handle the different action dimensions across domains. All other components and the control data are unchanged, so only the language is removed.

Results:

Table I. Planning success rates (%) by action condition

TrainingAction conditionWallPointMazePush-TDROIDRoboCasa-RRoboCasa-PAverage
Single-domain (control)Language97.9 (1.5)66.7 (4.8)70.8 (4.6)38.3 (2.3)21.9 (7.3)31.2 (8.2)54.5 (2.2)
Joint (control)Numerical actions89.6 (3.1)56.2 (5.1)82.3 (3.9)39.3 (2.3)18.8 (6.9)31.2 (8.2)52.9 (2.2)
Joint (control)Language96.9 (1.8)64.6 (4.9)81.2 (4.0)41.5 (2.3)25.0 (7.7)34.4 (8.4)57.3 (2.2)
Joint (control + image/video)Language95.8 (2.0)56.2 (5.1)81.2 (4.0)40.2 (2.6)40.6 (8.7)46.9 (8.8)60.2 (2.4)
  • Language actions outperform numerical actions on average. Joint training with numerical actions even falls below single-domain training.
  • We conjecture that language gives the meaning of each value along with the value itself. With raw values such as [−0.5, 0.6], the predictor must learn for each domain the physical meaning of each value, whereas 0.5 up and 0.6 left states it directly, so the predictor may focus on learning dynamics.
  • We would also like to highlight that the numerical baseline relies on per-domain adapters, so each new action space would require a new adapter, which limits its real-world applicability.
AuthorsOpenReview ↗

Response to Reviewer BNhw (2/2)

W2. The title and abstract frame the work as scaling latent world models, but the paper has no real scaling study.

We agree that scaling can suggest model or data size, and we have toned down this claim in the introduction. We would like to clarify that scaling refers to extending what latent world models learn from and are used for. To be more specific, we detail these extensions below:

  1. From numerical actions to language: Language turns domain-specific training into domain-agnostic training.
  2. From control data to image and video data: The predictor learns from data without action labels.
  3. From planning to policy fine-tuning: The world knowledge learned from these data transfers to action prediction.

In the introduction, we now write "extending LWMs to" instead of "scaling LWMs across" and state what scaling means: "Overall, LaLa-World scales LWMs in what they learn from and are used for: from numerical actions to language, from control data to image and video data, and from planning to policy fine-tuning."

We are also open to revising the title, and we would be happy to receive further feedback.

Q2. Are 1-5 denoising steps and a 12-hour budget fair to RAEv2? How do results change with more steps?

We appreciate the reviewer's important question. To verify fairness, we conduct additional experiments in which we train RAEv2 for 48 hours and evaluate it with up to 20 denoising steps. Even under these settings, RAEv2 still performs below LaLa-World.

Experimental setup:

We follow the matched setup in Section 4.2 and change only the training budget and the number of denoising steps.

Results:

Table II. Planning success rates (%)

ModelTrainingDenoising stepsWallPush-TMetaWorld-RMetaWorld-RWRoboCasa-PAveragePlanning time (min/traj)
RAEv212 h568.8 (4.7)47.9 (5.1)87.5 (4.8)43.8 (7.2)28.1 (7.9)55.2 (2.7)102.9 (13.9×)
LaLa-World12 h–94.8 (2.3)52.1 (5.1)87.5 (4.8)62.5 (7.0)53.1 (8.8)70.0 (2.7)7.4
RAEv248 h570.8 (4.6)54.2 (5.1)87.5 (4.8)47.9 (7.2)34.4 (8.4)59.0 (2.8)102.9 (13.9×)
RAEv248 h2075.0* (10.8)50.0* (12.5)––31.2* (11.6)–399.9 (54.0×)
LaLa-World48 h–93.8* (6.1)56.2* (12.4)75.0 (6.2)52.1 (7.2)50.0* (12.5)65.4 (4.2)7.4

We evaluate the entries marked with * on 16 episodes per environment, as 20 denoising steps quadruple the planning time.

  • With 48 hours of training, RAEv2 improves from 55.2 to 59.0 on average, but LaLa-World still outperforms it.
  • Increasing the denoising steps from 5 to 20 yields comparable planning performance while quadrupling the planning time.
  • These results confirm that our setting is fair to RAEv2. We have added Table II to Appendix C.8.

We hope our responses and additional experiments have addressed the reviewer's concerns. If any concerns remain, we would be happy to address them during the discussion.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal

Dear Reviewer BNhw,

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would like to follow up to see if you have any further questions or anything else we can help clarify. We hope our responses have addressed your concerns, and if any remain, we would be glad to discuss them further.

Thank you,
Authors of paper 3899

Reviewer_a2smOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LaLa-World uses natural language as the action interface for a latent world model, allowing one predictor to learn from heterogeneous control trajectories and image/video transitions. A pretrained language model predicts future visual features conditioned on these descriptions. The model supports planning and provides an initialization for LaLa-Policy, which jointly predicts actions and future features.

Critical strengths
  1. The language interface provides a useful way to combine different action definitions and data sources. Experiments with numerical and qualitative language actions, including zero-shot RoboCasa planning, demonstrate capabilities beyond a single control setting.

  2. The evaluation connects world-model learning, planning, and policy initialization. The RAEv2 comparison matches the visual encoder, language interface, training data, and approximate predictor size, providing useful evidence for the performance and efficiency of latent prediction. Policy initialization ablations further support the value of world-model pretraining.

Critical weaknesses

The main comparison with existing latent world models has a planning confound. Appendix D.4.2 explains that LaLa sums actions across substeps and optimizes these sums, while the baselines concatenate substep actions. This changes both the CEM search dimensionality and the execution parameterization. Consequently, some planning gains may come from the simpler search space. This concern limits the interpretation of Table 1; the separately matched RAEv2 comparison remains informative.

Questions for authors

How much of the Table 1 advantage remains when DINO-WM and JEPA-WMs use the same substep aggregation, equal distribution across execution substeps, and CEM search dimensionality as LaLa?

4 comments
AuthorsOpenReview ↗

Response to Reviewer a2sm

We thank Reviewer a2sm for their time and constructive feedback. We are glad that the reviewer recognizes (1) the useful language interface combining different action definitions and data sources, and (2) the value of latent world models in terms of efficiency and policy initialization. We have carefully considered the concern raised and provide our response below.

W1. According to Appendix D.4.2, the baselines concatenate the five substep actions, whereas LaLa-World sums each component. Do the planning gains come from the lower-dimensional, simpler search space?

Q1. How much of the Table 1 advantage remains under the same substep aggregation (format B in Table I)?

We thank the reviewer for pointing out an important detail. To address this, we conduct additional experiments with substep aggregation for DINO-WM and JEPA-WMs and substep concatenation for LaLa-World. The results show that LaLa-World's gains do not come from the simpler search space, and that the baselines do not benefit from substep aggregation.

Experimental setup:

  • We retrain DINO-WM and JEPA-WMs with aggregated actions (format B in Table I), so they use the same substep aggregation, equal distribution across execution substeps, and CEM search dimensionality as LaLa-World.
  • We also train LaLa-World with concatenated language actions (format C).

Table I. Action formats, with examples from Push-T (5 substeps per step, 2-D action)

IndexAction formatExampleTraining pairUsed by
AConcatenated substep actions (numerical)[0.15, −0.27, 0.06, −0.09, −0.03, 0.04, −0.11, 0.05, −0.54, −0.04](xt,A,xt+1)(x_t, A, x_{t+1})DINO-WM, JEPA-WMs (Table 1)
BAggregated substep actions (numerical)[−0.47, −0.32](xt,B,xt+1)(x_t, B, x_{t+1})DINO-WM, JEPA-WMs (new)
CConcatenated substep actions (language)move the pusher 0.15 right and 0.27 up, then move the pusher 0.06 right and 0.09 up, then move the pusher 0.03 left and 0.04 down, then move the pusher 0.11 left and 0.05 down, then move the pusher 0.54 left and 0.04 up(xt,C,xt+1)(x_t, C, x_{t+1})LaLa-World (new)
DAggregated substep actions (language)move the pusher 0.47 left and 0.32 up(xt,D,xt+1)(x_t, D, x_{t+1})LaLa-World (Table 1)

Results:

Table II. Planning success rates (%)

ModelAction formatWallPointMazePush-TMetaWorld-RMetaWorld-RWAverage
DINO-WMA (Table 1)90.6 (3.0)59.4 (5.0)65.6 (4.8)52.1 (7.2)43.8 (7.2)62.3 (2.5)
DINO-WMB89.6 (3.1)57.3 (5.0)68.8 (4.7)43.8 (7.2)33.3 (6.8)58.5 (2.5)
JEPA-WMsA (Table 1)79.2 (4.1)60.4 (5.0)68.8 (4.7)66.7 (6.8)47.9 (7.2)64.6 (2.6)
JEPA-WMsB89.6 (3.1)43.8 (5.1)65.6 (4.8)68.8 (6.7)37.5 (7.0)61.0 (2.5)
LaLa-WorldD (Table 1)95.8 (2.0)56.2 (5.1)81.2 (4.0)75.0 (6.2)52.1 (7.2)72.1 (2.3)
LaLa-WorldC93.8 (2.5)60.4 (5.0)78.1 (4.2)70.8 (6.6)52.1 (7.2)71.0 (2.4)
  • With aggregated actions (format B), DINO-WM and JEPA-WMs drop by 3.8 and 3.6 points on average. A lower-dimensional search space does not help them. As mentioned in Appendix D.4.2 Line 1610, aggregation discards variation within a step, which likely causes this drop.
  • Compared with the baselines, LaLa-World is robust to the action format, achieving 71.0 and 72.1 on average with formats C and D, respectively.
  • These results indicate that the gains in Table 1 do not come from the simpler search space. We have added both tables and this discussion to Appendix D.4.2.

We hope our responses and additional experiments have addressed the reviewer's concerns. If any concerns remain, we would be happy to address them during the discussion.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal

Dear Reviewer a2sm,

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would like to follow up to see if you have any further questions or anything else we can help clarify. We hope our responses have addressed your concerns, and if any remain, we would be glad to discuss them further.

Thank you,
Authors of paper 3899

Thank you for the rebuttal. The new action-format experiments resolve my main concern. I maintain my original score of 3.

AuthorsOpenReview ↗

Thanks for your time and effort

We are glad that our responses addressed the reviewer’s main concern, and we sincerely appreciate the reviewer’s time and valuable feedback. If there is anything else we could clarify to further strengthen the paper, we would be happy to address it.

Thank you so much.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear ACs and Reviewers,

We thank all the Reviewers for their time and thoughtful comments. We are particularly grateful for the acknowledgment of the strengths of our work, including:

  • Our language interface is useful for combining different actions and data sources (a2sm and oPHe).
  • Our recipe is simple and also learns from video (BNhw).
  • Our multi-domain training results are encouraging (oPHe).
  • Our comparison with a video generation model, RAEv2, is interesting (BNhw) and shows a concrete efficiency advantage (a2sm, BNhw, and oPHe).
  • Our policy fine-tuning experiments support the value of latent world models (a2sm and oPHe).
  • Our evaluation is broad (BNhw), with particularly helpful diagnostics (oPHe).

We have carefully considered the Reviewers' feedback on the weaknesses and provided our responses as follows.

Clarifications & discussions

  • We clarified that scaling means extending latent world models to language actions, video, and policy, and toned down the claim. (BNhw-W2)
  • We discussed UniSim and clarified that a language interface enables multi-domain training, video learning, and policy learning for latent world models. (BNhw-W1-a)
  • We moved the vision encoder ablation from the appendix to the main paper, including the dual-encoder result. (oPHe-W2-b/Q2-b)
  • We clarified what is matched in the RAEv2 comparison setup, consolidating the main paper and appendix. (oPHe-W2-c/Q2-d)
  • We made explicit which RAEv2 setting our planning speedup is measured against. (oPHe-W2-d)
  • We clarified which pretrained model and data are used for policy fine-tuning. (oPHe-W2-f/Q3-b/Q3-c)

Additional experiments

  • We matched the action format in Appendix D.4.2 between LaLa-World and the baselines and showed that our gains remain. (a2sm-W1/Q1 and oPHe-W1-a/Q1)
  • We extended the ablation in Table C, showing that language actions outperform numerical actions when training across multiple domains. (BNhw-W1-b/Q1 and oPHe-W1-b)
  • We reported the planning time and memory of the dual-encoder model evaluated in Table D. (oPHe-W2-a/Q2-a)
  • We trained RAEv2 longer and evaluated it with more denoising steps, where LaLa-World retains its advantage. (BNhw-Q2)
  • We reported a planning-time breakdown, including text encoding, for LaLa-World and RAEv2. (oPHe-Q2-c)
  • We repeated policy fine-tuning with three seeds and confirmed that pretrained LaLa-World improves LaLa-Policy. (oPHe-W2-e/Q3-a)

We have incorporated the Reviewers' valuable feedback into the revised manuscript. We hope that our responses address all of the concerns and would be happy to further discuss any remaining points.

We again appreciate the constructive feedback and the opportunity to improve our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.