acceptodds
Under review as a conference paper at ICLR 2027

OneLA: Learning Many Worlds from Latent Actions

Abstract

Learning world models across diverse domains calls for a unified control space, yet action interfaces vary across domains and action annotations are often scarce. Latent actions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains remains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we introduce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions provide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the current visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real-scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent Action Model, World Model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5rJJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OneLA, a framework for learning a shared control representation from videos across games, navigation, driving, and robotics, where action interfaces differ and action labels are often unavailable. It learns continuous latent actions from frame pairs using an orthonormal motion dictionary, then maps native controls into that space and trains a shared world model to predict video and latent actions. The authors report improved action readout on most evaluated datasets and better cross-view transfer. In action-cycle transfer, they report 15.5% lower translation error and 24.8% lower rotation error than OLaF.

Critical strengths
  1. The paper addresses an important problem.Control interfaces differ across domains, while many videos lack action annotations. Learning a shared latent action interface from such videos could make heterogeneous data more useful for controllable world modeling.
  2. The system has broad scope. OneLA learns from games, navigation, driving, and robotics data, and evaluates these domains with a shared video world model. This provides a substantial testbed for multi-domain action representations.
  3. Several quantitative results are encouraging. In Table 2, OneLA improves linear-probe macro-F1 over the strongest reported baseline by 5.30 points on Game and 5.63 points on AgiBot. It also improves cross-view action readout, while Table 3 reports a 24.8% reduction in action-cycle transfer rotation error relative to OLaF.
Critical weaknesses
  1. The methodological novelty is not sufficiently clear, and its contribution is difficult to separate from training scale. The framework combines existing techniques for latent action modeling, orthogonal dictionaries, regularization, and control adaptation with a large collection of data from multiple domains. The paper’s own experiments show that increasing model size and training data can substantially improve performance. However, the baselines are not fully matched in their pretraining data and model capacity. It is therefore unclear how much of the improvement comes from the proposed design and how much comes from training scale.
  2. The orthogonal dictionary lacks direct evidence of its impact on downstream performance. This is a central design choice, but its evaluation mainly shows changes in similarity between action representations. These results indicate that the representation changes, but do not establish that the design improves action transfer or video prediction. The paper does not sufficiently quantify how the main results change when the orthogonal dictionary is removed, leaving its necessity and practical contribution unclear.
  3. Generalization to new domains and robots is insufficiently evaluated. Most evaluated domains are already included in the training mixture, and separate adapters are trained with labeled data for different control interfaces. The results demonstrate that one model can handle multiple trained domains, but provide limited evidence of its performance on a new robot or control interface. The cross-view experiments offer some evidence of transfer, yet do not establish whether the shared action space reduces the data requirements or adaptation effort for a new robot.
  4. The lack of closed-loop experiments on real robots limits the evidence of practical value. The robotics evaluation mainly consists of representation tests, offline video prediction, and generated videos of manipulation. It does not demonstrate a robot executing actions from current observations and adjusting its behavior using feedback. Whether the model can support task completion under physical contact, execution errors, or failed actions therefore remains unknown. Visually plausible manipulation videos do not establish successful physical execution, and the RT-1 video prediction results also fall below the strongest baseline. The paper consequently provides limited direct evidence of benefits for actual robot tasks.
Questions for authors
  1. Could the authors provide comparisons using the same amount of data, model capacity, and training budget to clarify how much improvement comes from the proposed method itself?
  2. Could the authors report action-transfer and video-prediction results without the orthogonal dictionary? Does it retain a clear advantage over simpler alternatives, such as normalizing the dictionary columns?
  3. Could the authors evaluate a robot or control interface excluded from training and compare the amount of data needed for adaptation, to establish the practical benefit of the shared action space?
  4. Have the authors conducted any closed-loop task experiments on real robots? If not, could they include at least one challenging manipulation task and compare task success and recovery from execution errors against a baseline, to assess whether the reported improvements translate into practical benefits? I would be willing to raise my score if the authors satisfactorily address these concerns through clarification or additional experimental evidence.
Reviewer_xXG1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OneLA, a framework that learns a shared continuous latent-action interface for world models across games, navigation, driving, and robotics. Its central contribution is a latent-action model that represents visual transitions through coefficients over a learned orthonormal motion dictionary. Training combines frame reconstruction with KL and sparsity regularization, without action labels or explicit cross-domain alignment objectives during latent-action pretraining. Lightweight MLPs subsequently map native control commands into this space, while a shared diffusion transformer jointly predicts video and realized-transition latent actions using labeled and unlabeled data.

Validation includes frozen-representation linear probes, single-dimension action readout, and cross-view transfer across six datasets spanning four domains. World-model experiments evaluate command-conditioned video prediction and action-cycle transfer, showing improved motion consistency and competitive visual fidelity relative to OLaF, AdaWorld, and DiLA. Ablations examine orthogonal parameterization, model and data scaling, training-data composition, latent-action conditioning, and auxiliary action prediction. Overall, the results support the usefulness of the shared latent-action interface for representation learning and controllable world modeling, although improvements in visual fidelity are not uniform across datasets and metrics.

Critical strengths
  1. OneLA combines an orthonormal motion dictionary with reconstruction, KL, and sparsity regularization, without introducing external supervisory cues during latent-action pretraining. Strong single-dimension readout and cross-view probing results suggest that this simple formulation learns accessible, transferable action information. This makes the approach promising for scaling latent-action learning to large, heterogeneous video collections without additional annotation requirements.
  2. The paper evaluates both directions of first-/third-person transfer on MIND, fitting probes only on source-view labels, and tests 40 action-transfer cycles across game and real scenes. These complementary protocols assess whether action information transfers across viewpoints and whether source motion survives transfer to another scene and back. The improved cross-view F1 and reduced cycle-transfer errors provide concrete quantitative support for the proposed shared action interface.
Critical weaknesses
  1. The latent-action comparisons are not controlled for pretraining data. OneLA is pretrained on a large, diverse collection of 1.79M video clips, whereas the baselines, including OLaF, use their released LAM checkpoints. Although downstream world-model training uses the same data mixture, this does not control for differences in latent-action pretraining. It therefore remains unclear whether the improved alignment and transfer reflect the proposed formulation or broader training-data coverage. A comparison that retrains OneLA and OLaF on the same video collection with comparable training budgets is needed to establish whether OneLA can achieve comparable or better alignment without additional supervisory cues.
Questions for authors

I appreciate the simplicity of the latent-action pretraining framework; however, given the differences in pretraining data, could the authors provide a comparison with Olaf-World under matched data and training budgets to clarify whether OneLA achieves comparable or better semantic alignment through its formulation rather than broader data coverage?

Reviewer_3pKXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OneLA learns continuous latent actions from videos using an orthonormal dictionary, reconstruction, KL regularization, and sparsity. Interface-specific MLPs map native controls into this space, which conditions a shared DiT video world model across games, navigation, driving, and robotics. Experiments show improvements in action readout, cross-view transfer, and several motion-prediction metrics, although video-quality gains vary across datasets.

Critical strengths
  • This work connects heterogeneous controls to one world-model backbone. Latent-action pretraining requires no action labels, while supervised command adapters provide a route to controllable generation. Using one shared checkpoint across the quantitative benchmarks strengthens this contribution.
  • The results are encouraging, particularly the cross-view readout gains on MIND. The world-model comparisons also match the training mixture and fine-tuning budget, making them more informative than comparisons between unrelated generation systems.
Critical weaknesses
  • The contribution of the orthogonal dictionary is not sufficiently isolated. Figure 4 shows improved prototype similarities, but does not establish how removing QR affects action readout or downstream motion prediction. Meanwhile, the released baseline LAMs have different pretraining setups, as documented for OLaF and DiLA. Matching downstream fine-tuning therefore leaves uncertainty about how much improvement comes from the proposed regularization.
  • The overall pipeline closely resembles DreamDojo, which also learns continuous latent actions from frame pairs to condition a video world model. Although OneLA introduces an orthogonal dictionary and joint video–action prediction, their added value over DreamDojo is not sufficiently established. Including DreamDojo as a baseline would help clarify the methodological contribution.
Questions for authors
  • Under the same architecture, data, and training budget, how do removing QR and sparsity affect action-readout F1 and downstream motion errors?
  • Do all world-model baselines receive equivalent command-adapter training and auxiliary action-prediction supervision, or are only the backbone, data, and update budget matched?
  • Can you directly evaluate motion preservation in the first transferred rollout, alongside round-trip RPE, and provide paired uncertainty estimates for the transfer gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.