- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents OneLA, a framework for learning a shared control representation from videos across games, navigation, driving, and robotics, where action interfaces differ and action labels are often unavailable. It learns continuous latent actions from frame pairs using an orthonormal motion dictionary, then maps native controls into that space and trains a shared world model to predict video and latent actions. The authors report improved action readout on most evaluated datasets and better cross-view transfer. In action-cycle transfer, they report 15.5% lower translation error and 24.8% lower rotation error than OLaF.
Critical strengths
- The paper addresses an important problem.Control interfaces differ across domains, while many videos lack action annotations. Learning a shared latent action interface from such videos could make heterogeneous data more useful for controllable world modeling.
- The system has broad scope. OneLA learns from games, navigation, driving, and robotics data, and evaluates these domains with a shared video world model. This provides a substantial testbed for multi-domain action representations.
- Several quantitative results are encouraging. In Table 2, OneLA improves linear-probe macro-F1 over the strongest reported baseline by 5.30 points on Game and 5.63 points on AgiBot. It also improves cross-view action readout, while Table 3 reports a 24.8% reduction in action-cycle transfer rotation error relative to OLaF.
Critical weaknesses
- The methodological novelty is not sufficiently clear, and its contribution is difficult to separate from training scale. The framework combines existing techniques for latent action modeling, orthogonal dictionaries, regularization, and control adaptation with a large collection of data from multiple domains. The paper’s own experiments show that increasing model size and training data can substantially improve performance. However, the baselines are not fully matched in their pretraining data and model capacity. It is therefore unclear how much of the improvement comes from the proposed design and how much comes from training scale.
- The orthogonal dictionary lacks direct evidence of its impact on downstream performance. This is a central design choice, but its evaluation mainly shows changes in similarity between action representations. These results indicate that the representation changes, but do not establish that the design improves action transfer or video prediction. The paper does not sufficiently quantify how the main results change when the orthogonal dictionary is removed, leaving its necessity and practical contribution unclear.
- Generalization to new domains and robots is insufficiently evaluated. Most evaluated domains are already included in the training mixture, and separate adapters are trained with labeled data for different control interfaces. The results demonstrate that one model can handle multiple trained domains, but provide limited evidence of its performance on a new robot or control interface. The cross-view experiments offer some evidence of transfer, yet do not establish whether the shared action space reduces the data requirements or adaptation effort for a new robot.
- The lack of closed-loop experiments on real robots limits the evidence of practical value. The robotics evaluation mainly consists of representation tests, offline video prediction, and generated videos of manipulation. It does not demonstrate a robot executing actions from current observations and adjusting its behavior using feedback. Whether the model can support task completion under physical contact, execution errors, or failed actions therefore remains unknown. Visually plausible manipulation videos do not establish successful physical execution, and the RT-1 video prediction results also fall below the strongest baseline. The paper consequently provides limited direct evidence of benefits for actual robot tasks.
Questions for authors
- Could the authors provide comparisons using the same amount of data, model capacity, and training budget to clarify how much improvement comes from the proposed method itself?
- Could the authors report action-transfer and video-prediction results without the orthogonal dictionary? Does it retain a clear advantage over simpler alternatives, such as normalizing the dictionary columns?
- Could the authors evaluate a robot or control interface excluded from training and compare the amount of data needed for adaptation, to establish the practical benefit of the shared action space?
- Have the authors conducted any closed-loop task experiments on real robots? If not, could they include at least one challenging manipulation task and compare task success and recovery from execution errors against a baseline, to assess whether the reported improvements translate into practical benefits? I would be willing to raise my score if the authors satisfactorily address these concerns through clarification or additional experimental evidence.