acceptodds
Under review as a conference paper at ICLR 2027

WALA: Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

Abstract

Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present , a framework named for . Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves % average success on RoboTwin and % on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation Learning, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iPpkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WALA learns latent actions from semantic and depth changes, then uses both encoder-derived distillation targets and decoder-based future prediction to supervise the policy tokens that directly generate robot actions. Robot action labels supervise labeled examples, while action-free videos provide auxiliary supervision. The LAM and future observations are unnecessary at inference. Evaluation covers RoboTwin, RoboCasa, and real-robot manipulation, including human-video scaling and target-task zero-shot/few-shot transfer. The central contribution is the combination of semantic-geometric latent learning and complementary supervision of the action-generating representation.

Critical strengths
  1. The controlled policy ablations provide convincing evidence of practical benefit. With the same policy backbone and robot data, WALA improves RoboCasa success from 54.2% to 75.2% (Table 3), exceeding world-model supervision alone (71.0%) and distillation alone (67.6%). This supports the usefulness of combining the two signals, rather than relying only on comparisons between independently developed systems. The fixed-label experiment also improves success from 53.9% to 67.8% as additional action-free robot videos are included during policy training (Figure 3).

  2. The real-robot experiments directly test the intended use of human videos. At 200 robot demonstrations per base task, adding 400 human videos per task raises average success from 75.0% to 83.3% (Table 4). With no target-task robot demonstrations, adding 150 target-task human videos improves bread pick-place from 9/30 to 24/30 and cabinet opening from 4/30 to 15/30 (Table 5). These are meaningful transfer results under the stated setting of a policy also trained on related robot tasks. The inference-time removal of auxiliary prediction machinery makes the approach practically attractive.

Critical weaknesses
  1. The evidence isolates the overall training recipe more clearly than its proposed representation-level explanation. There is no matched learned-LAM ablation replacing semantic/depth deltas with absolute features while holding architecture and pretraining fixed. Table 13 compares raw visual representations with linear probes, which does not establish the effect of delta modeling on the learned policy. Likewise, the 2.2-point geometry comparison uses separately pretrained LAMs with different architectures (Appendix A.3). Attention maps and retrieval examples are suggestive, but do not resolve these attribution questions. This limits how strongly the paper establishes why its particular representation is preferable.

  2. The main policy results lack reported variability across independent training runs; the three-seed representation probes do not address this uncertainty. This matters especially for the 2.2-point geometry gain, the 1.9-point readout comparison, and the 0.09-point RoboTwin lead of WALA-Wan over LingBot-VA. Moreover, WALA's 25,224-hour LAM pretraining is not matched across competing methods. Appendix A.3 usefully matches target-task training/evaluation budgets for Moto and UniVLA, but does not establish matched pretraining or backbone capacity. Thus, the strongest evidence is the internal ablation and video-scaling evidence, rather than a clean attribution of leaderboard differences to the proposed method.

Overall, I lean toward weak acceptance: the substantial controlled improvements and real human-video transfer outweigh these limitations, although the mechanistic and fine-grained comparative claims need more cautious support.

Questions for authors
  1. Representation attribution: Is there a matched experiment comparing learned delta-based and absolute-feature LAMs with the same architecture, data, and optimization? Table 13 does not appear to be such a control. Please also clarify how much of the semantic-only versus semantic-geometric comparison could be attributable to architecture differences. Existing controlled evidence would strengthen the representation claim; otherwise, please narrow the causal interpretation.

  2. Robustness of the reported differences: How many independently trained policies underlie the main tables, and what uncertainty can you report for the key Table 3 comparisons and the smaller readout/leaderboard differences? Please distinguish rollout variability from training-seed variability. This would affect whether the smaller gains support component-level conclusions, even though the larger overall gains are persuasive.

3 comments
AuthorsOpenReview ↗

Response to Reviewer iPpk (1/2)

Thank you for carefully reading our paper and providing constructive suggestions on representation design and experimental reliability. We appreciate the time and attention you have given to our experiments and their interpretation. Below, we clarify the comparisons and report additional experiments addressing your concerns.

W1 & Q1: Delta Modeling and Geometric Information

Delta versus absolute representations. We have added the requested comparison between two learned LAMs. Both use the same architecture, pretraining data, future-frame sampling, and optimization settings. The difference is whether the LAM encodes and predicts changes relative to the current observation or absolute features. We then train the corresponding policies with the same backbone, action head, robot data, and downstream training and evaluation settings.

Both stages use the same 10% RoboCasa episode subset. The results are summarized below.

LAM representationSuccess (%)
Absolute features47.8
Deltas51.3

Delta improves success by 3.5 percentage points. This experiment evaluates delta modeling through the performance of policies trained with the learned LAMs. It therefore adds direct downstream evidence to the raw-feature comparison in Table 13 of the original manuscript.

Adding geometric information. The architecture in Section 3.2 explains how geometric information enters the LAM. The semantic-only model encodes and predicts changes in DINOv3 features. Building on this design, the semantic–geometric model adds a depth-encoding branch and a corresponding depth-prediction branch. The encoder combines information from the semantic and geometric branches into shared latent-action tokens, and the decoder uses those tokens to predict both semantic and depth changes.

The depth branches provide the pathway through which geometric information enters the representation and its training objectives. The architectural difference between these variants is therefore part of introducing geometric information into our LAM. With the policy backbone and action head unchanged, the ablation evaluates the benefit of adding this geometric pathway and its depth supervision to semantic latent-action learning.

As described in Appendix A.3, the two variants have separately pretrained LAM checkpoints. We use the existing checkpoint for each configuration in the additional downstream experiments. The original comparison in Table 3 gives 68.8% for semantic-only prediction and 71.0% for the semantic–geometric world model. To assess whether this improvement is consistent across policy-training runs, we have now repeated downstream training for both configurations, as detailed below.

AuthorsOpenReview ↗

Response to Reviewer iPpk (2/2)

W2 & Q2: Repeated Runs and Method Comparisons

Variation across downstream training runs. The baseline results cited in our main benchmark tables are generally published as single success rates without standard deviations across independent training runs. We follow this format. WALA's original results use one training run and one evaluation per configuration.

We have now repeated downstream training for the semantic-only and semantic–geometric world-model-supervision variants in Table 3. For each configuration, we reuse its existing pretrained LAM checkpoint and train two additional policies with seeds 43 and 44. Together with the original seed-42 run, this gives three independently trained policies evaluated using the same benchmark protocol. The table reports the mean and sample standard deviation of their success rates.

ConfigurationSuccess rate (%, mean ± SD)
Semantic only68.9 ± 0.2
Semantic + geometry71.4 ± 0.3

Adding the geometric branch improves performance with every training seed and yields an average gain of 2.5 percentage points. These runs assess downstream training variability with the pretrained LAM held fixed within each configuration.

Action readout. This experiment tests whether a small set of latent-action tokens provides sufficient information for control. Matching the performance of a head that reads all backbone tokens, or achieving a modest improvement, already supports this design objective. Latent-only readout achieves 75.2%, compared with 73.3% using all backbone tokens (Appendix B.5). The attention analysis in the same appendix shows that, even when all backbone tokens are available, the action head primarily attends to latent-action tokens. The performance and attention results thus support using these compact tokens as the direct input to the action head.

Performance across benchmarks. A central goal of WALA is to learn transferable interaction knowledge from videos without action annotations and use it to improve robot control. LingBot-VA benefits from large-scale action-labeled pretraining and explicitly predicts future video states in latent space, using inverse dynamics to generate actions conditioned on those predictions. In contrast, WALA pretrains its LAM without action annotations and requires only the policy and action head at inference. Without inference-time future-video generation, WALA-Wan nevertheless achieves 92.33% on RoboTwin, compared with LingBot-VA's 92.24% (Table 1), matching this strong baseline with a slightly higher reported success rate.

To extend this comparison beyond RoboTwin, we trained LingBot-VA on the full RoboCasa training data, obtaining 56.8%, compared with WALA's 75.2% reported in Table 2, a difference of 18.4 percentage points. The table also reports efficiency measured on the same H100 platform at a per-GPU batch size of 1, using each method's own training and inference configuration.

MethodRoboCasa success (%)Training (s/step)Training memory (GB)Inference (s/call)
LingBot-VA56.812.99444.015.068
WALA75.20.63823.690.070

WALA uses 46.2% less training memory, with 20.4× faster training steps and 72.4× faster inference calls. These results show near-equal RoboTwin success, higher RoboCasa success, and lower measured training and inference costs.

Pretraining data and backbone capacity. As described in Appendix A.3, our original Moto and UniVLA comparisons match downstream training and evaluation budgets while using each method's own pretrained model and backbone. Our new comparison fixes the policy architecture and training data to address these differences.

We implement UniVLA-style, villa-X-style, and VLA-JEPA-style within WALA using the same Qwen3-VL-4B backbone, action head, 10% RoboCasa episode subset, and downstream training and evaluation settings. All methods, including WALA, use this same subset for both video pretraining and policy training. Each style follows the corresponding method's pretraining procedure and supervision design within the shared framework.

MethodSuccess (%)
VLA-JEPA-style43.1
villa-X-style45.8
UniVLA-style46.4
WALA51.3

WALA improves over VLA-JEPA-style, villa-X-style, and UniVLA-style by 8.2, 5.5, and 4.9 percentage points, respectively. These results demonstrate an advantage for WALA's representation and supervision design under matched data and policy architecture, complementing the comparisons between the original methods.

We will incorporate these additional experiments and discussions into the revised manuscript. We hope that our responses address all of your concerns. Please let us know if you have any further questions or would like clarification on any point. We would be happy to provide further details.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer iPpk,

Thank you again for your thoughtful comments on representation design and experimental reliability. We would appreciate your feedback on the matched delta-versus-absolute comparison and repeated policy-training results provided in our response. We hope these additions address your concerns and would welcome the opportunity to clarify any remaining questions.

Thank you for your time and continued consideration.

Best regards,

Authors

Reviewer_ZH7aOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

WALA pretrains a latent-action model on video without action labels. The encoder reads the current DINOv3 features and depth, plus their differences from four future frames; the decoder predicts those differences (Eqs. 2–6). During policy learning, the frozen encoder supplies a distillation target, the decoder predicts the same differences, and the action head reads only the policy’s latent-action tokens (Eqs. 8–11). The latent-action model is unused at inference. Qwen3-VL reaches 75.2% on RoboCasa-GR1-Tabletop, and Wan reaches 92.33% on RoboTwin (Tables 1–2). With 50 demonstrations and 400 human videos per base task, the real robot reaches 74.2% (Table 4).

Critical strengths

On one backbone, the three training signals add up. In Table 3, action-only training scores 54.2%. Semantic and depth prediction without pretraining reaches 67.8%; a pretrained semantic–geometric world model reaches 71.0%; distillation alone reaches 67.6%; all three reach 75.2%. Depth adds 2.2 points over semantic prediction alone (71.0 versus 68.8). The rows share the backbone and the robot data.

Matched human video raises success with the real-robot policy held fixed. At 50 demonstrations per base task, 0, 150, and 400 human videos yield 63.3%, 70.0%, and 74.2% over 30 trials (Table 4, Fig. 5b). At 200 demonstrations, 400 human videos reach 83.3%.

Critical weaknesses

Joint training of policy latent actions and a latent-action world model is already implemented, and WALA does not isolate what it changes. LAPA is the two-stage pipeline in Fig. 1(a) [1]. UniVLA learns DINO-space latent actions from robot and human video and decodes them into control [2]. Moto co-finetunes motion tokens with an action head [3]. villa-X predicts latent actions and robot actions together with visual and proprioceptive forward models [4]. UniT feeds one token set to both a VLA and a world model [5]. LDA-1B predicts DINO-space dynamics [6]. VLA-JEPA is the closest implementation [7]: the VLM emits latent actions, a world model decodes them into future latent states, robot data uses LFM+βLWM\mathcal{L}_{\mathrm{FM}}+\beta\mathcal{L}_{\mathrm{WM}} (Eq. 9 in [7]), action-free video uses only the world-model term, and the world model is dropped at inference. WALA does not cite it. WALA’s additions are DINOv3 and depth deltas (Eqs. 1–3) and distillation from a future-conditioned encoder (Eq. 7). Table 3 only sums WALA’s own losses. The reproduced UniVLA (51.2%) and Moto (45.6%) runs are those codebases finetuned on RoboCasa (Appendix A.3), not those objectives inside this Qwen3-VL-4B model and 25,224-hour corpus (Table 6).

The comparisons do not show action-label efficiency, a clear RoboTwin lead, a matched real-robot start, or human-video zero-shot. With 10% of the action labels, WALA scores 53.9% and the action-only baseline scores 18.1% (Fig. 3, left). This looks like a gain from the same small action budget. Appendix A.1 states that the latent-action model was pretrained on the full RoboCasa training videos, so it has already seen the other 90%. The baseline has not. The extra success includes that video, and the comparison does not show that WALA makes better use of the 10% labels. On RoboTwin, WALA-Wan’s 92.33% is 0.09 above the published LingBot-VA number, and standard WALA’s 91.71% is below both LingBot-VA and Fast-WAM (Table 1). Those numbers do not establish a lead. The real-robot policy is initialized from a WALA checkpoint already trained on RoboTwin (Appendix A.3). The comparison with π0 and π0.5 therefore starts the methods from different points. Bread Pick-Place succeeds on 9 of 30 trials with no target-task demonstrations and no human video (Table 5). That zero-shot residue comes from co-training on the other robot tasks, so the human videos are not the sole source of the transfer.

References

[1] Ye, Seonghyeon, et al. “Latent Action Pretraining from Videos.” International Conference on Learning Representations, 2025.

[2] Bu, Qingwen, et al. “UniVLA: Learning to Act Anywhere with Task-Centric Latent Actions.” arXiv preprint arXiv:2505.06111 (2025).

[3] Chen, Yi, et al. “Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos.” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 19752–19763.

[4] Chen, Xiaoyu, et al. “villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models.” arXiv preprint arXiv:2507.23682 (2025).

[5] Chen, Boyu, et al. “UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling.” arXiv preprint arXiv:2604.19734 (2026).

[6] Lyu, Jiangran, et al. “LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion.” arXiv preprint arXiv:2602.12215 (2026).

[7] Sun, Jingwen, et al. “VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model.” arXiv preprint arXiv:2602.10098 (2026).

Questions for authors
  1. On the same Qwen3-VL backbone, action head, and RoboCasa data, replace the delta model with VLA-JEPA’s joint latent-action world model (Eq. 9 in [1]), then with UniVLA [2] and villa-X [3]. Report success beside Table 3.
  2. Retrain the latent-action model for Fig. 3 (left) on only the videos tied to the 10% labeled subset. The reported model has already seen the full training set (Appendix A.1).

[1] Sun, Jingwen, et al. “VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model.” arXiv preprint arXiv:2602.10098 (2026).

[2] Bu, Qingwen, et al. “UniVLA: Learning to Act Anywhere with Task-Centric Latent Actions.” arXiv preprint arXiv:2505.06111 (2025).

[3] Chen, Xiaoyu, et al. “villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models.” arXiv preprint arXiv:2507.23682 (2025).

6 comments
AuthorsOpenReview ↗

Response to Reviewer ZH7a (1/2)

Thank you for carefully reading our paper, comparing it with related work, and providing concrete experimental suggestions. We appreciate the time and effort you have devoted to evaluating our work. Below, we clarify the methodological differences and report additional controlled experiments addressing your concerns.

W1 & Q1: Methodological Differences and Controlled Comparisons

Related work and WALA's contributions. In the original manuscript, we discussed LDA-1B in Section 2.1 and LAPA, UniVLA, Moto, villa-X, and UniT in Section 2.2, distinguishing their prediction targets, latent representations, and use in policy learning. We will add VLA-JEPA and expand the comparison. WALA's contributions center on semantic and geometric delta modeling and complementary encoder distillation and decoder prediction applied to the same latent-action tokens read directly by the action head. This design also enables action-free videos to improve downstream policy learning, as evaluated below.

Use of action-free videos during policy learning. VLA-JEPA's reported setup uses human videos during pretraining, followed by fine-tuning on robot demonstrations. WALA extends the use of action-free videos to downstream policy learning, where action-labeled robot demonstrations are jointly trained with human and robot videos without action supervision.

The original experiments directly evaluate the benefit of this downstream video supervision. With the robot action-label budget fixed, adding unlabeled robot videos raises simulation success from 53.9% to 67.8% (Fig. 3). With 50 robot demonstrations per real-world task, adding 0, 150, and 400 human videos raises success from 63.3% to 70.0% and 74.2%, respectively (Table 4 and Fig. 5b). These results show that action-free videos continue to improve the policy during downstream training, beyond their use in pretraining.

Direct latent-to-action pathway. The methods also differ in how the video-supervised representation is connected to robot actions. VLA-JEPA uses latent tokens to condition its world model and additional action tokens to condition its action head. These action tokens attend to the context, including the latent tokens. In WALA, the action head directly reads the latent-action tokens themselves. The same tokens receive latent-action distillation, semantic and geometric delta prediction, and robot action supervision, placing the video-supervised representation directly in the action-generation pathway.

The original appendix provides evidence for this design. Probes of the frozen LAM show that robot-action information can be decoded from its latent representations (Appendix B.3). When the action head can read all backbone tokens, its attention concentrates primarily on latent-action tokens. Reading only this compact token set achieves 75.2% success, compared with 73.3% using all backbone tokens (Appendix B.5). These results support the use of latent-action tokens as a compact representation sufficient for control, without requiring the action head to read the full backbone feature sequence.

Matched method comparisons. To evaluate the representation and supervision choices within a common policy architecture, we implemented UniVLA-style, villa-X-style, and VLA-JEPA-style in WALA. All use the same Qwen3-VL-4B backbone, action head, 10% RoboCasa episode subset, and downstream training and evaluation settings. The same subset is used for both video pretraining and policy training, including for WALA. Each style follows the corresponding method's pretraining procedure and supervision design within this shared framework.

MethodSuccess (%)
VLA-JEPA-style43.1
villa-X-style45.8
UniVLA-style46.4
WALA Absolute47.8
WALA Delta51.3

WALA improves over VLA-JEPA-style, villa-X-style, and UniVLA-style by 8.2, 5.5, and 4.9 percentage points, respectively. These comparisons show an advantage for WALA's representation and supervision design when the policy backbone, action head, and training data are held fixed.

To examine delta modeling specifically, the Delta and Absolute variants additionally share the same LAM architecture, future-frame sampling, and optimization settings. They differ in whether the LAM encodes and predicts changes or absolute features. Delta improves success by 3.5 percentage points, providing downstream policy evidence for the benefit of modeling visual changes.

AuthorsOpenReview ↗

Response to Reviewer ZH7a (2/2)

W2 & Q2: Data Efficiency and Interpretation of the Evaluation Results

Restricted-video experiment. As explained in Appendix A.1, the original label-efficiency experiment uses limited robot action labels while allowing LAM pretraining to access videos from the full RoboCasa training set. It therefore evaluates the setting in which action annotations are scarce but unlabeled training videos are available.

To directly examine the influence of the remaining 90% of videos, we retrained the LAM from scratch using only videos from the fixed 10% labeled RoboCasa subset, then trained the policy on those same episodes. Neither stage uses the other 90% of RoboCasa videos or the large-scale LAM pretraining corpus. This is the WALA model included in the controlled comparison above. It achieves 51.3%, compared with 53.9% in the original experiment, and remains 33.2 percentage points above the 18.1% action-only baseline in Fig. 3. Thus, a substantial improvement over action-only learning persists when both video access and action labels are restricted to the same 10% subset.

Performance across benchmarks. A central goal of WALA is to learn transferable interaction knowledge from videos without action annotations and use it to improve robot control. LingBot-VA benefits from large-scale action-labeled pretraining and explicitly predicts future video states in latent space, using inverse dynamics to generate actions conditioned on those predictions. In contrast, WALA pretrains its LAM without action annotations and requires only the policy and action head at inference. Without inference-time future-video generation, WALA-Wan nevertheless achieves 92.33% on RoboTwin, compared with LingBot-VA's 92.24% (Table 1), matching this strong baseline with a slightly higher reported success rate.

To extend this comparison beyond RoboTwin, we additionally trained LingBot-VA on the full RoboCasa training data. It achieves 56.8%, compared with WALA's 75.2% reported in Table 2, a difference of 18.4 percentage points. We also compared their computational efficiency on the same H100 platform with a per-GPU batch size of 1, using each method's own training and inference configuration.

MethodRoboCasa success (%)Training (s/step)Training memory (GB)Inference (s/call)
LingBot-VA56.812.99444.015.068
WALA75.20.63823.690.070

Compared with LingBot-VA, WALA uses 46.2% less training memory, with 20.4× faster training steps and 72.4× faster inference calls. Together, these results demonstrate that our framework combines competitive RoboTwin performance without inference-time future-video generation, higher success on RoboCasa, and substantially lower measured training and inference costs.

Real-robot initialization. As reported in Appendix A.3, we initialize WALA from a RoboTwin-trained checkpoint before real-world adaptation. This provides WALA with prior robot manipulation experience and brings its starting point closer to π0 and π0.5, both of which already benefit from extensive robot-action pretraining. The action-labeled pretraining datasets of π0 and π0.5 are substantially larger and more diverse than the RoboTwin data used for WALA's initialization. Using the RoboTwin checkpoint therefore helps balance the comparison by ensuring that all three methods begin real-world adaptation with robot-pretrained policies.

Target-task zero-shot transfer. Zero-shot refers to the absence of robot demonstrations for the target task, as described in Section 4.6 and Appendix A.6. Co-training on other robot tasks supplies transferable manipulation skills that can already support some target-task success without human videos. Human videos then provide supervision about the target objects and interactions, helping the policy apply those existing skills to the new task.

The comparison in Table 5 keeps robot co-training fixed and varies the target-task human-video budget. Adding 150 human videos raises bread pick-place success from 9/30 to 24/30 and cabinet opening from 4/30 to 15/30. These improvements isolate the contribution of target-task human videos beyond the skills already obtained from robot co-training.

We will incorporate these additional experiments and discussions into the revised manuscript. We hope that our responses address all of your concerns. Please let us know if you have any further questions or would like clarification on any point. We would be happy to provide further details.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer ZH7a,

Thank you again for your detailed comments and concrete experimental suggestions. We would welcome your feedback on whether our responses, new matched-data comparisons, and restricted-video experiment have addressed your concerns about the methodological differences and data efficiency. Please let us know if any points require further clarification; we would be happy to discuss them.

Thank you for your time and continued consideration.

Best regards,

Authors

Reviewer_eS8TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the high cost of collecting robot demonstrations and the difficulty of directly leveraging human videos without action labels, the paper proposes WALA. It first jointly trains a LAM encoder and decoder to represent semantic and geometric changes in videos through latent actions. It then trains a policy to generate latent actions for robot control through latent-action distillation, future-change prediction, and robot-action supervision. At inference time, only the policy network and action head are used.

Critical strengths

The paper addresses an important problem of broad interest: robot demonstrations are costly to collect, while human manipulation videos are abundant but typically lack action labels (lines 35–45).

The paper identifies limitations of prior work (Section 2.2) and proposes latent actions that integrate interaction semantics and spatial geometry. This design is reasonably motivated.

On RoboCasa-GR1-Tabletop, the proposed method improves success from the previous state of the art of 70.2% to 75.2% (Table 2).

Critical weaknesses

The paper’s LAM pretraining requires approximately 25,600 PPU-hours (line 990), representing a substantial computational cost. However, the paper does not comprehensively compare the total pretraining and downstream training costs across methods. It therefore remains unclear how much of the performance improvement comes from the method’s design rather than additional pretraining data and computational resources.

The two-stage training process requires aligning video-learned latent representations with policy-generated representations and the robot action space. Although the paper addresses this through distillation, adapters, and joint training, the existing experiments do not sufficiently evaluate the stability of this training pipeline.

The performance gains vary across benchmarks. On RoboCasa-GR1-Tabletop, the method improves success from the previous state of the art of 70.2% to 75.2% (Table 2). On RoboTwin, however, its best variant achieves 92.33%, essentially matching the previous best result of 92.24% (Table 1). The current results are therefore insufficient to establish a consistent advantage over strong baselines across benchmarks.

Questions for authors

What are the training costs of the prior methods compared in the paper? Do their pretraining data and computational budgets differ substantially from those of this work?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eS8T (1/2)

Thank you for carefully reading our paper and providing thoughtful, constructive feedback. We appreciate the time and effort you have devoted to evaluating our work. We address your concerns below with additional experimental results, detailed explanations, and references to the original manuscript.

W1 & Q1: Pretraining Cost and Computational Efficiency

Pretraining resources. WALA pretrains its LAM on 25,224 hours of human and robot videos, with all action annotations excluded (Appendix A.1, Table 6). The reported 25,600 PPU-hours (Appendix A.3) correspond to an estimated 4,000 H100-equivalent GPU-hours based on measured training throughput.

The table compares publicly reported pretraining data and computational budgets.

MethodPretraining dataPretraining compute
X-VLA290K episodes6,144 A100 GPU-hours
Motus1.01M sequences18,000 GPU-hours; GPU type unspecified
LingBot-VA16,000 hours1.4T training tokens
RLDX-11.5M episodes12,480 H200 GPU-hours
LDA-1B30,000+ hours4,608 H800 GPU-hours
WALA25,224 hours4,000 H100-equivalent GPU-hours

WALA pretrains its LAM on a video corpus comparable in scale to those of LingBot-VA and LDA-1B. An important difference is that the compared methods incorporate action supervision into their pretraining pipelines, whereas WALA learns transferable interaction representations from visual changes in human and robot videos without requiring action annotations during LAM pretraining. In terms of computation, WALA's estimated 4,000 H100-equivalent GPU-hours are comparable in scale to the reported GPU-hour budgets of LDA-1B and X-VLA, and substantially lower than those of Motus and RLDX-1. LingBot-VA reports its pretraining budget as 1.4 trillion training tokens rather than GPU-hours.

Measured training and inference efficiency. Beyond pretraining, we compare downstream training and inference costs with LingBot-VA and LDA-1B as strong, representative baselines. On the same H100 platform at per-GPU batch size 1, we measure mean training-step time, training memory, and inference-call latency using each method's own training and inference configuration.

MethodTraining (s/step)Training memory (GB)Inference (s/call)
LingBot-VA12.99444.015.068
LDA-1B1.35028.410.291
WALA0.63823.690.070

WALA has the lowest measured training-step time, training memory usage, and inference-call latency among the three methods. These results complement the pretraining comparison and highlight WALA's efficiency in downstream training and inference.

Separating the method's contribution from additional pretraining. To separate the effects of training design and pretraining, we examined the controlled ablations in Table 3 of the original manuscript, with the policy backbone and robot data fixed. The action-only base policy achieves 54.2% success. Adding a latent-action-conditioned world model without LAM pretraining raises success to 67.8%, a gain of 13.6 percentage points. With LAM pretraining, distillation alone achieves 67.6% (+13.4 points), world-model supervision alone achieves 71.0% (+16.8 points), and their combination achieves 75.2% (+21.0 points). All gains are relative to the same action-only baseline. The improvement without LAM pretraining establishes a contribution from the training design itself. Pretraining further strengthens world-model supervision, and combining world-model supervision with distillation produces the largest gain.

To examine whether these benefits persist without the large-scale pretraining corpus, we compare WALA with UniVLA-style, villa-X-style, and VLA-JEPA-style, using the same Qwen3-VL-4B backbone, action head, 10% RoboCasa episode subset, and downstream training and evaluation settings. All methods use this same subset for both video pretraining and policy training. Each style follows the corresponding method's pretraining procedure and supervision design within this shared framework. For the Delta versus Absolute comparison, we additionally hold the LAM architecture, future-frame sampling, and optimization settings fixed, changing whether the model encodes and predicts deltas or absolute features.

MethodSuccess (%)
UniVLA-style46.4
villa-X-style45.8
VLA-JEPA-style43.1
WALA Absolute47.8
WALA Delta51.3

WALA Delta improves over Absolute by 3.5 percentage points, supporting the benefit of modeling visual changes. It also exceeds UniVLA-style, villa-X-style, and VLA-JEPA-style by 4.9, 5.5, and 8.2 points, respectively. Together, these comparisons and the original ablations show that WALA's representation and supervision design contributes to performance gains under matched data and policy architecture, including when the large-scale LAM pretraining corpus is not used.

AuthorsOpenReview ↗

Response to Reviewer eS8T (2/2)

W2: Stability of the Two-Stage Training Pipeline

Multi-stage training is used by LAPA, UniVLA, Moto, and Motus, all discussed in our paper. LAPA, UniVLA, and Moto learn latent action or motion representations from videos, pretrain policies to predict them, and then adapt the policies for robot control. Motus uses video pretraining, latent-action pretraining, and robot-specific fine-tuning. WALA connects these stages through a consistent distillation reference, direct robot-action supervision, and prediction-pathway adaptation. We assess both optimization robustness and cross-stage alignment below.

Alignment mechanism. As described in Section 3.3, the pretrained LAM encoder remains frozen during policy training, providing consistent distillation targets. Adapters map policy tokens into the formats required for encoder-target alignment and decoder conditioning. The decoder is jointly optimized with the policy to accommodate policy-generated latent actions and downstream observations. The action head directly reads these same latent-action tokens, so the representation shaped by distillation and prediction also receives robot-action supervision. The frozen encoder provides a stable reference as the policy and decoder adapt to downstream interactions.

Robustness to loss weighting. We tested sensitivity to loss weighting. We fixed the action prediction loss weight at 1 and tested latent-action distillation and world-model prediction weights drawn from {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1}. Training converged stably across the tested configurations. Stable optimization across this broad range of auxiliary-loss weights shows that convergence does not depend on a single finely tuned balance.

Prediction-based alignment on downstream data. We also tested whether prediction feedback helps policy-generated latents drive the pretrained world model on new interactions. Table 14 in the original manuscript, “Effect of Decoder Supervision and Fine-Tuning,” uses RoboDojo videos unseen during LAM pretraining. All settings retain action supervision and latent-action distillation. Adding prediction supervision through a frozen decoder reduces semantic and depth prediction errors by 35.6% and 50.2%, respectively, relative to no decoder supervision. With the decoder fixed, these improvements reflect adaptation of the policy and its decoder adapter through prediction feedback. Jointly fine-tuning the decoder yields further reductions of 11.8% and 4.1%, respectively. These results show how joint training supports cross-stage alignment. Distillation anchors policy representations to the pretrained latent space, while prediction supervision refines their ability to express downstream interaction outcomes. This experiment uses three independent training seeds.

W3: Performance Across Benchmarks

WALA aims to learn transferable interaction knowledge from videos without action annotations and improve robot control. We pretrain the LAM on action-free human and robot videos, then transfer its representations through distillation and future-prediction supervision alongside robot-action supervision. At inference, only the policy and action head are needed, without generating future video states.

This motivates our comparison with LingBot-VA. LingBot-VA benefits from large-scale action-labeled pretraining and explicitly predicts future video states in latent space, using inverse dynamics to generate actions conditioned on those predictions. WALA-Wan achieves 92.33% success on RoboTwin, compared with LingBot-VA's 92.24% (Table 1). With action-free LAM pretraining and no inference-time future-video generation, WALA-Wan matches this strong baseline with a slightly higher reported success rate.

To extend the comparison to RoboCasa, we trained LingBot-VA on the full training data, obtaining 56.8%, compared with WALA's 75.2%, a difference of 18.4 percentage points. The table combines these results with the measurements in W1 and Q1, using the same H100 platform at per-GPU batch size 1 and each method's own training and inference configuration.

MethodRoboCasa success (%)Training (s/step)Training memory (GB)Inference (s/call)
LingBot-VA56.812.99444.015.068
WALA75.20.63823.690.070

Compared with LingBot-VA, WALA uses 46.2% less training memory, with 20.4× faster training steps and 72.4× faster inference calls. Together, these results show competitive RoboTwin performance without inference-time future-video generation, higher RoboCasa success, and substantially lower measured training and inference costs.

We will include these additional experiments and discussions in the revised manuscript. We hope our responses address all of your concerns. Please let us know if you have further questions or would like clarification on any point. We would be happy to provide further details.

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer eS8T,

Thank you again for your thoughtful comments. We would appreciate your feedback on whether our responses and additional experiments have addressed your concerns about pretraining resources, training stability, and performance across benchmarks. We would be happy to clarify any remaining points or discuss further questions you may have.

Thank you for your time and continued consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer eS8T,

As the author discussion period draws to a close, we would be grateful to hear whether our response and revised manuscript have addressed your concerns. If so, would you be willing to reconsider your rating in light of the additional evidence? We would be happy to clarify any remaining points.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We have now revised the manuscript in response to your comments, with additions and substantive changes highlighted with a light green background in the revised PDF. The main updates are summarized below; detailed results and responses are provided in our earlier individual replies.

  1. Pretraining resources. We added comparisons of pretraining data and compute, including WALA’s estimated 4,000 H100-equivalent GPU-hours based on measured throughput. We clarified that WALA pretrains only its LAM without action annotations and reuses it across downstream policies (Appendix A.9, Table 12).

  2. Performance and computational efficiency. We added a RoboCasa comparison using the same training data and evaluation protocol, where WALA achieves 75.2% success compared with 56.8% for LingBot-VA. H100 measurements against LingBot-VA and LDA-1B show that WALA has the lowest training-step time, training memory usage, and inference latency, with a 20.4× speedup per training step and a 72.4× inference speedup over LingBot-VA. We also clarified that WALA-Wan closely matches LingBot-VA on RoboTwin without inference-time future-video generation (Section 4.2, Tables 1–2; Appendices A.3 and A.9, Table 13).

  3. Related-work discussion. We added VLA-JEPA to the related-work discussion and clarified differences in latent-action supervision and action generation. We also highlighted WALA’s use of action-free videos alongside action-labeled robot demonstrations during downstream policy co-training, beyond video pretraining (Section 2.2).

  4. Matched-data method comparisons. We added UniVLA-style, villa-X-style, and VLA-JEPA-style comparisons using the same backbone, action head, and downstream settings, with both video pretraining and policy training restricted to the same 10% RoboCasa episode subset. WALA achieves 51.3%, exceeding these variants by 4.9, 5.5, and 8.2 percentage points, respectively (Section 4.4, Table 4; Appendix A.8).

  5. Restricted-video evaluation. In the same controlled experiment, WALA retains a 33.2-point advantage over the 18.1% action-only baseline when both LAM pretraining and policy training use only the same 10% RoboCasa episode subset. This demonstrates gains without access to the remaining videos or the large-scale LAM pretraining corpus (Section 4.3, Table 4; Appendix A.8).

  6. Delta versus absolute modeling. We added a controlled comparison between learned delta and absolute LAMs. Delta modeling improves downstream success from 47.8% to 51.3%, supporting the benefit of modeling interaction-induced changes (Section 4.4, Table 4; Appendix A.8).

Table references in our earlier individual replies follow the original manuscript. In the revised manuscript, original Tables 4, 5, 6, 13, and 14 correspond to Tables 5, 6, 7, 16, and 17, respectively. The section and figure numbers cited in those replies remain unchanged.

We welcome further questions and discussion and would be happy to clarify any remaining concerns. If the revised evidence addresses your concerns, we would be grateful if you could consider a more favorable assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.