acceptodds
Under review as a conference paper at ICLR 2027

EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning

13 views

Abstract

Egocentric human data offer a path to scaling robot learning beyond costly robot demonstrations, yet the embodiment gap makes raw human trajectories a poor supervisory target for control. Our key insight is that, although low-level actions are embodiment-specific, their underlying motion intent can capture task-relevant structure that transfers across humans and robots. We introduce EgoLAP, a VLA pre-training framework that jointly learns from human and robot trajectories through a shared language-based action chain-of-thought. EgoLAP expresses motion intent as structured, temporally abstracted language actions and pairs them with motion-level reasoning grounded in scene geometry, physics, and object affordances. Across extensive real-world and simulated experiments, EgoLAP transfers human experience to robot control more effectively than alternative action representations and reaches mean real-world task progress, a performance gain over alternative action representations. Motion-level reasoning also outperforms a composite reasoning format that combines subtask, object-box, and visual-trace reasoning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning from human video, reasoning, cross-embodiment transfer, robotic manipulation, action representations

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uDiwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce EgoLAP, a vision-language-action pre-training framework that learns from egocentric human and robot trajectories through shared language actions and motion-level reasoning. EgoLAP transfers human experience to robot control, achieving 80.1% mean real-world task progress on an unseen robot configuration and a 2.3× gain over alternative action representations.

Critical strengths
  1. Strong and consistent empirical gains. EgoLAP reaching 80.1% real-world task progress and 38.3% zero-shot simulation success. The controlled comparisons and ablations provide useful evidence for the effectiveness of language-based motion supervision.

  2. A practical approach to cross-embodiment learning. Language actions provide a common interface for human and robot trajectories without requiring a learned action tokenizer or text generation during control. The method is simple to deploy and demonstrates promising transfer to unseen tasks and robot configurations.

Critical weaknesses

see questions

Questions for authors

How much of the human-to-robot transfer benefit depends on the specific human datasets and their interaction with reasoning supervision?

The ablations show that human data improves performance and that motion-level reasoning further amplifies this gain. Can the authors clarify whether this interaction is consistent across different human-data subsets, or whether it is primarily driven by MECKA, the only human-data source with motion-level reasoning annotations? This would help assess the generality of the proposed co-training approach.

What exactly is unseen in the real-world zero-shot evaluation?

The real-world evaluation uses a customized bimanual YAM platform, while the training mixture includes demonstrations from the ABC YAM platform. Could the authors clarify which aspects of the robot embodiment, camera configuration, and control interface differ between training and evaluation, and whether any data collected on the evaluation hardware was used during pretraining? This would make the scope of the claimed zero-shot cross-embodiment transfer more precise.

2 comments

We thank the reviewer for the positive assessment and precise questions. We are glad the reviewer found the empirical gains strong and consistent, and the ablations useful. We also appreciate that the reviewer sees EgoLAP as a practical, simple-to-deploy approach to cross-embodiment learning, with no learned tokenizer and no text generation at control time.

Q1. Does the human-data × reasoning interaction depend on MECKA?

This is a fair question, and we acknowledge that our current experiments cannot fully separate the two factors. MECKA contributes 20% of each batch, out of 25% egocentric data (App. A.7). It is therefore both the only human source with reasoning labels and the majority of our human data. The existing results support the following:

  1. The language-action pathway transfers human data without any reasoning. With no reasoning, adding all human data (MECKA, Scale, Aria) raises sim success from 7.2% to 16.4%.
  2. The benefit of reasoning is not specific to MECKA. Robot-side reasoning alone (DROID, labeled by a separate pipeline) improves the robot-only policy from 7.2% to 18.4%.
  3. The interaction is real but likely carried by annotated human data. Human data adds +19.9 points with reasoning versus +9.2 points without (App. C.3). This is consistent with MECKA's motion-level rationales driving much of the amplification.

We agree that a per-subset study would make the generality claim more precise, e.g., MECKA only versus Scale + Aria only. We have revised the paper accordingly. It now states explicitly that all human-side reasoning comes from MECKA, and it scopes the human-data × reasoning interaction to this setting. We list a per-subset analysis as future work in the Limitations.

Q2. What exactly is unseen in the real-world zero-shot evaluation?

No data collected on the evaluation hardware or setup was used at any stage of training. The training mixture contains YAM data from the ABC platform and from MolmoAct2 (App. A.7). Our evaluation platform uses the same arm model but a different configuration from every training embodiment:

  • Arm placement: the two arms are mounted about 0.8 m apart, a spacing that differs from both the ABC and MolmoAct2 setups.
  • Cameras: a RealSense D435 base camera and two RealSense D405 wrist cameras. The base-camera placement and viewpoint differ from those in the training data.
  • Scenes and objects: our lab, with objects and layouts not in the training data.
  • Control interface: the arms are driven by our own custom controller built on the I2RT package, and no training data were collected through it. The policy outputs the same Cartesian delta end-effector actions as in training (Sec. 3.2), and our controller executes them on the arms.

The claim is therefore zero-shot transfer to a new configuration (arm placement, camera setup, scenes) of a seen arm type, not to a new arm. We have revised Sec. 4.1 to state this scope precisely. The shift is not trivial. Informally, during development we observed that aligning our setup more closely with the MolmoAct2 configuration improved performance. Under our shifted configuration, the baselines degraded considerably more than EgoLAP: EgoLAP reaches 56.0% full-task success, versus at most 18.0% for any other policy (Fig. 4(b)).

We thank the reviewer again and welcome further questions.

Thank you for the clarifications and revisions. Explicitly limiting the human-data × reasoning interaction to the current data mixture addresses my concern about overgeneralization, although per-subset validation remains valuable future work. The clarification that zero-shot evaluation uses a new configuration of a seen arm type, with no training data from the evaluation setup, also resolves my second concern.

I remain positive about the paper and maintain my Weak Acceptance rating.

Minor correction: in Appendix C.4, “19.9 versus 8.4 points” should read “19.9 versus 9.2 points,” consistent with Table 9.

Reviewer_9FQSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoLAP, a VLA pre-training framework that effectively leverages both human and robot data. The key insight is simple yet powerful. First, language actions provide a shared representation across human and robot data, enabling the model to learn more effectively through joint training. Second, language actions serve as an effective auxiliary supervision signal, bridging the gap between the VLM backbone and the action backbone. By representing actions as natural language, EgoLAP aligns action modeling with the text distribution that VLMs are inherently trained to produce, leading to superior performance compared to alternative action tokenization methods, such as FAST and OAT.

Critical strengths
  1. A key challenge in human-robot co-training is establishing a shared action representation across domains. EgoLAP addresses this by modeling both human and robot actions in natural language , providing a simple yet effective interface that facilitates knowledge transfer and joint training across human and robot data.
  2. Language actions serve as an effective auxiliary supervision signal, bridging the gap between the VLM backbone and the action backbone. By representing actions as text, EgoLAP aligns action prediction with the distribution that VLMs are naturally trained to model, enabling more effective learning and outperforming alternative action tokenization methods such as FAST and OAT.
Critical weaknesses
  1. I am mostly curious about the evaluation protocol. The paper reports task progress on real-world YAM tasks, but it would be better to clarify how task progress is calculated for each task and stage. The paper would also benefit from explaining why success rates are not reported for real-robot tasks. Most of the real-robot tasks do not appear particularly challenging or contact-rich, as many of them are simply pick-and-place tasks.

  2. Although language actions are effective, this idea was originally proposed in the LAP paper [1]. Using language actions for human-robot co-training seems like a relatively marginal extension of the original idea.

  3. The paper mostly studies VLA baselines, which may not be sufficient. It would be interesting to investigate whether language actions are truly an effective shared learning interface between human and robot data by comparing against methods such as OpenWAM [2]. Video could also serve as a natural shared interface for human and robot data, so including such a baseline would strengthen the evaluation.

[1] LAP: https://arxiv.org/abs/2602.10556

[2] OpenWAM: https://arxiv.org/pdf/2609.07398

Questions for authors
  1. Why does the paper report task progress instead of success rates for real-robot tasks? How is task progress calculated for each task?

  2. When language-action reasoning is enabled at test time, do the generated language actions actually align with the executed actions?

  3. Could the authors include OpenWAM as an additional baseline for a more detailed comparison?

11 comments

We thank the reviewer for the review and for recognizing that language actions are a simple and effective shared interface that outperforms FAST and OAT.

W1 / Q1. Evaluation protocol, success rates, and task difficulty.

Progress definition. Task progress is defined in App. B. Each task ii is split into KiK_i ordered stages, a trial is scored pi=1Ki∑ksikp_i = \frac{1}{K_i}\sum_k s_{ik} with sik∈{0,1}s_{ik}\in\{0,1\}, and we report the mean over trials. For example, Fold Pants has five stages: grasp one end, grasp both ends, lift the ends, produce a partial overlap, and complete the fold. Progress separates policies that fail at the first stage from those that fail at the last, which matters with 10–12 trials per task.

Full-task success rates (%). These come from the same trials as Fig. 4(b).

TaskABC-VLAEgoOAT (no reas.)EgoOATEgoFAST (no reas.)EgoFASTEgoLAP (no reas.)EgoLAP
Pick Place (Simple)302030603080100
Pick Place (Complex)000008.350
Handover & Place000001050
Wipe Object00000030
Fold Pants300030102050
Mean success12.04.06.018.08.023.756.0
Mean progress25.618.025.635.226.144.580.1

Task difficulty. Only one of the five tasks is plain pick-and-place.

  • Fold Pants: deformable manipulation, with coordinated bimanual grasps on a garment whose shape changes.
  • Wipe Object: sustained contact, where the policy must regulate pressure while covering a surface.
  • Handover & Place: requires bimanual coordination and avoiding contact between the two arms.
  • Pick Place (Complex): grounding the instructed object and container in clutter, possibly with a handover.

All of these run zero-shot on a robot configuration not seen in training. Under full success, every non-EgoLAP policy scores 0% on the three hardest tasks, while EgoLAP reaches 30–50%.

W2. "Language actions for human–robot co-training is a marginal extension of LAP."

We respectfully think this misses the main contribution. Language actions themselves come from LAP, and we credit this (Secs. 2, 3.3). EgoLAP's contributions are different:

  1. Motion-level reasoning, a new supervision target that explains why a one-second motion is appropriate (contact, geometry, affordance). This is different from the high-level plans, subtasks and traces in prior reasoning work (ECoT, ECoT-Lite, RAD). It targets motion-level generalization across objects and embodiments, not long-horizon planning.
  2. The reasoning–action-representation interaction. Under an identical recipe, motion-level reasoning adds +35.6 real-world points to language actions but reduces FAST from 35.2% to 26.1%. Human data adds +19.9 sim points to language actions versus +7.3 to FAST (App. C.4). The ECoT-style composite reasoning hurts (16.4% → 5.3%).

Language actions alone (no reasoning) reach 44.5% real-world progress. The EgoLAP contributions nearly double this to 80.1%, and raise full success from 23.7% to 56.0%. The gain therefore comes primarily from what is new in EgoLAP, not from the LAP representation.

W3 / Q3. Comparison with a video-based interface (OpenWAM).

Following the suggestion, we evaluated the publicly released OpenWAM checkpoint (OpenWAM/OpenWAM-Alpha-Pretrain-Foundation-Model, the variant we found most likely to succeed) in the same zero-shot settings: in simulation, it achieves 0/1400 successful episodes on the same simulation benchmark we evaluate on. It did not pick up an object in any episode, and qualititively the action is not task-relavent.

We read this result with care. OpenWAM's paper reports results only after per-benchmark fine-tuning. So the zero-shot cross-embodiment setting is outside what it was designed for. Still, the result is informative for the reviewer's question. Within this zero-shot transfer regime, a shared video interface pretrained at scale did not transfer to our unseen robot configuration, whereas a shared action-level language interface did. We also note that OpenWAM was posted on arXiv on Sep 7, 2026, shortly before the submission deadline. We have added the result and a discussion of world–action models to the paper.

Q2. Do generated language actions align with the executed actions?

Yes. On held-out frames, we decode the language action from the backbone and compare its translation direction with that of the action chunk executed by the action expert. We report per-axis sign agreement and cosine similarity (left arm):

No reasoningReasoning
Decoded language action vs. executed actionsign 0.80, cos 0.81sign 0.85, cos 0.86
Decoded vs. ground-truth language actionsign 0.83, cos 0.86sign 0.82, cos 0.77

The language action the backbone generates closely agrees with the motion the expert executes, and the agreement is higher with motion-level reasoning. This is notable because the action expert never attends to text tokens (Fig. 3). The alignment therefore emerges from the shared backbone representation rather than from conditioning on decoded text, so control requires no test-time text generation.

We hope the new success-rate results, the OpenWAM evaluation and the clarification of our contributions address the reviewer's concerns. We would be grateful if the reviewer would consider revisiting the score, and we are happy to answer any further questions.

Thank you for the detailed rebuttal and additional experiments. I appreciate the further analysis. However, my main concern remains the methodological novelty. In my view, combining reasoning supervision with a language-based action representation is not fundamentally new. While the paper frames its contributions as “motion-level reasoning” and “the reasoning–action-representation interaction,” I see these primarily as refinements and empirical observations built upon existing techniques, rather than sufficiently novel methodological contributions. The additional experiments strengthen the empirical findings but do not fully address my novelty concern. Therefore, I will maintain my original score.

Reviewer_japTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of transferring egocentric human demonstrations to bimanual robot control under embodiment discrepancies. The authors propose EgoLAP, which leverages deterministic language actions and motion-level textual rationales as shared supervision signals to jointly train on human and robot data. The key insight is that, although the specific actions differ across embodiments, the underlying motion intents can be shared. The effectiveness of the proposed approach is validated through experiments on the ABC (Scalable Behavior Cloning) simulation benchmark and real-world robotic tasks, demonstrating improvements in performance.

Critical strengths
  1. A promising supervision design for cross-embodiment learning.

The most significant strength of this work lies in its supervision design for Vision-Language-Action (VLA) models. By introducing language-based action representations that capture shared motion intents across embodiments, the proposed approach provides a promising direction for leveraging egocentric human demonstrations to improve cross-embodiment generalization. This is particularly relevant to the broader challenge of effectively incorporating human demonstration data into robot policy learning.

  1. Encouraging experimental results with relatively comprehensive validation.

The experimental results suggest that language-based action supervision facilitates effective learning from human demonstrations. The paper also provides a relatively extensive set of ablation studies, with the real-world deployment results being particularly valuable. These experiments strengthen the empirical support for the proposed approach and demonstrate its potential practical applicability.

Critical weaknesses
  1. Potential limitations in data scalability and annotation requirements.

I do not identify any major conceptual or methodological flaws in the proposed approach. However, one concern is whether the current framework can readily scale to large collections of in-the-wild egocentric human videos, particularly those available on the Internet.

Specifically, the proposed action supervision requires access to 3D hand positions, orientations, and hand open/close states over temporal windows. Such information is generally not directly available in ordinary Internet videos, and obtaining reliable annotations retrospectively may be challenging and costly. These requirements could limit the scalability of the proposed framework and its applicability to broader sources of human demonstration data.

While this concern does not necessarily undermine the validity of the current experimental findings, it raises questions about the extent to which the proposed approach can benefit from large-scale, readily available human video data in future applications.

Questions for authors

Q1. Generalization to more diverse robot configurations.

The real-world evaluation primarily focuses on a customized bimanual YAM robot. Although its arm spacing and camera configuration differ from those of the robots represented in the training data, the overall embodiment and system configuration still appear relatively similar.

Have the authors considered evaluating the proposed approach on robot configurations with more substantial embodiment differences? Additionally, would aligning the robot's arm spacing and camera configuration more closely with those in the training data lead to better performance? It would be helpful to know whether any further adjustments or ablation studies have been conducted to investigate the effects of arm spacing and camera configuration on cross-embodiment generalization.

Q2. Clarification regarding training steps.

The main text mentions that the proposed model is trained for 30K steps, whereas other models are trained for 40K steps. However, Appendix A.6 states that all models are trained for 40K steps.

My understanding is that the 40K-step setting in the appendix may have been used to ensure a fair comparison, while the 30K-step result in the main text may be intended to highlight the faster convergence of the proposed method. Could the authors clarify this discrepancy? A brief explanation in the appendix would help avoid potential confusion regarding the experimental setup and comparison fairness.

Q3. Reporting of evaluation metrics across simulation and real-world tasks.

The paper reports that EgoLAP achieves an average progress of 80.1% on real-world tasks, representing a 2.3× improvement over alternative action representations, and a success rate of 38.3% on simulated tasks.

Could the authors also report the success rate on real-world tasks and the average progress on simulated tasks, if available? Providing both metrics for the two evaluation settings would facilitate a more consistent comparison and offer a more complete understanding of the model's performance.

Q4. Suggestion for future investigation: Robustness to imperfect human motion annotations.

Given the potential difficulty of obtaining accurate motion annotations from in-the-wild human videos, an interesting direction would be to investigate the robustness of the proposed method to annotation noise and missing information.

For example, one could fix the same set of human demonstration videos and progressively introduce perturbations such as hand-pose noise, missing trajectory segments, scale estimation errors, or camera-pose errors. Evaluating whether EgoLAP maintains its advantage over baseline methods under these conditions could provide valuable insights into its potential for scaling to less structured human video datasets.

I recognize that conducting additional experiments may not be feasible within the rebuttal period. Therefore, this is intended as a suggestion for discussion and future investigation, rather than a requirement or a factor affecting my current assessment.

3 comments

We thank the reviewer for the encouraging review. We are glad the reviewer found that language-based action representations capturing shared motion intent offer a promising supervision design for cross-embodiment learning and for incorporating human demonstrations. We also appreciate that the reviewer found the ablations relatively comprehensive, considered the real-world deployment results particularly valuable, and identified no major conceptual or methodological flaws.

W1. Scalability to in-the-wild / Internet egocentric video.

We agree that this is an important consideration, and we have added it explicitly to the Limitations. The required signals are becoming increasingly accessible. For example, HaMeR (Pavlakos et al., CVPR 2024) reconstructs 3D hands from single RGB images with about 6 mm PA-MPJPE on FreiHAND. HaWoR (Zhang et al., CVPR 2025) recovers world-space hand trajectories directly from monocular egocentric video.

More importantly, language actions are inherently tolerant to annotation noise. Each language action summarizes the net motion over a one-second window, quantized to 1 cm in translation and 5° in rotation (Sec. 3.3). Intra-chunk details are left out, so jitter and small pose errors that estimated hand tracks typically contain are largely absorbed. Learned tokenizers such as FAST and OAT instead encode the full high-frequency action chunk, so noise in the action data propagates directly into their training targets. This is consistent with our observation that the advantage of language actions over FAST and OAT is largest when co-training with human data (16.4% vs. 1.6% and 2.7% sim success without reasoning, Sec. 4.2). We therefore expect language actions to be a favorable representation for noisier, estimated hand trajectories from in-the-wild video. We also agree that learning from raw video alone, e.g., with latent or pseudo-labeled actions, is a promising complementary direction for future work.

Q1. More diverse robot configurations; effect of arm spacing and camera.

Thank you for the question. We have not run a controlled ablation over arm spacing or camera placement. Informally, during development we observed that aligning our setup more closely with the MolmoAct2 YAM configuration in the training data improved performance. Under our shifted evaluation configuration, the baselines degraded considerably more than EgoLAP (Fig. 4(b)). Beyond the real YAM, Tab. 1 shows that EgoLAP also adapts well after fine-tuning to a different embodiment, the Aloha-AgileX robot in RoboTwin 2.0. We agree that a systematic study of configuration shifts, and evaluation on embodiments with substantially different kinematics, would be valuable.

Q2. 30k vs. 40k training steps.

Thank you for catching this inconsistency. All models, including EgoLAP, were trained for 40k steps under identical settings (App. A.6). For each model, we select the checkpoint with the lowest offline validation loss. For EgoLAP, the 30k checkpoint has the best validation loss, reflecting its faster convergence; for the baselines, the best checkpoints are around 40k. The training budget is therefore matched across all methods. We have corrected the wording in Sec. 4.1 and App. A.8 to state this consistently.

Q3. Real-world success rate and simulation progress.

Real world, full-task success rate (%), same trials as Fig. 4(b):

TaskABC-VLAEgoOAT (no reas.)EgoOATEgoFAST (no reas.)EgoFASTEgoLAP (no reas.)EgoLAP
Pick Place (Simple)302030603080100
Pick Place (Complex)000008.350
Handover & Place000001050
Wipe Object00000030
Fold Pants300030102050
Mean success12.04.06.018.08.023.756.0

Simulation, mean task progress (%), same 1,400 episodes as Fig. 4(a):

GroupABC-VLAEgoOAT (no reas.)EgoOATEgoFAST (no reas.)EgoFASTEgoLAP (no reas.)EgoLAP
Pick14.323.621.016.323.432.346.3
Place40.010.24.312.823.347.256.1
Color27.58.912.712.727.426.260.9
Fold26.442.466.00.560.247.477.8
Overall31.715.213.412.626.641.056.2

The conclusions hold under both metrics in both settings. EgoLAP is best overall and in every group. Motion-level reasoning helps language actions the most: +15.2 points in sim progress and +32.3 points in real success. Both tables have been added to the appendix.

Q4. Robustness to imperfect human motion annotations.

We think this is an excellent suggestion. A controlled study that perturbs the same human data would give exactly the evidence needed to scale to less structured video: hand-pose noise, missing segments, scale errors and camera-pose errors. It would also directly test our hypothesis from W1. Because language actions keep only the quantized net motion of each window, we expect them to degrade more gracefully than learned tokenizers such as FAST and OAT, which encode the full noisy trajectory. Measuring whether EgoLAP's advantage holds, or grows, under increasing perturbation is a natural next step. We plan to study it in future work.

We appreciate the reviewer's constructive feedback.

Thank you for your clarification and detailed response. I appreciate your efforts in addressing my comments.

Reviewer_3ZjeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoLAP, which trains a VLA model using human recordings with tracked hand poses and robot demonstrations. During training, the model learns from language actions and motion explanations as COT, and at test time predicts continuous robot actions instead, without generating text COT. Experiments on 14 simulation tasks and 5 real-world tasks show clear improvements over the tested alternatives.

Critical strengths
  1. Substantial improvements in robot control. Adding motion explanations raises simulation success from 16.4% to 38.3% and average real-world task progress from 44.5% to 80.1%. These gains show that the additional language supervision improves continuous robot control, even without generating CoT text at test time.

  2. Ablations support the effectiveness of human data. The simulation experiments compare robot-only and human–robot training, each w and w/o motion explanations. A separate ablation also shows that training the action expert on human trajectories improves the policy performance.

  3. Useful ablations on supervision design. In the tested setting, motion explanations help most when paired with language actions, compared against adding plans +object boxes+visual traces. This provides useful empirical findings for robot action CoT design.

Critical weaknesses
  1. Limited novelty. Most of the components and proposed CoT types have been explored before. E.g., LAP (Zha et al., 2026) represents actions in language, RAD (Clark et al., 2025) uses movement explanations to learn from human videos, and ECoT-Lite (Chen et al., 2025b) uses reasoning supervision without generating reasoning at test time. EgoLAP’s main advance is the specific combination for human–robot training and the accompanying empirical findings through ablations, so I consider the novelty modest.

  2. The real-world contribution of human data is not directly established. The simulation ablations clearly support the benefit of human data, but the corresponding real-world comparison against robot-only training with motion explanations seems not reported.

  3. Limited evidence on scaling (minor). The paper compares training with and without human data, but does not ablate the amounts of data. It remains unclear whether increasing the data further would continue to improve performance or whether the gains would level off.

Questions for authors
  1. Could you report complete-task success rates for the five real-world tasks, alongside mean task progress?

  2. Which exact checkpoint was used to initialize the PaliGemma backbone?

  3. For each variant in Figure 7, what are the simulation success rates before and after GRPO?

3 comments

We thank the reviewer for the careful review and for recognizing the substantial control improvements, the ablations supporting human data, and the usefulness of our supervision-design findings.

W1. Limited novelty (LAP, RAD, ECoT-Lite).

We respectfully disagree that the novelty is modest. EgoLAP makes two contributions that, to our knowledge, prior work has not studied: a new level of reasoning, and the interaction between reasoning and the action representation.

(1) Motion-level reasoning is a new level of reasoning, not a variant of existing ones. Prior embodied reasoning, including ECoT, ECoT-Lite and RAD, reasons about what to do next at the task level: plans, subtasks, object locations, or a coarse movement direction. It is shown to help long-horizon and task-level generalization. Motion-level reasoning instead explains why this specific motion, at this moment, produces the intended physical effect. It grounds the next one-second motion in contact, geometry and object affordances. Its target is motion-level generalization: performing the same skill on new objects (folding one garment to fold another) and on new embodiments (from a human hand to a robot gripper, Fig. 2). To our knowledge, this form of reasoning has not been studied, let alone at scale on human and robot data.

ECoT / ECoT-LiteRADEgoLAP (motion-level)
Reasons aboutplan, subtask, boxes, gripper traceECoT chain + direction-only move primitive ("move left")physical rationale of the next motion: contact, geometry, affordance → effect
Targetstask-level / long-horizontask-level, from human videomotion-level, across objects and embodiments
Human motion used as–action-free (reasoning only)action target (language + continuous)
Text at inferenceyes / noyes (7B autoregressive)no

The two levels are empirically distinct. Under an identical recipe, an ECoT-style composite (subtask + boxes + visual trace) lowers sim success from 16.4% to 5.3%, and placing it before motion-level reasoning caps the policy at 14.4%. Motion-level reasoning alone reaches 38.3% (Sec. 4.4).

(2) How actions interact with text-space reasoning is an important, under-explored question. As foundation models (e.g., Astra) increasingly reason in text before acting, a central open question is how actions should be represented so that text-space reasoning actually improves them. This matters for robot policies and for agentic systems more broadly. Prior reasoning VLAs keep a standard action representation (binned tokens, FAST) and implicitly assume that reasoning helps regardless. We show that this assumption fails. Reasoning pays off only when the action target lives in the same language space, so that reasoning and action form one continuous chain of thought:

Action targetΔ from motion-level reasoning (real, progress)Δ (sim, success)
Language actions (EgoLAP)+35.6+21.9
FAST−9.1 (35.2 → 26.1)+7.1
OAT+7.6+0.7

The same reasoning supervision nearly doubles real-world progress with language actions but hurts FAST. The interaction extends to human data: with reasoning, adding human data gains +19.9 points for language actions versus +7.3 for FAST (App. C.4). We believe identifying this interaction, and an action representation that unlocks reasoning, is a key contribution in its own right. We will revise Sec. 2 to make both points explicit.

W2. Real-world contribution of human data.

We acknowledge that we did not run a real-world robot-only + reasoning variant. The submission provides three pieces of evidence:

  1. Simulation, matched recipe (App. C.3). With reasoning, robot-only reaches 18.4% and robot + human reaches 38.3%. Without reasoning, the numbers are 7.2% and 16.4%.
  2. Wipe Object (real). Wiping appears in our egocentric human data but in none of the robot pre-training corpora (App. B), so any success must come from transfer out of human data. EgoLAP reaches 70% progress and 30% full success, while every other variant has 0% full success.
  3. Representation analysis (Sec. 4.6). Reasoning supervision organizes human and robot frames by shared skill. Cross-embodiment R@1 rises from 51% to 76% (robot→human) and from 42% to 63% (human→robot).

W3. Data scaling.

We agree that scaling behavior is important. Our current evidence is the step from robot-only to robot + human data at fixed robot data: 18.4% → 38.3% with reasoning and 7.2% → 16.4% without. The larger gain under reasoning suggests that the benefit of human data grows with richer supervision. A systematic sweep over human-data hours is a clear next step; we have added it to the Limitations and leave it to future work.

Q1. Full-task success rates on the real robot.

Below are full-task success rates (all stages completed) from the same trials as Fig. 4(b), alongside mean progress. Pick Place (Complex) has 12 trials; the other tasks have 10.

TaskABC-VLAEgoOAT (no reas.)EgoOATEgoFAST (no reas.)EgoFASTEgoLAP (no reas.)EgoLAP
Pick Place (Simple)302030603080100
Pick Place (Complex)000008.350
Handover & Place000001050
Wipe Object00000030
Fold Pants300030102050
Mean success (%)12.04.06.018.08.023.756.0
Mean progress (%)25.618.025.635.226.144.580.1

The ranking is unchanged under full success, and the gap widens. EgoLAP is the only policy with non-zero success on all five tasks, and every non-EgoLAP policy has 0% success on the three hardest tasks. We will add this table to the appendix.

Q2. PaliGemma checkpoint.

The backbone is initialized from Google's pretrained paligemma-3b-pt-224 checkpoint (https://huggingface.co/google/paligemma-3b-pt-224). This is the pretrained ("pt") version, not the instruction-mixed ("mix") one. The full VLM has about 3B parameters. We have added this to App. A.1.

Q3. Simulation success before and after GRPO (Fig. 7).

VariantBefore GRPORelative gainAfter GRPO
Reasoning pre-training, sampling language actions only38.3%+19.4%≈45.7%
Reasoning pre-training, sampling reasoning + language actions38.3%+12.7%≈43.2%
Without reasoning pre-training16.4%+14.2%≈18.7%

The gains in Fig. 7 are relative (e.g., 38.3% × 1.194 ≈ 45.7%). We have made this explicit in the caption and added the table to App. A.9.

We hope these responses address the reviewer's concerns and are happy to discuss any remaining questions!

I thank the authors for clarifying the details and providing the real-world success rates. On the downside I still see overlap with prior work. I would maintain my weak accept recommendation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of contributions, reviewer feedback, and rebuttal

We thank the AC and all reviewers for their time and constructive feedback. Below we summarize the paper's contributions and how the rebuttal addresses the main questions.

Contributions.

  1. Motion-level reasoning. This is a new level of embodied reasoning. It explains why the next one-second motion produces its intended physical effect, grounding it in contact, geometry and affordance. Prior reasoning (ECoT, ECoT-Lite, RAD) targets task-level plans. Motion-level reasoning instead targets motion-level generalization across objects and embodiments, including from human hands to robots.
  2. Bridging text-space reasoning with low-level actions. As robotics moves toward agentic systems that reason in language before acting, this has become a central yet under-explored question, and it is exactly what we study. We show that reasoning pays off only when the action representation lives in the same language space. Under identical training, motion-level reasoning adds +35.6 real-world points to language actions but hurts FAST (35.2 → 26.1).
  3. EgoLAP. A VLA pre-training framework that co-trains on egocentric human and robot data through this language-based action chain-of-thought. It requires no text generation at inference.

Strong results. EgoLAP reaches 80.1% real-world task progress and 56.0% full-task success (≤18.0% for any baseline), and 38.3% zero-shot success in simulation. Reviewers described these as "substantial improvements in robot control" (3Zje) and "strong and consistent empirical gains" (uDiw), and the key insight as "simple yet powerful" (9FQS).

Controlled experiments and ablations. All variants share the same backbone, data, training budget and evaluation, and differ only in the factor under study. The ablations cover action representation × reasoning, human data × reasoning, motion-level versus ECoT-style reasoning, and human motion for the action expert. Reviewers found that "the controlled comparisons and ablations provide useful evidence for the effectiveness of language-based motion supervision" (uDiw) and offer "useful empirical findings for robot action CoT design" (3Zje).

Comprehensive evaluation. The evaluation spans five zero-shot real-world tasks on an unseen bimanual configuration (folding, wiping, handover, cluttered pick-place), 14 simulation tasks (1,400 episodes), RoboTwin 2.0 fine-tuning, and GRPO post-training. Reviewer japT noted "relatively comprehensive validation", with "the real-world deployment results being particularly valuable".

Common questions addressed in the rebuttal.

  • Novelty vs. LAP / RAD / ECoT-Lite (3Zje, 9FQS): EgoLAP is not an extension of any of these works. LAP contributes only the language-action format. Contributions 1 and 2 above appear in none of LAP, RAD or ECoT-Lite. Those works reason at the task level, treat human video as action-free (RAD), or pair reasoning with standard action tokens. We added a side-by-side comparison (response to 3Zje) on reasoning level, use of human motion, and test-time text generation. Empirically, these new components carry most of the gain. Language actions alone reach 44.5% progress and 23.7% success, and EgoLAP reaches 80.1% and 56.0%. An ECoT-style reasoning composite instead hurts (16.4% → 5.3% in sim).
  • Real-world success rates (3Zje, japT, 9FQS): we added full-task success rates for all variants. The ranking is unchanged, and the gap widens; all baselines score 0% on the three hardest tasks.
  • Video / world-action-model baseline (9FQS): the released OpenWAM checkpoint scores 0% zero-shot in both simulation (0/304 episodes) and the real world.
  • Language-action / action alignment (9FQS): decoded language actions agree with the executed actions (sign 0.85, cosine 0.86 with reasoning), even though the action expert never attends to text.
  • Human-data contribution and zero-shot scope (3Zje, uDiw, japT): we clarified the evidence in sim, on the human-only Wipe task, and in representation retrieval. We also stated precisely what is unseen on the evaluation robot.
  • Other clarifications: the matched 40k-step training budget, the PaliGemma checkpoint, GRPO numbers, simulation progress, and robustness to estimated hand poses.

We have revised the paper accordingly and thank the reviewers again for helping us improve it.

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.