- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a full-stack system for language-steerable bimanual dexterous manipulation. Its main idea is to pre-train a VLA on large-scale egocentric human video and then post-train it on a moderate amount of real-robot data. There are three components:
- EgoSmith, a four-stage curation pipeline: heuristic pre-filtering; 4D hand/camera reconstruction built on HaWoR, with DROID-SLAM replaced by DPVO plus Any4D metric-depth scale recovery; five-level language labeling with Qwen3.5-VL; and episode/chunk/frame-level post-filtering. Applied to 12 datasets, it yields a 9.6K-hour corpus (2.29M episodes), about 84% of which comes from Egocentric-100K. On 512 episodes with ground truth, it reports lower camera and hand errors than HaWoR and an 8.3× speedup (Table 1).
- Robot Stack, a shared teleoperation/inference/intervention stack (gloves + Vive trackers, mink IK) with a relative motion mapping for jump-free human takeover. It was used to collect 187 h / 55K trajectories over 193 tasks and 8.3 h of DAgger corrections.
- EgoSteer, a Qwen3-VL-2B backbone with a ~300M DiT flow-matching action expert. The action space is "morphology-aligned": bimanual wrist SE(3) plus fingertip keypoints in the camera frame. Training-time RTC handles asynchronous execution, and a training-only ~70M "world-model expert" regresses DINOv3 features of the frame at the end of the action chunk, conditioned on the ground-truth action chunk and the camera motion.
The authors curate a 9,606-hour training corpus from 12 sources and adopt a unified representation of wrist poses and fingertip keypoints. After pre-training on human video, the model is adapted through fine-tuning on robot demonstrations and DAgger-based correction. The authors report that the final multi-task policy achieves an average success rate of about 75% across 45 tasks. Separately, the pre-trained model is adapted to each of five long-horizon tasks, reaching an average progress rate of about 79% with 100 demonstrations per task. The main contribution is a complete system that integrates data, model and real-robot control, together with its empirical validation. According to the paper, the code and model checkpoints are released.
Critical strengths
-
The work is highly open. According to the reproducibility statement, the code of the full stack and the pre-trained (EgoSteer-9.6K-PT) and final (EgoSteer-9.6K-DG) checkpoints are released. The 9.6K-hour curated human-video corpus and the 187-hour, 193-task teleoperation dataset are promised upon publication. This makes it realistic for a complete dexterous-hand VLA training framework to be adopted in practice. The system is also highly integrated, and many details receive careful engineering optimization. Overall, this is a highly complete piece of work.
-
The method is evaluated on two robot platforms and a wide range of tasks: The 45-task suite (15 trials per task, 675 real-robot trials in total) covers seven manipulation categories in randomized, cluttered scenes. It also includes compositional, unseen and mid-execution instruction-switching conditions, with 75% overall success. Few-shot adaptation is demonstrated on five long-horizon tasks of about one minute each, on RealMan and AgiBot G1. With 100 demonstrations, EgoSteer-9.6K-FS reaches about 79% average progress, versus about 29% for π0.5 (Fig. 7).
Critical weaknesses
-
The system combines egocentric pre-training, a flow-matching VLA, auxiliary future-feature prediction, RTC, and DAgger. While this integration has clear practical value, I find the methodological and conceptual advances less clearly established than the engineering contribution. My concern is not the use of existing components itself, nor the absence of exhaustive ablations, but the limited evidence connecting the principal design choices to the stronger explanations offered for the results. For example, Table 2c supports the utility of the world-model objective and data filtering, but the filtering ablation does not isolate the downstream benefit of improved motion reconstruction. Likewise, Table 5 demonstrates DAgger gains on four seen tasks, whereas Section 6.2 additionally claims that the resulting recovery capabilities generalize to novel tasks. These distinctions matter because the paper’s scientific contribution depends not only on the final system working, but also on what the experiments establish about why it works.
-
Limited characterization of fine-grained dexterity. The contact-rich tasks in Figure 5 and the long-horizon tasks in Figure 7 provide meaningful evidence of dexterous manipulation. However, the evaluation emphasizes task diversity and successful completion more than it characterizes the fine-grained dexterity these tasks require, such as coordinated finger movements, adjustments of finger–object contacts, and in-hand object manipulation. Moreover, the more challenging long-horizon tasks are evaluated through task-specific few-shot adaptation rather than with the generalist policy. Consequently, while the results demonstrate broad language-guided manipulation capabilities, the extent to which the multi-task policy handles interactions requiring fine-grained finger-level control remains insufficiently characterized.
Questions for authors
This is a very complete work and also demonstrates good results. I would like to ask the following questions:
-
DAgger and Generalization. Table 2a demonstrates the effectiveness of DAgger on four seen tasks. However, the paper further claims that the resulting failure-recovery capabilities generalize to novel tasks. Could the authors provide quantitative evidence supporting this claim, particularly the recovery performance on unseen or compositional tasks? If such results are unavailable, the corresponding claim should be qualified.
-
Steerability and Language Grounding. The paper reports a 75% average success rate across 45 tasks and demonstrates instruction-following through qualitative examples. However, task success alone does not distinguish language-grounding failures from physical execution failures. Could the authors provide a quantitative breakdown of these failure types, particularly cases where the policy executes an unintended but plausible task versus cases where it correctly interprets the instruction but fails during execution? If available, results from language-ablated evaluations (e.g., empty or shuffled instructions) would further strengthen the claim that the policy genuinely grounds language rather than relying on visual or task-specific priors.
-
Although Table 1 shows EgoSmith's improvements in motion reconstruction accuracy and processing efficiency, and Table 2c also verifies the positive effect of data filtering on model performance, it remains unclear to what extent the improvement in motion reconstruction accuracy improves downstream robot policy performance. Could the authors provide existing quantitative experimental results to illustrate the relationship between motion reconstruction quality and downstream robotic manipulation performance?
3 comments
Response to Reviewer 7dA4 (Part 2/2)
Characterization of fine-grained, finger-level dexterity
Summary: Characterizing the policy's fine-grained dexterity beyond task completion.
Response: Thank you for this question. We would like to clarify that many of our evaluation tasks inherently require delicate, contact-rich finger coordination, thereby directly characterizing the policy's fine-grained dexterity.
In the complex long-horizon tasks, box folding requires using the right thumb to stabilize the cardboard while coordinating the index and middle fingers to press the upper edge into the box. Similarly, sport stacking requires the ring finger and pinky finger of the right hand to press against the stack at appropriately timed moments, preventing the bottom cups from slipping out when lifting the stacked cups.
Likewise, the multi-task generalist policy executes diverse tasks that depend strictly on fine-grained finger coordination. For example, lifting a teapot requires hooking four fingers through the narrow handle, and placing a tiny toy chick onto a small eggshell demands a precise two-finger pinch grasp followed by delicate, steady placement onto a tiny target area.
These behaviors confirm that both our generalist policy and few-shot policies exhibit coordinated finger-level control. We warmly invite you to inspect the rollout videos on our project website (https://ego-steer.github.io/) to observe these fine-grained finger interactions.
Disentangling language grounding from physical execution in steerability
Summary: Disentangling language-grounding failures from physical execution failures, and verifying genuine instruction compliance through failure breakdowns or language-ablated evaluations.
Response: Thank you for this question. Across all failure cases, the majority occur when the model successfully follows the instruction and demonstrates the correct manipulation intent, but fails during execution due to inaccuracies in action prediction. In contrast, a smaller portion of failures stem from failing to follow the instruction correctly.
Regarding evaluations under empty or shuffled instructions, we consider this beyond the scope of this paper. In practical deployment, these instructions can be readily filtered out by an upstream AI agent to avoid taking action, whereas our policy focuses on accurately executing valid, feasible user commands.
Furthermore, extensive experiments in Section 6.1 demonstrate that EgoSteer genuinely grounds language instructions rather than memorizing spurious visual-action correlations. A prime example is illustrated in Figure 13a, where the tabletop features a paper ball, an open trash can, and a laptop. When commanded to put the paper ball onto the laptop, the policy faithfully places the paper ball onto the laptop instead of defaulting to discarding it into the trash can.
We sincerely appreciate your comprehensive review, which has helped us strengthen the paper. We hope our response and the additional results have satisfactorily addressed your concerns, and we remain fully available to discuss any further questions.
Dear Reviewer 7dA4,
We sincerely appreciate the time and effort you have devoted to reviewing our work. As the discussion phase draws to a close, we would like to respectfully follow up on our response and paper revision. Specifically, we have provided new real-robot evaluations in Section 6.2 showing that DAgger-trained failure-recovery and adjustment capabilities generalize to novel tasks, alongside further explanations regarding data filtering and motion reconstruction, as well as our policy’s fine-grained finger coordination and language grounding. We hope our response and the additional results have satisfactorily addressed your concerns, and we remain more than happy to provide further clarifications should you have any remaining questions. Thank you again for your thoughtful consideration.
Best regards,
The Authors
Response to Reviewer 7dA4 (Part 1/2)
We sincerely appreciate your thorough and insightful review. We are delighted that you recognize our work as highly open and highly complete, noting that many details receive careful engineering optimization and that our system is supported by extensive evaluations with compelling results. In what follows, we carefully address each of your comments and questions in detail.
Response: Thank you for raising this point. To clarify, the ablation in Table 2c was specifically designed to evaluate the role of data filtering, so it was not intended to measure the isolated effect of motion reconstruction accuracy. Regarding motion reconstruction, EgoSmith's superior reconstruction quality (Table 1) provides more physically plausible and kinematically consistent demonstrations, which naturally benefits downstream policy performance. Furthermore, establishing a quantitative relationship between reconstruction error and policy performance would require training separate robot policies across a spectrum of varying reconstruction qualities. Because our entire framework is built directly upon the high-quality reconstructions produced by EgoSmith, characterizing such an empirical curve remains difficult in practice.
Response: Thank you for this constructive comment. To substantiate our claim that DAgger-trained failure-recovery and adjustment capabilities generalize to novel tasks, we expand our evaluation by benchmarking EgoSteer-9.6K-DG against EgoSteer-9.6K-FT across four failure-prone seen tasks, two unseen tasks, and two compositional tasks, with 10 randomized trials per task:
As shown in the table above, DAgger refinement not only boosts performance on seen tasks, but also yields substantial improvements on novel tasks, raising unseen tasks from 40% to 70% on average and compositional tasks from 20% to 65% on average. This indicates that DAgger enables EgoSteer to internalize key human intervention patterns, specifically adjusting actions at critical manipulation steps and promptly recovering from failures, thereby generalizing these strategies to unseen tasks and boosting overall policy performance. We have updated Section 6.2, Table 2, and Table 5 in the revision to reflect these quantitative results, with all updates highlighted in blue.