- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into dexterous robot trajectories and synchronized robot observation videos. The method first reconstructs hand-object interaction using hand-anchored initialization, contact-guided multi-start pose refinement, and local contact refinement. It then transfers the reconstructed motion to a dexterous robot through keypoint-based retargeting and reinforcement learning. Successful simulated trajectories are jointly rendered with the robot hands and manipulated objects to produce synchronized visual observations, and the same interaction can be transferred to compatible target backgrounds through a rigid task-frame transformation. The paper evaluates EgoMatrix on TACO, OakInk2, and a self-collected dataset. It reports an overall simulated trajectory-transfer success rate of 54.84% (34/62), compared with 14.52% for Do As I Do and 12.90% for DexImit. It also reports competitive hand-object reconstruction results, ablations of the two refinement stages, qualitative visual synthesis comparisons, feature-space alignment with real-robot observations, and representative real-robot trajectory replays.
Critical strengths
-
EgoMatrix addresses an important and challenging data-generation problem with a coherent end-to-end pipeline. The method connects monocular hand-object reconstruction, dexterous retargeting, physics-based policy learning, and synchronized robot observation synthesis. The use of contact-aware refinement is well motivated because small hand-object gaps or penetrations can significantly affect downstream manipulation.
-
The method achieves strong reported improvements in simulated trajectory transfer and reconstruction quality. EgoMatrix reaches 54.84% overall transfer success on the evaluated demonstrations, compared with 14.52% for Do As I Do and 12.90% for DexImit. On the public reconstruction benchmarks, it obtains the best result on most of the reported metrics, including object geometry and trajectory-related measures.
-
The ablations support the role of the proposed refinement stages. The cumulative ablation shows that multi-start pose refinement and local contact refinement substantially improve end-to-end transfer. For example, success on TACO increases from 5.41% after hand-anchored initialization to 27.03% after multi-start refinement and 48.65% after local contact refinement. This provides useful evidence that interaction geometry is important for the downstream simulation transfer.
Critical weaknesses
-
The paper does not directly validate whether the synthesized demonstrations improve downstream policy learning. Although EgoMatrix is presented as a data engine for robot and VLA training, the paper does not train a downstream policy using the generated demonstrations and measure improvement on real or simulated tasks. The current results show that EgoMatrix can transfer selected trajectories and render observations, but they do not yet demonstrate that the generated dataset provides the claimed learning benefit.
-
The real-robot evaluation is qualitative and limited in scope. The paper shows representative trajectory replays on real robot platforms, but does not report quantitative success rates, failure modes, repeated trials, or coverage across the evaluated tasks. Consequently, the relationship between the reported simulation success and actual hardware reliability remains unclear.
-
The end-to-end evaluation uses a relatively small and heterogeneous set of demonstrations. The overall evaluation contains 62 demonstrations, including 37 from TACO, 14 from OakInk2, and 11 self-collected demonstrations with one example per task. Some individual task groups contain only a small number of demonstrations, which limits the strength of conclusions about generalization. The self-collected rollouts are also manually judged, making them less directly comparable to the public benchmark results.
-
The visual-fidelity evaluation does not fully establish physical faithfulness. The source-background comparison is mainly qualitative, while the target-background comparison measures feature-space distance over 52 frames from one manipulation sequence. These measurements provide useful evidence of visual similarity, but they do not directly evaluate contact correctness, temporal consistency, or the usefulness of the synthesized videos for policy training.
-
Several reconstruction metrics focus on relative rather than absolute physical accuracy. The object geometry evaluation uses Sim(3) alignment, and several other metrics use root or trajectory alignment. These protocols are appropriate for evaluating relative hand-object interaction, which is the stated design goal, but they can remove some scale and pose discrepancies. The results therefore support improved interaction consistency more directly than they support fully metric-accurate scene reconstruction.
Questions for authors
-
Can the authors train a downstream manipulation policy or VLA using EgoMatrix-generated data and compare it against training with real demonstrations, existing synthetic data, or no additional data?
-
Can the authors provide quantitative real-robot replay results with repeated trials, task-level success rates, and representative failure cases?
-
How sensitive are the reported 54.84% and 48.65% success rates to the selected task subsets and the small number of demonstrations per task? Can the authors report confidence intervals or results under alternative aggregation protocols?
-
Can the authors clarify the success-accounting protocol for multi-hand demonstrations, particularly the partial-credit treatment for DexImit and whether equivalent task-level criteria are applied to all methods?
-
Can the authors provide a more direct evaluation of synthesized observation fidelity, such as contact-level annotation, temporal consistency, human preference evaluation, or downstream policy performance?
7 comments
W3/Q3 — Evaluation Coverage and Statistical Robustness
We thank the reviewer for raising the concern about evaluation scale, task selection, and aggregation.
W3. “Some individual task groups contain only a small number of demonstrations, which limits the strength of conclusions about generalization.”
Evaluation coverage. For each selected TACO task group, we evaluate all available demonstrations, rather than choosing individual favorable examples. For instance, all seven videos labeled “brush, brush, bowl” are included. These sequences share the same semantic action–tool–object category but differ in object shapes, grasp configurations, and hand-object interactions. The evaluated TACO, OakInk2, and self-collected tasks also cover different manipulation patterns.
Evaluation protocol across datasets. We revised Table 1 to use task-specific physical success conditions and the same binary, bimanual task-level accounting for every method. Public-dataset successes must additionally satisfy the original object-trajectory thresholds. Self-collected tasks lack ground-truth object trajectories and are assessed using their defined physical criteria (Appendix E). We report results by dataset as well as overall.
Q3. “Can the authors report confidence intervals or results under alternative aggregation protocols?”
Alternative aggregation and subset sensitivity. We report micro-averages (equal weight per demonstration), task-macro-averages (equal weight per task group), and leave-one-task-group-out rates (recomputing the micro-average after removing each complete group).
| Method | Public micro SR | Task-macro SR | Leave-one-group-out SR |
|---|---|---|---|
| DexImit | 0.00% | 0.00% | 0.00% |
| Do As I Do | 11.76% | 13.22% | 9.09–15.38% |
| EgoMatrix | 49.02% | 51.79% | 45.45–53.85% |
Public includes the six evaluated task groups in TACO and OakInk2. On TACO alone, EgoMatrix achieves 45.95% (micro) and 49.57% (task-macro), with leave-one-group-out rates of 40.00–52.00%.
Across all 62 demonstrations, EgoMatrix achieves 53.23% with micro-aggregation and 58.61% when the three datasets are weighted equally. The corresponding dataset-macro rates are 18.94% for Do As I Do and 15.15% for DexImit. EgoMatrix retains its advantage under these aggregation and subset protocols.
Q4/W5 — Task Success Criteria and Reconstruction Metrics
We thank the reviewer for highlighting the need for consistent bimanual success accounting and clearer interpretation of alignment-based reconstruction metrics.
Q4. "Can the authors clarify the success-accounting protocol for multi-hand demonstrations, particularly the partial-credit treatment for DexImit and whether equivalent task-level criteria are applied to all methods?"
Unified success evaluation. We thank the reviewer for pointing out this inconsistency. In our original Table 1, only DexImit received 0.5 credit when one of its two hand-object trajectories satisfied the success threshold. EgoMatrix and Do As I Do already required both trajectories to satisfy the thresholds. Although this more permissive treatment did not favor our method, we agree that all methods should be evaluated under the same task-level standard.
We have therefore revised Table 1 to use binary task-level success for all methods, without partial credit, and introduced task-specific physical success criteria to evaluate actual task completion beyond trajectory agreement.
For the six TACO/OakInk2 task groups, the criteria are:
| Task | Task-specific physical success criteria |
|---|---|
| Brush | Brush–bowl contact force N and relative brushing displacement cm |
| Cut | Knife–surface contact force N and cutting displacement cm |
| Screw | Screwdriver tip within 1 cm of the target and contacting the box, with accumulated axial rotation relative to the box |
| Smear | Nozzle within 1 cm of the target surface and surface-relative motion cm |
| Stir | Tool tip cm below the container rim and lateral motion cm relative to the container |
| Rearrange | Tableware within 2 cm of its target placement and stable for s after release |
Success also requires both hands to fulfill their respective roles without unintended object drops. Contact-based conditions must hold for at least 0.2 s.
For TACO and OakInk2, these physical criteria are applied together with the original object-trajectory thresholds ( m / rad). Self-collected tasks lack ground-truth object trajectories and are evaluated using their corresponding physical success criteria, detailed in Appendix E.
Under this unified protocol, EgoMatrix achieves 53.23% (33/62) overall success, compared with 14.52% (9/62) for Do As I Do and 8.06% (5/62) for DexImit. The revised protocol is described in Section 4.1 and Appendix E.
W5. “Several reconstruction metrics focus on relative rather than absolute physical accuracy.”
Reconstruction objectives and evaluation alignment. We appreciate the reviewer's observation that alignment can remove absolute scale and pose errors. We agree that aligned reconstruction metrics should not be interpreted as evidence of metric-accurate global scene recovery. However, this is not the reconstruction objective of EgoMatrix.
As emphasized in our paper, our reconstruction focuses on recovering hand-object relative geometry, contact, and motion sufficiently accurately for downstream manipulation transfer, while maintaining consistency with the source video. We estimate a common scene scale, but do not aim to recover the entire scene in an exact global metric coordinate frame.
This objective also motivates our evaluation design. Alongside conventional reconstruction metrics, we report hand-relative interaction metrics, , SD, and , to assess the geometric properties most relevant to manipulation. More importantly, we evaluate whether the reconstructed interactions can be successfully transferred to dexterous robot manipulation, now with explicit task-specific physical success criteria.
Regarding alignment, Sim(3) is applied only to canonical object geometry evaluation (CD/F@5), while hand MPJPE is root-aligned. Object trajectories use a single rigid, fixed-scale alignment per demonstration, without per-frame alignment or scale correction. These evaluation alignments do not correct the reconstructed data used for downstream transfer.
We will further clarify the scope of these alignment-based metrics and their relationship to our reconstruction objectives in the revised manuscript.
W1/Q1, W2/Q2, W4/Q5 — Downstream Learning and Real-World Evaluation
Dear Reviewer jd3m,
Thank you for your thoughtful review and constructive suggestions. We fully agree that downstream policy learning and real-robot evaluation are among the most direct ways to validate the practical value of EgoMatrix-generated data. We have therefore made these experiments a priority. However, policy training and repeated hardware evaluations take considerable time, and the results are not yet ready. We are actively working on both and are confident that we can report the complete results before the discussion period ends. We also expect these results to provide stronger evidence for our central claims.
In the meantime, we would like to share our responses to the concerns we have already addressed, rather than wait until all experiments are finished. We hope these additional analyses and revisions can resolve some of your questions and leave enough time for further discussion. We would sincerely appreciate your feedback on the points below while we complete the remaining experiments.
Downstream policy learning. We agree that successful trajectory generation alone does not establish the learning value of paired observation-action data. We are fine-tuning a VLA policy using EgoMatrix-generated demonstrations and preparing an action-matched comparison against EgoEngine-style visual synthesis [1]. We keep robot actions, policy architecture, and training settings fixed to isolate the contribution of observation generation. The trained policies will be evaluated through real-robot task execution under varied initial conditions.
Real-world evaluation through VLA deployment. Our original real-robot replays demonstrated that EgoMatrix-generated trajectories can be executed on hardware. We agree that these examples did not quantify replay reliability. However, repeating the same trajectories would mainly measure their execution reliability under the robot controller and given initial conditions. It would not test a central goal of EgoMatrix: generating observation–action data that can train real-world robot policies.
We therefore prioritize the downstream VLA evaluation described above. We are fine-tuning a VLA on EgoMatrix-generated data and will evaluate it through repeated real-robot trials under varied initial conditions, reporting task-level success rates and representative failures. Unlike replaying predefined trajectories, the VLA must use visual feedback to determine its actions during execution. We believe its real-robot performance provides more direct evidence of whether our generated data can support deployable robot policies, which is why we prioritize this evaluation over additional quantitative replay experiments.
Visual fidelity and downstream policy learning. We agree that global image-feature distances cannot directly measure hand-object contact correctness or temporal consistency. EgoMatrix addresses these aspects by jointly rendering robot hands and manipulated objects from the same successful simulation rollout, ensuring that the generated observations are consistent with the recorded robot states and actions.
To further validate the benefit of this design, we are conducting the action-matched VLA comparison introduced in W1/Q1. We keep the robot actions, VLA architecture, and training settings identical, changing only the visual observation synthesis method between EgoEngine-style synthesis and our joint rendering. The resulting real-robot task success rates will therefore help isolate how observation synthesis affects downstream policy learning. We believe this directly addresses the reviewer's suggestion to evaluate visual fidelity through downstream policy performance, while providing evidence of whether our joint rendering produces more useful training data for real-world manipulation.
We welcome feedback on the completed revisions while the remaining experiments are in progress.
Reference
[1] Liu et al., “EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations,” CoRL, 2026.