acceptodds
Withdrawn from ICLR 2027

EgoAct4D: Estimating Bimanual 4D Hand Motion from Egocentric Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Turning egocentric video into action supervision requires accurate and efficient hand motion estimation. Multi-camera capture and wearable sensing provide accurate motion supervision but are costly to scale. Monocular reconstruction could enable large-scale supervision from abundant Internet videos, but existing pipelines require multiple stages and incur motion errors of several centimeters. To address these limitations, we present EgoAct4D, a simpler pipeline for more accurate bimanual 4D hand motion estimation from monocular egocentric RGB videos. Its single-stage hand estimator predicts bimanual states from short clips, including occluded and out-of-view poses, while a separate estimator directly predicts metric camera poses. The pipeline reuses hand predictions and stitches overlapping camera trajectories to express hand motion in each window's first-frame camera coordinates and pair it with the starting observation. Since existing alignment-based evaluations can mask position and motion errors, we measure camera-space joint error (Absolute MPJPE) and hand-trajectory error (Action MPJPE) in sliding windows without ground-truth alignment. EgoAct4D achieves state-of-the-art results on both metrics on the HOT3D benchmark. We then build the EgoAct4D dataset and demonstrate full-pipeline data scaling beyond HOT3D's training scale. On its test set, EgoAct4D achieves 7.95 mm Absolute MPJPE and 10.98 mm Action MPJPE at an end-to-end throughput of 49.3 FPS. This accuracy and efficiency can support reusable motion supervision for vision-language-action pretraining and human-to-robot motion retargeting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric vision, Bimanual 4D hand motion estimation, Camera pose estimation, Motion supervision, Data scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4fYqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoAct4D estimates bimanual hand motion from egocentric video using separate hand and camera estimators. It emphasizes absolute hand placement and short trajectories without ground-truth alignment. The paper evaluates on HOT3D and a newly collected EgoAct4D dataset, with data-scaling, architecture and inference ablations.

Critical strengths
  1. The HOT3D results show improvements in absolute hand placement and action trajectories, including the broader evaluation with out-of-view states. These metrics are relevant to using reconstructed motion as action supervision.
  2. The pipeline is practical. Prediction reuse and camera stitching improve throughput, and the paper includes useful camera-source and data-scaling experiments.
Critical weaknesses
  1. External validation is limited to HOT3D. The additional EgoAct4D experiments are useful, but do not replace evaluation on another established benchmark or transfer to an unseen dataset. This leaves robustness across capture conditions, subjects and interaction types insufficiently demonstrated.
  2. The construction of EgoAct4D is insufficiently documented. Section 4.3 and Appendix C.1 describe its composition and splits, but do not explain how the 3D hand and camera ground truth was obtained or independently validated. Without the capture setup, annotation pipeline and label accuracy, the reported near-centimeter errors are difficult to interpret.
  3. The contribution relative to existing geometry-based and world-space hand reconstruction is not sufficiently isolated. The hand-encoder ablation compares DA3 initialization with a generic ViT and Qwen3-VL, but not other geometry-pretrained alternatives such as MapAnything or VGGT-Omega. It is encouraged to test these pretrained models that supply strong geometric cues.
Questions for authors

please clarify concerns raised in the weaknesses section.

Reviewer_65YaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoAct4D, a pipeline for estimating bimanual 4D hand motion from monocular egocentric videos. It combines a single-stage hand estimator using metric-depth pretraining and structured temporal queries with a separate estimator of metric camera poses. Hand prediction reuse and camera-trajectory stitching enable efficient processing of continuous video. The paper also adopts an alignment-free sliding-window evaluation protocol and introduces a dataset containing 55.3 hours of training videos. Experiments demonstrate substantial improvements in absolute hand placement on HOT3D, consistent improvements with additional training data, and 10.975 mm Action MPJPE at 49.26 FPS on the new dataset.

Critical strengths
  1. Well-Motivated Problem Formulation. Existing hand reconstruction methods often prioritize local hand pose accuracy while paying less attention to absolute hand localization in metric 3D space, which is particularly important for robotic manipulation. The paper addresses this limitation through two complementary designs. First, it leverages metric geometric features from DA3Metric to improve absolute hand localization. Second, it transforms hand states across time into the coordinate system of the initial camera frame, producing temporally coherent trajectories suitable for action supervision. I find this shift from local hand pose reconstruction to camera-anchored motion estimation meaningful and practically relevant. The proposed technical designs are also well aligned with the underlying motivation.

  2. Simple, Efficient, and Practical Framework. Rather than relying on a complicated multi-stage reconstruction pipeline, the proposed method directly estimates bimanual hand states using full-image geometric features and a structured temporal decoder. The authors also explicitly consider the computational cost of processing large-scale video data, achieving a 9.27× throughput improvement through hand prediction reuse and camera trajectory stitching. I particularly appreciate that the reported end-to-end throughput accounts for camera estimation, data preprocessing, and trajectory assembly, rather than focusing solely on individual model components. This makes the efficiency gains more convincing and relevant to practical deployment.

Critical weaknesses
  1. Limited Evidence of Downstream Utility for Robot Learning. One of the paper's central motivations is to convert egocentric human videos into scalable action supervision for VLA pretraining and human-to-robot retargeting. However, while the proposed method achieves lower Action MPJPE, it remains unclear to what extent these improvements translate into better downstream robot learning performance. Since the paper primarily focuses on hand motion estimation, I do not think a comprehensive robot learning study is necessary. Nevertheless, given the emphasis on scalable action supervision, even a lightweight downstream evaluation would substantially strengthen the practical significance of the work.

  2. Lack of Cross-Dataset Generalization Evaluation. The paper aims to scale action supervision by leveraging diverse egocentric videos available online, which naturally raises questions about generalization across scenes, devices, and tasks. However, the current evaluations use models trained separately on HOT3D and EgoAct4D, without sufficiently examining whether a model trained on one dataset can generalize directly to another. This is particularly important because differences in camera intrinsics, viewpoints, and motion patterns may affect metric depth estimation and camera pose estimation. Without cross-dataset evaluation, it is difficult to assess how robust the proposed method would be when applied to large-scale videos from diverse sources.

Questions for authors

Overall, I lean toward a weak acceptance. I appreciate the well-motivated problem formulation and the substantial improvements in estimation accuracy and efficiency. However, I still have some concerns regarding the downstream utility and generalization capability of the proposed method. Addressing the following questions, ideally with additional experimental evidence, would help strengthen my confidence in the work.

  1. Downstream Utility for Robot Learning. Could the authors provide a lightweight downstream evaluation to demonstrate whether improved action supervision leads to tangible benefits for robot learning? For example, the authors could train the same policy model using action labels extracted by different methods and compare task success rates. Alternatively, a human-to-robot retargeting experiment could also help assess whether the predicted trajectories are feasible under robot kinematic constraints.

  2. Cross-Dataset Generalization. Could the authors include a zero-shot cross-dataset evaluation without any fine-tuning, such as training on EgoAct4D and evaluating directly on HOT3D, and vice versa? This would help assess the robustness of the proposed method to variations in camera parameters, viewpoints, and scene distributions, while providing stronger evidence of its potential for processing large-scale real-world videos.

Reviewer_w5SLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EgoAct4D for estimating bimanual 4D hand motion from monocular egocentric videos. It uses a pretrained Depth Anything 3 encoder to estimate hand states, together with a separate camera estimator and a sliding-window pipeline to combine predictions across video clips. The authors also introduce a dataset with 55.3 hours of training videos and report improvements in motion accuracy and inference speed over existing methods. However, the method mainly builds on existing pretrained models and standard architectural components, and its technical novelty is limited. The dataset annotation process also needs to be better documented, and the experiments do not fully establish the method's generalization or its benefits for downstream robot learning. Overall, I do not think the current version provides a sufficiently strong methodological contribution for ICLR.

Critical strengths
  1. The paper tackles the problem of recovering hand motion from egocentric videos, which could provide useful action data for imitation learning and VLA training.
  2. Reusing hand predictions across overlapping windows and stitching camera trajectories reduces redundant computation while keeping motion accuracy largely unchanged.
  3. The paper evaluates 3D hand positions and trajectories without ground-truth alignment, allowing errors in absolute position and camera motion to be captured rather than removed during evaluation.
  4. The experiments include both HOT3D and the proposed EgoAct4D dataset, with ablations examining the encoder, camera estimator, inference stride, and training data size.
Critical weaknesses

1.The method mainly combines existing pretrained models and standard components, with limited methodological innovation.

2.The paper lacks sufficient details on how the 3D hand and camera annotations are obtained and validated.

3.The modified baseline pipelines and shared camera estimates make it difficult to attribute the improvements solely to better hand estimation.

4.Experiments on HOT3D and EgoAct4D do not establish generalization to other datasets or recording environments.

5.The scaling experiments use subsets of a single dataset and do not support broader claims about large-scale Internet videos.

6.The paper does not show whether the estimated hand motions improve robot policy learning or manipulation.

Questions for authors
  1. How are the 3D hand and camera annotations in EgoAct4D obtained and validated, and what is their expected measurement error?
  2. Can the authors provide a comparison with existing methods using their original pipelines to verify that the improvements are not caused by changes in baseline processing?
  3. Can the proposed method generalize to other egocentric datasets and recording devices without additional fine-tuning?
  4. Does EgoAct4D improve downstream VLA training or robot manipulation compared with existing hand-motion supervision methods?
Reviewer_bSsgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method contains a metric-depth-pretrained hand encoder, temporal bimanual queries, a separate metric camera estimator, and trajectory stitching. It reports 15.773 mm Action MPJPE on HOT3D’s in-frame subset and 10.975 mm at 49.26 FPS on its own dataset.

Critical strengths
  • Relevant objective: Metric trajectories expressed in the starting camera frame address action-supervision directly and removed necessary post-processing for wrist-centered hand poses.

  • Large metric gains: The proposed method improves HOT3D in-frame Action MPJPE improves from HaWoR’s 42.382 to 15.773 mm, a 62.8% reduction.

Critical weaknesses
  • Lack of novelty Or lack of justification: The model architecture and training design extremely lacks technical novelty, the only "kind of new" thing that I see is the usage of DA3 as encoder, then all the rest are simple transformer encoder-decoder designs. However, given the proposed method achieves strong metric improvment over previous baselines, I would like to see where those improvements are from? Is it from the depth-pretrained encoder? or from the head design? I would like to see more ablations on this.

  • Confusing details in "EFFICIENT VIDEO INFERENCE":

    • L253: "The hand estimator runs at stride 16, reusing its per-frame predictions across output windows". I am sorry but I don't really understand what is this, what is the
    • How "common reference frame" was choosed?
    • Why this design is efficient? It will be nice to have a figure that shows which part of the compute was saved
  • Missing Benchmark: It will be nice to see the proposed method's performance on the more benchmarks on the same task, for example:

    • ARCTIC
    • H2O
    • EgoDex
    • HOI4D
    • ...
  • Missing modern competitors: ViDiHand, ACE-Ego-Hand, HandFlow, and UniHand are relevant alternatives; the current table cannot establish superiority over the available temporal reconstruction methods.

  • I will be nice if more in-the-wild results can be shown, especially in such cases:

    • results when the hands are severely occluded
    • results on different camera intrinsics
    • results on multi-view cameras
    • results on fish-eye cameras, e.g., on the quest-subset of HOT3D
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.