acceptodds
Withdrawn from ICLR 2027

InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

World-space hand motion estimation from egocentric video requires recovering 3D articulated hand geometry while tracking camera egomotion. Existing approaches heavily rely on cascading independent hand pose estimators and SLAM systems, resulting in error accumulation, complex pipelines, and severe computational overhead. To address these limitations, we present InfiniHand, an end-to-end streaming feed-forward framework that jointly estimates MANO parameters, camera trajectories, and hand locations directly from uncalibrated egocentric video. InfiniHand integrates persistent spatiotemporal memory with hand-centered visual features, explicitly coupling camera motion with local hand geometry within a unified architecture. We train InfiniHand in two progressive stages by first learning robust camera-space hand priors and then extending to streaming world-space reconstruction. To support this process, we aggregate a pretraining corpus of approximately 5,000 hours of egocentric data across multiple public datasets. Extensive evaluations demonstrate that InfiniHand outperforms state-of-the-art baselines on in-domain benchmarks, achieving a 21.4% reduction in ARCTIC PA-p compared to ViDiHand while substantially mitigating world-space drift. Furthermore, InfiniHand generalizes robustly to in-the-wild videos and operates at 11.19 FPS, delivering more than twice the throughput of HaWoR.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Space Hand Motion Reconstruction, Streaming Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Fm1qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Different from previous world-space hand motion reconstruction method that relies on separate models for hand localization, MANO parameter prediction, and camera trajectory estimation, the paper tries to build a unified framework that jointly estimates MANO parameters, camera trajectories, and hand locations directly from uncalibrated egocentric video. The proposed framework is pretrained on a larger set of data by aggregating existing public egocentric datasets with MANO or 3D keypoint annotations. The framework is shown to achieve SOTA accuracy in worldspace hand motion estimation with remarkable efficiency, and strong generalization on in-the-wild videos.

Critical strengths

The paper addresses an important and timely problem: estimating world-space hand motion from egocentric video. This direction is highly relevant to embodied AI, robot learning, dexterous manipulation, and large-scale hand-motion data generation. Compared with methods that only estimate camera-space hand pose, world-space hand motion estimation is more directly useful for downstream robotics and human-object interaction applications.

Besides, the paper aggregates and cleans existing public egocentric datasets into a standardized, high-quality corpus, which could facilitate the community to build a stronger model for egocentric hand motion reconstruction.

Critical weaknesses

Despite the importance of the problem and the potential of the system, I have several major concerns about the current submission.

  1. The baseline comparison does not appear fully fair. The paper emphasizes that InfiniHand is trained with around 5,000 hours of egocentric data, while the compared methods do not seem to use training data at the same scale. Under this setting, it is difficult to determine whether the reported improvements come from the proposed architecture or mainly from much larger-scale pretraining. The authors should provide more controlled comparisons, such as comparing InfiniHand and strong baselines under the same training-data scale, or reporting an ablation where InfiniHand is trained with a smaller amount of data.

  2. On ARCTIC and HOT3D, some metrics of the proposed method are worse than the baselines. This weakens the overall value of the contribution. If the proposed method still underperforms existing methods on some important benchmarks or metrics, the authors should analyze these cases more explicitly rather than only emphasizing favorable results. The current presentation is not sufficient to convincingly show that the method has a stable advantage in world-space hand motion estimation.

  3. Important implementation details are missing or under-specified. The paper should more clearly describe the network architecture of each module, the input and output dimensions, the feature-fusion strategy, the organization of different prediction heads, and the information flow between modules. Without these details, it is difficult to assess the technical soundness of the method and to reproduce the results.

  4. The claim that the method is end-to-end is questionable. The proposed framework relies on a set of models/tools for camera-space reconstruction and world-space reconstruction. The limitation section further mentions that an additional post-processing alignment model is needed to handle metric scale. If the final world-space output depends on such cascading steps and an external alignment step, then it is unclear in what sense the method can still be called end-to-end. Thus, I don't see intrinsic difference of this paper from previous world-space hand reconstruction baseline of HaWoR which also relies on cascading components.

Questions for authors
  1. The comparison of parameter count and efficiency is not sufficiently detailed. What is the parameter count of the full system? Please provide a detailed efficiency table including parameter count, hardware setup, and inference latency.

  2. Does the reported 11.19 FPS include the full pipeline, including camera estimation, bundle adjustment, metric-scale alignment, and other post-processing steps? If not, please report the runtime of each stage separately.

  3. How are the large-scale training corpus and the test sets of ARCTIC, HOT3D, EgoDex, and other benchmarks separated? Are test videos, scenes, objects, and annotations strictly excluded from all training and data-cleaning stages?

  4. Please clarify whether the post-processing module is part of the proposed method, how it is trained, whether it is used during evaluation, whether the same treatment is applied to all baselines, and how much it affects the final results.

Reviewer_uF9mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors proposed InfiniHand, an end-to-end streaming feed-forward framework that jointly estimates MANO hand parameters, hand locations, and camera trajectories from uncalibrated egocentric video. The method bases full-image geometric features that are extracted from a 3D foundation model and then adds WiLoR features and enforces persistent spatiotemporal memory for long video sequences. Overall pipeline is trained using 5000 hours of egocentric videos, followed by a bundle adjustment to reduce long-term camera drift. Experiments on ARCTIC, HOT3D, EgoDex, HOI4D show good performance with 11.19 FPS.

Critical strengths
  • Novel unified design: One streaming model constitutes hand detector, hand estimator and SLAM, giving better world-space accuracy.

  • Fast speed: The pipeline is again well designed to have high throughput, even if it has complex recon. pipeline.

  • Dataset contribution: Authors cleaned 5,000 hours of videos to secure the data sanity.

Critical weaknesses
  • The claim of feed-forwarding network becomes weaker by involving the bundle adjustment step.

  • In-the-wild generalization is only qualitatively given.

  • Less clear whether the good performance is coming from the larger pre-training data or from the proposed architecture. More ablation could be there to clearly analyze this.

  • Authors may need to compare to this, even though it is an ArXiv preprint. It is reporting the performance using the same table and seemingly has better performance: ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery, ArXiv2608.

Questions for authors
  • Please discuss the possibility of removing the bundle adjustment or further converting it into the feed-forwarding method.

  • Please discuss the effect of large data vs. architecture.

Reviewer_82g1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a method called InfiniHand, which is a streaming framework for world-space hand motion reconstruction from egocentric video. It combines LingBot-Map geometry with WiLoR appearance features, jointly trains hand and camera prediction, and refines trajectories through sparse bundle adjustment. Trained on approximately 5,000 hours of curated public data, it reports improved reconstruction accuracy and 11.19 FPS throughput.

Critical strengths
  1. The research question has practical value.
  2. The constructed large-scale training data is valuable.
  3. The method improves computational efficiency while maintaining competitive reconstruction accuracy.
Critical weaknesses

1.The contribution beyond combining existing methods is unclear. The main components are derived from previous studies, but this paper does not explicitly state which designs are new, or what problems these designs solve beyond directly integrating existing methods. The component-removal ablation experiments in Table 3 indicate that these components are useful, but they do not prove that the proposed integration method is more effective than a simple combination. Please clearly describe the new designs, explain their necessity, and provide corresponding comparative analyses.

2.The current experiments do not isolate the benefits of joint hand–camera learning. Introducing LingBot-Map simultaneously changes the geometric features, camera estimator, and pre-trained priors. Removing the second stage in Table 3 also means simultaneously removing camera adaptation, world-space supervision, and joint training. This highlights the importance of the training stage, but does not prove that joint learning is superior to separate training. A useful control experiment is to independently adapt the same LingBot-Map and WiLoR components using the same training data and comparable training budgets, and then combine them using the same scale recovery and BA methods. Equivalent decoupled training ablation experiments can also address this concern.

3.The composition and quality of the training corpus are insufficiently introduced. This paper reports approximately 5,000 hours of data and during the data cleaning process, 30% to 40% of the candidate data were eliminated. However, it does not provide information on the data volume, supervision type, sampling weight of each source before and after cleaning, or the amount of data used in the first and second stages. Please provide these statistics, distinguish between native and fitted MANO annotations, and describe the training/testing split and cross-source deduplication process. The lack of information alone does not imply data leakage. Additionally, the agreement between Sapiens 2 predictions and projected MANO joints can serve as an effective screening signal, but it is not sufficient to fully verify 3D pose, depth, or metric scale. Independent quality checks, filtering thresholds, and statistics on the retention of difficult samples will help strengthen the contribution of this dataset.

4.The benefits of data scale, cleaning, and diversity are not directly evaluated. Table 3 only ablates components and training stages, and does not compare the data configurations under a fixed model. Therefore, the improvement of the overall system cannot directly prove the contribution brought by the scaling of diverse supervision. Priority experiments should include comparisons of data scales, as well as comparisons between the cleaned data and the equally sized randomly selected uncleaned candidate data. The claim of generalization performance also needs to be explained: Xperience-10M appears in both the training data source and the qualitative examples, so the sample split needs to be clarified. On the HOI4D task, PA-p has improved, while MP-p and wrist position error have deteriorated, indicating that the generalization advantage does not cover all geometric metrics.

5.Efficiency and long-sequence stability require more targeted evaluation. The reported 11.19 FPS was achieved using multiple prediction workers and background BA, but the hardware configuration, resource allocation, and complete timing boundaries were not specified. The implementation method and cost of the auxiliary scale recovery model are also unclear. Please clarify whether the throughput covers the entire pipeline, whether the reported accuracy corresponds to the initial prediction or the prediction after BA optimization, and report lookahead and latency for the initial and final outputs. Although BA ablation experiments support the contribution of BA to overall accuracy, they cannot directly quantify long-term drift. Camera ATE/RPE, errors grouped by sequence length, and error-over-time curves will provide more direct evidence.

Questions for authors

Please solve the concerns in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.