acceptodds
Under review as a conference paper at ICLR 2027

RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning

Abstract

Recently, approaches that leverage human video datasets for robot policy training have become increasingly prevalent. However, most existing hand trackers regress pose from cropped frames with limited priors on hand motion and object interaction, resulting in inaccurate and physically inconsistent estimates. Moreover, the lack of physical cues, e.g., contact and force, limits the use of human videos for robot policy training. To this end, we propose RLHND, a video foundation model-based hand tracking model that jointly estimates hand pose and realistic tactile information from monocular egocentric videos. RLHND turns the pre-trained Cosmos 3 video diffusion backbone into a deterministic clip-level feature extractor via clean-latent conditioning, carrying its learned priors on hand motion and hand-object interaction into tracking. For pose estimation, RLHND (i) predicts hand poses with anatomically plausible joint angles and (ii) enables optional conditioning on the shape parameter to maintain consistent hand shape within the same video and even across videos recorded by the same actor. For tactile estimation, a separate tactile expert stream, trained with the pose stream frozen, predicts dense contact and force over the hand surface. We further adopt LBS-based feature spreading to enable vertex-wise feature extraction without costly per-vertex attention. RLHND achieves state-of-the-art performance across various benchmark datasets for pose estimation, while also achieving state-of-the-art performance in contact and force estimation. Moreover, we demonstrate the utility of RLHND for robot learning through retargeting results and real-world robot experiments. The code will be publicly available at https://anonymous4898.github.io/rlhnd/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-to-robot, hand motion reconstruction, tactile estimation, robot learning, human egocentric video

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5L8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLHND, a video foundation model based hand tracking framework that jointly estimates 3D hand motion, contact, and force information from monocular egocentric videos. The method uses Cosmos 3 as a video feature extractor and introduces shape caching, anatomical pose constraints, and a decoupled tactile expert. Experiments on hand reconstruction, tactile estimation, and robot learning benchmarks demonstrate the effectiveness of RLHND.

Critical strengths
  1. The paper proposes a physically grounded hand tracking framework for robot learning. Different from traditional methods that only estimate hand pose, RLHND jointly predicts hand motion, contact, and force information, making human videos more useful for robot learning.

  2. Using video foundation model priors is an interesting and valuable direction. The paper leverages the motion and hand-object interaction priors learned by Cosmos 3 to improve video-level hand reconstruction consistency, instead of relying only on independent frame-level predictions.

  3. Besides hand reconstruction and tactile estimation, the paper demonstrates retargeting to multiple dexterous hands and robot policy training, which improves the practical significance of the work.

Critical weaknesses
  1. RLHND heavily relies on the Cosmos 3 backbone and builds additional designs on top of it. The main contributions, including shape caching, anatomical constraints, and tactile expert design, are useful but appear more like engineering improvements. More analysis is needed to show their independent contributions compared with previous approaches.

  2. The paper claims that video foundation models provide strong priors for hand motion and interaction. However, it is unclear how much of the improvement comes from Cosmos 3 pre-training and how much comes from the proposed RLHND designs. Comparisons with other backbones or smaller video encoders would help clarify the main source of the improvements.

  3. RLHND predicts contact and force from visual input, but physical interaction signals are strongly related to real contact dynamics. Current experiments mainly evaluate prediction accuracy, while lacking analysis of whether the predicted tactile information can reflect real interaction states, such as grasp stability or failure detection.

  4. The paper demonstrates retargeting and Dexterous Point Policy training, but these experiments mainly show that RLHND outputs are useful. More experiments are needed to understand whether RLHND directly improves robot learning performance, such as evaluations on more tasks, more environments, or comparisons with other human-video-to-robot learning approaches.

Questions for authors
  1. How much of the performance improvement comes from the Cosmos 3 backbone? If another video foundation model or a standard video encoder is used, do the proposed designs, such as shape caching and anatomical pose parameterization, remain effective?

  2. Can the predicted tactile information (contact and force) improve robot manipulation performance in downstream tasks? More robot experiments based on RLHND tactile predictions would strengthen the claim.

  3. Shape caching reduces hand-size fluctuation, but how does it perform when the hand undergoes significant appearance changes, such as wearing gloves, large viewpoint changes, or different operators?

  4. The current robot experiments mainly focus on retargeting and policy training. Can the authors provide direct comparisons with other human-video-to-robot learning methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5L8U (1/2)

We thank the reviewer for recognizing the value of jointly predicting hand motion, contact, and force, and the potential of video foundation model priors for hand tracking.

Distinct contributions of RLHND (W1). RLHND introduces several task-specific designs for adapting video foundation models to hand tracking and tactile estimation. First, it uses the native clean-latent conditioning interface of Cosmos 3, rather than passing clean frames through the denoising interface at σ=0 as in prior video-diffusion trackers. Second, external shape conditioning and caching enforce a consistent hand shape, while the anatomically constrained pose parameterization restricts predictions to anatomically plausible joint configurations. Third, the tactile expert uses MANO skinning weights to map token features to mesh vertices, introducing an anatomical correspondence for dense contact and force prediction. Replacing this mapping with a learnable matrix reduces contact F1 on all three benchmarks (T1 in Table 5).

Contributions of the backbone and proposed designs (W2, Q1). We extended the pose ablations in Table 5 of the revised paper from Cosmos 3 Nano to two additional backbones: Wan2.2 (used by ACE-Ego-Hand) and Cosmos 3 Edge. On all three backbones, ARCTIC MPJPE-p rises by 0.4-1.5 mm without external shape conditioning and caching (A2, B2, C2), and on Wan2.2 and Edge, EgoDex MPJPE-p rises by 0.8-2.2 mm without the anatomically constrained pose parameterization (A1, B1). These results also clarify what the backbone contributes. With Wan2.2 in place of Cosmos 3, RLHND without shape input is roughly level with ACE-Ego-Hand re-trained on our training mixture (ARCTIC 16.62 vs. 16.33 mm, EgoDex 21.57 vs. 21.74 mm), so the backbone is responsible for much of the accuracy. What the backbone cannot supply is a way to use a known hand shape, and this is where shape conditioning is more than an engineering refinement. Feeding the operator's ground-truth β lowers ARCTIC MPJPE-p by 1.1 mm on Wan2.2 and 1.3 mm on Edge (A4, B4), reaching 15.55 mm on Wan2.2, below re-trained ACE-Ego-Hand (16.33 mm), whose hand tokens start from fixed learnable parameters and have no input for such a measurement. On Cosmos 3 Nano, which already recovers hand shape well from video, the gain is 0.2 mm and appears mainly in retargeting (Q-err 10.12 → 9.69). A one-time hand calibration during robot data collection can therefore be turned into more accurate labels, particularly when the visual prior is weaker. For tactile estimation, the corresponding ablations show that both the Cosmos 3 backbone (T0) and LBS-based feature spreading (T1) improve contact estimation (Table 5).

Tactile Signals and Real Interaction (W3, Q2). We added a simulation experiment that holds hand motion fixed and varies only the contact-based control. We replayed all 55 segments of the ARCTIC egocentric validation split in which an object is lifted at least 3 cm from the table without articulation. Hand keypoints are mapped to world coordinates with the ground-truth camera and retargeted to two floating Shadow Hands in MuJoCo with DexPilot. The object is initialized at its ground-truth pose on the table and subsequently moves only through physical contact with the hands.

Following DPP, we add a 0.3 rad closing offset to the flexion targets of each finger predicted to be in contact (at least five MANO vertices above 0.8). A trial succeeds if the object reaches at least half its ground-truth height above the table in at least 80% of the lifted frames. We report mean success over five runs with randomized object mass and friction.

With ground-truth hand motion, RLHND contact increases success from 37.1% to 58.2%, compared with 62.2% using ground-truth contact. Applying the same closing offset to all fingers in every frame achieves only 45.8%, supporting the value of selecting when and where to apply the offset using predicted contact. With RLHND-estimated hand motion, predicted contact increases success from 28.0% to 32.7%, while slightly increasing the object pose error.

Details and a rollout example are in the new Appendix B.2 (Table 9 and Figure 14).

ConfigHand motion (ARCTIC replay)Contact offsetSuccess (%)Object error (cm)
P1HaWoRnone9.133.8
P2ACE-Ego-Handnone22.524.0
P3RLHNDnone28.019.8
P4RLHNDRLHND (Ours)32.721.2
P5Ground Truthnone37.117.6
P6Ground Truthall fingers, every frame45.820.7
P7Ground TruthRLHND (Ours)58.215.5
P8Ground TruthGround Truth (upper bound)62.215.9
AuthorsOpenReview ↗

Response to Reviewer 5L8U (2/2)

Real-Robot Contact Ablation (W3, Q2, continued). On a real robot, in a setup separate from Table 4, we followed the Dexterous Point Policy (DPP) recipe. We used RLHND to annotate contact on approximately 1M egocentric human video episodes and pre-trained a bimanual policy for two 20-DoF hands to predict keypoints and per-keypoint contact intensities. At deployment, the predicted intensity determines the magnitude of a keypoint target offset along the finger-closing direction.

We evaluated pick-and-place with four objects and 64 trials per variant, fine-tuning on 10% of the teleoperated demonstrations. As shown below, removing RLHND contact labels, using binary contact, or disabling the contact-driven offsets all reduce success, as objects often slipped during lifting without the offsets.

Together, the simulation and real-robot results show that RLHND contact predictions provide actionable information for maintaining grasps beyond hand pose alone. These experiments assess the utility of predicted contact for grasp maintenance rather than directly measuring physical force.

Real robot, pick-and-place (64 trials)Success rate (%)
w/o RLHND contact labels in pre-training43.8
Binary instead of graded contact37.5
w/o contact-driven offset at deployment25.0
Full (graded RLHND contact + contact-driven offset)56.3

Comparison with Human-Video-to-Robot Methods (W4, Q4). We additionally evaluate VITRA, a vision-language-action model that learns dexterous manipulation from egocentric human videos annotated using HaWoR. We trained two VITRA variants on the same human and robot episodes used in Table 4: one using the original HaWoR annotations and the other using RLHND annotations. The architecture, initialization, robot data, and training schedule were identical. Both variants were evaluated using the same robot setup (RB-Y1 with two 20-DoF hands), objects, and number of trials.

Replacing HaWoR annotations with RLHND annotations increases VITRA's overall success rate from 73.4% to 80.7%, with improvements on both pick-and-place tasks (82.8% to 89.1%) and bimanual tasks (54.7% to 64.1%). Table 4 of the revised paper and the table below report these results alongside DPP results under the same evaluation protocol. These results demonstrate that RLHND annotations improve two human-video-to-robot learning methods with different policy architectures and action spaces.

In the physics replay of W3, which involves no policy learning, RLHND hand motion without contact offsets achieves 28.0% lift success, compared with 22.5% for ACE-Ego-Hand and 9.1% for HaWoR.

Method (real robot, same human and robot episodes)Pick & Place avgBimanual avgAll
VITRA (HaWoR labels)82.854.773.4
VITRA + RLHND89.164.180.7
DPP (HaWoR keypoints)89.164.180.7
DPP + RLHND93.875.087.5

Shape Caching under Appearance Changes (Q3). RLHND targets single-worker recordings, consistent with the tracking setup of egocentric hand datasets such as EgoDex and HOT3D. Operator changes within a video fall outside this setting, as stated in the limitations of the revised paper (Section 5).

Within this setting, appearance changes that persist throughout the sequence do not affect tracking stability. As shown in the new Appendix B.6 (Figure 16), we evaluated RLHND on 30 egocentric episodes recorded with white tactile gloves and 30 bare-hand episodes of the same packing task, captured on the same day with the same rig. Articulation accuracy is unchanged (PA-MPJPE 8.0 vs. 8.1 mm), with detection rates of 99.0% and 99.1% on frames with a visible hand. For viewpoint changes, exchanging cached shapes between the simultaneous egocentric and exocentric ARCTIC cameras changes MPJPE-p by at most 0.45 mm (table below), well below the 2.12 mm increase caused by another operator's shape (Table 6 in Appendix A.1 of the revised paper).

ARCTIC MPJPE-p (mm, lower is better)Cache from the same viewCache from another viewChange
Egocentric view (cache from exocentric views)13.4413.15-0.29
Exocentric landscape views (cache from egocentric view)16.1516.57+0.42
Exocentric portrait views (cache from egocentric view)17.5918.04+0.45

The remaining case is a change in hand appearance within a video, e.g., putting on gloves midway. Since shape caching is optional and per-clip shape estimation achieves comparable accuracy (within 0.08 mm MPJPE-p across benchmarks, Table 1), such videos can be processed with per-clip shape estimation, which adapts the shape to each clip without degrading accuracy.

AuthorsOpenReview ↗

Response to Reviewer 5L8U (Follow-up)

Dear Reviewer 5L8U,

Thank you again for your review. As the discussion period ends soon, we wanted to briefly check whether our response has addressed your concerns. In summary, we added (1) backbone ablations on Wan2.2 and Cosmos 3 Edge showing that shape conditioning/caching and the anatomical pose parameterization remain effective across backbones, (2) simulation and real-robot experiments showing that predicted contact improves grasp maintenance, (3) a direct comparison with VITRA, where RLHND labels improve success from 73.4% to 80.7%, (4) analyses of gloves and viewpoint changes for shape caching, and (5) a backbone-matched comparison (posted in our reply to Reviewer E7AX): with the same Cosmos 3 Nano backbone and training data, RLHND still lowers MPJPE-p over ACE-Ego-Hand on all benchmarks (HOT3D 13.25 → 13.01, ARCTIC 14.59 → 13.36, EgoDex 20.49 → 19.90 mm), isolating the contribution of our design choices beyond the backbone. If any concerns remain, we would be glad to clarify them before the deadline.

Best regards, The Authors

Reviewer_E7AXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RLHND, a video foundation model-based hand tracking model that jointly estimates hand pose and realistic tactile information from monocular egocentric videos. RLHND leverages the pre-trained Cosmos 3 video diffusion backbone for downstream tasks.

Critical strengths
  • This paper seems interesting and addresses the problem of hand pose estimation as well as tactile estimation.
  • Authors provide some demos.
Critical weaknesses
  • The performance seems good, but it may stem from the stronger backbone i.e. Cosmos 3. Please provide more detailed results about the performance gain in terms of backbone and proposed designs.
  • This paper lacks failure cases.
Questions for authors
  • Provide more detailed performance ablations.
  • Provide limitations and failure cases.
5 comments
AuthorsOpenReview ↗

Response to Reviewer E7AX

We thank the reviewer for the positive assessment and for finding the joint estimation of hand pose and tactile signals interesting.

Backbone vs. Design (W1, Q1). We agree that the backbone contributes to performance and provide two complementary analyses to distinguish the effects of training data, backbone choice, and the proposed designs.

First, we retrained the strongest pose baseline, ACE-Ego-Hand, on the same training mixture as RLHND and evaluated both models on identical dataset splits. Under matched training data, RLHND achieves 1.8-3.0 mm lower MPJPE-p across all three test sets, as shown below and in Table 14 (Appendix B.9) of the revised paper. Thus, differences in training data alone do not explain the improvement over ACE-Ego-Hand.

MPJPE-p (mm, lower is better)HOT3DARCTIC-egoEgoDex
ACE-Ego-Hand, re-trained on our data14.8716.3321.74
RLHND13.0113.3619.90

Second, we extended the pose ablations in Table 5 of the revised paper to two additional backbones: Wan2.2, used by ACE-Ego-Hand, and Cosmos 3 Edge. Removing external shape conditioning and caching (A2, B2, C2) increases ARCTIC MPJPE-p by 0.4-1.5 mm on all three backbones, and removing the anatomically constrained pose parameterization (A1, B1) increases EgoDex MPJPE-p by 0.8-2.2 mm on Wan2.2 and Edge. The backbone accounts for a large share of the gain. With the Wan2.2 backbone of ACE-Ego-Hand, RLHND without shape input is on par with re-trained ACE-Ego-Hand (16.62 vs. 16.33 mm on ARCTIC, 21.57 vs. 21.74 mm on EgoDex), while replacing Wan2.2 with Cosmos 3 Nano, used through its native clean-latent conditioning interface, lowers ARCTIC MPJPE-p by 3.3 mm (A3 vs. C3). Alongside the component gains above, selecting and adapting this backbone is a key part of our design. However, RLHND additionally exposes a shape interface. With a strong backbone (Cosmos 3 Nano), RLHND already recovers hand shape reliably from video, so the ground-truth shape brings only a small gain (0.2 mm on ARCTIC), mainly improving retargeting (Q-err 10.12 → 9.69). When the visual prior is weaker, the same interface becomes more valuable (1.1-1.3 mm on Wan2.2 and Edge), and on Wan2.2 it lets RLHND surpass re-trained ACE-Ego-Hand (15.55 vs. 16.33 mm), which initializes its hand tokens from fixed learnable parameters and has no means to use a calibrated shape. Shape conditioning thus acts as a safeguard: a one-time hand calibration in robot data collection can be exploited when visual cues are insufficient.

Together, these results show that the Cosmos 3 backbone accounts for a substantial part of the improvement, while the proposed components yield consistent gains on every tested backbone. In addition, shape conditioning lets RLHND exploit a calibrated hand shape that ACE-Ego-Hand cannot use. For tactile estimation, Table 5 separately evaluates the backbone (T0) and LBS-based feature spreading (T1), and both improve contact estimation.

Failure Cases and Limitations (W2, Q2). We moved the limitations to the main text of the revised paper (Section 5) and added a failure-case figure (Figure 5). As shown in Figure 5, RLHND can predict an anatomically plausible finger configuration that differs from the ground truth under partial occlusion, even when the wrist and palm are localized accurately. RLHND also assumes a single operator per video and may incorrectly assign a hand slot to another person's hand in scenes with multiple people. Distinguishing the camera wearer's hands from those of other people remains a direction for future work. Section 5 also discusses the limited diversity of force supervision in our training data.

Reviewer_E7AXOpenReview ↗

Bakcbone

Thanks for the response! Do the results in the table simply replace “backbone” with “ACE-Ego-Hand” while leaving everything else unchanged? If not, please do so to get a clearer picture of backbone’s performance. Of course, it’s fine to use a more powerful Backbone.

AuthorsOpenReview ↗

Response to Reviewer E7AX (Backbone-matched comparison)

We thank the reviewer for the follow-up. The table compares the two full models trained on the same data, with each model retaining its respective backbone. To further isolate the effect of the backbone, we additionally trained an ACE-Ego-Hand variant with its Wan2.2-TI2V-5B backbone replaced by Cosmos 3 Nano, while keeping its standard MANO head, learnable hand tokens, training data, and 20k-step schedule unchanged.

MPJPE-p (mm, lower is better)BackboneHOT3DARCTIC-egoEgoDex
ACE-Ego-Hand, re-trained on our dataCosmos 3 Nano (our adaptation)13.2514.5920.49
RLHNDCosmos 3 Nano13.0113.3619.90

With the same backbone and training data, RLHND lowers MPJPE-p consistently on every benchmark. We will add this comparison to the revised paper.

Both models use Cosmos 3 Nano through the same clean-latent conditioning interface (Section 3.1), so the comparison below isolates the remaining design choices of RLHND (shape conditioning and caching, and the anatomically constrained pose parameterization).

Additionally, beyond pose accuracy, the remaining design choices provide properties that ACE-Ego-Hand lacks and that matter for robot learning. RLHND outputs anatomically constrained joint angles that can be retargeted directly, and it accepts a calibrated hand shape, which ACE-Ego-Hand cannot use because its hand tokens start from fixed learnable parameters.

Reviewer_E37eOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RLHND, a video foundation modelbased hand tracking model that jointly estimates hand pose and tactile information from monocular hand-object interaction videos. For pose estimation, RLHND exploits the strong Cosmos-3 backbone as visual encoder, converts the original 45-DoF MANO labels into anatomically plausible joint angles, and introduces optional conditioning on the shape parameter to maintain consistent hand shape. For tactile estimation, a separate tactile expert stream is used and LBS-based feature spreading is adopted to enable vertex-wise feature extraction. Extensive experiments are conducted to validate the effectiveness of RLHND across various benchmark datasets for pose estimation and contact/force estimation. Moreover, the utility of RLHND for robot learning is also demonstrated.

Critical strengths
  1. Important problem and potential impact for robot learning.
    The paper addresses a meaningful and practically relevant problem: converting egocentric human hand videos into useful supervision of 3D hand pose and contact/force information for robot learning. If the method is robust and cost-effective, it could be useful for generating supervision from large amounts of human video data, especially for dexterous manipulation tasks where contact and hand-pose annotations are difficult to obtain.

  2. Broad evaluation coverage.
    The paper reports results across multiple tasks, including hand pose reconstruction, tactile/contact estimation, retargeting to dexterous robot hands, and real-robot learning experiments. This breadth is valuable and shows that the authors are trying to evaluate the method beyond a single benchmark.

Critical weaknesses
  1. Possible failures and adaptation problem for shape caching.
    Section 3.2 estimates the shape parameter from the first sufficiently visible clip and then reuses it throughout the video. While this design may reduce hand-size fluctuation, it also introduces a potential failure mode: if the early clip is affected by motion blur, incorrect hand detection, or unusual pose, the cached shape error may propagate through the entire sequence. This issue may become more serious in videos involving different actors, different hands, or multi-person/multi-hand scenarios. Table 1 reports the RLHND (β-cache) result, but the paper does not provide a systematic analysis of impacts from possible failures of shape caching, or multi-actor scenarios. Without such analysis, it is difficult to assess whether shape caching is robust enough for real-world deployment.

  2. The baseline comparison is not fully fair or well controlled.
    Section 4 states that RLHND is trained using a weighted mixture of egocentric and exocentric hand-video datasets, and that the tactile stage uses force-labeled glove/pressure-pad recordings plus contact-only labels. In contrast, all baselines are evaluated using their official weights and inference code. Therefore, the comparisons in Table 1/2 against existing works do not isolate the effects of model architecture and training data. This makes it hard to determine how much of the gain is due to the proposed method itself rather than stronger data and model capacity.

Questions for authors
  1. How does shape caching behave when extreme situations occur, such as occlusion, motion blur, unusual hand pose, or a different actor/hand from later clips? Can the authors report failure rates or sensitivity to the visibility threshold?

  2. Can the authors provide a controlled comparison where the strongest baseline, e.g., ACE-Ego-Hand, is retrained using the same training data mixture?

  3. How much of RLHND’s improvement comes from the Cosmos 3 backbone, larger training data, shape caching, anatomical constraints, and tactile stream? A more detailed ablation study could better illustrate how RLHND really works.

3 comments
AuthorsOpenReview ↗

Response to Reviewer E37e

We thank the reviewer for recognizing the practical relevance of our problem and its potential impact on robot learning, as well as the breadth of our evaluation.

Shape Caching (W1, Q1). The cached shape is not necessarily taken from the first clip. As detailed in Algorithm 1 and Figure 6 in Appendix A.1, the cache is initialized separately for each hand using the first clip in which that hand is predicted to be present in more than half of the frames. If no clip satisfies this criterion, each clip retains its own shape estimate, corresponding to the per-clip RLHND setting in Table 1. Each hand has a separate cache, and all caches are reset for every video. RLHND assumes a single operator throughout each video, and operator changes and hands from multiple people are not currently handled, as clarified in the revised limitations (Section 5).

Shape caching is optional: disabling it changes MPJPE-p by at most 0.08 mm across the evaluated benchmarks. Its purpose is to maintain a consistent hand size within a video with little effect on pose accuracy. This differs from removing shape conditioning during training (A2, B2, C2 in Table 5), which degrades accuracy.

To assess sensitivity to cache initialization (Table 6 in Appendix A.1 of the revised paper), we vary the hand-presence threshold used to select the caching clip from 0.1 to 0.9. MPJPE-p varies by at most 0.06 mm across these thresholds, indicating low sensitivity to the selection threshold on the evaluated datasets. We also test sensitivity to shape mismatch by replacing the default cached shape with another participant's shape. This increases MPJPE-p by 0.83 mm on HOT3D and 2.12 mm on ARCTIC, while using the ground-truth shape reduces it to 12.75 mm on HOT3D and 13.15 mm on ARCTIC. These results support the use of an operator-specific shape, which can be estimated from the video or supplied directly through one-time calibration in robot data collection.

Baseline Control (W2, Q2). To distinguish the contribution of the model from that of the training data, we retrained the strongest pose baseline, ACE-Ego-Hand, on the same training mixture as RLHND and evaluated both models on identical dataset splits. RLHND achieves lower pose error on all three test sets, as shown below and in Table 14 (Appendix B.9), so the gain cannot be attributed solely to training data.

MPJPE-p (mm, lower is better)HOT3DARCTIC-egoEgoDex
ACE-Ego-Hand, re-trained on our data14.8716.3321.74
RLHND13.0113.3619.90

For tactile estimation, the strongest baseline, HOPE, is already trained on the same force data as RLHND, so the force comparison in Table 2 uses matched training data. For contact, we re-trained HOPE with the additional contact datasets used by RLHND (HOT3D, HO3D, HRDexDB, and OakInk2), as shown in the table below.

Contact F1 / AUROC (higher is better)OpenTouchDexYCBHOT3DARCTIC
HOPE (official)0.663 / 0.8940.506 / 0.8680.197 / 0.7620.591 / 0.914
HOPE (re-trained)0.640 / 0.8890.330 / 0.9130.369 / 0.9600.374 / 0.926
RLHND0.696 / 0.9800.572 / 0.9150.589 / 0.9590.602 / 0.935

Component Ablation (Q3). To assess whether the proposed components remain effective across backbones, we extended the pose ablations in Table 5 of the revised paper to Wan2.2, the backbone used by ACE-Ego-Hand, and Cosmos 3 Edge, in addition to Cosmos 3 Nano. For each backbone, we train the full model and two ablated variants using the same training data: one without the anatomically constrained pose parameterization (A1, B1, C1) and one without external shape conditioning and caching (A2, B2, C2). On every backbone, the full model gives the lowest EgoDex MPJPE-p, and removing shape conditioning and caching increases ARCTIC MPJPE-p by 0.4-1.5 mm. Removing the constrained pose parameterization increases EgoDex MPJPE-p by 0.8-2.2 mm on Wan2.2 and Edge, while its effect is negligible on Nano. The full model also achieves the lowest ARCTIC EPE2D-p among the variants without ground-truth shape. The backbone accounts for a large share of the gain: with the Wan2.2 backbone, RLHND without shape input is on par with re-trained ACE-Ego-Hand (16.62 vs. 16.33 mm on ARCTIC). Shape conditioning, however, provides a capability that ACE-Ego-Hand lacks. On Nano, the ground-truth shape (C4) mainly improves retargeting (Q-err 10.12 → 9.69), while on Wan2.2 and Edge it lowers ARCTIC MPJPE-p by 1.1-1.3 mm (A4, B4). On Wan2.2, this lets RLHND surpass re-trained ACE-Ego-Hand (15.55 vs. 16.33 mm), which cannot use a calibrated shape.

For tactile estimation, Table 5 separately ablates the backbone (T0), LBS-based feature spreading (T1), and contact-only training data (T2). The tactile stream does not affect pose predictions, as it is trained with the pose stream frozen and its outputs do not feed back into pose estimation.

Reviewer_E37eOpenReview ↗

Concerns resolved

Thanks for the response! My main concerns are well addressed.

AuthorsOpenReview ↗

Response to Reviewer E37e (Thank you)

Thank you for the follow-up and for the constructive feedback throughout the review. We are glad that our responses addressed your main concerns. The clarifications and new results from this discussion, including the comparison under matched training data and the extended ablations, are included in the revised paper, and we will keep refining it for the final version. Please let us know if any further questions arise.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (to all reviewers)

We thank all reviewers for their careful reading and constructive feedback. We are delighted that the reviewers found the work to address an important and practically relevant problem, turning egocentric human videos into hand-pose and contact/force supervision for robot learning (E37e, E7AX, 5L8U). The reviewers also found the joint estimation of hand pose and tactile signals interesting (E7AX), noting that predicting contact and force rather than pose alone goes beyond traditional hand trackers and makes human videos more useful for robot learning (5L8U), and that building on video foundation model priors is an interesting and valuable direction for consistent video-level reconstruction (5L8U). We are also glad that the evaluation was found broad, spanning hand reconstruction, tactile estimation, retargeting to multiple dexterous hands and real-robot policy learning (E37e, 5L8U), and that the performance was found good and supported by demos (E7AX).

We also note that the appendix was inadvertently included in the supplementary material rather than appended to the main submission PDF. The revised PDF now includes it, and we would be grateful if the reviewers could kindly consult it, since several of the points raised are addressed there.

Summary of revisions. The main changes in the revised PDF are:

  • Human-video-to-robot comparison (Section 4.3, Table 4): VITRA trained on the same episodes with HaWoR and RLHND labels (5L8U).
  • Ablations across backbones (Section 4.4, Table 5): pose ablations extended from Cosmos 3 Nano to Wan2.2 and Cosmos 3 Edge (E37e, E7AX, 5L8U).
  • Pose baseline under matched training data (Appendix B.9, Table 14): ACE-Ego-Hand re-trained on our training mixture (E37e, E7AX).
  • Limitations and failure cases (Section 5, Figure 5): limitations moved to the main text, with failure cases under occlusion and with multiple people in view (E37e, E7AX, 5L8U).
  • Shape caching (Section 4.1, Appendix A.1, Table 6): corrected description of cache initialization, the visibility threshold Tv = 0.5, and a sensitivity analysis of the cached shape (E37e, 5L8U).
  • Physics replay (Appendix B.2, Table 9, Figure 14): MuJoCo replay on ARCTIC evaluating estimated hand motion and contact (5L8U).
  • Gloved hands (Appendix B.6, Figure 16): evaluation on gloved and bare hands of the same task (5L8U).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.