- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RLHND, a video foundation model based hand tracking framework that jointly estimates 3D hand motion, contact, and force information from monocular egocentric videos. The method uses Cosmos 3 as a video feature extractor and introduces shape caching, anatomical pose constraints, and a decoupled tactile expert. Experiments on hand reconstruction, tactile estimation, and robot learning benchmarks demonstrate the effectiveness of RLHND.
Critical strengths
-
The paper proposes a physically grounded hand tracking framework for robot learning. Different from traditional methods that only estimate hand pose, RLHND jointly predicts hand motion, contact, and force information, making human videos more useful for robot learning.
-
Using video foundation model priors is an interesting and valuable direction. The paper leverages the motion and hand-object interaction priors learned by Cosmos 3 to improve video-level hand reconstruction consistency, instead of relying only on independent frame-level predictions.
-
Besides hand reconstruction and tactile estimation, the paper demonstrates retargeting to multiple dexterous hands and robot policy training, which improves the practical significance of the work.
Critical weaknesses
-
RLHND heavily relies on the Cosmos 3 backbone and builds additional designs on top of it. The main contributions, including shape caching, anatomical constraints, and tactile expert design, are useful but appear more like engineering improvements. More analysis is needed to show their independent contributions compared with previous approaches.
-
The paper claims that video foundation models provide strong priors for hand motion and interaction. However, it is unclear how much of the improvement comes from Cosmos 3 pre-training and how much comes from the proposed RLHND designs. Comparisons with other backbones or smaller video encoders would help clarify the main source of the improvements.
-
RLHND predicts contact and force from visual input, but physical interaction signals are strongly related to real contact dynamics. Current experiments mainly evaluate prediction accuracy, while lacking analysis of whether the predicted tactile information can reflect real interaction states, such as grasp stability or failure detection.
-
The paper demonstrates retargeting and Dexterous Point Policy training, but these experiments mainly show that RLHND outputs are useful. More experiments are needed to understand whether RLHND directly improves robot learning performance, such as evaluations on more tasks, more environments, or comparisons with other human-video-to-robot learning approaches.
Questions for authors
-
How much of the performance improvement comes from the Cosmos 3 backbone? If another video foundation model or a standard video encoder is used, do the proposed designs, such as shape caching and anatomical pose parameterization, remain effective?
-
Can the predicted tactile information (contact and force) improve robot manipulation performance in downstream tasks? More robot experiments based on RLHND tactile predictions would strengthen the claim.
-
Shape caching reduces hand-size fluctuation, but how does it perform when the hand undergoes significant appearance changes, such as wearing gloves, large viewpoint changes, or different operators?
-
The current robot experiments mainly focus on retargeting and policy training. Can the authors provide direct comparisons with other human-video-to-robot learning methods?
5 comments
Response to Reviewer 5L8U (2/2)
Real-Robot Contact Ablation (W3, Q2, continued). On a real robot, in a setup separate from Table 4, we followed the Dexterous Point Policy (DPP) recipe. We used RLHND to annotate contact on approximately 1M egocentric human video episodes and pre-trained a bimanual policy for two 20-DoF hands to predict keypoints and per-keypoint contact intensities. At deployment, the predicted intensity determines the magnitude of a keypoint target offset along the finger-closing direction.
We evaluated pick-and-place with four objects and 64 trials per variant, fine-tuning on 10% of the teleoperated demonstrations. As shown below, removing RLHND contact labels, using binary contact, or disabling the contact-driven offsets all reduce success, as objects often slipped during lifting without the offsets.
Together, the simulation and real-robot results show that RLHND contact predictions provide actionable information for maintaining grasps beyond hand pose alone. These experiments assess the utility of predicted contact for grasp maintenance rather than directly measuring physical force.
| Real robot, pick-and-place (64 trials) | Success rate (%) |
|---|---|
| w/o RLHND contact labels in pre-training | 43.8 |
| Binary instead of graded contact | 37.5 |
| w/o contact-driven offset at deployment | 25.0 |
| Full (graded RLHND contact + contact-driven offset) | 56.3 |
Comparison with Human-Video-to-Robot Methods (W4, Q4). We additionally evaluate VITRA, a vision-language-action model that learns dexterous manipulation from egocentric human videos annotated using HaWoR. We trained two VITRA variants on the same human and robot episodes used in Table 4: one using the original HaWoR annotations and the other using RLHND annotations. The architecture, initialization, robot data, and training schedule were identical. Both variants were evaluated using the same robot setup (RB-Y1 with two 20-DoF hands), objects, and number of trials.
Replacing HaWoR annotations with RLHND annotations increases VITRA's overall success rate from 73.4% to 80.7%, with improvements on both pick-and-place tasks (82.8% to 89.1%) and bimanual tasks (54.7% to 64.1%). Table 4 of the revised paper and the table below report these results alongside DPP results under the same evaluation protocol. These results demonstrate that RLHND annotations improve two human-video-to-robot learning methods with different policy architectures and action spaces.
In the physics replay of W3, which involves no policy learning, RLHND hand motion without contact offsets achieves 28.0% lift success, compared with 22.5% for ACE-Ego-Hand and 9.1% for HaWoR.
| Method (real robot, same human and robot episodes) | Pick & Place avg | Bimanual avg | All |
|---|---|---|---|
| VITRA (HaWoR labels) | 82.8 | 54.7 | 73.4 |
| VITRA + RLHND | 89.1 | 64.1 | 80.7 |
| DPP (HaWoR keypoints) | 89.1 | 64.1 | 80.7 |
| DPP + RLHND | 93.8 | 75.0 | 87.5 |
Shape Caching under Appearance Changes (Q3). RLHND targets single-worker recordings, consistent with the tracking setup of egocentric hand datasets such as EgoDex and HOT3D. Operator changes within a video fall outside this setting, as stated in the limitations of the revised paper (Section 5).
Within this setting, appearance changes that persist throughout the sequence do not affect tracking stability. As shown in the new Appendix B.6 (Figure 16), we evaluated RLHND on 30 egocentric episodes recorded with white tactile gloves and 30 bare-hand episodes of the same packing task, captured on the same day with the same rig. Articulation accuracy is unchanged (PA-MPJPE 8.0 vs. 8.1 mm), with detection rates of 99.0% and 99.1% on frames with a visible hand. For viewpoint changes, exchanging cached shapes between the simultaneous egocentric and exocentric ARCTIC cameras changes MPJPE-p by at most 0.45 mm (table below), well below the 2.12 mm increase caused by another operator's shape (Table 6 in Appendix A.1 of the revised paper).
| ARCTIC MPJPE-p (mm, lower is better) | Cache from the same view | Cache from another view | Change |
|---|---|---|---|
| Egocentric view (cache from exocentric views) | 13.44 | 13.15 | -0.29 |
| Exocentric landscape views (cache from egocentric view) | 16.15 | 16.57 | +0.42 |
| Exocentric portrait views (cache from egocentric view) | 17.59 | 18.04 | +0.45 |
The remaining case is a change in hand appearance within a video, e.g., putting on gloves midway. Since shape caching is optional and per-clip shape estimation achieves comparable accuracy (within 0.08 mm MPJPE-p across benchmarks, Table 1), such videos can be processed with per-clip shape estimation, which adapts the shape to each clip without degrading accuracy.
Response to Reviewer 5L8U (Follow-up)
Dear Reviewer 5L8U,
Thank you again for your review. As the discussion period ends soon, we wanted to briefly check whether our response has addressed your concerns. In summary, we added (1) backbone ablations on Wan2.2 and Cosmos 3 Edge showing that shape conditioning/caching and the anatomical pose parameterization remain effective across backbones, (2) simulation and real-robot experiments showing that predicted contact improves grasp maintenance, (3) a direct comparison with VITRA, where RLHND labels improve success from 73.4% to 80.7%, (4) analyses of gloves and viewpoint changes for shape caching, and (5) a backbone-matched comparison (posted in our reply to Reviewer E7AX): with the same Cosmos 3 Nano backbone and training data, RLHND still lowers MPJPE-p over ACE-Ego-Hand on all benchmarks (HOT3D 13.25 → 13.01, ARCTIC 14.59 → 13.36, EgoDex 20.49 → 19.90 mm), isolating the contribution of our design choices beyond the backbone. If any concerns remain, we would be glad to clarify them before the deadline.
Best regards, The Authors
Response to Reviewer 5L8U (1/2)
We thank the reviewer for recognizing the value of jointly predicting hand motion, contact, and force, and the potential of video foundation model priors for hand tracking.
Distinct contributions of RLHND (W1). RLHND introduces several task-specific designs for adapting video foundation models to hand tracking and tactile estimation. First, it uses the native clean-latent conditioning interface of Cosmos 3, rather than passing clean frames through the denoising interface at σ=0 as in prior video-diffusion trackers. Second, external shape conditioning and caching enforce a consistent hand shape, while the anatomically constrained pose parameterization restricts predictions to anatomically plausible joint configurations. Third, the tactile expert uses MANO skinning weights to map token features to mesh vertices, introducing an anatomical correspondence for dense contact and force prediction. Replacing this mapping with a learnable matrix reduces contact F1 on all three benchmarks (T1 in Table 5).
Contributions of the backbone and proposed designs (W2, Q1). We extended the pose ablations in Table 5 of the revised paper from Cosmos 3 Nano to two additional backbones: Wan2.2 (used by ACE-Ego-Hand) and Cosmos 3 Edge. On all three backbones, ARCTIC MPJPE-p rises by 0.4-1.5 mm without external shape conditioning and caching (A2, B2, C2), and on Wan2.2 and Edge, EgoDex MPJPE-p rises by 0.8-2.2 mm without the anatomically constrained pose parameterization (A1, B1). These results also clarify what the backbone contributes. With Wan2.2 in place of Cosmos 3, RLHND without shape input is roughly level with ACE-Ego-Hand re-trained on our training mixture (ARCTIC 16.62 vs. 16.33 mm, EgoDex 21.57 vs. 21.74 mm), so the backbone is responsible for much of the accuracy. What the backbone cannot supply is a way to use a known hand shape, and this is where shape conditioning is more than an engineering refinement. Feeding the operator's ground-truth β lowers ARCTIC MPJPE-p by 1.1 mm on Wan2.2 and 1.3 mm on Edge (A4, B4), reaching 15.55 mm on Wan2.2, below re-trained ACE-Ego-Hand (16.33 mm), whose hand tokens start from fixed learnable parameters and have no input for such a measurement. On Cosmos 3 Nano, which already recovers hand shape well from video, the gain is 0.2 mm and appears mainly in retargeting (Q-err 10.12 → 9.69). A one-time hand calibration during robot data collection can therefore be turned into more accurate labels, particularly when the visual prior is weaker. For tactile estimation, the corresponding ablations show that both the Cosmos 3 backbone (T0) and LBS-based feature spreading (T1) improve contact estimation (Table 5).
Tactile Signals and Real Interaction (W3, Q2). We added a simulation experiment that holds hand motion fixed and varies only the contact-based control. We replayed all 55 segments of the ARCTIC egocentric validation split in which an object is lifted at least 3 cm from the table without articulation. Hand keypoints are mapped to world coordinates with the ground-truth camera and retargeted to two floating Shadow Hands in MuJoCo with DexPilot. The object is initialized at its ground-truth pose on the table and subsequently moves only through physical contact with the hands.
Following DPP, we add a 0.3 rad closing offset to the flexion targets of each finger predicted to be in contact (at least five MANO vertices above 0.8). A trial succeeds if the object reaches at least half its ground-truth height above the table in at least 80% of the lifted frames. We report mean success over five runs with randomized object mass and friction.
With ground-truth hand motion, RLHND contact increases success from 37.1% to 58.2%, compared with 62.2% using ground-truth contact. Applying the same closing offset to all fingers in every frame achieves only 45.8%, supporting the value of selecting when and where to apply the offset using predicted contact. With RLHND-estimated hand motion, predicted contact increases success from 28.0% to 32.7%, while slightly increasing the object pose error.
Details and a rollout example are in the new Appendix B.2 (Table 9 and Figure 14).