- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MetricHand, aiming to reconstruct four-dimensional hand motion with real metric scale from first-person videos. The method leverages a pretrained monocular depth model to provide geometric priors, associates RGB-D features with hand joints through topology-aware sampling, and completes hand motion reconstruction using quality-aware conditioning, temporal memory queues, and Flow Matching. Experiments on datasets such as HOT3D and ARCTIC report results superior to several existing methods.
Critical strengths
-
This paper focuses on four-dimensional hand reconstruction from first-person videos, particularly the recovery of hand motion in the world coordinate system with real metric scale. This task holds potential applications for human manipulation data collection, dexterous manipulation learning, and the training of Vision-Language-Action (VLA) models.
-
This paper utilizes a pretrained monocular depth model to provide geometric priors and associates local RGB-D features with hand joints through topology-aware sampling, offering a relatively clear geometric modeling approach for hand pose estimation and spatial localization.
-
By combining quality-aware conditioning, cross-window temporal memory, and Flow Matching, this paper attempts to leverage historical information to improve hand motion reconstruction under occlusion or unreliable local observations, rather than relying solely on per-frame predictions.
-
The paper evaluates reconstruction performance in both camera coordinate and world coordinate systems on HOT3D and ARCTIC, while also examining wrist localization error and motion acceleration error, providing a comprehensive presentation of the method's performance in terms of pose estimation, spatial localization, and temporal consistency.
Critical weaknesses
-
MetricHand uses a frozen monocular geometry model to predict metric depth, which is then employed for coarse hand estimation, world-coordinate localization, and the construction of geometric conditions in the subsequent generation process. In contrast, it has not been demonstrated that the multiple comparison methods in Table 1 use equivalent depth inputs, geometric features, or the same pretrained depth model. Therefore, the current comparison is not conducted under fully identical information conditions. The performance improvement of MetricHand may partially come from the geometric information provided by the additional input, rather than entirely from its proposed designs such as topology-aware sampling, temporal memory, or Flow Matching.
-
Does this paper actually enhance the model's own ability to recover three-dimensional hand geometry from video, or does it rely on an external depth model to pre-provide a substantial portion of the critical geometric information, which is then used to improve the final reconstruction results?
-
The paper reports ablation results for modules including depth representation, topology-aware sampling, Flow Matching, quality-aware conditioning, and temporal memory. However, the current experiments mainly compare performance after gradually adding or removing modules, making it still difficult to completely rule out interaction effects among different modules, as well as the impact of differences in model capacity and input information.
-
The paper reports substantial improvements in world-coordinate localization error on HOT3D and ARCTIC, while simultaneously acknowledging that errors from camera trajectory and monocular depth estimation propagate to the final reconstruction results. Taking ARCTIC as an example, the reported camera-space MPJPE is 17.54 mm, whereas the absolute world-coordinate MPJPE is 120.86 mm, revealing a significant gap between the two.
Questions for authors
- How does the proposed method compare with existing methods when all methods are given the same depth input?
- How much of the performance gain comes from the proposed architecture, rather than the pretrained depth model?
- Does the method remain effective when the estimated depth is inaccurate or noisy?
7 comments
Rebuttal for Q2
Q2: How much of the performance gain comes from the proposed architecture, rather than the pretrained depth model?
A2: Our core contribution is the construction and sustained use of structure-aware 3D HOI geometry, rather than depth prediction itself. The pretrained depth model provides useful geometric information, while controlled ablations show that our architecture achieves further gains beyond generic depth fusion by constructing a topology-aware 3D HOI representation and continuously using it throughout reconstruction.
Specifically, we aggregate local RGB–D information around projected anatomical joint anchors and construct geometric attention biases using image distance, log-depth differences, and backprojected surface distance. This enables the representation to preserve both hand topology and spatial relationships with nearby interaction surfaces. The Metric Head uses fused features and absolute-depth statistics to refine translation in viewing-ray and log-depth coordinates. Quality-aware conditioning and cross-window memory preserve reliable, joint-associated geometric evidence throughout flow generation. The core contribution is the construction and sustained use of structured 3D HOI geometry, rather than depth prediction itself.
Table 2 of the paper quantifies these gains through incremental controls, with the first four configurations sharing the same Metric Head. On ARCTIC, generic RGB–D cross-attention reduces MPJPE from 36.34 to 29.64 mm. With the same depth input and regression head, topology-aware HOI aggregation further reduces it to 19.98 mm, a 32.6% reduction relative to cross-attention. Subsequently, replacing direct regression with flow matching while retaining the geometric conditions, memory queue, and window length further reduces MPJPE to 17.54 mm, an additional 12.2% reduction.
Table 3 of the paper also verifies the importance of continuously providing geometric information during generation: using depth only for the coarse estimate or providing geometric conditions only at the first flow step yields ARCTIC MPJPE of 38.95 and 28.95 mm, respectively, compared with 17.54 mm for the full method. Thus, the depth prior alone is insufficient to achieve the full method's accuracy. Organizing it into joint-associated interaction geometry and using it throughout motion generation are important sources of the improvement.
Table R2. Depth representation ablation on ARCTIC
| Configuration | MPJPE ↓ | PA-MPJPE ↓ |
|---|---|---|
| RGB-only regression | 36.34 | 6.78 |
| RGB–D regression: concatenation | 31.65 | 6.26 |
| RGB–D regression: cross-attention | 29.64 | 6.01 |
| RGB–D regression: topology-aware | 19.98 | 5.79 |
| Topology-aware conditions + flow | 17.54 | 5.18 |
Errors are in mm. The first four configurations share the same Metric Head.
Rebuttal for Q3
Q3: Does the method remain effective when the estimated depth is inaccurate or noisy?
A3: Average reconstruction errors change little under the tested depth-noise conditions; in contrast, directly removing informative observations leads to more pronounced errors.
We keep the model weights, runtime environment, and all other settings unchanged, perturb only the predicted depth, and rerun inference without retraining. Here, 10% and 20% denote the relative standard deviation of mean-preserving multiplicative noise.
As shown in Table R3, average reconstruction errors under noisy depth remain close to those obtained with the original predicted depth. At 20% noise, MPJPE changes by +0.108 mm on HOT3D and +0.019 mm on ARCTIC. These results indicate that average camera-space reconstruction errors have low sensitivity to depth noise under the tested perturbations.
Table R3. Error changes under depth noise
| Depth input | HOT3D MPJPE | HOT3D RA-MPJPE | ARCTIC MPJPE | ARCTIC RA-MPJPE |
|---|---|---|---|---|
| Original predicted depth | 8.60 | 5.84 | 17.54 | 11.28 |
| 10% noise (Δ) | +0.045 | +0.001 | +0.014 | +0.003 |
| 20% noise (Δ) | +0.108 | +0.002 | +0.019 | +0.015 |
Errors are in mm.
In addition, we separately test joint masking of RGB and depth to assess performance when observations are missing. As shown in Table R4, masking generally increases reconstruction errors, with larger changes in MPJPE than in PA-MPJPE, indicating a more pronounced effect on absolute hand placement. We will include successful and failure cases in the appendix to further illustrate both reasonable reconstructions under partial occlusion and errors caused by insufficient remaining information. The appendix also provides additional visualizations; see Figures 5–10.
Thus, tolerance to depth noise is not equivalent to robustness to missing observations: the remaining geometric information can still be used under moderate perturbations, whereas missing key visual and geometric evidence can still lead to errors in absolute position and articulation.
Table R4. Error changes under joint RGB and depth masking
| Dataset | Occlusion condition | Δ MPJPE | Δ RA-MPJPE | Δ PA-MPJPE |
|---|---|---|---|---|
| HOT3D | Mask 30% | 4.85 | 1.651 | 0.433 |
| HOT3D | Mask 50% | 4.96 | 2.789 | 0.932 |
| ARCTIC | Mask 30% | 1.132 | 0.27 | 0.291 |
| ARCTIC | Mask 50% | 2.487 | 0.49 | 0.835 |
Errors are in mm. Δ denotes the error with masked inputs minus that with the original inputs; positive values indicate increased error. The 30% / 50% levels refer to coverage of the projected hand region accumulated over the clip, rather than the area of the entire image. This table reports a controlled masking diagnostic on a subset using the archived model.
Under the evaluation protocols in the paper, MetricHand reduces absolute camera-space MPJPE by 49.3%, 51.6%, and 63.2%, respectively, compared with the second-ranked methods on HOT3D and ARCTIC, and on the internal HOI4D held-out split evaluated in a zero-shot setting (Tables 1 and 8). These results demonstrate the effectiveness of the method. Together with the ablation studies, they also indicate that a structured 3D hand–object interaction (HOI) representation and its continued use throughout motion generation benefit 4D hand reconstruction, providing a reference for future research.
Thank you again for your time and constructive feedback. We hope these clarifications, matched-depth comparisons, and robustness analyses address your questions and better explain the role of the structured 3D HOI representation in reconstruction. If these explanations and results resolve your concerns, we would sincerely appreciate your reconsideration of our paper and, if appropriate, an increase in your score. Thank you very much for your time and consideration.
Rebuttal for Q1
Thank you for your careful review and constructive questions about fair comparisons under the same depth input, the contribution of our architecture, and robustness to depth noise. Your feedback helps us clarify our method's contribution and its experimental support.
We address your questions individually below.
Q1: How does the proposed method compare with existing methods when all methods are given the same depth input?
A1: MetricHand goes beyond introducing depth as an additional modality: it organizes depth and RGB information into a metric-scale 3D hand–object interaction (HOI) representation that continuously provides joint-associated geometric evidence throughout reconstruction. Even when given the same predicted depth, MetricHand retains an advantage over baselines that incorporate depth through generic feature fusion.
We select WiLoR and HaWoR as baselines and augment each with an RGB–D cross-attention module. These variants use depth predictions from the same frozen MoGe-3 model and adopt the same depth encoding as MetricHand: following Sec. 3.2 and Appendix E of the paper, normalized log-depth, validity masks, and spatial gradients are encoded into depth features. RGB and depth features are fused through cross-attention before MANO parameter regression, while the rest of each baseline architecture remains unchanged.
As shown in Table R1, adding depth improves both baselines but does not close the gap to MetricHand. Compared with HaWoR + RGB–D cross-attention, MetricHand reduces MPJPE from 14.68 to 8.60 mm on HOT3D and from 30.12 to 17.54 mm on ARCTIC, corresponding to reductions of 41.4% and 41.8%, respectively. Root-relative errors also decrease further on both datasets.
These results indicate that the gains arise not only from the additional depth input but also from how depth and appearance information are organized into a geometric representation for reconstruction. The controlled ablations in Table 2 of the paper further demonstrate the benefits of the topology-aware HOI representation over generic RGB–D fusion, as detailed in our response to the next question.
Table R1. Comparison with depth-augmented baselines
Errors are in mm. The original-configuration rows are included for reference; the matched-depth comparison is between the two RGB–D-augmented baselines and MetricHand.