acceptodds
Under review as a conference paper at ICLR 2027

MetricHand: Geometry-Aware Flow Matching for 4D Hand Reconstruction in Metric World Space

18 views

Abstract

Egocentric hand motion provides physical motion supervision for dexterous manipulation and vision-language-action pretraining. Reliable use of this supervision requires accurate and temporally coherent 4D hand reconstruction in metric world space. Existing methods typically lift framewise estimates produced under weak perspective into world coordinates. By ignoring joint-dependent depth variation at close range, this approximation couples errors in translation, orientation, and articulation that persist after world lifting and worsen under occlusion. We present MetricHand, a geometry-aware flow matching framework for 4D hand reconstruction in metric world space. MetricHand encodes depth priors from hand–object interaction regions into joint-associated representations that preserve local spatial structure and metric scale. Fused with visual features, this geometry guides coarse metric estimation and conditions every step of temporal generation. The model jointly recovers world-space translation, global orientation, MANO articulation, and shape, while structural priors and temporal context support estimation under partial occlusion. MetricHand reduces camera-space MPJPE by 49.3% and 52.9%, absolute world-space MPJPE by 28.7% and 21.7%, and wrist localization error by 48.5% and 37.9% relative to HaWoR, respectively. World-space acceleration error decreases by 44.2% and 64.3%, demonstrating improved temporal fidelity alongside more accurate absolute localization. Our project code is available at https://anonymous.4open.science/api/repo/metrichand-review-C230/file/index.html.

Primary area:
generative models
Keywords:
4D Hand Reconstruction, Geometry Priors, 3D Representations, Egocentric Videos

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1hYAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MetricHand, aiming to reconstruct four-dimensional hand motion with real metric scale from first-person videos. The method leverages a pretrained monocular depth model to provide geometric priors, associates RGB-D features with hand joints through topology-aware sampling, and completes hand motion reconstruction using quality-aware conditioning, temporal memory queues, and Flow Matching. Experiments on datasets such as HOT3D and ARCTIC report results superior to several existing methods.

Critical strengths
  1. This paper focuses on four-dimensional hand reconstruction from first-person videos, particularly the recovery of hand motion in the world coordinate system with real metric scale. This task holds potential applications for human manipulation data collection, dexterous manipulation learning, and the training of Vision-Language-Action (VLA) models.

  2. This paper utilizes a pretrained monocular depth model to provide geometric priors and associates local RGB-D features with hand joints through topology-aware sampling, offering a relatively clear geometric modeling approach for hand pose estimation and spatial localization.

  3. By combining quality-aware conditioning, cross-window temporal memory, and Flow Matching, this paper attempts to leverage historical information to improve hand motion reconstruction under occlusion or unreliable local observations, rather than relying solely on per-frame predictions.

  4. The paper evaluates reconstruction performance in both camera coordinate and world coordinate systems on HOT3D and ARCTIC, while also examining wrist localization error and motion acceleration error, providing a comprehensive presentation of the method's performance in terms of pose estimation, spatial localization, and temporal consistency.

Critical weaknesses
  1. MetricHand uses a frozen monocular geometry model to predict metric depth, which is then employed for coarse hand estimation, world-coordinate localization, and the construction of geometric conditions in the subsequent generation process. In contrast, it has not been demonstrated that the multiple comparison methods in Table 1 use equivalent depth inputs, geometric features, or the same pretrained depth model. Therefore, the current comparison is not conducted under fully identical information conditions. The performance improvement of MetricHand may partially come from the geometric information provided by the additional input, rather than entirely from its proposed designs such as topology-aware sampling, temporal memory, or Flow Matching.

  2. Does this paper actually enhance the model's own ability to recover three-dimensional hand geometry from video, or does it rely on an external depth model to pre-provide a substantial portion of the critical geometric information, which is then used to improve the final reconstruction results?

  3. The paper reports ablation results for modules including depth representation, topology-aware sampling, Flow Matching, quality-aware conditioning, and temporal memory. However, the current experiments mainly compare performance after gradually adding or removing modules, making it still difficult to completely rule out interaction effects among different modules, as well as the impact of differences in model capacity and input information.

  4. The paper reports substantial improvements in world-coordinate localization error on HOT3D and ARCTIC, while simultaneously acknowledging that errors from camera trajectory and monocular depth estimation propagate to the final reconstruction results. Taking ARCTIC as an example, the reported camera-space MPJPE is 17.54 mm, whereas the absolute world-coordinate MPJPE is 120.86 mm, revealing a significant gap between the two.

Questions for authors
  1. How does the proposed method compare with existing methods when all methods are given the same depth input?
  2. How much of the performance gain comes from the proposed architecture, rather than the pretrained depth model?
  3. Does the method remain effective when the estimated depth is inaccurate or noisy?
7 comments
AuthorsOpenReview ↗

Rebuttal for Q1

Thank you for your careful review and constructive questions about fair comparisons under the same depth input, the contribution of our architecture, and robustness to depth noise. Your feedback helps us clarify our method's contribution and its experimental support.

We address your questions individually below.

Q1: How does the proposed method compare with existing methods when all methods are given the same depth input?

A1: MetricHand goes beyond introducing depth as an additional modality: it organizes depth and RGB information into a metric-scale 3D hand–object interaction (HOI) representation that continuously provides joint-associated geometric evidence throughout reconstruction. Even when given the same predicted depth, MetricHand retains an advantage over baselines that incorporate depth through generic feature fusion.

We select WiLoR and HaWoR as baselines and augment each with an RGB–D cross-attention module. These variants use depth predictions from the same frozen MoGe-3 model and adopt the same depth encoding as MetricHand: following Sec. 3.2 and Appendix E of the paper, normalized log-depth, validity masks, and spatial gradients are encoded into depth features. RGB and depth features are fused through cross-attention before MANO parameter regression, while the rest of each baseline architecture remains unchanged.

As shown in Table R1, adding depth improves both baselines but does not close the gap to MetricHand. Compared with HaWoR + RGB–D cross-attention, MetricHand reduces MPJPE from 14.68 to 8.60 mm on HOT3D and from 30.12 to 17.54 mm on ARCTIC, corresponding to reductions of 41.4% and 41.8%, respectively. Root-relative errors also decrease further on both datasets.

These results indicate that the gains arise not only from the additional depth input but also from how depth and appearance information are organized into a geometric representation for reconstruction. The controlled ablations in Table 2 of the paper further demonstrate the benefits of the topology-aware HOI representation over generic RGB–D fusion, as detailed in our response to the next question.

Table R1. Comparison with depth-augmented baselines
MethodHOT3D MPJPE ↓HOT3D RA-MPJPE ↓ARCTIC MPJPE ↓ARCTIC RA-MPJPE ↓
WiLoR-SLAM53.7816.5836.2314.23
WiLoR + RGB–D cross-attention47.3915.1933.9513.96
HaWoR16.9712.6537.2622.45
HaWoR + RGB–D cross-attention14.6811.5430.1219.05
MetricHand8.65.8417.5411.28

Errors are in mm. The original-configuration rows are included for reference; the matched-depth comparison is between the two RGB–D-augmented baselines and MetricHand.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2: How much of the performance gain comes from the proposed architecture, rather than the pretrained depth model?

A2: Our core contribution is the construction and sustained use of structure-aware 3D HOI geometry, rather than depth prediction itself. The pretrained depth model provides useful geometric information, while controlled ablations show that our architecture achieves further gains beyond generic depth fusion by constructing a topology-aware 3D HOI representation and continuously using it throughout reconstruction.

Specifically, we aggregate local RGB–D information around projected anatomical joint anchors and construct geometric attention biases using image distance, log-depth differences, and backprojected surface distance. This enables the representation to preserve both hand topology and spatial relationships with nearby interaction surfaces. The Metric Head uses fused features and absolute-depth statistics to refine translation in viewing-ray and log-depth coordinates. Quality-aware conditioning and cross-window memory preserve reliable, joint-associated geometric evidence throughout flow generation. The core contribution is the construction and sustained use of structured 3D HOI geometry, rather than depth prediction itself.

Table 2 of the paper quantifies these gains through incremental controls, with the first four configurations sharing the same Metric Head. On ARCTIC, generic RGB–D cross-attention reduces MPJPE from 36.34 to 29.64 mm. With the same depth input and regression head, topology-aware HOI aggregation further reduces it to 19.98 mm, a 32.6% reduction relative to cross-attention. Subsequently, replacing direct regression with flow matching while retaining the geometric conditions, memory queue, and window length further reduces MPJPE to 17.54 mm, an additional 12.2% reduction.

Table 3 of the paper also verifies the importance of continuously providing geometric information during generation: using depth only for the coarse estimate or providing geometric conditions only at the first flow step yields ARCTIC MPJPE of 38.95 and 28.95 mm, respectively, compared with 17.54 mm for the full method. Thus, the depth prior alone is insufficient to achieve the full method's accuracy. Organizing it into joint-associated interaction geometry and using it throughout motion generation are important sources of the improvement.

Table R2. Depth representation ablation on ARCTIC
ConfigurationMPJPE ↓PA-MPJPE ↓
RGB-only regression36.346.78
RGB–D regression: concatenation31.656.26
RGB–D regression: cross-attention29.646.01
RGB–D regression: topology-aware19.985.79
Topology-aware conditions + flow17.545.18

Errors are in mm. The first four configurations share the same Metric Head.

AuthorsOpenReview ↗

Rebuttal for Q3

Q3: Does the method remain effective when the estimated depth is inaccurate or noisy?

A3: Average reconstruction errors change little under the tested depth-noise conditions; in contrast, directly removing informative observations leads to more pronounced errors.

We keep the model weights, runtime environment, and all other settings unchanged, perturb only the predicted depth, and rerun inference without retraining. Here, 10% and 20% denote the relative standard deviation of mean-preserving multiplicative noise.

As shown in Table R3, average reconstruction errors under noisy depth remain close to those obtained with the original predicted depth. At 20% noise, MPJPE changes by +0.108 mm on HOT3D and +0.019 mm on ARCTIC. These results indicate that average camera-space reconstruction errors have low sensitivity to depth noise under the tested perturbations.

Table R3. Error changes under depth noise
Depth inputHOT3D MPJPEHOT3D RA-MPJPEARCTIC MPJPEARCTIC RA-MPJPE
Original predicted depth8.605.8417.5411.28
10% noise (Δ)+0.045+0.001+0.014+0.003
20% noise (Δ)+0.108+0.002+0.019+0.015

Errors are in mm.

In addition, we separately test joint masking of RGB and depth to assess performance when observations are missing. As shown in Table R4, masking generally increases reconstruction errors, with larger changes in MPJPE than in PA-MPJPE, indicating a more pronounced effect on absolute hand placement. We will include successful and failure cases in the appendix to further illustrate both reasonable reconstructions under partial occlusion and errors caused by insufficient remaining information. The appendix also provides additional visualizations; see Figures 5–10.

Thus, tolerance to depth noise is not equivalent to robustness to missing observations: the remaining geometric information can still be used under moderate perturbations, whereas missing key visual and geometric evidence can still lead to errors in absolute position and articulation.

Table R4. Error changes under joint RGB and depth masking
DatasetOcclusion conditionΔ MPJPEΔ RA-MPJPEΔ PA-MPJPE
HOT3DMask 30%4.851.6510.433
HOT3DMask 50%4.962.7890.932
ARCTICMask 30%1.1320.270.291
ARCTICMask 50%2.4870.490.835

Errors are in mm. Δ denotes the error with masked inputs minus that with the original inputs; positive values indicate increased error. The 30% / 50% levels refer to coverage of the projected hand region accumulated over the clip, rather than the area of the entire image. This table reports a controlled masking diagnostic on a subset using the archived model.

Under the evaluation protocols in the paper, MetricHand reduces absolute camera-space MPJPE by 49.3%, 51.6%, and 63.2%, respectively, compared with the second-ranked methods on HOT3D and ARCTIC, and on the internal HOI4D held-out split evaluated in a zero-shot setting (Tables 1 and 8). These results demonstrate the effectiveness of the method. Together with the ablation studies, they also indicate that a structured 3D hand–object interaction (HOI) representation and its continued use throughout motion generation benefit 4D hand reconstruction, providing a reference for future research.

Thank you again for your time and constructive feedback. We hope these clarifications, matched-depth comparisons, and robustness analyses address your questions and better explain the role of the structured 3D HOI representation in reconstruction. If these explanations and results resolve your concerns, we would sincerely appreciate your reconsideration of our paper and, if appropriate, an increase in your score. Thank you very much for your time and consideration.

Reviewer_sKihOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MetricHand, which reconstructs 4D hand motion at metric scale from monocular video and camera intrinsics. The system uses a frozen monocular depth estimation model and DROID-SLAM to provide geometry and camera motion information. RGB-D fusion and a Metric Head produce coarse hand states, which are then transformed into world coordinates. Using projected joints as anchors, the model performs deformable sampling with geometric biases over multi-scale features. Observation-quality weighting and cross-window memory are incorporated to construct joint-associated geometric conditions.

Critical strengths
  1. For robot learning, motion supervision, and spatial interaction, accurate local hand geometry may still be unusable if the wrist is incorrectly positioned. The paper reports MPJPE, WristErr, root-relative errors, similarity-aligned errors, and world-space trajectory metrics, helping distinguish errors associated with pose and camera motion.

  2. The experimental results support improvements in relative shape and motion dynamics. Ablation studies also provide direct evidence for the effectiveness of the geometric conditioning representation.

Critical weaknesses
  1. ABS-W-MPJPE should be interpreted as world-space error under the specified evaluation protocol. It does not fully characterize accumulated drift over an entire long video in a single fixed reference frame.

  2. On ARCTIC, camera-space MPJPE is 17.54 mm, whereas ABS-W-MPJPE remains 120.86 mm, indicating that camera trajectory estimation and scale recovery remain practical bottlenecks.

Questions for authors
  1. Table 8 explicitly states that RR-MPJPE subtracts the wrist position, which matches the definition of RA-MPJPE in the main text. Do these two metrics refer to the same quantity? If so, should the abbreviations be unified?

  2. Could the authors provide the complete training splits, dataset mixing ratios, and training configurations, together with the checkpoint, frontend, and fine-tuning status of each baseline? What are the default depth model, the source of the hand masks, and the camera configuration used for HaWoR in Table 1?

4 comments
AuthorsOpenReview ↗

Rebuttal For MetricHand

Thank you for your careful reading and constructive comments on metric terminology and reproducibility. Your feedback helps us clarify the evaluation definitions and implementation details needed to interpret and reproduce the results.

We address your questions below.

Q1: Table 8 explicitly states that RR-MPJPE subtracts the wrist position, which matches the definition of RA-MPJPE in the main text. Do these two metrics refer to the same quantity? If so, should the abbreviations be unified?

A1: Yes. RR-MPJPE in Table 8 denotes the same quantity as RA-MPJPE in the main text and Appendix C. Both subtract the predicted and ground-truth wrist positions from their respective joint coordinates before computing MPJPE, removing translation while retaining orientation and scale errors. This is a notation inconsistency rather than a separate metric. We will replace RR-MPJPE with RA-MPJPE in Table 8 and its caption to ensure consistent terminology throughout the paper.

Q2: Could the authors provide the complete training splits, dataset mixing ratios, and training configurations, together with the checkpoint, frontend, and fine-tuning status of each baseline? What are the default depth model, the source of the hand masks, and the camera configuration used for HaWoR in Table 1?

A2: Thank you for pointing out these reproducibility details. The joint training stage of MetricHand uses the training sets of HOT3D, ARCTIC, DexYCB, and HO3D in equal proportions, with a per-GPU batch size of 16 on 8 GPUs for 50,000 training steps. We use the AdamW optimizer, with learning rates of 2×10⁻⁶ and 4×10⁻⁵ for the initialized modules and the newly added Flow modules, respectively, a weight decay of 10⁻⁴, and a 500-step warm-up followed by cosine decay. All baselines in Table 1 use the code and publicly released checkpoints provided by their official GitHub repositories. We did not perform any additional training or fine-tuning on these baselines. HaWoR’s runtime configuration and evaluation protocol follow the instructions in its official GitHub repository. MetricHand uses MoGe-3 to provide depth priors by default. Table 4 in the appendix additionally reports comparisons using Metric3D v2 and sensor depth on DexYCB, where Metric3D v2 uses the same version as HaWoR. Our hand bounding boxes are generated by the same hand-pose frontend used by HaWoR. Evaluation follows the standard splits: DexYCB uses the s0 test split, HO3D uses the official evaluation split, ARCTIC uses the P2 validation split, and HOT3D uses the same validation split and evaluation protocol as HaWoR (as specified on GitHub). We will include the above training and implementation details in the revised manuscript.

Under the evaluation protocols in the paper, MetricHand reduces absolute camera-space MPJPE by 49.3%, 51.6%, and 63.2%, respectively, compared with the second-ranked methods on HOT3D and ARCTIC, and on the internal HOI4D held-out split evaluated in a zero-shot setting (Tables 1 and 8). These results demonstrate the effectiveness of the method. Together with the ablation studies, they also indicate that a structured 3D hand–object interaction (HOI) representation and its continued use throughout motion generation benefit 4D hand reconstruction, providing a reference for future research.

Thank you again for identifying these points and helping us improve the clarity and reproducibility of the paper. We hope the clarification of the metric notation and the implementation details above address your questions. If these explanations resolve your concerns, we would be grateful if you would reconsider your assessment and consider an upward revision of your score. We sincerely appreciate your time and consideration.

AuthorsOpenReview ↗

Rebuttal for weakness

W1 & W2: World-space error and SLAM drift. ABS-W-MPJPE directly measures the mean Euclidean distance between predicted and ground-truth hand joints in the same world reference frame, without additional hand alignment:

\mathrm{ABS\text{-}W\text{-}MPJPE} = \frac{1}{TJ}\sum_{t=1}^{T}\sum_{j=1}^{J}\left\\|\widehat{\mathbf{J}}^{W}_{t,j} - \mathbf{J}^{W,\mathrm{GT}}_{t,j}\right\\|_2.

Since world-space reconstruction relies on camera poses provided by an external SLAM module, this metric reflects both hand reconstruction errors and residual camera trajectory errors. On ARCTIC, our camera-space MPJPE is relatively low (17.54 mm), whereas ABS-W-MPJPE is considerably higher (120.86 mm). This gap suggests that trajectory errors from the external DROID-SLAM module may substantially affect the final world-space reconstruction. Prolonged camera motion can also introduce accumulated drift that is not fully removed by blockwise rigid alignment, further degrading reconstruction accuracy.

AuthorsOpenReview ↗

Follow-up Comment

Dear Reviewer sKih,

Thank you again for your thoughtful review. We would like to briefly follow up on our clarifications regarding evaluation terminology, reproducibility, and world-space accuracy.

Our responses explain the equivalence of RR-MPJPE and RA-MPJPE and outline the notation correction. We also provide further details on training splits, dataset mixing, training settings, and baseline and frontend configurations. For world-space evaluation, we clarify the interpretation of ABS-W-MPJPE under the specified protocol and discuss how the accuracy and accumulated drift of the external SLAM trajectories influence the reported results.

We would be grateful for any further feedback when you have a chance, particularly on whether these clarifications address the points you raised. We would be happy to provide any additional details. Thank you for your time and consideration.

AuthorsOpenReview ↗

Follow-up Comment

Dear Reviewer sKih,

Thank you again for your thoughtful review. We would like to briefly follow up on our clarifications regarding evaluation terminology, reproducibility, and world-space accuracy.

Our responses explain the equivalence of RR-MPJPE and RA-MPJPE and outline the notation correction. We also provide further details on training splits, dataset mixing, training settings, and baseline and frontend configurations. For world-space evaluation, we clarify the interpretation of ABS-W-MPJPE under the specified protocol and discuss how the accuracy and accumulated drift of the external SLAM trajectories influence the reported results.

We would be grateful for any further feedback when you have a chance, particularly on whether these clarifications address the points you raised. We would be happy to provide any additional details. Thank you for your time and consideration.

Reviewer_hZG9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors presented a geometry-aware flow-matching framework for reconstructing temporally consistent 3D hand motion in metric world coordinates from RGB videos. The method combines RGB features, monocular depth estimates, and camera trajectories to obtain coarse MANO predictions, then leverages topology-aware hand–object interaction geometry, learned observation-quality weighting, and cross-window temporal memory to refine hand translation, orientation, articulation, and shape. A conditional flow-matching model jointly generates the motion of both hands over time. Experiments on HOT3D, ARCTIC, DexYCB, and HO3D, along with zero-shot evaluation on HOI4D, show improvements over existing methods in camera-space and world-space MPJPE, wrist localization, and temporal consistency.

Critical strengths
  • The paper uses local hand–object interaction geometry to inform both world-space placement and pose recovery, providing a coherent approach to two closely related reconstruction challenges.
  • The topology-aware sampling associates local surface structure with individual hand joints, offering a potentially effective way to exploit geometric cues under occlusion and ambiguous visual evidence.
  • The learned quality-weighting mechanism allows the model to adapt to varying confidence in visual and depth observations without requiring explicit quality labels.
  • Combining cross-window memory with conditional flow matching provides a unified framework for leveraging temporal context and improving motion consistency.
  • The paper evaluates on multiple datasets and includes zero-shot testing on unseen videos for generalization evaluation, and reports substantial improvements in world-space localization, wrist accuracy, and temporal consistency over existing baselines.
Critical weaknesses
  • Limited conceptual novelty: The framework combines existing components—RGB-D fusion (WiLoR, Monocular frozen depth model, DROIR-SLAM), confidence weighting, temporal memory, and flow matching. The paper should more clearly establish which methodological contribution is fundamentally new beyond their integration.
  • Insufficient positioning against prior work: The related-work section largely lists existing methods rather than critically discussing their differences, making it difficult to assess the paper's novelty relative to recent hand-reconstruction and flow-matching approaches.
  • Overly complex presentation: The method introduces numerous modules and terminology, making the core technical contribution difficult to identify. The main figure is also difficult to follow.
  • Lack of geometry-specific evaluation: Although geometry is central to the paper's motivation and design, the evaluation primarily focuses on pose accuracy and temporal consistency. The absence of explicit geometric metrics (e.g., hand–object surface distance, contact consistency, or joint-to-surface alignment) makes it difficult to verify whether the proposed geometric reasoning actually improves geometric fidelity.
Questions for authors
  • What is the precise algorithm contribution beyond HandFlow-style [1] condtional flow matching with joint associated RGB-D features ?
  • Which components are fundametally new, and which are adaptations of existing techniques ?
  • Are there any failure cases for the method ? does it struggle with specific objects or with specific occlusions ?
  • The proposed method uses flow matching to reconstruct hand motion conditioned on RGB images. What advantages does this formulation offer over deterministic approaches? In particular, does the stochastic sampling process produce different reconstructions for the same input observations ? [1] HandFlow: Fully Generative 4D Hand Recovery with Flow Matching (2026)
7 comments
AuthorsOpenReview ↗

Rebuttal for Q1

Thank you for your careful review and constructive questions about our algorithmic contribution, the role of individual components, failure cases, and stochastic reconstruction. Your feedback helps us make the distinction from existing approaches and the supporting evidence clearer.

We address your questions below.

Q1: What is the precise algorithmic contribution beyond HandFlow-style conditional flow matching with joint-associated RGB–D features [1]?

A1: Our contribution is not simply conditioning flow matching on RGB–D features, but constructing a topology-aware, metric-scale 3D HOI representation that supports hand reconstruction through explicit depth constraints and per-joint perspective projection, while maintaining its geometric evidence throughout motion generation.

The benefit extends beyond generic RGB–D fusion. On ARCTIC, adding depth through concatenation or cross-attention reduces camera-space MPJPE from 36.34 mm to 31.65 mm or 29.64 mm, respectively. With the same metric regression head, our topology-aware HOI representation further reduces this error to 19.98 mm, a 32.6% reduction relative to cross-attention. ABS-W-MPJPE also decreases from 142.14 to 127.49 mm. These controls isolate the benefit of organizing geometric evidence before introducing flow decoding.

Representation and decoderCamera MPJPE ↓PA-MPJPE ↓ABS-W-MPJPE ↓
RGB-only + regression36.346.78151.74
RGB–D concatenation + regression31.656.26147.36
RGB–D cross-attention + regression29.646.01142.14
Topology-aware RGB–D HOI + regression19.985.79127.49
Topology-aware conditions + flow17.545.18120.86

ARCTIC results; errors are in mm. The first four controls share the same metric regression head.

The algorithmic distinction is how 3D hand–object interaction evidence is structured and reused. Rather than merging generic appearance and depth tokens, we anchor multi-scale RGB–D sampling at projected anatomical joints and introduce geometric attention biases based on image distance, log-depth differences, and backprojected 3D surface distance. This produces joint-associated descriptors of local hand–object geometry. A metric head uses absolute-depth statistics to refine translation, while a joint-indexed cross-window memory retains geometric evidence for every flow update. The representation therefore connects metric placement, local articulation, and temporal correspondence within the reconstruction process.

HandFlow [1], by comparison, conditions its hand-motion generator on RGB features and skeletal rays without explicit depth-derived surface geometry; depth in its released pipeline supports SLAM. Our full method achieves 17.54 mm camera MPJPE and 120.86 mm ABS-W-MPJPE on ARCTIC. In the HOT3D full-method comparison, camera MPJPE / RA-MPJPE decreases from HandFlow’s 34.89 / 14.26 mm to 8.60 / 5.84 mm.

Thus, the contribution is a structured, metric-scale 3D HOI representation and its persistent use for reconstruction, rather than merely an additional depth modality, with improvements in both absolute localization and root-relative reconstruction.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2: Which components are fundamentally new, and which are adaptations of existing techniques?

A2: MetricHand’s central contribution is to jointly leverage RGB and depth to construct a metric-scale 3D hand–object interaction (HOI) representation for absolute hand localization, while continuously supporting motion reconstruction through joint-quality conditioning and temporal memory. The innovation goes beyond adding a depth modality: it organizes predicted depth into geometric conditions consistent with hand topology, interaction surfaces, and temporal correspondences.

  1. Geometry-aware 3D HOI representation: the core representational innovation

Using anatomical keypoints as indices, we aggregate spatially aligned RGB and depth features to construct a 3D HOI representation that connects hand topology with interaction surfaces and is used throughout motion generation, rather than performing generic RGB–D fusion. In a regression setting with a shared Metric Head, this representation reduces MPJPE from 29.64 to 19.98 compared with RGB–D cross-attention, a 32.6% reduction (Table 1). In the full model, removing topology-aware sampling or HOI region context increases MPJPE from 17.54 to 38.91 and 32.42, respectively. These results indicate that the gains arise not only from depth input, but also from how it is organized around joints and interaction geometry.

  1. Metric Head beyond weak perspective: absolute localization at real-world scale

Reconstruction requires not only an accurate hand shape after alignment, but also a metric-scale 3D position (tx,ty,tz)(t_x,t_y,t_z) in camera coordinates. As illustrated in Figure 1(a) of the paper, localization based solely on weak-perspective scale conversion lacks an explicit metric-depth basis, and depth variations at close range also affect translation estimates. The distinction is not whether a baseline uses any scale information, but whether depth-derived metric geometry directly informs hand translation and articulation during reconstruction.

To address this, the Metric Head uses fused RGB–D features to refine 3D translation along the viewing ray and in log-depth space, explicitly incorporating the depth prior into localization rather than relying entirely on weak-perspective projection. Table 2 reports the localization performance of the overall method. On HOT3D, MetricHand achieves a camera-coordinate MPJPE of 8.60, compared with 16.97 for HaWoR and 34.89 for HandFlow. Its world-coordinate ABS-W-MPJPE is 20.58, compared with 28.88 and 42.59, respectively. These results support the overall metric reconstruction approach, but should not be interpreted as an isolated ablation gain from the Metric Head.

  1. Quality-aware conditioning and temporal memory: combining per-joint reliability with historical evidence

We jointly organize camera-ray cues, topology-aware HOI features, and observation quality as generative conditions, and predict hand-level and joint-level reliability. This allows occluded joints to reduce the influence of unreliable local observations while retrieving spatial context and historical evidence for the corresponding joints. Removing quality weighting while retaining validity masks increases MPJPE from 17.54 to 25.35; removing cross-window memory increases it to 25.67 (Table 1). These results support the value of adapting existing confidence-conditioning and queue mechanisms to operate at the joint level with explicit geometric associations.

Table 1: Experimental results for HOI representation, quality conditioning, and temporal memory

Adapted from Tables 2–3 of the paper. All results are on ARCTIC. Errors are reported in mm; lower is better. The regression configurations share the same Metric Head.

SettingConfigurationCamera-coordinate MPJPE ↓World-coordinate ABS-W-MPJPE ↓
RegressionRGB-only36.34151.74
RegressionRGB–D concatenation31.65147.36
RegressionRGB–D cross-attention29.64142.14
RegressionTopology-aware HOI representation19.98127.49
GenerativeFull model17.54120.86
GenerativeWithout topology-aware sampling38.91154.36
GenerativeWithout HOI region context32.42139.67
GenerativeWithout quality weighting25.35132.69
GenerativeWithout cross-window memory25.67136.24

Table 2: Absolute localization accuracy compared with existing methods

Adapted from Table 1 of the paper, using the evaluation configurations reported therein.

MethodHOT3D: Camera MPJPE ↓HOT3D: ABS-W-MPJPE ↓ARCTIC: Camera MPJPE ↓ARCTIC: ABS-W-MPJPE ↓
HaMeR-SLAM67.8773.2854.56166.19
WiLoR-SLAM53.7859.2136.23152.89
HaWoR16.9728.8837.26154.36
HandFlow34.8942.59——
MetricHand8.620.5817.54120.86
AuthorsOpenReview ↗

Rebuttal for Q3

Q3: Are there any failure cases for the method? Does it struggle with specific objects or specific occlusions?

A3: MetricHand shows robustness across the evaluated occlusion conditions. On DexYCB, PA-MPJPE is 4.03, 4.12, and 4.14 mm for the 25–50%, 50–75%, and 75–100% occlusion groups, respectively (Table 6). ARCTIC also contains naturally occluded hand–object interactions.

We additionally conducted controlled input-masking experiments on HOT3D and ARCTIC at 30% and 50% masking levels. As shown below, masking increases reconstruction errors, with smaller changes in PA-MPJPE than in MPJPE, suggesting that Procrustes-aligned pose is less affected than absolute reconstruction.

DatasetOcclusion conditionΔ MPJPE (mm)Δ RA-MPJPE (mm)Δ PA-MPJPE (mm)
HOT3DMask 30%+4.85+1.651+0.433
HOT3DMask 50%+4.96+2.789+0.932
ARCTICMask 30%+1.132+0.27+0.291
ARCTICMask 50%+2.487+0.49+0.835

Prolonged occlusion across consecutive frames can still produce failure cases when insufficient visual evidence remains. We will include such examples alongside successful reconstructions in the appendix. Existing visualizations are provided in Figures 5–10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.