acceptodds
Under review as a conference paper at ICLR 2027

HandCrafter: Monocular 3D Hand Motion Reconstruction with Multi-View Video Diffusion Priors

23 views

Abstract

Monocular 3D hand motion reconstruction remains challenging in hand-object interaction videos due to persistent occlusions and the resulting spatial ambiguity. We present HandCrafter, a generative multi-view framework that reconstructs 3D hand motion by synthesizing complementary viewpoints from a single monocular video. We first leverage an off-the-shelf geometry foundation model to warp the input video into novel viewpoints, producing incomplete but spatially aligned conditions that are subsequently completed by an adapted video diffusion model. Hand motion is then recovered from the resulting multi-view diffusion representations. Instead of relying on hand detection and cropping, our estimator uses sparse 2D joint anchors for spatial localization and decodes unified MANO parameters from cross-view diffusion representations, enabling crop-free 3D hand motion reconstruction. To mitigate the scarcity of multi-view HOI data, we further mix limited multi-view data with scalable monocular supervision, enabling cross-view and temporal modeling to benefit from a broader training distribution. Extensive experiments demonstrate superior performance under severe occlusions and strong generalization across diverse in-the-wild HOI scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-View Diffusion;3D Hand Estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8q11OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HandCrafter, a monocular hand reconstruction method based on multi-view generation, aiming to address hand reconstruction under severe occlusion.

In terms of methodology, it uses a pretrained Wan2.1 video diffusion model, designs cross-view hand attention and view attention, extracts hand features from the DiT latent space, and designs a Decoder to decode MANO parameters. During training, a multi-view video synthesis method is constructed to train the model's visual priors in the first stage, followed by supervised training using hand data in the second stage.

Experiments show that the method achieves improvements on multiple metrics, especially demonstrating outstanding performance under severe occlusion.

Critical strengths
  1. The research idea has a certain degree of novelty, using multi-view generation to address the occlusion problem in hand reconstruction. In particular, hand features are extracted from the DiT latent space, and a trainable Decoder is used to directly predict MANO parameters, achieving end-to-end prediction.

  2. In terms of training strategy, multi-view data and monocular videos are first used for multi-view synthesis training, fully utilizing the prior knowledge learned through cross-view supervision. The completion training on monocular data also expands the amount of training data.

  3. In terms of experimental results, relatively comprehensive evaluations are conducted on three datasets, with particularly significant improvements in severe occlusion scenarios. The ablation experiments also demonstrate the effectiveness of multi-view information.

Critical weaknesses
  1. Lack of direct evaluation of the quality of generative multi-view reconstruction. This paper uses multi-view generation to compensate for multi-view information for hand reconstruction, but the performance of multi-view prediction itself is relatively limited. Moreover, the authors also mention that there are some missing parts when constructing multi-view training data from a single view. It is suggested to supplement a comparative analysis between the multi-view reconstruction results and the hand reconstruction results.

  2. The authors cite ViDiHand, which is highly related to this work, as both methods leverage pretrained video diffusion models and intermediate DiT features for hand reconstruction. However, the authors do not include ViDiHand in the comparative experiments.

  3. The description of the MANO Decoder is unclear. Section 3.4 describes two independent Decoders, but the architecture diagram in Figure 2 is drawn as a single Decoder structure. It is unclear whether the two Decoders have similar structures or actually share some parameters. The authors are advised to clarify this. In addition, if there are two Decoders, it is suggested to supplement an ablation study comparing them with a single Decoder.

Questions for authors

It is suggested to clarify the Decoder architecture, as mentioned in Weakness 3.

4 comments
AuthorsOpenReview ↗

Official Response by Authors - Part 1

We appreciate the insightful comments and constructive feedback, and hope our responses below address the reviewer's concerns.

Analysis of Video Reconstruction Quality

Thanks for your suggestion. We have investigated the quality of the generated novel-view videos and the performance of 3D hand motion reconstruction across different denoising steps. This analysis has been added to Appendix A.7 of the revised manuscript, with qualitative visualizations shown in Figure 6 and quantitative results (Table 9) reported below:

Denoising StepsPSNR ↑SSIM ↑LPIPS ↓Hand-crop LPIPS ↓MPJPE-p (mm) ↓MPVPE-p (mm) ↓EPE-p (px) ↓CT-p (m) ↓
121.060.7640.5950.59213.7013.2916.850.154
320.510.7540.3940.46513.1512.7716.120.134
520.100.7430.3570.42813.3012.9116.360.141

As the number of denoising steps increases, the generated frames transition from blurry to sharp with progressively clearer details (Figure 6), which is consistently reflected by the decrease in both LPIPS and Hand-crop LPIPS. Note that PSNR and SSIM show an opposite trend because pixel-level similarity metrics inherently favor blurry, over-smoothed images that average out high-frequency uncertainty, whereas sharper details synthesized at larger denoising steps may incur slight pixel-wise misalignments with the ground truth.

Meanwhile, 3D hand pose reconstruction performance remains largely insensitive to the visual quality of the generated frames across different denoising steps. Precisely because a single denoising step already provides sufficient spatial cues for accurate hand motion reconstruction, our model only requires a single forward pass during inference, enabling the favorable inference cost reported in Appendix A.5.

AuthorsOpenReview ↗

Official Response by Authors - Part 2

(continued...)

Comparison with ViDiHand

Thank you for pointing this out. Since the code and pretrained checkpoints of ViDiHand have not been released so far, we re-implemented the model according to the details described in its paper and trained it on the identical dataset used in our work across 8 NVIDIA A100 80GB GPUs for three days until convergence. We have added the experimental results and analysis to Appendix A.8 (Table 8) of the revised manuscript, with the quantitative comparisons on each benchmark presented below:

ARCTIC Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.99200.996418.518.8617.898.5920.4913.230.0549
HandCrafter (Ours)0.99490.996014.026.5713.606.3818.1611.200.1593

DexYCB Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.56970.634219.8410.7619.0110.2426.0015.260.0603
HandCrafter (Ours)0.97540.915114.616.7414.066.4213.5611.410.0430

HO-Cap Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.60990.863422.237.7021.357.4425.7116.940.0666
HandCrafter (Ours)0.96700.980121.098.2720.257.9724.4415.790.0652

Furthermore, we have included qualitative comparisons with ViDiHand in Appendix A.8 (Figure 7) as well as on our project page. Together, these experimental results indicate that our method achieves superior overall performance compared to ViDiHand, which underscores the value of leveraging multi-view generative priors for 3D hand pose estimation.

AuthorsOpenReview ↗

Official Response by Authors - Part 3

(continued...)

Dual-decoder Design

Thank you for your question and constructive suggestion. To clarify, in our implementation, the two decoders (DaggD_{agg} and DpvD_{pv}) only share a similar architectural structure and do not share any parameters.

Furthermore, we have added an ablation study on the dual-decoder design in Appendix A.10 of the revised manuscript. Specifically, we remove the per-view decoder DpvD_{pv} and retain only the aggregation decoder DaggD_{agg} (One Decoder). Instead of predicting the global orientation o\boldsymbol{o} and translation t\boldsymbol{t} in each viewpoint's camera coordinate system via DpvD_{pv}, we modify DaggD_{agg} to directly output o\boldsymbol{o} and t\boldsymbol{t} in the camera coordinate system of the input view. The quantitative comparison (Table 11) is reported below:

MethodFAcc ↑Recall ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
One Decoder0.89540.94920.947244.6019.2643.2418.6784.3537.420.5762
Dual Decoder0.95720.98120.979435.4015.8034.2915.3350.5827.110.3943

These experimental results validate the effectiveness of our dual-decoder design. This is because the two decoders naturally decouple the parameters shared across viewpoints (such as hand shape β\boldsymbol{\beta} and pose θ\boldsymbol{\theta}) from per-view localization (o\boldsymbol{o} and t\boldsymbol{t}). Supervising hand localization in each viewpoint's corresponding camera coordinate system in a per-view manner both reduces the learning difficulty and provides denser supervision signals across views.

Dear Reviewer 8q11,

Thank you for your valuable time and constructive feedback. As the discussion period is approaching its end, we would like to kindly follow up and hope that our responses have adequately addressed your concerns. We would greatly appreciate any further feedback you may have.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_rAfdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HandCrafter, a framework for monocular 3D hand motion reconstruction using generative multi-view video priors. To address severe hand-object occlusions, the method first warps monocular observations into novel viewpoints using estimated geometry and then employs an adapted video diffusion model to complete the missing visual content. A parallel hand branch extracts cross-view diffusion features, which are decoded into unified MANO parameters without requiring hand detection and cropping.

The authors further propose a two-stage hybrid training strategy combining multi-view hand-object interaction datasets with monocular video supervision. Experiments on ARCTIC, DexYCB, and HO-Cap demonstrate competitive reconstruction accuracy, particularly under severe occlusions, while ablations investigate the effects of synthesized viewpoints and cross-view attention.

Critical strengths
  1. An interesting use of generative multi-view priors for hand reconstruction. The paper addresses the challenging problem of monocular 3D hand motion reconstruction under persistent hand-object occlusions. Instead of relying solely on temporal information from the observed viewpoint, HandCrafter synthesizes complementary viewpoints and exploits their diffusion representations for 3D reconstruction. This is a meaningful direction that extends the use of generative priors beyond conventional image or video enhancement.

  2. Strong quantitative results and useful ablation studies. The proposed method achieves substantial improvements on ARCTIC and competitive results on the unseen HO-Cap dataset. In particular, Table 2 reports a reduction in PA-MPJPE from 17.03 mm to 6.57 mm on ARCTIC. The occlusion analysis in Table 3 and the view-attention ablations in Table 5 further support the effectiveness of multi-view modeling. These results constitute strong empirical evidence for the overall framework.

Critical weaknesses
  1. The contribution of genuinely new geometric information from synthesized views is not sufficiently isolated. This is my primary concern. The proposed framework generates additional viewpoints from a single monocular video, but these views are synthesized using learned priors rather than independent observations. Therefore, the generated viewpoints cannot provide additional measured geometric evidence in the same sense as real multi-view recordings. Although Tables 4 and 5 demonstrate improvements from additional viewpoints and cross-view attention, they do not fully distinguish the benefit of complementary geometric reasoning from the benefit of increased model capacity, additional conditioning, or stronger learned motion priors. A more controlled comparison with an equally capable monocular diffusion-based estimator, especially ViDiHand, would better substantiate the central claim.

  2. The fairness of comparisons with existing methods requires clarification. Table 2 reports very large improvements over several established methods, particularly on ARCTIC. However, HandCrafter uses a two-stage training strategy with multiple annotated multi-view datasets and additional monocular video supervision, whereas the compared methods use their released pretrained checkpoints. Differences in training distributions, supervision, and model capacity may contribute to the performance gap. Although these comparisons are useful for evaluating the complete system, they do not establish how much of the improvement comes specifically from the proposed multi-view diffusion mechanism. Furthermore, the closely related ViDiHand method is discussed in the paper but is not included in the main quantitative comparisons.

  3. The generalization and efficiency of the approach remain insufficiently characterized. The framework relies on monocular depth estimation, geometric warping, and multi-view diffusion generation, introducing several potential sources of error and substantial computational complexity. However, the main experiments provide limited analysis of sensitivity to depth inaccuracies and the trade-off between reconstruction accuracy and inference cost. Additionally, the paper acknowledges that its current formulation is less suitable for significant camera motion, including egocentric videos. These limitations narrow the practical applicability of a method otherwise motivated by challenging real-world hand-object interactions.

Questions for authors
  1. Multi-view contribution: Could the authors clarify how much of the improvement comes from genuinely complementary spatial information rather than additional diffusion model capacity? Is there an existing controlled comparison with a monocular diffusion-based reconstruction model of comparable capacity?

  2. Baseline comparison: Why is ViDiHand, a closely related method using pretrained video diffusion representations for hand reconstruction, not included in Table 2? Could the authors also clarify differences in training data and supervision between HandCrafter and the evaluated baselines?

  3. Experimental consistency: Table 6 reports an MPJPE-p of 35.40 mm for the eight-query configuration, whereas Table 2 reports 14.02 mm for the full model on ARCTIC. Could the authors clarify the differences in training configurations and whether these results are directly comparable?

  4. Robustness and efficiency: How sensitive is HandCrafter to errors in monocular depth estimation and geometric warping? Could the authors report or clarify the inference cost associated with synthesizing multiple viewpoints?

5 comments
AuthorsOpenReview ↗

Official Response by Authors - Part 1

We appreciate the insightful comments and constructive feedback, and hope our responses below address the reviewer's concerns.

Comparison with ViDiHand

Thank you for this valuable comment. Up to now, ViDiHand has not publicly released its source code or pretrained checkpoints. To conduct a direct and controlled evaluation, we reproduced ViDiHand based on the methodology described in its paper. Like our method, ViDiHand also adopts Wan 2.1 1.3B as its base model, so its model capacity is essentially comparable to HandCrafter. Moreover, we trained it on the exact same dataset as ours across 8 NVIDIA A100 80GB GPUs for three days until convergence. We have included the experimental results and corresponding analysis in Appendix A.8 (Table 8) of the revised manuscript, with the quantitative results on each dataset summarized below:

ARCTIC Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.99200.996418.518.8617.898.5920.4913.230.0549
HandCrafter (Ours)0.99490.996014.026.5713.606.3818.1611.200.1593

DexYCB Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.56970.634219.8410.7619.0110.2426.0015.260.0603
HandCrafter (Ours)0.97540.915114.616.7414.066.4213.5611.410.0430

HO-Cap Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.60990.863422.237.7021.357.4425.7116.940.0666
HandCrafter (Ours)0.96700.980121.098.2720.257.9724.4415.790.0652

The experiments demonstrate that our approach outperforms ViDiHand overall across the evaluated benchmarks, confirming that under comparable model capacity, generating multi-view priors brings clear benefits to 3D hand pose estimation. In addition, we have supplemented qualitative comparisons with ViDiHand in Appendix A.8 (Figure 7) and on our project page.

AuthorsOpenReview ↗

Official Response by Authors - Part 2

(continued...)

Training Data and Supervision of Baselines

We summarize and compare the training datasets used by HandCrafter and all evaluated baselines in the table below:

MethodTraining Datasets
HaMeRFreiHAND, HO3D, MTC, RHD, InterHand2.6M, H2O3D, DexYCB, COCO-WholeBody, Halpe, MPII-NZSL
WiLoRDetection: WHIM; Hand Reconstruction: FreiHAND, HO3D, MTC, RHD, InterHand2.6M, H2O3D, DexYCB, COCO-WholeBody, Halpe, MPII-NZSL, Re:InterHand, HOT3D
HaWoRHOT3D, ARCTIC, DexYCB, HO3D (using frozen WiLoR backbone)
HaPTICImage Training: FreiHAND, HO3D, MTC, RHD, InterHand2.6M, H2O3D, DexYCB, COCO-WholeBody, Halpe, MPII-NZSL; Video Training: DexYCB, ARCTIC, H2O, HO3D, InterHand2.6M
OmniHandsInterHand2.6M, Re:InterHand, FreiHAND, HO3D, MTC, RHD, H2O3D, DexYCB, COCO-WholeBody, Halpe, MPII-NZSL
HandCrafter (Ours)w/ Hand Pose Annotations: ARCTIC, DexYCB, TACO, OakInk-v2, GigaHands; w/o Hand Pose Annotations: SynCamVideo, HOIGen-1M

Regarding the supervision schemes, the similarities and differences between HandCrafter and the evaluated baselines are summarized as follows:

  • Similarities: In the hand reconstruction stage, HandCrafter adopts supervision signals similar to those of the evaluated baselines, including constraints on MANO parameters, 3D geometry (joints or meshes), and 2D reprojection, to map visual features into accurate hand poses and spatial locations.
  • Differences: HandCrafter decouples generative representation learning and hand motion estimation into two stages. Stage 1 uses RGB videos as the supervision target, learning cross-view and temporal representations for hand reconstruction through generation supervision on synchronized multi-view videos and degradation-restoration supervision on monocular videos; notably, this stage requires no hand pose annotations. Stage 2 freezes the video generation branch and leverages hand pose annotations to train the hand branch and decoders.

Training Configurations in Table 2 and Table 6

Thank you for your question. As stated in Section 4.4 (Number of MANO Query Tokens), since training on the full dataset until convergence is excessively time-consuming, we randomly sampled 20% of the data from the entire training set for the ablation study in Table 6. All three experimental configurations (1, 4, and 8 query tokens) maintained identical training settings and were trained on this exact same 20% subset for the same number of steps. Because the amount of training data used here is smaller than that of our main model, the results reported in Table 6 differ from those in Table 2.

AuthorsOpenReview ↗

Official Response by Authors - Part 3

(continued...)

Sensitivity Analysis of Depth Reconstruction

Thank you for your question and constructive suggestion. In both training and testing, we employ VGGT-Ω\Omega for depth reconstruction. We observed that retaining all raw depth values introduces substantial floating points into the back-projected point clouds; therefore, in both training and testing, we apply an appropriate threshold to the confidence scores output by VGGT-Ω\Omega to filter out low-confidence points.

To evaluate the sensitivity of HandCrafter to depth reconstruction quality, we conducted three additional experiments on the DexYCB test set under degraded depth settings: (1) Relaxed Filtering (τ=1.3\tau = 1.3), which filters depth predictions using a confidence threshold of 1.3 and retains a moderate amount of floating points; (2) w/o Confidence Filtering, which applies no filtering and retains heavy floating-point artifacts; and (3) w/o Depth Reconstruction, which omits depth reconstruction entirely so that the warped novel-view conditions are blank frames. The quantitative evaluation results on the DexYCB test set are summarized below:

Depth Reconstruction SettingF1 ↑MPJPE-p (mm) ↓MPVPE-p (mm) ↓EPE-p (px) ↓CT-p (m) ↓
Default Filtering (Ours)0.915114.6114.0613.560.043
Relaxed Filtering (τ=1.3\tau = 1.3)0.914715.5114.9315.540.045
w/o Confidence Filtering0.914715.4614.8915.490.045
w/o Depth Reconstruction0.916117.9517.3315.700.050

As shown in the results, introducing more floating points and lowering the quality of depth reconstruction leads to a moderate decline in 3D hand pose estimation performance. When depth reconstruction is completely omitted, the performance degradation becomes more pronounced, confirming the effectiveness of geometric warping and multi-view priors. Nevertheless, the model still maintains a reasonable level of performance even without depth reconstruction; this is because when the multi-view priors are completely stripped away, the model degenerates into a monocular hand pose estimator and remains capable of reconstructing hand motion directly from the input-view observations.

Inference Cost

Thank you for this helpful suggestion. In Appendix A.5 (Table 7) of the revised manuscript, we have included an evaluation of inference efficiency for both the preprocessing stage and the main backbone of HandCrafter under varying numbers of synthesized novel viewpoints (V∈{1,2,3}V \in \lbrace1, 2, 3\rbrace), alongside the inference costs of the evaluated baselines:

MethodFPS ↑GPU Memory (GB) ↓
WiLoR8.474.27
HaMeR1.5111.60
HaWoR1.8912.23
HaPTIC1.3611.71
OmniHands1.5012.01
HandCrafter (V=1V=1)Preprocessing: 8.98, Main Backbone: 27.81Preprocessing: 13.23, Main Backbone: 8.73
HandCrafter (V=2V=2)Preprocessing: 7.62, Main Backbone: 17.88Preprocessing: 13.26, Main Backbone: 9.60
HandCrafter (V=3V=3)Preprocessing: 6.61, Main Backbone: 12.90Preprocessing: 13.30, Main Backbone: 10.47

These experiments show that HandCrafter achieves competitive inference efficiency. Unlike image-based baselines that perform estimation frame by frame, our model predicts the hand motion across an entire video sequence in one go, which contributes considerably to accelerating the overall inference speed.

Reviewer_EC8oOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HandCrafter, a framework for monocular 3D hand motion reconstruction using multi-view video diffusion priors, particularly under severe hand-object occlusions. The method first uses a pretrained geometry model to warp the input video into novel viewpoints, which are then completed by a video diffusion model. It further extracts cross-view diffusion features and employs a structured MANO decoder to estimate 3D hand pose and shape without relying on hand detection and cropping. A two-stage training strategy combines multi-view and monocular video data. Experiments on multiple benchmarks demonstrate competitive reconstruction accuracy, robustness to severe occlusions, and generalization to unseen scenarios.

Critical strengths
  1. The paper explores using generated multi-view information to improve 3D hand pose estimation under severe occlusions, which is an interesting direction.

  2. The paper provides extensive evaluations on ARCTIC, DexYCB, and HO-Cap, achieving competitive results with both quantitative and qualitative comparisons.

Critical weaknesses
  1. The experiments lack a direct comparison with ViDiHand, which also uses video diffusion models for 3D hand motion reconstruction.

  2. Although the paper includes ablations on the number of generated views, it remains unclear how much of the performance improvement comes from mixing multiple training datasets. Hence, I suggest providing some experimental results about the contributions of training data and generated multi-view information.

  3. The paper does not report inference time or computational cost. Since the method uses a large video diffusion model and multiple generated views, I suggest reporting FPS, GPU memory usage, and runtime compared with existing hand reconstruction methods.

Questions for authors

see the weakness

4 comments
AuthorsOpenReview ↗

Official Response by Authors - Part 1

We appreciate the insightful comments and constructive feedback, and hope our responses below address the reviewer's concerns.

Comparison with ViDiHand

Thanks for this review. To date, neither the official code nor pretrained checkpoints of ViDiHand have been publicly released. We therefore reproduced ViDiHand following the architectural design and training formulation described in its paper, and trained it using the exact same training data as HandCrafter on 8 NVIDIA A100 80GB GPUs for three days until convergence. Quantitative comparisons on ARCTIC, DexYCB, and the unseen HO-Cap dataset are reported below:

ARCTIC Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.99200.996418.518.8617.898.5920.4913.230.0549
HandCrafter (Ours)0.99490.996014.026.5713.606.3818.1611.200.1593

DexYCB Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.56970.634219.8410.7619.0110.2426.0015.260.0603
HandCrafter (Ours)0.97540.915114.616.7414.066.4213.5611.410.0430

HO-Cap Dataset

MethodFAcc ↑F1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
ViDiHand0.60990.863422.237.7021.357.4425.7116.940.0666
HandCrafter (Ours)0.96700.980121.098.2720.257.9724.4415.790.0652

The results show that HandCrafter outperforms ViDiHand overall across all three benchmarks, demonstrating the value of multi-view generative priors for 3D hand pose estimation. We have added both the quantitative comparison (Table 8) and qualitative comparison (Figure 7) with ViDiHand to Appendix A.8 of the revised manuscript, and provided additional qualitative comparisons on our project page.

Contributions of Training Data

While multi-view HOI datasets provide direct cross-view supervision, they are captured in controlled settings with limited visual diversity; we therefore incorporate scalable monocular videos from HOIGen-1M in Stage I. To evaluate the contribution of this scalable monocular training data, we conducted an ablation study comparing models trained without and with HOIGen-1M. Quantitative results on the unseen HO-Cap dataset (Table 10 in Appendix A.9) are reported below:

Training DataF1 ↑MPJPE-p (mm) ↓PA-MPJPE-p (mm) ↓MPVPE-p (mm) ↓PA-MPVPE-p (mm) ↓EPE-p (px) ↓GO-p (°) ↓CT-p (m) ↓
w/o HOIGen-1M0.638939.4813.8538.1113.2864.1334.930.1319
w/ HOIGen-1M0.806638.7910.9637.3210.5250.1331.570.1170

Qualitatively (Figure 8), novel-view synthesis comparisons show that training with HOIGen-1M generates better videos on in-the-wild cases. Quantitatively (Table 10), incorporating HOIGen-1M further improves hand motion reconstruction on unseen data. Together, these results demonstrate that introducing scalable monocular video data significantly enhances model generalization ability. Detailed analyses have been added to Appendix A.9 of the revised manuscript.

AuthorsOpenReview ↗

Official Response by Authors - Part 2

(continued...)

Inference Cost

Thanks for your suggestion. We have added a report on the inference cost of our model in terms of inference throughput (FPS) and peak GPU memory consumption. Taking the generation of 3 novel views (V=3V=3) as an example, the inference costs of HandCrafter in the preprocessing and main backbone stages compared with existing baselines (Table 7 in Appendix A.5) are reported below:

MetricHandCrafter PreprocessingHandCrafter Main BackboneWiLoRHaMeRHaWoRHaPTICOmniHands
FPS ↑6.6112.908.471.511.891.361.50
GPU Memory (GB) ↓13.3010.474.2711.6012.2311.7112.01

The results demonstrate that our model achieves favorable computational efficiency. Compared with image-based methods that process frames individually, our method estimates the hand motion of an entire video clip in a single forward pass, which effectively accelerates inference. More detailed analyses and descriptions have been added to Appendix A.5 of the revised manuscript.

Dear Reviewer EC8o,

Thank you for your valuable time and constructive feedback. As the discussion period is approaching its end, we would like to kindly follow up and hope that our responses have adequately addressed your concerns. We would greatly appreciate any further feedback you may have.

Thank you again for your time and consideration.

Best regards, The Authors

Thank you for the detailed rebuttal and additional experiments. My concerns have been adequately addressed, and I maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their detailed reviews and suggestions!

We have updated the manuscript with the following revisions based on the reviewers' suggestions. All revisions in the updated manuscript are highlighted in blue:

Add more experiments

  • Add inference speed (FPS) and peak GPU memory comparison of HandCrafter across different numbers of novel views against existing baselines (reviewer EC8o and rAfd, Appendix A.5 line 982-1002, Table 7).
  • Add quantitative and qualitative evaluation of novel-view video reconstruction quality and its relationship with 3D hand pose estimation across different denoising steps (reviewer 8q11, Appendix A.7 line 1060-1077, Table 9 and Figure 6).
  • Reimplemented and trained ViDiHand on the exact same dataset as our method. Add quantitative and qualitative comparison with reproduced ViDiHand across ARCTIC, DexYCB, and the unseen HO-Cap dataset (reviewer EC8o, 8q11 and rAfd, Appendix A.8 line 1098-1127, Table 8 and Figure 7).
  • Add quantitative ablation study on the contribution of the monocular HOIGen-1M training data (reviewer EC8o, Appendix A.9 line 1129-1173, Table 10).
  • Add ablation study comparing our dual-decoder design with a single-decoder variant (reviewer 8q11, Appendix A.10 line 1175-1189, Table 11).
  • Add sensitivity analysis of depth reconstruction (response to reviewer rAfd).

Add more discussions and details

  • Add comparison of training datasets and supervision schemes across all baselines (response to reviewer rAfd)
  • Add clarification of the training configurations in Table 2 and Table 6 (response to reviewer rAfd).
  • Add clarification on parameter decoupling between the per-view decoder and the aggregation decoder (response to reviewer 8q11).

Update project page

  • Add qualitative video comparisons with reproduced ViDiHand on our anonymous project page (reviewer EC8o, 8q11 and rAfd).

Thanks again for all the effort and time, and we look forward to further discussions if there are any more questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.