- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HandCrafter, a monocular hand reconstruction method based on multi-view generation, aiming to address hand reconstruction under severe occlusion.
In terms of methodology, it uses a pretrained Wan2.1 video diffusion model, designs cross-view hand attention and view attention, extracts hand features from the DiT latent space, and designs a Decoder to decode MANO parameters. During training, a multi-view video synthesis method is constructed to train the model's visual priors in the first stage, followed by supervised training using hand data in the second stage.
Experiments show that the method achieves improvements on multiple metrics, especially demonstrating outstanding performance under severe occlusion.
Critical strengths
-
The research idea has a certain degree of novelty, using multi-view generation to address the occlusion problem in hand reconstruction. In particular, hand features are extracted from the DiT latent space, and a trainable Decoder is used to directly predict MANO parameters, achieving end-to-end prediction.
-
In terms of training strategy, multi-view data and monocular videos are first used for multi-view synthesis training, fully utilizing the prior knowledge learned through cross-view supervision. The completion training on monocular data also expands the amount of training data.
-
In terms of experimental results, relatively comprehensive evaluations are conducted on three datasets, with particularly significant improvements in severe occlusion scenarios. The ablation experiments also demonstrate the effectiveness of multi-view information.
Critical weaknesses
-
Lack of direct evaluation of the quality of generative multi-view reconstruction. This paper uses multi-view generation to compensate for multi-view information for hand reconstruction, but the performance of multi-view prediction itself is relatively limited. Moreover, the authors also mention that there are some missing parts when constructing multi-view training data from a single view. It is suggested to supplement a comparative analysis between the multi-view reconstruction results and the hand reconstruction results.
-
The authors cite ViDiHand, which is highly related to this work, as both methods leverage pretrained video diffusion models and intermediate DiT features for hand reconstruction. However, the authors do not include ViDiHand in the comparative experiments.
-
The description of the MANO Decoder is unclear. Section 3.4 describes two independent Decoders, but the architecture diagram in Figure 2 is drawn as a single Decoder structure. It is unclear whether the two Decoders have similar structures or actually share some parameters. The authors are advised to clarify this. In addition, if there are two Decoders, it is suggested to supplement an ablation study comparing them with a single Decoder.
Questions for authors
It is suggested to clarify the Decoder architecture, as mentioned in Weakness 3.
4 comments
Official Response by Authors - Part 2
(continued...)
Comparison with ViDiHand
Thank you for pointing this out. Since the code and pretrained checkpoints of ViDiHand have not been released so far, we re-implemented the model according to the details described in its paper and trained it on the identical dataset used in our work across 8 NVIDIA A100 80GB GPUs for three days until convergence. We have added the experimental results and analysis to Appendix A.8 (Table 8) of the revised manuscript, with the quantitative comparisons on each benchmark presented below:
ARCTIC Dataset
| Method | FAcc ↑ | F1 ↑ | MPJPE-p (mm) ↓ | PA-MPJPE-p (mm) ↓ | MPVPE-p (mm) ↓ | PA-MPVPE-p (mm) ↓ | EPE-p (px) ↓ | GO-p (°) ↓ | CT-p (m) ↓ |
|---|---|---|---|---|---|---|---|---|---|
| ViDiHand | 0.9920 | 0.9964 | 18.51 | 8.86 | 17.89 | 8.59 | 20.49 | 13.23 | 0.0549 |
| HandCrafter (Ours) | 0.9949 | 0.9960 | 14.02 | 6.57 | 13.60 | 6.38 | 18.16 | 11.20 | 0.1593 |
DexYCB Dataset
| Method | FAcc ↑ | F1 ↑ | MPJPE-p (mm) ↓ | PA-MPJPE-p (mm) ↓ | MPVPE-p (mm) ↓ | PA-MPVPE-p (mm) ↓ | EPE-p (px) ↓ | GO-p (°) ↓ | CT-p (m) ↓ |
|---|---|---|---|---|---|---|---|---|---|
| ViDiHand | 0.5697 | 0.6342 | 19.84 | 10.76 | 19.01 | 10.24 | 26.00 | 15.26 | 0.0603 |
| HandCrafter (Ours) | 0.9754 | 0.9151 | 14.61 | 6.74 | 14.06 | 6.42 | 13.56 | 11.41 | 0.0430 |
HO-Cap Dataset
| Method | FAcc ↑ | F1 ↑ | MPJPE-p (mm) ↓ | PA-MPJPE-p (mm) ↓ | MPVPE-p (mm) ↓ | PA-MPVPE-p (mm) ↓ | EPE-p (px) ↓ | GO-p (°) ↓ | CT-p (m) ↓ |
|---|---|---|---|---|---|---|---|---|---|
| ViDiHand | 0.6099 | 0.8634 | 22.23 | 7.70 | 21.35 | 7.44 | 25.71 | 16.94 | 0.0666 |
| HandCrafter (Ours) | 0.9670 | 0.9801 | 21.09 | 8.27 | 20.25 | 7.97 | 24.44 | 15.79 | 0.0652 |
Furthermore, we have included qualitative comparisons with ViDiHand in Appendix A.8 (Figure 7) as well as on our project page. Together, these experimental results indicate that our method achieves superior overall performance compared to ViDiHand, which underscores the value of leveraging multi-view generative priors for 3D hand pose estimation.
Official Response by Authors - Part 3
(continued...)
Dual-decoder Design
Thank you for your question and constructive suggestion. To clarify, in our implementation, the two decoders ( and ) only share a similar architectural structure and do not share any parameters.
Furthermore, we have added an ablation study on the dual-decoder design in Appendix A.10 of the revised manuscript. Specifically, we remove the per-view decoder and retain only the aggregation decoder (One Decoder). Instead of predicting the global orientation and translation in each viewpoint's camera coordinate system via , we modify to directly output and in the camera coordinate system of the input view. The quantitative comparison (Table 11) is reported below:
| Method | FAcc ↑ | Recall ↑ | F1 ↑ | MPJPE-p (mm) ↓ | PA-MPJPE-p (mm) ↓ | MPVPE-p (mm) ↓ | PA-MPVPE-p (mm) ↓ | EPE-p (px) ↓ | GO-p (°) ↓ | CT-p (m) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| One Decoder | 0.8954 | 0.9492 | 0.9472 | 44.60 | 19.26 | 43.24 | 18.67 | 84.35 | 37.42 | 0.5762 |
| Dual Decoder | 0.9572 | 0.9812 | 0.9794 | 35.40 | 15.80 | 34.29 | 15.33 | 50.58 | 27.11 | 0.3943 |
These experimental results validate the effectiveness of our dual-decoder design. This is because the two decoders naturally decouple the parameters shared across viewpoints (such as hand shape and pose ) from per-view localization ( and ). Supervising hand localization in each viewpoint's corresponding camera coordinate system in a per-view manner both reduces the learning difficulty and provides denser supervision signals across views.
Dear Reviewer 8q11,
Thank you for your valuable time and constructive feedback. As the discussion period is approaching its end, we would like to kindly follow up and hope that our responses have adequately addressed your concerns. We would greatly appreciate any further feedback you may have.
Thank you again for your time and consideration.
Best regards, The Authors
Official Response by Authors - Part 1
We appreciate the insightful comments and constructive feedback, and hope our responses below address the reviewer's concerns.
Analysis of Video Reconstruction Quality
Thanks for your suggestion. We have investigated the quality of the generated novel-view videos and the performance of 3D hand motion reconstruction across different denoising steps. This analysis has been added to Appendix A.7 of the revised manuscript, with qualitative visualizations shown in Figure 6 and quantitative results (Table 9) reported below:
As the number of denoising steps increases, the generated frames transition from blurry to sharp with progressively clearer details (Figure 6), which is consistently reflected by the decrease in both LPIPS and Hand-crop LPIPS. Note that PSNR and SSIM show an opposite trend because pixel-level similarity metrics inherently favor blurry, over-smoothed images that average out high-frequency uncertainty, whereas sharper details synthesized at larger denoising steps may incur slight pixel-wise misalignments with the ground truth.
Meanwhile, 3D hand pose reconstruction performance remains largely insensitive to the visual quality of the generated frames across different denoising steps. Precisely because a single denoising step already provides sufficient spatial cues for accurate hand motion reconstruction, our model only requires a single forward pass during inference, enabling the favorable inference cost reported in Appendix A.5.