- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SilHand, a multi-view 3D hand reconstruction method that bypasses the synthetic-to-real appearance gap by throwing away RGB features like color, texture, and lighting, and relying entirely on binary masks. The core idea is to train a model on synthetic binary masks—which are easy to scale up and offer perfect geometric supervision—and then apply it to real-world multi-view setups where camera intrinsics/extrinsics are known. Specifically, the pipeline first aggregates multi-view mask features and Plücker-ray features onto a metric 3D grid via Point-Wise Multi-View Feature Aggregation. Then, a Global-to-Canonical Hand Reconstruction module estimates the global pose from palm joints, maps the representations to a canonical space to decode the articulation, and projects them onto the MANO shape basis. The model is trained on 60k free-hand samples and 46.8k grasping configurations, with hand-shape statistics and camera distributions drawn from real datasets. For evaluation, the authors test 3D reconstruction accuracy on five datasets using reference-rendered masks. To test on real images (with SAM3 segmentations), they measure the silhouette IoU on held-out views on two datasets.
Critical strengths
-
The mask-centric training route is a very practical and elegant way to avoid the sim-to-real domain gap. The scaling behavior is convincing: as the training set scales up from 1.7k to 106.8k, the PA-V error steadily decreases (Figure 2b), with Appendix A.3 confirming a consistent 1,000,000-step budget across all scales. More importantly, the real-world evaluation using SAM3 inputs is quite strong. The model achieves median held-out-view IoUs of 81.9% and 83.1% on ARCTIC-MV and InterHand-MV, outperforming POEMv2 (79.0% and 81.8%), which uses RGB inputs from the same views. Since the evaluation cameras are withheld from both models, this is a solid proof of projection consistency rather than just overfitting input silhouettes.
-
The proposed structured geometric reconstruction is well-designed and supported by thorough ablations under controlled inputs. In Table 9, we can see that removing canonicalization or reverting to mean pooling leads to a clear drop in both MPVPE and PA-V (e.g., SilHand's clean-mask MPVPE/PA-V of 2.86/1.92 mm degrades to 3.18/2.24 mm without canonicalization, and to 3.01/2.06 mm with mean pooling). These variants also degrade similarly under corruptions C1–C4, and the improvements in PA-V show that canonicalization benefits more than just global alignment. Furthermore, comparing SilHand to a baseline of Visual Hull + MANO fitting (which has access to ground-truth global pose and shape) shows that SilHand's learned geometric reconstruction performs significantly better across all five datasets. This validates the effectiveness of the proposed learned components.
Critical weaknesses
-
My main concern is the decision to discard all RGB appearance cues at the reconstruction stage. Appearance is not just a source of domain shift; it also carries useful information about the hand. For example, visible boundaries between overlapping fingers, nail locations, and shading can help distinguish finger configurations and surface orientation, but these cues are lost in a binary mask. Multiple calibrated views clearly reduce ambiguity, as discussed in Section 3.2, but they do not necessarily remove it when different poses produce similar silhouettes across the available views. In those cases, the model has to rely more on its learned hand prior to choose a pose. A plausible hand is not necessarily the hand that was actually observed. The low 3D errors in Table 1 show that masks contain substantial useful information, so I am not arguing that mask-only reconstruction cannot work. What I find missing is a direct assessment of the information it gives up: the paper does not test cases with similar multi-view silhouettes but different 3D poses, or provide a controlled comparison showing whether retaining some appearance cues helps. Without this, I am not convinced that removing appearance altogether is a well-supported trade-off for recovering full hand articulation, rather than mainly a convenient way to simplify synthetic training.
-
The real-input evaluation does not resolve this concern. In Table 1, SilHand receives masks rendered from the reference hand geometry, so its input contours are consistent with the 3D target by construction. This is a useful controlled test, but the comparison with RGB methods on real images does not isolate the benefit or cost of discarding appearance. When using actual SAM3 inputs, Table 2 reports held-out-view silhouette IoU rather than 3D reconstruction errors. The held-out views make this a meaningful test of cross-view consistency, but matching another silhouette still does not tell us whether the finger articulation is correct when several poses have similar projections. I therefore find the evidence stronger for recovering silhouette-consistent hands than for recovering the actual 3D hand state from real observations. This is especially important here because the information omitted by the model is precisely what could help distinguish some of those alternatives.
Questions for authors
-
Within the intended calibrated multi-view setting, can you examine cases where different hand poses have similar masks across all input views, and report how SilHand distinguishes them? A targeted analysis of overlapping fingers or configurations whose differences mostly lie inside the silhouettes would help. Alternatively, a small controlled comparison between mask-only reconstruction and a version that also uses appearance cues, with matched views, training data, and supervision, would help assess the cost of removing appearance. I am not asking the method to work outside its stated multi-view setting, but to clarify this trade-off within it.
-
Can you report MPVPE/MPJPE and PA-V for real SAM3 inputs against a same-view RGB baseline on a subset with reasonably reliable 3D references? I understand the annotation concerns raised in the paper, so a carefully checked subset with those uncertainties made explicit would be useful. Results broken down by finger visibility or overlap would be particularly informative: do the gains in silhouette IoU also correspond to more accurate 3D articulation in these cases?
6 comments
Q2. 3D reconstruction accuracy under real masks
Thank you for highlighting the important distinction between 2D mask consistency and 3D articulation accuracy. We agree that higher held-out-view mask IoU does not necessarily imply more accurate 3D hand reconstruction. Therefore, it is important to verify whether improvements in silhouette agreement are accompanied by corresponding improvements in 3D geometric accuracy.
To directly investigate this question under real-world observations, we additionally evaluate SilHand on the PALM dataset [1], which provides high-precision scanned hand surfaces as 3D ground truth. We evaluate all 12,908 frames from 263 subjects, using SAM3-predicted masks for SilHand and RGB images for the official POEMv2 model, under the same calibrated camera configuration.
We report the mean prediction-to-scan surface distance in millimeters without Procrustes alignment. Unlike MPVPE and MPJPE, which require corresponding MANO mesh vertices or joints, this metric directly measures geometric agreement with the high-precision scanned surfaces without introducing errors from MANO fitting.
| Method | Surface distance | IoU |
|---|---|---|
| SilHand | 3.109 | 83.1 |
| POEMv2 | 3.498 | 80.6 |
The results show that the improvement in silhouette IoU is accompanied by a corresponding improvement in 3D reconstruction accuracy. Specifically, SilHand achieves a 2.5-percentage-point higher silhouette IoU and an 11.1% lower mean 3D surface distance compared with POEMv2. This demonstrates that SilHand's advantage is not limited to better silhouette agreement but also extends to more accurate reconstruction of the underlying 3D hand geometry.
[1] Fan, Zicong, et al. "Palm: A dataset and baseline for learning multi-subject hand prior." 2026 International Conference on 3D Vision (3DV). IEEE, 2026.
Thanks for the detailed responses and additional experiments. The PALM evaluation addresses an important gap in the original submission: it provides direct 3D evidence with real predicted masks, rather than relying only on held-out-view silhouette IoU. I also appreciate the controlled articulation experiment, which makes the limitations of mask-only observations more concrete and shows that the model responds better when the silhouettes contain distinguishable changes. I still have some concerns: prediction-to-scan surface distance does not directly establish correct finger articulation, and the ambiguity experiment does not tell us how much appearance cues would help in those cases. Results broken down by finger overlap or visibility would strengthen the evidence further.
Thank you for your insightful review and for highlighting the inherent limitations of mask-only observations. To further investigate this issue, we conducted additional analyses on the PALM dataset [1], which provides high-precision 3D scans of hand surfaces. Specifically, we evaluate 3D reconstruction accuracy using real-world masks predicted by SAM3 on the PALM dataset and further analyze the reconstruction ambiguities arising from mask-based observations based on the synthetic dataset.
Q1. Different articulations with nearly identical multi-view masks
Thank you for this insightful question. We agree that the inherent ambiguity of mask-based reconstruction is an important limitation that warrants further investigation. To better understand how the distinguishability of input masks affects SilHand's ability to recover articulation changes, we conducted an additional controlled experiment using randomly sampled hand articulations from the synthetic test set under PALM's seven-camera configuration.
For each articulation, we perturb one finger's joint parameter in a randomly selected positive or negative direction, while keeping hand shape, global pose, other joints, and camera parameters fixed. We consider two groups:
(i) Nearly identical masks: Small articulation perturbations of 5$^\circ$-10$^\circ$, with mask IoU ≥ 0.99 in every camera view before and after perturbation. These cases represent highly ambiguous observations, where articulation changes introduce almost no distinguishable silhouette information.
(ii) Distinguishable masks: Larger articulation perturbations of 30$^\circ$-45$^\circ$, with mask IoU ≤ 0.90 in at least one camera view. These cases represent scenarios where at least one view provides additional geometric evidence of the articulation change.
All configurations satisfy anatomical joint limits and self-collision constraints. We randomly select 5000 valid perturbed instances (1000 perturbations per finger) for each group and evaluate fingertip position errors before and after Procrustes alignment (PA).
To directly measure articulation discrimination, we evaluate whether the predicted fingertip displacement between the original and perturbed poses agrees with the ground-truth displacement. Let p0,p1 denote the ground-truth fingertip positions before and after perturbation, and p^0,p^1 their corresponding predictions. We define the relative displacement-vector error as
e=∥p1−p0∥2∥(p^1−p^0)−(p1−p0)∥2×100.
Importantly, we introduce the relative displacement-vector error to normalize for differences in perturbation magnitude between the two groups.
For nearly identical masks, SilHand exhibits a relative fingertip displacement error of 29.70% (25.65% after PA), indicating difficulty in recovering articulation changes when the input silhouettes provide insufficient discriminative information. In contrast, when at least one camera view captures distinguishable silhouette changes, the relative error decreases substantially to 7.18% (5.27% after PA), demonstrating that SilHand can effectively respond to informative mask variations and recover the corresponding fingertip motion. These findings highlight both SilHand's sensitivity to observable articulation changes and the inherent ambiguity of mask-only reconstruction when different articulations produce nearly identical silhouettes.
We agree with the reviewer's insightful suggestion that incorporating appearance cues could help resolve such ambiguities. We believe that retaining binary masks as a robust geometric representation while introducing complementary appearance descriptors is a promising direction for improving reconstruction under ambiguous silhouettes. However, our current synthetic-data generation pipeline does not incorporate appearance information, and extending it to support appearance-based supervision would require substantial additional development and evaluation that may be beyond the scope of this work. To clarify this trade-off, we have explicitly discussed this limitation and identified the integration of complementary appearance cues as an important direction for future research in the revised manuscript.