acceptodds
Under review as a conference paper at ICLR 2027

Hand in Silhouette: Synthetic-to-Real Multi-View Hand Reconstruction via Binary Masks

17 views

Abstract

Calibrated multi-view hand reconstruction has advanced rapidly, yet the effectiveness of existing methods depends strongly on real captured data, which is costly to acquire, limited in hand-geometry diversity, and often affected by imperfect annotations. Synthetic data offers a scalable alternative with accurate supervision, but a substantial synthetic-to-real appearance gap hinders its use. To address this, we introduce **Hand in Silhouette (SilHand)**, a calibrated multi-view hand reconstruction framework that operates on binary hand masks—obtainable at low cost either automatically or manually—to reduce the synthetic-to-real observation gap. Rather than relying on RGB features for 3D lifting, SilHand projects metric 3D hypotheses into each view to query and fuse mask-ray evidence for hand reconstruction. This mask-centric formulation enables training on a large-scale synthetic dataset comprising 46.8k grasping poses and 60k free-hand samples, providing broad coverage of hand geometry while directly transferring to real captured hand geometry. Across five real-world benchmarks, SilHand improves relative reconstruction accuracy from **49.2%** to **63.4%** using masks rendered from reference hands. Moreover, reconstruction accuracy improves consistently as the synthetic training set grows, indicating that broader geometric coverage translates into better real-world performance. While achieving such diversity is difficult with real captured data, it becomes practical under our mask-centric formulation. Finally, SilHand remains robust under mask corruption, with reconstruction error increasing by at most 1.63 mm. Together, these results support binary masks as an effective interface for learning hand reconstruction from synthetic geometry.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D hand reconstruction, multi-view geometry, binary hand masks, synthetic data, synthetic-to-real transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BN82OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SilHand, a multi-view 3D hand reconstruction method that bypasses the synthetic-to-real appearance gap by throwing away RGB features like color, texture, and lighting, and relying entirely on binary masks. The core idea is to train a model on synthetic binary masks—which are easy to scale up and offer perfect geometric supervision—and then apply it to real-world multi-view setups where camera intrinsics/extrinsics are known. Specifically, the pipeline first aggregates multi-view mask features and Plücker-ray features onto a metric 3D grid via Point-Wise Multi-View Feature Aggregation. Then, a Global-to-Canonical Hand Reconstruction module estimates the global pose from palm joints, maps the representations to a canonical space to decode the articulation, and projects them onto the MANO shape basis. The model is trained on 60k free-hand samples and 46.8k grasping configurations, with hand-shape statistics and camera distributions drawn from real datasets. For evaluation, the authors test 3D reconstruction accuracy on five datasets using reference-rendered masks. To test on real images (with SAM3 segmentations), they measure the silhouette IoU on held-out views on two datasets.

Critical strengths
  1. The mask-centric training route is a very practical and elegant way to avoid the sim-to-real domain gap. The scaling behavior is convincing: as the training set scales up from 1.7k to 106.8k, the PA-V error steadily decreases (Figure 2b), with Appendix A.3 confirming a consistent 1,000,000-step budget across all scales. More importantly, the real-world evaluation using SAM3 inputs is quite strong. The model achieves median held-out-view IoUs of 81.9% and 83.1% on ARCTIC-MV and InterHand-MV, outperforming POEMv2 (79.0% and 81.8%), which uses RGB inputs from the same views. Since the evaluation cameras are withheld from both models, this is a solid proof of projection consistency rather than just overfitting input silhouettes.

  2. The proposed structured geometric reconstruction is well-designed and supported by thorough ablations under controlled inputs. In Table 9, we can see that removing canonicalization or reverting to mean pooling leads to a clear drop in both MPVPE and PA-V (e.g., SilHand's clean-mask MPVPE/PA-V of 2.86/1.92 mm degrades to 3.18/2.24 mm without canonicalization, and to 3.01/2.06 mm with mean pooling). These variants also degrade similarly under corruptions C1–C4, and the improvements in PA-V show that canonicalization benefits more than just global alignment. Furthermore, comparing SilHand to a baseline of Visual Hull + MANO fitting (which has access to ground-truth global pose and shape) shows that SilHand's learned geometric reconstruction performs significantly better across all five datasets. This validates the effectiveness of the proposed learned components.

Critical weaknesses
  1. My main concern is the decision to discard all RGB appearance cues at the reconstruction stage. Appearance is not just a source of domain shift; it also carries useful information about the hand. For example, visible boundaries between overlapping fingers, nail locations, and shading can help distinguish finger configurations and surface orientation, but these cues are lost in a binary mask. Multiple calibrated views clearly reduce ambiguity, as discussed in Section 3.2, but they do not necessarily remove it when different poses produce similar silhouettes across the available views. In those cases, the model has to rely more on its learned hand prior to choose a pose. A plausible hand is not necessarily the hand that was actually observed. The low 3D errors in Table 1 show that masks contain substantial useful information, so I am not arguing that mask-only reconstruction cannot work. What I find missing is a direct assessment of the information it gives up: the paper does not test cases with similar multi-view silhouettes but different 3D poses, or provide a controlled comparison showing whether retaining some appearance cues helps. Without this, I am not convinced that removing appearance altogether is a well-supported trade-off for recovering full hand articulation, rather than mainly a convenient way to simplify synthetic training.

  2. The real-input evaluation does not resolve this concern. In Table 1, SilHand receives masks rendered from the reference hand geometry, so its input contours are consistent with the 3D target by construction. This is a useful controlled test, but the comparison with RGB methods on real images does not isolate the benefit or cost of discarding appearance. When using actual SAM3 inputs, Table 2 reports held-out-view silhouette IoU rather than 3D reconstruction errors. The held-out views make this a meaningful test of cross-view consistency, but matching another silhouette still does not tell us whether the finger articulation is correct when several poses have similar projections. I therefore find the evidence stronger for recovering silhouette-consistent hands than for recovering the actual 3D hand state from real observations. This is especially important here because the information omitted by the model is precisely what could help distinguish some of those alternatives.

Questions for authors
  1. Within the intended calibrated multi-view setting, can you examine cases where different hand poses have similar masks across all input views, and report how SilHand distinguishes them? A targeted analysis of overlapping fingers or configurations whose differences mostly lie inside the silhouettes would help. Alternatively, a small controlled comparison between mask-only reconstruction and a version that also uses appearance cues, with matched views, training data, and supervision, would help assess the cost of removing appearance. I am not asking the method to work outside its stated multi-view setting, but to clarify this trade-off within it.

  2. Can you report MPVPE/MPJPE and PA-V for real SAM3 inputs against a same-view RGB baseline on a subset with reasonably reliable 3D references? I understand the annotation concerns raised in the paper, so a carefully checked subset with those uncertainties made explicit would be useful. Results broken down by finger visibility or overlap would be particularly informative: do the gains in silhouette IoU also correspond to more accurate 3D articulation in these cases?

6 comments

Thank you for your insightful review and for highlighting the inherent limitations of mask-only observations. To further investigate this issue, we conducted additional analyses on the PALM dataset [1], which provides high-precision 3D scans of hand surfaces. Specifically, we evaluate 3D reconstruction accuracy using real-world masks predicted by SAM3 on the PALM dataset and further analyze the reconstruction ambiguities arising from mask-based observations based on the synthetic dataset.

Q1. Different articulations with nearly identical multi-view masks

Thank you for this insightful question. We agree that the inherent ambiguity of mask-based reconstruction is an important limitation that warrants further investigation. To better understand how the distinguishability of input masks affects SilHand's ability to recover articulation changes, we conducted an additional controlled experiment using randomly sampled hand articulations from the synthetic test set under PALM's seven-camera configuration.

For each articulation, we perturb one finger's joint parameter in a randomly selected positive or negative direction, while keeping hand shape, global pose, other joints, and camera parameters fixed. We consider two groups:

(i) Nearly identical masks: Small articulation perturbations of 5$^\circ$-10$^\circ$, with mask IoU ≥\ge 0.99 in every camera view before and after perturbation. These cases represent highly ambiguous observations, where articulation changes introduce almost no distinguishable silhouette information.

(ii) Distinguishable masks: Larger articulation perturbations of 30$^\circ$-45$^\circ$, with mask IoU ≤\le 0.90 in at least one camera view. These cases represent scenarios where at least one view provides additional geometric evidence of the articulation change.

All configurations satisfy anatomical joint limits and self-collision constraints. We randomly select 5000 valid perturbed instances (1000 perturbations per finger) for each group and evaluate fingertip position errors before and after Procrustes alignment (PA).

To directly measure articulation discrimination, we evaluate whether the predicted fingertip displacement between the original and perturbed poses agrees with the ground-truth displacement. Let p0,p1\mathbf{p}_0,\mathbf{p}_1 denote the ground-truth fingertip positions before and after perturbation, and p^0,p^1\hat{\mathbf{p}}_0,\hat{\mathbf{p}}_1 their corresponding predictions. We define the relative displacement-vector error as

e=∥(p^1−p^0)−(p1−p0)∥2∥p1−p0∥2×100.e = \frac{\left\|(\hat{\mathbf{p}}_1-\hat{\mathbf{p}}_0)-(\mathbf{p}_1-\mathbf{p}_0) \right\|_2}{\|\mathbf{p}_1-\mathbf{p}_0\|_2} \times 100.

Importantly, we introduce the relative displacement-vector error to normalize for differences in perturbation magnitude between the two groups.

MetricNearly identical masksDistinguishable masks
Relative displacement-vector error before PA (%)29.707.18
Relative displacement-vector error after PA (%)25.655.27

For nearly identical masks, SilHand exhibits a relative fingertip displacement error of 29.70% (25.65% after PA), indicating difficulty in recovering articulation changes when the input silhouettes provide insufficient discriminative information. In contrast, when at least one camera view captures distinguishable silhouette changes, the relative error decreases substantially to 7.18% (5.27% after PA), demonstrating that SilHand can effectively respond to informative mask variations and recover the corresponding fingertip motion. These findings highlight both SilHand's sensitivity to observable articulation changes and the inherent ambiguity of mask-only reconstruction when different articulations produce nearly identical silhouettes.

We agree with the reviewer's insightful suggestion that incorporating appearance cues could help resolve such ambiguities. We believe that retaining binary masks as a robust geometric representation while introducing complementary appearance descriptors is a promising direction for improving reconstruction under ambiguous silhouettes. However, our current synthetic-data generation pipeline does not incorporate appearance information, and extending it to support appearance-based supervision would require substantial additional development and evaluation that may be beyond the scope of this work. To clarify this trade-off, we have explicitly discussed this limitation and identified the integration of complementary appearance cues as an important direction for future research in the revised manuscript.

Q2. 3D reconstruction accuracy under real masks

Thank you for highlighting the important distinction between 2D mask consistency and 3D articulation accuracy. We agree that higher held-out-view mask IoU does not necessarily imply more accurate 3D hand reconstruction. Therefore, it is important to verify whether improvements in silhouette agreement are accompanied by corresponding improvements in 3D geometric accuracy.

To directly investigate this question under real-world observations, we additionally evaluate SilHand on the PALM dataset [1], which provides high-precision scanned hand surfaces as 3D ground truth. We evaluate all 12,908 frames from 263 subjects, using SAM3-predicted masks for SilHand and RGB images for the official POEMv2 model, under the same calibrated camera configuration.

We report the mean prediction-to-scan surface distance in millimeters without Procrustes alignment. Unlike MPVPE and MPJPE, which require corresponding MANO mesh vertices or joints, this metric directly measures geometric agreement with the high-precision scanned surfaces without introducing errors from MANO fitting.

MethodSurface distanceIoU
SilHand3.10983.1
POEMv23.49880.6

The results show that the improvement in silhouette IoU is accompanied by a corresponding improvement in 3D reconstruction accuracy. Specifically, SilHand achieves a 2.5-percentage-point higher silhouette IoU and an 11.1% lower mean 3D surface distance compared with POEMv2. This demonstrates that SilHand's advantage is not limited to better silhouette agreement but also extends to more accurate reconstruction of the underlying 3D hand geometry.

[1] Fan, Zicong, et al. "Palm: A dataset and baseline for learning multi-subject hand prior." 2026 International Conference on 3D Vision (3DV). IEEE, 2026.

Thanks for the detailed responses and additional experiments. The PALM evaluation addresses an important gap in the original submission: it provides direct 3D evidence with real predicted masks, rather than relying only on held-out-view silhouette IoU. I also appreciate the controlled articulation experiment, which makes the limitations of mask-only observations more concrete and shows that the model responds better when the silhouettes contain distinguishable changes. I still have some concerns: prediction-to-scan surface distance does not directly establish correct finger articulation, and the ambiguity experiment does not tell us how much appearance cues would help in those cases. Results broken down by finger overlap or visibility would strengthen the evidence further.

Reviewer_K256OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a multi-view reconstruction method from binary masks, which aims to solve the synthetic-to-real problem in Multi-view 3D pose estimation. While previous methods mainly focus on photorealistic rendering to use highly realistic hand images to improve the estimation performance, SilHand focuses on mask-based reconstruction to reduce synthetic-to-real gap. Using multi-view corrupted masks and Plucker ray as inputs, SilHand reconstructs the corresponding 3D hand. It synthesizes 3D hand geometry supervisions to train SilHand in two ways: (1) free poses are generated via MANO (2) grasping poses are collected from Dexonomy dataset. Experiments across five multi-view hand reconstruction datasets validate its effectiveness.

Critical strengths
  1. SilHand provides a new perspective for solving the synthetic-to-real gap in multi-view 3D hand reconstruction. The synthetic-to-real gap can be caused by rendering realism or pose/shape distribution. SilHand as it says, sidesteps the photorealistic rendering and realizes reconstruction from masks, which mitigates the gap from an intermediate shared modality.
  2. The experimental results are promising. Using around 100k synthetic geometry data to train SilHand, it shows strong performance on multiple benchmarks. The metric results seem good and competitive.
  3. The overall writing and presentation are good. The paper is easy to follow and the provided experimental details are overall clear.
Critical weaknesses
  1. This method can only be used in the multi-view reconstruction scenario since only masks as inputs can have more ambiguity when applied to single-view reconstruction.
  2. More experimental details are encouraged to provide. Is SilHand additionally trained with the training sets of different benchmarks when evaluating on their test sets, or just doing testing?
  3. More discussions regarding the composition of the synthetic geometry would be better. For free-hand samples and grasping samples, which is more important? What ratio should we choose to generate to produce better results? The importance of camera parameter perturbation is also encouraged to discuss.
Questions for authors

For Fig.2(b), which test set does SilHand evaluate on? Is the performance converging at the 106.8k number of samples? Why is it converging so fast, since 100k is not a large size for synthetic data regarding the hand reconstruction scenatio?

3 comments

We thank the reviewer for the positive assessment and for highlighting the need for further clarification regarding our evaluation protocol, scaling behavior, and synthetic-data design.

Q1. Experiment protocol

Thank you for raising this question. We agree that these important experimental details should be clearly stated. To clarify, SilHand is trained exclusively on synthetic data and is not additionally trained or fine-tuned on the real training split of any benchmark. The trained model is directly evaluated on the designated evaluation split of each benchmark. We have included these important experimental details in the revised manuscript.

For the question of the evaluation set in Fig.2 (b), we would like to clarify that all experiments are evaluated on the same test set from the five real-world datasets. We have added this clarification to the revised manuscript.

Q2. Scaling beyond 106.8k samples

Thank you for raising this important question. We agree that this interesting finding warrants further investigation. To explore this, we conducted an additional experiment in which we doubled the training dataset size while keeping the number of training steps unchanged. Our results show that further increasing the dataset size does not lead to noticeable performance improvements.

Training samplesMedian PA-V
106.8k1.61
213.6k1.74

We hypothesize that this saturation is related to the inherently constrained and highly coupled nature of hand articulation, which results in a relatively low-dimensional space of plausible hand configurations. Consequently, increasing the number of samples drawn from the same distribution may increasingly introduce redundant configurations rather than substantially expanding geometric coverage. In contrast, although real-world video datasets contain many frames, temporally adjacent frames often exhibit highly similar hand configurations, which can be readily interpolated by the learned network and therefore do not necessarily provide equivalent amounts of new geometric information.

Q3. Free-hand and grasping sample composition

Thank you for this insightful question. Our initial choice of a 1:1 sampling ratio was motivated by the complementary characteristics of free-hand and grasping configurations. Specifically, free-hand samples capture hand poses without object interaction, where occlusion primarily arises from the hand itself (self-occlusion). In contrast, grasping samples capture hand-object interactions, where the grasped objects can introduce additional and potentially severe occlusions. As both scenarios are important for achieving comprehensive geometric coverage and robust hand reconstruction, we assume them equally important and therefore adopt a 1:1 ratio during the initial sampling stage.

After applying different anatomical and collision-based filtering procedures to the two categories, we obtain 60k free-hand samples (56.2%) and 46.8k grasping configurations (43.8%). The slight deviation from the initial 1:1 ratio is thus a consequence of the different filtering procedures and sample retention rates, rather than an intentional imbalance. We have clarified this sampling rationale and the resulting distribution in the revised manuscript.

We acknowledge that a systematic investigation of how different sampling ratios affect reconstruction performance could provide further insights into the optimal balance between free-hand and grasping configurations. We have explicitly acknowledged this limitation in the revised manuscript and identified the exploration of more effective sampling strategies as an important direction for future work.

Q4. Camera-parameter perturbation

Thank you for raising this important point. We agree that evaluating sensitivity to camera calibration errors is important for understanding the robustness and operating range of SilHand.

Keeping input masks and reference hand geometry fixed, we jointly perturbed camera translation, rotation, focal length, and principal point. L0 denotes unperturbed calibration.

Perturbation magnitudeL1L2L3L4
Translation (mm)0.100.250.501.00
Rotation (∘^{\circ})0.0250.050.100.25
Focal-length change (%)0.100.250.501.00
Principal-point displacement (% of image short side)0.0250.050.100.25
DatasetL0L1L2L3L4
DexYCB-MV2.072.092.052.313.62
HO3D-MV2.792.792.822.903.19
OakInk-MV2.953.093.203.585.42
ARCTIC-MV2.993.063.234.208.05
InterHand-MV3.523.553.623.694.54

We acknowledge the dependence of SilHand on accurate camera parameters, as calibration errors can lead to inaccurate 3D-to-2D projections and misaligned cross-view geometric evidence. We have added this limitation to the revised manuscript and discussed improving robustness to camera calibration uncertainty as an important direction for future work.

I would like to thank the authors for their rebuttal. It addresses most of my questions. I will maintain my original positive score.

Reviewer_eMnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The method uses binary masks as input, projects 3D mesh points onto multiple camera views to extract mask features and Plücker ray features, aggregates them via cross-view attention and 3D CNNs, and recovers hand pose and shape through global pose estimation, canonical space transformation, and a token-query decoder. These components have direct technical precedents in geometric reconstruction and multi-view learning. Binary silhouettes constraining 3D shapes, multi-view geometry fusion with known camera parameters, mesh-based feature aggregation, and MANO parametric decoding are not novel individually. The paper needs to clarify which key designs are original contributions versus combinations of existing components. In particular, the authors characterize their main contribution as a "mask-centric reconstruction architecture," yet fail to articulate its essential differences from existing multi-view reconstruction frameworks. Is the performance gain primarily due to binary mask input representation, cross-view attention, 3D mesh aggregation, global-to-canonical decoding, or larger synthetic training data? Current experiments cannot disentangle these factors.

Critical strengths
  1. The paper addresses an important problem in multi-view hand reconstruction and presents a coherent framework for reconstructing 3D hand geometry from binary silhouettes.

  2. The proposed cross-view feature fusion mechanism provides a reasonable way to aggregate geometric information across camera views, while the canonicalization and decoding stages offer a structured approach to pose and shape estimation.

  3. The method is evaluated on multiple established benchmarks, and the experiments suggest promising reconstruction performance under the tested settings.

Critical weaknesses
  1. The paper's core claim is that models trained on synthetic data can effectively transfer to real hand reconstruction. However, in the main annotation-oriented evaluation, the authors render input masks from dataset-annotated reference hand geometry (preserving object occlusions) and compare reconstructions against the same reference geometry.

  2. Table 1 compares SilHand with RGB-based methods (POEM, MLPHand, POEMv2, JUMP-Hand) plus Visual Hull as a mask baseline. The issue: these methods differ in input modality, training data format, and information used. SilHand uses binary masks and large-scale synthetic hand geometry for training, while comparison methods use RGB features with different training data and supervision. Directly comparing final reconstruction errors without isolating the effects of input representation, training data scale, synthetic supervision, and network architecture makes it unclear whether gains come from the proposed architecture or from specialized training/evaluation settings.

  3. The authors construct a synthetic dataset with 60k free-hand samples and 46.8k grasp pose samples, showing decreasing reconstruction error as training samples increase. This demonstrates that, under the current generation and training configuration, more samples help, but doesn't prove improvement comes primarily from expanded geometric coverage. Increasing sample count may simultaneously alter pose distribution, shape distribution, grasp type coverage, and effective sample diversity during training. The paper visualizes geometric distribution using the first two principal components, but such 2D projections cannot adequately measure high-dimensional hand pose/shape space coverage or confirm whether new samples fill gaps in real data's most lacking geometric patterns.

  4. Abstract, introduction, experimental analysis, and conclusion repeatedly emphasize that binary masks reduce synthetic-to-real domain gaps and that expanding synthetic geometric coverage improves performance, but largely restate conclusions without deeper analysis. Some contribution claims are grand but lack corresponding experimental support. The paper does exhibit template-like phrasing and repetitive arguments.

Questions for authors
  1. How does the proposed method compare with existing approaches when all methods are trained using the same data, supervision, and computational budget?

  2. Since the main evaluation uses masks rendered from reference hand geometry, how well does the method perform when using masks predicted directly from real RGB images?

  3. Can the authors provide more comprehensive quantitative evaluations of 3D reconstruction accuracy under realistic conditions, rather than relying primarily on silhouette-based metrics?

  4. How sensitive is the method to the number of camera views, camera calibration errors, and inaccurate or noisy silhouettes?

  5. To what extent does the performance improvement come from the proposed architecture rather than the large-scale synthetic training data? Can this be verified through controlled experiments?

3 comments

We thank the reviewer for identifying the need to disentangle architectural and data effects and to strengthen evaluation under realistic inputs. We address these concerns below.

Q1. Comparison under the same training data, supervision, and computational budget

Thank you for this valuable question. We agree that additional controlled comparisons are important for a fair evaluation. To this end, we adapted the open-source POEMv2 architecture to support binary masks and trained it using the same synthetic dataset as SilHand, with a matched training budget of 20 GPU hours on a single NVIDIA RTX 5090. For a fair comparison, both methods were evaluated using clean masks under the same five-dataset evaluation protocol.

MethodMPVPEMPJPE
SilHand3.333.39
POEMv2-Mask54.4852.89

Q2-Q3. Reconstruction from real predicted masks and quantitative 3D accuracy

Thank you for raising this important point. We agree that evaluating reconstruction from real-image-predicted masks is essential for fair evaluation. To address this, we additionally evaluated all samples from 263 subjects in PALM [1], which provides high-precision scanned 3D hand surfaces. SilHand uses SAM3-generated masks, while the official POEMv2 uses RGB inputs from the same cameras. We report mean prediction-to-scan surface distance in millimeters, avoiding introducing MANO-fitting error into the reference.

MethodSurface distance
SilHand3.109
POEMv23.498

This supplies direct quantitative evidence from real predicted masks and independently scanned surfaces, complementing the annotation-oriented and held-out-view results in the paper.

[1] Fan, Zicong, et al. "Palm: A dataset and baseline for learning multi-subject hand prior." 2026 International Conference on 3D Vision (3DV). IEEE, 2026.

Q4. Sensitivity to view count, calibration errors, and mask noise

Thank you for raising this important point. We agree that evaluating sensitivity to input views, camera calibration errors, and mask noise is important for understanding the robustness and operating range of SilHand. To address this, we have added three complementary analyses.

View count. We evaluated the frozen SilHand model on nested subsets of available cameras using rendered masks. The following table reports MPVPE in millimeters.

Dataset2 views3 views4 views5 views6 views7 views8 views
DexYCB-MV34.478.234.512.782.442.212.07
HO3D-MV11.234.443.492.79---
OakInk-MV12.503.912.95----
ARCTIC-MV74.9432.1310.005.813.543.142.99
InterHand-MV11.115.964.624.223.953.713.52

We acknowledge this as a limitation of SilHand, which relies on multi-view masks to provide sufficient geometric constraints for accurate reconstruction. We have clarified this limitation and included additional qualitative results in the "Discussion and Limitations" section of the revised manuscript.

Calibration. Keeping input masks and reference hand geometry fixed, we jointly perturbed camera translation, rotation, focal length, and principal point. L0 denotes unperturbed calibration.

Perturbation magnitudeL1L2L3L4
Translation (mm)0.100.250.501.00
Rotation (∘^{\circ})0.0250.050.100.25
Focal-length change (%)0.100.250.501.00
Principal-point displacement (% of image short side)0.0250.050.100.25
DatasetL0L1L2L3L4
DexYCB-MV2.072.092.052.313.62
HO3D-MV2.792.792.822.903.19
OakInk-MV2.953.093.203.585.42
ARCTIC-MV2.993.063.234.208.05
InterHand-MV3.523.553.623.694.54

We acknowledge the dependence of SilHand on accurate camera parameters, as calibration errors can lead to inaccurate 3D-to-2D projections and misaligned cross-view geometric evidence. We have added this limitation to the revised manuscript and discussed improving robustness to camera calibration uncertainty as an important direction for future work.

Mask noise. We have added evaluations under four progressively more severe mask-corruption levels (C1-C4) in Tables 1 and 3 of the revised manuscript. These levels progressively increase corruption severity by combining forearm extension, fragment dropout, morphological and boundary perturbations, and localized boundary artifacts.

Q5. Architectural contributions versus synthetic-data scale

Thank you for this important question. To disentangle architectural contributions from the effects of synthetic-data scale, we have included controlled ablation studies in Appendix A.4 and Table 9. All variants use the same full synthetic training set and training budget, with one architectural component modified at a time. The consistent performance degradation when replacing cross-view attention, removing joint-based global-pose estimation, or omitting canonicalization demonstrates the individual contributions of these components under controlled training conditions.

Other questions

Thank you for the other insightful observations. In the revised manuscript, we clarify that the increased geometric coverage comes primarily from sampling more diverse hand poses. We have also expanded our analysis in the revised manuscript to provide a more comprehensive discussion.

Thank you for the rebuttal. It was helpful but ultimately did not change my assessment. I am keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.