acceptodds
Withdrawn from ICLR 2027

Human-Robot Alignment for Dexterous Hand Recognition with Geometry-aligned Synthetic Pairs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Anthropomorphic robot hands imitate human hand kinematics, but the appearance gap degrades the performance of SOTA human-hand detectors and pose estimators, while the scarcity of labeled robot-hand images limits finetuning and reduces accuracy for human hands. We present SynDex, which adapts pretrained human-hand models to robot hands without any real robot training images. SynDex renders depth and normal maps from annotated human hands and generates paired human-robot images from the same geometry. A vision-language model checks each candidate against its real reference, yielding 47K pixel-aligned human–robot pairs. A frozen human-domain teacher then supervises a student. Student features on each robot image regress teacher features on the paired human image, while a preservation term on real human images limits forgetting. SynDex improves the WiLoR's robot-hand detection AP from 9.55 to 38.45 on RealDex and from 2.93 to 60.66 on HRDexDB, and lowers the robot-hand PA-MPJPE of HaMeR and WiLoR by up to 1.85 mm, while keeping human-hand pose error within 0.4 mm of the original models on FreiHAND and HO3Dv2.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
dexterous robot hand pose estimation, dexterous robot hand detection, human-robot domain alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jGsZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a dexterous-hand recognition method that transfers a human-hand detector to dexterous hands via paired-data generation and feature alignment; experiments show that the method works.

Critical strengths

The paper is well formatted and reported results are reasonably good.

Critical weaknesses

Overall. The significance of this research remains unclear. Why is vision needed to estimate the pose of a dexterous hand, rather than reading proprioceptive or other onboard sensor data directly? Even if vision were used as ground truth for sensor calibration, the ground truth itself would need to be highly accurate—e.g., multi-view motion capture. As currently motivated, the value of the problem setting is not convincing.

The paper’s novelty is limited, and the proposed method fundamentally overlooks several important factors. The method largely combines established techniques and does not introduce a clearly new idea, finding, or formulation. The core component reads as a relatively straightforward data-augmentation strategy. There is no retargeting step, and the paper does not discuss morphological differences such as joint/link length discrepancies between human hands and dexterous hands. The training schedule and loss design appear equivalent to training a dexterous-hand detector directly with human-hand ground-truth supervision; the distinctive benefit of the proposed pipeline is therefore not well justified. The claim of being a cross-domain method seems overstated. The proposed method appears to degrade human-hand estimation accuracy. Given that dexterous-hand joint lengths differ from those of human hands, and as shown in Figure 4, the predicted keypoints often do not land on the actual joint positions. An ablation of baseline + L_align (without L_task) would be informative, the current ablation study is incomplete.

Questions for authors

Please refer to weaknesses. I would be willing to consider a higher recommendation If all of my concerns are adequately addressed.

Reviewer_qLyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SynDex, a vision backbone alignment framework that adapts pretrained human-hand detection/pose estimation models to dexterous robot hands. The method consists of two parts: (1) generating pose-aligned human-robot image pairs by conditioning a multi-conditional diffusion model on depth and normal maps rendered from annotated human hand meshes, followed by VLM-driven automatic filtering (~47K pairs); (2) a teacher-student feature alignment loss that maps robot-hand features into a frozen human feature space, plus a knowledge preservation term on real human images to prevent catastrophic forgetting. Experiments on RealDex and HRDexDB raise WiLoR's detection AP50 from 9.55 to 38.45 and from 2.93 to 60.66, reduce robot-hand PA-MPJPE by up to ~1.9mm, and largely preserve human-hand accuracy on FreiHAND/HO3D.

Critical strengths

The method proposed in the paper is somewhat novel, the paper is well organized, the English writing is clear, and sufficient supplementary material is provided.

Critical weaknesses
  1. A dexterous hand's skeleton can be obtained directly via forward kinematics from joint encoders and its known 3D model, without any visual estimation. The introduction only briefly mentions that proprioceptive sensing degrades under mechanical wear and sensor drift, but provides no concrete evidence or scenarios showing why visual perception is irreplaceable. For example: what is the typical magnitude of joint-angle error caused by encoder drift? Where does vision outperform encoders? In safety-critical interaction, how should vision complement rather than replace proprioception? Please strengthen this argument; otherwise the research motivation does not hold.
  2. In Fig. 1, Fig. 3, Tabs. 1 and 2, the authors only compare against HaMeR and WiLoR—human-hand models in zero-shot mode on robot hands. Due to the domain gap these baselines are near-failure to begin with, so outperforming broken baselines does not demonstrate the method's superiority. Please add comparisons against: (a) detectors/pose estimators fine-tuned on rendered/synthetic robot-hand data; (b) domain adaptation methods (e.g., HRAlign-style contrastive alignment); (c) a simple yet strong baseline of direct supervised fine-tuning on the generated data (task-only fine-tuning appears only in the Tab. 6 ablation on the validation set, not as a main-table baseline on the test sets).
  3. The concrete values and rationale for the three loss weights λ_align, λ_cons, λ_task in Eq. (3) are not clearly reported in the main text. Appendix Tab. 10 only ablates λ_align and λ_task (defaults 1.0 and 0.01); the value of λ_cons and its sensitivity analysis are missing. Please report all three default values, the justification for choosing them, and whether the weights are shared across the detection and pose tasks.
  4. Real-time performance is critical for robotic applications such as visual teleoperation and online imitation learning, yet the paper does not report inference time at all. Although the paper states that only the student backbone is deployed at inference (Fig. 2), please report FPS / per-frame latency for both detection and pose estimation, compared against the HaMeR/WiLoR baselines, to demonstrate that the method meets real-time requirements.
  5. Appendix A.6 states that pose estimation is evaluated using ground-truth crops and 3D joints. This means pose accuracy is measured on pre-cropped, pre-localized images, bypassing the detection stage and overestimating end-to-end accuracy relative to a real "detect-then-estimate" pipeline. Are the results in Tab. 2 based on GT crops? If so, please add end-to-end results using predicted bounding boxes for cropping.
  6. The detection gains are dramatic (AP50 from 2.93 to 60.66), but pose gains are modest: on RealDex, HaMeR improves only from 15.08 to 14.24mm (<1mm), and WiLoR by only 0.29mm; on FreiHAND, human-hand PA-MPJPE actually degrades from 5.5 to 5.9mm. Moreover, PA-MPJPE absorbs scale/translation errors through Procrustes alignment. How do the results look on unaligned MPJPE and F@5/F@15? Are the millimeter-level improvements above the noise floor introduced by annotation error and the training-free GMR retargeting used at evaluation?
Questions for authors
  1. A dexterous hand's skeleton can be obtained directly via forward kinematics from joint encoders and its known 3D model, without any visual estimation. The introduction only briefly mentions that proprioceptive sensing degrades under mechanical wear and sensor drift, but provides no concrete evidence or scenarios showing why visual perception is irreplaceable. For example: what is the typical magnitude of joint-angle error caused by encoder drift? Where does vision outperform encoders? In safety-critical interaction, how should vision complement rather than replace proprioception? Please strengthen this argument; otherwise the research motivation does not hold.
  2. In Fig. 1, Fig. 3, Tabs. 1 and 2, the authors only compare against HaMeR and WiLoR—human-hand models in zero-shot mode on robot hands. Due to the domain gap these baselines are near-failure to begin with, so outperforming broken baselines does not demonstrate the method's superiority. Please add comparisons against: (a) detectors/pose estimators fine-tuned on rendered/synthetic robot-hand data; (b) domain adaptation methods (e.g., HRAlign-style contrastive alignment); (c) a simple yet strong baseline of direct supervised fine-tuning on the generated data (task-only fine-tuning appears only in the Tab. 6 ablation on the validation set, not as a main-table baseline on the test sets).
  3. The concrete values and rationale for the three loss weights λ_align, λ_cons, λ_task in Eq. (3) are not clearly reported in the main text. Appendix Tab. 10 only ablates λ_align and λ_task (defaults 1.0 and 0.01); the value of λ_cons and its sensitivity analysis are missing. Please report all three default values, the justification for choosing them, and whether the weights are shared across the detection and pose tasks.
  4. Real-time performance is critical for robotic applications such as visual teleoperation and online imitation learning, yet the paper does not report inference time at all. Although the paper states that only the student backbone is deployed at inference (Fig. 2), please report FPS / per-frame latency for both detection and pose estimation, compared against the HaMeR/WiLoR baselines, to demonstrate that the method meets real-time requirements.
  5. Appendix A.6 states that pose estimation is evaluated using ground-truth crops and 3D joints. This means pose accuracy is measured on pre-cropped, pre-localized images, bypassing the detection stage and overestimating end-to-end accuracy relative to a real "detect-then-estimate" pipeline. Are the results in Tab. 2 based on GT crops? If so, please add end-to-end results using predicted bounding boxes for cropping.
  6. The detection gains are dramatic (AP50 from 2.93 to 60.66), but pose gains are modest: on RealDex, HaMeR improves only from 15.08 to 14.24mm (<1mm), and WiLoR by only 0.29mm; on FreiHAND, human-hand PA-MPJPE actually degrades from 5.5 to 5.9mm. Moreover, PA-MPJPE absorbs scale/translation errors through Procrustes alignment. How do the results look on unaligned MPJPE and F@5/F@15? Are the millimeter-level improvements above the noise floor introduced by annotation error and the training-free GMR retargeting used at evaluation?
Reviewer_PiibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles 3D hand pose estimation for anthropomorphic robot hands. To address the scarcity of training data, the paper proposes a method that uses a multi-conditional diffusion model to generate paired and aligned human and robot hand data with ground-truth (GT) annotations. The generated paired data are used to train hand detection and pose estimation models with a feature alignment loss to improve performance. The experimental results show that the proposed method significantly outperforms the baseline methods on robot hand pose estimation while maintaining performance on human hand pose estimation.

Critical strengths
  • The paper is easy to follow.
  • The source code is included in the submission, which is important for reproducibility and for future work in this field.
  • The proposed method greatly improves robot hand pose estimation without requiring real robot data for training, while maintaining performance on human hand pose estimation.
Critical weaknesses
  • The motivation for appearance-based 3D pose estimation of anthropomorphic robot hands is not sufficiently clear. In real-world applications where robot hands are operated, their 3D joint coordinates can usually be obtained from the robot’s software. In addition, other sensors besides a single RGB camera are generally available. The paper should clearly discuss in which cases and applications the 3D pose of a robot hand needs to be estimated from a single RGB camera alone.
  • The motivation for the proposed approach is also not sufficiently clear. The paper states that “real dexterous robot data are difficult to scale as capture environments are confined to fixed laboratories (Lim et al., 2026), and 3D pose annotation requires costly multi-view capture (Richardson et al., 2025)” in L. 63-65. However, a straightforward alternative would be to capture robot hand data in a laboratory and augment the backgrounds with image synthesis or generation. This alternative has two advantages: (1) accurate 3D pose GT can be obtained from the robot’s software, which may be more accurate than the multi-view-based annotations of human hand datasets, and (2) the appearance of the foreground (i.e., the robot hand surface) is real. The paper would be stronger if it clarified the motivation and advantages of the proposed approach over such alternatives, or added a comparison with them.
  • The oracle performance (i.e., the performance of a model trained on target-domain data) is not reported.
Questions for authors

What are the motivation for appearance-based 3D pose estimation of anthropomorphic robot hands and the proposed approach? (See weaknesses)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.