- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TactoHand, a framework that predicts three dense hand-surface interaction fields (contact probability, signed-distance proximity, and 3D vertex forces) on the canonical MANO mesh directly from monocular video. To overcome the scarcity of large-scale tactile sensor recordings, the authors curate and harmonize 24 datasets, deriving geometry-based contact/proximity labels and motion-derived 3D vertex forces.
Critical strengths
- Unifying contact, proximity, and 3D force on the canonical MANO topology across 24 diverse datasets is timely and useful. The geometric contact-labeling rules (normal-consistency filtering + multiray inside-mesh recovery, Eq. 3) sensibly resolve common ground-truth pose penetration and backface false-positive artifacts seen in pure distance-thresholding methods like HACO.
- The method explicitly accounts for the non-uniqueness of recovering vertex forces from object motion. Its sequence-level convex program (Eq. 6) combines polyhedral friction constraints, vertex weights based on signed distances, temporal regularization, and estimated tabletop support to select a force allocation consistent with the motion-derived net wrench. This provides a structured approach to deriving physically constrained force targets from existing motion and geometry annotations without collecting tactile measurements.
Critical weaknesses
-
I am quite concerned with potential test-set leakage as checkpoint selection uses the evaluation sets. Appendix C.3, lines 1101–1103, explicitly states that checkpoints are selected using the nominal DexYCB test split and ARCTIC P2 validation split, and results are reported on those same sets. Selecting on the DexYCB test split compromises its role as an independent evaluation (ARCTIC results likewise reflect validation performance). This affects the interpretation of both the main comparisons and the force ablation.
-
It seems that converting TactoHand’s predicted local force components into camera-frame forces requires ground-truth posed MANO normals and tangents (Eq. 7). Without evaluation using estimated hand geometry, the practical reliability of these force predictions remains unclear. The downstream reconstruction improvements provide no additional validation of the force component, since those experiments use only contact and proximity.
Questions for authors
See weakness section.
3 comments
Thank you for addressing my concerns. I’ll revise my score upward to reflect your clarifications.
Thank you for taking another look at the revised experiments. Your questions about checkpoint selection and estimated hand poses helped us strengthen the evaluation.
Thank you for raising the checkpoint selection and hand geometry concerns.
Evaluation splits
We retrained all four TactoHand variants and reran the evaluations. For DexYCB, we followed the official S0 split, using 800 sequences for training, 40 for validation, and 160 for final evaluation. For ARCTIC, we held out subject s04 from the official P2 training split for validation. Because the official P2 test annotations are unavailable, we report final results on the egocentric sequences of s05, the official P2 validation subject. We kept HOCap held out from both training and validation and used it for cross-dataset evaluation.
The revised results differ slightly from those in the original submission. On DexYCB, for example, contact Macro-F1 increased from 59.5% to 63.64% for the temporal model and decreased from 58.5% to 58.31% for the single-frame model. The temporal variant of TactoHand still achieves the highest Macro-F1 and the lowest proximity ADE on benchmarks in the updated Table 1.
Evaluation of the predicted forces
We first clarify when camera-frame forces are needed and how posed hand geometry is used to compute them. We predict vertex forces in local coordinates so that each vector is expressed relative to the hand surface. Whether these components need to be converted to camera coordinates depends on the application. To represent the force distribution on the canonical MANO surface, the local components are sufficient, just as contact and proximity are defined on that surface. Computing camera-frame wrenches requires an accurate estimate of the hand pose. To assess the effect of hand pose estimation, we estimated hand poses with HaWoR on the DexYCB, HOCap, and H2O test sets. Table 9 in Appendix D.3 reports camera-frame MPJPE and compares rollout errors using ground-truth and HaWoR hand poses. Using HaWoR changes position ADE by at most 0.16 cm. Rotation ADE increases on all three datasets, with the largest increase on H2O (38.83° to 56.57°). Results show that hand pose errors can degrade rollout accuracy.
We also extended the force and rollout evaluation in Table 2 to HOCap and H2O. For the temporal model, rollout supervision reduces both position and rotation ADE across all three datasets.
The hand–object reconstruction experiments in Table 3 were designed to test how much predicted contact and proximity improve alignment. We agree that the downstream use of predicted forces still needs to be validated. One direction is to apply the predicted forces in simulation to assess whether they produce physically plausible motion. We discuss the scope of our current evaluation in the Limitations section.