acceptodds
Desk rejected at ICLR 2027

TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce , a model that predicts three dense hand-surface interaction fields—contact, proximity, and 3D force—from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand–object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand-object interaction, Contact estimation, Proximity estimation, Visual force estimation, Differentiable physics, Monocular video

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gChYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TactoHand, a framework that predicts three dense hand-surface interaction fields (contact probability, signed-distance proximity, and 3D vertex forces) on the canonical MANO mesh directly from monocular video. To overcome the scarcity of large-scale tactile sensor recordings, the authors curate and harmonize 24 datasets, deriving geometry-based contact/proximity labels and motion-derived 3D vertex forces.

Critical strengths
  1. Unifying contact, proximity, and 3D force on the canonical MANO topology across 24 diverse datasets is timely and useful. The geometric contact-labeling rules (normal-consistency filtering + multiray inside-mesh recovery, Eq. 3) sensibly resolve common ground-truth pose penetration and backface false-positive artifacts seen in pure distance-thresholding methods like HACO.
  2. The method explicitly accounts for the non-uniqueness of recovering vertex forces from object motion. Its sequence-level convex program (Eq. 6) combines polyhedral friction constraints, vertex weights based on signed distances, temporal regularization, and estimated tabletop support to select a force allocation consistent with the motion-derived net wrench. This provides a structured approach to deriving physically constrained force targets from existing motion and geometry annotations without collecting tactile measurements.
Critical weaknesses
  1. I am quite concerned with potential test-set leakage as checkpoint selection uses the evaluation sets. Appendix C.3, lines 1101–1103, explicitly states that checkpoints are selected using the nominal DexYCB test split and ARCTIC P2 validation split, and results are reported on those same sets. Selecting on the DexYCB test split compromises its role as an independent evaluation (ARCTIC results likewise reflect validation performance). This affects the interpretation of both the main comparisons and the force ablation.

  2. It seems that converting TactoHand’s predicted local force components into camera-frame forces requires ground-truth posed MANO normals and tangents (Eq. 7). Without evaluation using estimated hand geometry, the practical reliability of these force predictions remains unclear. The downstream reconstruction improvements provide no additional validation of the force component, since those experiments use only contact and proximity.

Questions for authors

See weakness section.

3 comments

Thank you for raising the checkpoint selection and hand geometry concerns.

Evaluation splits

We retrained all four TactoHand variants and reran the evaluations. For DexYCB, we followed the official S0 split, using 800 sequences for training, 40 for validation, and 160 for final evaluation. For ARCTIC, we held out subject s04 from the official P2 training split for validation. Because the official P2 test annotations are unavailable, we report final results on the egocentric sequences of s05, the official P2 validation subject. We kept HOCap held out from both training and validation and used it for cross-dataset evaluation.

The revised results differ slightly from those in the original submission. On DexYCB, for example, contact Macro-F1 increased from 59.5% to 63.64% for the temporal model and decreased from 58.5% to 58.31% for the single-frame model. The temporal variant of TactoHand still achieves the highest Macro-F1 and the lowest proximity ADE on benchmarks in the updated Table 1.

Evaluation of the predicted forces

We first clarify when camera-frame forces are needed and how posed hand geometry is used to compute them. We predict vertex forces in local coordinates so that each vector is expressed relative to the hand surface. Whether these components need to be converted to camera coordinates depends on the application. To represent the force distribution on the canonical MANO surface, the local components are sufficient, just as contact and proximity are defined on that surface. Computing camera-frame wrenches requires an accurate estimate of the hand pose. To assess the effect of hand pose estimation, we estimated hand poses with HaWoR on the DexYCB, HOCap, and H2O test sets. Table 9 in Appendix D.3 reports camera-frame MPJPE and compares rollout errors using ground-truth and HaWoR hand poses. Using HaWoR changes position ADE by at most 0.16 cm. Rotation ADE increases on all three datasets, with the largest increase on H2O (38.83° to 56.57°). Results show that hand pose errors can degrade rollout accuracy.

We also extended the force and rollout evaluation in Table 2 to HOCap and H2O. For the temporal model, rollout supervision reduces both position and rotation ADE across all three datasets.

The hand–object reconstruction experiments in Table 3 were designed to test how much predicted contact and proximity improve alignment. We agree that the downstream use of predicted forces still needs to be validated. One direction is to apply the predicted forces in simulation to assess whether they produce physically plausible motion. We discuss the scope of our current evaluation in the Limitations section.

Thank you for addressing my concerns. I’ll revise my score upward to reflect your clarifications.

Thank you for taking another look at the revised experiments. Your questions about checkpoint selection and estimated hand poses helped us strengthen the evaluation.

Reviewer_2UntOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a model to jointly learn hand-object physical features, such as the force applied to the object, the amount of contact between the object and hand, and the proximity of the hand edges (fingers) to the object. While others have attempted to predict such features, this paper seems to be the first to jointly learn all three. To achieve this, it: (1) creates a large dataset by aggregating and reformatting existing datasets; (2) builds a joint loss function based on techniques from previous papers (e.g., contact and proximity losses) while introducing a new "motion rollout" loss inspired by robotics work, which matches the object pose to a rigid body; and (3) uses query-based learning to simultaneously capture the three features from attended video tokens generated by a spatiotemporal video transformer encoder. In experimental evaluations, the method performs very well on predicting Contact and Proximity features, though less so on Force.

Critical strengths

– I think the problem is important, as predicting physical features from images is an important yet under-explored field of research, with many implications for robotics, etc.

– I think there are a few interesting technical points in this paper, such as the addition of a differentiable rigid-body physics layer, the rollout motion loss, and the use of query-based learning.

– The curation of data, which reformats and aligns existing datasets, might be useful for further research in this field.

Critical weaknesses

I think the paper ultimately does not show that it can properly learn the exact 3D per-vertex force magnitude, which is indeed a hard task—yet perhaps the most interesting one. This might be due to the creation of the labels (synthetic, rather than coming from real tactile hardware), or simply because predicting force features from video alone is non-deterministic (e.g., holding an object by squeezing only the thumb and index finger vs. pressing all fingers, which may look visually similar).

Questions for authors

none

3 comments

Thank you for raising the ambiguity of estimating per-vertex forces from video.

Problem of predicting contact forces from video

We agree with the reviewer that visually similar grasps can involve different per-vertex forces. Our labels come from a regularized optimization that enforces physical constraints and matches the net wrenches inferred from observed object motion. In practice, people often grip more firmly than the minimum needed to explain the motion to keep the object stable. Calibrating our labels to reflect the forces people actually apply would therefore require direct tactile force measurements.

Because these datasets lack tactile force measurements, we use object motion error to assess whether the predicted forces are consistent with the observed motion. We added force and rollout results for HOCap and H2O to Table 2. With rollout supervision, the temporal model achieves lower position and rotation errors across all three datasets, indicating that its predicted forces better reproduce the observed object motion in the rollout evaluation.

We discuss the need for tactile calibration in the limitations and leave it for future work.

Reviewer_2UntOpenReview ↗

thanks

Thanks for responding. Overall, I support acceptance.

Thank you for your support and for raising the ambiguity of per-vertex forces. Your feedback helped us clarify what the motion-based evaluation can actually establish.

Reviewer_KMxXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TactoHand jointly predicts contact, proximity, and three-component hand-surface force fields from monocular video. Geometry provides contact and proximity targets, while eligible object trajectories support regularized inverse-dynamics force allocation. A temporal model shares visual features across the three outputs, and a differentiable rigid-body objective encourages motion consistency. The work combines a broad heterogeneous collection and evaluates direct field prediction and controlled interaction reconstruction. Its geometry results are stronger evidence than its interpretation as recovery of real distributed physical force.

Critical strengths

A common MANO-indexed interface for several interaction quantities is practically useful. The paper is explicit that supervision availability differs by source and that invalid force annotations are masked. The approximately 49 million captured frames are collection counts, not a claim that every frame trains the force head. The paper also clearly acknowledges non-unique force allocation, geometry-derived proxy labels, and reference-assisted short-horizon evaluation. Controlled reconstruction variants retain shared settings, making their improvements meaningful within that assisted setup.

Critical weaknesses

Evaluation concerns in Appendix C.3: the nominal DexYCB test split and ARCTIC P2 validation split are used for checkpoint selection, and reported performance is measured on those same sets. This is selection-and-reporting reuse, not evidence that test examples entered gradient training. Nevertheless, it weakens the interpretation of those results as untouched final-test estimates. An independent sealed evaluation or a new development/test partition is needed, ideally with comparable selection budgets for the baselines. HOCap remains a useful counterbalance and should not be discarded simply because other evaluation sets are reused.

Questions for authors
  1. Can all principal results be repeated on a set not used for checkpoint or hyperparameter selection?
  2. How many captured frames, unique episodes, valid force frames, and rollout-eligible windows actually contribute to training?
1 comment

Thank you for raising the problem of evaluation splits and training data coverage.

Evaluation splits

We retrained all four TactoHand variants and reran the evaluations. For DexYCB, we followed the official S0 split, using 800 sequences for training, 40 for validation, and 160 for final evaluation. For ARCTIC, we held out subject s04 from the official P2 training split for validation. Because the official P2 test annotations are unavailable, we report final results on the egocentric sequences of s05, the official P2 validation subject. We kept HOCap held out from both training and validation and used it for cross-dataset evaluation.

The revised results differ slightly from those in the original submission. On DexYCB, for example, contact Macro-F1 increased from 59.5% to 63.64% for the temporal model and decreased from 58.5% to 58.31% for the single-frame model. The temporal variant of TactoHand still achieves the highest Macro-F1 and the lowest proximity ADE on benchmarks in the updated Table 1.

Training data coverage

The curated collection contains 49.0M source frames. The training pool contains 29.1M frames across 114K sequences, with 4.6M frames that have valid force labels. In the training pool, 195K clips are eligible for rollout supervision. The validation sets contain 104.9K frames, and the test sets used in Table 1 contain 176K frames. Appendix A.1 reports the collection and split counts in Tables 4 and 5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Discussion reminder

Dear Reviewers,

Thank you again for your feedback. As the discussion period draws to a close, we would appreciate it if you could take a look at our November 12 responses and let us know if you have any remaining concerns. We would be happy to clarify them.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of the TactoHand discussion

Dear Area Chair and Reviewers,

Thank you for the discussion. Below is a brief summary of our revisions.

Evaluation protocol. We retrained all four TactoHand variants using separate splits for checkpoint selection and final evaluation. Under the revised protocol, our temporal model continues to achieve the highest contact Macro-F1 and the lowest proximity ADE across the benchmarks in Table 1.

Force estimation. We extended the rollout evaluation to H2O and HOCap. Across all three datasets, adding the rollout loss reduced both position and rotation errors for the temporal model (Table 2). To isolate the effect of hand geometry, we also repeated the evaluation using hand poses estimated by HaWoR while keeping the predicted local forces fixed. Rotation error increased on all three datasets, showing that rollout accuracy is sensitive to hand pose errors.

Training coverage. We clarified the dataset coverage in Appendix A.1. Of the 49.0M source frames in the collection, the training pool contains 29.1M frames across 114K sequences, including 4.6M frames with valid force labels. Within that pool, 195K clips qualify for rollout supervision.

Supplementary material. We added an offline website with an overview video and interactive visualizations of TactoHand’s predictions.

Clarification on force non-uniqueness. We clarified that different distributions of per-vertex forces can produce the same object motion. Our solver uses physical constraints to select one plausible distribution. Calibrating these estimates against the forces people actually apply would require tactile measurements.

We appreciate the reviewers’ questions, which helped us strengthen the evaluation and clarify the paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.