- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a framework to predict whole-hand tactile from visual inputs. The network is trained from glove tactile data collected by the authors (100 hours, 3 glove types, 6 scenes and 200 tasks). The proposed network outperforms baselines such as egotactile, touchanything.
Critical strengths
The tactile sensor template mapping is reasonable choice. The proposed network architecture looks solid and the efforts of dataset building is respectable.
Critical weaknesses
Despite the efforts, I do not find the technical depth and novelty of this paper for a venue like ICLR. This paper seems to include lots of engineering about mapping glove tactiles to a normalized hand, and some standard transformer details.
The predicted tactiles are heatmap instead of real force. The heatmap are not verified with any downstream applications. Most of the hand object interactions in the attached videos are object holding behavior without complicated interactions. So the tactile pattern mostly look similar. Also the proposed method does not work with robotics tactile (much more difficult to normalize into a standard template) or shear force. In sum, I do not learn any technical insight after reading this paper, maybe ICLR is not the right venue for it.
Questions for authors
See weakness.
3 comments
Response to Reviewer tYBb (Part 2/2)
Two downstream applications.
In the initial submitted version, we have already conducted two downstream applications to validate the effectiveness of our tactile representation and vision-to-tactile prediction model, which is clearly presented in Section 4.4, Figures 5–6, and Appendix Sec. D.1–D.2:
-
3D Hand-Object Interaction Reconstruction: As shown in Fig. 5, and Appendix Table 2, incorporating our predicted tactile fields as additional guidance improves the reconstruction of the RGB-only baseline. Quantitatively, we evaluate on the OakInk benchmark and demonstrate that our tactile prediction helps to reduce hand MPJPE from 11.24 mm to 10.31 mm, improve Contact IoU from 13.25% to 24.06%, and avoid hand-object interpenetration from 22.11% to 3.18%.
-
Dexterous Robot Skill Imitation: In Section. 4.4, Fig. 6, and Appendix Sec. D.2, we also clearly demonstrate that our predicted tactile fields can be leveraged to guide dexterous imitation learning in the simulator, consistently improving task success rate and accelerating the convergence process.
The presented “holding” interactions are important and not simple
Object holding is a fundamental form of human-object interaction and an essential component of most everyday manipulation actions. Across different object geometries, weights, and grasp poses, holding naturally involves distinct contact regions and physical force distributions across the hand, providing key tactile cues for downstream interaction modeling. Our experiment shows that, even for the state-of-the-art RGB-only baseline, the 3D HOI reconstruction results of “holding” still suffer from severe penetrations and inaccuracy, which can be effectively improved by introducing our tactile predictions.
Robotics tactile and shear forces are out of the scope.
As stated in our title, abstract, and introduction, the research scope of this work is human full-hand tactile learning from human egocentric videos, rather than standard tactile patterns of grippers, etc. Also, our application of tactile-guided dexterous imitation learning, to some extent, has verified the practical value of transferring predicted human full-hand tactile to isomorphic robots such as dexterous hands. Also, shear forces are out of scope since the research target of our work and all the related works (e.g., OpenTouch, TouchAnything, EgoTouch, EgoTactile, HumanTouch, HOPE, EgoTac, etc.) is normal forces. And, to the best of our knowledge, there does not exist a publicly available dataset capturing human full-hand tactile with shear forces, which we agree is a promising research direction in the future, but not related to this work.
We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have tried our best to provide responses to all of your questions and we hope our responses would address your concerns.
As the discussion period nears its conclusion, we would be grateful for any final feedback you may have. Please don't hesitate to let us know if further clarifications would be helpful - we remain ready to provide additional details as needed.
Thank you again for your valuable insights and constructive feedback throughout this process.
Response to Reviewer tYBb (Part 1/2)
Thank you for recognizing that our tactile template mapping is a reasonable choice, our network architecture is solid, and our 100-hour dataset collection effort is respectable. Although no specific questions or constructive suggestions are given, we hope the following explanations can help you to better understand our work.
Technical contribution of our work
Unifying tactile data from heterogeneous hardware into a transferable learning space is challenging and has not been solved well by prior methods. Also, our vision-to-tactile model, with a simple yet effective architecture, tackles the challenge of predicting full-hand tactile without 3D hand pose trackers. Rather than ad-hoc engineering or generic Transformer stacking, our study establishes two key representation insights:
Peak-Preserving Geometric Unification: Direct Mapping introduces sharp boundary discontinuities and overlap peak dilution, while unmapped raw regression locks models to a single glove's channel count. By contrast, our geodesic Gaussian upper-envelope representation preserves physical sensor peaks—reducing Peak Err. from 7.13 to 1.80 on EgoTouch in Table 4—improves native-sensor accuracy over Direct Mapping and Mean aggregation in Reviewer v4gG Q2, outperforms direct raw-signal regression in Common Q2, and unlocks cross-dataset pre-training gains in Reviewer 7YcA Q4.
Tracker-Free Multi-Level Fusion and Force-Guided Supervision over Generic Video Transformers: Pairing multi-level DINOv3 spatial fusion with our universal force-guided objective in Eq. 5 and Table 5 captures fine-grained contact cues under egocentric occlusion—whereas naively adding standard cross-frame spatiotemporal self-attention actually degrades V-IoU and force MAE by over-smoothing rapid contact transitions, as shown in Reviewer 7YcA Q10.
Misunderstanding of heatmap visualization and real physical force prediction.
The presented heatmap in our figures is just a visualization of our results, not our predictions and learning targets. All the ground-truth tactile signals are carefully calibrated and normalized for training, and the prediction of our model will be de-normalized back to real physical force in Newtons. Our experiments are also quantitatively evaluated as physical forces in Newtons—for example, reducing physical force MAE from 0.3800 N to 0.2229 N on EgoTactile and achieving 0.1442 N on OpenTouch (please refer to Table 2).