- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Touch-JEPA learns hand-surface representations from hand-object videos through geometry-guided masked latent prediction. Estimated MANO geometry provides anatomical correspondences, while interaction and anatomical masking encourage inference from spatial and temporal context. The framework also supports joint training, combining self-supervision across all videos with contact and magnitude supervision on a tactile-labeled subset, with gradient control applied at the representation level. The learned representations are evaluated using supervised MLP readouts with frozen encoders. Across three human-hand benchmarks, the self-supervised (SSL) variant outperforms the retrained supervised baselines in pooled contact AUROC and F1, as well as magnitude nRMSE; joint training further improves most human-hand metrics. Additional experiments assess human-to-robot tactile transfer and action-conditioned manipulation in simulation. These results support the utility of video-based learning for tactile representations, although fine-grained temporal magnitude prediction remains weak on two benchmarks, and joint training yields poorer transfer than SSL to the evaluated robotic hands.
Critical strengths
-
Effective geometry-guided representation learning. Without tactile supervision during encoder pretraining, SSL features support supervised readouts that outperform all four retrained baselines in pooled AUROC, F1, and nRMSE across three human-hand datasets. Visual evidence, geometric structure, and masking each contribute to this performance.
-
Strong controlled and downstream results. Joint training improves AUROC and nRMSE over same-architecture SSL and Labels-only controls across all three datasets. In simulated manipulation, Touch-JEPA-AC achieves 86% grasp-lift and 84% placement success, compared with 54% and 46% for the matched V-JEPA 2.1 planner.
Critical weaknesses
Post-rebuttal update: The new experiments largely address my concerns about label efficiency and the training setup. Tracking magnitude changes within a contact remains difficult, especially on OpenTouch, but the additional analysis helps clarify this limitation.
-
Limited isolation of supervision and joint-training effects. The experiments do not isolate gains from additional unlabeled video sources or establish the benefit of gradient projection/capping over ordinary joint training. Only one tactile-label budget is evaluated, and the Labels-only control also changes the weighting schedule and gradient handling. This limits conclusions about label efficiency and which components drive the gains.
-
Limited evidence for fine-grained tactile dynamics. Within-contact temporal magnitude correlations are only 0.035 on OpenTouch and 0.109 on HandSense. On OpenTouch, SSL also yields near-chance temporal AUROC (0.527) and worse nRMSE than the static prior (0.914 vs. 0.898). Despite stronger temporal results on PVDB, these findings leave limited evidence that the representations capture detailed tactile dynamics beyond spatial contact regularities.
Questions for authors
Post-rebuttal update:The authors have answered my main questions through the additional experiments and clarifications. I have no further questions at this stage.
-
Contribution attribution and label efficiency. Can controlled ablations isolate the benefits of additional unlabeled videos and gradient projection/capping relative to SSL on the tactile datasets and ordinary weighted joint training, respectively? How does joint training compare with Labels-only under reduced tactile-label budgets?
-
Robustness of the gains. Do the AUROC improvements over Labels-only persist across training seeds and recording- or participant-level resampling? Please clarify whether test windows overlap and how dependence between clips is accounted for.
-
Scope of robot transfer. Do the evaluated HRDexDB robot episodes share corresponding human interactions, objects, or scenes with encoder training data? Please clarify the train–test separation and report the number of Inspire evaluation samples.
5 comments
Response to Reviewer K4U7 (2/2)
W2. Limited evidence for fine-grained tactile dynamics. Within-contact temporal magnitude correlations are only 0.035 on OpenTouch and 0.109 on HandSense. On OpenTouch, SSL also yields near-chance temporal AUROC (0.527) and worse nRMSE than the static prior (0.914 vs. 0.898). Despite stronger temporal results on PVDB, these findings leave limited evidence that the representations capture detailed tactile dynamics beyond spatial contact regularities.
Thank you for this point. As our General Response explains with Tab R3, contacts on OpenTouch and HandSense vary far less in measured magnitude than those on PVDB, which leaves t-ρ little to track. Despite this, Joint tracks the onset and release of contact rather than only its usual location on PVDB and HandSense, reaching tAUC 0.802 and 0.753 against 0.749 and 0.649 for the best retrained baseline, whereas a static prior of usual contact locations stays at chance (0.500).
For magnitude, joint training brings OpenTouch nRMSE below the static prior (0.879 vs. 0.898), and even SSL is far below the prior on PVDB and HandSense (0.893 vs. 0.955 and 0.896 vs. 1.047).
These results demonstrate that our representations capture contact timing beyond spatial contact regularities on PVDB and HandSense, and that joint training keeps magnitude error below the static prior on all three datasets.
Q2. Robustness of the gains. Do the AUROC improvements over Labels-only persist across training seeds and recording- or participant-level resampling? Please clarify whether test windows overlap and how dependence between clips is accounted for.
As shown in Tab R5 of our General Response above, the contact AUROC improvement is significant across all three seeds on PVDB and in two on OpenTouch, while the nRMSE gain is significant in all nine conditions. Under participant- and recording-level resampling, the contact AUROC gain over Labels-only is significant on PVDB and OpenTouch at every training length of Tab R12.
Tab R12. Joint − Labels-only contact AUROC with 95% intervals, significant differences in bold
| Dataset | 6k steps | 9k steps | 12k steps |
|---|---|---|---|
| PVDB | +0.009 [0.005, 0.011] | +0.006 [0.003, 0.008] | +0.005 [0.004, 0.006] |
| OpenTouch | +0.006 [0.002, 0.010] | +0.005 [0.001, 0.009] | +0.005 [0.001, 0.010] |
| HandSense | +0.017 [−0.010, 0.035] | +0.015 [−0.017, 0.033] | +0.014 [−0.018, 0.029] |
-
On PVDB, the gain is positive for each of the six participants at all three training lengths (two-sided sign test, p = .03 at each).
-
Only the OpenTouch test windows overlap (2,519 windows over 8,925 distinct frames), and only within a recording. Because our intervals resample whole recordings or participants, overlapping and correlated clips always stay together.
These results show that the gain over Labels-only on PVDB and OpenTouch is not an artifact of clip-level resampling or training length.
Q3. Scope of robot transfer. Do the evaluated HRDexDB robot episodes share corresponding human interactions, objects, or scenes with encoder training data? Please clarify the train–test separation and report the number of Inspire evaluation samples.
No robot video and no robot tactile data enter encoder training, so every robot episode is evaluated zero-shot, and the readouts are fit on human HandSense data only. On the human side, we split HRDexDB-Human by whole recordings (373 for training and 42 held out), and only the training recordings enter encoder training. A robot episode’s paired human demonstration is therefore in training for 80 of 89 Allegro and 332 of 367 Inspire episodes, while the remaining 9 and 35 have a held-out paired recording. Capture setups and objects recur across recordings, so all scenes and objects appear in training. The Inspire evaluation comprises 367 episodes, 46,053 frames and 229,903 valid frame × fingertip entries over 5 fingertip channels.
Tab R13. Contact AUROC ↑ on episodes without paired human video in training
| Robot (episodes) | Ours (SSL) | Labels-only | HOPE |
|---|---|---|---|
| Allegro (9) | 0.681 | 0.653 | 0.668 |
| Inspire (35) | 0.648 | 0.608 | 0.560 |
As shown in Tab R13, transfer also holds on the episodes whose paired human video is not in training, where SSL has the highest contact AUROC of the three methods on both robots, significantly above chance. These results establish zero-shot robot transfer without robot training data.
In addition to the revisions announced in our General Response, the revision will add the gradient-control and warm-up ablations to the appendix, report the intervals of Tab R12 as primary in Table 9, and detail the robot evaluation setup in App. H and Sec. 4.3. We hope these experiments resolve your concerns and welcome further questions.
Gentle Reminder to Reviewer K4U7
Dear Reviewer K4U7,
We truly thank you for your time and thoughtful feedback on our manuscript. We have addressed your major comments as follows:
-
Label Efficiency: Reducing encoder-training labels from 100% to 1% increases Joint's average contact AUROC improvement over Labels-only from +0.011 to +0.139, and its tactile magnitude nRMSE reduction from 0.056 to 0.180 (Tab R1 & R2).
-
Temporal Dynamics: Our analysis (Tab R3) indicates that the relatively low t-ρ values may largely stem from limited within-contact magnitude variations in the OpenTouch and HandSense datasets. For larger variations, Joint model achieves t-ρ values of 0.282 on HandSense and 0.362 on PVDB, while also achieving the highest OpenTouch tAUC of 0.540.
-
Robustness: The Joint model achieves statistically significant contact AUROC gains over Labels-only across three different training lengths on PVDB and OpenTouch (Tab R12). Its nRMSE improvements are also significant in the three-seed experiments (Tab R5).
As the author–reviewer discussion period draws to a close, we would like to kindly follow up in case you have any additional comments. We would be very glad to address any remaining concerns.
Best regards,
Authors
Response to Reviewer K4U7 (1/2)
Thank you for the detailed review and specific questions. We address your comments below, with supporting results in Tabs. R1–R5 of our General Response above.
Tab R4 of our General Response above shows that training SSL with the additional unlabeled HOI videos, rather than on the tactile datasets alone, improves it on the human benchmarks and on both robots without any tactile label.
The label-fraction study of Tab R1 and R2 in our General Response shows the same benefit from the other side. With only 1% of the tactile labels for encoder training, Joint, which still learns from all of the video, keeps 93–96% of its full-label contact AUROC margin above chance, compared with 48–72% for Labels-only.
These results demonstrate that video recorded without any tactile sensor strengthens Touch-JEPA on both human and robot hands.
Gradient control improves HandSense magnitude estimation and Inspire contact transfer beyond ordinary weighted joint training. We trained Joint for 6k training steps with the same losses and schedule but without projection and cap. Removing these controls significantly increases HandSense nMAE and nRMSE by 0.046 and 0.053 and lowers Inspire contact AUROC from 0.604 to 0.511.
Even without these controls, Joint still significantly outperforms Labels-only in contact AUROC on PVDB and OpenTouch (+0.007 on both) and in nRMSE on all three human benchmarks (reductions of 0.018, 0.020 and 0.078 on PVDB, OpenTouch and HandSense, respectively).
With the cap raised from 0.4 to 1.0, contact AUROC changes by at most 0.002, and Joint remains significantly better than Labels-only in contact AUROC on PVDB and OpenTouch and in nRMSE on OpenTouch and HandSense.
These results demonstrate that gradient control adds protection for HandSense magnitude estimation and Inspire contact transfer on top of the gain that joint training already provides.
Neither difference accounts for Joint’s gain over Labels-only. Projection and cap are defined relative to the SSL gradient, which the Labels-only encoder does not receive, so W1-2 removes them from Joint instead.
These results demonstrate that Joint’s gain over Labels-only remains when Labels-only adopts Joint’s weighting schedule and when gradient control is removed from Joint, and thus comes mainly from adding the SSL objective.
As Tab R1 and R2 of our General Response show, Joint’s advantage over Labels-only grows as the tactile-label budget for encoder training shrinks.
The contact AUROC gap grows from 0.009 to 0.111 on PVDB, from 0.006 to 0.094 on OpenTouch and from 0.017 to 0.211 on HandSense as the budget falls from 100% to 1%, and at 1% Joint is also significantly ahead in tAUC on PVDB and HandSense (0.103 and 0.202).
Our General Response further shows that even SSL, trained without any tactile label, beats the 1% Labels-only encoder in contact AUROC on all three datasets.
These results show that the benefit of Touch-JEPA over supervised training increases as tactile labels become scarce.