acceptodds
Under review as a conference paper at ICLR 2027

Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision

32 views

Abstract

Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Representation Learning, Tactile Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K4U7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Touch-JEPA learns hand-surface representations from hand-object videos through geometry-guided masked latent prediction. Estimated MANO geometry provides anatomical correspondences, while interaction and anatomical masking encourage inference from spatial and temporal context. The framework also supports joint training, combining self-supervision across all videos with contact and magnitude supervision on a tactile-labeled subset, with gradient control applied at the representation level. The learned representations are evaluated using supervised MLP readouts with frozen encoders. Across three human-hand benchmarks, the self-supervised (SSL) variant outperforms the retrained supervised baselines in pooled contact AUROC and F1, as well as magnitude nRMSE; joint training further improves most human-hand metrics. Additional experiments assess human-to-robot tactile transfer and action-conditioned manipulation in simulation. These results support the utility of video-based learning for tactile representations, although fine-grained temporal magnitude prediction remains weak on two benchmarks, and joint training yields poorer transfer than SSL to the evaluated robotic hands.

Critical strengths
  1. Effective geometry-guided representation learning. Without tactile supervision during encoder pretraining, SSL features support supervised readouts that outperform all four retrained baselines in pooled AUROC, F1, and nRMSE across three human-hand datasets. Visual evidence, geometric structure, and masking each contribute to this performance.

  2. Strong controlled and downstream results. Joint training improves AUROC and nRMSE over same-architecture SSL and Labels-only controls across all three datasets. In simulated manipulation, Touch-JEPA-AC achieves 86% grasp-lift and 84% placement success, compared with 54% and 46% for the matched V-JEPA 2.1 planner.

Critical weaknesses

Post-rebuttal update: The new experiments largely address my concerns about label efficiency and the training setup. Tracking magnitude changes within a contact remains difficult, especially on OpenTouch, but the additional analysis helps clarify this limitation.

  1. Limited isolation of supervision and joint-training effects. The experiments do not isolate gains from additional unlabeled video sources or establish the benefit of gradient projection/capping over ordinary joint training. Only one tactile-label budget is evaluated, and the Labels-only control also changes the weighting schedule and gradient handling. This limits conclusions about label efficiency and which components drive the gains.

  2. Limited evidence for fine-grained tactile dynamics. Within-contact temporal magnitude correlations are only 0.035 on OpenTouch and 0.109 on HandSense. On OpenTouch, SSL also yields near-chance temporal AUROC (0.527) and worse nRMSE than the static prior (0.914 vs. 0.898). Despite stronger temporal results on PVDB, these findings leave limited evidence that the representations capture detailed tactile dynamics beyond spatial contact regularities.

Questions for authors

Post-rebuttal update:The authors have answered my main questions through the additional experiments and clarifications. I have no further questions at this stage.

  1. Contribution attribution and label efficiency. Can controlled ablations isolate the benefits of additional unlabeled videos and gradient projection/capping relative to SSL on the tactile datasets and ordinary weighted joint training, respectively? How does joint training compare with Labels-only under reduced tactile-label budgets?

  2. Robustness of the gains. Do the AUROC improvements over Labels-only persist across training seeds and recording- or participant-level resampling? Please clarify whether test windows overlap and how dependence between clips is accounted for.

  3. Scope of robot transfer. Do the evaluated HRDexDB robot episodes share corresponding human interactions, objects, or scenes with encoder training data? Please clarify the train–test separation and report the number of Inspire evaluation samples.

5 comments
AuthorsOpenReview ↗

Response to Reviewer K4U7 (1/2)

Thank you for the detailed review and specific questions. We address your comments below, with supporting results in Tabs. R1–R5 of our General Response above.

W1. Limited isolation of supervision and joint-training effects. The experiments do not isolate gains from additional unlabeled video sources or establish the benefit of gradient projection/capping over ordinary joint training. Only one tactile-label budget is evaluated, and the Labels-only control also changes the weighting schedule and gradient handling. This limits conclusions about label efficiency and which components drive the gains.

Q1. Contribution attribution and label efficiency. Can controlled ablations isolate the benefits of additional unlabeled videos and gradient projection/capping relative to SSL on the tactile datasets and ordinary weighted joint training, respectively? How does joint training compare with Labels-only under reduced tactile-label budgets?

W1-1. Additional unlabeled video

  • Tab R4 of our General Response above shows that training SSL with the additional unlabeled HOI videos, rather than on the tactile datasets alone, improves it on the human benchmarks and on both robots without any tactile label.

  • The label-fraction study of Tab R1 and R2 in our General Response shows the same benefit from the other side. With only 1% of the tactile labels for encoder training, Joint, which still learns from all of the video, keeps 93–96% of its full-label contact AUROC margin above chance, compared with 48–72% for Labels-only.

These results demonstrate that video recorded without any tactile sensor strengthens Touch-JEPA on both human and robot hands.

W1-2. Gradient projection and cap

Gradient control improves HandSense magnitude estimation and Inspire contact transfer beyond ordinary weighted joint training. We trained Joint for 6k training steps with the same losses and schedule but without projection and cap. Removing these controls significantly increases HandSense nMAE and nRMSE by 0.046 and 0.053 and lowers Inspire contact AUROC from 0.604 to 0.511.

  • Even without these controls, Joint still significantly outperforms Labels-only in contact AUROC on PVDB and OpenTouch (+0.007 on both) and in nRMSE on all three human benchmarks (reductions of 0.018, 0.020 and 0.078 on PVDB, OpenTouch and HandSense, respectively).

  • With the cap raised from 0.4 to 1.0, contact AUROC changes by at most 0.002, and Joint remains significantly better than Labels-only in contact AUROC on PVDB and OpenTouch and in nRMSE on OpenTouch and HandSense.

These results demonstrate that gradient control adds protection for HandSense magnitude estimation and Inspire contact transfer on top of the gain that joint training already provides.

W1-3. Weighting schedule and gradient handling

Neither difference accounts for Joint’s gain over Labels-only. Projection and cap are defined relative to the SSL gradient, which the Labels-only encoder does not receive, so W1-2 removes them from Joint instead.

  • Labels-only trained with Joint’s warm-up of the tactile-loss weight λ stays significantly behind Joint in contact AUROC on PVDB and OpenTouch (0.011 and 0.007) and in nRMSE on all three datasets (0.043, 0.023 and 0.131).

These results demonstrate that Joint’s gain over Labels-only remains when Labels-only adopts Joint’s weighting schedule and when gradient control is removed from Joint, and thus comes mainly from adding the SSL objective.

W1-4. Tactile-label budgets

As Tab R1 and R2 of our General Response show, Joint’s advantage over Labels-only grows as the tactile-label budget for encoder training shrinks.

  • The contact AUROC gap grows from 0.009 to 0.111 on PVDB, from 0.006 to 0.094 on OpenTouch and from 0.017 to 0.211 on HandSense as the budget falls from 100% to 1%, and at 1% Joint is also significantly ahead in tAUC on PVDB and HandSense (0.103 and 0.202).

  • Our General Response further shows that even SSL, trained without any tactile label, beats the 1% Labels-only encoder in contact AUROC on all three datasets.

These results show that the benefit of Touch-JEPA over supervised training increases as tactile labels become scarce.

AuthorsOpenReview ↗

Response to Reviewer K4U7 (2/2)

W2. Limited evidence for fine-grained tactile dynamics. Within-contact temporal magnitude correlations are only 0.035 on OpenTouch and 0.109 on HandSense. On OpenTouch, SSL also yields near-chance temporal AUROC (0.527) and worse nRMSE than the static prior (0.914 vs. 0.898). Despite stronger temporal results on PVDB, these findings leave limited evidence that the representations capture detailed tactile dynamics beyond spatial contact regularities.

Thank you for this point. As our General Response explains with Tab R3, contacts on OpenTouch and HandSense vary far less in measured magnitude than those on PVDB, which leaves t-ρ little to track. Despite this, Joint tracks the onset and release of contact rather than only its usual location on PVDB and HandSense, reaching tAUC 0.802 and 0.753 against 0.749 and 0.649 for the best retrained baseline, whereas a static prior of usual contact locations stays at chance (0.500).

For magnitude, joint training brings OpenTouch nRMSE below the static prior (0.879 vs. 0.898), and even SSL is far below the prior on PVDB and HandSense (0.893 vs. 0.955 and 0.896 vs. 1.047).

These results demonstrate that our representations capture contact timing beyond spatial contact regularities on PVDB and HandSense, and that joint training keeps magnitude error below the static prior on all three datasets.

Q2. Robustness of the gains. Do the AUROC improvements over Labels-only persist across training seeds and recording- or participant-level resampling? Please clarify whether test windows overlap and how dependence between clips is accounted for.

As shown in Tab R5 of our General Response above, the contact AUROC improvement is significant across all three seeds on PVDB and in two on OpenTouch, while the nRMSE gain is significant in all nine conditions. Under participant- and recording-level resampling, the contact AUROC gain over Labels-only is significant on PVDB and OpenTouch at every training length of Tab R12.

Tab R12. Joint − Labels-only contact AUROC with 95% intervals, significant differences in bold

Dataset6k steps9k steps12k steps
PVDB+0.009 [0.005, 0.011]+0.006 [0.003, 0.008]+0.005 [0.004, 0.006]
OpenTouch+0.006 [0.002, 0.010]+0.005 [0.001, 0.009]+0.005 [0.001, 0.010]
HandSense+0.017 [−0.010, 0.035]+0.015 [−0.017, 0.033]+0.014 [−0.018, 0.029]
  • On PVDB, the gain is positive for each of the six participants at all three training lengths (two-sided sign test, p = .03 at each).

  • Only the OpenTouch test windows overlap (2,519 windows over 8,925 distinct frames), and only within a recording. Because our intervals resample whole recordings or participants, overlapping and correlated clips always stay together.

These results show that the gain over Labels-only on PVDB and OpenTouch is not an artifact of clip-level resampling or training length.

Q3. Scope of robot transfer. Do the evaluated HRDexDB robot episodes share corresponding human interactions, objects, or scenes with encoder training data? Please clarify the train–test separation and report the number of Inspire evaluation samples.

No robot video and no robot tactile data enter encoder training, so every robot episode is evaluated zero-shot, and the readouts are fit on human HandSense data only. On the human side, we split HRDexDB-Human by whole recordings (373 for training and 42 held out), and only the training recordings enter encoder training. A robot episode’s paired human demonstration is therefore in training for 80 of 89 Allegro and 332 of 367 Inspire episodes, while the remaining 9 and 35 have a held-out paired recording. Capture setups and objects recur across recordings, so all scenes and objects appear in training. The Inspire evaluation comprises 367 episodes, 46,053 frames and 229,903 valid frame × fingertip entries over 5 fingertip channels.

Tab R13. Contact AUROC ↑ on episodes without paired human video in training

Robot (episodes)Ours (SSL)Labels-onlyHOPE
Allegro (9)0.6810.6530.668
Inspire (35)0.6480.6080.560

As shown in Tab R13, transfer also holds on the episodes whose paired human video is not in training, where SSL has the highest contact AUROC of the three methods on both robots, significantly above chance. These results establish zero-shot robot transfer without robot training data.

In addition to the revisions announced in our General Response, the revision will add the gradient-control and warm-up ablations to the appendix, report the intervals of Tab R12 as primary in Table 9, and detail the robot evaluation setup in App. H and Sec. 4.3. We hope these experiments resolve your concerns and welcome further questions.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer K4U7

Dear Reviewer K4U7,

We truly thank you for your time and thoughtful feedback on our manuscript. We have addressed your major comments as follows:

  • Label Efficiency: Reducing encoder-training labels from 100% to 1% increases Joint's average contact AUROC improvement over Labels-only from +0.011 to +0.139, and its tactile magnitude nRMSE reduction from 0.056 to 0.180 (Tab R1 & R2).

  • Temporal Dynamics: Our analysis (Tab R3) indicates that the relatively low t-ρ values may largely stem from limited within-contact magnitude variations in the OpenTouch and HandSense datasets. For larger variations, Joint model achieves t-ρ values of 0.282 on HandSense and 0.362 on PVDB, while also achieving the highest OpenTouch tAUC of 0.540.

  • Robustness: The Joint model achieves statistically significant contact AUROC gains over Labels-only across three different training lengths on PVDB and OpenTouch (Tab R12). Its nRMSE improvements are also significant in the three-seed experiments (Tab R5).

As the author–reviewer discussion period draws to a close, we would like to kindly follow up in case you have any additional comments. We would be very glad to address any remaining concerns.

Best regards,

Authors

Reviewer_fxDxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Touch-JEPA learns representations of the hand surface from interaction videos, using masked feature prediction guided by hand geometry. The paper also adds joint training with tactile labels on a subset of the videos. The frozen representations are tested in three ways: supervised readouts of contact and magnitude on three human-hand datasets, zero-shot transfer to robotic hands, and simulated manipulation. Joint training helps on human hands, while the purely self-supervised model transfers better to robot hands. In simulation, the representation beats V-JEPA 2.1 on grasp-lift and placement success.

Critical strengths
  1. The motivation makes sense. Tactile data is hard to collect, so getting useful contact information out of interaction video is much easier and scalable.
  2. Using hand geometry to match points across frames makes sense. The ablations suggest that geometry alone isn’t enough, and that visual evidence, the geometry encoder, and both masking strategies each contribute.
  3. The static prior and temporal metrics are helpful to catch cases where a model can obtain reasonable aggregate scores without tracking contact changes.
Critical weaknesses
  1. The results for learning tactile dynamics is mixed. Temporal AUROC is still low compared to AUROC, only 0.527 and 0.540, respectively. Within-contact magnitude correlations are also low: 0.035 on OpenTouch and 0.109 on HandSense for the joint model. This implies the challenging temporal behavior remains unresolved.
  2. In Appendix Table 8, the “Labels only” model seems to have comparable performance with the joint trained method or the SSL method, so its unclear how much benefits does SSL method actually bring. SSL does show improvement on the robot transfer so an experiment with labels only on robot transfer is needed to justify the benefits of SSL.
  3. Tactile labels cover 27.5% of the training corpus, and labels are also used to fit the final readouts. The experiments show that labeled and unlabeled data can be combined. They don’t show how much annotation is actually needed. Curves that vary the amount of supervision, separately for encoder training and for the readouts, would be much clearer. It would also help to train without the extra unlabeled datasets, to isolate what broader video coverage adds.
Questions for authors
  1. How does performance change as tactile labels are reduced, separately for encoder training and readout fitting?
  2. Is there any analysis on why joint training hurts robot transfer?
6 comments
AuthorsOpenReview ↗

Response to Reviewer fxDx (1/3)

Thank you for the thoughtful review. Tab R1–R5 in our General Response above report the label-fraction study and the unlabeled-video ablation (W3/Q1), the ground-truth dynamics (W1) and the three-seed analysis (W2), and we kindly ask you to refer to them.

W1. The results for learning tactile dynamics is mixed. Temporal AUROC is still low compared to AUROC, only 0.527 and 0.540, respectively. Within-contact magnitude correlations are also low: 0.035 on OpenTouch and 0.109 on HandSense for the joint model. This implies the challenging temporal behavior remains unresolved.

Thank you for raising this point. Tab R3 of our General Response above shows that the sensors change much less during a contact on OpenTouch and HandSense than on PVDB, so the low within-contact correlations you cite stem largely from these datasets themselves.

Even so, Touch-JEPA captures when contact happens, not only where. On PVDB and HandSense, Joint’s tAUC is 0.802 and 0.753 against 0.749 and 0.649 for the best baseline, and on PVDB its t-ρ is 0.249 against 0.076. The quartile analysis under Tab R3 further shows that Joint’s t-ρ peaks on every dataset where the magnitude changes most within a contact.

  • tAUC removes the credit that pooled AUROC gives for knowing where contact usually occurs, so every method in Table 1 has a lower tAUC than pooled AUROC. The static prior, which knows only where contact usually occurs, stays at chance in tAUC (0.500) on every dataset, whereas Joint is far above chance on PVDB and HandSense.

  • Even without tactile labels in encoder training, SSL reaches tAUC 0.783 and t-ρ 0.239 on PVDB against 0.749 and 0.076 for the best baseline. Tab R5 of our General Response also shows that Joint significantly exceeds Labels-only in PVDB tAUC and t-ρ in all three training seeds.

These results demonstrate that Touch-JEPA captures when contact happens on PVDB and HandSense rather than just where it usually occurs, and that its magnitude predictions follow within-contact changes most closely where these changes are large.

W2. In Appendix Table 8, the “Labels only” model seems to have comparable performance with the joint trained method or the SSL method, so its unclear how much benefits does SSL method actually bring. SSL does show improvement on the robot transfer so an experiment with labels only on robot transfer is needed to justify the benefits of SSL.

W2-1. Labels-only on robot transfer

We ran the requested Labels-only robot-transfer experiment with the same HandSense readout (Tab R9). Without any tactile label in encoder training, SSL shows no significant difference in robot contact AUROC from Labels-only, which uses all tactile labels. SSL is higher on Allegro (0.676 vs. 0.648), and on Inspire none of the four metrics of the paper’s Table 10 differs significantly between the two.

Tab R9. Zero-shot robot transfer with the HandSense readout, contact AUROC ↑ / ρ ↑ (12k training steps)

MethodAllegro (89 episodes)Inspire (367 episodes)
HOPE0.582 / 0.1590.568 / 0.014
Labels-only (all tactile labels)0.648 / 0.2740.633 / 0.066
Ours (SSL, no tactile labels)0.676 / 0.1830.608 / 0.059
Ours (Joint)0.628 / 0.1060.509 / −0.015
  • The 89 Allegro episodes cover 21 objects. Computed per object, SSL has the higher contact AUROC on 17 of the 21 objects (two-sided sign test, p = .007). Over all Allegro episodes, it also has a significantly higher F1, while Labels-only, which learns from human magnitude labels, ranks magnitude better (ρ 0.274 vs. 0.183, significant). As Tab R9 shows, SSL exceeds HOPE, the strongest retrained baseline on the robots, in contact AUROC and ρ on both robots.

  • Against Labels-only encoders from three training seeds, SSL is never significantly lower in contact AUROC across the six seed–robot comparisons and is significantly higher in one, by 0.031.

These results demonstrate that our SSL objective, without any tactile label, transfers contact to robot hands on par with an encoder trained with all tactile labels and better than every retrained baseline.

AuthorsOpenReview ↗

Response to Reviewer fxDx (2/3)

W2-2. How much benefit the SSL method brings

  • When labels are scarce, SSL is clearly ahead. Against each of two Labels-only encoders trained independently with 1% of the labels, SSL has a significantly higher contact AUROC on both robots, by 0.218 and 0.089 on Allegro and by 0.162 and 0.059 on Inspire. Tab R1 and R2 of our General Response show that, with 1% of the labels, Labels-only falls far behind on the human benchmarks in both contact AUROC and nRMSE, while Joint stays close to its full-label performance. Our General Response further shows that even the label-free SSL encoder remains ahead of that 1% Labels-only encoder on all three datasets.

  • Tab R5 of our General Response isolates the objective at equal labels, which the SSL and Labels-only rows of Table 8 cannot do because they differ in both the tactile labels and the training objective. Over three training seeds, Joint is significantly ahead of Labels-only in 31 of the 54 combinations and never significantly behind, in line with its higher contact AUROC and lower nRMSE on all three datasets in Table 8. SSL alone comes within 0.019–0.032 contact AUROC of Labels-only in Table 8 and exceeds all four retrained baselines in contact AUROC in Table 1.

These results demonstrate that the SSL objective helps both when tactile labels are scarce and when all labels are available.

W3. Tactile labels cover 27.5% of the training corpus, and labels are also used to fit the final readouts. The experiments show that labeled and unlabeled data can be combined. They don’t show how much annotation is actually needed. Curves that vary the amount of supervision, separately for encoder training and for the readouts, would be much clearer. It would also help to train without the extra unlabeled datasets, to isolate what broader video coverage adds.

Q1. How does performance change as tactile labels are reduced, separately for encoder training and readout fitting?

Tab R1 and R2 of our General Response above give these curves for encoder training. Joint stays ahead of Labels-only in every cell, and the gap widens on all three datasets as the labeled recordings shrink from 100% to 1%. These results demonstrate that Touch-JEPA needs far fewer tactile labels in encoder training than supervised training.

In Tab R10 and R11 below, we additionally experimented with fitting the readout using {100%, 10%, 1%} of the training recordings while keeping the trained encoders fixed, namely the Labels-only and Joint encoders of the paper’s Table 8.

Tab R10. Contact AUROC ↑, higher is better, fewer labels for the readout only

# readout labelsPVDBOpenTouchHandSense
100%0.942→0.946 (+0.005)0.861→0.867 (+0.005)0.921→0.934 (+0.014)
10%0.935→0.940 (+0.005)0.850→0.858 (+0.008)0.909→0.927 (+0.018)
1%0.929→0.932 (+0.003)0.824→0.835 (+0.011)0.888→0.919 (+0.031)

Tab R11. Tactile Magnitude nRMSE ↓, lower is better, fewer labels for the readout only

# readout labelsPVDBOpenTouchHandSense
100%0.877→0.864 (−0.013)0.879→0.864 (−0.016)0.854→0.748 (−0.106)
10%0.969→0.969 (|Δ| < 0.001)0.903→0.872 (−0.031)0.894→0.770 (−0.124)
1%0.969→0.968 (|Δ| < 0.001)1.079→1.077 (−0.002)1.687→1.080 (−0.607)
  • Joint has the better value in all 18 cells, by less than 0.001 in two of them. Contact AUROC needs very few readout labels, as Joint loses only 0.01–0.03 with 1% of the recordings.

  • Magnitude needs more readout labels than contact, likely because the readout has to map the latent space to each sensor’s units. With 1% of the recordings, and on PVDB already with 10%, the readout effectively stops working as a readout of magnitude, so magnitude estimation with very few labels is limited by readout supervision. Even there, Joint has the lower nRMSE. Tab R2 of our General Response also shows that, with enough readout labels, its nRMSE stays at 0.80–0.89 even with 1% of the labels in encoder training.

  • Our readout is deliberately light, a single 256-unit hidden layer (App. E), so with enough labels the scores reflect the frozen representation rather than the readout.

These results demonstrate that contact can be read out from Touch-JEPA’s representation with very few labels, and that its magnitude information is preserved and recovered once the readout has enough labels.

Finally, Tab R4 of our General Response above isolates what broader video coverage adds. SSL trained without the additional unlabeled HOI videos has a lower contact AUROC on all five datasets, most clearly on the robots.

AuthorsOpenReview ↗

Response to Reviewer fxDx (3/3)

Q2. Is there any analysis on why joint training hurts robot transfer?

We traced the gap across seeds, objects and training components. Human tactile sensors and the robot fingertip sensors differ in how and where they measure contact. Joint fits the human sensors most closely, with the highest contact AUROC and the lowest nRMSE on all three human benchmarks, and we attribute its weaker zero-shot transfer to this close fit. SSL shapes its latent space from video alone, without fitting any sensor, so we use the label-free SSL encoder for robot hands and Joint for human tactile estimation, as in the submission.

  • The gap is consistent. Labels-only stays ahead of Joint on both robots in all three training seeds, significantly in ρ in all six comparisons, and on Allegro Joint is below Labels-only in ρ on 18 of the 21 objects (p = .001).

  • The human labels themselves do not cause the gap, because Labels-only uses the same labels and transfers contact as well as SSL (Tab R9). Nor does a lack of relevant video, because Joint is trained on the same unlabeled video as SSL.

  • Joint’s warm-up of the tactile-loss weight λ explains part of the ρ gap. Added to Labels-only at 6k training steps, it significantly lowers the Allegro ρ from 0.291 to 0.228, still far above Joint’s 0.076.

These results show that the gap comes from joint training itself rather than from the human labels or the video, in line with Joint’s closer fit to the human sensors, while SSL’s sensor-free latent space transfers well, which supports this division of roles.

Beyond the revisions listed in our General Response, the revised paper will include the diagnostics of W1 and the readout sweep of Tab R10 and R11 in the appendix, add Tab R9 and the seed-level and per-object analysis of the Joint gap to App. H, and state in Sec. 4.3 that SSL leads the retrained baselines in contact AUROC and ρ on both robots. We hope these results address your concerns, and we would be glad to clarify any remaining point during the discussion.

Reviewer_VKmDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Touch-JEPA learns tactile representations using hand-object interaction video dataset with limited tactile supervision. The SSL framework learns a visual-only feature space from the videos using a V-JEPA backbone. A joint training step further adds tactile supervision. With a supervised readout module, the proposed method outperforms supervised baseline methods.

Critical strengths
  1. Tactile features from limited tactile labels: Even without any tactile labels during training, the model is able to outperform all supervised baseline methods on tactile estimation tasks, including contact estimation and magnitude estimation.

  2. The learned features transfer to robot hand videos: The self-supervised representation transfers zero-shot to real Allegro-hand videos even without any robot tactile data. This suggests that the learned representation is not constrained to a specific sensor or embodiment, making it applicable to more robotic settings.

Critical weaknesses
  1. Label efficiency is not measured strictly: the paper claims that the framework is learning with "limited tactile supervision". However, the 30k+ labeled clips cover all training data of HandSense, PressureVisionDB and OpenTouch, and the evaluation is also run on the three benchmarks. As a result, the amount of the tactile supervision is never varied, which means that it is not showing that the joint training is more label-efficient than supervised training. It would be interesting if an experiment can be done to compare the joint-training and the labels-only training with a varied number of training labels (e.g. 1%, 5%, 20%, 80%), and show that the benefit from the joint-training can grow as the label number shrinks.

  2. SSL's beating supervised baselines might come from architecture instead of training objective. According to Table 8, when the encoder architecture is fixed, the labels-only variant outperforms SSL on almost every metric. This is suggesting that the SSL's advantage over the baseline models in Table 1 and Table 2comes mainly from a stronger V-JEPA backbone instead of SSL framework. It would be interesting to run the SSL vs labels-only training using the same backbone as the baseline methods.

Questions for authors

Corresponding to the weakness points above, I have two major questions:

  1. Since the joint model uses all tactile training labels, would it be possible to run a controlled experiment that trains joint and labels-only model with varied fractions of tactile labels and report the performance on the metrics? If the joint training's advantage grows significantly as the number of labels decreases, then it would support the claim in the paper that the proposed method is more label-efficient.

  2. Would it be possible to change the encoder architecture to the same as the baseline methods and see if the SSL training alone is able to achieve a better performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VKmD (1/2)

Thank you for your constructive feedback and insightful comments. We address each point in more detail below.

W1. Label efficiency is not measured strictly: the paper claims that the framework is learning with "limited tactile supervision". However, the 30k+ labeled clips cover all training data of HandSense, PressureVisionDB and OpenTouch, and the evaluation is also run on the three benchmarks. As a result, the amount of the tactile supervision is never varied, which means that it is not showing that the joint training is more label-efficient than supervised training. It would be interesting if an experiment can be done to compare the joint-training and the labels-only training with a varied number of training labels (e.g. 1%, 5%, 20%, 80%), and show that the benefit from the joint-training can grow as the label number shrinks.

Q1. Since the joint model uses all tactile training labels, would it be possible to run a controlled experiment that trains joint and labels-only model with varied fractions of tactile labels and report the performance on the metrics? If the joint training's advantage grows significantly as the number of labels decreases, then it would support the claim in the paper that the proposed method is more label-efficient.

In Tab R1 and R2 of our General Response, we evaluated label efficiency using {100%, 10%, 1%} of the labeled training recordings. Joint is ahead of Labels-only in all 18 cells (three datasets, two metrics and three label fractions), and its advantage in both contact AUROC and magnitude nRMSE widens on all three datasets as fewer labeled recordings are used for encoder training.

  • In Tab R1, the contact AUROC gap widens most sharply when the label fraction drops from 10% to 1% on all three datasets. From 100% to 1%, Joint loses only about 0.02 contact AUROC on each dataset, whereas Labels-only loses 0.11–0.22.

  • Tab R2 shows a similar pattern for tactile magnitude. Reducing the label fraction from 100% to 1% increases Joint’s nRMSE by at most 0.04 on each dataset, compared with increases of 0.09–0.22 for Labels-only. On PVDB, for example, Joint’s nRMSE is nearly unchanged (0.870 to 0.871), while the nRMSE for Labels-only rises from 0.879 to 0.969. At 1%, Joint’s nRMSE is significantly lower than that of Labels-only on all three datasets.

  • The advantage also holds in contact timing. At 1%, Joint exceeds Labels-only in tAUC by 0.103 on PVDB and 0.202 on HandSense, and both differences are significant.

We attribute this to the SSL objective, which still trains the encoder on the video of every training dataset, labeled or not, while Labels-only must adapt the same pretrained backbone from a few tactile labels. On every dataset, Joint’s contact AUROC gain at 1% significantly exceeds its gain with all labels. These results demonstrate that Joint’s advantage grows significantly as tactile labels decrease, which is the criterion you proposed for the paper’s claim that Touch-JEPA is more label-efficient than supervised training.

AuthorsOpenReview ↗

Response to Reviewer VKmD (2/2)

W2. SSL's beating supervised baselines might come from architecture instead of training objective. According to Table 8, when the encoder architecture is fixed, the labels-only variant outperforms SSL on almost every metric. This is suggesting that the SSL's advantage over the baseline models in Table 1 and Table 2 comes mainly from a stronger V-JEPA backbone instead of SSL framework. It would be interesting to run the SSL vs labels-only training using the same backbone as the baseline methods.

Q2. Would it be possible to change the encoder architecture to the same as the baseline methods and see if the SSL training alone is able to achieve a better performance?

Results with DINOv3 (HOPE’s visual backbone)

Thank you for this suggestion. To evaluate Touch-JEPA on a different backbone, we replaced V-JEPA 2.1 with DINOv3 ViT-B/16, the visual backbone used by HOPE, and trained SSL and Joint with half the training budget of the retrained baselines (App. E). The results are reported in Tab R6 and R7 below. In Tab R6–R8, “Best retrained baseline” is the strongest of the four supervised baselines for each dataset and metric, which can be a different method in each column.

Tab R6. Contact AUROC ↑ with DINOv3

MethodPVDBOpenTouchHandSense
HOPE (retrained, Table 1)0.9100.7620.831
Best retrained baseline0.9120.7850.831
Ours-DINOv3 (SSL)0.9160.8290.850
Ours-DINOv3 (Joint)0.9350.8610.917

Tab R7. Tactile Magnitude nRMSE ↓ with DINOv3

MethodPVDBOpenTouchHandSense
HOPE (retrained, Table 2)0.9560.9720.931
Best retrained baseline0.9270.9560.931
Ours-DINOv3 (SSL)0.9070.9370.953
Ours-DINOv3 (Joint)0.8980.8970.820
  • As shown in Tab R6, SSL exceeds retrained HOPE and the best retrained baseline on all three datasets, most clearly on OpenTouch (0.829 vs. 0.762 for HOPE).

  • As shown in Tab R7, Joint has the lowest nRMSE on all three datasets, and SSL beats the best baseline on PVDB and OpenTouch. Over all six metrics of the paper’s Tables 1–2, Ours-DINOv3 (Joint) beats the best retrained baseline in 16 of the 18 cells.

These results demonstrate that Touch-JEPA’s contact AUROC advantage over the supervised baselines, even with SSL alone, persists when V-JEPA 2.1 is replaced with HOPE’s visual backbone.

Why we matched DINOv3

The four baselines do not share one architecture. PressureVision and PressureVision++ are single-image networks, our PressureFormer implementation uses frozen per-frame HaMeR features, and HOPE uses frozen DINOv3 features, so none of these pipelines can host our masked training on 16-frame clips. We therefore adopted HOPE’s backbone, which shares the ViT-B/16 configuration of V-JEPA 2.1 and leaves our training objective unchanged.

Analysis of frozen V-JEPA 2.1 features

To test the backbone alone, we read out frozen V-JEPA 2.1 with the same supervised readout on the common test data of Table 1.

Tab R8. Contact AUROC ↑ with frozen and trained V-JEPA 2.1

MethodPVDBOpenTouchHandSense
Best retrained baseline0.9120.7850.831
V-JEPA 2.1 (frozen)0.9120.7120.876
Ours (SSL)0.9250.8360.889
Ours (Joint)0.9490.8620.934
  • In Tab R8, frozen V-JEPA 2.1 matches the best retrained baseline on PVDB, falls below it on OpenTouch (0.712 vs. 0.785) and exceeds it on HandSense, while our SSL encoder raises contact AUROC above both on all three datasets and Joint raises it further.

These results demonstrate that our advantage over the baselines goes beyond the pretrained V-JEPA 2.1 features, as our training improves contact AUROC over the frozen backbone on every dataset.

The SSL objective with the architecture fixed

On any backbone, a gap between SSL and Labels-only mixes the effect of the labels with that of the objective, as in Table 8. Tab R5 of our General Response therefore holds the labels fixed as well, and Joint significantly outperforms Labels-only in 31 of the 54 seed–dataset–metric combinations. The same section shows that SSL without any tactile label exceeds the 1% Labels-only encoder in contact AUROC on all three datasets. Since both comparisons keep the backbone unchanged, these gaps cannot come from the backbone.

Together with the revisions in our General Response, the revised paper will add the backbone controls of Tab R6–R8 to a new appendix. We hope these results address both of your concerns, and we would be grateful if you took them into account in your final assessment.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer VKmD

Dear Reviewer VKmD,

We sincerely appreciate your time and thoughtful efforts in reviewing our submission. We have addressed your two major concerns regarding (1) label efficiency and (2) performance specific to the V-JEPA backbone, as follows:

  • Label Efficiency: Compared with Labels-only, our Joint model achieves average contact AUROC improvements of +0.011, +0.023, and +0.139, and tactile magnitude nRMSE reductions of 0.056, 0.065, and 0.180 at 100%, 10%, and 1% label fractions, respectively (Tab R1 & R2).

  • Different Encoder Backbones: To demonstrate that our improvements are not specific to V-JEPA, we additionally applied our method to DINOv3. Compared with the best retrained baseline, our Joint model achieves an average contact AUROC improvement of +0.062 and a tactile magnitude nRMSE reduction of 0.066. Even with SSL alone, our DINOv3-based model outperforms the best retrained baseline in contact AUROC across all three datasets (Tab R6 & R7).

We hope that these additional experiments adequately address your concerns. If so, we would appreciate it if you could reconsider your initial rating in light of these additional results and clarifications. If you have any remaining questions or concerns, please do not hesitate to let us know. We would be very happy to address them during the author–reviewer discussion period.

Best regards,

Authors

AuthorsOpenReview ↗

Follow-up on Our Responses and Revised Manuscript

Dear Reviewer VKmD,

As the discussion period draws to a close, we would like to kindly follow up once again on our author response and revised manuscript.

We aim to resolve all reviewers' concerns as thoroughly as possible during the author–reviewer discussion period. We are grateful that the other two reviewers have carefully considered our responses and acknowledged that their main concerns have been resolved. We also greatly value your constructive feedback and would be grateful to hear whether our responses and additional experiments have sufficiently addressed your remaining concerns.

If you find that your main concerns have been largely resolved, we would greatly appreciate it if you could consider raising your rating. We would also welcome any final comments you may have.

Thank you again for your valuable feedback on our submission.

Best regards,

Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for the time and care they devoted to our submission. Their constructive comments and suggestions have been extremely helpful. Below, we first address the questions shared by several reviewers and then reply to each reviewer’s specific comments.

Label efficiency with fewer tactile labels in encoder training

Our central claim is that Touch-JEPA learns effective tactile representations under limited tactile supervision. We acknowledge the absence of a direct label-efficiency study in the submission and thank the reviewers for suggesting it.

To highlight the superior label efficiency of our model, we conducted additional controlled experiments. We trained both the Joint and Labels-only encoders for 6k training steps using {100%, 10%, 1%} of the labeled training recordings. At each fraction, the readout is fit on the exact same training clips, and every model is evaluated on all valid test entries. The experimental results are reported in Tab R1 and R2, where each cell reports [Labels-only → Joint (Joint − Labels-only)] computed before rounding. Throughout our responses, we call a difference significant when its 95% paired bootstrap interval excludes zero, using 1,000 resamples of the whole test data.

Tab R1. Contact AUROC ↑, higher is better

# labelsPVDBOpenTouchHandSense
100%0.936→0.944 (+0.009)0.857→0.863 (+0.006)0.916→0.932 (+0.017)
10%0.909→0.933 (+0.025)0.830→0.852 (+0.021)0.896→0.919 (+0.023)
1%0.814→0.925 (+0.111)0.745→0.839 (+0.094)0.699→0.910 (+0.211)

Tab R2. Tactile Magnitude nRMSE ↓, lower is better

# labelsPVDBOpenTouchHandSense
100%0.879→0.870 (−0.009)0.894→0.867 (−0.027)0.891→0.760 (−0.131)
10%0.896→0.866 (−0.030)0.915→0.882 (−0.033)0.945→0.812 (−0.133)
1%0.969→0.871 (−0.098)1.024→0.891 (−0.132)1.109→0.799 (−0.310)
  • The Joint model outperforms Labels-only in both contact and tactile magnitude estimation across all 18 cells of Tab R1 and R2. Its contact AUROC gain grows from 0.009, 0.006 and 0.017 with all labels to 0.111, 0.094 and 0.211 at 1%, and the gain at 1% is significantly larger than with all labels on every dataset. Similarly, its magnitude nRMSE reduction expands from 0.009, 0.027, and 0.131 with all labels to 0.098, 0.132, and 0.310 at 1%.

  • At 1%, Joint retains 93–96% of its full-label contact AUROC margin above chance (AUROC − 0.5), while Labels-only retains 48–72%. Even the best constant prediction, the mean of the test measurements, has an nRMSE of 0.96, 1.03 and 1.08. At 1%, Labels-only falls to about this level (0.969, 1.024 and 1.109), whereas Joint stays well below it (0.871, 0.891 and 0.799), even below that of Labels-only trained with all labels (0.879, 0.894 and 0.891).

These results demonstrate that Touch-JEPA is substantially more label-efficient than supervised training, which directly supports our central claim of learning tactile representations with limited tactile supervision.

Tactile dynamics within a contact

Reviewers fxDx and K4U7 noted the low within-contact magnitude correlation (t-ρ) of Joint on OpenTouch and HandSense. Tab R3 shows that the measured magnitude changes far less within a contact on these two benchmarks than on PVDB, so t-ρ has much less signal to follow there.

Tab R3. Within-contact change and contact transitions in the test ground truth (16-frame sensor series)

StatisticPVDBOpenTouchHandSense
Median within-contact magnitude range (training-set SD units)1.120.700.31
Median within-contact coefficient of variation0.470.260.12
Series with a contact onset or release17%23%30%
  • When the test series are sorted by the within-contact magnitude range of Tab R3, Joint’s t-ρ rises from the lowest to the highest quartile on every dataset (0.135 → 0.362 on PVDB, 0.012 → 0.088 on OpenTouch, 0.043 → 0.282 on HandSense), and in the top quartile it is far above HOPE under the same condition (0.362 vs. 0.125 on PVDB and 0.282 vs. 0.152 on HandSense).

These results demonstrate that the low t-ρ on OpenTouch and HandSense reflects how little the measured magnitude changes in these benchmarks, and that Touch-JEPA tracks within-contact changes better the larger they are.

AuthorsOpenReview ↗

General Response

Additional unlabeled video

Reviewers fxDx and K4U7 asked what the additional unlabeled video adds. We trained SSL for 6k training steps on the three tactile-sensor datasets only, without the additional unlabeled hand–object interaction (HOI) videos, and compared it with SSL trained with them. Each cell of Tab R4 reports [SSL on the tactile datasets only → SSL with the HOI videos]. Allegro and Inspire are the zero-shot robot-hand evaluations of the paper’s Tables 4 and 10, which report no nRMSE.

Tab R4. SSL trained on the tactile datasets only vs. with the additional HOI videos

MetricPVDBOpenTouchHandSenseAllegroInspire
Contact AUROC ↑0.915→0.9220.835→0.8400.877→0.8860.579→0.6860.581→0.632
nRMSE ↓0.880→0.8810.905→0.8960.918→0.893––
  • As shown in Tab R4, SSL trained with the HOI videos has a higher contact AUROC on all five datasets, significantly on PVDB, OpenTouch and both robots (+0.107 on Allegro and +0.051 on Inspire). Across all six metrics of the paper’s Tables 1–2, it is significantly better in 7 of the 18 human-benchmark cells and significantly worse in none.

These results demonstrate that additional unlabeled video improves the encoder without any tactile label.

The effect of the SSL objective at equal labels

Reviewers VKmD and fxDx inquired about the additional benefits of the SSL objective over tactile labels alone, whereas reviewer K4U7 asked if the performance gains persist across training seeds. In the paper’s Table 8, SSL uses no tactile labels while Labels-only uses all of them, so that comparison changes the labels and the objective at once. At equal labels, the effect of adding our SSL objective is the difference between Joint and Labels-only, which Tab R5 reports for three training seeds, with significant differences in bold.

Tab R5. Joint − Labels-only over three training seeds

Dataset, metricSeed 1Seed 2Seed 3
PVDB AUROC ↑+0.009+0.008+0.008
PVDB F1 ↑+0.020+0.058+0.045
PVDB tAUC ↑+0.018+0.015+0.018
PVDB nMAE ↓−0.002−0.001−0.003
PVDB nRMSE ↓−0.009−0.015−0.010
PVDB t-ρ ↑+0.028+0.027+0.029
OpenTouch AUROC ↑+0.006+0.009+0.002
OpenTouch F1 ↑+0.008+0.015+0.007
OpenTouch tAUC ↑+0.005+0.004+0.002
OpenTouch nMAE ↓−0.001−0.001−0.006
OpenTouch nRMSE ↓−0.027−0.015−0.015
OpenTouch t-ρ ↑+0.007+0.003−0.003
HandSense AUROC ↑+0.017+0.007+0.010
HandSense F1 ↑+0.036+0.015+0.028
HandSense tAUC ↑−0.012+0.018+0.008
HandSense nMAE ↓−0.107−0.116−0.073
HandSense nRMSE ↓−0.131−0.137−0.090
HandSense t-ρ ↑+0.031+0.013+0.054
  • Joint is significantly better than Labels-only in 31 of the 54 seed–dataset–metric combinations and significantly worse in none, and its nRMSE is significantly lower in all nine combinations of seed and dataset.

  • With the same architecture and training length, SSL trained with the HOI videos in Tab R4, without any tactile label in encoder training, also has a significantly higher contact AUROC than the 1% Labels-only encoder of Tab R1 on all three datasets (e.g., 0.922 vs. 0.814 on PVDB and 0.840 vs. 0.745 on OpenTouch). Because the two encoders share the full architecture, including the Geometry Encoder, this gap comes from the training objective rather than from the architecture.

These results demonstrate that our SSL objective improves the tactile representation beyond what tactile labels alone provide, consistently across training seeds and with the architecture held fixed.

The revised paper will add the label-fraction study, the ground-truth statistics of Tab R3, the unlabeled-video ablation of Tab R4 and the three-seed analysis to a new appendix and will replace the single-seed statements with the three-seed results for Joint and Labels-only. We address each reviewer’s specific comments in the individual responses below, which refer to Tab R1–R5 above.

Area_Chair_PGbsOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

AuthorsOpenReview ↗

Summary of Revisions

We are grateful to the reviewers for their constructive feedback and valuable insights. Based on their comments, we have revised the manuscript, with all changes marked in blue. The main revisions are summarized below:

  • Introduction & Conclusion: Highlighted the growing advantage of joint training as tactile labels become scarce.

  • Experiments: Explained the low within-contact correlations observed in tactile datasets, expanded the ablation analysis, and added a label-efficiency study.

  • Appendix A: Reported ground-truth statistics on variation in within-contact tactile dynamics in the test data.

  • Appendices E & F: Adopted paired bootstrap confidence intervals that account for correlations among clips, and added ablations on gradient control and loss weighting.

  • Appendix H.1: Compared the SSL model with a Labels-only encoder on the Allegro & Inspire robot hands.

  • Appendices I & J: Added studies on label efficiency, the effect of additional unlabeled video, seed robustness, and experiments using a DINOv3 backbone.

We hope this summary clarifies how we have addressed the reviewers' concerns and further strengthened the paper's technical contributions in the revised manuscript. As there is still one day remaining in the author-reviewer discussion period, we would appreciate any further suggestions on our responses and revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.