acceptodds
Under review as a conference paper at ICLR 2027

TouchAnnotator: Multi-Source Tactile Supervision with Privileged-View Distillation

Abstract

Egocentric video captures human–object interaction at scale but rarely includes tactile labels for hidden contact and pressure. We introduce TouchAnnotator, an ego-only bimanual tactile annotator trained with multi-source tactile supervision and privileged-view distillation. A canonical hand field unifies glove pressure, geometry-derived human contact, and robot fingertip deformation from six datasets and three embodiments, while source-aware masks preserve valid targets. During training, an EMA teacher observes synchronized ego and wrist views and distills hand-local representations and tactile predictions into a student that uses only ego video and estimated pose at inference. Compact past-clip memory supplies temporal context. We evaluate on EgoTouch, zero-shot FPHA transfer, and 860 synchronized glove recordings. The model transfers to bare-hand contact and produces qualitatively coherent tactile responses across bare-hand, gloved-hand, and robot embodiments. On measured recordings, it improves temporal accuracy by 5.38 points and Volumetric IoU by 5.02 points under a shared protocol. These results show that heterogeneous human–robot supervision and privileged training views can turn egocentric video into dense tactile annotations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
tactile prediction, touch annotator

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dz86OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TouchAnnotator, a framework for predicting hand contact and relative pressure from egocentric RGB videos and estimated hand poses. It integrates six existing datasets into a unified tactile representation and introduces privileged-view distillation and temporal memory. The method is evaluated on existing benchmarks and a newly collected tactile evaluation set.

Critical strengths
  1. Meaningful problem with practical potential. Predicting tactile information from RGB videos is an interesting and practically valuable research direction. If sufficiently accurate, the proposed framework could enable large-scale tactile annotation of human demonstration videos without requiring additional tactile sensors, potentially benefiting embodied learning.

  2. Integration of heterogeneous tactile supervision sources. The paper makes a meaningful effort to unify different tactile supervision sources, including sensor measurements and geometry-derived contact labels. The source-aware training strategy is reasonable, and the additional evaluation on newly collected real tactile data provides useful evidence beyond existing benchmarks.

Critical weaknesses
  1. Insufficient validation of the unified tactile supervision. Although the paper provides evaluations on real tactile data, the accuracy and consistency of the unified tactile mapping remain insufficiently validated. In particular, geometry-derived contact labels are based on proximity thresholds rather than direct contact measurements, while pressure mappings across different sensors may introduce systematic discrepancies. Since this unified representation is fundamental to the proposed framework, its reliability requires stronger quantitative evidence.

  2. Insufficient evidence supporting the effectiveness of the proposed method. The supplementary material reports several ablation configurations but explicitly acknowledges that confounding factors prevent isolating the effects of privileged-view distillation and temporal memory. This makes it difficult to assess the independent contributions of these key components. Furthermore, without strong pose-only baselines, it remains unclear to what extent the model captures tactile-specific information beyond common visual and hand-pose correlations.

Questions for authors

Q1. Can the authors provide quantitative validation of the unified tactile mapping, particularly the agreement between geometry-derived contact labels and real tactile measurements? How do the authors ensure consistency across different sensor modalities and pressure normalization schemes?

Q2. Can the authors provide a strong pose-only baseline to better demonstrate the model's tactile-specific prediction capability? Additionally, the FPHA evaluation appears to use annotated hand poses in the supplementary material but estimated hand poses in the main paper. Could the authors clarify the actual input setting and its implications for the reported results?

Q3. Given the confounding factors acknowledged in the supplementary material, can the authors provide controlled ablations that independently evaluate privileged-view distillation and temporal memory under identical training data, initialization, and evaluation settings? Such experiments would be important to establish whether these components provide consistent improvements over simpler alternatives.

3 comments

We sincerely thank Reviewer dz86 for your time and thoughtful questions about tactile supervision and pose inputs. We appreciate your recognition of the practical value of tactile annotation.

R3Q1: Geometry–tactile agreement and pressure normalization

Q1. Can the authors provide quantitative validation of the unified tactile mapping, particularly the agreement between geometry-derived contact labels and real tactile measurements? How do the authors ensure consistency across different sensor modalities and pressure normalization schemes?

How the two sources supervise the model. The contact and pressure heads have separate output weights but share the features from which they predict. During joint training, geometry-derived binary contact labels and glove pressure measurements both update those shared features. The common hand grid thus represents contact location and relative tactile intensity across egocentric datasets. Pressure is reported on a normalized scale.

Adding geometry-derived contact labels. On unseen EgoTouch objects, adding HOI4D and TACO to glove training improved temporal accuracy and both spatial-overlap metrics. Pressure MAE increased.

Training data comparisonT.Acc. ↑C.IoU ↑V.IoU ↑MAE ↓
TouchAnything reproduction.8329.4207.3371.0631
With HOI4D and TACO added.8861.4600.3749.0917

Agreement before and after target normalization. We align HumanTouch to the EgoTouch relative-pressure distribution by monotone quantile mapping. To test sensitivity to the scoring target, we held both models’ predictions fixed and scored the same valid cells across 860 synchronized glove recordings (139,102 frames).

Glove target used for scoringTouchAnything C.IoU / V.IoU (%)TouchAnnotator C.IoU / V.IoU (%)
Device-reported readings5.88 / 10.887.38 / 15.54
Baseline-subtracted, normalized per trajectory6.89 / 9.1010.73 / 14.12

TouchAnnotator leads on both overlap metrics under either target. Its V.IoU advantage is 4.66 points before and 5.02 points after normalization.

R3Q2: Pose-only baseline and FPHA inputs

Q2. Can the authors provide a strong pose-only baseline to better demonstrate the model's tactile-specific prediction capability? Additionally, the FPHA evaluation appears to use annotated hand poses in the supplementary material but estimated hand poses in the main paper. Could the authors clarify the actual input setting and its implications for the reported results?

Pose-only baseline. With the WiLoR pose sequence held fixed, adding ego RGB improves C.IoU by 8.80 points and V.IoU by 7.26 points on unseen EgoTouch objects, showing that visual cues help localize contact and predict its relative intensity (Table G3, General Comment).

FPHA. We use hand poses annotated by WiLoR as input. During teacher training, we additionally incorporate RGB videos from three camera views. In contrast, during student distillation, we use only the annotated hand poses and egocentric RGB videos to simulate the practical setting of bare-hand egocentric data collection.

R3Q3: Independent effects of privileged views and memory

Q3. Given the confounding factors acknowledged in the supplementary material, can the authors provide controlled ablations that independently evaluate privileged-view distillation and temporal memory under identical training data, initialization, and evaluation settings? Such experiments would be important to establish whether these components provide consistent improvements over simpler alternatives.

Table G2 in the General Comment addresses the two effects separately. With memory fixed, giving the teacher wrist views improves the student’s V.IoU. With teacher views fixed, adding memory also improves V.IoU. Both paired V.IoU intervals lie above zero on unseen trajectories, while the corresponding C.IoU intervals include zero.

Thank you for the additional experiments. While G1-G3 address some of my concerns, the reliability and cross-source consistency of the unified tactile supervision remain insufficiently validated. In addition, the ablation result do not seem to provide sufficiently strong evidence for the paper’s core claims, particularly given the marginal and inconsistent improvements form privileged-view distillation. Therefore, I maintain my original rating.

Thank you for clarifying your remaining concerns. We understand that G1–G3 leave an important question about the supervision itself.

How the sources are combined. Geometry-derived labels supervise binary contact, while tactile measurements supervise relative pressure on the shared hand grid. The two prediction heads share learned features and are jointly trained. This allows the supervision sources to contribute to a common representation while retaining their respective target meanings.

The additional data-source experiment provides evidence of their practical compatibility. Adding HOI4D and TACO improves unseen C.IoU from .4207 to .4600 and V.IoU from .3371 to .3749. Together with the consistent model ranking before and after glove-target normalization, this supports the utility of combining the sources.

What the distillation ablations establish. A−B isolates privileged wrist-view access with distillation and memory fixed, yielding +0.82 V.IoU points [95% paired CI +0.38, +1.28]. B−C separately tests distillation from an ego-only teacher, yielding +2.00 T.Acc. points and −0.59 V.IoU points. Thus, the mixed B−C result concerns a teacher receiving the same camera view as the student. A−B supports a modest benefit from additional teacher views.

We will make these distinctions explicit and align the claims with the demonstrated benefits and remaining validation limits.

Reviewer_2rMyOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TouchAnnotator, a model that predicts tactile information from ego-centric video during test time. Given the fact that datasets with tactile information are rare and heterogeneous, TouchAnnotator harmonizes six datasets from three embodiments to form a unified training corpus. TouchAnnotator also distills privileged information not available during test time through an EMA teacher. TouchAnnotator shows superior performance on EgoTouch, in particular on unseen objects. The paper also shows strong result on FPHA and self-collected data, and demonstrates signs of cross-embodiment generalization.

Critical strengths
  1. The method is well designed and sensible, e.g., unifying different datasets, distilling from privileged information, using video encoder for visual features.

  2. The experiments are well designed and the empirical results are convincing.

Critical weaknesses
  1. While the method design is sensible, it could benefit from some ablation to help readers understand how different components contribute to the performance.
Questions for authors

N/A

1 comment

We sincerely thank Reviewer 2rMy for your time and encouraging assessment of our experimental design. We appreciate your positive feedback on combining heterogeneous tactile supervision and your suggestion to quantify what each component contributes.

R2Q1: Component contributions (from Critical Weaknesses)

While the method design is sensible, it could benefit from some ablation to help readers understand how different components contribute to the performance.

The clearest contributions are to unseen V.IoU: access to wrist views during teacher training adds 0.82 points, and past-clip memory adds 0.90 points. Distillation from an ego-only teacher has a different effect: temporal accuracy rises while V.IoU slightly falls. Table G2 in the General Response gives the matched settings and intervals.

Reviewer_i4YUOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a training framework, TouchAnnotator, to predict tactile maps from egocentric videos and hand poses. It mixed six datasets with different supervision signals and used a multi-view EMA teacher to guide the egocentric-view-only student through distillation. The experiments include tactile prediction on EgoTouch, contact transfer to FPHA, and agreement on new glove measurements.

Critical strengths
  1. The paper is well-motivated, aiming to solve an important problem of video-to-tactile learning.
  2. The paper addresses practical challenges: a. combining heterogeneous supervision signals across sensors (pressure, geometry, deformation, etc.) b. dealing with head-camera view only during inference time through teacher-student training. The evaluation extends to cross-dataset bare-hand contact evaluation and compares against newly collected tactile measurements, showing the generalizability of the method.
Critical weaknesses

I spotted some issues in the experiments section.

  1. The comparison against TouchAnything on EgoTouch is not fully controlled. TouchAnnotator is evaluated on the authors’ local test split, while the TouchAnything scores are taken directly from the original paper. The authors explicitly state that the equivalence of these splits has not been verified yet, which affects the soundness of the results on EgoTouch comparisons with TouchAnything.
  2. The ablation experiments setup does not sufficiently isolate components in the framework. Privileged-view distillation, heterogeneous supervision, and temporal memory are central to the paper. However, Appendix C shows that the no-distillation run also changes the training mixture, and the memory and teacher-view comparisons involve different initializations. Controlled comparisons are needed to establish which proposed mechanisms drive the improvements.
Questions for authors
  1. Given that TouchAnything was evaluated on FPHA and the newly collected recordings, could the authors also evaluate it on the same EgoTouch test split used for TouchAnnotator, rather than relying on published scores?
  2. Can the authors provide controlled ablations that hold the training data, initialization, and optimization budget fixed while varying distillation, teacher-view access, and memory separately?
3 comments

We sincerely thank Reviewer i4YU for the time and care you put into this review. We appreciate your recognition of the bare-hand and glove evaluations, and your helpful suggestions on comparing the two models on identical episodes and isolating the effects of each training component.

R1Q1: Matched EgoTouch comparison

Given that TouchAnything was evaluated on FPHA and the newly collected recordings, could the authors also evaluate it on the same EgoTouch test split used for TouchAnnotator, rather than relying on published scores?

Yes. Table G1 in the General Response gives the requested same-split comparison. TouchAnnotator achieves higher spatial overlap on unseen objects. The reproduced TouchAnything results match the published scores.

R1Q2: Controlled ablations

Can the authors provide controlled ablations that hold the training data, initialization, and optimization budget fixed while varying distillation, teacher-view access, and memory separately?

Yes. Table G2 in the General Response isolates teacher wrist access (A−B), distillation with an ego-only teacher (B−C), and past-clip memory (A−D). Wrist access and memory each improve performance. The ego-only teacher improves temporal accuracy, with a slight decrease in V.IoU.

Thank you for the additional experiments. However, after considering the discussion, I remain concerned that the reliability and cross-source consistency of the unified tactile supervision are insufficiently validated. The ablations also show modest gains from privileged views and mixed effects from distillation. These limitations reduce my confidence in the core claims, and I am revising my rating from 3 to 2.

Thank you for reviewing the additional experiments and explaining your remaining concerns. We appreciate the opportunity to clarify the scope of the evidence.

Cross-source supervision. Adding geometry-derived supervision from HOI4D and TACO improves unseen EgoTouch C.IoU from .4207 to .4600 and V.IoU from .3371 to .3749. This supports the utility of combining these sources for contact localization.

Normalization reliability. Across the same 860 glove recordings, with predictions and valid cells fixed, TouchAnnotator retains its V.IoU advantage using both device-reported and normalized targets (+4.66 and +5.02 points, respectively). This supports the robustness of the model comparison to target normalization. Our pressure output represents relative tactile intensity on a shared hand grid.

Component contributions. We agree that the gains should be described precisely. Table G2 shows modest positive V.IoU effects from privileged views (+0.82 points) and memory (+0.90 points), with paired intervals above zero. Ego-only distillation improves frame-level contact accuracy (+2.00 points) .

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and Additional Results

We thank the reviewers for their careful reading and constructive suggestions. Several reviews raised two related questions: how TouchAnnotator compares with TouchAnything on identical test episodes, and which training components account for the observed gains.

Table G1. Matched EgoTouch evaluation.

We evaluated TouchAnything and the submitted TouchAnnotator Full checkpoint on the same 184 seen and 76 unseen trajectories, with identical ego RGB, WiLoR poses, scoring masks, and episode weighting.

SplitEgo-only modelSourceT.Acc. ↑C.IoU ↑V.IoU ↑MAE ↓
UnseenTouchAnythingTable 2, published reference.8271.4396.3743.0615
Unseen (76)TouchAnythingMatched re-evaluation.8329.4207.3371.0631
Unseen (76)TouchAnnotator FullTable 2, local evaluation.8687.4779.3821.0638
SeenTouchAnythingTable 2, published reference.8393.4792.4311.0456
Seen (184)TouchAnythingMatched re-evaluation.8492.4961.4087.0409
Seen (184)TouchAnnotator FullTable 2, local evaluation.8613.5051.4478.0479

Our TouchAnything reproduction performs broadly in line with its published EgoTouch results, supporting its use as a baseline on other benchmarks such as FPHA, where we evaluate both models on the same sequences with the same WiLoR poses. Note that on unseen objects, the matched gains are 5.72 C.IoU points (95% paired CI 4.60–6.88) and 4.50 V.IoU points (3.64–5.41).

Table G2. Controlled ablations on unseen EgoTouch.

We trained four students with the same six-source mixture, initialization, seed, and 1,024 updates. All use ego RGB and WiLoR poses at inference. Each row changes one training condition and reports the first student minus the second, in percentage points on the same 76 unseen trajectories. For both metrics, a positive difference favors the first model and a negative difference favors the second.

Setting testedWhat differs between the two studentsΔ T.Acc. ↑Δ V.IoU ↑ [95% paired CI]
Teacher views (A − B)Both use distillation and memory. A’s teacher sees ego plus wrist views when available. B’s teacher sees ego only.+2.02+0.82 [+0.38, +1.28]
Distillation (B − C)Both use memory and the same supervised tactile losses. B also learns from an ego-only teacher. In C, the distillation loss weight is zero.+2.00−0.59 [−0.79, −0.39]
Past-clip memory (A − D)Both use the same multiview teacher and distillation. A can use information from two preceding clips. D uses only the current eight-frame clip.+1.34+0.90 [+0.62, +1.18]
  1. Teacher wrist views (A − B). Wrist cameras are available for only a subset of the training data. Both students use distillation and memory, but only A’s teacher can see those wrist views when available. At test time, both students receive the same ego RGB and estimated hand pose. A gains 2.02 points in T.Acc. and 0.82 points in V.IoU, showing that the additional training views provide more information even when the student has no wrist input at inference.

  2. Distillation from an ego-only teacher (B − C). Both students learn from the same tactile labels, use the same input augmentation, and retain past-clip memory. B also learns from the outputs of an ego-only teacher. C assigns zero weight to that loss. This isolates what distillation contributes when the teacher has no extra camera view. It improves T.Acc. by 2.00 points while reducing V.IoU by 0.59 points. T.Acc. measures whether each frame is correctly classified as contact or no contact. V.IoU also depends on where pressure falls on the hand and how strong it is. Together, these results support using ego-only distillation for frame-level contact detection. The separate A−B comparison shows why we give the teacher wrist views when available: they improve the student’s pressure-map overlap (+0.82 V.IoU on unseen objects).

  3. Past-clip memory (A − D). Both students receive the same current eight-frame clip and train with the same multiview teacher setup. A can also use a stored representation of the two preceding clips, whereas D predicts from the current clip alone. Access to this recent history improves both T.Acc. (+1.34 points) and V.IoU (+0.90 points) on unseen trajectories.

Table G3. Pose-only ablation on EgoTouch unseen.

We conduct an ablation study on the EgoTouch unseen split to evaluate the effectiveness of pose-only inputs without RGB videos.

InputT.Acc. ↑C.IoU ↑V.IoU ↑MAE ↓
WiLoR poses only (w/o RGB)0.78650.39190.30950.0832
TouchAnnotator.8687.4779.3821.0638

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.