- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TouchAnnotator, a framework for predicting hand contact and relative pressure from egocentric RGB videos and estimated hand poses. It integrates six existing datasets into a unified tactile representation and introduces privileged-view distillation and temporal memory. The method is evaluated on existing benchmarks and a newly collected tactile evaluation set.
Critical strengths
-
Meaningful problem with practical potential. Predicting tactile information from RGB videos is an interesting and practically valuable research direction. If sufficiently accurate, the proposed framework could enable large-scale tactile annotation of human demonstration videos without requiring additional tactile sensors, potentially benefiting embodied learning.
-
Integration of heterogeneous tactile supervision sources. The paper makes a meaningful effort to unify different tactile supervision sources, including sensor measurements and geometry-derived contact labels. The source-aware training strategy is reasonable, and the additional evaluation on newly collected real tactile data provides useful evidence beyond existing benchmarks.
Critical weaknesses
-
Insufficient validation of the unified tactile supervision. Although the paper provides evaluations on real tactile data, the accuracy and consistency of the unified tactile mapping remain insufficiently validated. In particular, geometry-derived contact labels are based on proximity thresholds rather than direct contact measurements, while pressure mappings across different sensors may introduce systematic discrepancies. Since this unified representation is fundamental to the proposed framework, its reliability requires stronger quantitative evidence.
-
Insufficient evidence supporting the effectiveness of the proposed method. The supplementary material reports several ablation configurations but explicitly acknowledges that confounding factors prevent isolating the effects of privileged-view distillation and temporal memory. This makes it difficult to assess the independent contributions of these key components. Furthermore, without strong pose-only baselines, it remains unclear to what extent the model captures tactile-specific information beyond common visual and hand-pose correlations.
Questions for authors
Q1. Can the authors provide quantitative validation of the unified tactile mapping, particularly the agreement between geometry-derived contact labels and real tactile measurements? How do the authors ensure consistency across different sensor modalities and pressure normalization schemes?
Q2. Can the authors provide a strong pose-only baseline to better demonstrate the model's tactile-specific prediction capability? Additionally, the FPHA evaluation appears to use annotated hand poses in the supplementary material but estimated hand poses in the main paper. Could the authors clarify the actual input setting and its implications for the reported results?
Q3. Given the confounding factors acknowledged in the supplementary material, can the authors provide controlled ablations that independently evaluate privileged-view distillation and temporal memory under identical training data, initialization, and evaluation settings? Such experiments would be important to establish whether these components provide consistent improvements over simpler alternatives.
3 comments
Thank you for the additional experiments. While G1-G3 address some of my concerns, the reliability and cross-source consistency of the unified tactile supervision remain insufficiently validated. In addition, the ablation result do not seem to provide sufficiently strong evidence for the paper’s core claims, particularly given the marginal and inconsistent improvements form privileged-view distillation. Therefore, I maintain my original rating.
Thank you for clarifying your remaining concerns. We understand that G1–G3 leave an important question about the supervision itself.
How the sources are combined. Geometry-derived labels supervise binary contact, while tactile measurements supervise relative pressure on the shared hand grid. The two prediction heads share learned features and are jointly trained. This allows the supervision sources to contribute to a common representation while retaining their respective target meanings.
The additional data-source experiment provides evidence of their practical compatibility. Adding HOI4D and TACO improves unseen C.IoU from .4207 to .4600 and V.IoU from .3371 to .3749. Together with the consistent model ranking before and after glove-target normalization, this supports the utility of combining the sources.
What the distillation ablations establish. A−B isolates privileged wrist-view access with distillation and memory fixed, yielding +0.82 V.IoU points [95% paired CI +0.38, +1.28]. B−C separately tests distillation from an ego-only teacher, yielding +2.00 T.Acc. points and −0.59 V.IoU points. Thus, the mixed B−C result concerns a teacher receiving the same camera view as the student. A−B supports a modest benefit from additional teacher views.
We will make these distinctions explicit and align the claims with the demonstrated benefits and remaining validation limits.
We sincerely thank Reviewer dz86 for your time and thoughtful questions about tactile supervision and pose inputs. We appreciate your recognition of the practical value of tactile annotation.
R3Q1: Geometry–tactile agreement and pressure normalization
How the two sources supervise the model. The contact and pressure heads have separate output weights but share the features from which they predict. During joint training, geometry-derived binary contact labels and glove pressure measurements both update those shared features. The common hand grid thus represents contact location and relative tactile intensity across egocentric datasets. Pressure is reported on a normalized scale.
Adding geometry-derived contact labels. On unseen EgoTouch objects, adding HOI4D and TACO to glove training improved temporal accuracy and both spatial-overlap metrics. Pressure MAE increased.
Agreement before and after target normalization. We align HumanTouch to the EgoTouch relative-pressure distribution by monotone quantile mapping. To test sensitivity to the scoring target, we held both models’ predictions fixed and scored the same valid cells across 860 synchronized glove recordings (139,102 frames).
TouchAnnotator leads on both overlap metrics under either target. Its V.IoU advantage is 4.66 points before and 5.02 points after normalization.
R3Q2: Pose-only baseline and FPHA inputs
Pose-only baseline. With the WiLoR pose sequence held fixed, adding ego RGB improves C.IoU by 8.80 points and V.IoU by 7.26 points on unseen EgoTouch objects, showing that visual cues help localize contact and predict its relative intensity (Table G3, General Comment).
FPHA. We use hand poses annotated by WiLoR as input. During teacher training, we additionally incorporate RGB videos from three camera views. In contrast, during student distillation, we use only the annotated hand poses and egocentric RGB videos to simulate the practical setting of bare-hand egocentric data collection.
R3Q3: Independent effects of privileged views and memory
Table G2 in the General Comment addresses the two effects separately. With memory fixed, giving the teacher wrist views improves the student’s V.IoU. With teacher views fixed, adding memory also improves V.IoU. Both paired V.IoU intervals lie above zero on unseen trajectories, while the corresponding C.IoU intervals include zero.