- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EgoTouch, an egocentric dataset of bimanual hand–object interaction with dense tactile labels, and TouchAnything, a baseline for predicting these labels from video. EgoTouch contains about 20 h of recordings at 30 Hz (2.1M timepoints) from 10 participants: 1,891 episodes of 208 tasks with over 1,000 objects across five environment categories (Home, Workbench, Office, Retail, Outdoor). The wearable rig combines a head-mounted RGB camera, two wrist-mounted fisheye RGB cameras, Rokoko motion-capture gloves (21 joints per hand) worn under full-hand fabric tactile gloves with 256 sensing points per hand, and HTC Vive trackers on the head and wrists. The streams are associated in software on a common 30 Hz timeline by taking the latest available measurement of each sensor, and the 16×16 tactile arrays are normalized to [0, 1] and remapped to a canonical 21×21 hand-shaped grid. The benchmark asks a model to predict both hands’ tactile maps under four camera configurations (ego only, ego plus either wrist view, and all three views) and evaluates seen and unseen objects with Temporal Accuracy, Contact IoU, Volumetric IoU, and MAE. TouchAnything encodes each view with a frozen DINOv2 ViT-B/14, fuses the views with one softmax weight per view and frame, and applies a temporal transformer over 8-frame clips; hand joints estimated by WiLoR from the ego view then query the visual tokens through cross-attention, and a decoder produces a 21×21 map per hand. View dropout (each wrist view dropped with p = 0.3) lets one checkpoint serve all four configurations. With ego-only input, this multi-view-trained model outperforms an independently trained ego-only model, TA-Ego (seen Volumetric IoU 0.4311 vs. 0.4023), and adding both wrist views raises seen Contact IoU from 0.4792 to 0.5030 and Volumetric IoU from 0.4311 to 0.4575 (unseen: 0.4396 to 0.4496 and 0.3743 to 0.3852; Tab. 2). An adapted PressureVision++ reaches a Volumetric IoU of 0.175/0.150. Ablations show that view dropout prevents a collapse when wrist views are missing (Tab. 3) and that performance improves with training data (Fig. 6).
Critical strengths
-
A unique combination of modalities for learning contact from human video. From a robotics perspective, contact is the information that egocentric video lacks and that dexterous manipulation depends on. Among the ten prior datasets in Tab. 1, only OpenTouch measures touch in the wild, and it covers a single hand; none provides wrist views. EgoTouch adds bimanual full-hand pressure, two wrist views, and diverse scenes (208 tasks, over 1,000 objects, five environment categories), and all participants consented to public release (L103–104). Its layout of a scene camera plus wrist cameras also resembles many robot manipulation platforms, which may make the data easier to relate to robot observations than head-camera-only datasets. The data-scaling study adds that unseenobject Volumetric IoU rises from 0.2466 to 0.3852 between 25% and 100% of the training data (Fig. 6, L428–431), which supports the value of the dataset’s size. This is the main reason I do not recommend a clear rejection: the dataset is a genuine contribution, and my reservations below concern the claims built on it rather than the data itself.
-
Evaluation design that separates training-view and inference-view effects. The comparisons in Tab. 2 are well controlled. TA-Ego and Ego (MV) differ only in the views used for training, and the four configurations of one checkpoint differ only in the RGB inputs, because the pose input is held fixed (L079–080, L462–464). Episodes are kept intact across splits, unseen objects are excluded from training and validation (L302–306), and the metric aggregation, including empty-union cases, is specified precisely (App. A.3.7). The authors are also candid about what their comparisons do not show, for example that the gap to PV++ cannot be attributed to fusion (L407–410) and that the wristview benefit has not been stratified by occlusion (L483–485, App. A.1.3). This makes the results easy to interpret. The view-dropout ablation in Tab. 3 also shows a clear effect. Without dropout, removing the wrist views at inference lowers Volumetric IoU from 0.4441 to 0.3233 (−27.20%), below TA-Ego’s 0.4023; with dropout the drop is only 5.78%, and all-view performance also rises from 0.4441 to 0.4575. Together with the gain of Ego (MV) over TA-Ego on both seen and unseen objects, this suggests that wrist cameras help during training even if a deployed system has only a head camera. This matters for my assessment because, with controlled comparisons and clearly stated limits, much of what is missing below can be computed from data the authors already have.
Critical weaknesses
-
The motivating use, touch labels for sensor-free egocentric video, is untested and hard to reach with gloved-hand data. The abstract motivates vision-to-touch as a way to “extend tactile supervision to recordings acquired without tactile sensors” (L014–016), that is, to ordinary egocentric video such as Ego4D or EgoDex, in which hands are bare. In EgoTouch, however, every participant wears a black fabric tactile glove over a Rokoko glove (Fig. 2, L132), so the model only ever sees gloved hands; the one failure case the paper reports involves the black glove blending into black shorts (App. A.7, L1406–1411). The WiLoR pose input is likewise estimated on gloved hands. The authors acknowledge that “transfer to bare hands and unseen participants remains unevaluated” and that the hardware “may alter dexterity and grasping” (L478–480). The only mitigation is a glove color augmentation (L991), whose effect is not reported. The value of the dataset for its stated purpose therefore rests on an untested assumption. This is the first of the two reasons I lean towards rejection: labeling sensor-free video is one of the two claims that give the paper its purpose, and it is not yet supported (see Question 3).
-
The evidence for the multi-view claims is thin, and the benchmark lacks reference points. The abstract reports relative gains (7.2%, 5.0%, and 6.1%; L024–027) without stating that they are relative or that they refer to seen objects. In absolute terms they are 0.024–0.029 IoU, and on unseen objects adding the wrist views gains only 0.010 Contact IoU and 0.011 Volumetric IoU (Tab. 2). No standard deviations are reported, and the limitations list repeated-seed studies as future work (L483–485), so it is unclear whether these gains exceed run-to-run variance. The gains are also mixed across scenarios: on unseen objects, Workbench Temporal Accuracy drops from 0.8096 to 0.7538, and Outdoor Temporal Accuracy and Office Contact IoU drop as well (L415–417). A single wrist view accounts for essentially all of the aggregate gain (L417–419), so a benefit of the second wrist camera, the bimanual-specific part of the setup, is not shown. Finally, the paper motivates wrist views by occlusion (L048–051), but the only evidence that they help where the ego view is occluded is one qualitative example (Fig. 7), which, as the authors note, does “not establish the frequency of such recoveries” (L464–465). Beyond the view comparisons, the benchmark lacks reference points. The only external baseline, an adapted single-frame PV++, differs in backbone, temporal context, pose conditioning, and output formulation (L407–410). There is no pose-only model (WiLoR joints without RGB), no RGB-only model, and no trivial baseline such as a per-task mean map, so it is unclear how much of the 0.43– 0.46 Volumetric IoU on seen objects comes from visual evidence rather than from hand configuration or task priors; this matters most for seen objects, where the same participants repeat the same tasks. The cross-view fusion of contribution (iii) is also not compared with simple alternatives such as token concatenation, even though it mixes patches that share an index but not a location across cameras (L257–258). This is the second reason I lean towards rejection: the multi-view benefit is the other claim that gives the paper its purpose, and without variance estimates, an occlusion analysis, or simple baselines, the current numbers do not yet establish it (see Questions 1 and 2).
Minor Issues
- Tactile units and sensor quality. The tactile values are normalized to [0, 1] (L831–832), but the normalization constant is not given (a fixed sensor range, or a per-episode or per-participant value?). The results are in normalized units without physical calibration (L481–482), and the contact threshold τ = 0.1 (L309–310), which also defines the up-weighted region of the loss (L294–296), is not justified. Since contact is defined by a fixed threshold, please specify the normalization and reportbasic characteristics of the fabric glove, such as noise floor, drift, and hysteresis, for example from a known-load test.
- Synchronization. The streams are associated by taking the latest cached value at each 30 Hz tick, and the authors note that “a shared frame index alone does not establish a bound on inter-sensor timing error” (L824–825). Missing auxiliary values may be filled from the nearest valid measurement (L826–827), and duplicated wrist-camera frames occur (L849). Contact onsets and offsets are where tactile labels matter most, so please report measured offsets between streams, for example from a tap visible in all modalities, and how often values are reused or frames duplicated.
- Scope of the generalization and scale claims. All 10 participants appear in both training and test data (L306–307), so generalization to new people is untested, and the per-participant distribution of the data is not reported. The “large-scale” claim (L017, L052) should be qualified: by frame count, EgoTouch (2.1M) is smaller than EgoPressure (4.3M) and EgoDex (90M) in Tab. 1, and what distinguishes it is the combination of modalities rather than size.
- Presentation. The title reads “TouchAnything: A Dataset and Framework”, while the dataset is called EgoTouch and TouchAnything is the model, which is confusing. The abstract should state that the reported percentages are relative changes on seen objects. The appendix describes the tactile gloves as “custom pressure-sensing gloves” (L794), whereas the main text names a JQ-Industries glove (Fig. 2); please clarify. Evaluation also uses bidirectional attention within each clip (L370– 371, L909), so the numbers do not carry over to causal, online use such as closed-loop robot control; this is acknowledged (L482–483) but worth stating next to the results.
- No downstream evidence. The introduction motivates the dataset by embodied intelligence (L033– 035), but no experiment uses the predicted touch in a downstream task. Even a small proof of concept, for example adding predicted touch to a manipulation policy’s input or using it for contactaware retargeting, would strengthen the case.
Questions for authors
- Variance and baselines (additional analysis). Repeated-seed results with standard deviations for Tab. 2, and pose-only, RGB-only, and trivial (e.g., per-task mean map) baselines.
- Occlusion (additional analysis). An occlusion-stratified analysis of the wrist-view gain. The dataset already provides a proxy: the per-frame error of the ego-view WiLoR joints against the Rokoko joints indicates how occluded the hands are in the ego view.
- Bare hands (additional analysis). Any evidence on transfer to bare hands: (a) at least qualitative results on bare-hand egocentric clips and, ideally, a small bare-hand test set with touch labels, for example from a pressure-sensing surface as in EgoPressure; (b) an analysis of sensitivity to hand appearance, for example with and without the glove color augmentation; and (c) WiLoR accuracy on gloved hands against the Rokoko joints that the dataset already contains, which would separate pose errors from touch errors.
- Tactile normalization and timing (clarification). The tactile normalization procedure and measured timing offsets between streams (Minor issues 1 and 2).
10 comments
Part 2/9: Direct test for Reason 1 (W1 / Q3); Minor Issue 5
W1 / Q3 — The motivating use, touch labels for sensor-free video, is untested and hard to reach with gloved-hand data
We agree with the reviewer's assessment: the submission motivates the work by "extending tactile supervision to recordings acquired without tactile sensors", but all experiments are on hands wearing black gloves, and there is no direct evidence. We first test this use directly (this part) and then answer items (a), (b) and (c) of the reviewer's Q3 (Parts 3 and 4).
Direct test: labeling bare-hand video and using the labels for downstream training.
- EgoDex is egocentric video of bare hands, with no tactile sensors and only an egocentric view. We take a subset of about 400 hours and label it with TouchAnything (the Ego (MV) configuration of Table 2).
- We add a tactile branch to OpenWAM, which jointly predicts future video and actions, and run one intermediate training stage on the EgoDex subset + AgiBot + SharpaDex.
- We compare three settings. OpenWAM has no tactile branch. The other two both have the tactile branch and both use the touch that comes with SharpaDex; they differ only in whether the EgoDex tactile labels come from the adapted PV++ or from TouchAnything.
- The three models are fine-tuned and then evaluated in closed loop on two simulation benchmarks, UniVTAC (8 tasks, 100 trials each) and Bench2Dex (3 tasks, 50 trials each).
| Method | Insert HDMI | Insert Hole | Insert Tube | Grasp Classify | Lift Can | Lift Bottle | Pull-out Key | Put Bottle in Shelf | Average |
|---|---|---|---|---|---|---|---|---|---|
| OpenWAM | 28% | 79% | 88% | 97% | 73% | 97% | 90% | 92% | 80.5% |
| OpenWAM + PV++ labels | 31% | 78% | 84% | 92% | 76% | 97% | 88% | 88% | 79.3% |
| OpenWAM + TouchAnything labels | 42% | 94% | 89% | 97% | 92% | 98% | 96% | 98% | 88.3% |
| Method | Can placement (26) | Tool tray (32) | Puzzle (73) | Mean of three tasks | Total over three tasks |
|---|---|---|---|---|---|
| OpenWAM | 50% (25/50) | 22% (11/50) | 16% (8/50) | 29.3% | 44/150 |
| OpenWAM + PV++ labels | 48% (24/50) | 24% (12/50) | 12% (6/50) | 28.0% | 42/150 |
| OpenWAM + TouchAnything labels | 52% (26/50) | 34% (17/50) | 30% (15/50) | 38.7% | 58/150 |
- TouchAnything labels improve success. The average success rate on UniVTAC rises from 80.5% to 88.3%, with 7 of the 8 tasks improving; the three-task mean on Bench2Dex rises from 29.3% to 38.7%.
- PV++ labels do not (79.3% and 28.0%). The last two settings have identical model structure, training data and recipe; only the source of the tactile labels on EgoDex differs, so the difference comes from label quality.
The labels that a model trained on gloved-hand data produces for bare-hand video are useful for downstream training.
Minor Issue 5 — No downstream evidence
Training a robot manipulation model with predicted tactile labels is reported above (average success rate on UniVTAC from 80.5% to 88.3%). A second evaluation corresponds to the contact-aware retargeting the reviewer mentions: retargeting human-hand video to a robot requires the start and end time of every grasp, usually obtained with a heuristic on the thumb–index distance. With the events derived from measured touch as ground truth:
| Event source | F1: Seen | F1: Unseen | Timing error (frames): Seen | Timing error (frames): Unseen |
|---|---|---|---|---|
| Thumb–index distance heuristic (WiLoR) | 0.53 | 0.50 | 8.6 | 9.4 |
| Touch predicted by PV++ (adapted) | 0.38 | 0.33 | 11.2 | 12.5 |
| TouchAnything, ego only | 0.71 | 0.65 | 4.6 | 5.4 |
| TouchAnything, all views | 0.74 | 0.67 | 4.1 | 5.0 |
With the touch predicted by TouchAnything, already with the egocentric view as the only input, F1 is 0.18 (seen) and 0.15 (unseen) above the heuristic, with about half of its timing error.
Part 3/9: Bare hands and appearance (W1 / Q3, continued)
W1 / Q3 (continued) — Items (a) and (b): results on bare hands and sensitivity to hand appearance
Models compared. Every model in (a) and (b) is TouchAnything; the rows differ only in the training-time data augmentation or in the input. At test time all of them use only the egocentric view, the same configuration as for labeling EgoDex:
- TouchAnything (paper setting): Ego (MV) in Table 2. It is trained with the color augmentation of the paper: with probability 0.2 per clip, the glove region in the egocentric frames, given by the glove segmentation mask, is replaced by a randomly chosen appearance out of 20, including skin colors, other fabric colors and patterns such as stripes.
- TouchAnything (no color augmentation): the color augmentation above is removed; everything else is the same as the paper setting.
- TouchAnything (+ generative augmentation): on top of the color augmentation, one more training-data augmentation is added, which replaces the glove in the training videos with a bare hand; the architecture and the inputs of the model are unchanged. Concretely, the video generation model Wan2.1 VACE repaints the glove region of the egocentric video: the segmentation mask of the glove region is given by SAM3, a rendered hand mesh serves as the hand-shape prompt, and all pixels outside the region are kept from the original video. The tactile labels are those of the original video.
- TouchAnything (hand masked): the hand pixels in the input images are set to gray, so the model does not see the appearance of the hand at all.
- TouchAnything (pose-only input): no image input, hand joints only.
(a) Results on bare hands. 3 participants, wearing no gloves at all, repeatedly press, slide and lift their fingers and palms on a pressure-sensing mat, for a total of 36 clips and about 24,000 frames; the setup is similar to EgoPressure, which the reviewer mentions. The mat directly measures the pressure between the hand and its surface: frames whose reading exceeds the unloaded noise level are contact frames, 52% of all frames. Contact-state accuracy of each model:
| Model | Contact-state accuracy |
|---|---|
| Always predict contact (reference) | 0.520 |
| TouchAnything (pose-only input) | 0.604 |
| TouchAnything (hand masked) | 0.801 |
| TouchAnything (paper setting) | 0.822 |
| TouchAnything (+ generative augmentation) | 0.851 |
- TouchAnything can determine the contact state on bare hands. In the paper setting its accuracy is 0.822, which is 0.302 above always predicting contact. For reference, its Temporal Accuracy on the seen / unseen test sets of EgoTouch is 0.8393 / 0.8271.
- The contact state is determined mainly from the image. With pose-only input the accuracy is only 0.604, which is 0.218 below the paper setting.
- The decision does not depend on the appearance of the hand. With the hand masked, the accuracy is still 0.801; with the generative augmentation added, it rises to 0.851.
On bare-hand clips from EgoDex, the predicted contact timing also agrees with the grasp and release motions visible in the video.
(b) Sensitivity to hand appearance. We collected a new appearance-transfer test set with tactile ground truth: 2 participants and 42 episodes (14 per appearance), with a thin nitrile glove in skin color, white or blue worn over the tactile glove. Volumetric IoU of each model:
| Model | Black glove (standard test set, seen) | Skin color | White | Blue |
|---|---|---|---|---|
| TouchAnything (pose-only input) | 0.3052 | 0.2981 | 0.3007 | 0.3012 |
| TouchAnything (no color augmentation) | 0.4336 | 0.3284 | 0.3571 | 0.3629 |
| TouchAnything (hand masked) | 0.3905 | 0.3712 | 0.3748 | 0.3761 |
| TouchAnything (paper setting) | 0.4311 | 0.3795 | 0.3962 | 0.4008 |
| TouchAnything (+ generative augmentation) | 0.4287 | 0.4046 | 0.4095 | 0.4118 |
- TouchAnything remains effective under the three new appearances. In the paper setting its V.IoU is 0.3795–0.4008, which is 88%–93% of the value on the black glove, and clearly above pose-only input (about 0.30) under every appearance.
- Appearance augmentation is effective. On the skin-colored glove, V.IoU is 0.3284 without color augmentation, 0.3795 in the paper setting, and 0.4046 with the generative augmentation added, which lowers the result on the standard test set by only 0.0024.
- Most of the accuracy does not depend on the appearance of the glove. With the hand masked, the model does not see the appearance of the hand at all and stays at about 0.37 under the three appearances.
Part 1/9: Overview
We thank the reviewer for the exceptionally careful review, and for pointing out which analyses can be done with the data we already have. We carried out every analysis the reviewer suggested and, in addition, collected three small new sets of data. The reviewer gives two reasons for leaning towards rejection; we first summarize our response to these two and then go through every point.
Overview