acceptodds
Under review as a conference paper at ICLR 2027

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Abstract

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

Primary area:
datasets and benchmarks
Keywords:
Egocentric Vision, Tactile Sensing, Vision-to-Touch Prediction, Bimanual Manipulation, Hand-Object Interaction, Multimodal Learning, Embodied AI, Multi-View Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_asV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoTouch, an egocentric dataset of bimanual hand–object interaction with dense tactile labels, and TouchAnything, a baseline for predicting these labels from video. EgoTouch contains about 20 h of recordings at 30 Hz (2.1M timepoints) from 10 participants: 1,891 episodes of 208 tasks with over 1,000 objects across five environment categories (Home, Workbench, Office, Retail, Outdoor). The wearable rig combines a head-mounted RGB camera, two wrist-mounted fisheye RGB cameras, Rokoko motion-capture gloves (21 joints per hand) worn under full-hand fabric tactile gloves with 256 sensing points per hand, and HTC Vive trackers on the head and wrists. The streams are associated in software on a common 30 Hz timeline by taking the latest available measurement of each sensor, and the 16×16 tactile arrays are normalized to [0, 1] and remapped to a canonical 21×21 hand-shaped grid. The benchmark asks a model to predict both hands’ tactile maps under four camera configurations (ego only, ego plus either wrist view, and all three views) and evaluates seen and unseen objects with Temporal Accuracy, Contact IoU, Volumetric IoU, and MAE. TouchAnything encodes each view with a frozen DINOv2 ViT-B/14, fuses the views with one softmax weight per view and frame, and applies a temporal transformer over 8-frame clips; hand joints estimated by WiLoR from the ego view then query the visual tokens through cross-attention, and a decoder produces a 21×21 map per hand. View dropout (each wrist view dropped with p = 0.3) lets one checkpoint serve all four configurations. With ego-only input, this multi-view-trained model outperforms an independently trained ego-only model, TA-Ego (seen Volumetric IoU 0.4311 vs. 0.4023), and adding both wrist views raises seen Contact IoU from 0.4792 to 0.5030 and Volumetric IoU from 0.4311 to 0.4575 (unseen: 0.4396 to 0.4496 and 0.3743 to 0.3852; Tab. 2). An adapted PressureVision++ reaches a Volumetric IoU of 0.175/0.150. Ablations show that view dropout prevents a collapse when wrist views are missing (Tab. 3) and that performance improves with training data (Fig. 6).

Critical strengths
  1. A unique combination of modalities for learning contact from human video. From a robotics perspective, contact is the information that egocentric video lacks and that dexterous manipulation depends on. Among the ten prior datasets in Tab. 1, only OpenTouch measures touch in the wild, and it covers a single hand; none provides wrist views. EgoTouch adds bimanual full-hand pressure, two wrist views, and diverse scenes (208 tasks, over 1,000 objects, five environment categories), and all participants consented to public release (L103–104). Its layout of a scene camera plus wrist cameras also resembles many robot manipulation platforms, which may make the data easier to relate to robot observations than head-camera-only datasets. The data-scaling study adds that unseenobject Volumetric IoU rises from 0.2466 to 0.3852 between 25% and 100% of the training data (Fig. 6, L428–431), which supports the value of the dataset’s size. This is the main reason I do not recommend a clear rejection: the dataset is a genuine contribution, and my reservations below concern the claims built on it rather than the data itself.

  2. Evaluation design that separates training-view and inference-view effects. The comparisons in Tab. 2 are well controlled. TA-Ego and Ego (MV) differ only in the views used for training, and the four configurations of one checkpoint differ only in the RGB inputs, because the pose input is held fixed (L079–080, L462–464). Episodes are kept intact across splits, unseen objects are excluded from training and validation (L302–306), and the metric aggregation, including empty-union cases, is specified precisely (App. A.3.7). The authors are also candid about what their comparisons do not show, for example that the gap to PV++ cannot be attributed to fusion (L407–410) and that the wristview benefit has not been stratified by occlusion (L483–485, App. A.1.3). This makes the results easy to interpret. The view-dropout ablation in Tab. 3 also shows a clear effect. Without dropout, removing the wrist views at inference lowers Volumetric IoU from 0.4441 to 0.3233 (−27.20%), below TA-Ego’s 0.4023; with dropout the drop is only 5.78%, and all-view performance also rises from 0.4441 to 0.4575. Together with the gain of Ego (MV) over TA-Ego on both seen and unseen objects, this suggests that wrist cameras help during training even if a deployed system has only a head camera. This matters for my assessment because, with controlled comparisons and clearly stated limits, much of what is missing below can be computed from data the authors already have.

Critical weaknesses
  1. The motivating use, touch labels for sensor-free egocentric video, is untested and hard to reach with gloved-hand data. The abstract motivates vision-to-touch as a way to “extend tactile supervision to recordings acquired without tactile sensors” (L014–016), that is, to ordinary egocentric video such as Ego4D or EgoDex, in which hands are bare. In EgoTouch, however, every participant wears a black fabric tactile glove over a Rokoko glove (Fig. 2, L132), so the model only ever sees gloved hands; the one failure case the paper reports involves the black glove blending into black shorts (App. A.7, L1406–1411). The WiLoR pose input is likewise estimated on gloved hands. The authors acknowledge that “transfer to bare hands and unseen participants remains unevaluated” and that the hardware “may alter dexterity and grasping” (L478–480). The only mitigation is a glove color augmentation (L991), whose effect is not reported. The value of the dataset for its stated purpose therefore rests on an untested assumption. This is the first of the two reasons I lean towards rejection: labeling sensor-free video is one of the two claims that give the paper its purpose, and it is not yet supported (see Question 3).

  2. The evidence for the multi-view claims is thin, and the benchmark lacks reference points. The abstract reports relative gains (7.2%, 5.0%, and 6.1%; L024–027) without stating that they are relative or that they refer to seen objects. In absolute terms they are 0.024–0.029 IoU, and on unseen objects adding the wrist views gains only 0.010 Contact IoU and 0.011 Volumetric IoU (Tab. 2). No standard deviations are reported, and the limitations list repeated-seed studies as future work (L483–485), so it is unclear whether these gains exceed run-to-run variance. The gains are also mixed across scenarios: on unseen objects, Workbench Temporal Accuracy drops from 0.8096 to 0.7538, and Outdoor Temporal Accuracy and Office Contact IoU drop as well (L415–417). A single wrist view accounts for essentially all of the aggregate gain (L417–419), so a benefit of the second wrist camera, the bimanual-specific part of the setup, is not shown. Finally, the paper motivates wrist views by occlusion (L048–051), but the only evidence that they help where the ego view is occluded is one qualitative example (Fig. 7), which, as the authors note, does “not establish the frequency of such recoveries” (L464–465). Beyond the view comparisons, the benchmark lacks reference points. The only external baseline, an adapted single-frame PV++, differs in backbone, temporal context, pose conditioning, and output formulation (L407–410). There is no pose-only model (WiLoR joints without RGB), no RGB-only model, and no trivial baseline such as a per-task mean map, so it is unclear how much of the 0.43– 0.46 Volumetric IoU on seen objects comes from visual evidence rather than from hand configuration or task priors; this matters most for seen objects, where the same participants repeat the same tasks. The cross-view fusion of contribution (iii) is also not compared with simple alternatives such as token concatenation, even though it mixes patches that share an index but not a location across cameras (L257–258). This is the second reason I lean towards rejection: the multi-view benefit is the other claim that gives the paper its purpose, and without variance estimates, an occlusion analysis, or simple baselines, the current numbers do not yet establish it (see Questions 1 and 2).

Minor Issues

  1. Tactile units and sensor quality. The tactile values are normalized to [0, 1] (L831–832), but the normalization constant is not given (a fixed sensor range, or a per-episode or per-participant value?). The results are in normalized units without physical calibration (L481–482), and the contact threshold τ = 0.1 (L309–310), which also defines the up-weighted region of the loss (L294–296), is not justified. Since contact is defined by a fixed threshold, please specify the normalization and reportbasic characteristics of the fabric glove, such as noise floor, drift, and hysteresis, for example from a known-load test.
  2. Synchronization. The streams are associated by taking the latest cached value at each 30 Hz tick, and the authors note that “a shared frame index alone does not establish a bound on inter-sensor timing error” (L824–825). Missing auxiliary values may be filled from the nearest valid measurement (L826–827), and duplicated wrist-camera frames occur (L849). Contact onsets and offsets are where tactile labels matter most, so please report measured offsets between streams, for example from a tap visible in all modalities, and how often values are reused or frames duplicated.
  3. Scope of the generalization and scale claims. All 10 participants appear in both training and test data (L306–307), so generalization to new people is untested, and the per-participant distribution of the data is not reported. The “large-scale” claim (L017, L052) should be qualified: by frame count, EgoTouch (2.1M) is smaller than EgoPressure (4.3M) and EgoDex (90M) in Tab. 1, and what distinguishes it is the combination of modalities rather than size.
  4. Presentation. The title reads “TouchAnything: A Dataset and Framework”, while the dataset is called EgoTouch and TouchAnything is the model, which is confusing. The abstract should state that the reported percentages are relative changes on seen objects. The appendix describes the tactile gloves as “custom pressure-sensing gloves” (L794), whereas the main text names a JQ-Industries glove (Fig. 2); please clarify. Evaluation also uses bidirectional attention within each clip (L370– 371, L909), so the numbers do not carry over to causal, online use such as closed-loop robot control; this is acknowledged (L482–483) but worth stating next to the results.
  5. No downstream evidence. The introduction motivates the dataset by embodied intelligence (L033– 035), but no experiment uses the predicted touch in a downstream task. Even a small proof of concept, for example adding predicted touch to a manipulation policy’s input or using it for contactaware retargeting, would strengthen the case.
Questions for authors
  1. Variance and baselines (additional analysis). Repeated-seed results with standard deviations for Tab. 2, and pose-only, RGB-only, and trivial (e.g., per-task mean map) baselines.
  2. Occlusion (additional analysis). An occlusion-stratified analysis of the wrist-view gain. The dataset already provides a proxy: the per-frame error of the ego-view WiLoR joints against the Rokoko joints indicates how occluded the hands are in the ego view.
  3. Bare hands (additional analysis). Any evidence on transfer to bare hands: (a) at least qualitative results on bare-hand egocentric clips and, ideally, a small bare-hand test set with touch labels, for example from a pressure-sensing surface as in EgoPressure; (b) an analysis of sensitivity to hand appearance, for example with and without the glove color augmentation; and (c) WiLoR accuracy on gloved hands against the Rokoko joints that the dataset already contains, which would separate pose errors from touch errors.
  4. Tactile normalization and timing (clarification). The tactile normalization procedure and measured timing offsets between streams (Minor issues 1 and 2).
10 comments

Part 1/9: Overview

We thank the reviewer for the exceptionally careful review, and for pointing out which analyses can be done with the data we already have. We carried out every analysis the reviewer suggested and, in addition, collected three small new sets of data. The reviewer gives two reasons for leaning towards rejection; we first summarize our response to these two and then go through every point.

Overview

  • Reason 1 (W1 / Q3): the use of labeling sensor-free video with touch is untested. We tested it directly: labeling bare-hand EgoDex videos with TouchAnything and training a manipulation model on these labels clearly raises downstream success (Part 2). On a bare-hand test with ground truth from a pressure-sensing mat, the contact-state accuracy of TouchAnything is 0.822 (0.520 when always predicting contact, 0.604 with hand pose as the only input), and 0.851 once the generative glove-to-hand augmentation is added (Part 3).
  • Reason 2 (W2 / Q1 / Q2): the evidence for the multi-view claims is thin, and the benchmark lacks reference points.
    • Variance. On unseen objects, the V.IoU gain from the wrist views is +0.0109, with a 95% confidence interval of [0.0038, 0.0180]; the within-seed difference over 3 random seeds is +0.0109 ± 0.0018. (Part 5)
    • Occlusion. The gain from the wrist views increases monotonically with occlusion: +0.014, +0.026 and +0.039 over the three tiers on seen objects. (Part 6)
    • Second wrist camera. In the recordings with low visibility of the opposite hand, the V.IoU with both wrists is higher than with a single wrist by 0.0275 (seen) and 0.0143 (unseen). (Part 7)
    • Reference baselines. V.IoU on seen objects: per-task mean map 0.2853, pose only 0.3052, RGB only 0.3517, TA-Ego 0.4023. (Part 8)
    • Fusion. Across four fusion schemes, the all-view V.IoU lies between 0.4548 and 0.4652 on seen and between 0.3836 and 0.3907 on unseen objects, and the multi-view conclusions hold for all of them. (Part 6)
  • Minor Issues 1–5. The normalization constant is computed per episode, and in the tap test the offset of touch relative to the egocentric camera is about 1 frame (Part 9); results on unseen participants (Part 4), with causal attention (Part 9), and a second downstream evaluation (Part 2) are added.

Part 2/9: Direct test for Reason 1 (W1 / Q3); Minor Issue 5

W1 / Q3 — The motivating use, touch labels for sensor-free video, is untested and hard to reach with gloved-hand data

We agree with the reviewer's assessment: the submission motivates the work by "extending tactile supervision to recordings acquired without tactile sensors", but all experiments are on hands wearing black gloves, and there is no direct evidence. We first test this use directly (this part) and then answer items (a), (b) and (c) of the reviewer's Q3 (Parts 3 and 4).

Direct test: labeling bare-hand video and using the labels for downstream training.

  • EgoDex is egocentric video of bare hands, with no tactile sensors and only an egocentric view. We take a subset of about 400 hours and label it with TouchAnything (the Ego (MV) configuration of Table 2).
  • We add a tactile branch to OpenWAM, which jointly predicts future video and actions, and run one intermediate training stage on the EgoDex subset + AgiBot + SharpaDex.
  • We compare three settings. OpenWAM has no tactile branch. The other two both have the tactile branch and both use the touch that comes with SharpaDex; they differ only in whether the EgoDex tactile labels come from the adapted PV++ or from TouchAnything.
  • The three models are fine-tuned and then evaluated in closed loop on two simulation benchmarks, UniVTAC (8 tasks, 100 trials each) and Bench2Dex (3 tasks, 50 trials each).
MethodInsert HDMIInsert HoleInsert TubeGrasp ClassifyLift CanLift BottlePull-out KeyPut Bottle in ShelfAverage
OpenWAM28%79%88%97%73%97%90%92%80.5%
OpenWAM + PV++ labels31%78%84%92%76%97%88%88%79.3%
OpenWAM + TouchAnything labels42%94%89%97%92%98%96%98%88.3%
MethodCan placement (26)Tool tray (32)Puzzle (73)Mean of three tasksTotal over three tasks
OpenWAM50% (25/50)22% (11/50)16% (8/50)29.3%44/150
OpenWAM + PV++ labels48% (24/50)24% (12/50)12% (6/50)28.0%42/150
OpenWAM + TouchAnything labels52% (26/50)34% (17/50)30% (15/50)38.7%58/150
  • TouchAnything labels improve success. The average success rate on UniVTAC rises from 80.5% to 88.3%, with 7 of the 8 tasks improving; the three-task mean on Bench2Dex rises from 29.3% to 38.7%.
  • PV++ labels do not (79.3% and 28.0%). The last two settings have identical model structure, training data and recipe; only the source of the tactile labels on EgoDex differs, so the difference comes from label quality.

The labels that a model trained on gloved-hand data produces for bare-hand video are useful for downstream training.

Minor Issue 5 — No downstream evidence

Training a robot manipulation model with predicted tactile labels is reported above (average success rate on UniVTAC from 80.5% to 88.3%). A second evaluation corresponds to the contact-aware retargeting the reviewer mentions: retargeting human-hand video to a robot requires the start and end time of every grasp, usually obtained with a heuristic on the thumb–index distance. With the events derived from measured touch as ground truth:

Event sourceF1: SeenF1: UnseenTiming error (frames): SeenTiming error (frames): Unseen
Thumb–index distance heuristic (WiLoR)0.530.508.69.4
Touch predicted by PV++ (adapted)0.380.3311.212.5
TouchAnything, ego only0.710.654.65.4
TouchAnything, all views0.740.674.15.0

With the touch predicted by TouchAnything, already with the egocentric view as the only input, F1 is 0.18 (seen) and 0.15 (unseen) above the heuristic, with about half of its timing error.

Part 3/9: Bare hands and appearance (W1 / Q3, continued)

W1 / Q3 (continued) — Items (a) and (b): results on bare hands and sensitivity to hand appearance

Models compared. Every model in (a) and (b) is TouchAnything; the rows differ only in the training-time data augmentation or in the input. At test time all of them use only the egocentric view, the same configuration as for labeling EgoDex:

  • TouchAnything (paper setting): Ego (MV) in Table 2. It is trained with the color augmentation of the paper: with probability 0.2 per clip, the glove region in the egocentric frames, given by the glove segmentation mask, is replaced by a randomly chosen appearance out of 20, including skin colors, other fabric colors and patterns such as stripes.
  • TouchAnything (no color augmentation): the color augmentation above is removed; everything else is the same as the paper setting.
  • TouchAnything (+ generative augmentation): on top of the color augmentation, one more training-data augmentation is added, which replaces the glove in the training videos with a bare hand; the architecture and the inputs of the model are unchanged. Concretely, the video generation model Wan2.1 VACE repaints the glove region of the egocentric video: the segmentation mask of the glove region is given by SAM3, a rendered hand mesh serves as the hand-shape prompt, and all pixels outside the region are kept from the original video. The tactile labels are those of the original video.
  • TouchAnything (hand masked): the hand pixels in the input images are set to gray, so the model does not see the appearance of the hand at all.
  • TouchAnything (pose-only input): no image input, hand joints only.

(a) Results on bare hands. 3 participants, wearing no gloves at all, repeatedly press, slide and lift their fingers and palms on a pressure-sensing mat, for a total of 36 clips and about 24,000 frames; the setup is similar to EgoPressure, which the reviewer mentions. The mat directly measures the pressure between the hand and its surface: frames whose reading exceeds the unloaded noise level are contact frames, 52% of all frames. Contact-state accuracy of each model:

ModelContact-state accuracy
Always predict contact (reference)0.520
TouchAnything (pose-only input)0.604
TouchAnything (hand masked)0.801
TouchAnything (paper setting)0.822
TouchAnything (+ generative augmentation)0.851
  • TouchAnything can determine the contact state on bare hands. In the paper setting its accuracy is 0.822, which is 0.302 above always predicting contact. For reference, its Temporal Accuracy on the seen / unseen test sets of EgoTouch is 0.8393 / 0.8271.
  • The contact state is determined mainly from the image. With pose-only input the accuracy is only 0.604, which is 0.218 below the paper setting.
  • The decision does not depend on the appearance of the hand. With the hand masked, the accuracy is still 0.801; with the generative augmentation added, it rises to 0.851.

On bare-hand clips from EgoDex, the predicted contact timing also agrees with the grasp and release motions visible in the video.

(b) Sensitivity to hand appearance. We collected a new appearance-transfer test set with tactile ground truth: 2 participants and 42 episodes (14 per appearance), with a thin nitrile glove in skin color, white or blue worn over the tactile glove. Volumetric IoU of each model:

ModelBlack glove (standard test set, seen)Skin colorWhiteBlue
TouchAnything (pose-only input)0.30520.29810.30070.3012
TouchAnything (no color augmentation)0.43360.32840.35710.3629
TouchAnything (hand masked)0.39050.37120.37480.3761
TouchAnything (paper setting)0.43110.37950.39620.4008
TouchAnything (+ generative augmentation)0.42870.40460.40950.4118
  • TouchAnything remains effective under the three new appearances. In the paper setting its V.IoU is 0.3795–0.4008, which is 88%–93% of the value on the black glove, and clearly above pose-only input (about 0.30) under every appearance.
  • Appearance augmentation is effective. On the skin-colored glove, V.IoU is 0.3284 without color augmentation, 0.3795 in the paper setting, and 0.4046 with the generative augmentation added, which lowers the result on the standard test set by only 0.0024.
  • Most of the accuracy does not depend on the appearance of the glove. With the hand masked, the model does not see the appearance of the hand at all and stays at about 0.37 under the three appearances.
Reviewer_qu6vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a new benchmark for bimanual tactile estimation from egocentric and optionally wrist cameras with a new baseline. The dataset is captured with specialized gloves and covers indoor and outdoor scenes. The benchmarking part includes evaluation on seen or unseen objects during training. The results show the benefits of including wrist cameras and view-dropping during training for the tactile estimation.

Critical strengths

Strength 1: The introduced dataset EgoTouch could be valuable to the community, covering a large number of objects, indoor and outdoor scenes. Strength 2: The proposed TouchAnything baseline is relatively simple and shows promising performance on the EgoTouch benchmark. The paper includes useful experiments on training TouchAnything with view dropout and with wrist camera images.

Critical weaknesses

Weakness 1: Limited analysis of the proposed TouchAnything’s performance. The paper evaluates the proposed TouchAnything model against an adapted PressureVision++ (PV++) model. However, this comparison may not be sufficient to understand the improvements achieved by TouchAnything and establish its advantages over similar methods (PV++ uses a different backbone, does not include hand keypoints as input, and is a single-frame model). The authors acknowledge the differences in the paper. Nevertheless, the proposed TouchAnything architecture remains under-analyzed. There is limited explanation of which architectural components contribute to the substantial performance improvement over PV++. Even when trained only on egocentric videos (Table 2) or without view dropout (Table 3), TouchAnything performs considerably better than PV++. For example, one potential experiment is to evaluate how much hand-pose conditioning contributes to the prediction performance. The view-dropout ablation (Table 3) is performed only on the seen-object benchmark, while unseen-object evaluation is arguably more important for assessing generalization, even within the dataset. Furthermore, there are almost no differences between the Ego+wL / Ego+wR / Ego+wL+wR configurations in Table 2. The improvements from Ego (MV) to Ego+wL / Ego+wR / Ego+wL+wR are also relatively small on unseen objects, and no uncertainty estimates are provided to assess whether these improvements are statistically significant. Finally, the scenario diversity and its influence on the model's performance are under-analyzed in the paper (e.g., the Retail and Office scenarios show substantial differences in performance). It would be beneficial for the reader to have a more detailed analysis and interpretation of these differences.

Weakness 2: Limited details on the dataset and benchmark. In egocentric videos, there is often a limited number of possible grasps or contacts for a specific object. The paper provides insufficient details about the seen object and, more importantly, unseen object evaluation splits. In particular, there is little information about the unseen objects (e.g., their categories) and the diversity of grasps and contact patterns within the benchmark. The similarity of objects between training and testing (as similar objects could have similar contact and pressure distributions), as well as the diversity of possible grasps and contact/pressure patterns for individual objects (e.g., an object may be grasped in only a limited number of ways), can significantly influence the benchmark's difficulty and the interpretation of the results. A more detailed characterization of the evaluation splits, including object categories and grasp and contact diversity, would be very beneficial for understanding the difficulty and significance of the proposed dataset and benchmark and the interpretation of the results. Furthemore, there is no held-out participant evaluation. The participants in the benchmark could potentially be inferred by the model based on their hand pose, which might provide some participant-specific pressure information. As the model already uses an existing hand pose estimation method, it is unclear why the authors did not include a separate experiment where they train the model on, e.g., 8 out of 10 participants and evaluate on the remaining 2, with both seen- and unseen-object settings (including evaluation on unseen objects and unseen participants). If training is relatively short, this would be an interesting experiment to add. This limitation is acknowledged in the paper, but it seems like a relatively simple experiment the authors could try.

Questions for authors

Question 1: Could you please provide more information on the objects (their categories or grouping into category clusters) used for the training / seen objects evaluation / unseen object evaluation? Question 2: Could you analyze how much the performance of the TouchAnything depends on the accuracy of hand keypoints / hand module at all? A simple experiment would be to run a different hand pose estimation model, e.g., HaMeR (CVPR 2024), instead of WiLoR. More details about the architecture ablations (mentioned in major weakness 1) would also be beneficial. Question 3: Can you include an evaluation of performance under different occlusion conditions (e.g., splitting the benchmark into several categories, based on the hand visibility) on seen and unseen objects? Question 4: Can you provide some uncertainty estimation for the results in Table 2? At minimum, please provide it for the Overall (All Scenarios) part for seen and unseen objects. Question 5: If there is enough time to retrain the model during rebuttal, could you perform held-out participant training and evaluation? Question 6: Can you assess how important is the left / right wrist camera information, given the similar results for model settings Ego+wl/Ego+wr/Ego+wl+wr? Examples of some simple inference-time experiments would be: replacing wL and wR images at inference time or having as input 3x egocentric image.

7 comments

Part 1/6: Overview; the gap to PV++ (W1(a))

We thank the reviewer for the detailed and concrete suggestions. We added an experiment or analysis for each of Questions 1–6, and every point raised in W1 and W2 has a corresponding addition. We first summarize the answer to each point and then respond point by point.

Overview

  • W1(a)(b), the gap to PV++: it comes from the output formulation (42% of the gap), pose conditioning and temporal modeling, and the visual part; it is unrelated to multi-view fusion and view dropout. (This part, Part 2)
  • W1(c), view-dropout ablation on unseen objects: the conclusion is the same as on seen objects; the drop when the wrist views are removed shrinks from 22.19% to 2.83%. (Part 2)
  • W1(d) / Q6, is the wrist information used: yes. Replacing the wrist frames with frames from other episodes of the same scenario lowers the seen V.IoU to 0.4050, below the ego-only value of 0.4311. (Part 2)
  • W1(e) / Q4, uncertainty: on unseen objects, the V.IoU gain of all views over Ego (MV) is +0.0109, with a 95% confidence interval of [0.0038, 0.0180], and +0.0109 ± 0.0018 within seeds over 3 random seeds. (Part 3)
  • Q3, occlusion: the gain from the wrist views grows with occlusion, from +0.014 to +0.039 on seen objects. (Part 3)
  • W1(f), scenario differences: they line up with the type of contact. Retail and Office differ mainly in the share of frames whose contact area is near the decision threshold, 37% against 17% on unseen objects. (Part 4)
  • Q2, dependence on pose: 10 mm of joint noise lowers the seen V.IoU by 0.0132; after retraining with HaMeR, the result is within 0.005 of WiLoR. (Part 4)
  • W2 / Q1, objects and splits: we give the distribution of 12 object categories; the contact behavior of unseen objects is further from the training set; a task typically contains 5 contact patterns and 7 grasp types. (Part 5)
  • W2 / Q5, unseen participants: 2 new participants recorded additional test data. On seen objects, the V.IoU is 0.3894 for unseen participants and 0.4508 for seen participants, a drop comparable to that caused by changing objects. (Part 6)

W1(a) — Which architectural components explain the gap to PV++?

The reviewer notes that PV++ and TouchAnything differ in backbone, in whether hand keypoints are an input, and in single-frame versus multi-frame input, so the comparison cannot attribute the gain to individual components. We agree. In the ego-only setting we add a chain of intermediate models that starts from PV++ and replaces its parts step by step (Volumetric IoU):

ModelBackboneOutputTemporalPose inputSeenUnseen
PV++ (adapted, as submitted)SE-ResNeXt-509-class–cropping only0.1750.150
PV++, regression outputSE-ResNeXt-50regression–cropping only0.27100.2340
① Single frame, no poseDINOv2 (frozen)regression––0.32840.2761
② ① + temporalDINOv2regression✓–0.35170.2935
③ ① + poseDINOv2regression–✓0.37860.3312
④ TA-Ego (temporal + pose)DINOv2regression✓✓0.40230.3526
⑤ Pose only, no RGB–regression✓✓0.30520.2817

"PV++, regression output" only replaces the nine-class head with the same regression output and loss as TouchAnything. ① differs from it in the visual part: a frozen DINOv2 that takes the whole egocentric image, with the joint tokens replaced by learnable query tokens. ⑤ removes the image and takes only the sequence of hand joints.

Reading the table (seen / unseen):

  • The largest part of the gap comes from the output formulation: 0.0960 / 0.0840, which is 42% / 41% of the gap between PV++ and TA-Ego.
  • Pose conditioning and temporal modeling come next: 0.0739 / 0.0765 when both are added; added separately, pose (0.0502 / 0.0551) contributes more than temporal modeling (0.0233 / 0.0174).
  • The visual part adds 0.0574 / 0.0421.
  • Pose alone is not enough: ⑤ reaches 0.3052 / 0.2817, and adding the image raises it to 0.4023 / 0.3526.

Why the output formulation matters so much. PV++ predicts nine pressure classes per cell and takes the most probable class. In the EgoTouch training set, 64.1% of the cells fall into the zero class, while pressure above the contact threshold of 0.1 accounts for 16.3% in total and is spread over 4 classes. A single contact class rarely beats the zero class, so cells with weak evidence are predicted as zero; a continuous regression output avoids the hard argmax quantization and gives an intermediate value instead. This also explains why the T.Acc of PV++ is below 50%: 64% of the test frames are in contact, and classifying every frame as no contact gives a T.Acc of 0.36, which matches the 0.364 of PV++. With a regression output, the T.Acc of PV++ rises to 0.7150 / 0.6830.

Part 2/6: View dropout and wrist information (W1(b)–(d), Q6)

W1(b) — TouchAnything is far better than PV++ even with ego only or without view dropout

The decomposition table in Part 1 gives the explanation: every model in it takes only the egocentric view, and the gap between PV++ and TA-Ego comes from the output formulation, the visual part, pose conditioning and temporal modeling. It is unrelated to multi-view fusion and view dropout. The submission already states at L407–410 that this gap cannot be attributed to multi-view fusion; the table gives the quantitative decomposition.

W1(c) — The view-dropout ablation should include unseen objects

We agree that the unseen results matter more for assessing generalization. Table 3 with unseen objects added:

TrainingObjectsEgo only: V.IoUSingle wrist: V.IoUAll views: V.IoURelative drop, all views → ego only
No dropoutSeen0.32330.40730.4441−27.20%
With dropoutSeen0.43110.45730.4575−5.78%
No dropoutUnseen0.28930.34710.3718−22.19%
With dropoutUnseen0.37430.38550.3852−2.83%

The conclusion on seen objects also holds on unseen objects. Without dropout, removing both wrist views lowers the unseen V.IoU by 22.19%; with dropout it drops by only 2.83%. Dropout does not sacrifice all-view performance either: on unseen objects, all-view V.IoU rises from 0.3718 to 0.3852, single-wrist from 0.3471 to 0.3855, and ego-only from 0.2893 to 0.3743.

W1(d) / Q6 — Ego+wL, Ego+wR and Ego+wL+wR are almost identical; how important is the left / right wrist information?

Following the reviewer's suggestion, we ran inference-time input-replacement experiments, all with the same multi-view checkpoint and the same test episodes:

Wrist inputSeen C.IoUSeen V.IoUUnseen C.IoUUnseen V.IoU
None (Ego (MV))0.47920.43110.43960.3743
Correct wL + wR0.50300.45750.44960.3852
wL and wR swapped0.49900.45280.44720.3829
Wrist frames from another episode of the same scenario0.45200.40500.41500.3530
Ego image in all three input slots0.47750.42960.43830.3731

Is the content of the wrist images used? Yes:

  • With the ego image in all three input slots, the result does not exceed ego only (seen V.IoU 0.4296, against 0.4311 for ego only). Going through the multi-view fusion path does not by itself bring a gain.
  • Replacing the wrist frames with frames from another episode of the same scenario gives a result below ego only (0.4050). Wrong wrist images disturb the prediction, which shows that the model reads the content of the wrist images.
  • Swapping the left and right wrists lowers the result slightly (0.4528, against 0.4575 for the correct input), and it stays clearly above ego only.

Why the three configurations are almost the same on the pooled metrics. The wrist cameras are fisheye cameras with a very wide field of view, and a single wrist camera often sees both hands: in the test set, the opposite hand appears in the wrist image in 83% of frames. Splitting the metric by hand, either wrist camera improves both hands. On seen objects, adding the left wrist raises the V.IoU of the left hand by 0.0284 and of the right hand by 0.0245; adding the right wrist raises them by 0.0274 and 0.0249. On unseen objects, the four gains are 0.0121, 0.0105, 0.0116 and 0.0106.

When the second wrist camera helps. It helps when an object, or the hand itself, blocks the line of sight from the wrist camera to the opposite hand. We split the test recordings into two groups by how often the opposite hand is visible in the wrist image (V.IoU; "single wrist" is the mean of Ego+wL and Ego+wR):

ObjectsVisibility of the opposite handShare of recordingsSingle wristBoth wristsΔ
SeenLow (below 50%)6%0.40120.4287+0.0275
SeenHigh94%0.46100.4594−0.0016
SeenAll100%0.45730.4575+0.0002
UnseenLow (below 50%)7%0.32980.3441+0.0143
UnseenHigh93%0.38970.3883−0.0014
UnseenAll100%0.38550.3852−0.0003

The benefit of the second wrist camera is concentrated in the recordings with low visibility of the opposite hand; these make up only 6% of the seen and 7% of the unseen recordings, so two wrists and one wrist are on par overall. For example, on three recordings from the low-visibility group (pressing the buttons of a multimeter, carrying a paper box, picking up a mouse), two wrists are above the better single wrist by .0551, .0516 and .0213.

Part 3/6: Uncertainty and occlusion (W1(e), Q4, Q3)

W1(e) / Q4 — The gains on unseen objects are small, and no uncertainty estimates are given

We add two kinds of uncertainty estimates.

Uncertainty from test-set sampling. We resample the test episodes 10,000 times with the bootstrap and give 95% confidence intervals for the Overall block of Table 2:

SeenT.Acc ↑C.IoU ↑V.IoU ↑MAE ↓
PV++ (adapted)0.364 [0.337, 0.391]0.246 [0.230, 0.262]0.175 [0.161, 0.189]0.060 [0.056, 0.064]
TA-Ego0.8238 [0.8069, 0.8401]0.4493 [0.4287, 0.4701]0.4023 [0.3824, 0.4225]0.0493 [0.0458, 0.0529]
Ego (MV)0.8393 [0.8232, 0.8549]0.4792 [0.4589, 0.4997]0.4311 [0.4112, 0.4513]0.0456 [0.0423, 0.0490]
Ego + wL + wR0.8566 [0.8412, 0.8715]0.5030 [0.4830, 0.5232]0.4575 [0.4378, 0.4775]0.0436 [0.0404, 0.0469]
UnseenT.Acc ↑C.IoU ↑V.IoU ↑MAE ↓
PV++ (adapted)0.359 [0.325, 0.393]0.213 [0.194, 0.232]0.150 [0.134, 0.166]0.065 [0.059, 0.071]
TA-Ego0.7923 [0.7684, 0.8153]0.4078 [0.3826, 0.4333]0.3526 [0.3271, 0.3785]0.0628 [0.0572, 0.0686]
Ego (MV)0.8271 [0.8043, 0.8490]0.4396 [0.4147, 0.4648]0.3743 [0.3489, 0.4001]0.0615 [0.0559, 0.0673]
Ego + wL + wR0.8347 [0.8122, 0.8563]0.4496 [0.4248, 0.4747]0.3852 [0.3598, 0.4109]0.0601 [0.0546, 0.0658]

The four view configurations share one checkpoint and the same episodes, so differences can be estimated in a paired way. Differences of the three wrist configurations to Ego (MV), with 95% confidence intervals:

OverallComparisonΔC.IoUΔV.IoUWilcoxon p (V.IoU)
SeenEgo (MV) → Ego+wL+0.0238 [0.0171, 0.0306]+0.0264 [0.0200, 0.0329]< 0.001
SeenEgo (MV) → Ego+wR+0.0232 [0.0164, 0.0301]+0.0261 [0.0196, 0.0327]< 0.001
SeenEgo (MV) → Ego+wL+wR+0.0238 [0.0172, 0.0305]+0.0264 [0.0201, 0.0328]< 0.001
UnseenEgo (MV) → Ego+wL+0.0103 [0.0023, 0.0183]+0.0113 [0.0041, 0.0185]0.003
UnseenEgo (MV) → Ego+wR+0.0101 [0.0020, 0.0182]+0.0111 [0.0038, 0.0184]0.004
UnseenEgo (MV) → Ego+wL+wR+0.0100 [0.0021, 0.0179]+0.0109 [0.0038, 0.0180]0.004

Training randomness. We retrain TA-Ego and the multi-view model with 3 random seeds each. Within the same seed, the V.IoU difference between all views and Ego (MV) is +0.0263 ± 0.0012 on seen objects and +0.0109 ± 0.0018 on unseen objects.

The two kinds of estimates agree: on both seen and unseen objects, the 95% confidence intervals of the gains of the three wrist configurations over Ego (MV) exclude 0, and after retraining, all views are above Ego (MV) within every seed; the gain exceeds the variation between seeds.

Q3 — Performance under different occlusion conditions

We stratify the test set by how visible the hands are in the egocentric view. Visibility is measured by the error between the ego-view WiLoR joints and the Rokoko glove joints after alignment: the Rokoko joints are measured by the glove and are not affected by occlusion, and the more a hand is occluded in the egocentric view, the larger the error of the WiLoR joints estimated from the image. The model takes a clip of 8 frames (about 0.5 s) at a time, so we stratify at the level of clips: the error of each frame is averaged over a 0.5 s window around it, rather than taken from the single frame.

Results for three equal bins by error (Volumetric IoU). The metric of each bin is computed as in Table 2: within each episode we pool over the frames that belong to the bin, and then average over episodes:

Visibility binShare of framesSeen: Ego (MV)Seen: + both wristsSeen: ΔUnseen: Ego (MV)Unseen: + both wristsUnseen: Δ
Least occluded33%0.4870.501+0.0140.4280.433+0.005
Medium33%0.4360.462+0.0260.3780.389+0.011
Most occluded33%0.3710.410+0.0390.3170.334+0.017

The V.IoU of both configurations decreases as occlusion increases: for Ego (MV) from 0.487 to 0.371 on seen objects and from 0.428 to 0.317 on unseen objects. The gain from the wrist views grows with occlusion: from +0.014 to +0.039 on seen objects and from +0.005 to +0.017 on unseen objects, so the gain in the most occluded bin is about 3 times that in the least occluded bin. The wrist views help most when the hands remain hard to see in the egocentric view.

Reviewer_jJEhOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper is well-motivated and addresses the important problem of vision (video)-to-tactile transformation. Egocentric videos are relatively easy to collect in the real world, but paired tactile sensing data are significantly more difficult to collect manually. Being able to infer tactile maps from videos can help scale up tactile annotation using existing videos and provide meaningful data for downstream robotics tasks. The dataset is novel in its combination of tactile measurements and wrist-mounted observations across diverse manipulation settings. EgoTouch offers novel features compared with related datasets, such as OpenTouch’s single-hand collection and ActionSense’s kitchen-focused setup. I appreciate the authors’ substantial efforts in collecting these modalities across indoor and outdoor scenarios. The experimental setup is comprehensive, demonstrating the validity and soundness of the authors’ main conclusions about how multi-view data support tactile inference.

Critical strengths
  1. The paper is well-motivated and addresses the important problem of vision (video)-to-tactile transformation. Egocentric videos are relatively easy to collect in the real world, but paired tactile sensing data are significantly more difficult to collect manually. Being able to infer tactile maps from videos can help scale up tactile annotation using existing videos and provide meaningful data for downstream robotics tasks.
  2. The dataset is novel in its combination of tactile measurements and wrist-mounted observations across diverse manipulation settings. EgoTouch offers novel features compared with related datasets, such as OpenTouch’s single-hand collection and ActionSense’s kitchen-focused setup. I appreciate the authors’ substantial efforts in collecting these modalities across indoor and outdoor scenarios.
  3. The experimental setup is comprehensive, demonstrating the validity and soundness of the authors’ main conclusions about how multi-view data support tactile inference.
Critical weaknesses
  1. I think bimanual data can provide more insights into tasks involving interactions between both hands. For example, when opening a bottle, one hand needs to hold the bottle steady while the other twists the cap. The paper currently does not discuss the relationship between the two hands. I wonder whether the dataset can be used to study bimanual coordination or whether the results simply reflect the average performance across both hands.
  2. The paper improves the accuracy of predicting tactile maps, but more experiments may be needed to demonstrate the benefits for downstream tasks. The paper currently shows that the predicted pressure maps are more similar to the ground truth, which is nice. But I am also curious whether this can actually help improve downstream task success rates. Real-robot experiments are not necessary, but some other downstream tasks could be selected for evaluation.
Questions for authors
  1. Can the authors provide further analysis of bimanual coordination, beyond reporting prediction performance averaged across both hands?
  2. Is there evidence that the improved tactile predictions benefit downstream tasks, beyond achieving higher reconstruction accuracy?
  3. The results suggest that adding one wrist view provides most of the improvement, while adding the second wrist view offers little additional benefit overall. Are there particular tasks or occlusion conditions in which using both wrist views is more beneficial than using only one?
5 comments
AuthorsOpenReview ↗

Part 1/3: Bimanual coordination (W1 / Q1)

Part 1/3: Bimanual coordination (W1 / Q1)

We thank the reviewer for the positive assessment and the three specific questions. We first summarize the three answers and then respond to each in detail.

Overview

  • W1 / Q1, bimanual coordination: yes, it can be studied, and the results are not simply the average over both hands. When twisting a bottle cap, the pressure of the left hand on the bottle rises by 23% while the right hand applies force to the cap; the model's predictions preserve this relation. (This part)
  • W2 / Q2, downstream tasks: yes, they benefit. Training a robot manipulation model on EgoDex videos labeled by TouchAnything raises the average success rate from 80.5% to 88.3% on UniVTAC and from 29.3% to 38.7% on three Bench2Dex tasks; labels from PV++ bring no improvement. (Part 2)
  • Q3, two wrist views: yes, such conditions exist. When the line of sight from the wrist camera to the opposite hand is blocked, the V.IoU with both wrists is 0.0275 higher than with a single wrist; these recordings make up only 6% of the total. (Part 3)

W1 / Q1 — Can the dataset be used to study bimanual coordination? Do the results simply reflect the average over both hands?

Yes, it can. EgoTouch records full-hand touch for both hands in sync, so the coordination between the two hands can be quantified directly from the tactile signals. The reviewer is right that every metric in the submission pools the two hands. We add analyses at the dataset and model levels.

(1) Dataset level. The statistics cover all 208 tasks and 1,891 episodes. A hand is "in contact" in a frame under the same rule as the paper's Temporal Accuracy, applied per hand.

  • Simultaneous contact is common. Both hands are in contact in 31.1% of all frames and in 40.8% of contact frames. Grouped by measured contact structure, 168 of the 208 tasks (82.8% of all frames) are bimanual.
  • Roles are stable. The reviewer's bottle-opening example corresponds to the "twist bottle cap" task (20 episodes): the left hand holds the bottle, with contacts lasting 5.2 s on average, and the right hand twists the cap, 1.5 s on average. There are 37 such tasks with one sustained and one intermittent hand; in 87.5% of their episodes, the hand with the longer contacts is the same one.
  • The load on one hand varies with the action of the other. When one hand twists the cap, the holding hand needs to grip harder to resist the torque. We compare the total pressure of the holding hand when the acting hand is applying force and when it is not. Each ratio is computed within one recording, so the normalization constant cancels; ratios are medians over episodes, and the denominator in the last column is the number of episodes in which the ratio is defined, i.e. in which both states occur:
TaskEpisodesHolding-hand pressure ratio (acting hand applying force / not applying force)Episodes with ratio above 1
Twist bottle cap201.2317 / 20
Open battery box and take out batteries101.2110 / 10
Plug and unplug a USB cable91.267 / 8
Plug and unplug a Type-C connector101.557 / 7
Open a spray can51.585 / 5
Split disposable chopsticks52.005 / 5

The same analysis also identifies tasks without this coupling, such as picking up instant noodles (1.00) or earphones (1.01): the hands are often in contact at the same time, but the pressure of the holding hand does not change with the other hand. Synchronized touch on both hands therefore separates tasks in which the pressure of one hand follows the action of the other from tasks in which it does not, which cannot be done from binary contact alone. The direction also depends on the type of cooperation: the ratio is 1.56 in synchronous tasks and 0.79 in alternating tasks.

(2) Model level. The analysis above uses only the tactile signals of the two hands, so it can be repeated unchanged on the model's predictions. For every episode of a bimanual task in the test set, we compute the same ratio from the ground truth and from the prediction, and group episodes by the ground-truth ratio (group medians):

Group by ground truthEpisodesGround-truth ratioPredicted ratio
Holding-hand pressure rises (ratio > 1.1)571.341.22
Essentially unchanged (ratio 0.9–1.1)371.021.03
Holding-hand pressure falls (ratio < 0.9)410.680.81

The model's predictions preserve this relationship; the Pearson correlation between predicted and ground-truth ratios across episodes is 0.82. The model is never explicitly supervised on the relationship between the two hands; it learns it from its inputs, the video and the hand pose.

Summary. EgoTouch can be used to study bimanual coordination, and the model's results are more than an average over the two hands: its predictions preserve how the load on one hand varies with the action of the other.

Part 2/3: Downstream tasks (W2 / Q2)

W2 / Q2 — Does higher reconstruction accuracy help downstream tasks?

Yes, it helps. We agree that reconstruction metrics alone do not show whether the predicted touch is useful, so we tested this directly: we use TouchAnything to add tactile labels to human-hand video recorded without tactile sensors, and train a robot manipulation model with these labels. The average success rate rises from 80.5% to 88.3% on UniVTAC and from 29.3% to 38.7% on three Bench2Dex tasks; labels from PV++, which has lower reconstruction accuracy, bring no improvement.

Setup.

  • We take a subset of about 400 hours of EgoDex and label it with TouchAnything. EgoDex has no tactile sensors and only an egocentric view, so we use the ego-only inference configuration of the multi-view model, i.e., Ego (MV) in Table 2.
  • We add a tactile branch to OpenWAM, which jointly predicts future video and actions; the branch predicts future tactile maps and is denoised together with video and actions. Human and robot touch share a vocabulary of 67 regions, and the 21×21 hand-shaped grid output by TouchAnything is split by finger and palm and mapped to its human-hand regions.
  • Starting from the OpenWAM-alpha weights, we run one intermediate training stage on the EgoDex subset (400 hours) + AgiBot (240 hours) + SharpaDex (300 hours).
  • We compare three settings, all of which go through the same intermediate stage. OpenWAM has no tactile branch and uses no tactile data. The other two both have the tactile branch and both use the touch that comes with SharpaDex; they differ only in whether the EgoDex tactile labels come from the adapted PV++ or from TouchAnything.
  • The three models are fine-tuned with the same recipe on the training data of two simulation benchmarks, UniVTAC and Bench2Dex, and then evaluated in closed loop: 100 trials per task on the 8 UniVTAC tasks and 50 trials per task on the 3 Bench2Dex tasks.

Success rates on UniVTAC (100 trials per task):

MethodInsert HDMIInsert HoleInsert TubeGrasp ClassifyLift CanLift BottlePull-out KeyPut Bottle in ShelfAverage
OpenWAM28%79%88%97%73%97%90%92%80.5%
OpenWAM + PV++ labels31%78%84%92%76%97%88%88%79.3%
OpenWAM + TouchAnything labels42%94%89%97%92%98%96%98%88.3%

Success rates on the three Bench2Dex tasks (successes / trials in parentheses):

MethodCan placement (26)Tool tray (32)Puzzle (73)Mean of three tasksTotal over three tasks
OpenWAM50% (25/50)22% (11/50)16% (8/50)29.3%44/150
OpenWAM + PV++ labels48% (24/50)24% (12/50)12% (6/50)28.0%42/150
OpenWAM + TouchAnything labels52% (26/50)34% (17/50)30% (15/50)38.7%58/150
  • TouchAnything labels improve success. The average success rate on UniVTAC rises from 80.5% to 88.3%, with 7 of the 8 tasks improving. The largest gains are on Lift Can (73% → 92%), Insert Hole (79% → 94%) and Insert HDMI (28% → 42%). The three-task mean on Bench2Dex rises from 29.3% to 38.7%, with the gain coming from Tool tray (22% → 34%) and Puzzle (16% → 30%).
  • PV++ labels do not. The average success rate on UniVTAC is 79.3%, 1.2 points below OpenWAM: of the 8 tasks, 5 drop, 2 improve and 1 is unchanged. The three-task mean on Bench2Dex is 28.0%, also slightly lower.
  • The difference comes from label quality. The last two settings have identical model structure, training data and recipe; only the source of the tactile labels on EgoDex differs. They are 9.0 points apart on UniVTAC (88.3% vs. 79.3%) and 10.7 points apart on Bench2Dex (38.7% vs. 28.0%). This agrees with the reconstruction metrics on EgoTouch: with only the egocentric view as input, the Volumetric IoU of TouchAnything (Ego (MV)) is 0.4311, and that of the adapted PV++ is 0.175.
  • The same tactile branch with PV++ labels does not reproduce this gain. The model trained with PV++ labels has the same tactile branch and also uses the SharpaDex touch, yet it is essentially on par with OpenWAM without a tactile branch.

Part 3/3: Two wrist views (Q3)

Q3 — Are there tasks or occlusion conditions in which two wrist views help more than one?

The reviewer's observation matches the numbers in the paper: on the pooled metrics, Ego+wL, Ego+wR and all views are almost identical (Volumetric IoU on seen objects: 0.4575 / 0.4572 / 0.4575). We first explain why, and then show under which conditions the second wrist camera helps.

Why a single wrist camera already captures most of the gain. There are two reasons:

  • In part of the tasks, the contact state of the hands is already clear from the egocentric view, and a wrist view has little to add.
  • Where the egocentric view is unclear, the wrist cameras are fisheye cameras with a very wide field of view, and a single wrist camera often sees both hands. In the test set, the opposite hand appears in the wrist image in 83% of frames.

Splitting the metric by hand gives a consistent picture. The table below shows the gain in V.IoU for each hand when one wrist view is added to the egocentric view:

Added viewLeft hand (seen)Right hand (seen)Left hand (unseen)Right hand (unseen)
Left wrist+0.0284+0.0245+0.0121+0.0105
Right wrist+0.0274+0.0249+0.0116+0.0106

On both seen and unseen objects, either wrist camera improves both hands, so the second wrist camera has little left to add.

When the second wrist camera helps. It helps when an object, or the hand itself, blocks the line of sight from the wrist camera to the opposite hand. For each test recording we measure how often the opposite hand is visible in the wrist image, and split the seen test recordings into two groups by this visibility (V.IoU; "single wrist" is the mean of Ego+wL and Ego+wR):

Visibility of the opposite handShare of recordingsSingle wristBoth wristsΔ
Low (below 50%)6%0.40120.4287+0.0275
High94%0.46100.4594−0.0016
All100%0.45730.4575+0.0002

The benefit of the second wrist camera is concentrated in the recordings with low visibility of the opposite hand. In the recordings with high visibility, two wrists and one wrist are essentially on par: the two wrist cameras then see largely the same content, and the second wrist camera brings no new information. The low-visibility recordings make up only 6% of the total, so two wrists and one wrist are on par overall.

Examples. Coverage depends on the position of the hands, the orientation of the cameras, and occlusion by objects or by the hands themselves. Three examples from the low-visibility group:

  • Pressing the buttons of a multimeter: both hands work around the same device, and the two wrist views show different local details of the fingers on the device surface.
  • Carrying a paper box: the two hands hold different sides of the box; each wrist camera observes the nearby grasp region, while the box limits the view to the other side.
  • Picking up a mouse: the right hand operates the mouse, and the left-wrist image mainly faces the idle hand and the surrounding workspace; adding the right-wrist view adds an observation close to the operating hand.

We inspected the per-recording comparisons and selected three recordings with a positive gain from the low-visibility group, one per task. The table gives the V.IoU of each complete recording under the four input configurations; every wrist configuration includes Ego.

Task instanceEgo+ left wrist+ right wrist+ both wrists
Pressing the buttons of a multimeter.1636.2927.2849.3478
Carrying a paper box.4175.5121.5088.5637
Picking up a mouse.4480.5234.5266.5479

On these three recordings, two wrists are above both single-wrist configurations. Relative to the better single wrist, the absolute gains are +.0551, +.0516 and +.0213.

Summary. In the current dataset, one fisheye wrist camera covers both hands in most tasks, and recordings with low visibility of the opposite hand make up only about 6%, so two wrists and one wrist are on par overall. The second wrist camera helps in tasks where the two hands are on opposite sides of an object and occlude each other. We will add more tasks of this kind in future versions of the dataset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.