acceptodds
Under review as a conference paper at ICLR 2027

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

16 views

Abstract

Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, but visual observations alone do not directly measure contact and force during physical interaction. Recent visual-tactile datasets have begun to fill this gap, but the available volume of synchronized tactile data remains far smaller than that of human video, and the largest existing resources often combine data collected with different sensors or annotation procedures. Datasets containing hundreds of hours of human vision and touch captured through one consistent sensing pipeline remain scarce. We introduce TouchScale, a 500-hour dataset of contact-rich human interaction collected through a consistent sensing pipeline, comprising approximately 2K predefined task descriptions spanning everyday activities and structured manipulation. Each recording contains time-aligned egocentric RGB-D, bilateral wrist RGB and tactile measurements. At this scale, we ask a simple question: what does scale unlock for visual-tactile learning? We investigate this via cross-dataset generalization, transferable representation learning, and robotic manipulation. Using the same vision-to-touch architecture, training on TouchScale instead of EgoTouch improves zero-shot cIoU on EgoTactile by 35.1%. Pretraining on TouchScale improves downstream action recognition accuracy by 49.3% compared with other visual-tactile pretraining datasets. Using TouchScale for visual-tactile pretraining improves contact-rich manipulation success rate from 40% to 70%. Together, our scalable collection paradigm, large-scale dataset, and study establish a foundation for human visual-tactile learning and its transfer to robotic manipulation.

Primary area:
datasets and benchmarks
Keywords:
Visual Tactile Robot Learning, Egocentric Datasets

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_igQXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TouchScale, a 500-hour human visual–tactile dataset containing approximately 87,000 episodes, 1,964 task descriptions, and more than 1,500 objects. Recordings combine head-mounted RGB-D video, two wrist-mounted RGB views, and dense bimanual tactile measurements. The authors evaluate the dataset through cross-dataset tactile prediction, tactile-supervised visual pretraining for action recognition, and mid-training for a robot policy. The reported results include improved zero-shot tactile prediction and an increase in average success rate from 22.5% to 57.5% across four real-robot tasks.

Critical strengths
  • Substantial synchronized data collection. The scale and dense bimanual tactile coverage make the dataset a potentially valuable resource for studying visual–tactile learning.

  • Evaluation beyond a single benchmark. The paper examines tactile prediction, visual representation transfer, and robot policy learning. The real-robot gains are encouraging and make the dataset’s practical relevance more concrete.

Critical weaknesses
  • The dataset’s distinction from closely related work is insufficiently analyzed. Table 1 compares broad statistics, but datasets such as EgoTouch use a notably similar head-camera, dual-wrist-camera, and bimanual-glove setup. HumanTouch is also particularly relevant; TouchSight reports using 500 hours of its recordings. A more direct comparison of capture hardware, actual data diversity, quality, and downstream utility is needed to establish what TouchScale uniquely contributes beyond scale and sensor density.

  • Dataset diversity and quality are not characterized sufficiently to substantiate the headline statistics. The paper reports nine collection settings and over 800 scene configurations, yet many displayed examples appear to use similar tabletops and backgrounds. It does not define what constitutes a distinct scene configuration or report how much the physical backgrounds and work surfaces vary. Likewise, the quality-control procedure is described, but the paper does not report the raw-to-retained data ratio or the proportions assigned PASS, REVIEW, and REJECT. These omissions make it difficult to assess how diverse and usable the 500 hours are in practice.

  • The policy experiments leave the source and scope of the gains unclear. They compare TouchScale mid-training only against skipping that stage. There is no matched mid-training comparison with another visual–tactile dataset, and no evaluation of whether TouchScale improves generalization to new objects, backgrounds, or settings. All policy results come from one robot platform and four tasks. A two-arm task, even using grippers, would additionally help test whether this bimanual egocentric dataset benefits more general manipulation. Such an experiment would probe the value of the visual data; a separate control would still be needed to isolate the contribution of tactile supervision.

Questions for authors
  • How is a distinct scene configuration defined? How many distinct physical tabletops, backgrounds, and collection locations are represented, and are any held out during evaluation?
  • How many episodes or hours were recorded before quality control? What fractions received PASS, REVIEW, and REJECT, and what fraction was retained in the released dataset?
  • Does TouchScale mid-training improve policy success on held-out objects or visual environments? How does it compare with mid-training on another dataset under a matched training budget?
4 comments
AuthorsOpenReview ↗

Response to igQX

Thank you for the constructive feedback. Your review raises three concerns and three questions:

  • W1: distinction from EgoTouch and HumanTouch
  • W2/Q1/Q2: scene diversity and quality control
  • W3/Q3: source and scope of the policy gains

We hope the responses below address them.

W1. Distinction from EgoTouch and HumanTouch

We agree that EgoTouch and HumanTouch are the closest datasets, and we respectfully note that TouchScale differs from both in tactile density, egocentric depth, and task coverage. HumanTouch reports more recorded hours (1,000+, of which TouchSight trains on 500), so we will remove the claim that TouchScale offers the most recorded hours and add HumanTouch to Table 1.

EgoTouchHumanTouchTouchScale
Hours20~100 initial release (1,000+ reported)500
Tasks20810 initial release (100+ reported)1,964 descriptions
Objects1,000+not reported1,500+
Head cameraRGBRGBRGB-D
Taxels per hand256 (16×16)360880

Task counts follow each dataset's own definition. For downstream utility, TouchScale outperforms EgoTouch at matched hours in zero-shot tactile prediction (0.181 vs. 0.134 cIoU, Table 2) and as robot mid-training data under a matched budget (57.5% vs. 25.0%, W3/Q3). Beyond scale and tactile density, we study how scaling data within TouchScale affects zero-shot tactile prediction and downstream robot manipulation, with fixed sensing hardware and training protocols. As data volume increases, zero-shot cIoU rises from 0.311 to 0.383 and robot success from 30.0% to 57.5% (our response to R2). Thus, TouchScale combines the densest tactile sensing in this comparison with egocentric depth and broad task coverage.

W2/Q1/Q2. Scene diversity and quality control

Thanks for raising this question. TouchScale contains 800+ distinct scene configurations, defined by the object instances and their initial spatial arrangement, including positions, orientations, and relative layout. The same task can have multiple configurations, so these represent different object arrangements, not separate physical environments. Data were collected at one location, with five tabletop types and four background styles. The unseen split in Table 2 holds out entire task descriptions with all of their scene configurations, so it measures transfer to unseen tasks and arrangements on the same tabletops and backgrounds. Transfer to new environments is measured on EgoTactile (Table 2) and at our separate robot site (W3/Q3). Before quality control, we collected approximately 531 hours. Automated screening classified 63.7% PASS, 31.7% REVIEW, and 4.5% REJECT at the episode level, with REVIEW cases further inspected by human reviewers, and we retained 500 hours (94.2%). In conclusion, TouchScale varies tasks, objects, and arrangements at one site and retains 94.2% of the recorded hours. We will include more details into the revision.

W3/Q3. Source and scope of the policy gains

We expanded the policy evaluation as suggested. For the matched mid-training comparison, we compare EgoTouch and TouchScale mid-training under the same model initialization, training budget, 50 robot post-training demonstrations per task, and evaluation protocol. For EgoTouch mid-training, we map its tactile data to the same tactile representation format used in robot post-training.

Mid-trainingSortingCap RemovalTube TransferWipeAvg.
None10%40%30%10%22.5%
EgoTouch20%45%20%15%25.0%
TouchScale60%70%60%40%57.5%

TouchScale outperforms EgoTouch on all four tasks, by 32.5 points on average (46/80 vs. 20/80, two-sided Fisher exact test, p<0.001p < 0.001). For new objects and environments, our robot site, objects, and scenes are absent from the human data. The same objects appear in the 50 robot demonstrations, so this setting measures transfer from human data to new objects and scenes. For two-arm tasks(Appendix E, Fig. 13), using two xArm6 arms with two BrainCo dexterous hands, we added Bimanual Liquid Transfer and USB Insertion, and TouchScale mid-training improves success from 10% to 30% and from 0% to 30%, respectively, over 20 trials per task (12/40 vs. 2/40 pooled, two-sided Fisher exact test, p=0.006p = 0.006). For a control that isolates tactile supervision, we randomly re-pair visual and tactile observations during mid-training while keeping the training objective, underlying data, training budget, and robot post-training procedure unchanged. Average robot success drops from 57.5% with aligned pairs to 28.75% with randomly re-paired pairs (46/80 vs. 23/80, two-sided Fisher exact test, p<0.001p < 0.001), close to the 22.5% without mid-training (p=0.47p = 0.47). Therefore, TouchScale beats EgoTouch under a matched budget, extends to dual-arm tasks, and helps only when vision and touch stay aligned.

Thank you for the detailed rebuttal. I’ve read it carefully and appreciate the experiments you added in response to my review. The comparison with EgoTouch under a matched training budget helps clarify the downstream gains, and the bimanual experiments and vision–touch alignment control strengthen the paper.

My main remaining concern is the environmental diversity of the TouchScale dataset. The evaluation at a separate robot site provides evidence of transfer to a new setting. For the dataset itself, the collection at a single site with four background styles still limits the range of environments represented.

Taking the new results into account, I think a score of 3 remains the best fit for the paper’s overall novelty and the scope of its contributions on this year’s four-level scale. I will keep my original rating.

Thank you for your thoughtful response and for recognizing the value of our additional experiments. We sincerely appreciate your constructive feedback and positive assessment of our work.

Thank you again for your follow-up and for recognizing the value of our additional experiments.

Following your feedback, we have further updated the revised PDF. Sec. 3.2 now explicitly clarifies the single-site collection setup, including five tabletop types, four background styles, and the definition of the 800+ distinct scene configurations. We have also acknowledged environmental diversity as a limitation in Sec. 5. Our next collection round will include more sites and backgrounds.

Therefore, these revisions clarify the environmental diversity of the current dataset and outline our plans to broaden its coverage in future collections. We sincerely appreciate your constructive feedback and support for our work.

Reviewer_N98yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TouchScale, a large-scale human visual-tactile dataset with 500 hours of synchronized vision and touch data. The dataset is collected with a unified wearable system, including head-mounted RGB-D cameras, wrist RGB cameras, and dense tactile sensors on both hands. It covers around 2K task descriptions, more than 1.5K objects, and diverse daily manipulation scenarios. The authors study how large-scale visual-tactile data benefits three areas: tactile prediction, visual representation learning, and robotic manipulation. Experiments show that TouchScale improves zero-shot tactile prediction on unseen sensors, action recognition performance, and real-world robot manipulation success.

Critical strengths
  1. Compared with previous visual-tactile datasets, TouchScale provides much larger data with a unified sensing setup and synchronization pipeline. This enables a better study of how data scale affects visual-tactile learning.

  2. The paper provides comprehensive evaluations from perception to robot control.The authors evaluate tactile prediction, visual representation learning, and real robot manipulation, which makes the contribution more complete than a simple dataset release.

  3. The authors study the effect of increasing training data while keeping the sensor and collection protocol fixed. This provides useful evidence for understanding the role of large-scale visual-tactile data.

Critical weaknesses
  1. The effect of data scale is not fully separated from other advantages of the dataset. Compared with previous datasets, TouchScale is not only larger but also has denser tactile sensing, different task distributions, and a more consistent collection pipeline. Therefore, it is difficult to determine how much improvement comes from scale itself rather than other dataset properties.

  2. The paper shows that TouchScale mid-training improves robot performance, but it is still unclear what tactile information the model learns and why future tactile prediction helps action prediction. The current analysis of this relationship is limited.

  3. The generalization ability of the dataset is not fully validated. Although TouchScale contains many tasks and objects, the robot experiments are limited to one robot platform and four manipulation tasks. It is still unclear whether the benefits can transfer to different robots, tactile sensors, and broader manipulation scenarios.

Questions for authors
  1. In the experiments showing the benefit of TouchScale, can the authors provide more controlled comparisons, such as keeping the number of tasks, objects, or sensor density fixed while only changing the number of training hours? This would better measure the pure effect of data scale.

  2. In the robot experiments, does the improvement mainly come from better visual representations, better tactile prediction, or the combination of both? Can the authors provide ablation studies on different inputs or training objectives?

  3. Since the dataset is collected from around 20 participants, does the dataset contain possible participant-specific bias? For example, can the model learn general human manipulation patterns rather than specific behaviors from some participants?

3 comments
AuthorsOpenReview ↗

Response to N98y

Thank you for the constructive feedback. Your review raises three concerns and three questions:

  • W1/Q1: separating data scale from other dataset properties
  • W2/Q2: what tactile information helps action prediction
  • W3: generalization to other robots, sensors, and scenarios
  • Q3: participant-specific bias

We hope the responses below address them.

W1/Q1. Separating data scale from other dataset properties

We clarify the comparison. In Fig. 4 and Fig. 6 (left), all training subsets cover the same tasks and objects, with fixed sensing hardware, tactile density, and training protocols, and only the number of recordings changes. Under these controls, zero-shot cIoU on EgoTactile rises from 0.311 to 0.383 as training data increases from 10% to 100%, and average robot success rises from 30.0% to 57.5% as mid-training data increases from 20% to 100% (24/80 vs. 46/80, two-sided Fisher exact test, p<0.001p < 0.001). We separately compare TouchScale and EgoTouch using approximately 16 hours of training data each, where TouchScale achieves 0.181 vs. 0.134 cIoU on EgoTactile, a 35.1% relative improvement at matched data volume (Table 2). Each dataset retains its native tactile layout during training, with predictions mapped to 12 common anatomical hand regions for evaluation (Appendix A.1). In conclusion, the scaling curves isolate data volume within one dataset, and the matched-hour comparison isolates dataset differences at equal volume.

W2/Q2. What tactile information helps action prediction

Thanks for raising this question. Our results show that the robot improvement mainly comes from learning future tactile changes through visual–tactile alignment. During TouchScale mid-training, we freeze the visual-language-action backbone and train only the tactile projection layer, tactile predictor, and reconstruction head for future-tactile prediction (Appendix E.2). On unseen robot observations, future-tactile Top-10 accuracy improves from 0.18% to 3.31%. We also observe a positive correlation between tactile and action prediction errors (r=0.119r=0.119, p=0.020p=0.020), compared with r=0.003r=0.003 without mid-training. To further examine visual–tactile alignment, we randomly re-pair visual and tactile observations during mid-training while keeping all other settings unchanged. Robot success drops from 57.5% to 28.75% (46/80 vs. 23/80, Fisher's exact test, p<0.001p < 0.001), close to the 22.5% without mid-training (p=0.47p = 0.47). Therefore, the gain comes from predicting touch from vision, and it largely disappears once the visual–tactile alignment is removed.

W3. Generalization to other robots, sensors, and scenarios

We expanded the robot evaluation as suggested. Using two xArm6 arms with two BrainCo dexterous hands, we added Bimanual Liquid Transfer and USB Insertion (Appendix E, Fig. 13), and TouchScale mid-training improves success from 10% to 30% and from 0% to 30%, respectively, over 20 trials per task (12/40 vs. 2/40 pooled, two-sided Fisher exact test, p=0.006p = 0.006). Both setups use the same xArm6 and BrainCo robot models, while the new tasks require bimanual coordination and precise insertion. For tactile sensors, Sec. 4.1 demonstrates zero-shot tactile prediction across different sensor setups, and robot-policy transfer to other tactile sensors remains a direction for future work. Therefore, the benefit of TouchScale extends from four single-arm tasks to two dual-arm tasks.

Q3. Participant-specific bias

Thanks for this question. The learned visual-tactile relations transfer beyond our participants: the model trained on 16 hours of TouchScale transfers zero-shot to the independently collected EgoTactile dataset and outperforms the model trained on 16 hours of EgoTouch (0.181 vs. 0.134 cIoU, Table 2). Within TouchScale, approximately 20 participants perform different executions of the same tasks with varied object instances and initial configurations (Sec. 3.2), and they exhibit different grasp choices, movement timing, hand coordination, and contact sequences. These results support transfer beyond TouchScale’s participants, sensors, and scenes.

Thank you again for the detailed review. Following W1/Q1, we have clarified the controlled within-TouchScale scaling analysis (Fig. 4) and the matched-hour comparison with EgoTouch (Table 2) in our response. The revised PDF now includes EgoTouch mid-training under a matched budget in Table 4 and the randomly re-paired control from W2/Q2 in Appendix E.2. We would be glad to clarify any remaining points before the discussion closes on November 18 (AoE).

Reviewer_N98yOpenReview ↗

Response to Rebuttal

Thanks for the response. My concerns have been addressed, so I'll keep my positive score.

Reviewer_u8RAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TouchScale introduces a large‑scale (500 hours) real‑world human visuo‑tactile dataset containing egocentric RGB‑D, wrist‑mounted RGB, bimanual full‑hand tactile measurements. It covers ~2k task descriptions and more than 1.5k+ distinct objects. It also shows the scaling law.

Critical strengths
  1. Very large-scale dataset with synchronized camera and tactile measurements.
  2. With the large dataset, the paper explicitly studys scaling laws for vision‑tactile learning.
  3. Downstream validation with real‑robot transfer.
Critical weaknesses
  1. No semantic labels, QA pairs for assessing tool‑use cognition/action segmentation (training VLMs).
  2. 3D hand‑pose is obtained by a third‑party pose estimator, its quality is questionable.
  3. The shown manipulations in the paper and supplementary seem not to be complex and dexterous.
Questions for authors

see the weakness

3 comments
AuthorsOpenReview ↗

Response to u8RA

Thank you for the constructive feedback. Your review raises three concerns:

  • W1: semantic labels and QA pairs
  • W2: quality of the estimated 3D hand poses
  • W3: complexity of the shown manipulations

We hope the responses below address them.

W1. Semantic labels and QA pairs

We added semantic annotations as suggested. Using Gemini 3.7 Flash, we annotated 10 hours (due to the limited response window) of TouchScale with temporal action segments, step-level captions, and video QA pairs covering tool use, action sequences, and temporal reasoning. These complement the 1,964 task descriptions in TouchScale (Sec. 3.2) and provide finer-grained supervision for video understanding. We will release the annotations and the annotation pipeline. In short, a 10-hour subset of TouchScale now carries action segments and QA pairs on tool use, together with the pipeline to extend them. We will release the annotation protocol.

W2. Quality of the estimated 3D hand poses

Precise 3D hand pose typically requires sensor fusion such as multi-view motion capture with inertial or electromagnetic hand tracking, which our capture setup does not include. In TouchScale, we focus on synchronized vision and touch, and hand poses are WiLoR estimates from video that serve only as inputs to the TouchAnything baseline, with the same estimator used for EgoTouch (Sec. 4.1). However, accurate hand-pose capture is our next focus, and we will discuss it in the limitations section.

W3. Complexity of the shown manipulations

We expanded the robot evaluation to dual-arm dexterous manipulation. Using two xArm6 arms with two BrainCo dexterous hands, we added Bimanual Liquid Transfer and USB Insertion(Appendix E, Fig. 13), and TouchScale mid-training improves success from 10% to 30% and from 0% to 30%, respectively, over 20 trials per task (12/40 vs. 2/40 pooled, two-sided Fisher exact test, p=0.006p = 0.006). On the human side, TouchScale covers fine-grained and bimanual tasks such as pipetting, screw tightening, and cloth wringing (Sec. 3.2, Fig. 3). Therefore, TouchScale mid-training now also improves dual-arm dexterous tasks.

Thanks for the authors' response. Most of my concerns are resolved. I still have one remaining concern regarding environmental diversity and manipulation diversity in the TouchScale dataset. I am keeping my original rating.

Thank you for your follow-up. We appreciate that most of your concerns have been resolved.

Following your feedback, we have further updated the revised PDF. For environmental diversity, Sec. 3.2 now clarifies the single-site collection setup, including five tabletop types, four background styles, and the definition of scene configurations. We also explicitly acknowledge the limitation of collecting at a single site in Sec. 5. Our next collection round will include more sites and backgrounds.

For manipulation diversity, we have added quantitative results for two dual-arm dexterous tasks, Bimanual Liquid Transfer and USB Insertion, in Appendix E.1, complementing the task and interaction diversity presented in Sec. 3.2 and Fig. 3.

Together, these revisions clarify the scope and limitations of TouchScale's data collection and extend our evaluation to more challenging bimanual manipulation. We sincerely appreciate your constructive feedback and support for our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank Reviewers u8RA (R1), N98y (R2), and igQX (R3) for their time and feedback. We appreciate the strengths you recognized:

  1. Scale with synchronized sensing: "Very large-scale dataset with synchronized camera and tactile measurements" (R1).
  2. A controlled scaling study: "useful evidence for understanding the role of large-scale visual-tactile data" (R2).
  3. Transfer to real robots: "The real-robot gains are encouraging" (R3).

We added the following results, detailed in the individual responses:

  • Visual-tactile alignment control (R2, R3): randomly re-pairing visual and tactile observations during mid-training reduces average robot success from 57.5% to 28.75% (p<0.001p < 0.001).
  • Matched-budget mid-training on EgoTouch (R3): average robot success of 25.0% with EgoTouch vs. 57.5% with TouchScale (p<0.001p < 0.001, full table in our response to R3).
  • Two dual-arm dexterous tasks (R1, R2, R3): success rises from 2/40 to 12/40 over Bimanual Liquid Transfer and USB Insertion (p=0.006p = 0.006).
  • Comparison with EgoTouch and HumanTouch (R3): 880 taxels per hand in TouchScale vs. 256 in EgoTouch and 360 in HumanTouch.
  • Collection setup and quality control (R3): 500 of approximately 531 collected hours (94.2%) retained.
  • Semantic annotations (R1): 10 hours with temporal action segments, step-level captions, and video QA pairs.

We hope these results address the reviewers' concerns.

AuthorsOpenReview ↗

Summary of Revisions and Discussion

We thank the reviewers for the constructive discussion. We have uploaded a revised PDF with all changes marked in blue:

  • Table 1: Added HumanTouch, and the caption no longer claims the most recorded hours (R3).
  • Sec. 3.2 and Appendix C: Clarified the definition of scene configurations, the single-site collection setup with five tabletop types and four background styles, and quality-control statistics (500 of approximately 531 hours retained) (R1, R3).
  • Table 4 and Sec. 4.3: Added EgoTouch mid-training under a matched budget (25.0% vs. 57.5%) (R3).
  • Appendix E.2: Added the randomly re-paired visual–tactile control (28.75% vs. 57.5%) (R2, R3).
  • Appendix E.1: Added success rates for two dual-arm tasks: Bimanual Liquid Transfer (10% to 30%) and USB Insertion (0% to 30%) (R1, R2, R3).
  • Appendix G: Added the 10-hour semantic annotations (R1).
  • Sec. 5: Expanded the limitations on environmental diversity, ground-truth hand pose, and policy transfer to other robots and tactile sensors (R1, R2, R3).

In the discussion, R1 noted that most concerns were resolved, and R3 recognized that the matched-budget comparison, bimanual experiments, and alignment control strengthen the paper. Their remaining concern regarding the environmental diversity of a single collection site is now explicitly discussed as a limitation. Transfer to a new tactile sensor is evaluated in Sec. 4.1 (Table 2), while transfer to a separate robot site is reported in Sec. 4.3 (Table 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.