acceptodds
Under review as a conference paper at ICLR 2027

Tac-DINO: Learning Tactile Features with Patch Alignment

Abstract

Multisensory perception is essential for robotic manipulation, with tactile sensing providing critical information for fine-grained, contact-rich interactions. Although prior work integrates tactile and spatial signals to improve perception, vision-tactile learning commonly relies on object-level alignment. Addressing this limitation requires systematically aligned 3D vision-tactile data. We develop a data collection system and introduce Touch3D, a dataset comprising 505 real-world objects and 20,025 tactile contacts. Building on this dataset, we establish the Contact-Centric Multimodal Tactile Representation Benchmark (CoMT-Bench), which expands evaluation from object-level retrieval and tactile properties to include patch-level contact evaluation. We further propose Geometry-Relation Distilled Tactile-Centric Cross-Modal Learning (GeoTAC), which explores tactile learning across all multisensory data. Experiments on the Touch3D test set, patch-level alignment improves the average score by 5.32 over object-level alignment, with GeoTAC's geometric supervision improving 4.48 more. In robotic manipulation, GeoTAC improves TP by 19.14 and 11.34 over DINOv2 (Vision-only) and DINOv2 (Vision-Tactile), respectively.

Primary area:
datasets and benchmarks
Keywords:
Tactile Learning, Vision-Tactile Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y5CBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that vision-tactile learning has been confined to object-level alignment and proposes to move it to the contact-patch level. The authors build a data-collection platform and introduce Touch3D, a 3D vision-tactile dataset that pairs each tactile contact with a registered object mesh, its contact location, and material annotations. They define CoMT-Bench, which evaluates tactile representations at the object level, at the contact-patch level, and on material and object-category prediction. They further propose GeoTAC, a DINOv2-based tactile encoder combining tactile-centered cross-modal alignment, local-to-global distillation, and geometry-relation distillation. Against object-level alignment, and against CL, DAR, ImageBind, UniTouch and ViT-Lens, the authors report consistent gains from patch-level alignment and from GeoTAC, and better manipulation performance than the vision-only and vision-tactile DINOv2 baselines in a simulated benchmark.

Critical strengths
  1. The dataset is a substantial and timely resource. Touch3D contains 505 real objects and 20,025 contact samples with paired RGB-D observations, scanned 3D geometry, contact locations, and material labels, collected over roughly 700 hours. This is a meaningful scale-up over ObjectFolder Real and, unlike Touch and Go, provides scanned 3D geometry and contact-level annotations. Since the main bottleneck in this area is the lack of patch-aligned 3D vision-tactile data, the dataset is consequential: it enables the proposed benchmark and future work on contact-level tactile representation learning.
  2. The paper contributes a useful patch-level benchmark and a broad empirical study. CoMT-Bench moves from object-level to exact-contact retrieval across RGB, depth and pointmap, and the experiments compare object- vs. patch-level training, several multimodal baselines and module ablations. Patch-level alignment improves the reported average over its object-level counterpart, GeoTAC improves it further, and the downstream simulation results agree with the representation-level findings. This is consequential because it turns "patch alignment" into a measurable target.
Critical weaknesses
  1. The headline metric and the ablation evidence do not fully support the claims. “Avg.” is an unweighted mean of four heterogeneous metrics (object Top-5, patch E@5, material mAP, and category accuracy), and Table 3 reports ablations only for this average. Averaging hides trade-offs: Patch Tac+CL, for example, has lower object-level retrieval than Patch Tac (42.94 vs. 46.50) yet a higher average. Consequently, the ablation does not show whether Cross-Modal Alignment and Geometry Distillation improve individual metrics, or whether their gain in the average comes at the expense of a regression elsewhere. This matters because the per-component conclusions rest on a single aggregate that can mask per-metric failures.
  2. The paper lacks real-world robot manipulation experiments, even though the fundamental value of tactile representation learning lies in supporting robot manipulation. It evaluates only four tasks in the UniVTAC simulator and does not compare with recent object-level baselines such as AnyTouch and AnyTouch2. Except for grasp classification, GeoTAC's performance is modest, and on some tasks it even falls below the ACT baseline, which uses no tactile input (see https://univtac.github.io). This weakens the empirical case for the method.
  3. There are some typos and wording problems (e.g., “tactile sensing become,” “mainpulation”), and the sentence in Section 5.5, “The real-world results are consistent with the representation benchmark,” should refer to the UniVTAC simulation setting rather than real-world results.
Questions for authors

See Critical Weaknesses 1-3.

1 comment

Thanks for your hard works for helping us improving paper. We below do our best to solve your question at detailed ablation experiment results, real-world and simulator robot experiments, and typos and wording problems.

Q1: The headline metric and the ablation in Table 3 evidence do not fully support the claims.

Training configurationObject Top-5 (Tl→VgT_l \to V_g)RGB Patch E@5 (Tl↔VlT_l \leftrightarrow V_l)Depth Patch E@5 (Tl↔DlT_l \leftrightarrow D_l)Point Cloud Patch E@5 (Tl↔PlT_l \leftrightarrow P_l)Mat. mAPObj. Acc.V–T Avg. ↑
Patch Tac46.506.46——30.3756.4234.94
Patch Tac + CL42.9413.95——34.5059.0837.62
Patch Tac + CMA51.7713.943.151.8436.0359.9440.42
Patch Tac + GD51.6112.851.690.9534.9258.1039.40
GeoTAC (CMA + GD)50.5916.373.141.7138.7562.6942.10

A1: The top table shows the detailed results of the ablation study reported in Table 3 of the main paper.

From the results, we can easily identify the following phenomena.

  • Patch-aligned vision-tactile contrastive learning improves local vision-tactile alignment and tactile property prediction while slightly decreasing performance on local tactile-to-global visual retrieval.
  • Tactile-Centered Cross-Modal Alignment (the Patch Tac + CMA row) improves the alignment of tactile signals with depth and point clouds while retaining the advantages of patch alignment. The results for vision-tactile retrieval and tactile property prediction remain consistent.
  • Geometry distillation (the Patch Tac + GD row) further improves local vision-tactile alignment, which is consistent with our intuition that local geometric similarity helps local vision-tactile learning. It is worth noting that our GeoTAC shows a decrease in performance on local tactile-to-global visual retrieval, which we attribute to competition during the learning process.

Q2: The paper lacks real-world robot manipulation experiments, even though the fundamental value of tactile representation learning lies in supporting robot manipulation. It evaluates only four tasks in the UniVTAC simulator and does not compare with recent object-level baselines such as AnyTouch and AnyTouch2. Except for grasp classification, GeoTAC's performance is modest, and on some tasks it even falls below the ACT baseline, which uses no tactile input (see https://univtac.github.io). This weakens the empirical case for the method.

A2: We below detailed answer above question.

  • Real-world experimentals. Thanks for your recommendation to conduct real-world robot manipulation experiments. Our simulator experiments are conducted using the same dataset episodes and the same training and evaluation parameters. This provides some evidence for the efficiency of our tactile encoder. We will carefully consider your suggestion and conduct these experiments in future work.
  • AnyTouch and Anytouch2 vision-tactile baseline. The AnyTouch and AnyTouch2 methods build on the Tactile Dynamic Pyramid they collected and cannot be directly adapted to our dataset. Our proposed methods use vision, tactile, depth, and point cloud data, so a comparison with vision-tactile methods would be unfair because our methods use more modalities. In our experiments, we include commonly used multimodal learning methods such as ImageBind, UniTouch, and ViT-Lens.
  • DP vision-tactile lower than ACT vision-only. Different model has Inconsistent efficiency on robot tasks, as in ArtVIP (ICLR 2026) and Equivariant diffusion policy for sample-efficient robotic manipulation (IJRR). This phenomenon can be attributed to differences in policy models.

Q3: There are some typos and wording problems.

A3: Thanks for your careful reading and for helping us polish our paper. We will address all the problems to improve the formulations.

Reviewer_JVDAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors address the scale mismatch between local tactile readings and whole-object images via contact-localized representation learning. They introduce Touch3D (~20K localized GelSight presses on 505 3D-scanned objects with RGB-D and contact annotations) and CoMT-Bench for patch-level cross-modal retrieval. They also propose GeoTAC, a DINOv2 tactile encoder trained with local-to-global distillation, patch-level contrastive alignment, and 3D geometric supervision. Experiments show that patch-level training (+5.32) and GeoTAC (+4.48) significantly outperform object-level baselines on Touch3D, while boosting Diffusion Policy progress by ~19 points across four UniVTAC simulation tasks.

Critical strengths
  1. The dataset is the main strength. Collecting real tactile data that is registered to a scanned 3D shape is slow and painful; the authors report 700+ hours. The only comparable real dataset with meshes, ObjectFolder Real, has 100 objects. Touch3D covers 505 everyday objects, with several materials per object and a much wider category range. If it is released with meshes, contact annotations, and the raw tactile videos, I can see people using it for tactile-geometry and contact localization work even if they never use GeoTAC.
  2. The robot experiment asks the right question. A lot of tactile representation papers stop at retrieval/classification numbers. Here the encoder is plugged into Diffusion Policy, and the ranking roughly carries over: UniTouch and ViT-Lens encoders actually do worse than vision-only, and the patch-aligned ones do better.
Critical weaknesses
  1. The paper frames prior work as aligning touch with whole-object images, which is not really the state of the field. Touch and Go pairs each press with video frames recorded during the contact; TVL's HCT subset films every touch with the contact point in view; TaRF registers 19.3K real touches to 3D scenes and gives each one a spatially aligned visual view. More recently, Seeing Through Touch (arXiv 2604.11579) makes the same argument that global alignment misses local correspondence.

    • The method-side novelty comes down to the geometry-relation distillation. That is exactly the part the paper explains least: how tactile depth/point clouds are reconstructed, what the relation matrices R^r are computed over, and what "soft region matching" in LcL_c is are never stated. In Table 3 it adds +1.78 on its own.
  2. I don't think the experiments currently support the claims made from them.

    • Data split: The 4:1:1 split is not described.
    • Avg metric: The headline "Avg" mixes Top-5 retrieval, E@5, mAP, and probe accuracy.
    • Robot results: With saturated tasks, close success counts, and only simulated tactile input, I can't tell whether the encoder really matters for contact-rich control. A couple of real-robot tasks where touch is clearly needed (insertion, slip, or in-hand adjustment) would be far more convincing than more simulated ones.
Questions for authors

Clarifications that could change my assessment:

  1. Is the train/val/test split done by object? If not, could you report the main table with an object-disjoint split?
  2. How accurate are the manually clicked contact locations and the camera-to-mesh registration, compared with the crop size? Are the depth and point map crops taken from the depth camera or rendered from the scanned mesh?
  3. How exactly are the tactile depth and point clouds reconstructed, and what are LcL_c ("soft region matching") and RrR^r computed over?
  4. In the policy experiments, is the encoder frozen or fine-tuned together with Diffusion Policy?
  5. Will Touch3D (meshes, tactile videos, RGB-D, annotations) and the collection setup be released?

Suggestions:

  • Add a few real-robot tasks where tactile feedback is essential.
4 comments

Thanks for your exhausted works in reviewing our paper and proposing insightful questions. Below, we answer your questions about the comparison with existing work, real-world robot results, and implementation details.

Q1: The paper frames prior work as aligning touch with whole-object images, which is not really the state of the field. There need compare the existing works including Touch and Go, TVL's HCT subset, TaRF, and Seeing Through Touch.

A1: We show below the comparison with existing work in terms of data and methods. The visualization can be found in Figure 12 of the revised paper.

Data Collection Comparison

  • The Touch and Go dataset pairs tactile measurements with coarse third-person images and lacks fine-grained vision-tactile alignment labels.
  • TVL's HCT dataset records each touch with the contact point in view, as we do in our vision-tactile data collection. We also scan 3D shapes and align the vision-tactile data with them.
  • Seeing Through Touch uses the Touch and Go dataset, which pairs tactile measurements with recorded third-person images and lacks the fine-grained contact position labels found in our dataset and the ObjectFolder Real dataset.
  • TaRF collects data at the scene level, and our dataset focuses on objects, making the two datasets complementary. TaRF starts with a scene, reconstructs it using a NeRF, and collects paired 3D vision-tactile data within the scene.
  • Our Touch3D dataset starts with objects and provides 3D vision-tactile data with manually labeled contact positions, material labels, and object category labels.

Methods Comparison.

  • Touch and Go concentrates on multimodal self-supervised representation learning, tactile-driven image stylization, and touch prediction. Its self-supervised representation learning uses contrastive learning on vision-tactile data, which is consistent with our Tac + CL and Patch + CL.
  • Seeing Through Touch concentrates on tactile localization. They combined vision-tactile contrastive learning and a material diversity-based pairing strategy to extend the utilization of additional in-the-wild web images.
  • TVL studies vision-tactile question-answering tasks. TVL extends ImageBind to build its tactile encoder. Our baseline also extends ImageBind to align tactile, vision, depth, and point cloud data.
  • TaRF studies the ability to deep touch estimation with taking diffusion model closely follows Stable Diffusion.
  • Our methods concentrate on tactile feature learning with patch alignment across a tactile-centered multimodal dataset.

Q2: It is recommed to implemented in Real-robot tasks to explore encoder really matters for contact-rich control?

A2: Thanks for your suggestions. Our current simulator setup keeps the episodes, training steps, and learning parameters consistent. This provides some evidence for the efficiency of our tactile encoder. We will carefully consider your advice and implement our method in the real world in future work.

Q3: More detailed experiment implementation. 1. Is the train/val/test split done by object? If not, could you report the main table with an object-disjoint split? 2. How accurate are the manually clicked contact locations and the camera-to-mesh registration, compared with the crop size? Are the depth and point map crops taken from the depth camera or rendered from the scanned mesh? 3. How exactly are the tactile depth and point clouds reconstructed, and what are ("soft region matching") and computed over? 4. In the policy experiments, is the encoder frozen or fine-tuned together with Diffusion Policy? 5 Will Touch3D (meshes, tactile videos, RGB-D, annotations) and the collection setup be released?

A3: We below detailed answer above question.

  • Dataset split: The dataset split is based on objects. Contact- and frame-level splits are incorrect because different contact regions on the same object can have similar physical properties, which are reflected in their tactile signals. A simple split without manual selection is insufficient for constructing a test set.
  • Accuracy of manually clicked contact locations: The contact locations are selected within affordance areas, which makes the annotations robust to click errors and keeps them close to the intended contact points. Furthermore, we have a system to double-check the labeled data and filter out low-quality labels. See Figure 11 in the revised paper for detailed sample visualizations.
  • Depth and point map crops Resources: These are rendered from the scanned mesh.
  • The methods for tactile depth and point clouds reconstructed: We use the official GelSight Mini SDK for tactile depth and point cloud reconstruction. GelSight Mini SDK repository.
  • Computation of LcL_c and RrR^r. Lc\mathcal{L}_c is a bidirectional multi-positive InfoNCE loss inspired by “Learning Transferable Visual Models From Natural Language Supervision.” Tactile is the central modality, and each loss term pairs local tactile features with features from one other modality: local vision, depth, or point clouds.
= -\frac{1}{2B}\sum_{i=1}^{B} \left[ \log\frac{\sum_{j\in\mathcal{P}(i)}\exp(s_{ij}/\tau)} {\sum_{j=1}^{B}\exp(s_{ij}/\tau)} + \log\frac{\sum_{j\in\mathcal{P}(i)}\exp(s_{ji}/\tau)} {\sum_{j=1}^{B}\exp(s_{ji}/\tau)} \right].$$ $R^r$ is the pairwise cosine similarity matrix for each modality pair in direction $r$. The source pairs are local depth crops paired with tactile depth and local point cloud crops paired with tactile point clouds. The target pair is local vision paired with tactile. Let $\bar{\mathbf{x}}_i$ and $\bar{\mathbf{y}}_j$ be the pooled contact features for modalities $X$ and $Y$, respectively. $$[R_{X,Y}^{\rightarrow}]_{ij} = \frac{\bar{\mathbf{x}}_i^\top\bar{\mathbf{y}}_j} {\lVert\bar{\mathbf{x}}_i\rVert_2\\,\lVert\bar{\mathbf{y}}_j\rVert_2}, \qquad R_{X,Y}^{\leftarrow}=(R_{X,Y}^{\rightarrow})^\top.$$ - **Training setup of Diffusion Policy:** All parameters are fine-tuned in the policy experiments. - **Dataset and collection setup release:** We have finished preparing the data, code, and collection platform. We will release them after acceptance.

I thank the authors for providing an explanation of the existing work (and adding Figure 12 for clearer illustration) and details about the experimental implementation. However, I still have concerns regarding the lack of experiments on a physical robot. I keep my original rate.

Thanks for your response, and we are excited that we have addressed your other concerns. We are going to prepare real-robot experiments, which involve hard work on assembly, data collection, policy training for the baselines and our methods, and real-world inference. We will add these experiments in the final version.

Reviewer_VmxcOpenReview ↗
Rating
2
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to learn better tactile representations by aligning each touch with the corresponding local visual region and using 3D geometry. It introduces Touch3D, a dataset containing 505 real-world objects associated with visual, geometric, and annotation data. It also presents the associated benchmark for evaluation and the GeoTAC method.

Critical strengths
  1. I recognize the importance of the problem and concepts. If tactile signals can be matched with precise local texture, depth, and 3D geometry, performance will be better than pairing only with visual signals.
  2. The evaluation is comprehensive. The authors conduct multiple controlled experiments and test the method on downstream simulation tasks.
Critical weaknesses
  1. I find it difficult to identify the central contribution, as the paper presents several contributions: the dataset, the benchmark, and the method. Each requires different supporting evidence for its novelty. I see the improved scale and diversity of the dataset collected compared to prior datasets, but linking it to improved learning or generalization would help.
  2. The core idea of aligning touch with the corresponding local visual region has been explored in prior work. The authors could clarify what their approach adds beyond existing local-alignment methods. Existing comparisons in the paper are primarily against global-alignment methods to demonstrate the value of local alignment, which does not sufficiently establish the advantage over existing local-alignment approaches. Seeing Through Touch (Kim et al., 2026) learns dense local visual–tactile correspondences for tactile-conditioned material localization. TaRF (Dou et al., 2024) also establishes spatial correspondence between vision and touch within a shared 3D scene representation.
  3. Some important implementation details are missing. For example, Section 4.2 does not clearly define how the geometry-relation matrices are constructed or how “soft region matching” is implemented. The representation-pretraining setup is also under-specified.
  4. It is unclear whether the evaluation tests unseen objects or new contacts on previously observed objects. The paper reports the train/validation/test split ratio but does not specify the unit of splitting, such as object instance, contact, or frame.
Questions for authors

Could you clarify how GeoTAC goes beyond the local alignment used in Seeing Through Touch, and how does Touch3D’s spatial correspondence differ from that in TaRF? Could you explain how you compute the geometry-relation matrices and implement “soft region matching”? Please also report the main pretraining settings. Is the 4:1:1 split made by object, contact, or frame? Can the same physical object or contact sequence appear in both the training and test sets?

4 comments

Thank you for your hard work in reviewing our paper. Below, we address your questions about our central contribution, comparisons with related work, and implementation details.

Q1: I find it difficult to identify the central contribution, as the paper presents several contributions: the dataset, the benchmark, and the method. Each requires different supporting evidence for its novelty. I see the improved scale and diversity of the dataset collected compared to prior datasets, but linking it to improved learning or generalization would help.

A1: In this paper, we mainly propose and explore the question of learning tactile features with patch alignment methods across a tactile-centered multimodal dataset. To support this exploration, we propose a data collection platform and collect the largest object-level 3D vision-tactile dataset. We also explore methods for learning tactile features using the collected dataset.

Q2: Could you clarify how GeoTAC goes beyond the local alignment used in Seeing Through Touch, and how does Touch3D’s spatial correspondence differ from that in TaRF?

A2: Below, we describe the improvements to our approach in terms of data and methods. The visualization can be found in Figure 12 of the revised paper.

Data Collection Comparison

  • Seeing Through Touch uses the Touch and Go dataset, which pairs tactile measurements with recorded third-person images and lacks the fine-grained contact position labels found in our dataset and the ObjectFolder Real dataset.
  • TaRF collects data at the scene level, and our dataset focuses on objects, making the two datasets complementary. TaRF starts with a scene, reconstructs it using a NeRF, and collects paired 3D vision-tactile data within the scene.
  • Our Touch3D dataset starts with objects and provides 3D vision-tactile data with manually labeled contact positions, material labels, and object category labels.

Methods Comparison.

  • Seeing Through Touch concentrates on tactile localization. They combined vision-tactile contrastive learning and a material diversity-based pairing strategy to extend the utilization of additional in-the-wild web images.
  • TaRF studies the ability to deep touch estimation with taking diffusion model closely follows Stable Diffusion.
  • Our methods concentrate on tactile feature learning with patch alignment across a tactile-centered multimodal dataset.

Although we focus on studying tactile feature learning with patch alignment on a high-quality object-level 3D vision-tactile dataset, integrating coarse vision-tactile data, such as the Touch and Go dataset, and scene-level 3D vision-tactile data TaRF is part of our future work.

Q3: Could you explain how you compute the geometry-relation matrices and implement “soft region matching”? Please also report the main pretraining settings. Is the 4:1:1 split made by object, contact, or frame? Can the same physical object or contact sequence appear in both the training and test sets?

A3: Below, we do our best to provide a detailed description to clear up any confusion about the implementation. We will publicly release all data, code, and checkpoints and polish our paper to make the results easier to reproduce.

  • Geometry-relation matrices. The geometry-relation metrics are computed in the same way as those for tactile features tit_i and local RGB crops ViV_i, with local depth crops DiD_i or local point clouds PiP_i in place of the local RGB crops. We use the bidirectional exact-contact Top-5 retrieval metric (E@5), defined as follows.

sij(M)=cos⁡(ti,mj).s_{ij}^{(M)}=\cos(t_i,m_j).

\mathbf{1}\left\lbrace\operatorname{rank}_{j\in\mathcal{G}_i}(s_{ii}^{(M)};\\,s_{ij}^{(M)})\le 5\right\rbrace + \mathbf{1}\left\lbrace\operatorname{rank}_{j\in\mathcal{G}_i}(s_{ii}^{(M)};\\,s_{ji}^{(M)})\le 5\right\rbrace \right].$$ Here, $N$ is the number of evaluated contacts, and candidates are ranked in descending order of cosine similarity. - **Soft region matching.** We use a tactile-centered, bidirectional multi-positive InfoNCE loss inspired by “Learning Transferable Visual Models From Natural Language Supervision.” Each loss term pairs tactile features with one other modality: local vision, depth, or point clouds. $$\mathcal{L}_c(T_l,M_l) =-\frac{1}{2B}\sum_{i=1}^{B}\left[ \log\frac{\sum_{j\in\mathcal{P}(i)}\exp(s_{ij}/\tau)} {\sum_{j=1}^{B}\exp(s_{ij}/\tau)} + \log\frac{\sum_{j\in\mathcal{P}(i)}\exp(s_{ji}/\tau)} {\sum_{j=1}^{B}\exp(s_{ji}/\tau)} \right].$$ - **Pretraining setting parameters.** We initialized GeoTAC from the official DINOv2 ViT-L/14 checkpoint and pretrained it on Touch3D for 10,400 AdamW updates with a global batch size of $B=192$ contacts (24 per GPU across 8 GPUs) and four tactile frames per contact. The learning rate warmed up linearly to $10^{-4}$ over 2,080 updates and then followed a cosine decay schedule. We froze the external RGB, depth, and point-cloud encoders and set the cross-modal alignment and geometry-distillation loss weights to $\lambda_{\mathrm{align}}=\lambda_{\mathrm{geo}}=1$. - **Dataset Split.** The dataset is split by object. Contact- and frame-level splits are incorrect because different contact regions on the same object can have similar physical properties, which are reflected in their tactile signals. A simple split without manual selection is insufficient for constructing a test set.

Thank you for clarifying the object-level split and providing additional implementation details. I appreciate the dataset contribution. However, the methodological advance over existing local-alignment approaches remains unclear, and direct comparisons to these approaches are missing. My main concern remains. I will maintain my original score.

Thanks for your response. We describe the comparison of our methods with existing work in detail. We have added Figure 13 in the revised paper to show this comparison.

  • As shown in the dataset comparison in Figure 12, TaRF and Seeing Through Touch use scene-level vision–tactile data and coarse material-aligned vision–tactile data, respectively.
  • The objective of Seeing Through Touch is tactile localization. To achieve this, the paper uses the Touch and Go dataset and adds web data with matching material labels. The paired samples only share the same material label and may differ in physical attributes of touch, such as geometry and force. Vision–tactile contrastive learning uses positive pairs with the same material label and negative pairs with different material labels. Material is one of the physical attributes captured by tactile signals. Material-centered learning can limit capabilities such as depth estimation, object category estimation, and force perception, etc.
  • TaRF is a tactile generation method, while our method learns tactile features from multimodal signals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.