- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that vision-tactile learning has been confined to object-level alignment and proposes to move it to the contact-patch level. The authors build a data-collection platform and introduce Touch3D, a 3D vision-tactile dataset that pairs each tactile contact with a registered object mesh, its contact location, and material annotations. They define CoMT-Bench, which evaluates tactile representations at the object level, at the contact-patch level, and on material and object-category prediction. They further propose GeoTAC, a DINOv2-based tactile encoder combining tactile-centered cross-modal alignment, local-to-global distillation, and geometry-relation distillation. Against object-level alignment, and against CL, DAR, ImageBind, UniTouch and ViT-Lens, the authors report consistent gains from patch-level alignment and from GeoTAC, and better manipulation performance than the vision-only and vision-tactile DINOv2 baselines in a simulated benchmark.
Critical strengths
- The dataset is a substantial and timely resource. Touch3D contains 505 real objects and 20,025 contact samples with paired RGB-D observations, scanned 3D geometry, contact locations, and material labels, collected over roughly 700 hours. This is a meaningful scale-up over ObjectFolder Real and, unlike Touch and Go, provides scanned 3D geometry and contact-level annotations. Since the main bottleneck in this area is the lack of patch-aligned 3D vision-tactile data, the dataset is consequential: it enables the proposed benchmark and future work on contact-level tactile representation learning.
- The paper contributes a useful patch-level benchmark and a broad empirical study. CoMT-Bench moves from object-level to exact-contact retrieval across RGB, depth and pointmap, and the experiments compare object- vs. patch-level training, several multimodal baselines and module ablations. Patch-level alignment improves the reported average over its object-level counterpart, GeoTAC improves it further, and the downstream simulation results agree with the representation-level findings. This is consequential because it turns "patch alignment" into a measurable target.
Critical weaknesses
- The headline metric and the ablation evidence do not fully support the claims. “Avg.” is an unweighted mean of four heterogeneous metrics (object Top-5, patch E@5, material mAP, and category accuracy), and Table 3 reports ablations only for this average. Averaging hides trade-offs: Patch Tac+CL, for example, has lower object-level retrieval than Patch Tac (42.94 vs. 46.50) yet a higher average. Consequently, the ablation does not show whether Cross-Modal Alignment and Geometry Distillation improve individual metrics, or whether their gain in the average comes at the expense of a regression elsewhere. This matters because the per-component conclusions rest on a single aggregate that can mask per-metric failures.
- The paper lacks real-world robot manipulation experiments, even though the fundamental value of tactile representation learning lies in supporting robot manipulation. It evaluates only four tasks in the UniVTAC simulator and does not compare with recent object-level baselines such as AnyTouch and AnyTouch2. Except for grasp classification, GeoTAC's performance is modest, and on some tasks it even falls below the ACT baseline, which uses no tactile input (see https://univtac.github.io). This weakens the empirical case for the method.
- There are some typos and wording problems (e.g., “tactile sensing become,” “mainpulation”), and the sentence in Section 5.5, “The real-world results are consistent with the representation benchmark,” should refer to the UniVTAC simulation setting rather than real-world results.
Questions for authors
See Critical Weaknesses 1-3.
Thanks for your hard works for helping us improving paper. We below do our best to solve your question at detailed ablation experiment results, real-world and simulator robot experiments, and typos and wording problems.
A1: The top table shows the detailed results of the ablation study reported in Table 3 of the main paper.
From the results, we can easily identify the following phenomena.
A2: We below detailed answer above question.
A3: Thanks for your careful reading and for helping us polish our paper. We will address all the problems to improve the formulations.