- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MoDiTac, a diffusion-transformer framework for generating tactile images from visual observations. This is achieved while continuously controlling the trade-off between reconstruction fidelity and perceptual realism. The three main components—(a) tactile-guided visual selection, (b) modality-decoupled projections, and (c) preference-controlled global-context injection—address meaningful challenges in visual-to-tactile synthesis. Evaluation across TacQuad, Touch-and-Go, and ObjectsGrasping demonstrates consistent trade-offs between paired reconstruction metrics and perceptual metrics.
Critical strengths
The ablation studies support the proposed architectural components, and downstream experiments give good evidence that material recognition benefits from reconstruction-oriented outputs
Critical weaknesses
The main limitation is the scope of the evidence that supports the method’s broader utility. Although the experiments span three datasets, the central downstream results concern recognition tasks rather than physical manipulation or real-world robotic performance.
Improved FID or DISTS does not necessarily establish that generated tactile signals preserve physically meaningful forces, deformations, or contact dynamics.
The paper acknowledges that frames are generated independently; thus, its temporal recognition results do not demonstrate physically coherent tactile sequences.
Finally, results are reported primarily as point estimates, leaving uncertainty and run-to-run variability unclear.
Questions for authors
Recommendations for improvement
Provide confidence intervals or results across multiple training seeds, particularly for downstream accuracy and Macro-F1.
Add temporal-consistency and physically grounded evaluations, ideally including real sensor measurements or manipulation outcomes.
Report inference time, GPU memory, and computational cost, and discuss whether the benefits justify these requirements.
Strengthen comparisons with the closest feasible tactile-generation methods under matched data splits and evaluation protocols. You should more explicitly distinguish your contribution from closely related work, including AnyTouch (Feng et al., 2025), ControlTac (Luo et al., 2025), SplatTouch (Stefani et al., 2025), and Touch100K (Cheng et al., 2025). These papers address tactile representation learning, controllable tactile augmentation, explicit 3D vision–touch alignment, and large-scale touch-language-vision data. They are cited but not included as direct generation baselines; you should clarify which are technically comparable and, where feasible, provide additional comparisons.
5 comments
[W1/W2/Q2] (continued)
(2) AnyTouch feature consistency. AnyTouch is pretrained on data including TacQuad, so its features encode GelSight texture, deformation, and contact patterns. On TacQuad, gives the lowest paired cosine distance (0.033) and Feature MMD² (0.092) among all methods, and both decrease as increases (new Table 4; five-point results with 95% CIs in Appendix K.8). Perception-oriented outputs are thus closer to their paired real observations in a tactile-domain representation, not only under generic image metrics.
(3) Physical meaning of Press–Sliding, and why helps force-related cues. Pressing is dominated by normal loading, while sliding involves tangential load and shear deformation, so telling them apart requires contact-mechanics cues rather than appearance alone. Classical contact mechanics (the Cattaneo–Mindlin problem [3, 4]; Johnson [5]) shows that under tangential load the contact center sticks while partial slip first appears in an annulus at the contact edge, as also observed in GelSight shear and slip measurements [6]. Cues for distinguishing press from slide and for reading tangential force therefore concentrate at the contact periphery and appear as high-frequency detail such as shear deformation and texture displacement. Attribution on real touch agrees: only 16.0% (Press) and 22.4% (Sliding) of the action model's attribution falls in the central half, versus 39.7% and 42.9% for Fruit Identity. This is the axis that controls: larger adds high-frequency detail and extends the global-context correction to the contact periphery (see our response to R1), which explains why improves both Press–Sliding recognition and force estimation.
Summary. Physical validity is not one-dimensional. Contact-mechanics information (force, contact mode) is better preserved at , while object identity and material augmentation favor (Tables 3 and 17). A single generator can supply suitable data for different physical uses through , consistent with our central claim. We agree that force estimation and recognition do not replace real robot manipulation; full contact dynamics and closed-loop utility remain to be validated.
Revision: a physical-relevance paragraph and the AnyTouch table (Table 4) in §4.4; Appendix K.7 (force protocol and full results) and K.8 (AnyTouch); Appendix L clarifies the perception-utility scope and adds physical-measurement and closed-loop validation as future work.
[1] Engelbracht, T., Zurbrügg, R., Wohlrapp, M., Büchner, M., Valada, A., Pollefeys, M., Blum, H., & Bauer, Z. (2026). Hoi! – A multimodal dataset for force-grounded, cross-view articulated manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8880–8890).
[2] Higuera, C., Sharma, A., Bodduluri, C. K., et al. (2025). Sparsh: Self-supervised touch representations for vision-based tactile sensing. In Proceedings of the 8th Conference on Robot Learning, PMLR 270:885–915.
[3] Cattaneo, C. (1938). Sul contatto di due corpi elastici: distribuzione locale degli sforzi. Rendiconti della Reale Accademia Nazionale dei Lincei, Classe di Scienze Fisiche, Matematiche e Naturali, Serie 6, 27, 342–348.
[4] Mindlin, R. D. (1949). Compliance of elastic bodies in contact. Journal of Applied Mechanics, 16(3), 259–268.
[5] Johnson, K. L. (1985). Contact Mechanics. Cambridge University Press.
[6] Yuan, W., Li, R., Srinivasan, M. A., & Adelson, E. H. (2015). Measurement of shear and slip with a GelSight tactile sensor. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 304–311).
[W3] Does frame-wise generation ensure temporal consistency?
Thank you for raising this. We agree that MoDiTac generates frames independently, so the Press–Sliding experiment evaluates action-state cues preserved in sequences of frame-wise predictions rather than explicitly modeled tactile dynamics, as stated in Appendix L.
(1) Press–Sliding measures action-state cues and does not rely on strict temporal order. Keeping the trajectory-disjoint split, test frames, and five-frame windows fixed, we change only the frame order of real touch fed to the frozen classifier:
| Real tactile input | Original order | Reversed | Fixed shuffle |
|---|---|---|---|
| Accuracy | 90.00% | 85.00% | 88.33% |
Reversal and shuffling keep frame content but destroy temporal order, and accuracy drops by only 5.00 and 1.67 points. Discrimination mainly relies on in-window contact shape, local deformation, and texture, i.e., the normal-loading and shear cues discussed above. Press–Sliding therefore tests whether frame-wise generated touch preserves contact-mode information; we do not use it as evidence of temporal coherence.
(2) Local selection and global injection plug into joint video generators. Both mechanisms use only quantities available at each denoising step (the noisy tactile state, visual tokens, and tactile hidden states), so neither depends on MoDiTac's frame-wise structure. We inserted them as a plugin into two unified models with native video generation, Show-o2 [7] and Lance [8], after first adapting both to tactile video generation on TacQuad, and compared against no-plugin versions with the same backbone and training budget. Two findings: at the plugin improves all five metrics over the no-plugin model, so local selection remains effective in joint video generation; and sweeping yields the reconstruction–perception trade-off on both backbones (FID: Show-o2 101.74→84.63; Lance 85.04→66.04). Our mechanisms are thus compatible with temporal backbones: the host supplies temporal modeling and the plugin supplies preference control. Integration details, setup, and full results are in our response to R2 (Q3).
These evaluations still use frame-level metrics; we have not yet assessed the physical coherence of generated sequences (e.g., whether contact regions evolve as in real dynamics). Future work will add contact-evolution constraints to video backbones, study action-conditioned long-horizon tactile prediction, and validate with physical measurements and closed-loop robot tasks.
Revision: Appendix L states frame-wise generation and the action-state scope of Press–Sliding, and adds these future directions.
[7] Xie, J., Yang, Z., & Shou, M. Z. (2025). Show-o2: Improved native unified multimodal models. In Advances in Neural Information Processing Systems (NeurIPS).
[8] Fu, F., Huang, M., Wu, S., et al. (2026). Lance: Unified multimodal modeling by multi-task synergy. arXiv preprint arXiv:2605.18678.
We sincerely thank the reviewer for the careful review and constructive comments, and for recognizing that the ablations support each component and that material recognition benefits from reconstruction-oriented outputs. The questions on physical validity and temporal consistency helped us delimit the scope of our contribution more clearly.
[W1/W2/Q2] Do recognition tasks and perceptual metrics show that generated touch is physically meaningful?
Thank you for this important point. MoDiTac studies preference-controllable tactile generation for downstream perception, i.e., how generation characteristics affect task utility, rather than closed-loop robot manipulation. We agree that FID and DISTS measure appearance and distribution and do not by themselves establish that force, deformation, or contact dynamics are preserved; Appendix L already lists integration with physical measurements as future work. We therefore added two evaluations independent of the generator's training objective, which together with the original results form a four-level ladder: (1) generic image metrics; (2) tactile-domain representation under a frozen AnyTouch encoder (new); (3) interaction mode and material via Press–Sliding recognition and material augmentation; (4) a continuous physical quantity via an independent force probe (new). Every evaluator is a frozen pretrained model or a probe trained on external real data and never seen by the generator, so results cannot be inflated by optimizing for a metric.
(1) Force estimation. We evaluate on Hoi! [1] (DIGIT sensors). Generators are adapted on 20 regular scenes, and force is evaluated on held-out complete interaction trajectories from 6 of these scenes, with no shared raw frames; MoDiTac uses the hyperparameters of Appendix E, and all baselines use the same data and split. The estimator is a frozen Sparsh-DINOv2 encoder with a ForceLinearProbe [2], trained only on external real DIGIT force data; real and generated touch share the same input processing and force reference.
Table R4-A: Force RMSE (N) on held-out Hoi! trajectories.
MoDiTac at r=1 gives the lowest errors among the compared generators; relative to the same model at r=0, tangential, normal, and overall errors drop by 0.147, 0.276, and 0.166 N. Real DIGIT input remains best, marking the remaining gap to real measurement. This extends our utility evidence from discrete recognition to a continuous physical quantity.