acceptodds
Under review as a conference paper at ICLR 2027

MoDiTac: Modality-Decoupled Diffusion Transformer for Preference-Controlled Tactile Generation

Abstract

Tactile data capture local object deformation and contact properties, but collecting them requires specialized sensors and physical interaction, making large-scale acquisition more difficult than for visual data. Visual-to-tactile generation can alleviate this bottleneck by providing scalable synthetic data for tactile models. Existing generators primarily emphasize paired reconstruction quality, while paying less attention to whether the generated tactile data suit different downstream tasks. Diffusion models optimized with mean squared error tend toward conservative mean predictions, which can suppress the high-frequency textures that describe fine-grained surface deformation. Tactile detail recovery is further complicated by the mismatch between scene-level visual context and local tactile response, as well as conflicts between heterogeneous representations. MoDiTac addresses these challenges within a unified model. Tactile-guided visual selection extracts tactile-relevant visual evidence from the full scene according to the current generative state. Modality-decoupled projections reduce interference between visual and tactile representations before joint self-attention. Preference-controlled global-context injection maps a continuous preference coordinate to different reconstruction–perception operating points, allowing one model to provide tactile data for different task requirements. Experiments on multiple visual–tactile datasets show that MoDiTac improves both reconstruction and perceptual quality. Material recognition and Press–Sliding recognition favor different regions of the preference continuum, indicating that no single output is uniformly optimal across tactile tasks. MoDiTac provides an explicit interface for selecting task-appropriate tactile data.

Primary area:
generative models
Keywords:
Visual-to-Tactile Generation, Multimodal Learning, Diffusion Models, Tactile Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_7cN2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MoDiTac, a diffusion-transformer framework for generating tactile images from visual observations. This is achieved while continuously controlling the trade-off between reconstruction fidelity and perceptual realism. The three main components—(a) tactile-guided visual selection, (b) modality-decoupled projections, and (c) preference-controlled global-context injection—address meaningful challenges in visual-to-tactile synthesis. Evaluation across TacQuad, Touch-and-Go, and ObjectsGrasping demonstrates consistent trade-offs between paired reconstruction metrics and perceptual metrics.

Critical strengths

The ablation studies support the proposed architectural components, and downstream experiments give good evidence that material recognition benefits from reconstruction-oriented outputs

Critical weaknesses

The main limitation is the scope of the evidence that supports the method’s broader utility. Although the experiments span three datasets, the central downstream results concern recognition tasks rather than physical manipulation or real-world robotic performance.

Improved FID or DISTS does not necessarily establish that generated tactile signals preserve physically meaningful forces, deformations, or contact dynamics.

The paper acknowledges that frames are generated independently; thus, its temporal recognition results do not demonstrate physically coherent tactile sequences.

Finally, results are reported primarily as point estimates, leaving uncertainty and run-to-run variability unclear.

Questions for authors

Recommendations for improvement

Provide confidence intervals or results across multiple training seeds, particularly for downstream accuracy and Macro-F1.

Add temporal-consistency and physically grounded evaluations, ideally including real sensor measurements or manipulation outcomes.

Report inference time, GPU memory, and computational cost, and discuss whether the benefits justify these requirements.

Strengthen comparisons with the closest feasible tactile-generation methods under matched data splits and evaluation protocols. You should more explicitly distinguish your contribution from closely related work, including AnyTouch (Feng et al., 2025), ControlTac (Luo et al., 2025), SplatTouch (Stefani et al., 2025), and Touch100K (Cheng et al., 2025). These papers address tactile representation learning, controllable tactile augmentation, explicit 3D vision–touch alignment, and large-scale touch-language-vision data. They are cited but not included as direct generation baselines; you should clarify which are technically comparable and, where feasible, provide additional comparisons.

5 comments

We sincerely thank the reviewer for the careful review and constructive comments, and for recognizing that the ablations support each component and that material recognition benefits from reconstruction-oriented outputs. The questions on physical validity and temporal consistency helped us delimit the scope of our contribution more clearly.

[W1/W2/Q2] Do recognition tasks and perceptual metrics show that generated touch is physically meaningful?

Thank you for this important point. MoDiTac studies preference-controllable tactile generation for downstream perception, i.e., how generation characteristics affect task utility, rather than closed-loop robot manipulation. We agree that FID and DISTS measure appearance and distribution and do not by themselves establish that force, deformation, or contact dynamics are preserved; Appendix L already lists integration with physical measurements as future work. We therefore added two evaluations independent of the generator's training objective, which together with the original results form a four-level ladder: (1) generic image metrics; (2) tactile-domain representation under a frozen AnyTouch encoder (new); (3) interaction mode and material via Press–Sliding recognition and material augmentation; (4) a continuous physical quantity via an independent force probe (new). Every evaluator is a frozen pretrained model or a probe trained on external real data and never seen by the generator, so results cannot be inflated by optimizing for a metric.

(1) Force estimation. We evaluate on Hoi! [1] (DIGIT sensors). Generators are adapted on 20 regular scenes, and force is evaluated on held-out complete interaction trajectories from 6 of these scenes, with no shared raw frames; MoDiTac uses the hyperparameters of Appendix E, and all baselines use the same data and split. The estimator is a frozen Sparsh-DINOv2 encoder with a ForceLinearProbe [2], trained only on external real DIGIT force data; real and generated touch share the same input processing and force reference.

Table R4-A: Force RMSE (N) on held-out Hoi! trajectories.

Tactile inputTangential↓Normal↓Overall↓
Real DIGIT2.9733.4343.742
GVST3.7724.2944.466
TextToucher3.5214.0244.233
VTTac3.4633.8744.105
MoDiTac, r=0r=03.4794.0394.153
MoDiTac, r=1r=13.3323.7633.987

MoDiTac at r=1r=1 gives the lowest errors among the compared generators; relative to the same model at r=0r=0, tangential, normal, and overall errors drop by 0.147, 0.276, and 0.166 N. Real DIGIT input remains best, marking the remaining gap to real measurement. This extends our utility evidence from discrete recognition to a continuous physical quantity.

[W1/W2/Q2] (continued)

(2) AnyTouch feature consistency. AnyTouch is pretrained on data including TacQuad, so its features encode GelSight texture, deformation, and contact patterns. On TacQuad, r=1r=1 gives the lowest paired cosine distance (0.033) and Feature MMD² (0.092) among all methods, and both decrease as rr increases (new Table 4; five-point results with 95% CIs in Appendix K.8). Perception-oriented outputs are thus closer to their paired real observations in a tactile-domain representation, not only under generic image metrics.

(3) Physical meaning of Press–Sliding, and why r=1r=1 helps force-related cues. Pressing is dominated by normal loading, while sliding involves tangential load and shear deformation, so telling them apart requires contact-mechanics cues rather than appearance alone. Classical contact mechanics (the Cattaneo–Mindlin problem [3, 4]; Johnson [5]) shows that under tangential load the contact center sticks while partial slip first appears in an annulus at the contact edge, as also observed in GelSight shear and slip measurements [6]. Cues for distinguishing press from slide and for reading tangential force therefore concentrate at the contact periphery and appear as high-frequency detail such as shear deformation and texture displacement. Attribution on real touch agrees: only 16.0% (Press) and 22.4% (Sliding) of the action model's attribution falls in the central half, versus 39.7% and 42.9% for Fruit Identity. This is the axis that rr controls: larger rr adds high-frequency detail and extends the global-context correction to the contact periphery (see our response to R1), which explains why r=1r=1 improves both Press–Sliding recognition and force estimation.

Summary. Physical validity is not one-dimensional. Contact-mechanics information (force, contact mode) is better preserved at r=1r=1, while object identity and material augmentation favor r=0r=0 (Tables 3 and 17). A single generator can supply suitable data for different physical uses through rr, consistent with our central claim. We agree that force estimation and recognition do not replace real robot manipulation; full contact dynamics and closed-loop utility remain to be validated.

Revision: a physical-relevance paragraph and the AnyTouch table (Table 4) in §4.4; Appendix K.7 (force protocol and full results) and K.8 (AnyTouch); Appendix L clarifies the perception-utility scope and adds physical-measurement and closed-loop validation as future work.

[1] Engelbracht, T., Zurbrügg, R., Wohlrapp, M., Büchner, M., Valada, A., Pollefeys, M., Blum, H., & Bauer, Z. (2026). Hoi! – A multimodal dataset for force-grounded, cross-view articulated manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8880–8890).

[2] Higuera, C., Sharma, A., Bodduluri, C. K., et al. (2025). Sparsh: Self-supervised touch representations for vision-based tactile sensing. In Proceedings of the 8th Conference on Robot Learning, PMLR 270:885–915.

[3] Cattaneo, C. (1938). Sul contatto di due corpi elastici: distribuzione locale degli sforzi. Rendiconti della Reale Accademia Nazionale dei Lincei, Classe di Scienze Fisiche, Matematiche e Naturali, Serie 6, 27, 342–348.

[4] Mindlin, R. D. (1949). Compliance of elastic bodies in contact. Journal of Applied Mechanics, 16(3), 259–268.

[5] Johnson, K. L. (1985). Contact Mechanics. Cambridge University Press.

[6] Yuan, W., Li, R., Srinivasan, M. A., & Adelson, E. H. (2015). Measurement of shear and slip with a GelSight tactile sensor. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 304–311).

[W3] Does frame-wise generation ensure temporal consistency?

Thank you for raising this. We agree that MoDiTac generates frames independently, so the Press–Sliding experiment evaluates action-state cues preserved in sequences of frame-wise predictions rather than explicitly modeled tactile dynamics, as stated in Appendix L.

(1) Press–Sliding measures action-state cues and does not rely on strict temporal order. Keeping the trajectory-disjoint split, test frames, and five-frame windows fixed, we change only the frame order of real touch fed to the frozen classifier:

Real tactile inputOriginal orderReversedFixed shuffle
Accuracy90.00%85.00%88.33%

Reversal and shuffling keep frame content but destroy temporal order, and accuracy drops by only 5.00 and 1.67 points. Discrimination mainly relies on in-window contact shape, local deformation, and texture, i.e., the normal-loading and shear cues discussed above. Press–Sliding therefore tests whether frame-wise generated touch preserves contact-mode information; we do not use it as evidence of temporal coherence.

(2) Local selection and global injection plug into joint video generators. Both mechanisms use only quantities available at each denoising step (the noisy tactile state, visual tokens, and tactile hidden states), so neither depends on MoDiTac's frame-wise structure. We inserted them as a plugin into two unified models with native video generation, Show-o2 [7] and Lance [8], after first adapting both to tactile video generation on TacQuad, and compared against no-plugin versions with the same backbone and training budget. Two findings: at r=0r=0 the plugin improves all five metrics over the no-plugin model, so local selection remains effective in joint video generation; and sweeping rr yields the reconstruction–perception trade-off on both backbones (FID: Show-o2 101.74→84.63; Lance 85.04→66.04). Our mechanisms are thus compatible with temporal backbones: the host supplies temporal modeling and the plugin supplies preference control. Integration details, setup, and full results are in our response to R2 (Q3).

These evaluations still use frame-level metrics; we have not yet assessed the physical coherence of generated sequences (e.g., whether contact regions evolve as in real dynamics). Future work will add contact-evolution constraints to video backbones, study action-conditioned long-horizon tactile prediction, and validate with physical measurements and closed-loop robot tasks.

Revision: Appendix L states frame-wise generation and the action-state scope of Press–Sliding, and adds these future directions.

[7] Xie, J., Yang, Z., & Shou, M. Z. (2025). Show-o2: Improved native unified multimodal models. In Advances in Neural Information Processing Systems (NeurIPS).

[8] Fu, F., Huang, M., Wu, S., et al. (2026). Lance: Unified multimodal modeling by multi-task synergy. arXiv preprint arXiv:2605.18678.

Reviewer_bjtbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MoDiTac, a visual-to-tactile generation framework that predicts tactile images from visual observations and text conditions. It combines tactile-guided visual selection and modality-decoupled projections with a preference parameter rr that adjusts the balance between reconstruction fidelity and perceptual similarity. Experiments on three datasets assess generation quality, the contribution of each component, and how different downstream recognition tasks favor different values of rr.

Critical strengths

A key strength of this work is that a single generator can adjust the balance between reconstruction fidelity and perceptual similarity. The preference-parameter sweep and the same-dataset comparison of downstream tasks show that different settings are better suited to different tasks.

Critical weaknesses

The contribution of visual input is not fully isolated from the text condition. In particular, the TacQuad experiments use corresponding tactile descriptions. A within-model comparison using no text or a generic prompt would help establish how much tactile information the model predicts from vision itself.

Questions for authors

I suggest clarifying how the tactile text descriptions are obtained at test time and whether they contain information derived from real tactile measurements. An ablation using no text or a generic text prompt would help quantify the independent contribution of the visual input.

I suggest providing a method for automatically selecting the preference parameter r for a downstream task. If r must be chosen manually, please describe the practical selection procedure.

9 comments

We sincerely thank the reviewer for the careful review and constructive comments, and for recognizing that a single generator can adjust the trade-off between reconstruction fidelity and perceptual similarity, supported by the preference sweep and same-dataset downstream comparisons. We respond point by point.

[W1/Q1] Text sources, leakage risk, and the independent contribution of vision.

Thank you for raising this important question. We agree that separating the contributions of vision and text is key to confirming that MoDiTac predicts touch from vision. We describe the text sources, report a text ablation, and explain why text does not leak test tactile information.

(1) Text sources. None of the texts comes from test tactile measurements, and MoDiTac never uses test tactile images to construct text (Appendix E).

  • TacQuad: we use the descriptions provided with the dataset [1]. They were generated by the dataset authors from RGB images of the objects, without tactile images, and are object-level: all frames of the same object share one description. Simple descriptions contain only tactile attribute words (e.g., smooth, rough); rich descriptions additionally include the object name and material category. We use the rich ones.
  • Touch-and-Go: only material labels are available, from which we build a fixed template.
  • ObjectsGrasping: no descriptions are available, and all samples use the same prompt "An object is being touched."

All baselines that take text use the same text, and TextToucher additionally applies the text augmentation of its original paper (Appendix D). Text is therefore not an information source exclusive to MoDiTac.

(2) Text ablation. On the TacQuad frame-level split, with the same MoDiTac setup, we train with rich text, simple text, and no text.

Table R3-A: Text-condition ablation and baselines on TacQuad (baseline values from Tables 1 and 3).

MethodConditionrrMAE↓DISTS↓Material Acc↑
MoDiTacvision + rich text (original)04.1510.18590.03
16.2550.14788.52
MoDiTacvision + simple text04.2540.19788.53
16.4720.15585.42
MoDiTacvision only (no text)04.1970.19388.96
16.2840.16485.78
GVSTvision only—15.7590.20886.55
TextTouchertext only—9.0330.15787.99
VTTacvision + text—4.5900.17488.49

[W1/Q1] (continued)

(3) The independent contribution of vision, and the role of text.

  1. Most tactile information comes from vision. Without text, MAE at r=0r=0 only changes from 4.151 to 4.197 (about 1%), and material accuracy changes from 90.03% to 88.96%. Vision-only MoDiTac still outperforms all baselines in MAE and material accuracy, including VTTac and TextToucher, which use the same text. MoDiTac's advantage therefore does not depend on text.
  2. Text cannot carry frame-specific contact information. Descriptions are object-level and shared by all frames of an object, so they can only identify the object; they cannot specify where contact occurs in a given frame or its shape and deformation. Paired metrics such as MAE and SSIM measure exactly this frame-level correspondence, which must be supplied by vision. Under the frame-level split, object identity is also visible in the image, so text and vision are redundant here, which explains why removing text barely changes MAE.
  3. The paper already contains a full generic-prompt experiment. ObjectsGrasping uses a single generic prompt throughout, which is the setting the reviewer suggested. MoDiTac's advantage holds there: at r=1r=1, FID is 97.43 versus 134.23 for VTTac (Table 1), and Press–Sliding accuracy is 70.84% versus 55.26% for VTTac (Table 3).

(4) No test-time leakage in material recognition. The material experiment follows an augmentation protocol (Appendix K.1): generated data only replace the training subset B, the test set is isolated real touch, and neither the generator nor the text is used at test time. Conditioning generation on the material category is label-conditioned data augmentation, a common practice in training with synthetic data [2, 3].

In practice, vision is the required and main information source; specific and reliable object or material descriptions can be added as a semantic prior; descriptions with only vague attributes should be avoided; and a generic prompt suffices for contact-state tasks. This is consistent with our response to R2.

Revision: §4.2 summarizes the independent contribution of vision without text; Appendix K.9 gives the full text ablation, and Appendix E states the text sources of all three datasets, including the RGB-derived, object-level TacQuad descriptions and the generic ObjectsGrasping prompt.

[1] Feng, R., Hu, J., Xia, W., Gao, T., Shen, A., Sun, Y., Fang, B., & Hu, D. (2025). AnyTouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. In International Conference on Learning Representations (ICLR).

[2] Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., & Tian, Y. (2024). Scaling laws of synthetic images for model training ... for now. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3] You, Z., Zhong, Y., Bao, F., Sun, J., Li, C., & Zhu, J. (2023). Diffusion models and semi-supervised learners benefit mutually with few labels. In Advances in Neural Information Processing Systems (NeurIPS).

[Q2] How to choose the preference coordinate rr for a downstream task.

Thank you for the suggestion. Our selection procedure builds on an analysis of task properties and requires no generator retraining.

Basis: two task-property dimensions. On the same ObjectsGrasping setting we analyzed two tasks with opposite preferences (full analysis in our response to R1 and new Figure 5). Press–Sliding relies on peripheral, high-frequency cues: only 16.0%/22.4% of its attribution falls in the central half, and replacing the r=0r=0 high band with the r=1r=1 high band raises accuracy by 13.16 points. Fruit Identity relies on central, low-frequency structure: 39.7%/42.9% of its attribution is central, and replacing the low band lowers accuracy by 2.83 points. The task-weighted feature error changes with rr in line with their opposite downstream preferences. The preferred rr is thus determined by whether discriminative cues lie at the contact center or periphery, and whether they are low-frequency deformation or high-frequency texture.

Two-stage procedure.

  1. Task-informed initialization: using a task classifier trained on real touch, run spatial attribution and band swapping. Start from a small r0r_0 (e.g., 0.25) for tasks relying on the contact center and low-frequency deformation, and from a large r0r_0 (e.g., 0.75) for tasks relying on the periphery and high-frequency texture. This step analyzes only real data and needs no generation.
  2. Validation calibration: evaluate a few coordinates near r0r_0 on a held-out validation set, select r∗=arg⁡max⁡rScoreval(r)r^*=\arg\max_{r}\mathrm{Score}_{\mathrm{val}}(r), and fix it for testing; test data are not used. Because switching rr needs no retraining and inference time stays within 8.49–8.56 s per image across the five values of rr, this search is cheap.

Toward automatic selection. We have integrated MoDiTac's two mechanisms as a plugin into unified tactile understanding–generation models and verified that the reconstruction–perception trade-off holds there (see our response to R2); in the future, the understanding branch of such a model could select rr automatically from a task description.

Revision: the task-preference paragraph and figure in §4.4 summarize the spatial and frequency-band evidence; Appendix K.5 gives the full analysis and K.6 the selection procedure.

We thank the reviewer again and are happy to clarify any remaining questions during the discussion.

Reviewer_furqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The topic of this paper is about Visual-to-Tactile Generation task on different downstream task requirements. The authors propose a MoDiTac framework with a Tactile-guided Visual Selection, modality decoupled projections and preference-controlled Global-context injection. The proposed method has been evaluated on several benchmarks and achieves good performance.

Critical strengths
  • Exploring visual-to-tactile generation model's robustness on different downstream task requirements is meaningful.
  • The performance of the proposed method outperforms that of other SOTA methods on several benchmarks.
  • The idea of MoDiTac framework is interesting.
Critical weaknesses
  • Evaluate the proposed method on other experimental settings. The performance of the proposed method on tactile generation tasks for different tactile sensors. The cross-sensor tactile generation scenario is also important for tactile generation task.
  • Missing the analysis about the input modality selection for tactile generation task. For example, this paper mainly focuses on the visual-to-tactile generation task and the TextToucher method focuses on the text-to-tactile generation. In the practical tactile generation on different downstream task requirements, how to select the input information (single-modality, which single modality, or multi-modality)?
Questions for authors

Q1: To better demonstrate the high-quality of the generated tactile data, can the open-source VT-LLM (e.g., [a]) model be used to compare the generated tactile data and the corresponding real data? [a] Universal Visuo-Tactile Video Understanding for Embodied Interaction, NeurIPS2026

Q2: The implementation details about the proposed method should be complemented.

Q3: Some recent works(e.g., [b]) utilize a unified generation and understanding model for tactile data. For the Visual-to-Tactile Generation task on different downstream task requirements, can the proposed method be integrated on these models? [b] UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation, ECCV2026

5 comments

We sincerely thank the reviewer for the constructive comments, and for recognizing the significance of adapting visual-to-tactile generation to different downstream requirements, the performance of MoDiTac across benchmarks, and the overall framework design.

[W1] Cross-sensor validation.

Thank you for highlighting sensor coverage. "Cross-sensor" has two levels: whether the method holds on another type of sensor, and cross-sensor generation when the target sensor lacks training data. Our three original datasets mainly use GelSight-type sensors. We added experiments on the DIGIT sensor to address the first level; the second is a future direction (see Q3).

(1) Adaptation and force estimation on DIGIT. We adapt MoDiTac and all baselines to DIGIT touch in the Hoi! dataset [1]: adaptation uses 20 regular scenes, and evaluation uses held-out complete interaction trajectories from 6 of these scenes; training hyperparameters follow Appendix E. Since Hoi! provides force references, we evaluate generated touch with an independent force estimator: a frozen Sparsh-DINOv2 encoder with a ForceLinearProbe [2], trained only on external real DIGIT force data. MoDiTac at r=1r=1 achieves the lowest tangential, normal, and overall errors among the compared generators, with an overall RMSE of 3.987 N (VTTac 4.105 N, GVST 4.466 N; full table in our response to R4 and Appendix K.7). The same method thus adapts to another type of optical tactile sensor and preserves force-related information readable by an independent probe; preference control also remains effective, with lower force error at the perception-oriented endpoint.

(2) Why MoDiTac transfers easily to new sensors. Its two mechanisms act only on visual conditions and tactile latents, and the preference objectives (image-space L1 and a high-frequency wavelet target) are sensor-agnostic, so generation needs no sensor-specific prior. By contrast, VTTac injects an uncontacted tactile background as a low-frequency sensor prior (Appendix D). Moving to a new sensor therefore requires only paired data from that sensor for adaptation.

(3) Future direction. Cross-sensor generation without target-sensor training data requires a cross-sensor unified representation. Integrating our mechanisms into cross-sensor unified models (see Q3) is a feasible route to preference-controllable cross-sensor generation.

Revision: §4.4 summarizes the DIGIT force result; Appendix K.7 gives the protocol and full results; Appendix L adds cross-sensor generation as future work.

[1] Engelbracht, T., Zurbrügg, R., Wohlrapp, M., Büchner, M., Valada, A., Pollefeys, M., Blum, H., & Bauer, Z. (2026). Hoi! – A multimodal dataset for force-grounded, cross-view articulated manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8880–8890).

[2] Higuera, C., Sharma, A., Bodduluri, C. K., et al. (2025). Sparsh: Self-supervised touch representations for vision-based tactile sensing. In Proceedings of the 8th Conference on Robot Learning, PMLR 270:885–915.

[W2] How to choose input modalities for different downstream tasks.

Thank you for this practical question. TacQuad provides two granularities of descriptions: simple ones contain only tactile attribute words (e.g., smooth, rough), and rich ones additionally include the object name and material category; we use the rich ones. Touch-and-Go provides only material labels, from which we build a fixed template. ObjectsGrasping has no descriptions, and all samples use the prompt "An object is being touched." On TacQuad, we retrain the same MoDiTac setup with rich text, simple text, and no text (full table in our response to R3 and Appendix K.9). Four findings:

  1. Vision supplies most of the tactile information. Without text, MAE at r=0r=0 only changes from 4.151 to 4.197 and material accuracy remains 88.96%, whereas text-only TextToucher has an MAE of 9.033. Vision provides instance-level, spatially aligned contact evidence (object position, shape, and contact geometry), while text provides only category-level descriptions; paired reconstruction therefore relies mainly on vision.
  2. Reliable text adds a semantic prior. Rich text further improves perceptual quality and material recognition: at r=1r=1, DISTS drops from 0.164 to 0.147 and accuracy rises from 85.78% to 88.52%.
  3. Text must be specific. Simple text is worse than no text in both MAE and accuracy at both endpoints; attribute words such as "smooth" or "rough" are shared across many materials and can conflict with visual evidence.
  4. Contact-state tasks rely mainly on vision. ObjectsGrasping uses a single generic prompt, so the Press–Sliding and Fruit Identity results in the paper are mainly driven by vision; contact states change frame by frame and are hard to describe in text.

Selection principles. Vision is the default input for every task. Contact-state, action, and force-related tasks (e.g., Press–Sliding) can use a generic prompt. Material and category tasks benefit from text when reliable object or material descriptions are available; descriptions with only vague attributes should be avoided. Text-only conditioning is a fallback when no visual observation exists, at a clear cost in paired fidelity. The input modality determines what information the generator has, and rr determines whether it is rendered in a reconstruction- or perception-oriented way; the two can be chosen separately for each task. Revision: §4.2 summarizes the vision-only result; Appendix K.9 gives the full ablation; Appendix E states the text sources.

[Q1] Evaluating generation quality with a VT-LLM. Thank you for the suggestion. During the discussion period we could not find publicly released full weights for VTV-LLM; we will add this evaluation to the final version once they are available. Following the idea of comparing generated and real touch with a tactile understanding model, we evaluate feature consistency with a frozen AnyTouch encoder, a tactile-domain representation whose pretraining includes TacQuad, applied with the same pipeline to all methods. r=1r=1 gives the lowest paired cosine distance (0.033) and Feature MMD² (0.092) among all methods (new Table 4; full results with 95% CIs in Appendix K.8). Together with the original and new results, generated touch is now evaluated by frozen understanding models at four levels: representation (AnyTouch), semantics (material recognition), dynamic state (Press–Sliding), and physics (force estimation).

[Q2] Implementation details. Thank you. Appendices A, B, and E give the main implementation details, and the code is in the Supplementary Material. Appendix E now adds details of visual conditioning and global injection: the visual encoder is a frozen CLIP ViT-L/14 (224×224 input); local selection uses its 256 last-layer patch tokens; global context is provided by 32 slots projected from the pooled feature; in each block, global injection uses the tactile hidden state as query and these slots as K/V, and its output is scaled by rr and added to the block's attention output. Inference takes about 8.5 s per image on one A800 with 37.61 GiB peak memory.

[Q3] Integration into unified tactile understanding–generation models.

Thank you for this inspiring suggestion. UniTac unifies cross-sensor tactile understanding and generation in a single model; this is an advanced architecture and an important direction for tactile modeling. It motivated us to test whether MoDiTac's mechanisms can be integrated into unified models, so that they keep their understanding ability while gaining preference control for different downstream needs.

(1) Why the mechanisms can be integrated. Tactile-guided local selection and preference-controlled global injection use only quantities present in any Transformer-based generator (the current noisy target state, visual condition tokens, and tactile hidden states) and act through generic operations (an attention bias and a residual cross-attention). They do not depend on MoDiTac's own backbone. We package them as a generation-side plugin that attaches only to the generation path and leaves the understanding path unchanged, so the model's understanding ability is preserved.

(2) On UniTac. Its code and weights were not public during the discussion period, so we could not run it directly. Mechanistically, as long as its generation path reads visual conditions through attention and maintains hidden states for the tactile target, the plugin can be attached in the same way as below. Combined with UniTac's cross-sensor capability, this could yield preference-controllable cross-sensor tactile generation, the future direction noted in W1. We will attempt this integration once UniTac is released.

(3) Validation on two public unified models. We chose two unified understanding–generation models with public code and weights and substantially different architectures: Show-o2 [3] (unified Transformer with a generation head) and Lance [4] (dual-stream MoE with separate understanding and generation paths).

  • Tactile adaptation: both were built for general video, so we first adapt them to touch: conditioned on synchronized RGB video and text, they generate tactile video and are trained on the TacQuad frame-level split (MoDiTac's train/test sets; five-frame windows; 432×432), giving the "no plugin" models below.
  • Integration: Show-o2 adds the relevance bias in the generation head's attention and injects the global residual before the head; Lance aligns visual time slots to VAE time slices, applies local selection in MoT attention, and injects the global residual in the last four MoT layers. The global branch reads each host's own visual representation (SigLIP features in Show-o2, ViT semantic features in Lance).
  • Fair control: within each backbone, plugin and no-plugin versions share initialization, loss, 30,000 steps, and sampling settings; each plugin sweeps five values of rr with fixed test windows and noise.

Table R2-D: Plugin results in unified models (compare only within a backbone).

BackboneSettingMAE↓PSNR↑SSIM↑DISTS↓FID↓
Show-o2no plugin6.2529.040.8980.176124.32
plugin, r=0r=05.9630.770.9090.166101.74
plugin, r=1r=17.0629.530.8850.15384.63
Lanceno plugin6.3928.950.8870.16993.43
plugin, r=0r=05.7130.030.8920.16285.04
plugin, r=1r=16.4329.180.8810.14966.04

(4) Findings. First, at r=0r=0 the global residual is off, yet the plugin improves all five metrics over each no-plugin model, so tactile-guided local selection remains effective in unified models. Second, over the five-point sweep both backbones show the reconstruction–perception trade-off (MAE rises; DISTS/FID fall), so the unified models gain MoDiTac's control knob rr. Given the task-dependent preference verified on MoDiTac (Appendix K.4), the integrated models can supply tactile data with task-appropriate characteristics.

(5) Future work. Because unified models can understand, their understanding branch could select rr automatically from a task description. We will release the plugin and adaptation interface. Revision: Appendix L adds unified-model integration and automatic rr selection as future work.

[3] Xie, J., Yang, Z., & Shou, M. Z. (2025). Show-o2: Improved native unified multimodal models. In Advances in Neural Information Processing Systems (NeurIPS).

[4] Fu, F., Huang, M., Wu, S., et al. (2026). Lance: Unified multimodal modeling by multi-task synergy. arXiv preprint arXiv:2605.18678.

We thank the reviewer again and are happy to clarify any remaining questions during the discussion.

Reviewer_u7cUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MoDiTac, a model that generates tactile images from visual input. The authors argue that existing tactile generators, which produce reconstruction-oriented outputs, are not suitable for all downstream tasks, and propose a system in which a single generator can control its preference between reconstruction and perception. To this end, MoDiTac uses tactile-guided visual selection to weight visual information related to the current tactile state, modality-decoupled projection to apply separate projections for each modality, and preference-controlled global-context injection, where a coefficient r sets both the amount of injected global context and the balance between distortion and high-frequency texture objectives. In experiments, MoDiTac achieves the best MAE and SSIM at r=0 and the best DISTS and FID at r=1 among the compared methods, with metrics shifting as r changes within the same checkpoint. Material recognition perform best at r=0 and Press–Sliding recognition at r=1, from which the authors claim that different downstream tasks prefer different generation characteristics.

Critical strengths
  1. The paper raises the point that downstream tasks may require different properties from generated tactile images, and addresses this through the controllability of a single generative model. In particular, users can choose between reconstruction-oriented and perception-oriented outputs simply by adjusting r without retraining, which gives practical value for using generated data in different settings.

  2. The main claim is well supported by experiments. The trade-off is consistent across diverse settings, and the controlled task comparison shows that task preferences differ.

Critical weaknesses
  1. Limited analysis of why tasks prefer different r values. The paper does not analyze which task properties explain those preferences or how they relate to changes in the generated images as r varies. Although the results show differences in task performance, the link between each task’s requirements and the tactile image characteristics controlled by r remains unclear. Such an analysis could help users choose r for new tasks and improve the method’s practical usefulness.
  2. Additional validation of the design would clarify the role of each component.
    • Tactile-guided selection. The benefit of updating the map at every step is unclear. Table 7 (in Suppl) shows that tactile-state conditioning improves performance, but in Fig. 4, the relevance maps look visually similar and mainly cover the object region across all steps. Reporting the performance when the map from the first sampling step is reused for later steps, along with how much the map changes across steps, would show how much these updates are needed during sampling. If possible, a comparison with a model trained using a fixed saliency map as M, such as the CLS attention map from DINO, would also show whether the learned selection offers benefits beyond generic object saliency.
    • Preference-controlled global-context injection. Global context appears to play a major role in preference control. In Table 9 (in Suppl), most of the difference between r=0 and r=1 remains with global-context scaling alone, even after the preference loss is removed. The authors explain that the pooled global feature complements local visual evidence, but the choice of this feature for such an important pathway is not validated against other features. For example, a comparison with spatial features in the same pathway would help justify this design. If possible, a comparison with global features from other encoders would also be useful.
Questions for authors
  1. What task properties determine the preferred r, and is there a criterion for choosing r for a new task? (Weakness 1)
  2. How does performance change if the relevance map from the first sampling step is reused for later steps? Could you also report how much the map changes across steps? If possible, I would also like to see a comparison with a model trained using a fixed saliency map as M, such as the CLS attention map from DINO. (Weakness 2-1)
  3. How do the results change if spatial features are used instead of the pooled global feature in the global-context injection pathway? If possible, I would also like to see a comparison with features from other visual encoders. (Weakness 2-2)
7 comments

We sincerely thank the reviewer for the constructive comments, and for recognizing both the importance of task-dependent tactile generation and the practical value of switching between reconstruction- and perception-oriented outputs without retraining.

[W1/Q1] Which task properties determine the preferred rr? How to choose rr for a new task?

Thank you for this valuable suggestion. We agree that performance trends alone do not explain where the preference comes from. To exclude dataset differences, we study two tasks with opposite preferences on the same ObjectsGrasping setting (Fruit Identity prefers low rr; Press–Sliding prefers high rr; Appendix K.4) and build an evidence chain: task requirement → characteristics changed by rr → preservation of task-relevant features → downstream utility.

(1) Task requirement: the two tasks rely on different regions and frequency bands. On all 3,960 real GelSight test frames (792 five-frame windows), we compute attributions of both frozen classifiers in a common sensor field. The central-half attribution mass is 16.0%/22.4% for Press/Sliding, versus 39.7%/42.9% for Fruit Identity on the same windows; restricted to windows that both models classify correctly, the median paired Jensen–Shannon divergence is still 0.310 bit. For frequency, we Haar-decompose the r=0r=0 and r=1r=1 outputs, swap the low band LL and high band HH, and evaluate on the full test set with the frozen classifiers: replacing L0L_0 by L1L_1 lowers Fruit Identity accuracy by 2.83 points, and replacing H0H_0 by H1H_1 raises Press–Sliding accuracy by 13.16 points, while swapping the other band has little effect. This matches the physics of the tasks: fruit identity is determined by the overall shape and deformation at the contact center (low-frequency structure), whereas distinguishing press from slide depends on contact-boundary changes and local texture (high-frequency detail). Classical contact mechanics supports this: under tangential load the contact center sticks while partial slip first appears at the contact edge (the Cattaneo–Mindlin problem; see [1]), as also observed in GelSight shear and slip measurements [2].

(2) What rr changes. Appendix J shows that larger rr adds high-frequency content, consistent with the preference objective in Eq. (13) moving from image-space L1 toward a high-frequency wavelet target. Fixing conditions, initial noise, and schedule in the frozen model, we further find that in deeper blocks the relative residual at the global-context injection points becomes more spatially uniform as rr grows, so the global-context correction extends from a few positions to the whole sensor plane, including the contact periphery.

[W1/Q1] (continued)

(3) Preservation of task-relevant features. We feed paired generated and real tactile inputs at five rr values into each frozen classifier, compute the cosine distance between their spatial features, and weight it with the normalized real-tactile attribution map, giving a task-weighted feature error Et(r)E_t(r); lower means the generated data are closer to real data where the task looks. As rr goes from 0 to 1, EtE_t decreases monotonically for Press and Sliding and increases monotonically for Fruit Identity (new Figure 5; definition in Appendix K.5).

(4) Downstream utility. Appendix K.4 (Table 17) reports that from r=0r=0 to r=1r=1, Fruit Identity Acc/Macro-F1 drops from 71.15/64.52 to 62.00/54.00, while Press–Sliding rises from 50.72/49.86 to 70.84/70.17, in the same direction as (3).

Summary. The preferred rr is determined by two task properties: whether discriminative cues lie at the contact center or periphery, and whether they are low-frequency deformation or high-frequency texture. The high-frequency detail and peripheral correction added by larger rr are what Press–Sliding needs, whereas the paired low-frequency structure needed by Fruit Identity loses fidelity as rr grows.

Choosing rr for a new task (no retraining). (i) Using a task classifier trained on real tactile data, run attribution and band swapping; start from a small r0r_0 (e.g., 0.25) for center/low-frequency tasks and a large r0r_0 (e.g., 0.75) for periphery/high-frequency tasks. This step analyzes only real data. (ii) Evaluate a few coordinates near r0r_0 on a held-out validation set and fix r∗=arg⁡max⁡rScoreval(r)r^*=\arg\max_{r}\mathrm{Score}_{\mathrm{val}}(r) for testing.

Revision: a task-preference paragraph and the task-weighted feature-error figure (Figure 5) in §4.4; Appendix K.5 gives the attribution, band-swap, and feature-error protocols and results, and K.6 the selection procedure.

[1] Johnson, K. L. (1985). Contact Mechanics. Cambridge University Press.

[2] Yuan, W., Li, R., Srinivasan, M. A., & Adelson, E. H. (2015). Measurement of shear and slip with a GelSight tactile sensor. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 304–311).

[W2a/Q2] Is updating the relevance map at every step necessary? Is it better than generic object saliency?

Thank you for the careful observation. Table 8 shows that tactile-state conditioning helps, but not whether per-step updates matter. We added the controls below (original TacQuad frame protocol, r=1r=1, 25 steps).

Table R1-A: Map-update strategy and fixed-saliency controls.

SettingTypeMAE↓PSNR↑SSIM↑DISTS↓FID↓
Per-step M(s)M^{(s)} (full MoDiTac)—6.25527.5780.8930.14751.75
First-step M(0)M^{(0)} reused for 24 stepsinference-time, same model6.24327.4360.8910.15553.33
Fixed DINO saliency mapretrained7.12426.8640.8810.16357.34
Uniform M=1M=1 (Table 2)retrained6.83326.9270.8820.16761.46

First-step reuse and map change. Reusing M(0)M^{(0)} for the remaining 24 steps raises DISTS/FID from 0.147/51.75 to 0.155/53.33, while MAE is essentially unchanged. On the full test set, the step-0 and step-24 maps have Spearman correlation 0.917 and top-20% IoU 0.933, confirming the reviewer's observation that the main object region is largely stable. The remaining change is not random: in the Touch-and-Go example of the new comparison figure in Appendix G, the step-0 map responds to the text background, and by steps 12 and 24 this response fades as high responses concentrate on the probe and contact neighborhood. This small correction yields the DISTS/FID gains.

Why the first-step map needs correction. This follows from the gate g(σ)=clip(1−σ,0,1)g(\sigma)=\mathrm{clip}(1-\sigma,0,1) in Eq. (4). At the first step the noise is maximal and g(σ)≈0g(\sigma)\approx 0, so the queries are barely conditioned on the tactile state and the map mainly reflects visually salient patterns, which may include background text. As the noise decreases, the tactile state takes shape, the gate strengthens tactile conditioning, and the map is corrected toward the actual contact region. The first-step map provides an object-level prior; later steps perform the contact-level correction.

Fixed DINO saliency. We use the mean CLS-to-patch attention of the six last-layer heads of a pretrained DINO ViT-S/8 as a fixed MM and retrain; all five metrics are worse than the full model. DINO's CLS attention captures object-level visual saliency: any visually distinctive object receives high response, whether or not it is in contact, so in cluttered scenes the map spends much of its attention on the background (new Figure 6 in Appendix G: DINO's high responses cover large irrelevant background areas, whereas MoDiTac concentrates on the object and contact neighborhood). The tactile response, in contrast, is determined only by the local contact region; visual saliency is not tactile relevance. Tactile-guided selection targets exactly this gap: the map is conditioned on the current tactile state, and the response difference between relevance and irrelevance queries (Eq. (7)) suppresses regions that are salient but irrelevant to the current touch.

Revision: §4.3 briefly reports these controls; Appendix G gives the full table, map-change statistics, and the DINO–MoDiTac comparison figure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the reviewers and Area Chair for the time and care devoted to evaluating our work. Your valuable comments have helped us improve the paper. We also appreciate your recognition of the value of adapting tactile generation to different downstream needs. Our detailed responses provide additional experiments and analyses; we summarize four shared concerns and the corresponding findings below.

Why tasks prefer different rr values, and how to select rr. Under the same ObjectsGrasping setting, spatial attribution, frequency-band swaps, and task-feature analyses consistently show that Fruit Identity relies more on central, low-frequency deformation, whereas Press–Sliding relies more on peripheral, high-frequency detail. This explains their preference for smaller and larger rr, respectively. For a new task, these cues guide the initial choice of rr, which is then calibrated on a held-out validation set without retraining the generator (new Figure 5; Appendices K.5–K.6).

Physical relevance beyond recognition. We added evaluations using an independent force probe and the AnyTouch encoder. Among the compared generators, r=1r=1 achieves the lowest overall force-estimation error (3.987 N) and the lowest tactile-feature distances (paired cosine distance 0.033; Feature MMD² 0.092). Thus, preference control also affects force-related information accessible to downstream models. Full contact dynamics and closed-loop robot utility remain to be evaluated (new Table 4; Appendices K.7–K.8 and L).

Roles of vision and text, and input selection. Removing text on TacQuad changes the r=0r=0 MAE only from 4.151 to 4.197, showing that vision supplies most paired tactile evidence; reliable object or material descriptions add a semantic prior. We therefore use vision by default, a generic prompt for contact-state and force-related tasks, and reliable descriptions when available for material or category tasks. Inputs determine what information is available, while rr controls the generation preference (revised §4.2; Appendices E and K.9).

Extension to unified models and temporal backbones. We integrated both mechanisms into Show-o2 and Lance after adapting them to tactile video generation. On each backbone, the r=0r=0 plugin improves all five generation metrics over its no-plugin counterpart, while sweeping rr reproduces the reconstruction–perception trade-off. This supports compatibility with joint video generation; physical coherence of generated sequences still requires dedicated evaluation (Appendix L).

We thank the reviewers and Area Chair again for their valuable time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.