acceptodds
Withdrawn from ICLR 2027

Cortically Inspired Visuo-Tactile Alignment for Cross-Embodiment Tactile Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Observing touch on another person's body can recruit somatosensory sensations in observers, providing a striking example of how the brain translates visual events into bodily states. Neuroscientific evidence suggests that this visuo-tactile resonance relies on structured alignment between visual and somatosensory neural representations, yet the computational instantiation and functional role of such alignment remain unclear. Here we introduce MTNet, a cortically inspired dual-stream framework that learns visual-to-tactile prediction under semantic, distributional, and geometric alignment constraints. Given monocular RGB observations, MTNet predicts force fields across 1,140 taxels on a robotic hand at 1-mm spatial resolution. Compared with reconstruction-driven models, the alignment constraints substantially improve spatiotemporal prediction and reshape visual representations to match the tactile manifold geometry, thereby simplifying the mapping from vision to touch. By aligning paired visual observations of human and robotic hands within the learned visuo-tactile space, the physical robot generates corresponding tactile states from observed human touch, achieving cross-domain transfer without human tactile labels. These results offer a computational account of how the cross-modal alignment supports visuo-tactile resonance across embodiments, translating neuroscientific findings into a design principle for tactile inference in embodied artificial systems. Code and videos are available in the supplementary materials and will be publicly released upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Visuo-tactile alignment​, Mirror Touch, Tactile prediction, Cortical integration, Neuro-inspired computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pTj8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work is inspired by the mirror-touch phenomenon in neuroscience and the finding that an aligned mapping exists between the visual and somatosensory cortices, and investigates generating corresponding tactile signals from visual observations of human or robotic hands. Specifically, this work proposes MTNet, which predicts the 3D force field over 1,140 taxels across 11 sensors on a dexterous hand from RGB image sequences. It further proposes AMTNet, which enables human-hand images to be decoded by MTNet into paired robotic tactile signals.

Critical strengths
  1. The studied problem is novel with a well-motivated neuroscience grounding.
  2. The model architecture handles irregular taxel layouts well.
Critical weaknesses

The key weaknesses of this work are (1) the central claim that cortically inspired multi-level alignment drives the improvement is not supported by the experiments, and (2) the experimental evaluation is insufficient to validate the effectiveness and generalization of the proposed methods.

Specifically:

  1. The "Unaligned MTNet" baseline in Table 1 appears to be a flawed CVAE. During training, the visual branch samples from the posterior conditioned on the ground-truth tactile tokens, while inference samples from the visual prior. Without any KL term, the prior is never aligned with the posterior. Therefore, the performance collapse at inference is an expected train-inference mismatch rather than evidence that reconstruction-driven learning is insufficient. More appropriate reconstruction baselines (e.g., the deterministic regression with the same encoder and decoder and a standard CVAE with KL terms but without your cross-modal alignment KL terms) are necessary.
  2. The ablation in Table 1 does not support the multi-level alignment design and the claim "... that alignment constraints directly improve tactile prediction ...". Removing semantic or geometric alignment causes only marginal changes on most metrics, while only removing distributional alignment (which aligns the inference-time visual prior with the tactile posterior) leads to collapse. This suggests that the improvement mainly comes from fixing the CVAE prior, rather than from the cortically inspired multi-level alignment.
  3. The neuroscience claims are overstated. The paper claims to "offer a computational account" of visuo-tactile resonance, but provides no comparison with neural or behavioral data (for example, comparison of similarity between internal representation of the model and fMRI or electrophysiological data). The analogies also remain metaphorical. For example, the authors claim "motivated by the parallel processing of visual and somatosensory neural circuits and their subsequent integration in the parietal cortex" (line 157-158), "a shared parietal-like latent space" (Figure 2), but encoding two modalities separately and then map them to a shared latent space is the most common structure in multi-modal learning. What is the difference between calling shared latent space "the parietal cortex" and these multi-modal alignment works? They also claim the feature-wise gate is "Motivated by inhibitory modulation in biological somatosensory processing" (line 204-205), but this is actually just a simple Sigmoid gating, and the paper has not demonstrated that it exhibits lateral inhibition behavior (competition or inhibition between neighboring units) in the real inhibitory modulation [1]. Meanwhile, the "mirror-touch" demonstration relies on a pre-defined controller triggered by a 0.2 N force threshold. This demonstration can only demonstrate that the force predicted from the hand image exceeds 0.2 N, and cannot be used as evidence that the model "reproduced the mirror touch phenomenon". The methodological contribution of this work behind these neuroscience claims is very limited.
  4. The practical value of the studied setting is limited. All experiments are restricted to tactile prediction on a fully open robotic or human hand that is passively touched (as acknowledged in the Limitations: "our robotic hand is passively touched in the experiments"). However, most hand contacts in practice occur during complex grasping and manipulation, where the hand pose varies substantially and the contact regions are often severely occluded. Several concurrent works [2,3] (no direct comparison is required) study predicting dense tactile signals from vision during manipulation, which is considerably more challenging and practically meaningful in terms of both methodological contribution and application value. The authors should similarly discuss whether the proposed paradigm can be extended to active manipulation scenarios.
  5. The baseline comparison is insufficient. The only external baseline is a re-implemented and substantially modified tactile diffusion model with very low S-CosSim (0.063). Simpler and more direct baselines, such as CNN/ViT regression, visuo-tactile pre-training encoders.
  6. The evaluation metrics may inflate the reported performance. All five metrics are newly defined by the authors, and several assign perfect scores when no contact occurs (T-IoU, W-F1, and S-CCC are set to 1), which can substantially inflate the averages on highly sparse tactile data.
  7. The human-to-robot transfer lacks quantitative evaluation. The reported Bhattacharyya distance of the gate and the CKA similarity do not measure the accuracy of the predicted tactile signals. Since each human image in the Hand dataset is paired with a robot image that has ground-truth tactile measurements, the transfer error can be directly quantified, but it is not reported. Moreover, the method requires pose-matched human-robot image pairs, so the claim of "label-free" transfer is overstated. Transfer baselines (e.g., directly applying MTNet to human images, or image-to-image translation) are also missing.
  8. The generalization ability is not validated. The VT dataset contains only 500 episodes collected with a single robotic hand and a single camera viewpoint, and is split at the episode level, so the test distribution is highly similar to the training distribution. Generalization experiments to new viewpoints, lighting conditions, and objects and the quantitative results are not shown.
  9. Key implementation details are missing. The method involves 11 loss terms for MTNet and 5 for AMTNet, but none of the loss weights in Eq. 7 and Eq. 9 are reported, and no hyper-parameter experiments are provided.

Overall, although the studied problem is interesting, the main claims of this paper are not sufficiently supported by the current experiments, and both the baselines and the evaluation require substantial improvement.

[1] Fan, Linlin Z., et al. "All-optical electrophysiology reveals the role of lateral inhibition in sensory processing in cortical layer 1." Cell 180.3 (2020): 521-535.

[2] Patil, Soham, et al. "NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation." arXiv preprint arXiv:2606.24450 (2026).

[3] Zhou, Danyan, et al. "TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation." arXiv preprint arXiv:2609.20414 (2026).

Questions for authors

(1) What concrete computational account of visuo-tactile resonance the proposed method offers?

(2) How the neuroscience findings explicitly guide the design of specific components, beyond naming analogie?

(3) What the main methodological contributions of this work are?

7 comments
AuthorsOpenReview ↗

Response to Reviewer pTj8 (part 1)

We sincerely appreciate your careful review. Please see our point-by-point response below.

Response to Weakness

W1: The "Unaligned MTNet" baseline in Table 1 appears to be a flawed CVAE. During training, the visual branch samples from the posterior conditioned on the ground-truth tactile tokens, while inference samples from the visual prior. Without any KL term, the prior is never aligned with the posterior. Therefore, the performance collapse at inference is an expected train-inference mismatch rather than evidence that reconstruction-driven learning is insufficient. More appropriate reconstruction baselines (e.g., the deterministic regression with the same encoder and decoder and a standard CVAE with KL terms but without your cross-modal alignment KL terms) are necessary.

A1: Thank you for this insightful suggestion. We have corrected the unaligned MTNet baseline to include two intra-modal KL terms, and we also corrected the baseline without distributional alignment (MTNet w/o distributional alignment) to similarly contain two intra-modal KL terms. The updated results in Table 1 are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Unaligned MTNet0.01100.55190.85970.70610.9470
MTNet w/o distributional alignment0.01070.56070.85800.72630.9490
MTNet w/o semantic alignment0.01010.59210.86690.73640.9524
MTNet w/o geometric alignment0.00980.60780.87100.75730.9553
Aligned MTNet0.01020.60970.88140.79200.9664

The corrected unaligned MTNet baseline now exhibits basic visuo-tactile inference capability instead of collapsing, and each alignment constraint contributes to further performance gains. Applying all three alignment constraints yields the best overall performance.

We also updated the manifold analysis results for the improved unaligned MTNet (Fig. 5), showing that reconstruction-driven training alone improves the CKA score by approximately 0.15, and alignment constraints further boost it by around 0.10 over the unaligned baseline, making the visual and tactile manifolds more consistent. Together, these results demonstrate that alignment constraints effectively enhance model performance rather than serving as a prerequisite for training. This results strengthens the paper's conclusions, and the corresponding revisions have been made in the manuscript.

W2: The ablation in Table 1 does not support the multi-level alignment design and the claim "... that alignment constraints directly improve tactile prediction ...". Removing semantic or geometric alignment causes only marginal changes on most metrics, while only removing distributional alignment (which aligns the inference-time visual prior with the tactile posterior) leads to collapse. This suggests that the improvement mainly comes from fixing the CVAE prior, rather than from the cortically inspired multi-level alignment.

A2: Thank you for this comment. Based on the corrected results in A1, distributional, semantic, and geometric alignment complementarily improve the baseline performance, rather than being prerequisites for training. To further ablate the effect of alignment constraints, we trained AMTNet variants built upon the unaligned MTNet baseline and quantitatively evaluated their robot-to-human transfer performance using the corresponding robotic tactile signals as ground truth. The results are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Unaligned AMTNet0.01130.23950.76150.26790.8346
AMTNet w/o distributional alignment0.01100.35140.76830.37630.8847
AMTNet w/o semantic alignment0.01100.42730.76720.47080.8987
AMTNet w/o geometric alignment0.01100.40850.76480.46620.9176
Aligned AMTNet0.01070.44100.78980.52970.9244

It can be observed that alignment constraints also uniformly improve cross-embodied tactile inference, likely by reinforcing the correlation between the robotic visual and tactile spaces, thereby easing the integration of human visual observations. These results indicate that the three alignment constraints operate in a complementary manner, and the absence of any one of them degrades both the visuo-tactile mapping and transfer performance.

AuthorsOpenReview ↗

Response to Reviewer pTj8 (part 2)

This is the continued response to the remaining weakness 3.

Response to Weakness

W3 (a): The neuroscience claims are overstated. The paper claims to "offer a computational account" of visuo-tactile resonance, but provides no comparison with neural or behavioral data (for example, comparison of similarity between internal representation of the model and fMRI or electrophysiological data)

A3: We acknowledge that the present work lacks direct comparison with neural data. For computational flexibility, we drew more on principles from neuroscience rather than incorporating anatomical structures. We have revised the manuscript to avoid expressions such as "computational account/realization" and strengthened the discussion in the Limitations section.

W3 (b): The analogies also remain metaphorical. For example, the authors claim "motivated by the parallel processing of visual and somatosensory neural circuits and their subsequent integration in the parietal cortex" (line 157-158), "a shared parietal-like latent space" (Figure 2), but encoding two modalities separately and then map them to a shared latent space is the most common structure in multi-modal learning. What is the difference between calling shared latent space "the parietal cortex" and these multi-modal alignment works?

A4: The original intention of this passage was to differentiate our method from pure regression models, as learning a direct visuo-to-tactile mapping does not necessitate separate visual and tactile pathways, such as the deterministic ResNet-50 baseline included in A8. We adopted a dual-stream architecture to prepare for enforcing cross-modal consistency in the latent space. The term "parietal-like" has been removed to enhance clarity.

W3 (c): They also claim the feature-wise gate is "Motivated by inhibitory modulation in biological somatosensory processing" (line 204-205), but this is actually just a simple Sigmoid gating, and the paper has not demonstrated that it exhibits lateral inhibition behavior (competition or inhibition between neighboring units) in the real inhibitory modulation.

A5: Thank you for this comment. Lateral inhibition in biological somatosensory systems enhances perceptual contrast, a principle we considered relevant to the sparse nature of contact events. We therefore adopted a sigmoid gate to emphasize informative feature channels. We agree that the current implementation lacks direct biological grounding, and we have accordingly removed all references to "inhibitory modulation" from the revised manuscript.

W3 (d): Meanwhile, the "mirror-touch" demonstration relies on a pre-defined controller triggered by a 0.2 N force threshold. This demonstration can only demonstrate that the force predicted from the hand image exceeds 0.2 N, and cannot be used as evidence that the model "reproduced the mirror touch phenomenon". The methodological contribution of this work behind these neuroscience claims is very limited.

A6: We appreciate this concern. To clarify the robotic mirror touch experiment, the revised Fig. 6d presents the predicted force curves by AMTNet in real time from human-hand images, along with the motion trajectory of the robotic response. The predicted force is generated consistently during contact rather than as isolated pulses, and it remains synchronized with the movement of the robotic finger. Importantly, this prediction is achieved without tactile supervision from the human hand, and we believe that such artificial implementation of vicarious touch remains rare.

AuthorsOpenReview ↗

Response to Reviewer pTj8 (part 3)

This is the continued response to the remaining weakness 4-5.

Response to Weakness

W4: The practical value of the studied setting is limited. All experiments are restricted to tactile prediction on a fully open robotic or human hand that is passively touched (as acknowledged in the Limitations: "our robotic hand is passively touched in the experiments"). However, most hand contacts in practice occur during complex grasping and manipulation, where the hand pose varies substantially and the contact regions are often severely occluded. Several concurrent works [2,3] (no direct comparison is required) study predicting dense tactile signals from vision during manipulation, which is considerably more challenging and practically meaningful in terms of both methodological contribution and application value. The authors should similarly discuss whether the proposed paradigm can be extended to active manipulation scenarios.

A7: We appreciate the concern regarding practical value. We first clarify that the primary goal of this paper is cross-embodiment tactile inference rather than visual-to-tactile inference alone. Specifically, we aim to simulate a cognitive phenomenon in an artificial system: observing touch on another person elicits tactile sensations in oneself, even without direct somatosensory experience. To our knowledge, computational models for such cross-embodiment tactile inference remain scarce.

We acknowledge the practical significance of the two concurrent works mentioned. However, our setting differs from theirs in at least two major respects: (1) our tactile sensing is denser. NoContactNoWorries (Patil et al.) and TouchSight (Zhou et al.) employ binary contact labels at 4 fingertips and 275 trusted taxels per hand, respectively, whereas we use 1,140 independent taxels across the palm surface at 1-mm spatial resolution, deliberately reflecting the pronounced modality gap between human-like vision and tactile sensing. (2) our framework supports cross-domain transfer. While those works primarily learn tactile inference through paired visuo-tactile data, we investigate how to transfer inference capability from a robotic hand to a human hand solely through visual alignment, without requiring tactile labels on the human side.

We adopted a passive, open-hand touch setup as a controlled platform for two reasons: (1) It allows us to uniformly stimulate all sensing regions and maintain balanced contact distribution across the hand. (2) It avoids spurious contacts caused by robotic teleoperation imperfections, where visual contact may appear without actual force interaction. We agree on the importance of application value and plan to extend this paradigm to active manipulation in future work. For instance, AMTNet can serve as a domain adapter integrated into closed-loop robotic manipulation. A promising future direction is to leverage AMTNet’s transfer principle to generate tactile states from manipulation videos of robots with varying embodiments, thereby reducing the cost of collecting paired visuo-tactile data.

We have cited these related works and discussed the limitations of current hardware setup in the revised Limitations Section.

W5: The baseline comparison is insufficient. The only external baseline is a re-implemented and substantially modified tactile diffusion model with very low S-CosSim (0.063). Simpler and more direct baselines, such as CNN/ViT regression, visuo-tactile pre-training encoders.

A8: Following this suggestion, we added a deterministic ResNet-50 regression baseline. It was adapted to our task with two minimal modifications: (1) the first convolutional layer was expanded to accept a stacked input of K frames, and (2) the classification head was replaced with a multilayer perceptron that directly regresses the taxel-level force field under the same valid-taxel masking as MTNet. All other training details remained identical to those of MTNet. Thus, the baseline shares the same visual input, target representation, and reconstruction supervision with MTNet, differing only in the absence of any cross-modal structure. The results are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Deterministic ResNet-500.05000.60290.86890.37170.5887
Unaligned MTNet0.01100.55190.85970.70610.9470
Aligned MTNet0.01020.60970.88140.79200.9664

The prediction snapshots are also updated in Fig. 4. It shows that the regression baseline tends to produce signals in non-contact regions and frames, and its metric scores fall behind those of MTNet, indicating that pure numerical regression struggles to bridge the substantial modality gap between vision and touch. This additional baseline strengthens the completeness of our experimental comparison and has been incorporated into the revised manuscript.

Reviewer_rk9FOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Inspired by visuo-tactile resonance in the human cortex, this paper proposes a method for predicting tactile signals from visual observations. The proposed model learns to predict contact locations and three-dimensional force fields from images of a robotic hand using paired visual and tactile data. The authors then extend the model to human-hand images by aligning paired human and robot observations, without requiring human tactile labels. Experiments evaluate tactile prediction on the robotic hand and demonstrate that touch observed on a human hand can trigger corresponding responses in a physical robotic hand.

Critical strengths

The proposed approach to predicting tactile signals from visual input has some novelty and may offer useful insights for future research on tactile perception.

Critical weaknesses

A key weakness concerns the assessment of prediction accuracy on human-hand images. The authors should explain how they ensured consistency when manually reproducing the robot hand’s contact poses with a human hand. If pose matching relied only on subjective imitation, mismatched image pairs could affect the tactile predictions and the reliability of their evaluation.

Questions for authors

1.We suggest that the authors provide additional details on the types of contact interactions represented in the robot-hand and paired human-hand datasets, such as tapping, pressing, sliding, or grasping. Please also report the number of samples for each type and clarify whether the human-hand dataset covers all interaction types in the robot-hand dataset.

2.We suggest that the authors provide additional details on how consistency was ensured when manually reproducing the same contact poses with a human hand, including the matching criteria and quality checks for contact locations and hand poses. If no quantitative consistency check was performed, please discuss how pairing errors might affect training and the interpretation of the transfer results.

2 comments
AuthorsOpenReview ↗

Response to Reviewer rk9F

We sincerely thank you for taking the time to review. Please see our detailed response to the issues you have raised.

Response to Weakness

W: A key weakness concerns the assessment of prediction accuracy on human-hand images. The authors should explain how they ensured consistency when manually reproducing the robot hand’s contact poses with a human hand. If pose matching relied only on subjective imitation, mismatched image pairs could affect the tactile predictions and the reliability of their evaluation.

A1: We agree that the assessment of the frame pairing between robot and human hands is critical. Accordingly, in our response to Question 2 (A3), we detail the data collection protocol for the Hand dataset and provide supplementary experimental results on pairing errors.

Response to Questions

Q1: We suggest that the authors provide additional details on the types of contact interactions represented in the robot-hand and paired human-hand datasets, such as tapping, pressing, sliding, or grasping. Please also report the number of samples for each type and clarify whether the human-hand dataset covers all interaction types in the robot-hand dataset.

A2: We agree that dataset composition matters. During collection, we used different objects (human fingers, blue and red markers, a green cube, and a brown cylinder) to naturally and evenly touch the 11 sensing regions on the robot hand surface. Each episode includes multiple contact events, involving simultaneous contacts from multiple objects across regions. Four main contact types were identified: tapping and pinching by human fingers, and pressing and poking by the objects. The Hand dataset consists of 100 episodes selected from the VT dataset to represent these contact types. Below are the episode counts for each contact type in the two dataset:

TappingPinchingPressingPoking
VT dataset~468~170~88~117
Hand dataset~95~16~38~39

Each episode contains multiple events across various contact types. We provide example episodes in the supplementary material** and will release all datasets. These updates will be included in the revised manuscript.

Q2: We suggest that the authors provide additional details on how consistency was ensured when manually reproducing the same contact poses with a human hand, including the matching criteria and quality checks for contact locations and hand poses. If no quantitative consistency check was performed, please discuss how pairing errors might affect training and the interpretation of the transfer results.

A3: Thank you for raising this concern. Since our model infers touch from the motion context contained in the continuous RGB frames, we focused on ensuring consistency in object entry timing, motion trajectory, contact timing, and contact location when reproducing contact poses with the human hand. Every pair was inspected frame-by-frame during collection, and paired examples are provided in the supplementary material.

We agree that evaluating pairing consistency is essential. To quantify the sensitivity of the transfer to pairing errors, we conducted a corruption ablation by introducing mismatched samples at different ratios (ρ=25%\rho=25\%, 50%50\%) into the AMTNet training data. Each mismatched sample replaced the robot frame stack with one randomly drawn from another episode. Using corresponding robot tactile signals as ground truth, we evaluated AMTNet performance under these perturbations. The results are shown below:

NRMSE ↓S-CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
ρ\rho=0%0.01070.44100.78980.52970.9244
ρ\rho=25%0.01110.38700.75080.49130.9107
ρ\rho=50%0.01160.02470.73680.23770.8154

It can be seen that AMTNet performance degrades substantially as the corruption ratio increases, indicating that the learned robot-to-human transfer relies on reliable pairings. This results strengthen our evaluation and will be discussed in the revised manuscript.

I thank the authors for their responses and clarifications, which have addressed my questions.

Reviewer_A5AdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MTNet, which predicts force maps on a robotic hand from RGB images, training separate visual and tactile pathways with additional distributional, semantic and geometric alignment losses inspired by visuo-tactile alignment in the human brain. An extension, AMTNet, maps images of a touched human hand into the same space using paired human-robot images, so that observed human touch triggers reflexes on the robot hand.

Critical strengths
  • Inferring touch from vision, and transferring it across embodiments, is an interesting problem.
  • The supplementary code, including the baseline implementation, makes the method reproducible and easy to inspect.
Critical weaknesses
  • The setup is very constrained, with a single fixed camera looking at the hand from the top. The hand is flat and open, backgrounds are clean, and the contacts are just pokes on the passive hand. It's not clear how this would be practically useful, e.g., during manipulation, where the hand moves and contacts are occluded. And while the method predicts 3D forces, the evaluation mostly captures contact location and timing. I actually don't see how shear or slippage could be sensed without seeing it, and whether 5 frames are enough for any temporal correlation.
  • The evidence that alignment is what makes the method work is weak. The only baseline essentially fails at contact detection, and removing the semantic or geometric terms barely changes the results. The large gap only appears when the KL terms are removed.
Questions for authors
  1. The robot hand has four fingers, while the human hand has five. How is the correspondence defined in the Hand dataset, and what is predicted when the fifth human finger is touched?
  2. Could you report errors separately for normal and shear forces? How can shear or slippage be inferred from vision alone, and are 5 frames enough? An ablation on the number of frames would help.
  3. With only a top view of the hand, how would this be used in practice? How does the method perform with a different viewpoint or hand pose, cluttered backgrounds, or lighting changes?
  4. Figure 2 is hard to parse. Consider separating the architecture from the eleven losses.
  5. From the introduction, I expected human-hand transfer to come for free, but AMTNet needs an additional, essentially paired dataset (a person reproducing each robot touch pose) and further training. Please make this clear in the introduction.
  6. The semantic and geometric terms don't seem that strong, so why are they needed?
5 comments
AuthorsOpenReview ↗

Response to Reviewer A5Ad (part 1)

We sincerely thank you for your feedback of our work. Below, we would like to respond to each of your concerns point by point.

Response to Weaknesses

W1: The setup is very constrained, with a single fixed camera looking at the hand from the top. The hand is flat and open, backgrounds are clean, and the contacts are just pokes on the passive hand. It's not clear how this would be practically useful, e.g., during manipulation, where the hand moves and contacts are occluded. And while the method predicts 3D forces, the evaluation mostly captures contact location and timing. I actually don't see how shear or slippage could be sensed without seeing it, and whether 5 frames are enough for any temporal correlation.

A1: Thank you for the concern about practicality. We would first like to clarify that the primary goal of this work is to realize cross-embodied tactile inference, rather than robotic manipulation. That is, when observing another person being touched, one can generate corresponding tactile sensations even without direct somatosensory experience. To our knowledge, computational implementations of such a vicarious touch phenomenon remain scarce. Using a passive, open-hand setup as a controlled platform provides two advantages: (1) it uniformly stimulates all sensing regions, ensuring balanced contact distribution, and (2) it avoids spurious contacts caused by imperfect robot teleoperation, where visual contact occurs without actual force interaction.

We agree on the importance of practical value. In the revised Limitation section, we discuss the limitations of the current setup and plan to extend our method to more scenarios in future work. For example, the transfer framework of AMTNet can be leveraged to generate self-tactile states from robot manipulation videos of different morphologies, thereby reducing the cost of collecting visuo-tactile paired data. In our response to Question 2 (A4), we discuss the error of three-dimensional force components in detail and report ablation results on the number of input frames.

AuthorsOpenReview ↗

Response to Reviewer A5Ad (part 2)

This is the continued response to the remaining weaknesses 2.

Response to Weaknesses

W2: The evidence that alignment is what makes the method work is weak. The only baseline essentially fails at contact detection, and removing the semantic or geometric terms barely changes the results. The large gap only appears when the KL terms are removed.

A2: Thank you for raising this concern. We identified that the original unaligned MTNet collapsed due to the absence of two intra-modal KL terms (items 6-7 in Fig. 2e) that align the prior with the posterior within each modality. Therefore, we separated the two intra-modal KL terms from the alignment constraints, yielding seven reconstruction losses and four cross-modal alignment terms, and re-ran the ablation with the corrected unaligned MTNet. The results are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Unaligned MTNet0.01100.55190.85970.70610.9470
MTNet w/o distributional alignment0.01070.56070.85800.72630.9490
MTNet w/o semantic alignment0.01010.59210.86690.73640.9524
MTNet w/o geometric alignment0.00980.60780.87100.75730.9553
Aligned MTNet0.01020.60970.88140.79200.9664

The corrected unaligned baseline now performs basic visuo-tactile inference but remains markedly worse than the aligned MTNet. Each alignment constraint complementarily improves performance. This indicates that cross-modal alignment is an effective enhancement strategy rather than a training prerequisite. The updated manifold analysis (Fig. 5) further supports this: the unaligned MTNet raises the CKA score by 0.15 over the initial state, and the aligned MTNet further increases it by 0.10, yielding tighter coupling between visual and tactile manifolds.

In addition, we introduced a deterministic ResNet-50 regression baseline. We modified only the input layer and the output head to handle our task, while keeping all other training settings identical to those of MTNet (see revised Appendix A.5). Thus, this baseline shares the same visual input, target representation, and reconstruction supervision with MTNet, differing only in lacking any cross-modal structure. The comparative results are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Deterministic ResNet-500.05000.60290.86890.37170.5887
Unaligned MTNet0.01100.55190.85970.70610.9470
Aligned MTNet0.01020.60970.88140.79200.9664

Prediction snapshots were also updated in Fig. 4. The regression ResNet-50 baseline tends to produce signals in non-contact regions and frames, and its metric scores fall behind those of MTNet, indicating that pure numerical regression struggles to bridge the substantial modality gap between vision and touch.

To further ablate the contribution of alignment constraints, we trained AMTNet variants built upon unaligned baselines and quantitatively evaluated their robot-to-human transfer performance using the corresponding robotic tactile signals as ground truth. The results are shown below:

NRMSE ↓CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
Unaligned AMTNet0.01130.23950.76150.26790.8346
AMTNet w/o distributional alignment0.01100.35140.76830.37630.8847
AMTNet w/o semantic alignment0.01100.42730.76720.47080.8987
AMTNet w/o geometric alignment0.01100.40850.76480.46620.9176
Aligned AMTNet0.01070.44100.78980.52970.9244

It can be seen that alignment constraints also uniformly improve cross-embodiment tactile inference, and removing any single constraint degrades performance. This may be because alignment strengthens the correlation between the robotic visual and tactile spaces, thereby easing the establishment of mappings between human visual observations and robotic tactile. These results collectively indicate that the three alignment constraints complementarily contribute to both visuo-tactile mapping and transfer performance. These additions have been incorporated into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer A5Ad (part 3)

This is the continued response to the remaining questions 1-2.

Response to Questions

Q1: The robot hand has four fingers, while the human hand has five. How is the correspondence defined in the Hand dataset, and what is predicted when the fifth human finger is touched?

A3: Indeed, the robotic hand has only four fingers. We naturally mapped them to the first four human fingers (thumb, index, middle, ring), excluding the little finger, consistent with their original designations. Moreover, the robotic hand differs markedly from a human hand in size and color, making it a suitable subject for studying cross-embodiment tactile inference.

The trained AMTNet produced no tactile predictions when the little finger was touched. To ensure this robustness, we specifically collected negative samples with no valid contact (~29 episodes). These negative episodes involved movement and occlusion unrelated to valid contact events. For these inputs, AMTNet correctly remained silent. We attribute this to the alignment constraint between feature ZZ and contact probability c^\hat{c} (Fig. 3), which encode both the contact region and contact status.

Q2: Could you report errors separately for normal and shear forces? How can shear or slippage be inferred from vision alone, and are 5 frames enough? An ablation on the number of frames would help.

A4: Following this suggestion, we report the prediction errors separately for the three force components along the X, Y, and Z-axes as follows:

X-axis (-12.7 N, 12.7 N)Y-axis (-12.7 N, 12.7 N)Z-axis (0, 25.5 N)
RMSE0.099 N0.101 N0.309 N

It shows tha the normal force (Z-axis) error is the largest, likely due to varying magnitude ranges across axes. Our model predicts tactile forces primarily from the motion context within the continuous K-frame sequence. Due to unavoidable visual occlusion, distinguishing static force variations, including the mentioned shear forces, is challenging with only RGB input. In the current passive contact scenarios, shear force variations remain relatively small, while the normal force spans a much wider range, leading to larger prediction errors. Accordingly, we acknowledge in the Limitations Section that the current method is more reliable in estimating contact locations and coarse force magnitudes, which is also consistent with the coarse character of vicarious tactile perception in humans.

The choice of K was determined via the following ablation study:

NRMSE ↓S-CosSim ↑S-CCC ↑T-IoU ↑W-F1 ↑
K=30.01080.52480.86320.78780.9646
K=50.01020.60970.88140.79200.9664
K=70.01000.56830.86040.71760.9481

The results indicate that K=5 yields the best overall performance. In the supplementary material, we provide data samples showing that 5 consecutive frames form an appropriate window capturing object emergence, approach, and eventual contact. Since the visual encoder applies weighted averaging across K frames without encoding temporal positions explicitly, K=3 offers insufficient motion evidence, while K=7 risks diluting critical frames. Balancing metric performance and computational cost, we set K=5. These results will be included in the revised appendix.

Comments on the paper

1 comment
Area_Chair_TrG2OpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.