- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work is inspired by the mirror-touch phenomenon in neuroscience and the finding that an aligned mapping exists between the visual and somatosensory cortices, and investigates generating corresponding tactile signals from visual observations of human or robotic hands. Specifically, this work proposes MTNet, which predicts the 3D force field over 1,140 taxels across 11 sensors on a dexterous hand from RGB image sequences. It further proposes AMTNet, which enables human-hand images to be decoded by MTNet into paired robotic tactile signals.
Critical strengths
- The studied problem is novel with a well-motivated neuroscience grounding.
- The model architecture handles irregular taxel layouts well.
Critical weaknesses
The key weaknesses of this work are (1) the central claim that cortically inspired multi-level alignment drives the improvement is not supported by the experiments, and (2) the experimental evaluation is insufficient to validate the effectiveness and generalization of the proposed methods.
Specifically:
- The "Unaligned MTNet" baseline in Table 1 appears to be a flawed CVAE. During training, the visual branch samples from the posterior conditioned on the ground-truth tactile tokens, while inference samples from the visual prior. Without any KL term, the prior is never aligned with the posterior. Therefore, the performance collapse at inference is an expected train-inference mismatch rather than evidence that reconstruction-driven learning is insufficient. More appropriate reconstruction baselines (e.g., the deterministic regression with the same encoder and decoder and a standard CVAE with KL terms but without your cross-modal alignment KL terms) are necessary.
- The ablation in Table 1 does not support the multi-level alignment design and the claim "... that alignment constraints directly improve tactile prediction ...". Removing semantic or geometric alignment causes only marginal changes on most metrics, while only removing distributional alignment (which aligns the inference-time visual prior with the tactile posterior) leads to collapse. This suggests that the improvement mainly comes from fixing the CVAE prior, rather than from the cortically inspired multi-level alignment.
- The neuroscience claims are overstated. The paper claims to "offer a computational account" of visuo-tactile resonance, but provides no comparison with neural or behavioral data (for example, comparison of similarity between internal representation of the model and fMRI or electrophysiological data). The analogies also remain metaphorical. For example, the authors claim "motivated by the parallel processing of visual and somatosensory neural circuits and their subsequent integration in the parietal cortex" (line 157-158), "a shared parietal-like latent space" (Figure 2), but encoding two modalities separately and then map them to a shared latent space is the most common structure in multi-modal learning. What is the difference between calling shared latent space "the parietal cortex" and these multi-modal alignment works? They also claim the feature-wise gate is "Motivated by inhibitory modulation in biological somatosensory processing" (line 204-205), but this is actually just a simple Sigmoid gating, and the paper has not demonstrated that it exhibits lateral inhibition behavior (competition or inhibition between neighboring units) in the real inhibitory modulation [1]. Meanwhile, the "mirror-touch" demonstration relies on a pre-defined controller triggered by a 0.2 N force threshold. This demonstration can only demonstrate that the force predicted from the hand image exceeds 0.2 N, and cannot be used as evidence that the model "reproduced the mirror touch phenomenon". The methodological contribution of this work behind these neuroscience claims is very limited.
- The practical value of the studied setting is limited. All experiments are restricted to tactile prediction on a fully open robotic or human hand that is passively touched (as acknowledged in the Limitations: "our robotic hand is passively touched in the experiments"). However, most hand contacts in practice occur during complex grasping and manipulation, where the hand pose varies substantially and the contact regions are often severely occluded. Several concurrent works [2,3] (no direct comparison is required) study predicting dense tactile signals from vision during manipulation, which is considerably more challenging and practically meaningful in terms of both methodological contribution and application value. The authors should similarly discuss whether the proposed paradigm can be extended to active manipulation scenarios.
- The baseline comparison is insufficient. The only external baseline is a re-implemented and substantially modified tactile diffusion model with very low S-CosSim (0.063). Simpler and more direct baselines, such as CNN/ViT regression, visuo-tactile pre-training encoders.
- The evaluation metrics may inflate the reported performance. All five metrics are newly defined by the authors, and several assign perfect scores when no contact occurs (T-IoU, W-F1, and S-CCC are set to 1), which can substantially inflate the averages on highly sparse tactile data.
- The human-to-robot transfer lacks quantitative evaluation. The reported Bhattacharyya distance of the gate and the CKA similarity do not measure the accuracy of the predicted tactile signals. Since each human image in the Hand dataset is paired with a robot image that has ground-truth tactile measurements, the transfer error can be directly quantified, but it is not reported. Moreover, the method requires pose-matched human-robot image pairs, so the claim of "label-free" transfer is overstated. Transfer baselines (e.g., directly applying MTNet to human images, or image-to-image translation) are also missing.
- The generalization ability is not validated. The VT dataset contains only 500 episodes collected with a single robotic hand and a single camera viewpoint, and is split at the episode level, so the test distribution is highly similar to the training distribution. Generalization experiments to new viewpoints, lighting conditions, and objects and the quantitative results are not shown.
- Key implementation details are missing. The method involves 11 loss terms for MTNet and 5 for AMTNet, but none of the loss weights in Eq. 7 and Eq. 9 are reported, and no hyper-parameter experiments are provided.
Overall, although the studied problem is interesting, the main claims of this paper are not sufficiently supported by the current experiments, and both the baselines and the evaluation require substantial improvement.
[1] Fan, Linlin Z., et al. "All-optical electrophysiology reveals the role of lateral inhibition in sensory processing in cortical layer 1." Cell 180.3 (2020): 521-535.
[2] Patil, Soham, et al. "NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation." arXiv preprint arXiv:2606.24450 (2026).
[3] Zhou, Danyan, et al. "TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation." arXiv preprint arXiv:2609.20414 (2026).
Questions for authors
(1) What concrete computational account of visuo-tactile resonance the proposed method offers?
(2) How the neuroscience findings explicitly guide the design of specific components, beyond naming analogie?
(3) What the main methodological contributions of this work are?
7 comments
Response to Reviewer pTj8 (part 2)
This is the continued response to the remaining weakness 3.
Response to Weakness
W3 (a): The neuroscience claims are overstated. The paper claims to "offer a computational account" of visuo-tactile resonance, but provides no comparison with neural or behavioral data (for example, comparison of similarity between internal representation of the model and fMRI or electrophysiological data)
A3: We acknowledge that the present work lacks direct comparison with neural data. For computational flexibility, we drew more on principles from neuroscience rather than incorporating anatomical structures. We have revised the manuscript to avoid expressions such as "computational account/realization" and strengthened the discussion in the Limitations section.
W3 (b): The analogies also remain metaphorical. For example, the authors claim "motivated by the parallel processing of visual and somatosensory neural circuits and their subsequent integration in the parietal cortex" (line 157-158), "a shared parietal-like latent space" (Figure 2), but encoding two modalities separately and then map them to a shared latent space is the most common structure in multi-modal learning. What is the difference between calling shared latent space "the parietal cortex" and these multi-modal alignment works?
A4: The original intention of this passage was to differentiate our method from pure regression models, as learning a direct visuo-to-tactile mapping does not necessitate separate visual and tactile pathways, such as the deterministic ResNet-50 baseline included in A8. We adopted a dual-stream architecture to prepare for enforcing cross-modal consistency in the latent space. The term "parietal-like" has been removed to enhance clarity.
W3 (c): They also claim the feature-wise gate is "Motivated by inhibitory modulation in biological somatosensory processing" (line 204-205), but this is actually just a simple Sigmoid gating, and the paper has not demonstrated that it exhibits lateral inhibition behavior (competition or inhibition between neighboring units) in the real inhibitory modulation.
A5: Thank you for this comment. Lateral inhibition in biological somatosensory systems enhances perceptual contrast, a principle we considered relevant to the sparse nature of contact events. We therefore adopted a sigmoid gate to emphasize informative feature channels. We agree that the current implementation lacks direct biological grounding, and we have accordingly removed all references to "inhibitory modulation" from the revised manuscript.
W3 (d): Meanwhile, the "mirror-touch" demonstration relies on a pre-defined controller triggered by a 0.2 N force threshold. This demonstration can only demonstrate that the force predicted from the hand image exceeds 0.2 N, and cannot be used as evidence that the model "reproduced the mirror touch phenomenon". The methodological contribution of this work behind these neuroscience claims is very limited.
A6: We appreciate this concern. To clarify the robotic mirror touch experiment, the revised Fig. 6d presents the predicted force curves by AMTNet in real time from human-hand images, along with the motion trajectory of the robotic response. The predicted force is generated consistently during contact rather than as isolated pulses, and it remains synchronized with the movement of the robotic finger. Importantly, this prediction is achieved without tactile supervision from the human hand, and we believe that such artificial implementation of vicarious touch remains rare.
Response to Reviewer pTj8 (part 3)
This is the continued response to the remaining weakness 4-5.
Response to Weakness
W4: The practical value of the studied setting is limited. All experiments are restricted to tactile prediction on a fully open robotic or human hand that is passively touched (as acknowledged in the Limitations: "our robotic hand is passively touched in the experiments"). However, most hand contacts in practice occur during complex grasping and manipulation, where the hand pose varies substantially and the contact regions are often severely occluded. Several concurrent works [2,3] (no direct comparison is required) study predicting dense tactile signals from vision during manipulation, which is considerably more challenging and practically meaningful in terms of both methodological contribution and application value. The authors should similarly discuss whether the proposed paradigm can be extended to active manipulation scenarios.
A7: We appreciate the concern regarding practical value. We first clarify that the primary goal of this paper is cross-embodiment tactile inference rather than visual-to-tactile inference alone. Specifically, we aim to simulate a cognitive phenomenon in an artificial system: observing touch on another person elicits tactile sensations in oneself, even without direct somatosensory experience. To our knowledge, computational models for such cross-embodiment tactile inference remain scarce.
We acknowledge the practical significance of the two concurrent works mentioned. However, our setting differs from theirs in at least two major respects: (1) our tactile sensing is denser. NoContactNoWorries (Patil et al.) and TouchSight (Zhou et al.) employ binary contact labels at 4 fingertips and 275 trusted taxels per hand, respectively, whereas we use 1,140 independent taxels across the palm surface at 1-mm spatial resolution, deliberately reflecting the pronounced modality gap between human-like vision and tactile sensing. (2) our framework supports cross-domain transfer. While those works primarily learn tactile inference through paired visuo-tactile data, we investigate how to transfer inference capability from a robotic hand to a human hand solely through visual alignment, without requiring tactile labels on the human side.
We adopted a passive, open-hand touch setup as a controlled platform for two reasons: (1) It allows us to uniformly stimulate all sensing regions and maintain balanced contact distribution across the hand. (2) It avoids spurious contacts caused by robotic teleoperation imperfections, where visual contact may appear without actual force interaction. We agree on the importance of application value and plan to extend this paradigm to active manipulation in future work. For instance, AMTNet can serve as a domain adapter integrated into closed-loop robotic manipulation. A promising future direction is to leverage AMTNet’s transfer principle to generate tactile states from manipulation videos of robots with varying embodiments, thereby reducing the cost of collecting paired visuo-tactile data.
We have cited these related works and discussed the limitations of current hardware setup in the revised Limitations Section.
W5: The baseline comparison is insufficient. The only external baseline is a re-implemented and substantially modified tactile diffusion model with very low S-CosSim (0.063). Simpler and more direct baselines, such as CNN/ViT regression, visuo-tactile pre-training encoders.
A8: Following this suggestion, we added a deterministic ResNet-50 regression baseline. It was adapted to our task with two minimal modifications: (1) the first convolutional layer was expanded to accept a stacked input of K frames, and (2) the classification head was replaced with a multilayer perceptron that directly regresses the taxel-level force field under the same valid-taxel masking as MTNet. All other training details remained identical to those of MTNet. Thus, the baseline shares the same visual input, target representation, and reconstruction supervision with MTNet, differing only in the absence of any cross-modal structure. The results are shown below:
| NRMSE ↓ | CosSim ↑ | S-CCC ↑ | T-IoU ↑ | W-F1 ↑ | |
|---|---|---|---|---|---|
| Deterministic ResNet-50 | 0.0500 | 0.6029 | 0.8689 | 0.3717 | 0.5887 |
| Unaligned MTNet | 0.0110 | 0.5519 | 0.8597 | 0.7061 | 0.9470 |
| Aligned MTNet | 0.0102 | 0.6097 | 0.8814 | 0.7920 | 0.9664 |
The prediction snapshots are also updated in Fig. 4. It shows that the regression baseline tends to produce signals in non-contact regions and frames, and its metric scores fall behind those of MTNet, indicating that pure numerical regression struggles to bridge the substantial modality gap between vision and touch. This additional baseline strengthens the completeness of our experimental comparison and has been incorporated into the revised manuscript.
Response to Reviewer pTj8 (part 1)
We sincerely appreciate your careful review. Please see our point-by-point response below.
Response to Weakness
A1: Thank you for this insightful suggestion. We have corrected the unaligned MTNet baseline to include two intra-modal KL terms, and we also corrected the baseline without distributional alignment (MTNet w/o distributional alignment) to similarly contain two intra-modal KL terms. The updated results in Table 1 are shown below:
The corrected unaligned MTNet baseline now exhibits basic visuo-tactile inference capability instead of collapsing, and each alignment constraint contributes to further performance gains. Applying all three alignment constraints yields the best overall performance.
We also updated the manifold analysis results for the improved unaligned MTNet (Fig. 5), showing that reconstruction-driven training alone improves the CKA score by approximately 0.15, and alignment constraints further boost it by around 0.10 over the unaligned baseline, making the visual and tactile manifolds more consistent. Together, these results demonstrate that alignment constraints effectively enhance model performance rather than serving as a prerequisite for training. This results strengthens the paper's conclusions, and the corresponding revisions have been made in the manuscript.
A2: Thank you for this comment. Based on the corrected results in A1, distributional, semantic, and geometric alignment complementarily improve the baseline performance, rather than being prerequisites for training. To further ablate the effect of alignment constraints, we trained AMTNet variants built upon the unaligned MTNet baseline and quantitatively evaluated their robot-to-human transfer performance using the corresponding robotic tactile signals as ground truth. The results are shown below:
It can be observed that alignment constraints also uniformly improve cross-embodied tactile inference, likely by reinforcing the correlation between the robotic visual and tactile spaces, thereby easing the integration of human visual observations. These results indicate that the three alignment constraints operate in a complementary manner, and the absence of any one of them degrades both the visuo-tactile mapping and transfer performance.