acceptodds
Withdrawn from ICLR 2027

Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision and touch are naturally complementary for dexterous manipulation. Yet they are represented in fundamentally different forms: visual observations are dense and spatially structured, whereas tactile measurements are sparse, heterogeneous, and robot-centric. This mismatch makes it challenging to establish explicit spatial correspondence between the two modalities, particularly when they are fused in latent space. However, each tactile measurement is associated with a known location on the robot’s kinematic chain, offering a localization prior that could bridge this representational gap. This raises a natural question: can this localization prior be used to construct a shared representation that explicitly aligns touch with vision? We introduce **Tac2Pix**, an image-space visuo-tactile fusion framework that projects sparse tactile measurements onto the RGB image plane using forward kinematics and camera calibration, then renders them as force-aware saliency maps. By expressing touch in the same pixel space as vision, Tac2Pix directly leverages pretrained 2D visual representations. We further employ zero-initialized input adaptation to preserve the original visual pathway at the start of policy learning. To assess whether this geometry-based interface yields consistent policy gains, we evaluate Tac2Pix on three simulated and three real-world dexterous manipulation tasks. In simulation, Tac2Pix consistently improves visuo-tactile policy learning across tasks, policy architectures, and visual backbones. In real-world experiments, it improves average success under visual occlusion by percentage points over the strongest latent visuo-tactile fusion baseline, with gains persisting under both random and physical occlusions. Together, these results support image space as an effective interface for fusing sparse touch with pretrained visual representations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visuo-Tactile Fusion, Dexterous Manipulation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_U8ogOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Tac2Pix, a visuo-tactile fusion framework that projects sparse and normal-only tactile measurements onto the 2D RGB image plane using forward kinematics and camera calibration. These projections are rendered as force-aware Gaussian saliency maps and fused into visual encoders using a zero-initialized input adaptation mechanism. Evaluations across simulation and real-world manipulation tasks with the LEAP hand show performance gains, particularly under induced visual occlusion (masking out parts of image).

Critical strengths
  1. Provides a simple, geometry-grounded interface that allows sparse tactile data to leverage pre-trained 2D visual backbones.
  2. Tests across multiple policy architectures (BC-MLP, ACT, Diffusion Policy, SmolVLA), visual backbones (R3M, VIP, ResNet), and occlusion protocols (fixed, random, physical).
Critical weaknesses
  1. The method relies strictly on accurate forward kinematics and camera calibration. Real-world kinematic drift or calibration errors lead to spatial misalignment, which degrades policy success (see Table 7).
  2. Projecting 3D points onto a 2D image plane discards depth information, leading to self-occlusion ambiguities (e.g., front vs. back surface contacts map to the same image pixels).
  3. Conceptual mismatch in Zero-Initialized Adaptation. The paper adopts a ControlNet-inspired zero-initialization strategy to preserve pre-trained visual representations at the initialization of training. However, because the visual backbone seems to be fine-tuned end-to-end during policy learning, this original pathway is immediately altered, making it unclear whether pre-trained visual features are truly preserved. To properly evaluate the isolated impact of tactile saliency adaptation, this mechanism should be evaluated on top of a frozen, pre-trained vision-only policy; otherwise, performance gains cannot be cleanly disentangled from task-specific visual representation fine-tuning.
Questions for authors

1.Can this image-space projection framework easily scale to high-resolution optical tactile sensors (like GelSight), or is it strictly constrained to sparse FSR/custom grid taxels? 2. How does this force saliency map formulation extend to tactile modalities that measure directional shear forces or torques? Mapping inherently 3D force vectors to 2D scalar saliency maps risks discarding critical contact information depending on the underlying tactile sensor. 3. Can the authors clarify how much of the performance improvement stems from the explicit geometric saliency prior versus the unfreezing and fine-tuning of the visual backbone? 4. What are the primary advantages of explicitly projecting raw tactile measurements into 2D image space? Given that the policy receives both joint state/proprioception and camera views, couldn't a sufficiently expressive policy backbone implicitly learn this spatial cross-modal alignment during policy optimization? 5. The learned tactile representation remains task-dependent rather than yielding a general visuo-tactile feature space. 6. In the three simulated tasks (pick-and-place, cube-push, and rotate-cross), contact locations can be largely inferred directly from the visual workspace and hand proprioception. As a result, it is difficult to isolate the contribution of the tactile signal under normal visual conditions. To help demonstrate the advantage of image-space tactile grounding, the benchmark should also evaluate tasks driven by fine contact dynamics that cannot be easily perceived from RGB alone (e.g. slip, precise pressure regulation, etc).

Reviewer_qwZjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Tac2Pix, an image-space visuo-tactile fusion framework that projects tactile measurements onto RGB images using forward kinematics and camera calibration. The resulting force-aware saliency maps are integrated into pretrained visual encoders for dexterous manipulation.

Critical strengths

1.Practical and intuitive design. The proposed framework provides a simple and computationally efficient approach to visuo-tactile fusion. Its compatibility with pretrained 2D visual backbones makes it relatively easy to integrate into existing manipulation policies.

  1. Extensive experimental evaluation. The authors evaluate Tac2Pix across six simulated and real-world manipulation tasks, four policy architectures, and multiple visual backbones. The paper also includes several ablation studies investigating tactile rendering, fusion strategies, and spatial alignment.

  2. Promising real-world results. The experiments demonstrate substantial improvements under fixed visual occlusion, including a 26.7 percentage-point gain over the strongest implicit fusion baseline in the real-world evaluation. The inclusion of physical-occlusion experiments further strengthens the empirical evaluation.

Critical weaknesses

W1. Limited novelty and insufficient comparison with prior work. Zhang et al. (2018) [1] already proposed a highly similar approach that projects tactile contacts into image space using forward kinematics and camera calibration, constructs Gaussian-smoothed tactile heatmaps, and integrates them with visual inputs. SaTA and 3D-ViTac, both cited and compared by the authors, have also explored related ideas of geometrically grounded tactile localization, including FK-based spatial anchoring. The methodological novelty of Tac2Pix is therefore limited. Moreover, the comparisons with these spatial tactile baselines are not sufficiently convincing. The authors acknowledge that their poor performance may result from a mismatch between the original sensing configurations and the adapted hardware. More controlled comparisons in simulation, with matched sensing configurations and policy backbones, would help establish the actual advantage of the proposed representation.

W2. Limited practical relevance of the occlusion evaluation. Most experiments simulate visual occlusion by introducing artificial black masks only during evaluation. While this provides a robustness stress test, it introduces a substantial train-test distribution shift and does not faithfully reflect natural occlusions caused by the robot hand or manipulated objects. Although limited physical-occlusion experiments are provided, more realistic manipulation settings involving natural physical occlusions during both training and evaluation would offer stronger evidence of practical effectiveness.

W3. Confounding factors in the spatial alignment ablation. The spatial alignment ablation in Table 7 applies pixel offsets to tactile saliency maps only at test time. Consequently, the performance degradation may arise from train-test distribution mismatch rather than spatial misalignment itself. A more rigorous evaluation should train and test policies with matched spatial offsets or offset distributions to isolate the actual contribution of spatial alignment.

Reference [1] Zhang, M. M., ten Pas, A., Detry, R., and Daniilidis, K. (2018). Tactile-Visual Integration for Task-Aware Grasping. RSS Pioneers Workshop, Robotics: Science and Systems (RSS), Pittsburgh, PA, USA.

Questions for authors

See weakness

Reviewer_Qg2POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Tac2Pix, a method of visuo-tactile fusion that utilizes camera calibration and forward kinematics to project the tactile measurements into the RGB image space. Input adaptation with zero initialization is applied to integrate the resultant force-aware saliency maps into pretrained visual encoders. The authors evaluate their approach in terms of simulation and real-world manipulation tasks and observe improvements, particularly in cases of visual occlusions.

Critical strengths
  1. The method does not require any additional 3d tactile processing pipeline and is simple enough to work with current visual backbones.

  2. With encouraging real-world results, the evaluation covers 3 different tasks, policy architectures, and visual encoders.

  3. Good in terms of ablations, saliency rendering, fusion architecture, and spatial alignment are all great additions to the paper.

Critical weaknesses
  1. Lack of the tactile information contribution isolation. The proposed representation includes both tactile data and spatial embedding of the robot geometry. Saliency map-based binary ablation already accounts for tactile contact without considering the force value. An essential part of the control task comparison baseline that projects the sensor positions without using any tactile data is missing here. How could we estimate the influence of tactile sensing on the improvement without that?

  2. In some ablations, the isolation of design decision is not achieved. I am not sure which element is responsible for the improvement in Table 5, since force-aware rendering changes both spatial support and Gaussian amplitude, please specify. The improvement cannot be attributed only to the fusion phase since Table 6 compares fusion techniques with different architectures and processing branches. Modulo pixel shifts which do not model camera calibration errors are used for the spatial alignment experiment in Table 7.

  3. Additional experimental evidence is needed to support the robustness and spatial baselines comparison claims. Results with 3 seeds in Table 4 indicate less improvements over vision-only baseline under random (+5.0%) and physical (+6.7%) occlusions although Table 3 reports substantial improvement under fixed occlusion. I am not sure wehther these improvements are statistically significant. Furthermore, it is a bit hard to directly link the lower performance of the adapted 3D-ViTac and SaTA baselines to the choice of spatial representation because they have different sensing representations and policy configurations.

Questions for authors

How does Tac2Pix differ from a pose-only baseline, where no tactile readings are used to map the robot's tactile sensor positions to image coordinates?

Are the authors able to provide separate ablative studies for Gaussian amplitude and width, since force aware rendering influences both of these parameters? Also, have the comparisons been normalized by model capacity and number of parameters, taking into account different architectures of the fusion models in App C?

The improvements over Vision-Only in Table 4 are modest for both real and random occlusions. Do the authors have any statistical comparison or confidence intervals for these improvements? In addition, how are the conclusions in Table 4 affected by the sensing-representation mismatch and different policy setups of the altered 3D-ViTac and SaTA baselines?

Reviewer_TSB1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new feature channel for tactile sensing by leveraging robot kinematics and camera information to project tactile signals into image space. For image-space encoding, Pix2Tac uses Gaussian rendering and zero-initialized input adaptation to fuse features, then uses this as a general encoder for downstream policies.

Critical strengths
  1. The problem definition is critical; the paper points out that an alignment of heterogeneous tactile measurements into force-aware, pixel-aligned saliency maps for policy learning is an underexplored field.

  2. The new saliency channel provides an easy-to-use interface for incorporating tactile information into existing visual policies.

Critical weaknesses
  1. Limited task performance gains and evidence of an advantage for dexterous control. Table 1 shows that the benefit of Tac2Pix is moderate and depends more on the downstream policies. With SmolVLA, some simple cross-attention and concatenation methods can achieve on-par performance with Pix2Tac.

  2. For both the real-robot and simulation task suites, the tasks are mainly pick-and-place tasks and pushing, etc. These tasks do not clearly require dexterous manipulation or tactile feedback. The authors should demonstrate tasks that Tac2Pix can complete while competing methods cannot, or tasks on which it achieves a decisive advantage.

  3. In Tac2Pix, stronger Gaussian kernels can mask nearby weaker contacts under max aggregation, since S(u,v)=max⁡(G1(u,v),G2(u,v))S(u,v)=\max\bigl(G_1(u,v), G_2(u,v)\bigr). This can cause a stronger contact to mask a nearby weaker contact, so adding or removing the weaker contact can leave the saliency map unchanged even when their projected locations differ. This may contradict the paper's claim on fine-grained dexterous manipulation tasks.

Questions for authors
  1. Which specific manipulation behavior in the evaluated tasks benefits from pixel-aligned tactile information?

  2. How should the moderate and condition-dependent gains with SmolVLA be interpreted?

  3. Can the authors show a task case that requires both dexterous manipulation and tactile feedback based on Pix2Tac?

  4. How does the Pix2Tac learned representation handle a weak contact that is masked by a stronger neighboring Gaussian?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.