- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Tac2Pix, a visuo-tactile fusion framework that projects sparse and normal-only tactile measurements onto the 2D RGB image plane using forward kinematics and camera calibration. These projections are rendered as force-aware Gaussian saliency maps and fused into visual encoders using a zero-initialized input adaptation mechanism. Evaluations across simulation and real-world manipulation tasks with the LEAP hand show performance gains, particularly under induced visual occlusion (masking out parts of image).
Critical strengths
- Provides a simple, geometry-grounded interface that allows sparse tactile data to leverage pre-trained 2D visual backbones.
- Tests across multiple policy architectures (BC-MLP, ACT, Diffusion Policy, SmolVLA), visual backbones (R3M, VIP, ResNet), and occlusion protocols (fixed, random, physical).
Critical weaknesses
- The method relies strictly on accurate forward kinematics and camera calibration. Real-world kinematic drift or calibration errors lead to spatial misalignment, which degrades policy success (see Table 7).
- Projecting 3D points onto a 2D image plane discards depth information, leading to self-occlusion ambiguities (e.g., front vs. back surface contacts map to the same image pixels).
- Conceptual mismatch in Zero-Initialized Adaptation. The paper adopts a ControlNet-inspired zero-initialization strategy to preserve pre-trained visual representations at the initialization of training. However, because the visual backbone seems to be fine-tuned end-to-end during policy learning, this original pathway is immediately altered, making it unclear whether pre-trained visual features are truly preserved. To properly evaluate the isolated impact of tactile saliency adaptation, this mechanism should be evaluated on top of a frozen, pre-trained vision-only policy; otherwise, performance gains cannot be cleanly disentangled from task-specific visual representation fine-tuning.
Questions for authors
1.Can this image-space projection framework easily scale to high-resolution optical tactile sensors (like GelSight), or is it strictly constrained to sparse FSR/custom grid taxels? 2. How does this force saliency map formulation extend to tactile modalities that measure directional shear forces or torques? Mapping inherently 3D force vectors to 2D scalar saliency maps risks discarding critical contact information depending on the underlying tactile sensor. 3. Can the authors clarify how much of the performance improvement stems from the explicit geometric saliency prior versus the unfreezing and fine-tuning of the visual backbone? 4. What are the primary advantages of explicitly projecting raw tactile measurements into 2D image space? Given that the policy receives both joint state/proprioception and camera views, couldn't a sufficiently expressive policy backbone implicitly learn this spatial cross-modal alignment during policy optimization? 5. The learned tactile representation remains task-dependent rather than yielding a general visuo-tactile feature space. 6. In the three simulated tasks (pick-and-place, cube-push, and rotate-cross), contact locations can be largely inferred directly from the visual workspace and hand proprioception. As a result, it is difficult to isolate the contribution of the tactile signal under normal visual conditions. To help demonstrate the advantage of image-space tactile grounding, the benchmark should also evaluate tasks driven by fine contact dynamics that cannot be easily perceived from RGB alone (e.g. slip, precise pressure regulation, etc).