- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DexMuse, a multisensory module for dexterous manipulation. It constructs hand-region tokens by combining proprioception and touch with visual features sampled near calibrated fingertip projections; each region token then attends to recent audio. The tokens complement global image features and are integrated into ACT and π0.5. The paper also proposes availability-aware training that masks missing touch or audio while retaining the hand-region token structure. On seven real-world tasks, DexMuse reports average success rates of 0.70 with ACT and 0.80 with π0.5, compared with 0.35 and 0.52 for multimodal concatenation baselines.
Critical strengths
-
A clear and practical fusion design. Organizing contact information around hand regions is an intuitive physical prior. The module has a relatively simple interface and is demonstrated with both a task-specific policy and a pretrained VLA model.
-
Encouraging real-world results. The gains over matched concatenation baselines are substantial across seven single-arm and bimanual tasks. The modality ablations also show that touch and audio can each improve performance in the authors’ setup.
Critical weaknesses
-
The incremental contribution over spatially grounded vision–touch fusion is unclear. Prior work, including Tac2Pix and TAP-VLA, already uses robot geometry to associate tactile information with image locations. DexMuse implements this association differently and adds audio, but the paper does not compare its vision–touch configuration against these spatially grounded approaches under matched conditions. Consequently, the results do not establish how much the proposed hand-region design contributes beyond existing geometric alignment.
-
The proposed fusion design and the role of audio are insufficiently validated. The paper lacks ablations isolating fingertip-local visual features, geometric encoding, and the fusion gates. It also does not compare hand-region audio attention with alternatives such as attention from global visual tokens. Except for Sort Bank, the paper does not identify the acoustic cues that explain the gains on individual tasks. Since audio is relatively underexplored in dexterous manipulation and may be sensitive to recording conditions, evaluation under background noise, irrelevant sounds, microphone changes, or audio–visual misalignment would also be important. The reported modality ablations and training-time dropout do not establish whether the policy uses the intended sounds or generalizes under such interference.
Questions for authors
-
Evidence needed to assess novelty: How does DexMuse perform without audio against spatially grounded vision–touch methods such as RGB-S/Tac2Pix and TAP-VLA when trained on the same data and, where feasible, with matched policy backbones?
-
Evidence needed to assess the architecture: What are the separate contributions of local fingertip visual sampling, geometric encoding, and the fusion gates? How does region-conditioned audio attention compare with audio attention from global visual tokens, attention from both token types, or direct audio-token fusion under comparable parameter budgets?
-
Clarification and diagnostic evidence: What acoustic events are expected to help in tasks other than Sort Bank, particularly Flip Bagel and Insert Knife? Do test-time muting, temporal shuffling, or mismatched-audio evaluations show that the policy depends on correctly timed task sounds?
5 comments
Thank you for your reply. I’ll keep my rating.
Thank you for your prompt follow-up and your positive assessment of our work.
Could you please clarify which of your original concerns remain unresolved after the additional experiments and analyses? We would be happy to provide further clarification or evidence on any remaining points.
We thank the reviewer for the positive assessment and for recognizing DexMuse's practical design and encouraging real-world results. Below, we provide further clarifications and additional results in response to the reviewer's concerns.
Spatial vision–touch baselines and novelty
RGB-S/Tac2Pix was already included in the submitted comparison of related methods (Table 1). To provide additional quantitative evidence, we implemented a regional adaptation based on the Tac2Pix design described in its paper:
Even without audio, DexMuse outperforms the Tac2Pix regional adaptation on both tasks demonstrating the effectiveness of our regional vision–touch grounding design. Beyond spatial alignment, DexMuse further integrates region-conditioned audio and supports training with incomplete demonstrations, as summarized in Table 1. We have also expanded the TAP-VLA discussion in Appendix F (p. 20).
Architectural ablations
These results support the central design choices of DexMuse. On Pour Candies, disrupting fingertip-to-image correspondence reduces success from 0.77 to 0.54, while removing local appearance yields 0.54, highlighting the value of correctly grounded visual evidence. Sort Bank remains effective under these visual ablations, consistent with its greater reliance on touch and sound. Geometric grounding remains present when explicit coordinate encoding is removed, through fingertip projection and regional correspondence.
Regional audio queries outperform all three tested alternatives on both tasks, supporting retrieval conditioned on regional touch, proprioception, and local vision. Removing the fusion gates reduces success to 0.31 on both tasks, further supporting selective integration.
DexMuse enhances the observation representation supplied to ACT. Our baseline comparisons use matched training data, encoders, and training settings. These results support the effectiveness of the proposed multimodal fusion design, and the improvements with both ACT and π0.5 demonstrate its compatibility across policy backbones.
What audio contributes beyond Sort Bank
Audio provides feedback about contact onset and changes in interaction state, including tool–object interactions that can be difficult to observe visually, as shown in Figure 1. Because contact microphones capture vibrations transmitted through the hand and grasped tool, they complement fingertip touch with information about interactions beyond the immediate grasp.
The original Table 3 shows that adding audio to vision and touch improves success from 0.46 to 0.77 on Flip Bagel and from 0.23 to 0.62 on Insert Knife. The four new synchronized visualizations in Appendix G, Fig. 10(c-f) (p. 22), provide physical context for these gains. Flip Bagel exhibits distinct acoustic patterns during spatula grasping, pan adjustment, and lifting. Insert Knife shows changes across grasping, block stabilization, and insertion, with complementary signals from the two hands. Insert Stopper additionally exhibits a pronounced right-hand transient around the transition from bimanual interaction to hand separation.
Across the recordings per task, median right-hand 2–8 kHz power increases relative to the initial reference by 5.6 dB during pan adjustment and 6.4 dB during knife insertion. These measurements illustrate acoustic cues associated with contact-rich task stages, complementing the policy-level improvements in Table 3.
We additionally intervene on the fixed trained policy:
Muting reduces success, particularly on Sort Bank, while temporal shuffling and same-task mismatch further demonstrate the importance of timing and correspondence. Thus, the policy benefits from task-consistent acoustic feedback rather than merely the presence of an audio stream.
For recording consistency, contact microphones emphasize structure-borne interaction sounds and reduce airborne interference. We selected mounting locations after comparing feedback at different hand regions and synchronized audio, video, and robot states using timestamps. The modality ablations, acoustic interventions, and synchronized visualizations together support the value of audio for contact-rich manipulation across the evaluated tasks.