acceptodds
Under review as a conference paper at ICLR 2027

Grounding Vision, Touch, and Sound for Contact-rich Dexterous Manipulation

19 views

Abstract

Vision, touch, and sound provide complementary cues for contact-rich dexterous manipulation, yet their distinct spatial and temporal structures make effective fusion challenging. Moreover, requiring every demonstration to contain all modalities limits the amount of usable training data. We introduce DexMuse, a plug-in multisensory fusion module that anchors touch, sound, and local vision at the fingertips of a dexterous hand, where contact most often occurs. Calibrated fingertip projections associate local visual features with tactile and proprioceptive context, while the resulting region tokens attend to acoustic history and complement global visual observations. We further propose availability-aware mixed training, which lets demonstrations missing touch or sound supervise the same policy without placeholder tokens or synthesized signals. We integrate DexMuse into both ACT and the pretrained vision-language-action model π0.5 through backbone-specific interfaces. Across seven challenging real-world single-arm and bimanual tasks, DexMuse reaches average success rates of 0.70 with ACT and 0.80 with π0.5, compared with 0.35 and 0.52 for concatenation-based fusion, and outperforms all baselines on every task. At a fixed budget of complete demonstrations, adding incomplete demonstrations improves DexMuse by 0.19 on average, more than twice the gain of concatenation-based fusion. Real-world videos are available at dexmuse-submission.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multi-modal Robot Learning, Dexterous Manipulation, Multi-sensory Data Fusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sAE5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DexMuse, a multisensory module for dexterous manipulation. It constructs hand-region tokens by combining proprioception and touch with visual features sampled near calibrated fingertip projections; each region token then attends to recent audio. The tokens complement global image features and are integrated into ACT and π0.5. The paper also proposes availability-aware training that masks missing touch or audio while retaining the hand-region token structure. On seven real-world tasks, DexMuse reports average success rates of 0.70 with ACT and 0.80 with π0.5, compared with 0.35 and 0.52 for multimodal concatenation baselines.

Critical strengths
  • A clear and practical fusion design. Organizing contact information around hand regions is an intuitive physical prior. The module has a relatively simple interface and is demonstrated with both a task-specific policy and a pretrained VLA model.

  • Encouraging real-world results. The gains over matched concatenation baselines are substantial across seven single-arm and bimanual tasks. The modality ablations also show that touch and audio can each improve performance in the authors’ setup.

Critical weaknesses
  • The incremental contribution over spatially grounded vision–touch fusion is unclear. Prior work, including Tac2Pix and TAP-VLA, already uses robot geometry to associate tactile information with image locations. DexMuse implements this association differently and adds audio, but the paper does not compare its vision–touch configuration against these spatially grounded approaches under matched conditions. Consequently, the results do not establish how much the proposed hand-region design contributes beyond existing geometric alignment.

  • The proposed fusion design and the role of audio are insufficiently validated. The paper lacks ablations isolating fingertip-local visual features, geometric encoding, and the fusion gates. It also does not compare hand-region audio attention with alternatives such as attention from global visual tokens. Except for Sort Bank, the paper does not identify the acoustic cues that explain the gains on individual tasks. Since audio is relatively underexplored in dexterous manipulation and may be sensitive to recording conditions, evaluation under background noise, irrelevant sounds, microphone changes, or audio–visual misalignment would also be important. The reported modality ablations and training-time dropout do not establish whether the policy uses the intended sounds or generalizes under such interference.

Questions for authors
  • Evidence needed to assess novelty: How does DexMuse perform without audio against spatially grounded vision–touch methods such as RGB-S/Tac2Pix and TAP-VLA when trained on the same data and, where feasible, with matched policy backbones?

  • Evidence needed to assess the architecture: What are the separate contributions of local fingertip visual sampling, geometric encoding, and the fusion gates? How does region-conditioned audio attention compare with audio attention from global visual tokens, attention from both token types, or direct audio-token fusion under comparable parameter budgets?

  • Clarification and diagnostic evidence: What acoustic events are expected to help in tasks other than Sort Bank, particularly Flip Bagel and Insert Knife? Do test-time muting, temporal shuffling, or mismatched-audio evaluations show that the policy depends on correctly timed task sounds?

5 comments

We thank the reviewer for the positive assessment and for recognizing DexMuse's practical design and encouraging real-world results. Below, we provide further clarifications and additional results in response to the reviewer's concerns.

Spatial vision–touch baselines and novelty

RGB-S/Tac2Pix was already included in the submitted comparison of related methods (Table 1). To provide additional quantitative evidence, we implemented a regional adaptation based on the Tac2Pix design described in its paper:

Vision and touch methodPour Candies ↑Sort Bank ↑
Tac2Pix regional adaptation0.310.08
DexMuse without audio from original Table 30.690.54

Even without audio, DexMuse outperforms the Tac2Pix regional adaptation on both tasks demonstrating the effectiveness of our regional vision–touch grounding design. Beyond spatial alignment, DexMuse further integrates region-conditioned audio and supports training with incomplete demonstrations, as summarized in Table 1. We have also expanded the TAP-VLA discussion in Appendix F (p. 20).

Architectural ablations

SettingPour Candies ↑Sort Bank ↑
Full DexMuse with regional audio queries0.770.92
Finger-wise visual mismatch0.540.77
No local visual appearance0.540.77
No explicit geometric encoding0.690.85
No visual/audio fusion gates0.310.31
Global visual audio queries0.540.69
Separate regional/global queries with averaged outputs0.690.31
Shared pooled audio0.230.23

These results support the central design choices of DexMuse. On Pour Candies, disrupting fingertip-to-image correspondence reduces success from 0.77 to 0.54, while removing local appearance yields 0.54, highlighting the value of correctly grounded visual evidence. Sort Bank remains effective under these visual ablations, consistent with its greater reliance on touch and sound. Geometric grounding remains present when explicit coordinate encoding is removed, through fingertip projection and regional correspondence.

Regional audio queries outperform all three tested alternatives on both tasks, supporting retrieval conditioned on regional touch, proprioception, and local vision. Removing the fusion gates reduces success to 0.31 on both tasks, further supporting selective integration.

DexMuse enhances the observation representation supplied to ACT. Our baseline comparisons use matched training data, encoders, and training settings. These results support the effectiveness of the proposed multimodal fusion design, and the improvements with both ACT and π0.5 demonstrate its compatibility across policy backbones.

What audio contributes beyond Sort Bank

Audio provides feedback about contact onset and changes in interaction state, including tool–object interactions that can be difficult to observe visually, as shown in Figure 1. Because contact microphones capture vibrations transmitted through the hand and grasped tool, they complement fingertip touch with information about interactions beyond the immediate grasp.

The original Table 3 shows that adding audio to vision and touch improves success from 0.46 to 0.77 on Flip Bagel and from 0.23 to 0.62 on Insert Knife. The four new synchronized visualizations in Appendix G, Fig. 10(c-f) (p. 22), provide physical context for these gains. Flip Bagel exhibits distinct acoustic patterns during spatula grasping, pan adjustment, and lifting. Insert Knife shows changes across grasping, block stabilization, and insertion, with complementary signals from the two hands. Insert Stopper additionally exhibits a pronounced right-hand transient around the transition from bimanual interaction to hand separation.

Across the recordings per task, median right-hand 2–8 kHz power increases relative to the initial reference by 5.6 dB during pan adjustment and 6.4 dB during knife insertion. These measurements illustrate acoustic cues associated with contact-rich task stages, complementing the policy-level improvements in Table 3.

We additionally intervene on the fixed trained policy:

Audio inputPour Candies ↑Sort Bank ↑
Clean0.920.77
Muted0.690.23
Temporally shuffled0.310.54
Mismatched from another same-task recording0.000.00

Muting reduces success, particularly on Sort Bank, while temporal shuffling and same-task mismatch further demonstrate the importance of timing and correspondence. Thus, the policy benefits from task-consistent acoustic feedback rather than merely the presence of an audio stream.

For recording consistency, contact microphones emphasize structure-borne interaction sounds and reduce airborne interference. We selected mounting locations after comparing feedback at different hand regions and synchronized audio, video, and robot states using timestamps. The modality ablations, acoustic interventions, and synchronized visualizations together support the value of audio for contact-rich manipulation across the evaluated tasks.

Thank you for your reply. I’ll keep my rating.

Thank you for your prompt follow-up and your positive assessment of our work.

Could you please clarify which of your original concerns remain unresolved after the additional experiments and analyses? We would be happy to provide further clarification or evidence on any remaining points.

Reviewer_XD22OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DexMuse, a multisensory fusion module for dexterous manipulation. It uses robot kinematics and camera calibration to associate fingertip-local visual features with regional tactile and proprioceptive information. The resulting region tokens attend to acoustic history and complement global visual observations. An availability-aware masking scheme allows demonstrations missing touch or audio to supervise the same policy. DexMuse is integrated into ACT and π0.5 and evaluated on seven real-world single-arm and bimanual tasks. Experiments compare against multimodal baselines, ablate sensory inputs, and examine learning from additional demonstrations with withheld modalities.

Critical strengths

A well-motivated physical structure for multisensory fusion. Using hand regions to associate local vision, touch, and proprioception provides a useful inductive bias for contact-rich manipulation. Conditioning acoustic retrieval on these regional features offers a coherent way to incorporate interaction history while retaining global scene context.

Substantial improvements across two policy backbones and diverse real-world tasks. DexMuse improves average success from 0.35 to 0.70 with ACT and from 0.52 to 0.80 with π0.5 over their concatenation counterparts. The single-arm and bimanual evaluations, together with modality ablations, provide meaningful evidence that the complete system can exploit complementary sensory information.

Critical weaknesses

The architectural contributions are not sufficiently isolated by the ablations. DexMuse combines calibrated fingertip sampling, regional grouping, geometric embeddings, gated visual updates, region-conditioned acoustic attention, and backbone-specific integration. However, the experiments primarily vary sensor availability and demonstration completeness rather than these architectural choices. Consequently, they show that the complete system is effective without establishing which mechanisms are responsible for the gains or whether simpler alternatives would perform similarly.

This is particularly consequential because fingertip grounding is the central methodological claim. Controls that remove or shuffle the geometric associations while preserving the remaining architecture would directly test this claim. Likewise, pooled audio would help assess the need for regional acoustic attention. For ACT, the concatenation baseline also differs in memory organization and decoder integration, further complicating attribution. The feature and attention visualizations are informative but do not substitute for these controlled comparisons. This limits the strength of the methodological conclusions despite the promising aggregate performance.

Questions for authors

Could you share any available component ablations that isolate fingertip grounding and region-conditioned acoustic attention—for example, removing or shuffling the fingertip-to-image associations, or replacing acoustic attention with a shared pooled audio feature? For ACT, keeping the memory organization and decoder interface fixed in these comparisons would also help clarify how much of the improvement comes specifically from the proposed grounding mechanism.

4 comments
AuthorsOpenReview ↗

Architectural Attribution through Component Ablations and Feature Diagnostics

We thank the reviewer for recognizing DexMuse's physically grounded fusion design and real-world results. We have added controlled component ablations to address the architectural attribution concerns. We first present component ablations and feature diagnostics, followed by an additional real-robot control for ACT's memory organization and decoder integration.

SettingPour Candies SR ↑Sort Bank SR ↑Average SR ↑
Full DexMuse0.770.920.85
Finger-wise visual mismatch0.540.770.66
No local visual appearance0.540.770.66
No explicit geometric encoding0.690.850.77
No visual/audio fusion gates0.310.310.31
Shared pooled audio0.230.230.23

All entries report success rates over 13 real-world trials per setting.

1. Does correct fingertip-to-image correspondence matter?

Yes. Disrupting correspondence reduces success on both tasks while keeping the visual features available. The mismatch control assigns sampled local visual features to incorrect finger regions while retaining the recipient regions' geometry, touch, and proprioception. Success decreases from 0.77 to 0.54 on Pour Candies and from 0.92 to 0.77 on Sort Bank. Removing local visual appearance yields the same success rates. Thus, supplying local visual evidence without the correct regional correspondence does not recover the full method's performance.

2. Does regional acoustic retrieval outperform shared pooled audio?

Yes. Replacing regional retrieval with a shared temporally pooled audio feature reduces success to 0.23 on both tasks. Audio remains available, but this control removes region-conditioned retrieval and temporal selection. The comparison supports the proposed combination of regional conditioning and temporal acoustic evidence. Regional queries allow retrieval to depend on each region's current tactile, proprioceptive, and local visual context.

3. What do geometric encoding and fusion gates contribute?

Removing explicit geometric encoding yields 0.69/0.85 on Pour Candies/Sort Bank. This control retains forward-kinematic projection, local visual sampling, and regional grouping: it tests the supplementary coordinate embedding rather than removing geometric grounding itself.

Removing the regional visual and acoustic fusion gates jointly reduces success to 0.31 on both tasks, supporting selective integration of multimodal updates. These component controls retain DexMuse's ACT memory organization and decoder interface.

4. Do feature diagnostics support the behavioral ablations?

Following Figure 6, we analyze identical held-out observations for every setting: two Pour Candies demonstrations and three Sort Bank demonstrations, with 80 uniformly sampled frames per demonstration. Progress error averages the absolute difference in normalized progress to ten nearest frames from other demonstrations in the original transformer-input feature space, independently of t-SNE.

SettingPour Candies error (pp) ↓Sort Bank error (pp) ↓Average error (pp) ↓
Full DexMuse14.575.4710.02
Finger-wise visual mismatch17.095.7011.40
No local visual appearance17.885.8611.87
No explicit geometric encoding17.245.8011.52
No visual/audio fusion gates16.606.5111.56
Shared pooled audio17.515.9511.73

The full method has the highest reported robot success and the lowest progress error within these component comparisons. Appendix H (Table 6 and Figure 11) provides the corresponding t-SNE visualizations and diagnostic protocol. These representation results complement the controlled behavioral ablations, which provide our primary evidence for the proposed fusion mechanisms. Bold marks the best result in each column; arrows indicate the preferred direction.

AuthorsOpenReview ↗

Isolating Fusion Gains with a Fixed ACT Memory Organization and Decoder Interface

5. How do MM-ACT and DexMuse differ in their ACT integration?

In MM-ACT, independently encoded modality tokens are concatenated with image, robot-state, and latent tokens into one sequence. The ACT encoder processes this sequence, and the decoder attends to the resulting shared memory.

DexMuse constructs regional interaction tokens and processes the interaction sequence and global image sequence separately using the same encoder. Each decoder layer first attends to interaction memory, which includes regional, robot-state, and latent tokens, and then receives a gated cross-attention update from global scene memory. Both methods retain ACT's action-chunk prediction formulation and training objective.

6. What does the new control hold fixed?

The new aligned-input concatenation control retains calibrated fingertip sampling, geometric encoding, regional touch/proprioception encoders, and the same ordered audio tokens. It also retains DexMuse's regional token count, token dimensions and positional embeddings, interaction/scene memory organization, and ACT encoder/decoder interface, including the decoder's global-scene gate.

Within each region, we replace region-conditioned acoustic attention and gated local visual/acoustic updates with feature concatenation followed by a shared linear projection, GELU, and LayerNorm. All four ordered audio tokens are retained rather than temporally pooled. This keeps both spatial alignment and temporal acoustic information available while replacing the proposed fusion operations with a simpler alternative.

The control uses the same task demonstrations, modality encoders, 40,000-step training budget, and task-specific training settings as the corresponding DexMuse configuration. Its trainable parameter count is approximately 0.38% higher, so the comparison does not favor DexMuse through greater parameter capacity.

7. What do the new real-robot results show?

MethodPour Candies SR ↑Sort Bank SR ↑Average SR ↑
MM-ACT (original Table 2)0.460.540.50
Aligned-input concat with DexMuse ACT interface (new)0.310.620.47
Full DexMuse0.770.920.85

Success rates are measured over 13 real-world trials per setting.

DexMuse outperforms the new concatenation control on both tasks with the ACT memory organization and decoder interface held fixed: success increases from 0.31 to 0.77 on Pour Candies and from 0.62 to 0.92 on Sort Bank. The new control already receives calibrated regional inputs and retains the separate interaction/scene memories, yet simple concatenation does not reproduce the full method's performance. These results support the value of DexMuse's regional retrieval and selective fusion beyond its downstream ACT integration.

This comparison complements the component ablations above. The visual-mismatch control tests correct regional correspondence; pooled audio and gate removal test the corresponding fusion choices; and the new concatenation control tests the proposed fusion operations while retaining the grounded inputs and ACT interface. Together, these behavioral comparisons address the architectural attribution concern directly.

Thank you again for your thoughtful feedback. We have updated our response with component ablations, feature diagnostics, and a new real-robot concatenation control that retains DexMuse’s grounded inputs, ACT memory organization, and decoder interface. These additions directly address the architectural attribution concerns you raised.

We would greatly appreciate your updated assessment of this evidence. Please let us know if any points would benefit from further clarification.

AuthorsOpenReview ↗

DexMuse’s Contributions and New Evidence in Response to Your Review: A Summary for Your Reconsideration

Thank you again for your careful, modeling-focused reading of our paper. Your review made us test our own architectural claims more rigorously than we did in the original submission, and we believe the paper is stronger for it. You recognized the gains across two policy backbones and asked whether the benefit of fusion would persist if we held ACT’s memory organization and decoder interface fixed. Your question took us back to the real-world robots to understand those gains. We summarize both rebuttal comments here for your final assessment.

Your concerns, and where they are answered

  1. Does correct fingertip-to-image correspondence matter?

In the shuffle you suggested, every sampled image patch is still available. What changes is which finger receives it. Average success across Pour Candies and Sort Bank falls from 0.85 to 0.66. On each task, the shuffled patches yield the same success rate as removing them altogether. → [W1, Q1; rebuttal Q1]

  1. Would a shared audio feature do as well?

We gave every region the same feature pooled over the audio history. Each region still receives audio, but it can no longer use its own contact state to select a particular moment. Success falls to 0.23 on both tasks. → [W1, Q1; rebuttal Q2]

  1. Could ACT’s memory organization and decoder integration explain the gain?

Following your suggestion, we built a concatenation control with the same calibrated regional inputs and four ordered audio tokens as DexMuse. Both models use separate interaction and scene memories and the same decoder interface. → [W1, Q1; rebuttal Q5–Q7]

MethodPour Candies ↑Sort Bank ↑Average ↑
Original MM-ACT0.460.540.50
Concatenation with DexMuse’s ACT interface0.310.620.47
Full DexMuse0.770.920.85

With the ACT interface held fixed, regional acoustic retrieval and gated fusion improve success on both tasks. The two-task average rises from 0.47 to 0.85.

  1. How much weight should the visualizations carry?

We agree that the real-robot controls provide the primary evidence. The feature analysis supports those results. On the same held-out demonstrations, full DexMuse has the lowest task-progress error on both tasks. → [W1; rebuttal Q4, Appendix H]

Why grounding at the fingertips matters

  1. A finger often makes contact exactly where the camera’s view is occluded. The finger feels the contact, and vibrations can travel through the grasped object to the hand. The fingertip gives the policy a physical reference for relating what it sees, feels, and hears.
  2. Kinematics and camera calibration tell us where each fingertip is in the image. We can therefore pair a finger’s touch with its image patch and use that regional context to retrieve relevant sounds. Your shuffle tests that pairing directly.
  3. Acoustic retrieval and the fusion gates use these same regions. Giving all regions pooled audio lowers success to 0.23 on both tasks. In a separate experiment, jointly removing the visual and acoustic gates lowers it to 0.31 on both. These are the largest drops among the component controls. → [Rebuttal Q2–Q3]

Contribution from a robotics perspective

We also respectfully ask you to weigh the system-level contribution alongside the modeling one.

  1. Prior work has combined vision, touch, and sound. In Table 1, the policies using all three were evaluated with parallel grippers, while dexterous-hand approaches used vision and touch or omitted scene vision from the fusion module. DexMuse brings all three to multi-fingered hands in single-arm and bimanual settings, across two hand designs.
  2. On Insert Stopper and Insert Knife, two challenging bimanual tasks, the strongest baselines reach 0.38 and 0.31. DexMuse with π0.5 reaches 0.62 and 0.69, respectively. On Sort Bank, the robot must infer the contents of an opaque container by shaking it. Both concatenation baselines reach 0.54, while DexMuse with ACT reaches 0.92.
  3. Across all seven tasks, DexMuse improves both a task-specific policy (ACT, 0.35 → 0.70) and a pretrained VLA (π0.5, 0.52 → 0.80).
  4. At a fixed budget of complete demonstrations, adding demonstrations without touch or sound improves average success by 0.19 for DexMuse, versus 0.08 for concatenation. Availability-aware training thus reduces the need to collect fully instrumented demonstrations.
  5. All success rates, including the new ablations, come from 13 real-robot trials per method, task, and condition. The videos of all seven tasks (Clickable) show the contact and coordination involved more clearly than success rates alone.

We would be glad to run a further comparison if it would help. If the new controls address your concern about architectural attribution, we would be grateful if you would consider raising your score. Thank you again for helping us make the paper stronger.

Reviewer_q9rrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DexMuse, a plug-in multi-modal fusion module for contact-rich dexterous manipulation. It anchors tactile, local visual and acoustic observations at fingertip regions of a dexterous hand, and proposes an availability-aware mixed training strategy to learn from incomplete demonstrations where tactile or audio signals are missing. The module can be seamlessly plugged into ACT and π0.5\pi_{0.5} policy backbones. The authors evaluate on 7 real-world single-arm and bimanual dexterous tasks, showing consistent improvements over naive feature-concatenation multi-modal baselines.

Critical strengths

1)The problem is well-motivated: dexterous manipulation heavily suffers from visual occlusion, and complementary touch/audio signals carry critical contact state information. The fingertip-anchored regional fusion is a clean, physically grounded design, different from simple global feature concatenation in prior work. 2)The availability-aware mixed training is practical. Instead of hallucinating missing sensor readings or discarding entire modalities, the method masks only the missing evidence while keeping the geometric fingertip anchor intact. This makes good use of low-cost incomplete real-world demonstrations, which is a meaningful practical contribution for robot learning. 3)Good backward compatibility: DexMuse is plug-and-play for two representative policy families (ACT imitation learning and flow-based VLA), showing its generality. 4)Substantial real-robot evaluation. It covers 7 tasks, both single-arm and bimanual, on two backbones.

Critical weaknesses

1)The approach relies heavily on accurate hand-eye calibration and forward kinematics to project fingertips onto image space. The paper does not systematically evaluate performance degradation under calibration noise or drift, which is common in long-running real robot deployments. 2)All training is conducted via behavioral cloning from teleoperated demonstrations. There is no validation when coupled with online RL fine-tuning. It remains unclear whether the fused multi-modal representations can provide useful reward signals or uncertainty cues for online post-training. 3)The tasks are all medium-horizon single-episode manipulation. Long-horizon multi-stage tasks with sequential subgoals are not tested. It is unknown whether the region-level multi-modal memory can persist and support long sequential reasoning. 4)The sensor setup is relatively bespoke: fingertip tactile sensors plus hand-mounted contact microphones are not standard hardware for most robot labs. This raises the barrier for other groups to reproduce and adopt this method. 5)Small samples, no variance. Each cell has only 13 trials. For example, 10/13 has a 95% CI of about [0.50, 0.92] (my calculation), so most per-task rankings are not distinguishable. There is no multi-seed training.

Questions for authors

Q1. Have you tried adding Gaussian noise to fingertip 3D coordinates in simulation/real robot to test robustness against calibration error? Could you discuss potential mitigation strategies? Q2. When you mix incomplete demonstrations, is there a careful curriculum design for sampling ratio of full / partial-modal episodes? Have you swept over this ratio? Q3. For the audio branch: is the learned attention physically interpretable? For example, does the attention peak on the finger that physically contacts the object when the sound event occurs? Q4. Why does the paper not compare against representation-learning baselines that pre-train multi-modal encoders for tactile-audio-vision robot data?

4 comments

We thank the reviewer for recognizing DexMuse's physically grounded fusion and practical use of incomplete demonstrations. We address the reviewer's concerns below.

Q1 Calibration robustness

We agree the calibration has impact on the final policy performance so DexMuse already uses a neighborhood pooling around each fingertip projection as illustrated in Equation (2) and Figure 8. We also add test-time perturbations as requested:

SettingPour Candies ↑Sort Bank ↑Average ↑
Clean Input0.770.920.85
Fingertip XYZ Gaussian noise with sigma 5 mm0.690.690.69
Fixed extrinsic perturbation with ±10 mm translation and ±3 degree rotation per axis0.690.770.73

The fixed-calibration results show tolerance to moderate extrinsic offsets. The spatial pooling helps mitigate limited projection errors.

Q2 Complete and incomplete demonstrations

The original experiment mixes complete and incomplete episodes 1:1 without a curriculum. Following your suggestion, we evaluated additional complete-to-incomplete sampling ratios, with incomplete demonstrations containing RGB-only sensory observations.

Complete to incomplete ratioPour Candies ↑Sort Bank ↑Average ↑
1:30.310.310.31
1:1 (original in Table 4)0.460.540.50
3:10.690.920.81

The 3:1 mixture achieves the highest success on both tasks, whereas placing greater weight on RGB-only demonstrations (1:3) yields lower performance.

Q3 Acoustic attention interpretation

Attention operates over acoustic history for each hand region, but it can help identify the finger generating a sound. The two new visualizations in Appendix G, Fig. 10(a,b) (p. 22) show how regional attention changes during manipulation, including for visually occluded fingers. For example, during cloth manipulation, the thumb’s attention to the latest 1.5 seconds of audio increases from 0.321 to 0.443, while the index finger’s remains similar (0.335 to 0.344).

Q4 Representation learning baselines

We kindly refer the reviewer to Table 2, where we have already included comparisons with representation-learning baselines, namely Sparsh-X and FuSe. Their average success rates are 0.38 and 0.13, compared with 0.70 for DexMuse ACT and 0.80 for DexMuse π0.5.

Training variation and fair comparison

We conducted two additional independent training runs and evaluated them.

RunPour Candies ↑Sort Bank ↑Average ↑
Original0.770.920.85
Additional run 10.920.770.85
Additional run 20.690.920.81
Mean ± standard deviation0.79 ± 0.120.87 ± 0.090.83 ± 0.02

The results show consistently high success rates across the three independent training runs, with relatively low variability, supporting the reproducibility and robustness of DexMuse across different training runs. Each success-rate cell has 13 trials.

Scope and reproducibility

Our experiments focus on validating the central contribution of DexMuse under an imitation-learning setting. Online RL fine-tuning is an interesting direction for future work. We consider medium-horizon tasks that require fine-grained dexterity, precise contact control, and bimanual coordination. While long-horizon manipulation is an important direction, it is beyond the scope of the current study. Importantly, our observation-fusion module does not impose any explicit constraint on episode length. Evaluating its effectiveness in more complex long-horizon tasks is an interesting direction for future work.

The contact microphones cost approximately US$2.50 each in our setup and are readily procurable. DexMuse requires no particular tactile-sensor fabrication process and is evaluated on XHand and XHand Pro with different tactile layouts. This suggests that our approach is not tied to a particular hand-specific tactile hardware design. We will also release the code and hardware/setup information to support reproduction.

Thank you again for your time and constructive feedback. We have updated our rebuttal with additional results on calibration robustness, complete/incomplete demonstration mixing, training variability, and acoustic attention interpretation.

When you have a chance, could you please share your updated assessment and let us know whether any concerns remain unresolved? We would be happy to provide further clarification.

Thank you for the concise rebuttal. I retain my original score.

Thank you for considering our updated response and for your overall positive assessment of our work. Could you please briefly clarify whether any of your original concerns remain unresolved, or whether you have any additional concerns? We would be happy to provide further clarification. Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.