- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Sound-Triggered Mobile Manipulation, where robots respond to environmental sounds through navigation and physical interaction. It presents Habitat-Echo, the STMM-12K benchmark, and a hierarchical baseline for single- and multi-event tasks.
Critical strengths
The simulator and benchmark could be useful resources. Silent distractors and held-out sound recordings are sensible design choices. I also appreciate the per-skill evaluation and failure examples, which help explain where the system struggles.
Critical weaknesses
I’m not fully convinced that adding manipulation to audio navigation establishes a substantial new research problem here. The baseline uses audio for navigation and depth for manipulation. The main feedback described is that a source stops sounding after its event is resolved. The experiments don’t clearly show what we learn from this interaction, beyond errors building up across separate skills. A longer pipeline alone does not establish novelty.
The task goals feel prescribed rather than inferred. For example, why should relocating a ringing phone count as resolving its event? paper Appendix A.2 gives explicit sound-to-skill mappings, and the multi-event prompt already names the first target. This seems closer to sound recognition and fixed skill selection than broader goal inference.
Real-world relevance and generalization remain uncertain. The evaluation is simulation-only. A real-robot demo is not automatically required, but some validation of the acoustic assumptions would make the benchmark more convincing. The scene and object splits also need clarification. Separating sound recordings does not establish generalization to unfamiliar environments or objects.
Questions for authors
- Does planning success check target identity or only the skill chain? Would always predicting [Navigate, Pick, Place] get 100% planning success on single-event relocation?
- How much does silencing the first source help with the second task? Could you isolate this effect using the same post-interaction physical state?
- Does acoustic propagation update when objects move or doors open, beyond removing a resolved sound source? How are these updates validated?
- Which scenes and object assets are shared between training and testing? Are there results on unseen scenes or object instances?
2 comments
Rebuttal by Authors (Part 2)
Q4 Evaluation on the unseen scene
Table. Evaluation on an unseen scene. "Planning" and "Completion" denote Planning Success Rate and Completion Success Rate, respectively.
| Method | Single-event: Object Relocation Planning (↑) | Single-event: Object Relocation Completion (↑) | Single-event: State Transition Planning (↑) | Single-event: State Transition Completion (↑) | Multi-event Planning (↑) | Multi-event Completion (↑) |
|---|---|---|---|---|---|---|
| Oracle | 100.00 | 15.00 | 100.00 | 10.00 | 100.00 | 2.00 |
| MiniCPM-o-4.5 | 9.16 | 2.50 | 26.00 | 4.00 | 18.00 | 0.00 |
| Qwen3-Omni-30B | 69.16 | 3.30 | 28.00 | 4.00 | 12.00 | 0.00 |
| SSP | 88.33 | 10.83 | 42.00 | 6.00 | 32.00 | 0.00 |
Following previous work [1,2], we conduct the main experiments in a single scene with the same set of object assets across training and testing. To evaluate acoustic generalization, the 453 real-world sound instances are strictly divided into 311 training and 142 testing instances, with no overlap. In addition, Habitat-Echo randomizes (1) sound-source positions, (2)distractor positions, and (3) robot initial positions to increase layout diversity across episodes.
Following the reviewer's suggestion, we further evaluate our baseline on an unseen scene from [3], with 120, 50, and 50 episodes for Object Relocation, State Transition, and Multi-event, respectively. Compared with others, SSP retains some transfer capability, although performance drops in the unseen scene (see Table above and Figure 10 in the supplementary). This indicates that changes in scene layout remain challenging for the current policies. We consider broader generalization to unseen scenes and object assets an important direction for future work.
[1] A. Shukla, S. Tao, and H. Su, "ManiSkill-HAB: A benchmark for low-level manipulation in home rearrangement tasks," in ICLR, 2025.
[2] J. Gu, D. S. Chaplot, H. Su, and J. Malik, "Multi-skill mobile manipulation for object rearrangement," in ICLR, 2023.
[3] H. Ju, F. Yan, Z. Zhong, H. Yang, S. Huang, and Z. Zheng, "MuSTMM: Learning sound-triggered mobile manipulation across multiple scenes," arXiv preprint, 2026.
Rebuttal by Authors (Part 1)
Thank you for the constructive review. In the revised paper, we
We hope these additional experiments can address your concern. Below is our point-by-point response to your comments.
W1 Paradigm shift for the problem: from instruction following to event-triggered
STMM is not merely the modality expansion of mobile manipulation. It represents a paradigm shift in how robotic tasks are initiated and executed.
(1) Modality Expansion. Beyond vision and language, STMM incorporates environmental sound as an additional modality for perceiving and responding to events.
(2) Paradigm Shift. Existing instruction-driven robots passively execute tasks specified by human users. In contrast, STMM enables robots to actively perceive environmental events and autonomously execute appropriate physical interactions triggered by sound signals, without requiring per-episode human instructions.
W1&Q2 Effect of post-interaction sound persistence
Table. Success episodes of silencing the first sound source in the multi-event setting. Completion Success Rate (CSR) denotes the success rate of fully completing two events.
In our multi-event setting, the first sound source becomes silent after successful interaction to simulate common household scenarios where the first event is considered successfully resolved before the robot proceeds to the second event.
To evaluate the effect of this design, we conduct a controlled comparison between two settings: Resolved (Silent), the original setting, and Persistent (Active), where the first sound source remains active after the first interaction. Both settings share first-stage results (277 successful navigation episodes and 166 successful interaction episodes). Keeping the first sound source active reduces the second navigation success from 65 to 32 and the overall completion success rate from 11.25% to 4.00%. These results show that persistent sound interferes with subsequent navigation and interactions.
W2&Q3 Visualization of sound effect via Soundspaces
To visualize the qualitative sound distribution in SoundSpaces, we represent sound intensity in four directions using a quadrilateral, where a longer side indicates stronger sound intensity (see Figure 9 in supplementary). In (a), the forward sound intensity is clearly higher than in the other directions, especially the back. In (b), with the sound source fixed, moving the receiver farther away reduces the sound intensity in all directions. In (c), the leftward sound intensity is slightly stronger than the rightward. In (d), with the receiver fixed, moving the sound source to the front results in stronger forward sound intensity.
Q1 Same skill chain output: unified planner and non-trivial joint evaluation
We first clarify that object relocation and state transition (single-event setting) are evaluated jointly, without providing the interaction type to the planner. Object Relocation and State Transition share the same prompt for all Omni-MLLMs and a unified Setting-Specific Planner (see Prompt for Task Planning in supplementary). The task planner is not provided with the ground-truth interaction type and must infer the appropriate skill chain from the observations. Although always predicting [Navigate, Pick, Place] would achieve 100% Planning Success Rate on the Object Relocation, it would yield 0% on the State Transition. Therefore, no constant skill-chain prediction can solve the complete single-event setting.