acceptodds
Under review as a conference paper at ICLR 2027

From Instruction to Event: Sound-Triggered Mobile Manipulation

Abstract

Current mobile manipulation research predominantly follows an instruction-driven paradigm, where robots rely on predefined textual commands to execute tasks. However, this setting confines robots to a passive role, limiting robotic autonomy and the ability to react to dynamic environmental events. To address these limitations, we introduce Sound-Triggered Mobile Manipulation (STMM), where robots must actively perceive and interact with sound-emitting objects without explicit action instructions. To support STMM, we develop Habitat-Echo, a simulation platform that integrates sound rendering with physical interaction. We further propose a hierarchical baseline that translates high-level planning into low-level executions, where a task planner predicts a skill chain for policy models to execute sequentially. Experiments indicate the feasibility of perceiving auditory events and executing corresponding physical interactions without explicit instructions. Notably, in challenging multi-event scenarios, the robot successfully isolates the primary sources from overlapping acoustic interference to execute the first interactions, and subsequently proceeds to manipulate the secondary objects. These new challenges of planning-to-execution position STMM as a measurable research direction. The code and datasets will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
audio, mobile manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_15syOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Sound-Triggered Mobile Manipulation, where robots respond to environmental sounds through navigation and physical interaction. It presents Habitat-Echo, the STMM-12K benchmark, and a hierarchical baseline for single- and multi-event tasks.

Critical strengths

The simulator and benchmark could be useful resources. Silent distractors and held-out sound recordings are sensible design choices. I also appreciate the per-skill evaluation and failure examples, which help explain where the system struggles.

Critical weaknesses

I’m not fully convinced that adding manipulation to audio navigation establishes a substantial new research problem here. The baseline uses audio for navigation and depth for manipulation. The main feedback described is that a source stops sounding after its event is resolved. The experiments don’t clearly show what we learn from this interaction, beyond errors building up across separate skills. A longer pipeline alone does not establish novelty.

The task goals feel prescribed rather than inferred. For example, why should relocating a ringing phone count as resolving its event? paper Appendix A.2 gives explicit sound-to-skill mappings, and the multi-event prompt already names the first target. This seems closer to sound recognition and fixed skill selection than broader goal inference.

Real-world relevance and generalization remain uncertain. The evaluation is simulation-only. A real-robot demo is not automatically required, but some validation of the acoustic assumptions would make the benchmark more convincing. The scene and object splits also need clarification. Separating sound recordings does not establish generalization to unfamiliar environments or objects.

Questions for authors
  • Does planning success check target identity or only the skill chain? Would always predicting [Navigate, Pick, Place] get 100% planning success on single-event relocation?
  • How much does silencing the first source help with the second task? Could you isolate this effect using the same post-interaction physical state?
  • Does acoustic propagation update when objects move or doors open, beyond removing a resolved sound source? How are these updates validated?
  • Which scenes and object assets are shared between training and testing? Are there results on unseen scenes or object instances?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Thank you for the constructive review. In the revised paper, we

  • Add an experiment on persistent sounding of the first sound source in the multi-event setting (Table 5 in the supplementary)
  • Add visualization of the sound effect via Soundspaces (Figure 9 in the supplementary)
  • Add evaluation on the unseen scene (Table 8 and Figure 10 in the supplementary)

We hope these additional experiments can address your concern. Below is our point-by-point response to your comments.



W1 Paradigm shift for the problem: from instruction following to event-triggered

STMM is not merely the modality expansion of mobile manipulation. It represents a paradigm shift in how robotic tasks are initiated and executed.

(1) Modality Expansion. Beyond vision and language, STMM incorporates environmental sound as an additional modality for perceiving and responding to events.

(2) Paradigm Shift. Existing instruction-driven robots passively execute tasks specified by human users. In contrast, STMM enables robots to actively perceive environmental events and autonomously execute appropriate physical interactions triggered by sound signals, without requiring per-episode human instructions.

W1&Q2 Effect of post-interaction sound persistence

Table. Success episodes of silencing the first sound source in the multi-event setting. Completion Success Rate (CSR) denotes the success rate of fully completing two events.

SettingNavigation 1 (↑)Interaction 1 (↑)Navigation 2 (↑)Interaction 2 (↑)CSR/% (↑)
Resolved (Silent)277166654511.25
Persistent (Active)27716632164.00

In our multi-event setting, the first sound source becomes silent after successful interaction to simulate common household scenarios where the first event is considered successfully resolved before the robot proceeds to the second event.

To evaluate the effect of this design, we conduct a controlled comparison between two settings: Resolved (Silent), the original setting, and Persistent (Active), where the first sound source remains active after the first interaction. Both settings share first-stage results (277 successful navigation episodes and 166 successful interaction episodes). Keeping the first sound source active reduces the second navigation success from 65 to 32 and the overall completion success rate from 11.25% to 4.00%. These results show that persistent sound interferes with subsequent navigation and interactions.

W2&Q3 Visualization of sound effect via Soundspaces

To visualize the qualitative sound distribution in SoundSpaces, we represent sound intensity in four directions using a quadrilateral, where a longer side indicates stronger sound intensity (see Figure 9 in supplementary). In (a), the forward sound intensity is clearly higher than in the other directions, especially the back. In (b), with the sound source fixed, moving the receiver farther away reduces the sound intensity in all directions. In (c), the leftward sound intensity is slightly stronger than the rightward. In (d), with the receiver fixed, moving the sound source to the front results in stronger forward sound intensity.

Q1 Same skill chain output: unified planner and non-trivial joint evaluation

We first clarify that object relocation and state transition (single-event setting) are evaluated jointly, without providing the interaction type to the planner. Object Relocation and State Transition share the same prompt for all Omni-MLLMs and a unified Setting-Specific Planner (see Prompt for Task Planning in supplementary). The task planner is not provided with the ground-truth interaction type and must infer the appropriate skill chain from the observations. Although always predicting [Navigate, Pick, Place] would achieve 100% Planning Success Rate on the Object Relocation, it would yield 0% on the State Transition. Therefore, no constant skill-chain prediction can solve the complete single-event setting.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q4 Evaluation on the unseen scene

Table. Evaluation on an unseen scene. "Planning" and "Completion" denote Planning Success Rate and Completion Success Rate, respectively.

MethodSingle-event: Object Relocation Planning (↑)Single-event: Object Relocation Completion (↑)Single-event: State Transition Planning (↑)Single-event: State Transition Completion (↑)Multi-event Planning (↑)Multi-event Completion (↑)
Oracle100.0015.00100.0010.00100.002.00
MiniCPM-o-4.59.162.5026.004.0018.000.00
Qwen3-Omni-30B69.163.3028.004.0012.000.00
SSP88.3310.8342.006.0032.000.00

Following previous work [1,2], we conduct the main experiments in a single scene with the same set of object assets across training and testing. To evaluate acoustic generalization, the 453 real-world sound instances are strictly divided into 311 training and 142 testing instances, with no overlap. In addition, Habitat-Echo randomizes (1) sound-source positions, (2)distractor positions, and (3) robot initial positions to increase layout diversity across episodes.

Following the reviewer's suggestion, we further evaluate our baseline on an unseen scene from [3], with 120, 50, and 50 episodes for Object Relocation, State Transition, and Multi-event, respectively. Compared with others, SSP retains some transfer capability, although performance drops in the unseen scene (see Table above and Figure 10 in the supplementary). This indicates that changes in scene layout remain challenging for the current policies. We consider broader generalization to unseen scenes and object assets an important direction for future work.

[1] A. Shukla, S. Tao, and H. Su, "ManiSkill-HAB: A benchmark for low-level manipulation in home rearrangement tasks," in ICLR, 2025.

[2] J. Gu, D. S. Chaplot, H. Su, and J. Malik, "Multi-skill mobile manipulation for object rearrangement," in ICLR, 2023.

[3] H. Ju, F. Yan, Z. Zhong, H. Yang, S. Huang, and Z. Zheng, "MuSTMM: Learning sound-triggered mobile manipulation across multiple scenes," arXiv preprint, 2026.

Reviewer_3U8yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a robotic task where an agent must infer what to do from sounds, locate the target object, and physically interact with it without an explicit instruction. The authors build Habitat-Echo, a simulator combining spatial sound, visual observations, and physical interaction, and create a benchmark with 11k episodes. Results show that sounding object relocation is feasible in simulation, while articulated object state changes and multi-event scenarios remain challenging.

Critical strengths
  • The target of this benchmark is valuable, how to evaluate the capability of active perception, instead of passive question answering.
  • The benchmark scale is good.
Critical weaknesses
  • The scenarios or benchmark is a little toy, including the action space and the assests. Compared with more realistic environments as in HM3D or MP3D, STMM looks more like a game setting, instead of a real-world challenge. Also, simple SSP achieves best performance, super strong performance on Completeness. I would suggest that fitting this environment would fill the gap simply, instead of developing a better omni model or agent system.
  • I also doubt the sound effect. Not sure whether the stereo audio would differ in STMM. It builds on soundspace, which is built over five years ago and its sound effect is not that realistic in current perspective. Whether the location change would reflect in the sound effects is uncertain.
Questions for authors
  • I don't understand why even with oracle, Completion is still very low in Figure 5(a). Does this mean the benchmark is not solvable? Besides, Figure 5(a) is a table.
  • The evaluation results are not sufficient. At least we need some numbers of current frontier models like GPT or Claude, beyond qwen-series models.
  • The setting is interesting, but I would appreciate human performance on these tasks, which could serve as a real upper bound, instead of the "oracle" one.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Thank you for the valuable review. In the revised paper, we

  • Add visualization of the sound effect via Soundspaces (Figure 9 in the supplementary)
  • Show the solvability of STMM-12k by the action planner in the simulator (Table 7 in the supplementary)
  • Add additional Omni-MLLMs for task planning (Figure 5(a))

We hope these additional experiments can address your concern. We respond to your review points one by one below.



W1 Justification of typical household asset and atomic action space

Unlike scene datasets primarily designed for visual navigation (HM3D, MP3D), Habitat-Echo focuses on integrating spatial audio rendering with interactive physical objects, enabling robots to perceive and physically respond to environmental sound events in typical household scenarios. The skill library is intentionally built from atomic and reusable skills[1, 2, 3]. These reusable skills provide fundamental building blocks for constructing diverse sound-triggered interaction tasks.

[1] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, "ALFRED: A benchmark for interpreting grounded instructions for everyday tasks," in CVPR, 2020.

[2] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, "CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks," in RAL, 2022.

[3] Y. Jiang, A. Gupta, et al., "VIMA: Robot manipulation with multimodal prompts," in ICML, 2023.

W1 Explanation of performance of supervised SSP

(1) Better performance of supervised SSP. SSP achieves 99.03% and 99.49% PSR on Object Relocation and State Transition, respectively. But Omni-MLLMs achieve 88.56% and 41.47%. These results show that general-purpose Omni-MLLMs require task-aware adaptation. (2) Accurate Planning Success Rate (PSR) but unsatisfactory Completion Success Rate (CSR). High PSR indicates reliable task planning ability, rather than a complete solution to STMM. Unsatisfactory CSR shows that learning task planning alone is insufficient for reliable sound-triggered mobile manipulation.

W2 Visualization of sound effect via Soundspaces

To visualize the qualitative sound distribution in SoundSpaces, we represent sound intensity in four directions using a quadrilateral, where a longer side indicates stronger sound intensity (see Figure 9 in the supplementary). In (a), the forward sound intensity is clearly higher than in the other directions, especially the back. In (b), with the sound source fixed, moving the receiver farther away reduces the sound intensity in all directions. In (c), the leftward sound intensity is slightly stronger than the right. In (d), with the receiver fixed, moving the sound source to the front results in stronger forward sound intensity.

Q1 Oracle: Oracle provides perfect planning, not perfect execution

We clarify that Oracle provides only the ground-truth skill chain, not perfect navigation or interaction. Low-level skills are still executed by learned policies and can fail. The results show that low-level execution remains challenging even with perfect task planning.

Q1&Q3 Solvability of STMM-12k

Table. Solvability and upper bound performance of STMM-12k. "Planning" and "Completion" denote Planning Success Rate and Completion Success Rate, respectively.

MethodSingle-event: Obj Relocation Planning (↑)Single-event: Obj Relocation Completion (↑)Single-event: State Transition Planning (↑)Single-event: State Transition Completion (↑)Multi-event Planning (↑)Multi-event Completion (↑)
Action Planner100.0096.59100.0099.66100.0090.00

To validate the solvability of STMM-12K and establish a reliable execution upper bound, we evaluate an action planner that provides a reference independent of human performance. Given ground-truth task and physical states, the Action Planner plans collision-checked navigation paths and executes manipulation via inverse kinematics and joint-space control (see table above). These results show STMM-12k is solvable and has a high upper bound.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q2 Additional models for task planning

Table. Results of additional commercial MLLMs on STMM-12K. "Planning" and "Completion" denote Planning Success Rate and Completion Success Rate, respectively. †\dagger indicates that audio observations are converted into spectrograms as inputs.

MethodSingle-event: Object Relocation Planning (↑)Single-event: Object Relocation Completion (↑)Single-event: State Transition Planning (↑)Single-event: State Transition Completion (↑)Multi-event Planning (↑)Multi-event Completion (↑)
GPT-Realtime-2.1-mini7.305.359.043.7518.751.50
GPT-Realtime-2.120.4414.3634.4715.3661.505.50
Grok-4.7†25.0616.0649.6613.3152.504.00
GPT-6.1-Sol†81.7560.1075.4322.7064.256.25
GPT-6.1-Terra†59.1242.8276.9623.7268.256.25

Our original paper also includes MiniCPM-o-4.5. Following the reviewer's suggestion, we evaluate additional commercial and frontier MLLMs. We further evaluate GPT-series and Grok-4.7. For models without native audio input (†), we convert sound observations into spectrograms as image input. GPT-6.1-Terra achieves 76.96% and 68.25% planning success on State Transition and Multi-event, respectively, substantially outperforming Qwen3-Omni-30B (39.42% and 19.25%). Its Multi-event planning performance is also close to SSP (69.50%). These results provide broader comparisons with commercial MLLMs and highlight the challenges of STMM planning.

Reviewer_M2sUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Sound-Triggered Mobile Manipulation (STMM), where robots respond to environmental sounds by navigating to and interacting with sound-emitting objects without explicit instructions. The authors develop Habitat-Echo, a simulator integrating spatial sound rendering with physical interaction, and STMM-12K, a benchmark covering single-event and multi-event tasks. They propose a hierarchical baseline that combines audio-visual task planning with skill-specific policies for navigation and manipulation. Experiments demonstrate promising performance in single-event object relocation but reveal significant challenges in state transitions and multi-event task execution.

Critical strengths

The paper introduces an interesting event-driven manipulation setting and provides a unified simulator where sound perception and physical interaction are coupled. STMM-12K could be a useful benchmark for studying audio-guided mobile manipulation. Habitat-Echo integrates spatial sound rendering with physical interaction, allowing robot actions to change the acoustic environment. STMM-12K provides a structured benchmark for evaluating sound-triggered manipulation, which could support future research in this direction.

Critical weaknesses
  1. The planner relies on predefined mappings between sound categories and skill sequences, making the task largely a sound classification problem rather than high-level planning. Some mappings are also semantically questionable, such as opening a refrigerator in response to its alarm. Furthermore, all 11 object relocation categories share the same skill sequence, yet their reported Planning Success Rates vary significantly across models. This raises questions about what the planning metric actually evaluates.

  2. The manipulation environment uses simplified mechanisms such as grasp snapping and handle attachment, which reduce the complexity of real-world interactions. Even under these assumptions, completion success remains low for state transitions (28.67%) and multi-event tasks (7.75%). This limits the evidence for reliable sound-triggered mobile manipulation and raises questions about how well the benchmark reflects realistic robotic tasks.

Questions for authors
  1. Since all object relocation tasks share the same skill sequence [Navigate, Pick, Place], would a planner that always predicts this sequence achieve 100% planning success? If not, what additional criteria are used in the evaluation?

  2. Since the sound-to-skill mappings are predefined, how does the benchmark evaluate reasoning beyond sound classification? Additionally, how are the predefined actions justified for events such as refrigerator alarms or running dishwashers?

  3. How do the simplified grasping and articulated-object interaction mechanisms affect the benchmark's difficulty? Can the authors clarify how much of the reported completion performance depends on these abstractions?

  4. Why does navigation success decrease substantially from object relocation (81.02%) to state transition (44.20%) under oracle planning? Is this primarily due to differences in target geometry, navigation goal constraints, or acoustic localization difficulty? Could the authors provide a more detailed failure analysis?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Thank you for the insightful suggestion. In the revised paper, we

  • Add an ablation study of task planning without event-to-skill mapping (Table 3 in the supplementary)
  • Add an ablation study of interaction abstractions (Table 4 in the supplementary)

We hope these additions make the overall process clearer and directly address your concern. Below is our point‑by‑point response to your review.



W1&Q2 Event-to-skill mapping: task planning from sound to skill chain

Event-to-skill mapping is an essential intermediate step of task planning but not all part of it. We design this event-to-skill mapping for two reasons. (1) It follows a natural perception-to-action process: perceiving an environmental event, inferring the appropriate category, and translating it into executable skills. In this process, classification is a crucial intermediate step. (2) Predefined event-to-skill mappings establish a controlled and reproducible evaluation protocol, enabling consistent and quantitative evaluations across different planners and future research.

Table. Planning success rate/% without event-to-skill mapping.

MethodSingle-event: Object RelocationSingle-event: State TransitionMulti-event setting
MiniCPM-o-4_59.0013.4834.25
Qwen3-Omni-30B36.9830.0315.75

We further evaluate two representative task planners with direct reasoning (see table above). The system prompts describe the goals of Object Relocation, State Transition, and the Multi-event setting, as well as the skill library and the functions of individual skills, without providing any event-to-skill mappings. Given audio and visual observations, the planners are required to generate the corresponding skill chains. As shown in the table, both planners achieve worse results compared with event-to-skill mapping, showing that event-to-skill mapping is an important step in task planning.

W1&Q1 Same skill chain output: unified planner and non-trivial joint evaluation

In the single-event setting, Object Relocation and State Transition are evaluated jointly by a shared planner, without providing the ground-truth interaction type. Besides, object relocation and state transition share the same prompt for all Omni-MLLMs and a unified Setting-Specific Planner (see Prompt for Task Planning in supplementary). The task planner is not provided with the ground-truth interaction type and must infer the appropriate skill chain from the observations. Although always predicting [Navigate, Pick, Place] would achieve 100% Planning Success Rate on the Object Relocation, it would yield 0% on the State Transition. Therefore, no constant skill-chain prediction can solve the complete single-event setting.

W1 Variation of planning success rate across different models

In the single-event setting, Object Relocation and State Transition are evaluated jointly, by a shared planner, without providing the ground-truth interaction type. Although all 11 sound sources in object relocation share the same skill chain, task planners may incorrectly interpret relocation sounds as state-transition events and consequently generate incorrect skill chains. Therefore, the PSR measures event-conditioned skill-chain prediction rather than fine-grained sound classification.

Q2 Justification of refrigerator and dishwasher

We select several representative sound sources in household scenarios. For example, a refrigerator alarm indicates an abnormal internal condition, such as a temperature anomaly or malfunction. The robot navigates to the refrigerator and opens its door to inspect the interior and report the issue to the smart home system. Similarly, dishwasher sounds indicate either abnormal operation or completion of a washing cycle. In both cases, the robot navigates to and opens the dishwasher, either to inspect the abnormal condition or to release steam and facilitate drying after the washing cycle. These sound events are designed to instantiate representative physical interactions triggered by household sounds, rather than to prescribe universally optimal responses to real-world events. In the current benchmark, these mappings define controlled and reproducible event-conditioned interaction goals, allowing us to systematically evaluate sound perception, target navigation, and physical execution.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

W2 Clarification of physical interaction abstractions

Currently, our primary goal is to establish a controlled and reproducible platform (Habitat-Echo) for studying sound-triggered mobile manipulation, rather than focusing on complex contact dynamics. Habitat-Echo supports sound rendering, physical interaction, data generation, training, and evaluation.

Although some low-level contact mechanisms, such as grasp snapping and handle attachment, are simplified, the core manipulation process remains policy-controlled and non-trivial. (1)The interaction policies operate in an 8-DoF continuous action space, requiring coordinated control of seven arm joints and the gripper. (2) Articulated-object manipulation must follow object-specific kinematic constraints and achieve precise target joint states. (3) Successful execution requires accurate object placement or state transition, followed by arm retraction to its resting position. Policies can fail even after grasping an object or establishing contact with an articulated object (see Figure 7 in the supplementary). Thus, the abstractions simplify contact establishment rather than replace the actual manipulation process.

W2 Explanation of completion success rates in challenging tasks

The lower performance on State Transition (28.67%) and Multi-event (7.75%) reflects greater task complexity than Object Relocation (71.05%). As shown in Table 5, State Transition achieves 99.49% planning success, but only 44.25% navigation success and 65.12% subsequent interaction success, resulting in 28.67% overall completion. Notably, independently evaluated interaction skills achieve 94.20% success, compared with 65.12% during sequential execution.

This gap highlights the stricter navigation skill for articulated-object interactions. (1) In State Transition, robots must approach articulated objects from the front to ensure that manipulation follows their kinematic constraints and achieves the target joint states. (2) This challenge is further amplified in the multi-event setting, where 38.75% of episodes contain at least one articulated sound source, imposing the same strict navigation requirement on at least one interaction.

Q3 Ablation study of interaction abstractions

Table. Success rate of pick policy trained with different grasping thresholds.

Threshold/cmSuccess Rate/%
1376.88
1596.59
1796.83
Following previous mobile manipulation work [1], we set the grasp snapping threshold to 15 cm. To quantify the effect of this abstraction, we train and evaluate the policy with the 13 and 17 cm thresholds (see table above). A stricter threshold (13 cm) reduces the Pick success rate, while a relaxed threshold (17 cm) yields better performance. These results show that the grasping abstraction affects task difficulty, and 15 cm is a moderate threshold.

[1] J. Gu, D. S. Chaplot, H. Su, and J. Malik, "Multi-skill mobile manipulation for object rearrangement," ICLR, 2023.

Q4 Explanation of different navigation success rates

The gap mainly results from different navigation success criteria. For Object Relocation, the robot can approach rigid sound sources from any direction (front, back, left, or right), as they can be picked and placed regardless of the approach direction. In contrast, for State Transition, the robot must approach articulated sources from the front to ensure that the manipulation direction aligns with the object's kinematic constraints. This stricter navigation requirement leads to a lower navigation success rate.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers: Summary of Revisions

We are grateful to all reviewers for the constructive feedback. The questions and weaknesses raised in the reviews have improved the paper substantially. We address every comment from each reviewer separately below. The revised version of the paper has been uploaded, and it mainly includes the following:

  • Add an ablation study of task planning without event-to-skill mapping (Table 3 in the supplementary)
  • Add an ablation study to validate the effect of interaction abstractions (Table 4 in the supplementary)
  • Add visualizations of sound effects via Soundspaces (Figure 9 in the supplementary)
  • Show the solvability of STMM-12k by the action planner in the simulator (Table 7 in the supplementary)
  • Add additional Omni-MLLMs for task planning, including GPT-Realtime, GPT-6.1-Sol, GPT-6.1-Terra, and Grok-4.7 (Figure 5(a))
  • Add experiments on persistent sounding first sound source in the multi-event setting (Table 5 in the supplementary)
  • Add evaluation on the unseen scene (Table 8 and Figure 10 in the supplementary)

We hope these additional experiments address your concerns. Additionally, the proposed Sound-Triggered Mobile Manipulation (STMM) introduces the following to the community:

  • Paradigm Shift: In STMM, Robots actively perceive and respond to events triggered by environmental sound, rather than passively following text instructions. This opens a new research direction for the research community.

  • Closed-Loop & Open Platform: Habitat-Echo supports sound rendering and physical interaction, forming a closed loop of data generation, training, and evaluation. Users can also freely add custom assets. This provides a complete and flexible experimental platform for the research community.

  • Hierarchical Baseline: The proposed baseline provides an initial solution, validating the feasibility of STMM for the research community.

Best regards,

Authors of Submission 819

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.