- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OmniEchoBench to evaluate spatial audio-visual question answering and sound-guided navigation for embodied agents. The QA benchmark contains 2,372 temporal-spatial questions from 197 recorded audio-visual scenes and 600 cognitive-map questions derived from the navigation environments. The navigation benchmark comprises 900 episodes across 30 real-scanned environments, combining simulator-rendered visual observations with prerecorded FOA audio. The authors also develop a synthetic training-data pipeline. The proposed model, OmniEcho, augments Qwen3-Omni’s native semantic audio pathway with an FOA encoder and uses three-stage training: semantic alignment, query-conditioned localization, and multimodal instruction tuning. Experiments show improvements over the evaluated models on the proposed QA benchmark and navigation success rates close to those of early text-guided VLN baselines.
Critical strengths
- A relevant problem and valuable real-recorded data. The paper addresses spatial cues that conventional single-channel audio processing does not preserve. The benchmark covers off-screen sources, source motion, camera rotation, and navigation across rooms, providing useful settings for evaluating embodied spatial hearing.
- A unified perception-and-navigation framework. Combining spatial QA with closed-loop navigation enables evaluation of both explicit spatial predictions and downstream behavior. The parallel semantic and spatial audio pathways are a reasonable design, and Table 4 provides evidence that both contribute to overall QA performance within the proposed system.
Critical weaknesses
- The advantage of spatial-token integration over structured spatial predictions remains unclear. Several QA categories could potentially be addressed by estimating source direction, distance, or motion with a specialized model and passing these predictions to the language model. The paper already trains a query-conditioned localization head in Stage 2, making this a particularly relevant alternative. Table 4 supports the usefulness of the FOA pathway within the proposed system, but does not establish the benefit of passing latent spatial tokens rather than explicit spatial estimates. A comparison using the same language-model backbone and visual inputs would help isolate this design choice.
- The navigation experiments do not isolate the benefits of combining audio, vision, and language. Although OmniEchoBench-Nav is presented as a benchmark for audio-vision-language navigation, the experiments do not adequately establish the benefits of combining these modalities. Tables 2--3 compare different models receiving different forms of guidance: spatial audio versus text instructions. These comparisons do not establish whether text improves audio-visual navigation or whether audio improves vision-language navigation. A more informative evaluation would test OmniEcho on the same VLN-CE R2R episodes under three conditions: (1) vision and FOA audio, without text instructions; (2) vision and text instructions, without spatial audio; and (3) vision, FOA audio, and the same text instructions. Representative VLN and AVN methods could provide additional reference points. VLN methods could receive the original text instructions, while AVN methods could receive spatial audio emitted from the goal location and rendered online using SoundSpaces 2.0. Existing AVN models that use binaural audio should be adapted to FOA input and appropriately retrained, with the adaptation and training protocols clearly documented. These experiments would directly test whether joint audio-vision-language input improves over either navigation setting alone and would provide stronger evidence for the proposed model’s effectiveness. The claims that sound-guided navigation is inherently more challenging and achieves broadly comparable performance to VLN should be qualified. The reported results support a narrower comparison with selected baselines.
Questions for authors
- Could the authors reorganize the descriptions of synthetic training-data generation and real-world benchmark collection to make them easier to follow? Please also provide representative, complete audio-visual examples of both datasets in the supplementary material, with synchronized audio and explanations of how each example was generated or collected. The selected frames currently presented do not allow readers to assess temporal consistency or acoustic alignment adequately.
- Please proofread the text, figures, and tables carefully. Figure 1(a) contains inconsistent robot chest labels (“AI” and “IA”); the manuscript alternates among “audiovisual,” “audio-visual,” and “audio--visual”; and Figure 8 uses “SVABench-QA” in its title, whereas the caption and main text use “OmniEchoBench-QA.” Tables 4 and 15 contain the malformed header “Source textbfDirection,” and Table 15 additionally contains “Source textbfMotion.” Lines 889 and 1006 also contain the redundant expression “Eq. equation 1,” which should be corrected to “Eq. (1).” These are minor presentation issues that should be addressed consistently throughout the manuscript.
- What does “Echo” refer to in “OmniEcho”? Its literal meaning suggests reflected sound, whereas echo processing is not identified as a central component of the approach. A brief explanation would clarify the intended connection.
14 comments
Question 1: Organization of the data descriptions and complete examples
We have reorganized the description into two separate chronological workflows. For the synthetic training data, we explain scene specification, main-video and source-audio generation, visual-geometry correction, spatial-audio rendering, and subsequent augmentation and geometry-based QA construction. For the real-world benchmark, we separately describe scripted physical recording, source and camera annotations, time-stamped position/visibility/audibility records, camera-frame alignment of recorded spatial audio when applicable, and annotation-based QA construction. We will release the complete training-data annotation pipeline as part of our open-source codebase.
In supplementary material, we provide two complete audiovisual examples: a synthetic bicycle scene with a simulated camera turn and a recorded suitcase scene showing an off-screen-to-on-screen transition. Both preserve the full video and original four-channel audio without trimming or re-encoding, accompanied by generation or collection details, spatiotemporal annotations, and representative QA.
Question 2: Presentation and typographical consistency
Thank the reviewer for carefully identifying these issues. We will correct them.
Question 3: Meaning of “Echo” in “OmniEcho”
The real spatial audio collected for our benchmark naturally includes room reflections in indoor scenes. Although reflection processing is not the focus of our work, we chose “Echo” as a simple, memorable term that evokes the spatial characteristics of sound.
We believe our work opens a promising research direction and hope our responses have addressed your concerns. We welcome any further questions and would appreciate your consideration of a higher score. Thank you for your time and constructive feedback.
I apologize for the delayed response. I have read the authors’ response, the revised manuscript, and the supplementary material. I still have several questions that I hope the authors can clarify.
1.What is the core navigation task addressed by this paper?
The abstract states, "To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation." Elsewhere, the paper describes the task as "sound-guided navigation." Meanwhile, the response states that "the text in our benchmark serves solely as a system prompt specifying the navigation task and output format." Given these descriptions, is the central task VLN, AVN, or AVLN? Please clarify the core task that this paper aims to address, the role of language in that task, and the distinction between AVN and "sound-guided navigation."
2.What is the motivation for the proposed synthetic data generation pipeline?
Appendix A.1.2, "Sound-Guided Navigation Data Synthesis," describes the simulation pipeline, but the rationale for this design remains unclear. My understanding is that the scenes are generated by an LLM and do not provide explicit 3D scene geometry, while spatial audio is generated using Equation (1) and distance attenuation. Consequently, the acoustic rendering appears to be substantially simplified, without adequately accounting for scene structure and its effects on sound propagation. Why not directly use the large collection of existing MP3D/HM3D scenes and render spatial audio with SoundSpaces 2.0? The advantages of the proposed synthesis approach in terms of effectiveness and acoustic fidelity remain unclear. Moreover, the real recordings contain acoustic effects arising from the environment’s spatial structure, whereas the synthetic training data largely omit these effects. Would this mismatch not lead to a substantial sim-to-real gap when transferring from synthetic training data to real recordings?
3.What is Table 3 intended to demonstrate?
The response states that text serves solely as a system prompt. However, VLN-CE R2R provides complete navigation instructions. Were these original instructions provided to OmniEcho or VLN baselines in the experiments reported in Table 3? Please clarify exactly what inputs OmniEcho and the VLN baselines receive, and what conclusion this comparison is intended to support.
4.How strong are the comparisons provided in response to Weakness 2?
The experiments reported in response to Weakness 2 show that combining vision, FOA audio, and text yields better performance than the reduced-modality settings. However, the selected VLN and AVN baselines are among the simplest models, and comparisons with stronger recent methods are missing. These results would be acceptable as initial baseline experiments if the paper does not claim that the proposed model achieves the best performance on the proposed benchmark. Otherwise, stronger comparisons are needed to substantiate such a claim.
5.How were AV-Nav and AV-WaN trained in SoundSpaces 2.0?
Please provide a detailed description of the training procedure, including FOA audio processing, the agents’ action spaces, the training hardware, the FPS achieved during training, and the total number of environment steps used to complete training. Training these models with SoundSpaces 2.0 may take several days. Could the authors provide some clarification on how the training and responses to all reviewers were completed within the four-day response period?
We appreciate the reviewer’s careful reading and constructive feedback. We address the points raised one by one below.
Weakness 1: Latent spatial tokens versus explicit spatial estimates
Explicit spatial estimation is an alternative approach that offers transferability. However, its two-stage reasoning pipeline may be susceptible to error propagation and less suitable for tasks requiring joint understanding of visual and spatial audio information. Our experimental results show that latent spatial tokens improve overall QA accuracy by 4.88 percentage points over explicit spatial estimates, with the largest gains in Camera Rotation and Cognitive Map. Specifically, to isolate the effect of this design choice, we used the Stage 2 query-conditioned localization head mentioned by the reviewer to predict spatial quantities, including source azimuth, elevation, and distance, and provided these predictions to the LLM as text. Both methods use the same language-model backbone, visual inputs, and training setup, with no other differences in training.
OmniEcho improves Camera Rotation by 9.98 points and Cognitive Map by 21.33 points, while Source Direction decreases by 2.00 points. These results suggest that the benefit lies primarily in tasks requiring more complex spatial reasoning. Although explicit spatial estimates offer interpretability and transferability, their two-stage reasoning pipeline may be susceptible to error propagation. Retaining latent spatial representations may also better support the joint understanding of visual and spatial audio information required by these tasks.
Weakness 2: Controlled evaluation of audio, vision, and language in navigation
We would like to clarify that, as described in the paper, the text in our benchmark serves solely as a system prompt specifying the navigation task and output format; it provides no navigational guidance. Comparisons with instruction-guided VLN therefore involve different task settings, and our paper does not claim that multimodal fusion provides additional benefits.
Nevertheless, we conducted the experiments requested by the reviewer. Combining spatial audio and text instructions improves navigation over either guidance modality alone when the same OmniEcho model is evaluated on the same VLN-CE R2R episodes.
For this experiment, we additionally fine-tuned OmniEcho on a 1:1 mixture formed by taking half of the audio-navigation training data and half of the VLN-CE R2R text-guided training data. All three input conditions use this same model and the same test episodes, with identical text instructions in the two text-enabled conditions. SoundSpaces 2.0 renders FOA audio online from a source at the goal location. We also adapted AV-Nav and AV-WaN to the FOA input dimensions and retrained them.
† OmniEcho with the additional mixed-data fine-tuning described above.
The results support complementary benefits from audio and text in this setting. With vision + FOA, OmniEcho also has higher SR and SPL than the two evaluated AVN baselines. We will distinguish that setting from this additional fusion experiment and revise claims that sound-guided navigation is inherently harder or broadly comparable to instruction-guided VLN.
[1] SoundSpaces: Audio-Visual Navigation in 3D Environments. ECCV 2020.
[2] Learning to Set Waypoints for Audio-Visual Navigation. ICLR 2021.