acceptodds
Under review as a conference paper at ICLR 2027

OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

19 views

Abstract

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Sound, Omni model, MLLM, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gDBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniEchoBench to evaluate spatial audio-visual question answering and sound-guided navigation for embodied agents. The QA benchmark contains 2,372 temporal-spatial questions from 197 recorded audio-visual scenes and 600 cognitive-map questions derived from the navigation environments. The navigation benchmark comprises 900 episodes across 30 real-scanned environments, combining simulator-rendered visual observations with prerecorded FOA audio. The authors also develop a synthetic training-data pipeline. The proposed model, OmniEcho, augments Qwen3-Omni’s native semantic audio pathway with an FOA encoder and uses three-stage training: semantic alignment, query-conditioned localization, and multimodal instruction tuning. Experiments show improvements over the evaluated models on the proposed QA benchmark and navigation success rates close to those of early text-guided VLN baselines.

Critical strengths
  1. A relevant problem and valuable real-recorded data. The paper addresses spatial cues that conventional single-channel audio processing does not preserve. The benchmark covers off-screen sources, source motion, camera rotation, and navigation across rooms, providing useful settings for evaluating embodied spatial hearing.
  2. A unified perception-and-navigation framework. Combining spatial QA with closed-loop navigation enables evaluation of both explicit spatial predictions and downstream behavior. The parallel semantic and spatial audio pathways are a reasonable design, and Table 4 provides evidence that both contribute to overall QA performance within the proposed system.
Critical weaknesses
  1. The advantage of spatial-token integration over structured spatial predictions remains unclear. Several QA categories could potentially be addressed by estimating source direction, distance, or motion with a specialized model and passing these predictions to the language model. The paper already trains a query-conditioned localization head in Stage 2, making this a particularly relevant alternative. Table 4 supports the usefulness of the FOA pathway within the proposed system, but does not establish the benefit of passing latent spatial tokens rather than explicit spatial estimates. A comparison using the same language-model backbone and visual inputs would help isolate this design choice.
  2. The navigation experiments do not isolate the benefits of combining audio, vision, and language. Although OmniEchoBench-Nav is presented as a benchmark for audio-vision-language navigation, the experiments do not adequately establish the benefits of combining these modalities. Tables 2--3 compare different models receiving different forms of guidance: spatial audio versus text instructions. These comparisons do not establish whether text improves audio-visual navigation or whether audio improves vision-language navigation. A more informative evaluation would test OmniEcho on the same VLN-CE R2R episodes under three conditions: (1) vision and FOA audio, without text instructions; (2) vision and text instructions, without spatial audio; and (3) vision, FOA audio, and the same text instructions. Representative VLN and AVN methods could provide additional reference points. VLN methods could receive the original text instructions, while AVN methods could receive spatial audio emitted from the goal location and rendered online using SoundSpaces 2.0. Existing AVN models that use binaural audio should be adapted to FOA input and appropriately retrained, with the adaptation and training protocols clearly documented. These experiments would directly test whether joint audio-vision-language input improves over either navigation setting alone and would provide stronger evidence for the proposed model’s effectiveness. The claims that sound-guided navigation is inherently more challenging and achieves broadly comparable performance to VLN should be qualified. The reported results support a narrower comparison with selected baselines.
Questions for authors
  1. Could the authors reorganize the descriptions of synthetic training-data generation and real-world benchmark collection to make them easier to follow? Please also provide representative, complete audio-visual examples of both datasets in the supplementary material, with synchronized audio and explanations of how each example was generated or collected. The selected frames currently presented do not allow readers to assess temporal consistency or acoustic alignment adequately.
  2. Please proofread the text, figures, and tables carefully. Figure 1(a) contains inconsistent robot chest labels (“AI” and “IA”); the manuscript alternates among “audiovisual,” “audio-visual,” and “audio--visual”; and Figure 8 uses “SVABench-QA” in its title, whereas the caption and main text use “OmniEchoBench-QA.” Tables 4 and 15 contain the malformed header “Source textbfDirection,” and Table 15 additionally contains “Source textbfMotion.” Lines 889 and 1006 also contain the redundant expression “Eq. equation 1,” which should be corrected to “Eq. (1).” These are minor presentation issues that should be addressed consistently throughout the manuscript.
  3. What does “Echo” refer to in “OmniEcho”? Its literal meaning suggests reflected sound, whereas echo processing is not identified as a central component of the approach. A brief explanation would clarify the intended connection.
14 comments

We appreciate the reviewer’s careful reading and constructive feedback. We address the points raised one by one below.

Weakness 1: Latent spatial tokens versus explicit spatial estimates

Explicit spatial estimation is an alternative approach that offers transferability. However, its two-stage reasoning pipeline may be susceptible to error propagation and less suitable for tasks requiring joint understanding of visual and spatial audio information. Our experimental results show that latent spatial tokens improve overall QA accuracy by 4.88 percentage points over explicit spatial estimates, with the largest gains in Camera Rotation and Cognitive Map. Specifically, to isolate the effect of this design choice, we used the Stage 2 query-conditioned localization head mentioned by the reviewer to predict spatial quantities, including source azimuth, elevation, and distance, and provided these predictions to the LLM as text. Both methods use the same language-model backbone, visual inputs, and training setup, with no other differences in training.

QA categoryExplicit spatial estimates (%)OmniEcho (%)Difference (percentage points)
Source Direction24.6022.60−2.00
3D Localization13.8214.20+0.38
Source Motion23.6824.70+1.02
Camera Rotation31.7241.70+9.98
Cognitive Map26.1747.50+21.33
Overall accuracy23.6228.50+4.88

OmniEcho improves Camera Rotation by 9.98 points and Cognitive Map by 21.33 points, while Source Direction decreases by 2.00 points. These results suggest that the benefit lies primarily in tasks requiring more complex spatial reasoning. Although explicit spatial estimates offer interpretability and transferability, their two-stage reasoning pipeline may be susceptible to error propagation. Retaining latent spatial representations may also better support the joint understanding of visual and spatial audio information required by these tasks.

Weakness 2: Controlled evaluation of audio, vision, and language in navigation

We would like to clarify that, as described in the paper, the text in our benchmark serves solely as a system prompt specifying the navigation task and output format; it provides no navigational guidance. Comparisons with instruction-guided VLN therefore involve different task settings, and our paper does not claim that multimodal fusion provides additional benefits.

Nevertheless, we conducted the experiments requested by the reviewer. Combining spatial audio and text instructions improves navigation over either guidance modality alone when the same OmniEcho model is evaluated on the same VLN-CE R2R episodes. 

For this experiment, we additionally fine-tuned OmniEcho on a 1:1 mixture formed by taking half of the audio-navigation training data and half of the VLN-CE R2R text-guided training data. All three input conditions use this same model and the same test episodes, with identical text instructions in the two text-enabled conditions. SoundSpaces 2.0 renders FOA audio online from a source at the goal location. We also adapted AV-Nav and AV-WaN to the FOA input dimensions and retrained them.

ModelInputsSR (%) ↑SPL (%) ↑Oracle SR (%) ↑NE (m) ↓TL (m)
Seq2SeqVision + text21.0—28.07.818.39
CMAVision + text22.020.028.08.177.42
AV-Nav [1]Vision + FOA38.8327.2062.373.1911.64
AV-WaN [2]Vision + FOA48.9929.7253.724.1612.09
OmniEcho†Vision + text16.3714.7823.719.288.22
OmniEcho†Vision + FOA56.9335.8570.894.8325.87
OmniEcho†Vision + FOA + text60.9638.7574.814.3525.01

† OmniEcho with the additional mixed-data fine-tuning described above.

The results support complementary benefits from audio and text in this setting. With vision + FOA, OmniEcho also has higher SR and SPL than the two evaluated AVN baselines.  We will distinguish that setting from this additional fusion experiment and revise claims that sound-guided navigation is inherently harder or broadly comparable to instruction-guided VLN.

[1] SoundSpaces: Audio-Visual Navigation in 3D Environments. ECCV 2020.

[2] Learning to Set Waypoints for Audio-Visual Navigation. ICLR 2021.

Question 1: Organization of the data descriptions and complete examples

We have reorganized the description into two separate chronological workflows. For the synthetic training data, we explain scene specification, main-video and source-audio generation, visual-geometry correction, spatial-audio rendering, and subsequent augmentation and geometry-based QA construction. For the real-world benchmark, we separately describe scripted physical recording, source and camera annotations, time-stamped position/visibility/audibility records, camera-frame alignment of recorded spatial audio when applicable, and annotation-based QA construction.  We will release the complete training-data annotation pipeline as part of our open-source codebase.

In supplementary material, we provide two complete audiovisual examples: a synthetic bicycle scene with a simulated camera turn and a recorded suitcase scene showing an off-screen-to-on-screen transition. Both preserve the full video and original four-channel audio without trimming or re-encoding, accompanied by generation or collection details, spatiotemporal annotations, and representative QA. 

Question 2: Presentation and typographical consistency

Thank the reviewer for carefully identifying these issues. We will correct them.

Question 3: Meaning of “Echo” in “OmniEcho”

The real spatial audio collected for our benchmark naturally includes room reflections in indoor scenes. Although reflection processing is not the focus of our work, we chose “Echo” as a simple, memorable term that evokes the spatial characteristics of sound.

We believe our work opens a promising research direction and hope our responses have addressed your concerns. We welcome any further questions and would appreciate your consideration of a higher score. Thank you for your time and constructive feedback.

I apologize for the delayed response. I have read the authors’ response, the revised manuscript, and the supplementary material. I still have several questions that I hope the authors can clarify.

1.What is the core navigation task addressed by this paper?

The abstract states, "To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation." Elsewhere, the paper describes the task as "sound-guided navigation." Meanwhile, the response states that "the text in our benchmark serves solely as a system prompt specifying the navigation task and output format." Given these descriptions, is the central task VLN, AVN, or AVLN? Please clarify the core task that this paper aims to address, the role of language in that task, and the distinction between AVN and "sound-guided navigation."

2.What is the motivation for the proposed synthetic data generation pipeline?

Appendix A.1.2, "Sound-Guided Navigation Data Synthesis," describes the simulation pipeline, but the rationale for this design remains unclear. My understanding is that the scenes are generated by an LLM and do not provide explicit 3D scene geometry, while spatial audio is generated using Equation (1) and distance attenuation. Consequently, the acoustic rendering appears to be substantially simplified, without adequately accounting for scene structure and its effects on sound propagation. Why not directly use the large collection of existing MP3D/HM3D scenes and render spatial audio with SoundSpaces 2.0? The advantages of the proposed synthesis approach in terms of effectiveness and acoustic fidelity remain unclear. Moreover, the real recordings contain acoustic effects arising from the environment’s spatial structure, whereas the synthetic training data largely omit these effects. Would this mismatch not lead to a substantial sim-to-real gap when transferring from synthetic training data to real recordings?

3.What is Table 3 intended to demonstrate?

The response states that text serves solely as a system prompt. However, VLN-CE R2R provides complete navigation instructions. Were these original instructions provided to OmniEcho or VLN baselines in the experiments reported in Table 3? Please clarify exactly what inputs OmniEcho and the VLN baselines receive, and what conclusion this comparison is intended to support.

4.How strong are the comparisons provided in response to Weakness 2?

The experiments reported in response to Weakness 2 show that combining vision, FOA audio, and text yields better performance than the reduced-modality settings. However, the selected VLN and AVN baselines are among the simplest models, and comparisons with stronger recent methods are missing. These results would be acceptable as initial baseline experiments if the paper does not claim that the proposed model achieves the best performance on the proposed benchmark. Otherwise, stronger comparisons are needed to substantiate such a claim.

5.How were AV-Nav and AV-WaN trained in SoundSpaces 2.0?

Please provide a detailed description of the training procedure, including FOA audio processing, the agents’ action spaces, the training hardware, the FPS achieved during training, and the total number of environment steps used to complete training. Training these models with SoundSpaces 2.0 may take several days. Could the authors provide some clarification on how the training and responses to all reviewers were completed within the four-day response period?

Reviewer_31M1OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores integration of spatial audio (First-order-ambisonics) and vision tasks with LLMs. Previous work has mostly explore spatial audio LLM integration, so the audio-visual tasks exploration is new. The contribution is two-fold First, the authors contribute a spatial audio synchronized with video dataset with Question-answer (OmniEchoBench-QA) and navigation tasks (OmniEchoBench-Nav) . The datasets were recorded by the authors, so they are not synthetic, which is useful for future evals. The second contribution is a model (OmniEcho) built on top of Qwen LLM that can understand spatial audio-visual tasks. The authors conduct a validation on their benchmark (OmniEchoBench-QA and OmniEchoBench-Nav)

Critical strengths

The benchmark datasets, that the authors collected, is a strength of a paper. The datasets are specifically designed for spatial audio and video questions/answers and navigation. An important point is that the dataset was collected from real-world, and is not synthetic Currently such datasets do not exist in this field, and everyone comes up with their own, typically synthetic datasets in the papers. Therefore, if there was a common benchmark, it would be very valuable for this spatial audio field. It is not guaranteed that the dataset will be adapted (it has some flaws), but if it does, the impact on the field of spatial audio LLMs will be large. Personally, I would try using the dataset in my research, once it comes out.

Another critical strength is demonstration of audio-visual spatial understanding. For me, it is still impactful, but not as impactful as the first strength I listed (benchmarks). I have not seen work on audio-visual spatial integration, so it is interesting to see and valuable thing to do. Spatial audio and vision can do much more by complimenting each other. Interestingly, the authors approach does not appear very effective in terms of accuracy. That's quite valuable information, as if I were to implement audio-visual understanding, I would have done a similar approach, but now it makes me think ( and many other) of other approaches.

Critical weaknesses

The evals on the model on OmniEchoBench do not fully make sense to me. For instance, in Table 1, the model achieves accuracy of under 50 % for all the tasks, while the random baseline performs not a whole lot worse (12.5 - 33.6 % accuracy). Localization is just at 14.0% accuracy, which seems pretty low to me. The datasets might be difficult by nature, which is ok, and good for future work to climb. However, the authors should provide more explanation about the model performance, and show that performance is due to the dataset and not the model issues. Also, the authors could use other metrics besides accuracy, that are more lenient, and provide a better approximation of the performance.

While I applaud the authors for collecting the dataset, I think that there are some drawbacks to the collected dataset. For example, if the authors used optical tracking (like in STARSS23), the localization data could be much more accurate and complete. As I understand, the authors manually (or some CV algorithm) estimated the location from the grid on the floor. Obviously, the resolution of such data is not high, and it does not provide any elevation information. It would be good if the authors provided more rationale behind their data collection. Furthermore, it would be valuable if the dataset included transcripts of the speech in QA and navigation tasks. The collected dataset is not the first dataset for audio-visual tasks. Authors do not discuss existing datasets like the popular STARSS23, which has first order ambisonics audio, 360 video and optical tracking. One can imagine that a QA benchmark could be made directly from STARSS23, so authors should provide a solid comparison with previous work

Questions for authors

If the dataset ground truth was estimated from a 360 degree camera, how were X,Y,Z coordinates estimated precisely, especially the distance and elevation ?

Are there speech transcripts in the datasets? It would be valuable to have transcripts

How does the record dataset compare to STARSS23 ?

How was the accuracy metric calculated?

4 comments

We thank the reviewer for the thoughtful assessment and helpful suggestions. Below, we respond to each concern in detail.

Weakness 1: Low QA accuracy and interpretation of performance

OmniEcho has acquired some ability to distinguish turning directions and perceive coarse spatial directions, and its relatively low accuracy primarily reflects the difficulty of the benchmark. To assess these capabilities separately, we evaluated the final model using component-wise and relaxed metrics and computed random baselines under identical scoring criteria.

Subtask / metricSamplesOmniEchoRandom (matched scoring)
Camera rotation angle: left/right turn only17681.82%50.00%
Body-relative azimuth: left/right side only41662.98%50.00%
Coarse body-relative direction: exact eight-direction match65317.61%12.50%
Coarse body-relative direction: allow one adjacent bin (≤45°)65352.07%37.50%
Localization: direction error ≤45° only52151.06%37.50%

OmniEcho consistently outperforms the corresponding random baselines under the relaxed metrics, indicating that it has acquired foundational spatial audio-visual understanding capabilities. The gap between relaxed and strict evaluation results suggests that coarse spatial judgments are more tractable, whereas precise angle estimation, exact identification of combined spatial directions, and joint direction–distance estimation remain challenging. These distinctions help explain the relatively low accuracy under the benchmark’s strict scoring criteria and highlight the inherent difficulty of the dataset.

The Cognitive Map task provides complementary evidence: it assesses spatial understanding of stationary sound sources using a single reference image, reducing the need to integrate spatial information across changing visual observations. OmniEcho’s clear performance advantage on this task further supports its ability to reason about spatial relationships under less demanding conditions.

Overall, these results suggest that OmniEcho has acquired partial spatial understanding capabilities, while the benchmark remains challenging and offers substantial room for improvement in future models.

Weakness 2: Collection protocol, annotation precision, transcripts, and prior datasets

We agree that optical motion capture provides more precise trajectories and richer elevation information, but it requires additional infrastructure and is constrained by occlusion and tracking coverage. Our setup uses two cameras and an ambisonic microphone, with panoramic annotation support and visual–acoustic coordinate alignment, to facilitate collection across diverse environments (e.g. large open outdoor environments) for embodied spatial reasoning and navigation evaluation.

Horizontal positions are annotated using physical reference points, while source heights are estimated from the recording setup; fine-grained elevation is not a primary evaluation target. Our approach prioritizes deployment flexibility, while motion capture better supports precise 3D trajectory annotation.

Both OmniEchoBench and STARSS23 contain real spatial audio and synchronized video. STARSS23 provides a valuable resource for sound event detection and localization, whereas OmniEchoBench organizes evaluation around embodied spatial question answering and sound-guided navigation, encompassing changing viewpoints, sources outside the visible field, and recordings at multiple receiver locations. Our contribution therefore lies in the specific task design, collection protocol, and navigation evaluation setting. We provide a more detailed comparison in our response to Question 3.

Finally, we will include transcripts for speech-containing recordings in the public release.

Question 1: Estimation of source coordinates, distance, and elevation

Horizontal coordinates and distances are derived from physical floor references; height is estimated from the source's placement and is approximate. As described in Appendix A.2.1 (Annotation), the floor references anchor horizontal source coordinates, from which distances are calculated. The 360° camera assists source identification, including sources outside the model's visible field, rather than independently providing metric 3D reconstruction. Vertical coordinates use placement information, such as the height of a handheld phone or a source mounted on a floor-level toy vehicle. These annotations do not provide continuously tracked, high-precision vertical trajectories, and fine-grained elevation is not a primary evaluation target.

Question 2: Availability of speech transcripts

We will add them for speech-containing recordings in the public release, including those used in the QA and navigation tasks.

Question 3: Comparison with STARSS23

OmniEchoBench complements STARSS23 by evaluating embodied spatial reasoning and navigation under changing observations. Both datasets contain real multichannel spatial audio, synchronized video, and source-related spatial annotations. STARSS23's motion-capture annotations are better suited to precise trajectory analysis; OmniEchoBench adds task and collection designs for restricted-view QA and navigation.

AspectSTARSS23OmniEchoBench
Primary evaluation focusSound event detection and localizationEmbodied spatial QA and sound-guided navigation
Visual observations and viewpoint360° video supporting spatial sound analysisQA with a limited visual field, including camera rotation and sources entering or leaving the visible field
Navigation resourcesRecorded scenes primarily supporting detection and localizationScanned environments and real audio recordings at multiple receiver locations for closed-loop navigation evaluation
Main strengthFine-grained supervision for acoustic event analysis and localizationEvaluation of how spatial cues support reasoning under viewpoint changes and goal-directed actions

STARSS23 could support derived QA tasks, and we will acknowledge this explicitly. Our navigation setting additionally requires audio observations at multiple receiver locations in a navigable environment so that observations can change as the agent moves. This collection requirement, together with the embodied QA design, defines the contribution relative to prior audio-visual datasets.

Question 4: Definition and aggregation of evaluation metrics

As described in Appendix A.4 (Evaluation Details), for QA, each question receives a binary score, with no partial credit. The model's free-form response is first normalized into the relevant answer fields: direction, angle, distance, yes/no, or an option letter. An LLM is used only to extract these fields; it is not given the reference answer and does not make a subjective correctness judgment. Correctness is then determined by fixed rules.

QA categoryRule for a correct answer
Camera RotationFor rotation-angle questions, the turning direction must be correct and the absolute angular error must be ≤45°. For rotation-consistency questions, the yes/no answer must match the reference.
Source MotionFor motion-direction questions, the predicted direction set must exactly match the reference. For questions about a change in motion direction, the yes/no answer must match the reference.
Source DirectionFor direction questions, the direction set must exactly match the reference. For questions about turning toward a source, the turning direction must be correct and the absolute angular error must be ≤45°.
3D LocalizationThe direction set must exactly match the reference and the absolute distance error must be ≤2 m. Both conditions must hold. This is not a Euclidean 3D coordinate-error metric.
Cognitive MapThe extracted final option, A/B/C/D, must exactly match the reference option.

Directions are represented as subsets of {left, right, front, back}. For example, “back-left” corresponds to {left, back}. Direction and localization questions require exact set equality, so either omitting a required direction or adding an incorrect direction results in an incorrect answer. For angle questions, the rules also check the requested turning direction and reject conflicting opposite-direction predictions.

Category accuracy is the number of correct answers divided by the number of evaluated questions in that category. Overall accuracy is the total number of correct answers across all categories divided by the total number of evaluated questions. Thus, overall accuracy is weighted by the number of questions, rather than being an unweighted average of category accuracies.

For completeness, navigation metrics are calculated per episode and then averaged. Let d_i(t) denote the geodesic distance to the goal in episode i at time t, L_i* the shortest path length from the starting position to the goal, and P_i the actual path length accumulated by Habitat. The success threshold δ is 1 m on OmniEchoBench-Nav and 3 m on VLN-CE. Within each benchmark, SR and OS use the same threshold, consistent with the original evaluation protocol.

Navigation metricDefinition
SR ↑Mean of S_i, where S_i = 1 only if the agent issues STOP and its final geodesic distance satisfies d_i(T) < δ.
SPL ↑Mean of S_i × L_i* / max(L_i*, P_i). Unsuccessful episodes contribute zero.
OS / OSR ↑Fraction of episodes satisfying min_t d_i(t) < δ, regardless of whether STOP is issued within that region.
NE ↓Mean final geodesic distance d_i(T), in meters.
TLMean trajectory length, computed by summing Euclidean displacements between successive planar positions within each episode, in meters. In-place rotations do not increase this length.

We hope that our work offers a useful starting point for further research in this emerging area. If our responses have satisfactorily addressed your concerns, we would appreciate your consideration of a higher score. Thank you again for your time and valuable feedback.

I thank the authors for replying to my questions and comments in depth. I am satisfied with the provided answers, and I raised the score to clear accept. I believe that the paper and especially the dataset could be helpful in this research area.

Reviewer_ST2qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies spatial audio understanding for omni-modal embodied agents. The authors make three contributions. (1) OmniEchoBench, a real-world benchmark for spatial audio-visual perception and audiovision-language navigation. (2) A synthetic data pipeline that renders FOA data. (3) OmniEcho, which adds a separately pretrained FOA encoder to Qwen3-Omni-30B-A3B through a projector, while keeping the native audio pathway. The encoder is trained in three stages: semantic alignment, query-conditioned localization, and frozen integration. OmniEcho outperforms existing omni-modal models on the QA benchmark and is reported to approach text-guided VLN baselines on sound-guided navigation.

Critical strengths

Valuable real-world benchmark. Real FOA data with scripted, human-performed scenes and dense spatio-temporal annotation is impactful. The navigation set, with real FOA recordings on a dense receiver grid over scanned scenes.

Improvement in OmniEchoBench-QA benchmark is significant.

Critical weaknesses

Improvement in OmniEchoBench-Nav benchmark is not significant and can not match InternVLA-N1 and the absolute values are low.

At test time the agent receives the recording from the nearest receiver on a ~1 m grid, while each step moves 0.25 m. The audio observation is therefore piecewise constant over several steps, and the grid spacing equals the 1 m success radius. The paper should discuss how this affects stopping behavior.

Questions for authors

Since multiple actions are executed between model decisions (App. A.5.2), how does the decision interval affect navigation performance?

Qwen3-Omni encodes time via TM-RoPE. How are temporal position IDs assigned to the inserted FOA tokens, and how are they aligned with the audio and video tokens?

Is the LLM fully fine-tuned or adapted with LoRA? Does general audio/visual performance degrade compared with the base Qwen3-Omni?

How is the quality of the synthetic training data verified, e.g., the accuracy of source trajectories recovered from the generated videos?

What is OmniEcho’s performance on OmniEchoBench-Nav when given text instructions instead of FOA audio?

3 comments

We thank the reviewer for the careful evaluation and constructive suggestions. We address each concern below.

Weakness 1: Navigation performance and comparison with InternVLA-N1

OmniEcho uses spatial audio guidance, whereas InternVLA-N1 uses textual navigation instructions. These results therefore serve as reference points under different guidance settings and are not directly comparable. We acknowledge the limited absolute navigation performance, which may partly stem from the gap between synthetic training audio and real recordings. A core contribution of OmniEchoBench-Nav is an evaluation platform that combines scanned environments with spatial audio recorded at multiple receiver locations to assess sound-guided navigation. We will position the current results as an initial baseline with substantial room for improvement.

Weakness 2: Audio sampling resolution and stopping behavior

We would like to clarify a potential misunderstanding: spatial guidance may still vary as the agent rotates, because the source direction relative to the agent depends on its heading. On a regular two-dimensional square grid with 1 m spacing, the maximum Euclidean distance from any location to its nearest grid point is approximately 0.71 m. From a spatial coverage perspective, a receiver spacing of approximately 1 m therefore provides sufficient sampling granularity to support stopping decisions under a 1 m success radius.

Question 1: Effect of the decision interval

Executing four waypoints before replanning gives the best navigation performance among the three tested intervals. We fixed the prediction horizon at eight waypoints and varied the number executed per decision under identical spatial audio guidance. Compared with executing eight waypoints, executing four increases SR from 16.20% to 19.54% and SPL from 11.50% to 13.86%.

Waypoints executed per decisionSR (%) ↑SPL (%) ↑OS (%) ↑NE (m) ↓TL (m)
214.8910.9854.964.066.49
419.5413.8661.703.708.71
816.2011.5052.894.097.02

Executing only two waypoints may cause premature replanning and local oscillations, while executing eight may delay corrective feedback and allow errors to accumulate. We report the eight-waypoint setting in the paper to reduce the computational overhead of model calls. This experiment has been added to Appendix A.5.5 (Effect of the Decision Interval).

Question 2: FOA token positions and temporal alignment

First-order ambisonics (FOA) tokens use monotonically increasing, text-style position IDs, while their temporal correspondence with audio is implemented through ordered feature resampling within each segment. They do not share the audio or video timestamps assigned by TM-RoPE. For a valid segment with L audio tokens, we insert:

<foa_bos> <foa_pad> × L <foa_eos>

The block follows <|audio_end|> for standalone audio and <|vision_end|> for video with audio. The FOA encoder output is linearly interpolated to L features, projected, and substituted for the L placeholder embeddings. This matches the feature-sequence length and preserves temporal order within the segment; it does not enforce explicit timestamp-level alignment. The existing audio–video interleaving and TM-RoPE mechanism remain unchanged.

Question 3: Fine-tuning strategy and retention of general capabilities

The LLM is fully fine-tuned without LoRA. Our primary goal is to equip the model with a new capability—spatial audio-visual understanding—rather than to optimize for the retention of general capabilities. Nevertheless, as requested by the reviewer, we compared OmniEcho with the base Qwen3-Omni on four benchmarks to assess the impact on general capabilities. OmniEcho improves CountBenchQA by 3.26 percentage points, with modest decreases of 0.80–1.76 percentage points on the other three benchmarks.

BenchmarkQwen3-Omni-30b-a3b(%)OmniEcho(%)\delta
CountBenchQA83.3086.56+3.26
MMSU71.4870.42−1.06
VoiceBench OpenBookQA91.6589.89−1.76
HallusionBench72.1971.39−0.80

These results suggest that general audio and visual capabilities are largely retained on the evaluated benchmarks, despite fine-tuning exclusively on spatial audio-visual tasks. This evaluation has been added to Appendix A.5.7 (Retention of General Audio-Visual Capabilities).

Question 4: Verification of synthetic training-data quality

We verify synthetic-data quality by checking the generated visual and audio outputs against the prescribed trajectories and rendering parameters. The pipeline uses a shared geometric description for visual correction and spatial-audio rendering; FOA encoding and image-source reflections translate that geometry into directional acoustic cues. We then check visual reprojection, source-direction recovery, and waveform reconstruction:

Validation componentEvaluation scopeResult
Visual reprojection under the exact crop mapping47,067 held-out image correspondences across 120 videosMean error: 0.424 pixels; 100% within 5 pixels
Source-direction recovery from direct-path audio2,553 valid direct-path windowsMean angular error: 0.517°; 99.26% within 5°
Independent waveform reconstructionAll 120 videos, including 104 reverberant examplesAgreement with the pre-encoding audio to numerical precision

These checks support consistency between the generated data and the prescribed geometry, including subpixel mean reprojection error and subdegree mean directional error. 

Question 5: Navigation with text instructions

We would like to clarify that the text in our benchmark serves solely as a system prompt specifying the navigation task and output format, without providing navigational guidance. Our task is therefore distinct from text-guided navigation. Nevertheless, at the reviewer’s request, we compared text-guided and sound-guided navigation. Specifically, we evaluated OmniEcho on OmniEchoBench-Nav under three guidance configurations and included the base Qwen3-Omni model with text instructions as a reference. Visual observations are retained in all settings, and each model predicts and executes eight waypoints per decision.

ModelInputsSR (%) ↑SPL (%) ↑OS (%) ↑NE (m) ↓TL (m)
Qwen3-Omni-30b-a3bVision + text instructions0.330.304.416.160.92
OmniEchoVision + FOA audio16.2011.5052.894.097.02
OmniEchoVision + text instructions9.167.7836.904.784.37
OmniEchoVision + FOA audio + text instructions22.6516.4657.613.666.47

These results suggest that textual instructions and spatial audio provide complementary information for OmniEcho on this benchmark. The lower performance in the vision-plus-text setting is expected, as the base Qwen3-Omni lacks navigation capabilities (SR=0.33%) and our training data do not include corresponding text-guided navigation examples. This experiment has been added to Appendix A.5.6 (Navigation with Text Instructions).

We believe that our work opens up a promising new research direction and hope it will stimulate further interest in this area. If you feel that we have adequately addressed your concerns, we would be grateful if you would consider revising your score upward. Thank you for your time and constructive feedback.

Thanks the author for detailed response. Most of my concerns are addressed.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We sincerely thank you for your careful review and constructive feedback.

We introduce OmniEchoBench, a real-world benchmark for spatial audio-visual perception and sound-guided audio-visual navigation, together with a scalable synthetic training pipeline and OmniEcho, which integrates first-order ambisonics (FOA) spatial representations into an omni-modal model.

As the reviewers highlighted in their reviews, OmniEchoBench provides valuable real-world data for evaluating embodied spatial hearing (ST2q, 31M1, gDBe), with the potential to serve as a shared benchmark for future research (31M1). Reviewers also recognized the novelty and value of joint audio-visual spatial understanding (31M1), the unified evaluation of spatial predictions and navigation behavior (gDBe), and significant performance gains on OmniEchoBench-QA (ST2q).

We have responded to the concerns through additional experiments, detailed clarifications, and manuscript revisions. Revisions are highlighted in blue in the manuscript. We hope these updates clarify the benchmark's value and OmniEcho's advantages and contributions.

Thank you very much,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.