acceptodds
Under review as a conference paper at ICLR 2027

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

Abstract

Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce **SAVU-Bench**, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce **SAVU-Diag**, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce **SAVU-EA**, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Spatial Reasoning, Audio-Visual LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LeyvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. It further introduces SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, the paper introduces SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit.

Critical strengths
  • The paper introduces SAVU-Bench, a real-world benchmark curated from six diverse data sources that evaluates spatial audio-visual understanding across three capability levels including Perception, Alignment, and Reasoning and seven evaluation tasks.
  • It introduces SAVU-Diag, a scene-linked diagnostic set that connects reasoning questions with their prerequisite grounding and alignment tasks, enabling fine-grained analysis beyond aggregate task scores.
  • It evaluates 12 representative audio-visual models, showing that similar reasoning accuracy can arise from different prerequisite-error patterns, and introduces SAVU-EA, a training-free baseline showing that explicit spatial cues improve auditory grounding and joint matching.
Critical weaknesses
  1. Out of the 12 evaluated models, 11 do not support multichannel spatial audio input at all and are only fed mono audio. Mono audio already loses inter-channel time and level differences, so the models naturally cannot tell sound directions. The main table's claim that existing models are weak at audio spatial grounding is largely because they were fed mono audio, while Spatial-Omni is the only tested model that actually supports multichannel spatial audio input.
  2. The diagnostic setup in SAVU-Diag has a flaw: answering a standalone prerequisite localization question correctly does not prove that the model actually used that location information when answering the later reasoning question. In addition, the main benchmark uses cyclic permutation evaluation to avoid option-order bias, whereas SAVU-Diag only tests the default option order once, so random single-pass fluctuations can skew the failure attribution ratios.
Questions for authors
  1. If cyclic permutation evaluation is also applied to the prerequisite and reasoning questions in SAVU-Diag, how much do the failure attribution ratios in Figure 3 change, such as the 61.4% localization error rate for OmniVideo-7B?
  2. In Appendix A.1.3, even after directly giving the ground-truth localization and alignment facts to Gemini-3.6-Flash, it still gets 18 out of 81 reasoning questions wrong. What are the main causes of error in those 18 questions, such as perspective conversion or multi-step relational reasoning?
3 comments

Thank you for the constructive review. Results shared across reviewers (Tables R1–R3) are in the global response.

W1 (spatial input availability): We agree that mono input limits the interpretation of the audio grounding results. We have added six models with native FOA or binaural input; their setup and results are given in the global response (Tables R1–R3). These results supplement the observations in our manuscript. Spatial-audio input substantially improves Audio Grounding, to about 90% for the strongest models (MiDashengLM-Spatial 91.0%, Qwen3.8-Omni-Flash 90.4%), which confirms your point for this task.

The tasks that relate sound to vision, however, remain challenging even with spatial input: the strongest spatial-audio model, Qwen3.8-Omni-Flash, reaches 50.0% on Spatial Alignment (chance 50%) and 48.7% on Joint A2V (chance 33.3%), and on the human-evaluation subset the new models remain below humans on every task except Visual Grounding. We thank the reviewer for this suggestion, which has made our analysis more complete and robust.

Q1 (cyclic permutation in SAVU-Diag): Thank you for this suggestion. We now evaluate both the reasoning questions and their prerequisites under cyclic permutation, counting a prerequisite as correct only if it is answered correctly under all of its orderings (Table R8).

For OmniVideo-7B, the share of reasoning errors with grounding-only failures (G0A1) changes from 61.4% to 21.1%, while errors with both grounding and alignment failures (G0A0) account for 76.3%. For most models, G0A0 becomes the most frequent pattern, and models with the same reasoning accuracy still show distinct profiles: at 43.8%, 65.7% of Spatial-Omni SO-30B's reasoning errors are G0A1, compared with 38.0% for MiniCPM-o-4.5 and 11.1% for Qwen3.5-Omni-Plus (Tables R3 and R8).

The cyclic results reinforce our finding that most reasoning errors co-occur with prerequisite failures, while models with the same accuracy differ in which prerequisites they fail. The complete SAVU-Diag results under cyclic evaluation are now reported in Table 4 (Sec. 4.3) of the revised paper, together with an updated qualitative case (Fig. 3).

Table R8. SAVU-Diag outcomes (%) for the original models under cyclic evaluation (same definitions as Table R3). Reasoning correct equals the Spatial Reasoning accuracy in Table 2 of the paper.

ModelReasoning correctError: G0A1Error: G1A0Error: G0A0Error: G1A1
Qwen2.5-Omni-7B44.310.41.044.30.0
Spatial-Omni SO-7B45.87.80.545.80.0
MiniCPM-o-4.543.821.40.032.32.6
OmniVideo-7B40.612.51.045.30.5
video-SALMONN2+ 7B39.60.00.559.90.0
Nemotron-3-Nano-Omni44.314.18.930.72.1
DOTS-3-Note-Preview43.227.12.119.38.3
Qwen3.5-Omni-Plus43.86.38.340.11.6
Qwen3.5-Omni-Flash43.24.20.550.02.1
Gemini-2.5-Pro45.812.53.636.51.6
Gemini-3.6-Flash44.84.29.441.70.0
Doubao-Seed-2.0-Lite41.70.07.351.00.0

Q2 (errors given ground-truth prerequisites): We manually inspected the step-by-step rationale Gemini-3.6-Flash produced for each of the 18 errors and assigned each error to one of four categories, according to the first step at which the reasoning departs from a valid derivation (Table R9).

Table R9. Error categories of Gemini-3.6-Flash on the 18 reasoning questions answered incorrectly despite ground-truth prerequisite facts.

Error categoryCasesRepresentative observation
Sound-source binding6The model attributed footsteps from outside the field of view to a visible person and used that person's position instead of the supplied acoustic location.
Perspective / reference-frame conversion4The model carried a left–right relation from the recording frame into the imagined observer's frame without accounting for the change in viewpoint.
Multi-step relational reasoning / ordering4The model inferred an order between two objects from the fact that both were to the right of the same reference object, which does not determine their relative order.
Additional spatial attributes required4The question also depends on front–back position, orientation, or the identity of the target source, which go beyond the left–right relations supplied in the oracle condition.

The most frequent failure is sound-source binding (6/18): even when the acoustic location is explicitly supplied, the model tends to bind a sound to a salient visible entity and reason from that entity's position. The remaining errors are evenly split among reference-frame conversion, multi-step relational reasoning, and questions that require additional spatial attributes. We have added this analysis to our paper.

W2 (prerequisite correctness vs. evidence use): We agree that SAVU-Diag is correlational: it shows which prerequisite failures co-occur with reasoning failures, but a correctly answered standalone prerequisite does not prove that the model used it. Two observations test evidence use directly. (1) Oracle prerequisites: supplying the correct grounding and alignment facts raises Gemini-3.6-Flash from 49.4% to 77.8% on the 81 reasoning questions (single pass), so the model does use this information when it is given explicitly. (2) Conversely, the six sound-source binding errors in Table R9 are cases in which the correct location was supplied but not used.

In addition, under the cyclic protocol, where a prerequisite counts as correct only if it is answered correctly under every ordering, 10 of 17 models still fail on reasoning instances whose prerequisites are all correct (G1A1; up to 14.7% of reasoning errors for DOTS-3-Note-Preview and 9.1% for Qwen3.8-Omni-Flash). Even robustly acquired evidence is therefore not always composed correctly.

Dear Reviewer Leyv,

Thank you again for your valuable feedback. We have posted a detailed response and a revised manuscript to address your concerns.

As the discussion phase will close in about one day, we would greatly appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Best regards, The Authors

Reviewer_RCCAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a Q+A benchmark for spatial audio-visual reasoning. The paper curates an evaluation dataset of approximately 1,000 questions and answers using real-world spatial audio + video sampled from datasets such as STARSS23 and YT360. It uses a human+LLM pipeline to construct and validate question/answer pairs for the data.

The paper then evaluates a host of off-the-shelf models on the benchmark, finding that the models generally do well on semantic alignment of audio and video information, but perform poorly on spatial alignment and reasoning. The paper provides an estimated human accuracy topline for the data by having humans solve a subset of the questions. While the humans are not perfect, they generally outperform the models by a significant margin indicating that the benchmark has a decent amount of headroom to support future model development on spatial audio-visual understanding tasks.

Critical strengths
  • The paper is well-written and easy to follow
  • The paper presents a benchmark for an important task which is currently understudied. The data is drawn from real-world video with ambisonic audio, which is a plus because most of the existing benchmarks similar to the one presented use synthetic or simulated audio.
  • The paper conducts a human evaluation of the benchmark and demonstrates that there exists a substantial margin between current models and human performance on the task, indicating that the benchmark has enough headroom to be useful in the support of future model development.
  • The paper also presents a training-free baseline model (formed via the cascade of a SELD model with an LLM) that outperforms the off-the-shelf models
Critical weaknesses
  • The benchmark is relatively small in size, containing a little over 1,000 examples. It may not have the breadth necessary to cover a wide range of domains or sound event types, and it is not nearly large enough to be used for actually training models.

  • All but one of the models evaluated are only capable of ingesting mono (single-channel) audio so it is completely unsurprising that they would perform poorly on the question types requiring spatialized perception of the audio. There are multiple open-source LLMs trained for spatial audio understanding so it would have been nice to see these included.

Questions for authors
  • What are the results for models given a binaural or FOA-aware interface beyond Spatial-Omni (e.g., stereo-capable models), and with humans given mono?
  • The evaluation benchmark is small enough in size that I am concerned about the statistical significance of the results. Can you compute these?
2 comments

Thank you for the constructive review. Results shared across reviewers are in the global response.

W2 & Q1 (models with spatial-audio input and humans with mono audio): We agree that the original evaluation did not sufficiently cover models with native spatial-audio interfaces. We have added six models with native FOA or binaural input; their setup and results are given in the global response (Tables R1–R3).

These results supplement the observations in our manuscript. Spatial-audio input substantially improves Audio Grounding to about 90% for the strongest models, while the tasks that relate sound to vision remain challenging even with spatial input: no new model has a Semantic or Spatial Alignment interval that excludes chance, and on the human-evaluation subset the new models remain below humans on every task except Visual Grounding. We thank the reviewer for this suggestion, which has made our analysis more complete and robust.

Humans with mono audio. We additionally evaluated four new participants with mono audio on the same human subset (Table R7; Table 8 of the revised paper). Without spatial cues, accuracy falls on every task that requires locating sounds, while Semantic Alignment and Visual Grounding are unaffected. This confirms that the spatial tasks of SAVU-Bench require spatial audio.

Compared with the mono-input models on the same subset, mono participants are higher on Semantic Alignment (86.7 vs. at most 73.3), Joint A2V and V2A (45.8 / 49.2 vs. at most 40.0 / 43.3) and Spatial Reasoning (60.8 vs. at most 46.7), and within the models' range on Spatial Alignment (53.3 vs. 43.3–60.0). The new spatial-audio models exceed mono participants on Audio Grounding (80.0 and 73.3), as their spatial input allows, yet mono participants still match or exceed all of them on Semantic Alignment (86.7 vs. at most 63.3), Spatial Alignment (53.3 vs. at most 50.0) and Spatial Reasoning (60.8 vs. at most 53.3).

Table R7. Human accuracy (%) with binaural and mono audio on the human-evaluation subset (single pass).

ConditionA-Ground.V-Ground.Sem. Align.Spa. Align.Joint A2VJoint V2ASpatial Reas.
Chance50.050.050.050.033.333.345.8
Human, binaural (10 participants)89.393.386.771.766.765.385.0
Human, mono (4 participants)56.798.386.753.345.849.260.8

W1 & Q2 (benchmark scale and statistical significance): SAVU-Bench is an evaluation benchmark whose size is set by what can be verified on real recordings: every question comes from real spatial audio-visual recordings, is checked by human annotators, and is kept only if at least two of three domain experts answer it correctly. Its 1,066 questions are comparable in size to recent audio-visual benchmarks such as LVOmniBench [1] (1,014 questions). The questions span 36 sound-event categories with 1,475 annotated sound-event occurrences; speech (male, female and child) accounts for 29.1% and the ten most frequent categories for 66.1%, with the remainder covering musical instruments, environmental sounds and vehicles. The full distribution is added to Appendix A.2.1 (Table 9) of the revised paper.

To quantify statistical uncertainty, Table 2 of the revised paper reports approximate 95% confidence intervals for all models (± = 1.96 bootstrap standard errors; 10,000 resamples of questions, keeping all cyclic orderings of a question together). The intervals show which results are significant. All original models are far above chance on Visual Grounding, whereas on Audio Grounding only models with spatial-audio input exceed chance. On Semantic Alignment, Spatial Alignment and Spatial Reasoning, most models remain within chance and no new model's interval excludes it; on Joint Matching, only Qwen3.8-Omni-Flash is clearly above chance in both directions. Spatial Reasoning therefore remains challenging for current models: the oracle condition (77.8%) and humans (85.0%) are far above all of them.

[1] Tao, Keda, et al. "LVOmniBench: Pioneering long audio-video understanding evaluation for omnimodal LLMs." arXiv preprint arXiv:2603.19217 (2026).

Dear Reviewer RCCA,

Thank you again for your valuable feedback. We have posted a detailed response and a revised manuscript to address your concerns.

As the discussion phase will close in about one day, we would greatly appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Best regards, The Authors

Reviewer_25VrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SAVU-Bench, a benchmark of 1,066 multiple-choice questions constructed from six existing real-world spatial audio-visual datasets. It organizes seven evaluation tasks into three levels: Perception, Alignment, and Reasoning. The authors also introduce SAVU-Diag, a scene-linked diagnostic set that connects reasoning questions with independently evaluated grounding and alignment prerequisites. Experiments on 12 models, together with human evaluation on a subset, show substantially stronger visual grounding than auditory spatial grounding, joint matching, and spatial reasoning under the evaluated input interfaces. The paper further proposes SAVU-EA, a training-free baseline that supplements the original inputs with predicted spatial cues. This improves several tasks, while an additional oracle-prerequisite experiment suggests that supplying correct prerequisite facts substantially improves, but does not fully resolve, spatial reasoning for the tested model.

Critical strengths
  1. A useful real-recorded benchmark with deliberately constructed cross-modal spatial tasks. The benchmark combines six real-world data sources with human annotation and expert verification. Its spatial negatives preserve category-level semantic compatibility while changing spatial correspondence, which helps distinguish recognizing an event from correctly relating its acoustic and visual locations. The reasoning tasks also deliberately distribute evidence across visible anchors and out-of-view sound sources. These choices make the benchmark a useful resource for studying spatial audio-visual understanding rather than only semantic recognition.
  2. Scene-linked diagnosis is a meaningful contribution beyond leaderboard evaluation. SAVU-Diag links reasoning questions to prerequisites from the same scene, time interval, and reference frame, allowing models with similar final accuracy to be compared through different error profiles. The oracle experiment is a useful complement: providing correct prerequisite facts improves Gemini-3.6-Flash from 49.38% to 77.78% on the 81 reasoning questions. Together, these analyses provide a practical starting point for investigating evidence acquisition and relation composition, rather than treating every incorrect final answer as the same failure.
Critical weaknesses
  1. Baseline selection limit the generality of the empirical conclusions. The evaluation covers 12 models, but the inclusion criteria are not explained. In particular, it includes Qwen2.5-Omni-7B and Qwen3.5-Omni-Plus/Flash while omitting Qwen3-Omni, without discussing the reason. My concern is whether the selected baselines adequately support the paper’s conclusions. An explanation of the selection criteria and a targeted evaluation of the omitted baseline would strengthen this aspect.
Questions for authors
  1. Insufficient clarity in task taxonomy, overlapping capabilities, and coverage.

While the overall Perception–Alignment–Reasoning framework is reasonable, the seven evaluation tasks are not defined along consistent dimensions. Audio Grounding and Visual Grounding are distinguished by input modality; Semantic Alignment and Spatial Alignment are distinguished by the type of correspondence being evaluated; and Joint A2V and V2A Matching are distinguished primarily by the direction of matching. In particular, Joint Matching explicitly requires both semantic and spatial correspondence, making it a combination of the two preceding alignment capabilities rather than an entirely independent capability. A2V and V2A further instantiate this combined requirement in different matching directions.

In particular, Joint A2V/V2A Matching requires both Semantic Alignment and Spatial Alignment, with A2V and V2A differing primarily in the matching direction. How do the authors characterize the overlap and compositional relationships among these tasks? When interpreting the performance gap between individual alignment tasks and Joint Matching, how can one distinguish difficulties in integrating semantic and spatial information from additional challenges introduced by candidate comparison and input organization?

  1. Clarification and targeted validation: baseline selection What criteria determined model inclusion and exclusion? Why was Qwen3-Omni omitted while Qwen2.5-Omni and Qwen3.5-Omni variants were included? Were there specific technical or resource constraints? Results for this baseline, particularly on the reasoning and diagnostic tasks, would help assess the robustness of the reported conclusions
5 comments

Thank you for the constructive review. Results shared across reviewers are in the global response.

Q1 (1) (task taxonomy and overlap): The overlap among tasks is intentional: the taxonomy follows a progressive design rather than partitioning the seven tasks along one dimension. Perception evaluates spatial grounding within each modality; Alignment first evaluates semantic and spatial correspondence separately and then requires both in Joint Matching; Reasoning composes grounded and aligned relations into new spatial judgments. Table R4 makes the dimensions explicit.

Table R4. Dimensions of the seven evaluation tasks.

TaskInputCross-modal semantic correspondenceSpatial correspondenceRelation composition
Audio Groundingaudio–within audio–
Visual Groundingvideo–within video–
Semantic Alignmentaudio + video✓––
Spatial Alignmentaudio + video–✓–
Joint Matching (A2V, V2A)audio + video✓✓–
Spatial Reasoningaudio + video✓✓✓

Q1 (2) (candidate comparison and input organization): We concatenate the candidates because not all evaluated systems accept several audio or video inputs in one query; consecutive candidates are separated by 2-second silent black-screen intervals, following the compatibility strategy of STAR-Bench (in its Appendix C) [1]. We address the two parts of your question separately.

Input organization. We re-evaluated the models that accept multiple audio or video inputs with separately provided clips, using the same cyclic protocol and prompts as the main results (Table R5). The changes are small and have no consistent sign: at most about 11 of 534 question–ordering pairs on A2V (−2.0 points) and 21 of 549 on V2A (+3.8 points), with mean changes of −0.4 and +0.5 points. Concatenation therefore does not explain the low Joint Matching scores.

Table R5. Joint Matching accuracy (%) with separately provided clips; original concatenated-input scores in parentheses. Nemotron-3-Nano-Omni does not accept multiple video inputs and is evaluated on V2A only.

ModelJoint A2VChangeJoint V2AChange
Gemini-3.6-Flash36.5 (35.4)+1.146.6 (42.8)+3.8
Gemini-2.5-Pro27.5 (28.5)−1.039.9 (41.7)−1.8
Doubao-Seed-2.0-Lite29.2 (29.2)0.039.9 (39.7)+0.2
Nemotron-3-Nano-Omni––30.6 (29.5)+1.1
Qwen3.5-Omni-Flash30.6 (32.6)−2.033.9 (33.3)+0.6
Qwen3.5-Omni-Plus35.2 (35.2)0.034.4 (35.3)−0.9
Qwen3.8-Omni-Flash48.1 (48.7)−0.646.4 (45.9)+0.5

Integration versus candidate comparison. Each Joint Matching question has one semantic negative and one spatial negative, so the wrong candidate a model selects shows which correspondence it fails. As shown in Table R6, three patterns emerge.

(i) Most near-chance mono-input models select the match, the semantic negative and the spatial negative about equally often, i.e., they use neither correspondence in the three-way setting.

(ii) Models that exploit semantics fail to use spatial correspondence: on V2A, Gemini-3.6-Flash selects the semantic negative in only 14.8% of cases but chooses the spatial negative as often as the match (42.4% vs. 42.8%), and Gemini-2.5-Pro and Doubao-Seed-2.0-Lite behave the same way. Their errors reflect a failure to integrate spatial with semantic correspondence.

(iii) Qwen3.8-Omni-Flash is the only model that discriminates spatially (spatial negative 15.9% vs. match 48.7% on A2V), but on A2V it then selects the semantic negative in 35.4% of cases, so it still fails to satisfy both conditions jointly. In addition, A2V shows a candidate-comparison effect: several models, including the mono-input Qwen3-Omni-30B, which cannot exploit spatial cues, select the semantic negative well above chance (Qwen3-Omni-30B 51.1%, Spatial-Omni SO-30B 56.0%). V2A thus mainly reflects integration difficulty, while A2V additionally reflects the comparison among candidates.

Table R6. Share (%) of question–ordering pairs in which each candidate is selected (correct match / semantic negative / spatial negative).

ModelAudio inputJoint A2VJoint V2A
Qwen3.5-Omni-PlusMono35.2 / 33.5 / 31.335.3 / 30.6 / 34.1
Gemini-3.6-FlashMono35.4 / 22.7 / 41.942.8 / 14.8 / 42.4
Gemini-2.5-ProMono28.5 / 43.6 / 27.941.7 / 16.8 / 41.5
Doubao-Seed-2.0-LiteMono29.2 / 39.0 / 31.839.7 / 19.5 / 40.8
Qwen3-Omni-30BMono21.7 / 51.1 / 27.230.4 / 36.6 / 33.0
Spatial-Omni SO-30BFOA21.4 / 56.0 / 22.728.8 / 43.7 / 27.5
Qwen3.8-Omni-FlashFOA48.7 / 35.4 / 15.945.9 / 19.5 / 34.6

[1] Liu, Zihan, et al. "STAR-Bench: Probing deep spatio-temporal reasoning as audio 4D intelligence." International Conference on Learning Representations (ICLR), 2026.

W1 & Q2 (baseline selection): We included a model if (i) it accepts audio and video jointly in a single query, or, for spatial-audio models evaluated only on Audio Grounding, accepts FOA or binaural input; (ii) its weights or API were publicly available; and (iii) it represents a distinct model family, scale or deployment setting. Within this selection, Qwen2.5-Omni-7B serves as an established open-weight reference and is the backbone of Spatial-Omni SO-7B, while Qwen3.5-Omni-Plus/Flash cover commercial APIs. Qwen3-Omni met these criteria but was not included in the submission because of computational constraints at the time, which we have now resolved. We have evaluated Qwen3-Omni-30B (Qwen3-Omni-30B-A3B-Instruct) under the same protocol, together with six further models with spatial-audio input (global response, Table R1).

Qwen3-Omni-30B's results are consistent with our original conclusions. With mono input, its Audio Grounding is at chance (48.8 ± 7.1) and its Spatial Reasoning is indistinguishable from chance (39.1 ± 8.0 vs. 42.2). Its Joint A2V accuracy (21.7 ± 4.0) is significantly below chance because it selects the semantic negative in 51.1% of question–ordering pairs (Table R6). On SAVU-Diag (Table R3), 91.5% of its reasoning errors involve a failed grounding prerequisite, the same grounding-limited profile as the other mono-input models. The selection criteria are now stated in the revised paper.

Dear Reviewer 25Vr,

Thank you again for your valuable feedback. We have posted a detailed response to address your concerns.

As the discussion phase will close in about one day, we would greatly appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Best regards, The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their careful reading and constructive suggestions. Several reviews raised concerns about baseline coverage and spatial-audio input, so we have evaluated additional models, including several that process spatial audio natively, and summarize the new results here.

New models

We added seven models, evaluated in eight configurations. Six of them process spatial audio natively: Qwen3.8-Omni-Flash (newly released in September 2026), Spatial-Omni SO-30B and JAEGER take FOA together with video; Spatial-Omni SO-4B takes FOA audio alone; and BAT and MiDashengLM-Spatial take binaural audio alone. JAEGER is evaluated both in its original RGB-D setting and with an adapted video input. We also added Qwen3-Omni-30B (Qwen3-Omni-30B-A3B-Instruct), which processes mono audio. Audio-only models are evaluated on Audio Grounding, the task their input interface supports.

FOA models receive the original four-channel FOA signals. Qwen3.8-Omni-Flash is used through its API with FOA input and its thinking mode enabled. Binaural models receive binaural audio rendered from the same FOA recordings with the HRTF pipeline of our human study. JAEGER (Video) combines the Qwen2.5-Omni video processor with JAEGER's FOA encoder, while JAEGER (RGB-D) uses the middle frame of each clip with depth estimated by Depth Pro. As a single-frame input cannot represent the multiple candidate videos of Joint Matching, JAEGER (RGB-D) is not evaluated on that task.

Results on SAVU-Bench

Table R1. Accuracy (%) on SAVU-Bench under cyclic evaluation; ± = 1.96 bootstrap standard errors (approximate 95% intervals; 10,000 resamples of questions). "–": task not supported by the input interface.

ModelAudio inputA-Ground.V-Ground.Sem. Align.Spa. Align.Joint A2VJoint V2ASpatial Reas.
Chance–50.050.050.050.033.333.342.2
Qwen3.8-Omni-FlashFOA90.4 ± 5.497.9 ± 1.856.7 ± 7.250.0 ± 7.148.7 ± 5.845.9 ± 4.248.4 ± 10.2
Spatial-Omni SO-30BFOA70.2 ± 5.183.7 ± 4.949.0 ± 7.950.7 ± 8.021.4 ± 3.528.8 ± 3.243.8 ± 10.1
JAEGER (Video)FOA49.7 ± 5.362.6 ± 6.449.0 ± 4.749.3 ± 7.232.8 ± 1.033.7 ± 1.741.7 ± 5.2
JAEGER (RGB-D)FOA49.7 ± 5.356.8 ± 6.649.7 ± 3.548.0 ± 7.6––40.6 ± 6.2
Spatial-Omni SO-4BFOA49.1 ± 6.9––––––
BATBinaural78.9 ± 5.3––––––
MiDashengLM-SpatialBinaural91.0 ± 4.2––––––
Qwen3-Omni-30BMono48.8 ± 7.187.1 ± 4.555.0 ± 6.346.7 ± 7.321.7 ± 4.030.4 ± 1.839.1 ± 8.0

The new results are consistent with the three main observations in our manuscript and refine the first one:

(1) Qwen3-Omni-30B, which can only process mono audio, still shows the system-level bottleneck in auditory spatial perception that we observed for the other general-purpose models. Spatial-Omni SO-4B and JAEGER show a pattern similar to Spatial-Omni SO-7B; their low accuracy may be because they are trained primarily on simulated spatial audio, consistent with a simulation-to-real gap on real recordings. The other new models refine our previous analysis: with stronger spatial-audio processing, MiDashengLM-Spatial, Qwen3.8-Omni-Flash, BAT and Spatial-Omni SO-30B achieve much higher accuracy on Audio Grounding.

(2) Although some models improve substantially on perception tasks, their alignment performance remains poor, and all models still struggle with Spatial Reasoning. These results are consistent with the second and third observations in our manuscript: current models still struggle with cross-modal spatial integration and spatial reasoning.

Comparison with Humans

Table R2. Accuracy (%) of the new models on the human-evaluation subset (30 questions per task; single pass). Chance is the mean of 1/K over questions.

ModelAudio inputA-Ground.V-Ground.Sem. Align.Spa. Align.Joint A2VJoint V2ASpatial Reas.
Chance–50.050.050.050.033.333.345.8
Qwen3.8-Omni-FlashFOA80.0100.063.350.053.350.033.3
Spatial-Omni SO-30BFOA73.393.350.050.020.020.053.3
Spatial-Omni SO-4BFOA56.7––––––
JAEGER (Video)FOA46.756.750.040.033.336.736.7
JAEGER (RGB-D)FOA46.766.756.740.0––36.7
BATBinaural73.3––––––
MiDashengLM-SpatialBinaural86.7––––––
Qwen3-Omni-30BMono50.093.356.753.333.340.033.3
HumanBinaural89.393.386.771.766.765.385.0

On the human-evaluation subset, the new models remain below humans on every task except Visual Grounding. The gap is small on Audio Grounding (86.7 vs. 89.3) and large on Semantic Alignment (63.3 vs. 86.7), Spatial Alignment (53.3 vs. 71.7), Joint Matching (53.3 / 50.0 vs. 66.7 / 65.3) and Spatial Reasoning (53.3 vs. 85.0). Spatial-audio input thus closes most of the gap on Audio Grounding, but not on the tasks that relate sound to vision. This also validates our analysis.

SAVU-Diag Results

Table R3. SAVU-Diag outcomes (%) under cyclic evaluation. A prerequisite counts as correct only if it is answered correctly under all of its orderings; G1 (A1) requires all grounding (alignment) prerequisites of a reasoning question to be correct.

ModelReasoning correctError: G0A1Error: G1A0Error: G0A0Error: G1A1
Qwen3.8-Omni-Flash48.47.313.526.04.7
Spatial-Omni SO-30B43.837.00.014.64.7
JAEGER (Video)41.71.61.655.20.0
JAEGER (RGB-D)40.68.92.148.40.0
Qwen3-Omni-30B39.124.01.631.83.6

As Table R3 shows, even for the strongest spatial-audio model, most reasoning errors involve a failed prerequisite: for Qwen3.8-Omni-Flash, 64.6% of reasoning errors involve a failed grounding prerequisite and 76.8% a failed alignment prerequisite; for Spatial-Omni SO-30B, 91.7% involve a failed grounding prerequisite. Strong aggregate Audio Grounding therefore does not mean that the specific spatial facts a reasoning question needs are recovered reliably.

Revisions to the Manuscript

We are grateful to all reviewers for their suggestions, which have made our analysis more complete and robust. Based on the new results and the reviewers' comments, we have revised the manuscript, with all changes marked in blue. The main revisions are:

  • New models and analyses (Sec. 4): experiments with the new models and the corresponding analyses, including the main results (Table 2), a human comparison with all evaluated models (Table 3), and the complete SAVU-Diag results under cyclic evaluation (Table 4).
  • Presentation: a reorganized presentation of the main findings, together with a more representative SAVU-Diag case (Fig. 3).
  • Appendix: new sections on model selection and setup (A.1.2), Joint Matching (A.1.3), the oracle condition and human evaluation with mono audio (A.1.4), and the sound-event distribution (A.2.1).
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all reviewers for their time, thoughtful reviews, and constructive feedback. We are pleased that the reviewers recognized several key strengths of our work:

  • A real-recorded benchmark: all reviewers recognized the value of a benchmark built from real-world spatial audio-visual recordings, in contrast to existing benchmarks that rely on synthetic or simulated audio.
  • Scene-linked diagnosis with SAVU-Diag: Reviewers 25Vr and Leyv highlighted SAVU-Diag as "a meaningful contribution beyond leaderboard evaluation" (Reviewer 25Vr) that enables "fine-grained analysis beyond aggregate task scores" (Reviewer Leyv).

During the discussion, the main concerns were addressed:

  • Spatial-audio input (Reviewers Leyv and RCCA): we added six models with native FOA or binaural input. Spatial-audio input raises Audio Grounding accuracy to about 90%, while cross-modal alignment and spatial reasoning remain challenging for all models, which sharpens our main finding that the key bottleneck lies in cross-modal spatial integration.
  • Baseline coverage (Reviewer 25Vr): we stated our model selection criteria and evaluated Qwen3-Omni-30B, whose results are consistent with our conclusions.
  • Task taxonomy and Joint Matching (Reviewer 25Vr): we clarified the progressive task design, and additional experiments showed that the concatenated input format does not account for the Joint Matching results.
  • Robustness of SAVU-Diag (Reviewer Leyv): we applied cyclic evaluation to SAVU-Diag and reported the complete results, together with an analysis of the remaining errors in the oracle condition.

We are glad that our responses addressed all concerns of Reviewers 25Vr and Leyv, both of whom maintained their positive recommendations. Although Reviewer RCCA has not yet responded to our rebuttal, we believe these results address their concerns.

We have incorporated the reviewers' feedback into the revised manuscript and will further refine it in the final version. Thank you again for your time and effort in helping us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.