- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. It further introduces SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, the paper introduces SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit.
Critical strengths
- The paper introduces SAVU-Bench, a real-world benchmark curated from six diverse data sources that evaluates spatial audio-visual understanding across three capability levels including Perception, Alignment, and Reasoning and seven evaluation tasks.
- It introduces SAVU-Diag, a scene-linked diagnostic set that connects reasoning questions with their prerequisite grounding and alignment tasks, enabling fine-grained analysis beyond aggregate task scores.
- It evaluates 12 representative audio-visual models, showing that similar reasoning accuracy can arise from different prerequisite-error patterns, and introduces SAVU-EA, a training-free baseline showing that explicit spatial cues improve auditory grounding and joint matching.
Critical weaknesses
- Out of the 12 evaluated models, 11 do not support multichannel spatial audio input at all and are only fed mono audio. Mono audio already loses inter-channel time and level differences, so the models naturally cannot tell sound directions. The main table's claim that existing models are weak at audio spatial grounding is largely because they were fed mono audio, while Spatial-Omni is the only tested model that actually supports multichannel spatial audio input.
- The diagnostic setup in SAVU-Diag has a flaw: answering a standalone prerequisite localization question correctly does not prove that the model actually used that location information when answering the later reasoning question. In addition, the main benchmark uses cyclic permutation evaluation to avoid option-order bias, whereas SAVU-Diag only tests the default option order once, so random single-pass fluctuations can skew the failure attribution ratios.
Questions for authors
- If cyclic permutation evaluation is also applied to the prerequisite and reasoning questions in SAVU-Diag, how much do the failure attribution ratios in Figure 3 change, such as the 61.4% localization error rate for OmniVideo-7B?
- In Appendix A.1.3, even after directly giving the ground-truth localization and alignment facts to Gemini-3.6-Flash, it still gets 18 out of 81 reasoning questions wrong. What are the main causes of error in those 18 questions, such as perspective conversion or multi-step relational reasoning?
3 comments
Q2 (errors given ground-truth prerequisites): We manually inspected the step-by-step rationale Gemini-3.6-Flash produced for each of the 18 errors and assigned each error to one of four categories, according to the first step at which the reasoning departs from a valid derivation (Table R9).
Table R9. Error categories of Gemini-3.6-Flash on the 18 reasoning questions answered incorrectly despite ground-truth prerequisite facts.
| Error category | Cases | Representative observation |
|---|---|---|
| Sound-source binding | 6 | The model attributed footsteps from outside the field of view to a visible person and used that person's position instead of the supplied acoustic location. |
| Perspective / reference-frame conversion | 4 | The model carried a left–right relation from the recording frame into the imagined observer's frame without accounting for the change in viewpoint. |
| Multi-step relational reasoning / ordering | 4 | The model inferred an order between two objects from the fact that both were to the right of the same reference object, which does not determine their relative order. |
| Additional spatial attributes required | 4 | The question also depends on front–back position, orientation, or the identity of the target source, which go beyond the left–right relations supplied in the oracle condition. |
The most frequent failure is sound-source binding (6/18): even when the acoustic location is explicitly supplied, the model tends to bind a sound to a salient visible entity and reason from that entity's position. The remaining errors are evenly split among reference-frame conversion, multi-step relational reasoning, and questions that require additional spatial attributes. We have added this analysis to our paper.
W2 (prerequisite correctness vs. evidence use): We agree that SAVU-Diag is correlational: it shows which prerequisite failures co-occur with reasoning failures, but a correctly answered standalone prerequisite does not prove that the model used it. Two observations test evidence use directly. (1) Oracle prerequisites: supplying the correct grounding and alignment facts raises Gemini-3.6-Flash from 49.4% to 77.8% on the 81 reasoning questions (single pass), so the model does use this information when it is given explicitly. (2) Conversely, the six sound-source binding errors in Table R9 are cases in which the correct location was supplied but not used.
In addition, under the cyclic protocol, where a prerequisite counts as correct only if it is answered correctly under every ordering, 10 of 17 models still fail on reasoning instances whose prerequisites are all correct (G1A1; up to 14.7% of reasoning errors for DOTS-3-Note-Preview and 9.1% for Qwen3.8-Omni-Flash). Even robustly acquired evidence is therefore not always composed correctly.
Dear Reviewer Leyv,
Thank you again for your valuable feedback. We have posted a detailed response and a revised manuscript to address your concerns.
As the discussion phase will close in about one day, we would greatly appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.
Best regards, The Authors
Thank you for the constructive review. Results shared across reviewers (Tables R1–R3) are in the global response.
W1 (spatial input availability): We agree that mono input limits the interpretation of the audio grounding results. We have added six models with native FOA or binaural input; their setup and results are given in the global response (Tables R1–R3). These results supplement the observations in our manuscript. Spatial-audio input substantially improves Audio Grounding, to about 90% for the strongest models (MiDashengLM-Spatial 91.0%, Qwen3.8-Omni-Flash 90.4%), which confirms your point for this task.
The tasks that relate sound to vision, however, remain challenging even with spatial input: the strongest spatial-audio model, Qwen3.8-Omni-Flash, reaches 50.0% on Spatial Alignment (chance 50%) and 48.7% on Joint A2V (chance 33.3%), and on the human-evaluation subset the new models remain below humans on every task except Visual Grounding. We thank the reviewer for this suggestion, which has made our analysis more complete and robust.
Q1 (cyclic permutation in SAVU-Diag): Thank you for this suggestion. We now evaluate both the reasoning questions and their prerequisites under cyclic permutation, counting a prerequisite as correct only if it is answered correctly under all of its orderings (Table R8).
For OmniVideo-7B, the share of reasoning errors with grounding-only failures (G0A1) changes from 61.4% to 21.1%, while errors with both grounding and alignment failures (G0A0) account for 76.3%. For most models, G0A0 becomes the most frequent pattern, and models with the same reasoning accuracy still show distinct profiles: at 43.8%, 65.7% of Spatial-Omni SO-30B's reasoning errors are G0A1, compared with 38.0% for MiniCPM-o-4.5 and 11.1% for Qwen3.5-Omni-Plus (Tables R3 and R8).
The cyclic results reinforce our finding that most reasoning errors co-occur with prerequisite failures, while models with the same accuracy differ in which prerequisites they fail. The complete SAVU-Diag results under cyclic evaluation are now reported in Table 4 (Sec. 4.3) of the revised paper, together with an updated qualitative case (Fig. 3).
Table R8. SAVU-Diag outcomes (%) for the original models under cyclic evaluation (same definitions as Table R3). Reasoning correct equals the Spatial Reasoning accuracy in Table 2 of the paper.