acceptodds
Withdrawn from ICLR 2027

Empowering Large Audio-Language Models via Spatial Question and Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial audio understanding requires models to connect sound identities with locations, motion, and relations over time. Most large audio-language models (ALMs), however, process monaural audio and cannot directly access the inter-channel cues available in spatial recordings. We introduce SonicSpatial, a parameter-efficient framework that enhances the spatial understanding capabilities of pretrained ALMs. A question-conditioned spatial selection module uses learnable queries to compress dense features from a frozen spatial audio encoder into compact spatial tokens. These spatial tokens are then combined with the backbone's semantic audio tokens and question tokens as input to the LLM for answer generation. We also introduce SonicSpatial-Bench, comprising 3,045 multiple-choice questions over 576 recorded and simulated FOA audio instances across eight capability categories, including source motion, temporal relations, listener-centered transformations, and room-scale inference. On this benchmark, SonicSpatial with MOSS-Audio-8B achieves 60.5% overall accuracy, improving on its backbone by 22.0 percentage points; the Qwen2.5-Omni variant achieves 63.1%, improving on its backbone by 25.4 points. The two variants reach 64.3% and 65.4%, respectively, on our multiple-choice adaptation of SO-Bench. The spatial selection module can be integrated into diverse ALM backbones without modifying their audio encoders, yielding consistent gains across five evaluated models. These results demonstrate the broad applicability of SonicSpatial as a parameter-efficient spatial adaptation framework.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial audio understanding, large audio-language models, spatila audio benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RSHVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SonicSpatial adds spatial reasoning to audio-language models that normally process only mono audio. The original encoder reads the FOA W channel, while frozen GRAM features all four channels. A two-stage, question-conditioned Q-Former selects 32 spatial tokens for the LLM. Only the selector and LoRA weights are trained.

Critical strengths
  1. Building the labels from metadata and then re-checking them with a separate program after the LLM rewrite is a good practice. Removing samples whose meaning changes after rewriting is a nice touch.

  2. The benchmark covers things earlier spatial audio benchmarks mostly skip, such as longer clips (around 36 s on average), ego-rotation, and room size.

  3. The adapter is applied to five different ALM backbones and gives gains on all of them, which supports the plug-in claim.

  4. The efficiency point in the ablation is useful: 32 query tokens get close to the accuracy of passing all 4k+ spatial tokens, with roughly 3x faster training steps and half the inference latency.

Critical weaknesses
  1. Novelty of the module is limited. Question-conditioned Q-Former queries are already used in InstructBLIP and many video LLMs. Using a frozen spatial encoder and a Q-Former is a reasonable engineering choice, but the paper should be more modest about it. The ablations also do not test the central claim of "question conditioning." I would like to see: (a) the same queries without question conditioning, (b) different numbers of queries, and (c) a different spatial encoder in place of GRAM.

  2. The comparison is not apples to apples. SonicSpatial is trained on 22k questions made by the same generator, with the same templates, the same answer balancing, and the same STARSS23 and FOA-MEIR sources as the benchmark. All the baselines in Table 2 are zero-shot. A large part of the 20+ point gain could come from learning the question format, option style and answer priors, not from spatial hearing. The W-channel-only fine-tuning ablation in Figure 3 is the right control, but its numbers only appear in a bar chart. Please put the exact numbers for "W-only fine-tune" in a table next to Table 2

  3. Possible leakage and distribution overlap. STARSS23 train and test splits are split by recording, but do they share the same physical rooms and the same sound classes? Also, what was GRAM pretrained on? If GRAM saw STARSS23 or the same simulated rooms, that should be stated. For SO-Bench, the model is trained on 10k samples from SO-Dataset, so SO-Bench is also in-distribution for SonicSpatial but not for most baselines. The multiple-choice conversion of SO-Bench was done by an LLM, and the quality of those distractors is not checked.

Questions for authors
  1. What are the exact numbers for W-only fine-tuning on both benchmarks and all 8 categories?

  2. What input format did each baseline get?

  3. Is there any overlap in rooms, sound sources, or recording sessions between train and test in STARSS23?

  4. How well does the model do on recorded clips versus simulated clips, reported separately?

  5. Did you check performance on standard non-spatial audio benchmarks after training?

Reviewer_ZBqMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the problem of spatial sound understanding (using first order ambisonics or FOA audio) with LLMs. Rather than simply understanding the semantic content of an input sound (identify what sound is happening in a clip, transcribe the speech, etc.) like most audio-enabled LLMs, spatial sound understanding equips the LLM with multi-microphone audio input and tasks it with also inferring the direction, distance, movement, etc. of sound sources in a 3-D environment.

The paper makes two contributions:

  1. A dataset/benchmark of 3k evaluation examples and 22k training examples. The source waveforms are derived either from existing spatial audio datasets (STARSS23) or with commonly-used spatial audio simulators (SoundSpaces 2.0), and the authors create the question+answer pairs to accompany these waveforms.

  2. A model architecture that follows the standard recipe established by prior work for spatial audio-enabled LLMs (spatial audio encoder + adapter/projector + LoRA fine-tuning of the LLM backbone) but adds in a text-conditioned selector module. From my perspective, this effectively replaces the spatial audio encoder's adaptor/projector (which would normally preserve all of the spatial audio tokens as input to the LLM) with a Q-former that uses the input text as the query tokens (concatenated with an extra set of learnable query tokens)

The paper makes experimental comparisons to a range of audio LLMs and shows that the proposed model achieves higher accuracy on the proposed benchmark than these models, although it should be noted that all but one of these baseline models only process mono audio or were not explicitly trained to handle spatial/FOA audio data.

Critical strengths
  • The paper works on a timely and interesting research problem that is still relatively underdeveloped (lacking in large-scale datasets for training and evaluation, etc.)

  • The paper adds a couple of new question types to its benchmark (Ego-rotation, room acoustics) that prior benchmarks did not consider

  • The proposed text-conditioned audio token selection mechanism shows some promising improvements in computational efficiency

Critical weaknesses
  • The paper's experimental rigor is not up to the standard of ICLR in multiple ways:
  1. Although the paper acknowledges 8 different prior works (in Table 1) which each propose their own new spatial audio benchmark and model, when it comes to the experiments this paper basically ignores the majority this prior work. It does not evaluate its own proposed model on any of the benchmarks introduced by these prior papers, and it only evaluates 1 out of these 8 prior models on the benchmark proposed in this paper.

  2. All but 1 of the baseline models compared against in Table 2 are not meant to be capable of processing spatial audio input. They either were not trained/benchmarked on it, and/or can only take as input mono-channel audio. The fact that the proposed model's performance numbers are all bolded in Table 2 does not matter if the baseline models are effectively straw men.

  3. The proposed evaluation dataset is small enough that I am concerned about the statistical significance of the results; please include confidence intervals and significance testing.

Questions for authors

-How does your proposed dataset and model offer a tangible, objective improvement over the previously proposed spatial audio understanding datasets and models enumerated in Table 1? Currently, the paper does not sufficiently compare how the prior models perform on the proposed benchmark, how the proposed model performs on the prior benchmarks, and what practical new benefit the proposed dataset brings to the community.

-Are the experimental results statistically significant?

Reviewer_izREOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores how to improve LLM understanding of First order ambisonics (FOA) spatial audio. While the topic itself is not new, numerous papers worked on this concept, the paper has some novel ideas. The contributions of the paper are a two-fold, First, SonicSpatial encoder model that combines different approaches together to compress the spatial tokens seen by the LLM. The SonicSpatial compresses the dense spatial features from the off-the-shelf FOA encoder using Spatial Selection Module to select and combine relevant text and spatial features. Overall, I think that is an interesting concept worth exploring, as raw spatial tokens (or features) can be quite dense, taking up a large part of the LLMs context window. The second contribution is the SonicSpatial-Bench dataset, a spatial QA dataset derived from STARSS23 and synthetic data. There are not a lot of spatial audio eval datasets, so it is always good to see new eval datasets for future research to hill-climb on. The SonicSpatial model is validated on SonicSpatial-Bench dataset and show improvement over mono audio LLMs baselines.

Critical strengths

There is a lack of established eval datasets in the field of spatial audio LLMs. So the dataset that the authors composed could be very useful for the field. I think its valuable that it is composed on top of the popular STARSS23 recorded dataset, so there is room for modifications and future versions. The dataset has a range of diverse questions and seems to be well thought out by the authors.

The approach of using SSM (Spatial Selection Module) is interesting, and new to spatial audio encoding, as far as I know. So, there is value in learning about this approach. It seems to be effective in reducing the number of input tokens to the LLM by compressing spatial tokens, while keeping the performance metrics the same.

Critical weaknesses

The claim that the SSM makes spatial audio processing more efficient is not fully supported with baselines and metrics. While the SSM compressions approach reduces the number of tokens and inference time for LLM, it adds another model to the system, which adds compute. The authors should provide the overall numbers if possible, and not just the LLM. I imagine that the benefits of SSM might scale differently with the length of the audio input: as shorter audio might not get as much benefits. It would be interesting to see the tradeoffs between inputting spatial tokens directly into the LLM and SSM approach. I think digging deeper into efficiency is important, as SSM does not appear to provide other benefits (such as accuracy) in comparison to using dense spatial embeddings, while greatly increasing the complexity of the LLM. Intuitively, I would expect that SSM would improve performance on SonicSpatial-Bench, as compressed inputs are usually more effective. Also there are other potential issues, such as inability to make a steaming model. To strengthen the efficiency claims the authors could compare compute and accuracy numbers on SonicSpatial-Bench with other published FOA spatial encoders (e.g., Sci-Phi). The current baselines are non-spatial audio models, which understandably will not perform well on spatial audio. Also, as I understand, they were not finetuned on SonicSpatial-Bench. To better understand that, I would finetune without STARSS (perhaps on synthetic data only), and see the performance on STARSS derived QA set.
The authors could have also tried different audio encoders, and tried to finetune with drop-in replacement for GRAM. There are few other spatial FOA encoders.

Questions for authors

Does the SSM module approach work on a fixed length input ? Does it support steaming?

Why use GRAM as FOA encoder vs. other published or self-trained encoders?

Authors only compared with non-spatial audio models in Table 2. It would be interesting to see how does the proposed encoder compares to already published spatial encoders

Are there speech transcripts in the datasets? It would be valuable to have transcripts

Does SonicSpatial-Bench have overlapping speech and if so how well does the encoder work with overlapping speech?

What are the random baseline numbers for the SonicSpatial -Bench ?

Can the authors show ablation without learnable queries? I am having difficult time understanding the mechanics of how they benefit SonicSpatial

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.