acceptodds
Withdrawn from ICLR 2027

Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Audio carries phase relations that encode where sources are, yet existing large audio-language models (LALM) reduce it to single-channel content for stability of general audio understanding, leaving essential spatial information for answering questions about direction, distance, and source relations unused. We present SpheraAudio, which accepts recordings with any channel count — mono, stereo, First-Order Ambisonics (FOA), or higher — and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens that are interleaved with audio tokens at corresponding temporal positions, while we propose Spatial Multimodal RoPE (SMRoPE), which makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages — spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning — progressing from geometric supervision to language-based spatial decisions, with answers trained as an explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Considering that current benchmarks tie their questions to a fixed channel-count format and score answers without spatial reasoning evidence, we introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. On spatial tasks, it supports high-precision localisation, tracking, and reasoning over static and moving sound sources; beyond them, the model preserves strong general-audio capabilities. We will release the code, the SpheraQA benchmark, and its evaluation metrics to support reproducibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial audio, audio-language models, ambisonics, spatial reasoning, audio understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3YtQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the problem of spatial sound understanding with audio LLMs. It proposes the addition of a multi-format (mono, binaural, FOA) spatial audio encoder to a Qwen3-Omni-based LLM backbone. It follows prior work such as Spatial-Omni by having two audio encoders, a mono semantic encoder and a multichannel spatial encoder, and interleaves each spatial token with its paired semantic audio token, using a newly proposed SMRoPE positional encoding for the interleaving.

The paper presents a multi-stage training scheme (encoder pretraining followed by projector training followed by join SFT with the LLM and then GRPO) which uses a newly proposed benchmark, SpheraQA, for evaluation. SpheraQA contains 20k questions spanning audio perception, spatial relations, and spatial reasoning. Experimental results show that the proposed model generally outperforms Spatial Omni 7B and Gemini 3.1 Pro on the proposed benchmark, as well as the SO-Bench and TWNM benchmarks.

Critical strengths

-The paper studies an important and timely problem (spatial sound understanding with LLMs)

-The paper proposes a new model for spatial sound understanding, which seems to outperform prior models

-The paper also introduces a new spatial audio encoder which can handle multiple channel formats (stereo binaural and FOA) which is a useful innovation

Critical weaknesses
  • The evaluation conditions unfairly favor the proposed SpheraAudio model. SpheraAudio is fine-tuned with in-distribution spatial CoT data from the same generator used to construct SpheraQA test, but the baseline models are applied to SpheraQA in a zero-shot fashion. Additionally, Gemini 3.1 Pro receives only the mono W channel (so it cannot access the directional cues the tasks require). A fairer comparison would include Spatial-Omni or another baseline model fine-tuned on the same SpheraQA training data, as well as other very close architectural competitors (PhaseCoder, ST-AudioLM, BAT)

  • Some ablations are invalid. For example, Table 4c switches the RoPE type at inference (rather than training) on a model trained with SMRoPE

  • Regarding the benchmark itself, the "Spatial Reasoning" category contains only 120 questions (making statistical significance testing necessary), and according to appendix B, ~12k of the questions in the benchmark (more than half) are derived from only 168 STARSS23 recordings, so the majority of the evaluation samples are not independent from one another

  • Some of the baseline performances are strange. Gemini on mono beats SpheraAudio on lateral position (44.6 vs 41.5) and motion status (73.9 vs 73.4), which should be impossible for true lateral cues unless the questions can be solved semantically, or the metrics are noisy.

  • There is only a very vague discussion of what training data was used to train the model, and the majority of the important details about this are missing.

Questions for authors
  • Can you present experimental results in Table 1 for baseline models fine-tuned on identical data as SpheraAudio?
  • Can you state whether any SO-Bench or TWNM training splits were used in SFT/RL for SpheraAudio?
  • Can you describe what data was used to train the model? Please provide a table that details the training data used for each stage of training, including at minimum the data sources and number of examples/total duration of audio.
  • What are your release plans for the training data?
Reviewer_Fc9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpheraAudio, an audio-language model that combines a content encoder with a spatial encoder through temporally interleaved tokens and Spatial Multimodal RoPE (SMRoPE). Different recording formats are routed through format-specific preprocessing into a fixed four-component spatial interface, while mono inputs use the original content pathway. Training combines geometric pretraining, spatial chain-of-thought supervision, and outcome-based reinforcement learning. The paper also introduces SpheraQA and reports improvements on several spatial-audio QA benchmarks.

Critical strengths
  1. Unified encoding of multichannel spatial audio is an important, unresolved problem for LALMs.
  2. The proposed SMRoPE is an innovative approach to encoding temporal correspondence and modality identity.
  3. SpheraQA covers a diverse range of spatial audio evaluation tasks.
Critical weaknesses
  1. The “Any Channel Count” claim is overly broad and misleading. The spatial encoder uses a fixed four-component interface rather than directly encoding arbitrary spatial audio formats end to end. Equation (5) also lacks justification, as binaural (L-R)/2 is not equivalent to the FOA Y channel.

  2. Support for microphone-array and HOA inputs is insufficiently validated. The main text provides no dedicated results for arbitrary microphone-array inputs, nor does it analyze the impact of discarding higher-order HOA components, leaving the claim of HOA support insufficiently justified.

  3. Sim-to-real generalization is insufficiently analyzed. Although SpheraQA includes LOCATA and STARSS23, the paper does not adequately analyze the final model's sim-to-real generalization.

  4. A controlled comparison across input formats is missing. For at least one task, performance differences should be reported while holding all other task settings fixed and varying only the simulator’s output format (mono, binaural, FOA, HOA, and microphone array).

  5. Documentation and presentation are insufficient. Some relevant work is missing, e.g. JAEGER[1]. The simulator, simulation pipeline, and distributions of azimuth, elevation, distance, source count, and trajectory duration are unspecified. Figures 1 and 2 are raster images rather than vector graphics.

[1] Liu Z, et al. JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments. In Proc. ICML, 2026.

Questions for authors
  1. Is there prior literature or experimental evidence supporting the use of Equation (5)'s mid–side representation as a shared input with native FOA?
  2. Could you add microphone-array experiments to the main text and analyze how discarding higher-order HOA components affects model performance?
  3. Could you provide a breakdown of SpheraQA results on simulated and real recordings and analyze the model's sim-to-real generalization?
  4. For at least one task, could you hold the scene and task settings fixed, vary only the simulator's output format (mono, binaural, FOA, HOA, and microphone array), and report the performance differences?
  5. Could you evaluate on SpatialSceneQA's Audio DoA and Overlap Audio DoA tasks from JAEGER, reporting azimuth and elevation errors in degrees?
  6. Could you provide the simulator and data-generation pipeline details, along with the distributions of azimuth, elevation, distance, source count, and trajectory duration, as requested in Weakness 5?

If the authors could address my concerns, I am happy to raise my score.

Reviewer_H8RNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SpheraAudio, a spatial audio understanding model that encodes spatial audio in first-order ambisonics (FOA) format into a mono and a spatial audio token sequence. The mono audio encoder uses only one of the channels of the spatial audio, while the spatial encoder uses all four channels of the FOA audio. The time-synchronized mono and spatial tokens are interleaved and fed to the LLM backbone. The text query is prepended to this token sequence, and the LLM backbone produces text answers to the user query. Along with this new model, the paper introduces the SpheraQA benchmark to evaluate spatial audio understanding models. The benchmark consists of simple spatial audio perception tasks along with spatial relation understanding and reasoning tasks. The model is evaluated on several existing benchmarks and on the new benchmark.

Critical strengths

The paper introduces the idea of having synchronized spatial and mono audio tokens and interleaving them. This token sequencing scheme could be useful for audio-in, audio-out models that need to understand spatial audio and speak back to the user. This kind of real-time interaction might be harder to achieve with a scheme in which the spatial and mono tokens are concatenated. That being said, the advantages of this interleaving scheme are not sufficiently demonstrated in the paper and are not backed by results.

The authors show results on a newly introduced benchmark as well as existing benchmarks, and compare against other spatial audio understanding models and Gemini 3.1 Pro.

Critical weaknesses
  • Training on idealized synthetic data contradicts the paper's own critique of prior work. In the related work section, PhaseCoder is criticized for assuming free-floating omnidirectional microphones, whereas real arrays exhibit more complex wave effects such as scattering and reflections, so that identical microphone coordinates can result in distinct direction- and frequency-dependent responses. This is accurate; however, the authors then train their model on synthetic data that uses ideal first-order ambisonic responses without any physical array non-idealities or real physical array modeling, not even free-floating microphone array modeling. Furthermore, how room acoustics and reverberation were simulated is not explained.
  • Support for arbitrary and irregular microphone arrays is neither characterized nor evaluated. Performance on arbitrary and possibly irregular microphone arrays is neither quantified nor broken down in the evaluations. Microphone arrays encountered in real life, such as those on smart glasses, mobile phones, or robot heads, make spatial audio understanding and sound localization much more challenging than more favorable arrays such as the 32-microphone spherical Eigenmike array (e.g., from the STARSS23 dataset). While the evaluation includes recordings from LOCATA, it is never mentioned which LOCATA microphone arrays were used or how the model performs across different array geometries. Relatedly, the term Ec(f)E_c(f) in Equation 4 can be interpreted as a linear transform that maps an arbitrary microphone array signal to ambisonics, and the appendix mentions an external array-to-ambisonics encoding for calibrated arrays, but how this transformation is performed, the non-idealities or artifacts it introduces, and its effect on end-to-end results are not characterized. The minimum number of microphones and the microphone topologies for which the methodology works or does not work are likewise neither characterized nor discussed.
  • Higher-order ambisonic and multi-microphone inputs are reduced to first-order ambisonics, discarding spatial information. The paper claims to support higher-order ambisonic (HOA) signals (e.g., 9 channels for second-order or 16 channels for third-order ambisonics) and multi-microphone arrays with arbitrary channel counts. However, as shown in Table 6, the model reduces all inputs to the first four channels (first-order ambisonics) and discards the higher-order channels. Whereas an encoder such as PhaseCoder directly encodes all channels from a 9-microphone array, reducing a 9-microphone array or higher-order ambisonics down to four first-order ambisonic channels discards the higher-order spatial channels, resulting in a significant loss of spatial precision and angular resolution.
  • The advantages of interleaving spatial and mono tokens are overstated and not backed by results. The interleaving of time-synchronized spatial and mono audio tokens is a good idea that could be particularly useful for streaming models in the future, and its advantages are described in the manuscript. However, this advantage is not actually leveraged, and I see no evidence of these advantages in the evaluations or results. The paper frames Sci-Phi and PhaseCoder's prepending of spatial tokens to the audio token sequence as a poor design that interleaving solves, yet while token interleaving is compared against element-wise addition in Table 4a, there is no ablation comparing interleaving against prepending or concatenating the spatial and mono token sequences; the results could have been similar with simple concatenation. There are not even any evaluation tasks that predict the start and end times of sound activity. The significance of this sequencing is overstated without quantitative results comparing it to concatenation.
  • Insufficient detail on training data, spatial encoder pretraining, and benchmark construction renders the work unreproducible. How the training data and audio mixtures were prepared is not explained well, and the same goes for the SpheraQA evaluation set. The spatial encoder pretraining described in Appendix A.3 and in the manuscript body lacks substantial detail and is not reproducible. For SpheraQA, there is no explanation of how the recordings from the real datasets (LOCATA and STARSS23) were extracted; given the small number of recordings used from the real datasets, there must have been a selection process, but the details are not provided.
  • Spatial encoder pretraining models only a single sound source per time frame. Based on the pretraining methodology, the spatial encoder is supervised to predict only a single sound source (one azimuth, elevation, and distance) within a given time frame. If there are multiple active sound sources within the same time frame, the pretraining loss does not model them. This is a notable limitation, as real acoustic scenes often involve multiple overlapping sources, such as cocktail-party scenes with overlapping speakers. Meanwhile, the audio CoT supervision in Appendix A.5 (Optimization) asks the LLM to count distinct sound sources that can overlap in time, so the LLM is trained to operate with multiple sources while the spatial encoder is not. Although the encoder is later fine-tuned end-to-end with the LLM, the paper does not explain how the spatial encoder pretraining handles overlapping sources or how the encoder adapts to resolve multiple simultaneous sources.
  • The benchmark omits spatial speech transcription and offers limited novelty. The evaluations in the benchmark exclude practically relevant tasks such as spatial speech transcription with word error rate (WER) measurements. Various other manuscripts already explore tasks such as spatial perception, motion understanding, spatial relation understanding, and spatial reasoning (e.g., Spatial-Omni), so the evaluation tasks introduced in the benchmark offer minimal research novelty over prior work.
  • Evaluation relies entirely on thresholded QA accuracy. The evaluations comparing SpheraAudio against Spatial-Omni-7B and BAT (e.g., Table 1 and Table 3) rely entirely on thresholded QA accuracy percentages and task scores. While whole-answer accuracy offers a high-level overview, converting continuous spatial outputs into binary pass/fail decisions using fixed tolerances obscures the true error distribution and physical resolution of the models. The authors should also report continuous physical error metrics for the baseline models, such as Mean Absolute Error (MAE) in degrees for azimuth and elevation and in meters for distance, and, for categorical tasks, additional classification metrics such as F1 scores.
  • No evaluation of unconditioned joint sound event detection and localization. There is no evaluation where the user asks the model to identify the sound sources and their directions in the scene without conditioning, and the model answers with something like "the dog is barking from the left, and there is a male voice speaking to the right." Instead, the prompt conditions the model on the source identity (e.g., "What is the location of the dog barking?") or on the location of the source. This conditioning makes the task easier, and the simple question of identifying all sound sources and their locations is not evaluated.
  • The temporal correspondence and positional encoding ablations are uninformative. The temporal correspondence experiment in Table 4b shifts only the spatial stream at test time, which breaks the alignment between the audio and spatial tokens that the model was trained to expect, whereas shifting both streams together preserves their pairing. All this establishes is that misaligning the token streams at inference time in a way the model was not trained to handle drops accuracy, which is an obvious result. Similarly, the SMRoPE vs. Sequential RoPE ablation in Table 4c applies Sequential RoPE strictly at inference time to a model trained with SMRoPE. Because the attention weights were optimized under SMRoPE's coordinate scheme during SFT, swapping the positional encoding at test time acts as a severe out-of-distribution perturbation and does not demonstrate that a model trained natively with Sequential RoPE would be inferior. Neither experiment adds anything meaningful to the paper.
  • The SFT vs. RL comparison is selectively reported. In Section 5.5 (Table 5), the impact of reinforcement learning (GRPO) over supervised fine-tuning (SFT) is evaluated on only 10 selected task subsets from TWNM, stereo transfer, and SO-Bench, while entirely omitting any SFT vs. RL comparison on SpheraQA and on the remaining tasks in TWNM and SO-Bench. Reporting gains on only a handful of selected subsets appears to be selective reporting. To properly validate the benefit of RL, the authors should provide a complete comparison between the SFT and RL checkpoints across all evaluation benchmarks.
  • The waveform-analysis agent experiment in Appendix F is out of place and unfair. It is theoretically impossible to discern sound directions from single-channel audio, so it is no surprise that the agents provide incorrect answers to direction-based questions. Unlike a mono-only audio LLM, a coding agent can easily load multichannel audio and run signal processing or open-source sound perception tools, yet the agent was given only single-channel audio. In addition, the exact tools and prompt provided to the agent are not specified. This section seems like an unfair and incomplete attempt to provide another baseline, and it distracts from the rest of the manuscript.
Questions for authors
  • Array-to-ambisonics transformation. The term Ec(f)E_c(f) in Equation 4 can be interpreted as a linear transform mapping an arbitrary microphone array signal to ambisonics, and Appendix A.1 mentions an external array-to-ambisonics encoding for calibrated arrays. What specific preprocessing does this methodology involve to transform arbitrary array signals to ambisonics? Are the details of this encoding provided? If no assumptions are made about the preprocessing methodology, are the effects of different transformation methodologies on the end-to-end results discussed? Are the non-idealities or artifacts introduced by this transformation discussed, and are the effects of encoding different arbitrary microphone arrays on end-to-end performance characterized?
  • Microphone count and topology limitations. What is the minimum number of microphones for which this methodology would work? What are the limitations in terms of microphone topologies, and what kinds of topologies work or do not work?
  • Training and pretraining data preparation. How were the training datasets prepared? Was the training performed using only synthetic data, or was any real data included in the mix? What room simulator and reverberation parameters were used to simulate the rooms? What was the size of the training dataset, and how many unique examples did it contain? I assume that mono audio sources were spatialized to produce the spatial audio examples; where were the mono audio sources obtained from, and which open-source mono audio datasets were used to create the spatial audio datasets for both SFT and spatial encoder pretraining? For scenes with multiple sources, how were the sources mixed, what level of overlap was allowed, and what kinds of amplitude or delay augmentations were used?
  • Spatial speech transcription. Can the model spatially transcribe a single speech source, for example, transcribing only speech coming from a certain direction, or speech that is near or far?
  • Gemini 3.1 Pro results on direction-dependent tasks. In Table 1, Gemini 3.1 Pro performs best on the lateral position and motion direction tasks despite consuming only single-channel audio. Given the lack of multichannel audio cues, it is not possible for the model to discern movement along the azimuth and elevation axes. How, then, does this model perform best on these tasks? Is this because there is an insufficient number of motion direction examples that require understanding motion in the azimuth and elevation directions rather than movement closer and farther away? The same question applies to the relative position task: how does Gemini 3.1 Pro perform best on it, given that it cannot discern the aforementioned directions? This requires deeper analysis and discussion.
  • Catastrophic forgetting under full-parameter fine-tuning. The model employs full-parameter fine-tuning across the encoders, projectors, and language backbone, yet retains performance on several mono audio understanding tasks. How is catastrophic forgetting mitigated? The paper mentions including general-audio instruction instances in the SFT mixture, but this is underspecified: what kinds of general-audio instruction instances are in the SFT mixture, and in what proportions?
  • Spatial encoder behavior with overlapping sources. Given that the spatial encoder pretraining supervises only a single source per time frame while the LLM is trained to count distinct, possibly overlapping sound sources (Appendix A.5), how does the spatial encoder adapt to represent multiple overlapping sources? I would like to see a breakdown of the relevant evaluation metrics across segments of the evaluation data stratified by source overlap (e.g., from less to more overlap). How does the model perform when there is no overlap versus when there is heavier overlap?
  • Real-recording selection and input configuration for SpheraQA. How were the recordings from LOCATA and STARSS23 selected from the datasets? Which microphone arrays from LOCATA were used? For the evaluation datasets (especially LOCATA and TWNM), which specific input format and front-end operation from Table 6 were used to feed the spatial encoder?
  • Answer parsing for SpheraQA tasks. Which of the questions in Table 12 result in a numerical or categorical answer that can be directly parsed, versus which ones result in free-form answers that require processing with GPT-4o to extract the predicted values?
  • Waveform-analysis agent setup. Why was the waveform-analysis agent in Appendix F given only single-channel audio instead of the multichannel audio, and what exact tools and prompt were provided to the agent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.