- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SonicSpatial adds spatial reasoning to audio-language models that normally process only mono audio. The original encoder reads the FOA W channel, while frozen GRAM features all four channels. A two-stage, question-conditioned Q-Former selects 32 spatial tokens for the LLM. Only the selector and LoRA weights are trained.
Critical strengths
-
Building the labels from metadata and then re-checking them with a separate program after the LLM rewrite is a good practice. Removing samples whose meaning changes after rewriting is a nice touch.
-
The benchmark covers things earlier spatial audio benchmarks mostly skip, such as longer clips (around 36 s on average), ego-rotation, and room size.
-
The adapter is applied to five different ALM backbones and gives gains on all of them, which supports the plug-in claim.
-
The efficiency point in the ablation is useful: 32 query tokens get close to the accuracy of passing all 4k+ spatial tokens, with roughly 3x faster training steps and half the inference latency.
Critical weaknesses
-
Novelty of the module is limited. Question-conditioned Q-Former queries are already used in InstructBLIP and many video LLMs. Using a frozen spatial encoder and a Q-Former is a reasonable engineering choice, but the paper should be more modest about it. The ablations also do not test the central claim of "question conditioning." I would like to see: (a) the same queries without question conditioning, (b) different numbers of queries, and (c) a different spatial encoder in place of GRAM.
-
The comparison is not apples to apples. SonicSpatial is trained on 22k questions made by the same generator, with the same templates, the same answer balancing, and the same STARSS23 and FOA-MEIR sources as the benchmark. All the baselines in Table 2 are zero-shot. A large part of the 20+ point gain could come from learning the question format, option style and answer priors, not from spatial hearing. The W-channel-only fine-tuning ablation in Figure 3 is the right control, but its numbers only appear in a bar chart. Please put the exact numbers for "W-only fine-tune" in a table next to Table 2
-
Possible leakage and distribution overlap. STARSS23 train and test splits are split by recording, but do they share the same physical rooms and the same sound classes? Also, what was GRAM pretrained on? If GRAM saw STARSS23 or the same simulated rooms, that should be stated. For SO-Bench, the model is trained on 10k samples from SO-Dataset, so SO-Bench is also in-distribution for SonicSpatial but not for most baselines. The multiple-choice conversion of SO-Bench was done by an LLM, and the quality of those distractors is not checked.
Questions for authors
-
What are the exact numbers for W-only fine-tuning on both benchmarks and all 8 categories?
-
What input format did each baseline get?
-
Is there any overlap in rooms, sound sources, or recording sessions between train and test in STARSS23?
-
How well does the model do on recorded clips versus simulated clips, reported separately?
-
Did you check performance on standard non-spatial audio benchmarks after training?