- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents SEA-LM, a Spatial Audio Understanding model. First, it introduces FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. It trains a Multimodal Large Language Model to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, it introduces a Spatio-temporal Weighted Cross-Entropy Loss. On the evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.
Critical strengths
- The paper introduces a data generation and encoding pipeline that incorporates the device physics of variable-count and variable-position head-worn microphone arrays alongside Ambisonic Signal Matching beamforming to convert these signals to FOA.
- It introduces SEA-LM, an MLLM that jointly performs acoustic, semantic, and spatial reasoning while distinguishing wearer speech from bystander speech in multi-source environments, together with a Spatio-temporal Weighted Cross-Entropy objective that prioritizes sparse direction and timestamp tokens during SFT.
- It evaluates SEA-LM without array-specific retraining across 1,211 configurations derived from two smart-glasses geometries, spanning 4 to 9 microphones and varied microphone positions.
Critical weaknesses
- The whole paper is framed around smart-glasses wearable devices, yet all training and testing from start to finish use simulated shoebox rooms and stationary sound sources, without testing a single audio clip recorded on real smart glasses or testing moving wearers and moving sources. Frame noise, head movement, and complex acoustic reflections in real wearable use are not captured by this simulation.
- The paper claims to be stable across 1,211 microphone configurations, but Appendix H shows that all 1,211 configurations are just microphone subsets drawn from two smart-glasses models, and both training and testing randomly sample from this exact same pool. This only shows that the model memorized the array subsets seen during training and does not prove it works on a new unseen pair of glasses. Also, the 5 and 5 weights in the weighted cross-entropy loss were only chosen by validation loss search in Appendix O, and neither the main text nor the appendix reports downstream localization error and transcription WER for standard unweighted cross-entropy with weights 1 and 1, so I cannot tell how much this weighted loss actually helps.
Questions for authors
- If a subset of array configurations or one of the two smart-glasses geometries is completely held out from training, how much do localization and transcription performance drop on unseen array topologies?
- Under the exact same training setup, what are the downstream localization MAE, tIoU, and transcription WER of SEA-LM when trained with standard unweighted cross-entropy using weights 1 and 1?
2 comments
Request to review
Dear reviewer eNRU,
As the author-reviewer discussion phase is coming to an end, we would like to invite you to review our rebuttal and share your views, and let us know if it helped resolve your doubts and if you have any follow-up questions.
Rebuttal
Thank you for the review. We address both weaknesses and both questions below.
W1. Simulation only. We agree the evaluation is simulated, and Sec. 7 states this. Our claims are scoped to controlled comparisons on held-out simulated data (abstract: "On our evaluation set"); the contribution is a device-realistic simulation and encoding recipe and a model trained with it. One clarification on what the simulation covers: head and frame acoustic scattering are modeled, through BEM transfer functions computed on CAD models of the glasses fitted on a head-and-torso model (Sec. 3.2.1, App. F), together with the artifacts of converting these signals to FOA. Head movement, moving sources, structure-borne frame noise and non-shoebox room geometry are not modeled, and the final version's limitations will name each.
No public corpus supports the full task, which needs a wearable array with known transfer functions (to compute ASM), per-source 3D direction relative to the wearer, wearer/bystander labels, and event boundaries and transcripts for both speech and non-speech. For the closest candidates:
The real glasses datasets would therefore test zero-shot cross-device transfer, which we do not claim. The final version's Sec. 7 will name EasyCom as the first real-recording target.
W2a. Configurations are shared between training and test. Correct, and by design: the target scenario is one model serving every valid microphone subset of a known device (product variants, microphones disabled to save power, occluded or failed channels). Test scenes use disjoint source audio and disjoint simulated RIRs (Sec. 4.1). We agree this does not show the model works on an unseen pair of glasses; App. L says so, and the final abstract will read "a single checkpoint serving 1,211 microphone subsets (4-9 microphones) of two simulated glasses frames".
On memorization: the geometry mapping is not learned. ASM weights are computed in closed form from each configuration's transfer functions (Eq. 4), so FOACoder sees a common FOA format and needs only to be robust to the residual artifacts that differ across configurations (low-pass coloration, Z-channel attenuation; App. C). Fig. 8 shows performance is not significantly affected from 4 to 9 microphones, across the range of those artifacts in the pool. Prior spatial-audio LLMs either assume a fixed input format (binaural in BAT and OWL; native FOA in Sci-Phi and SPUR) or, as in PhaseCoder, accept arbitrary microphone positions without modeling device or head acoustics. To our knowledge, SEA-LM is the first spatial-audio LLM evaluated with a single checkpoint across a large set of wearable-array configurations whose signals include simulated head and frame scattering and FOA-conversion artifacts. Whether this extends to subsets or devices outside the pool is untested, and the final version's Sec. 7 will say so.
W2b. Weighted cross-entropy. The weight sweep (Sec. 3.3.4) included the uniform (1,1) setting, alongside (1,5), (5,1), (5,5) and (10,10). It ran in Stage 1, with selection on validation loss measured as uniform cross-entropy so the runs are directly comparable, and (5,5) reached lower uniform cross-entropy than (1,1), even though (1,1) optimizes that loss directly. Because a sweep through the full two-stage curriculum is several times more expensive, we selected the weights in Stage 1 and carried that choice into Stage 2. The final version will state this selection procedure in Sec. 3.3.4.
Q1. Holding out a geometry. We designed the study around the modeled device family: one checkpoint serving every valid microphone subset of a known device, with geometry normalized analytically by ASM. Holding out an entire design asks a different question, cross-device transfer, which requires retraining FOACoder (2M steps on 8 × H100) and both SFT stages on the remaining design. The final version's Sec. 7 will list held-out geometries as the first item of future work.
Q2. Unweighted cross-entropy. The uniform (1,1) setting was part of the Stage-1 sweep, where (5,5) reached lower uniform cross-entropy than (1,1) (see W2b); the full curriculum was then trained with (5,5).