acceptodds
Under review as a conference paper at ICLR 2027

SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays

Abstract

Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Audio, Large Audio Language Model, Spatial Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eNRUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents SEA-LM, a Spatial Audio Understanding model. First, it introduces FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. It trains a Multimodal Large Language Model to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, it introduces a Spatio-temporal Weighted Cross-Entropy Loss. On the evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.

Critical strengths
  • The paper introduces a data generation and encoding pipeline that incorporates the device physics of variable-count and variable-position head-worn microphone arrays alongside Ambisonic Signal Matching beamforming to convert these signals to FOA.
  • It introduces SEA-LM, an MLLM that jointly performs acoustic, semantic, and spatial reasoning while distinguishing wearer speech from bystander speech in multi-source environments, together with a Spatio-temporal Weighted Cross-Entropy objective that prioritizes sparse direction and timestamp tokens during SFT.
  • It evaluates SEA-LM without array-specific retraining across 1,211 configurations derived from two smart-glasses geometries, spanning 4 to 9 microphones and varied microphone positions.
Critical weaknesses
  1. The whole paper is framed around smart-glasses wearable devices, yet all training and testing from start to finish use simulated shoebox rooms and stationary sound sources, without testing a single audio clip recorded on real smart glasses or testing moving wearers and moving sources. Frame noise, head movement, and complex acoustic reflections in real wearable use are not captured by this simulation.
  2. The paper claims to be stable across 1,211 microphone configurations, but Appendix H shows that all 1,211 configurations are just microphone subsets drawn from two smart-glasses models, and both training and testing randomly sample from this exact same pool. This only shows that the model memorized the array subsets seen during training and does not prove it works on a new unseen pair of glasses. Also, the 5 and 5 weights in the weighted cross-entropy loss were only chosen by validation loss search in Appendix O, and neither the main text nor the appendix reports downstream localization error and transcription WER for standard unweighted cross-entropy with weights 1 and 1, so I cannot tell how much this weighted loss actually helps.
Questions for authors
  1. If a subset of array configurations or one of the two smart-glasses geometries is completely held out from training, how much do localization and transcription performance drop on unseen array topologies?
  2. Under the exact same training setup, what are the downstream localization MAE, tIoU, and transcription WER of SEA-LM when trained with standard unweighted cross-entropy using weights 1 and 1?
2 comments

Thank you for the review. We address both weaknesses and both questions below.

W1. Simulation only. We agree the evaluation is simulated, and Sec. 7 states this. Our claims are scoped to controlled comparisons on held-out simulated data (abstract: "On our evaluation set"); the contribution is a device-realistic simulation and encoding recipe and a model trained with it. One clarification on what the simulation covers: head and frame acoustic scattering are modeled, through BEM transfer functions computed on CAD models of the glasses fitted on a head-and-torso model (Sec. 3.2.1, App. F), together with the artifacts of converting these signals to FOA. Head movement, moving sources, structure-borne frame noise and non-shoebox room geometry are not modeled, and the final version's limitations will name each.

No public corpus supports the full task, which needs a wearable array with known transfer functions (to compute ASM), per-source 3D direction relative to the wearer, wearer/bystander labels, and event boundaries and transcripts for both speech and non-speech. For the closest candidates:

DatasetArraySupportsWhat prevents a faithful evaluation
STARSS2332-ch spherical Eigenmike, near-ideal FOATasks 1-2 on its 13 classesNo wearer, no transcripts, many moving sources; idealized FOA is the regime our pipeline moves away from
EasyCom4 frame mics + 2 in-ear mics; measured ATFs releasedTasks 4-6 (speech only)An array FOACoder never saw; continuous head motion; up to four tracked partners vs. at most two external sources in training; no labeled non-speech events
MMCSG (CHiME-8)Aria glasses, 7 mics; anechoic ATFs releasedTask 5; Task 4 without localizationNo partner direction labels; an array FOACoder never saw

The real glasses datasets would therefore test zero-shot cross-device transfer, which we do not claim. The final version's Sec. 7 will name EasyCom as the first real-recording target.

W2a. Configurations are shared between training and test. Correct, and by design: the target scenario is one model serving every valid microphone subset of a known device (product variants, microphones disabled to save power, occluded or failed channels). Test scenes use disjoint source audio and disjoint simulated RIRs (Sec. 4.1). We agree this does not show the model works on an unseen pair of glasses; App. L says so, and the final abstract will read "a single checkpoint serving 1,211 microphone subsets (4-9 microphones) of two simulated glasses frames".

On memorization: the geometry mapping is not learned. ASM weights are computed in closed form from each configuration's transfer functions (Eq. 4), so FOACoder sees a common FOA format and needs only to be robust to the residual artifacts that differ across configurations (low-pass coloration, Z-channel attenuation; App. C). Fig. 8 shows performance is not significantly affected from 4 to 9 microphones, across the range of those artifacts in the pool. Prior spatial-audio LLMs either assume a fixed input format (binaural in BAT and OWL; native FOA in Sci-Phi and SPUR) or, as in PhaseCoder, accept arbitrary microphone positions without modeling device or head acoustics. To our knowledge, SEA-LM is the first spatial-audio LLM evaluated with a single checkpoint across a large set of wearable-array configurations whose signals include simulated head and frame scattering and FOA-conversion artifacts. Whether this extends to subsets or devices outside the pool is untested, and the final version's Sec. 7 will say so.

W2b. Weighted cross-entropy. The weight sweep (Sec. 3.3.4) included the uniform (1,1) setting, alongside (1,5), (5,1), (5,5) and (10,10). It ran in Stage 1, with selection on validation loss measured as uniform cross-entropy so the runs are directly comparable, and (5,5) reached lower uniform cross-entropy than (1,1), even though (1,1) optimizes that loss directly. Because a sweep through the full two-stage curriculum is several times more expensive, we selected the weights in Stage 1 and carried that choice into Stage 2. The final version will state this selection procedure in Sec. 3.3.4.

Q1. Holding out a geometry. We designed the study around the modeled device family: one checkpoint serving every valid microphone subset of a known device, with geometry normalized analytically by ASM. Holding out an entire design asks a different question, cross-device transfer, which requires retraining FOACoder (2M steps on 8 ×\times H100) and both SFT stages on the remaining design. The final version's Sec. 7 will list held-out geometries as the first item of future work.

Q2. Unweighted cross-entropy. The uniform (1,1) setting was part of the Stage-1 sweep, where (5,5) reached lower uniform cross-entropy than (1,1) (see W2b); the full curriculum was then trained with (5,5).

AuthorsOpenReview ↗

Request to review

Dear reviewer eNRU,

As the author-reviewer discussion phase is coming to an end, we would like to invite you to review our rebuttal and share your views, and let us know if it helped resolve your doubts and if you have any follow-up questions.

Reviewer_QspfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors point out that existing audio-language models rely primarily on single-channel (mono) audio, which limits their capacity for spatial awareness. To address this, they propose 'SEA-LM,' a model designed for spatial audio understanding. Specifically, the model first employs 'FOACoder' to encode First-Order Ambisonics (FOA) data captured from various microphone configurations. Furthermore, it utilizes a two-stage curriculum learning approach encompassing six distinct tasks to train a Multimodal Large Language Model (MLLM) to interpret spatial audio embeddings. Finally, the authors introduce a 'Spatio-temporal Weighted Cross-Entropy loss' to emphasize spatial coordinates and temporal boundaries during the supervised fine-tuning phase. Evaluation results demonstrate that SEA-LM achieves superior performance and maintains consistent robustness across diverse microphone array configurations, such as those found in smart glasses.

Critical strengths

The paper explicitly addresses head/device scattering and artifacts introduced when converting irregular wearable microphone arrays into FOA, while distinguishing wearer speech from external sources. Incorporating these factors into data generation and representation learning addresses meaningful challenges that idealized spatial-audio inputs do not capture. This makes the formulation relevant to wearable assistants that must understand both their surroundings and their users.

Critical weaknesses
  1. Insufficient evidence isolating the contributions of the proposed components
  • The author set up an experiment using SLDNet as one of the baselines. Although both models (SLDNet baseline and SEA-LM) use the same synthetic data and training curriculum for downstream SFT, this stage updates only the projector and LLM LoRA adapters while keeping the spatial encoders frozen. As stated in Appendix D.4, SELDNet was trained on STARSS23 and was not adapted to the authors’ synthetic domain, whereas FOACoder was pretrained on that domain. Thus, the comparison does not isolate the benefit of the proposed encoder design from the benefit of domain-specific encoder pretraining.

  • Moreover, although the spatio-temporal weighted loss is presented as a key contribution, its benefit is not demonstrated through reported downstream comparisons against uniform cross-entropy. Other choices, such as adding a raw reference channel, are also not evaluated separately. Consequently, the results demonstrate the effectiveness of the complete system but do not sufficiently establish which proposed components drive the improvements.

  1. Limited validation beyond the authors’ synthetic dataset.
  • All evaluations are conducted on the authors’ own synthetic dataset, with no validation on independent external data or real-world recordings. Although source audio and room impulse responses are separated between training and testing, both splits are generated using the same simulation pipeline. External datasets such as EasyCom, which provides real egocentric multichannel recordings with speech transcriptions and participant poses, and STARSS23, which provides real spatial recordings with sound-event and localization annotations, offer potential avenues for evaluating supported subsets of the proposed capabilities, subject to appropriate preprocessing and task alignment. Furthermore, the 1,211 microphone configurations are derived from only two smart-glasses geometries, with no evaluation on held-out array topologies. Thus, transfer to real wearable environments and unseen hardware remains unverified, despite being central to the paper’s motivation.
Questions for authors
  1. Controlled encoder comparison: Have you evaluated a SELDNet encoder adapted to the same synthetic domain before downstream SFT? If available, please report this comparison to clarify whether FOACoder’s advantage persists when encoder pretraining domains are matched.

  2. Evidence for the weighted objective: What are the downstream localization, temporal grounding, and transcription results with uniform cross-entropy versus the proposed spatio-temporal weighting, holding other settings fixed? These results would help establish the contribution of a central proposed component.

  3. Evidence for generalization: Evaluation of supported tasks on datasets such as EasyCom or STARSS23, with appropriate preprocessing and task alignment, would help assess transfer beyond the proposed simulation pipeline. If such evaluation is infeasible, what specific input or annotation incompatibilities prevent it?

  4. Clarification of source counts: Section 5 and Appendix N state that Stage 2 uses scenes containing one to four sound sources, whereas Section 3.2 describes one wearer and two external sources, and Appendix I explicitly limits each generated scene to three sources. What source-count distribution was actually used in Stage 2, and how were any four-source scenes generated?

3 comments

Thank you for a precise review; each question maps onto a specific concern, and we answer them in order.

W1 / Q1. Controlled encoder comparison. Agreed: with SELDNet frozen and trained only on STARSS23, the comparison mixes encoder design with pretraining domain and capacity. We built SELDNet + Gemma-4 from the DCASE 2024 SELDNet checkpoint, following Sci-Phi's frozen-SELDNet baseline, with the same LLM, data and curriculum as SEA-LM. To our knowledge, the open SELD checkpoints, including larger pretrained ones such as PSELDNets, are trained on FOA from spherical arrays or idealized simulation rather than wearable arrays, so any of them carries the same domain gap. The baseline's numbers show this gap: azimuth MAE of 80.6∘80.6^\circ is close to the monaural model's 87.5∘87.5^\circ and to the ∼90∘\sim 90^\circ expected from uninformed guesses. An encoder trained on idealized FOA passes almost no direction on glasses-derived FOA, which is the gap our device-realistic training targets.

The comparison the paper does control is against the fine-tuned monaural Gemma, which shares SEA-LM's LLM, audio tower, training data and two-stage curriculum and lacks only the spatial pathway (Secs. 4.3 and 5). Against it, SEA-LM reduces Task 1 azimuth/elevation MAE from 87.5∘87.5^\circ/27.7∘27.7^\circ to 14.9∘14.9^\circ/12.8∘12.8^\circ, Task 6 external-source hallucination from 27.2% to 8.0%, and Task 6 WER from 38.1% to 32.5%. The final version will present the SELDNet row as an off-domain diagnostic and claim only that an encoder trained with egocentric objectives on device-realistic data gives the LLM usable spatial features, not that FOACoder's AST-style backbone outperforms a domain-matched SELD encoder.

W1 / Q2. Evidence for the weighted objective. We swept the weights in Stage 1 over (1,1), (1,5), (5,1), (5,5) and (10,10) and chose (5,5), which gave the lowest validation loss measured as uniform cross-entropy, so the runs are directly comparable. In that sweep, (5,5) reached lower uniform cross-entropy than (1,1), even though (1,1) optimizes that loss directly. Because a sweep through the full two-stage curriculum is several times more expensive, we selected the weights in Stage 1 and carried that choice into Stage 2. The final version will state this selection procedure in Sec. 3.3.4.

W1. Raw reference channel. App. C and Fig. 3 motivate it: ASM low-pass filters most reconstructed channels, so the raw channel restores high-frequency content at the encoder input. The final version will describe it as a design choice motivated by this analysis.

W2 / Q3. Generalization and the specific incompatibilities. Our claims are scoped to controlled comparisons on held-out simulated data (abstract: "On our evaluation set"). No public corpus supports the full task, which needs a wearable array with known transfer functions (to compute ASM), per-source 3D direction relative to the wearer, wearer/bystander labels, and event boundaries and transcripts for both speech and non-speech. For the datasets raised:

DatasetArraySupportsWhat prevents a faithful evaluation
STARSS2332-ch spherical Eigenmike, near-ideal FOATasks 1-2 on its 13 classesNo wearer, no transcripts, many moving sources; idealized FOA is the regime our pipeline moves away from; the SELDNet baseline encoder was trained on it
EasyCom4 frame mics + 2 in-ear mics; measured ATFs releasedTasks 4-6 (speech only)An array FOACoder never saw; continuous head motion; up to four tracked partners plus an untracked server vs. at most two external sources in training; no labeled non-speech events
MMCSG (CHiME-8)Aria glasses, 7 mics; anechoic ATFs releasedTask 5; Task 4 without localizationNo partner direction labels; an array FOACoder never saw

EasyCom and MMCSG would test zero-shot cross-device transfer. Our claim is narrower: one checkpoint serves every valid microphone subset of a modeled device, with geometry normalized in closed form by ASM (Eq. 4); Fig. 8 shows stable performance from 4 to 9 microphones. The final version will state this scope in the abstract ("a single checkpoint serving 1,211 microphone subsets (4-9 microphones) of two simulated glasses frames"), and its Sec. 7 will name EasyCom as the first real-recording target. Our simulation targets the device side of the sim-to-real gap (BEM transfer functions of the frame on a head-and-torso model, plus ASM artifacts), which FOA-based audio LLMs omit.

Q4. Source counts. Thank you for catching this. Each generated scene has at most three sources: the wearer plus up to two external sources (App. I). "One to four" in Sec. 5 and App. N is a typo; Stage 2 uses scenes with one to three sources. The final version will correct both passages and relabel the "3+" cohorts in App. L as "3".

I thank the authors for their response.

The planned revisions clarify the scope of the generalization claims (W2/Q3) and resolve the source-count inconsistency (Q4).

However, the response does not provide a domain-matched SELDNet comparison (W1/Q1), downstream task results for uniform versus weighted cross-entropy (W1/Q2). As the final version could include additional results, these would help substantiate the contributions of the individual components.

The monaural comparison supports the overall spatial pathway, but does not isolate the contribution of FOACoder.

I therefore decided to maintain my rating.

AuthorsOpenReview ↗

Response to reviewer comment

We thank the reviewer for the follow-up and are glad the scope and source-count points are resolved. We would like to clarify the two remaining points.

Isolating FOACoder. Besides the monaural comparison, Table 2 contains a controlled encoder swap. SELDNet + Gemma-4-E4B-IT and SEA-LM share the LLM, native audio tower, training data, two-stage curriculum and trainable/frozen split (projector in Stage 1; projector and LoRA in Stage 2; spatial encoder frozen throughout; Sec. 5). The only component that differs is the spatial encoder. Replacing SELDNet with FOACoder reduces Task 1 azimuth/elevation MAE from 80.6∘80.6^\circ/27.4∘27.4^\circ to 14.9∘14.9^\circ/12.8∘12.8^\circ, raises Task 2 query F1 from 76.2% to 94.9%, and lowers the Task 4 missing-source rate from 16.1% to 1.8% and the hallucination rate from 10.9% to 4.5%. This isolates the downstream contribution of FOACoder as trained, that is, its input representation and egocentric objectives on device-realistic data.

We also note how this baseline was obtained. To our knowledge, no publicly released spatial audio encoder is trained on wearable-array signals or provides wearer voice activity; the open SELD checkpoints are trained on FOA from spherical arrays or idealized simulation. Rather than report an off-the-shelf model, we built SELDNet + Gemma-4-E4B-IT ourselves, following Sci-Phi's frozen-SELDNet design, and trained its spatial projector and LoRA adapters through SEA-LM's full two-stage curriculum on the same data. Retraining an external encoder on a new array domain is not a standard baseline protocol for spatial-audio LLMs, nor for multimodal LLMs more broadly, where open- and closed-source baselines are compared as released rather than retrained on matched data and compute. Our objective is a spatial-audio LLM that works on wearable arrays, and this comparison shows that an existing encoder does not transfer to glasses-derived FOA even when integrated with the same LLM and trained through the same curriculum and data (Task 1 azimuth MAE of 80.6∘80.6^\circ versus 14.9∘14.9^\circ for SEA-LM).

A domain-matched SELDNet would also not be an architecture-only control. SELDNet takes four-channel FOA features, has no wearer-VAD output, and has 0.74M parameters; adapting it to this task means adding the raw reference channel, a wearer-VAD objective and over 60 times more capacity, which are exactly FOACoder's design choices. FOACoder's contribution is this encoder design for egocentric glasses audio: a five-channel input that restores the high-frequency content ASM removes (App. C), joint SELD and wearer-VAD objectives, and training on device-realistic data. At the encoder level, it reduces angular error from 44.01∘44.01^\circ to 15.06∘15.06^\circ, raises localization recall from 32.29% to 86.50%, and reaches 98.53% wearer-VAD F1 (Table 3), and the encoder swap above shows that the localization gains carry through to the downstream tasks.

Weighted cross-entropy. The weighting targets token dilution: direction and timestamp values occupy a small fraction of the target tokens, and that fraction shrinks further once Stage 2 adds dense transcripts. Prior audio-language work likewise finds uniform cross-entropy insufficient for sparse timestamp tokens. TAC (Kumar et al., 2026) up-weights timestamp tokens in the cross-entropy, with a weight of 5 as in our setting, because standard cross-entropy treats them equally with semantic tokens. TEMPO (Kulkarni et al., EMNLP 2026) adds an auxiliary timestamp loss because token-level cross-entropy penalizes all incorrect timestamps equally. We selected the weights by comparing (1,1), (1,5), (5,1), (5,5) and (10,10) on validation loss measured as uniform cross-entropy. (5,5) reached lower uniform cross-entropy than (1,1), even though (1,1) optimizes that loss directly, which indicates that up-weighting the sparse spatio-temporal tokens improves the fit to the full target sequence rather than trading it away. The final version will describe this selection procedure in Sec. 3.3.

Reviewer_qz5rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SEA-LM, an ego-centric spatial audio understanding framework for wearable microphone arrays. The proposed system consists of two main components: FOACODER, a spatial audio encoder that converts variable count and position microphone-array recordings from smart glasses into FOA-based spatial embeddings, and SEA-LM, a multimodal large language model that combines these spatial embeddings with a conventional monaural audio encoder. FOACODER is trained with wearer VAD and SELD objectives, while SEA-LM is trained on six spatial-audio understanding tasks covering sound localization, temporal grounding, source attribution, and both holistic and targeted transcription. The authors also introduce a spatio-temporal weighted cross-entropy loss to increase the training signal for sparse spatial and temporal tokens.\
The system is trained primarily on a large synthetic dataset generated using realistic head- and device-scattering simulations and approximately 1,211 microphone configurations derived from two smart-glasses geometries. On the held-out synthetic evaluation set, SEA-LM substantially outperforms monaural and SELDNet-based baselines on most localization, temporal, hallucination, and transcription metrics. The results demonstrate the potential of integrating spatial audio representations into audio-language models for ego-centric scene understanding.

Critical strengths
  1. Comprehensive ego-centric spatial audio understanding
    Comprehensive task formulation. SEA-LM goes beyond conventional sound localization by jointly addressing localization, temporal grounding, wearer-versus-bystander attribution, and source-selective transcription within a single MLLM framework. This is a meaningful step toward practical ego-centric spatial audio understanding, since wearable systems need to jointly reason about what is present, where it originates, when it occurs, and whether speech originates from the wearer or from the surrounding environment.

  2. Realistic microphone-array simulation and strong controlled results
    Realistic simulation and strong empirical improvements. The authors make a substantial effort to model practical wearable-array effects, including head and device scattering and the non-idealities introduced by converting irregular microphone arrays into FOA through Ambisonic Signal Matching. The resulting system shows large improvements over the evaluated baselines, while substantially reducing hallucinated and missing sources in transcription tasks.

Critical weaknesses
  1. Generalization to unseen hardware is not convincingly demonstrated
    The claimed hardware/layout generalization is not sufficiently validated. Although the paper evaluates SEA-LM across 1,211 microphone configurations, these configurations are derived from only two underlying smart-glasses geometries. Thus, the large number of configurations primarily represents different microphone subsets and placements within two form factors, rather than 1,211 independent hardware geometries. In particular, the experiments do not evaluate the model on an entirely unseen wearable form factor. Consequently, it remains unclear whether the proposed geometry-flexible representation can generalize to genuinely unseen wearable hardware, which is an important requirement for the broader hardware-generalization claim.

  2. Deployment feasibility is insufficiently evaluated
    The proposed system is motivated by wearable devices such as smart glasses, but the computational requirements and deployment feasibility of the complete SEA-LM system are not sufficiently characterized. The system combines a large multimodal LLM with a 46.18M-parameter FOACODER spatial encoder using a 30-second context. Although the paper discusses a 1-second FOACODER variant with reduced inference cost and latency, the actual compute, memory footprint, latency, and hardware requirements of the complete SEA-LM system are not reported. Without such analysis, it is difficult to assess whether the proposed system is practically deployable in the intended wearable or edge-computing setting.

Questions for authors

Q1. Unseen hardware and real-world generalization
Could the authors provide an evaluation on microphone geometries or wearable form factors that are completely held out during training and used only for evaluation? Such an experiment would more directly test the claimed generalization to unseen hardware configurations. In addition, could the authors evaluate SEA-LM on a real-world wearable dataset, such as the MMCSG Dataset (https://ai.meta.com/datasets/mmcsg-dataset), to assess its robustness to real acoustic environments and the sim-to-real gap? These experiments would help clarify whether the reported performance extends beyond the simulated configurations and form factors used during training.

Q2. Deployment feasibility
Could the authors provide more detailed information about the computational requirements and deployment feasibility of the complete SEA-LM system? Specifically, it would be helpful to report the parameter count, FLOPs/MACs, and memory footprint of the full pipeline, including both FOACODER and the language model. Additionally, could the authors clarify whether the intended deployment is fully on-device or assumes an external edge/cloud compute platform? This information would help assess the practical feasibility of SEA-LM for the proposed wearable setting.

Q3. Baseline comparisons
The current experiments compare FOACODER primarily against SELDNet, while FOACODER has substantially greater model capacity (46.18M parameters versus 0.74M for SELDNet). Could the authors provide comparisons with more recent and stronger SELD architectures under a comparable training protocol? This would help determine whether the observed improvements are attributable to the proposed geometry-flexible spatial representation or primarily to the substantially larger encoder capacity.

1 comment

Thank you for the careful summary and for recognizing the task formulation and the device-realistic simulation. We respond to each point below.

W1 / Q1. Unseen hardware and real data (MMCSG). We agree the 1,211 configurations are microphone subsets of two glasses frames, not 1,211 devices, and the paper does not claim transfer to an unseen form factor (App. L; Sec. 7). The final version will revise "layout-flexible" and "robust across 1,211 smart-glasses array configurations" to "a single checkpoint serving 1,211 microphone subsets (4-9 microphones) of two simulated glasses frames".

The claim we make is narrower: one model serves every valid microphone subset of a modeled device (product variants, microphones disabled to save power, occluded or failed channels) without per-configuration retraining. Geometry normalization is not learned. ASM weights are computed in closed form from each configuration's transfer functions (Eq. 4), so FOACoder sees a common FOA format and must only be robust to configuration-dependent residual artifacts (App. C); Fig. 8 shows performance is stable from 4 to 9 microphones. Prior spatial-audio LLMs either assume a fixed input format (binaural in BAT and OWL; native FOA in Sci-Phi and SPUR) or, as in PhaseCoder, accept arbitrary microphone positions without modeling device or head acoustics. To our knowledge, SEA-LM is the first spatial-audio LLM evaluated with a single checkpoint across a large set of wearable-array configurations whose signals include simulated head and frame scattering and the artifacts of converting them to FOA.

On real data, our claims are scoped to held-out simulated data. MMCSG provides wearer/partner transcripts on Aria glasses but no partner direction labels, so only Task 5 and the non-spatial part of Task 4 could be scored, and on an array FOACoder never saw; it would test zero-shot cross-device transfer rather than the claim we make. The final version's Sec. 7 will list unseen-hardware and real-recording evaluation as the first open problems.

W2 / Q2. Deployment feasibility. SEA-LM is meant to run on a server or tethered host, like other multimodal LLMs that serve wearables; the glasses capture and stream audio. We chose Gemma-4-E4B because it is the largest Gemma 4 variant with a pretrained audio encoder (the 26B and 31B variants take no audio input), not for on-device execution. The split falls naturally at ASM: its weights are fixed per configuration, so the device can apply it locally and stream a fixed 5-channel signal regardless of microphone count. Analytical estimates for one 30-s clip:

ComponentParametersPrefill compute
FOACoder46.2M≈\approx 28 GFLOPs (2 ×\times 46.2M ×\times 300 frames)
Spatial projector≈\approx 7.9M (512 →\rightarrow 2560 →\rightarrow 2560)negligible
Gemma-4-E4B with its $\sim$305M audio encoder4.5B effective; 8B with per-layer embeddings≈\approx 10 TFLOPs over $\sim$1,100 input tokens
Rank-128 LoRA adaptersrank 128 on attention and FFN projectionsnone once merged into the base weights

Weights total about 16 GB in bf16, so inference fits on a single data-center GPU. The spatial pathway's main cost is its 300 extra LLM input tokens, about a quarter of prefill; FOACoder itself is under 1% of prefill compute. The 1-s FOACoder in App. D matters for streaming latency of the spatial pathway on the server, not for on-glasses execution. The final version will add a deployment paragraph and these estimates.

Q3. Stronger SELD baselines and capacity. We built SELDNet + Gemma-4 from the DCASE 2024 SELDNet checkpoint, following Sci-Phi's frozen-SELDNet baseline, with the same LLM, data and curriculum as SEA-LM. Larger open checkpoints exist (e.g., PSELDNets), but to our knowledge all are trained on FOA from spherical arrays or idealized simulation rather than wearable arrays, so swapping one in would raise capacity while keeping the same domain gap. We agree it does not separate encoder design from domain and capacity; its numbers show the domain gap: azimuth MAE of 80.6∘80.6^\circ is close to the monaural model's 87.5∘87.5^\circ and to the ∼90∘\sim 90^\circ expected from uninformed guesses.

The comparison that isolates our contribution is against the fine-tuned monaural Gemma, which shares SEA-LM's LLM, audio tower, training data and curriculum and lacks only the spatial pathway. Against it, SEA-LM reduces Task 1 azimuth/elevation MAE from 87.5∘87.5^\circ/27.7∘27.7^\circ to 14.9∘14.9^\circ/12.8∘12.8^\circ and Task 6 external-source hallucination from 27.2% to 8.0%. We scope the claim to this: the spatial pathway, trained with egocentric objectives on device-realistic data, accounts for the gains. The final version will not imply that FOACoder's AST-style backbone outperforms SELD architectures, and will label the App. D SELDNet comparison an off-domain, unequal-capacity reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.