acceptodds
Under review as a conference paper at ICLR 2027

From Task Affinity to Shared Computation: Efficient Spatial Audio Encoding

Abstract

Spatial audio encoders jointly predict sound events, source distances, and directions of arrival. However, assigning a separate processing branch to each task repeats computation even when transformer parameters are shared. We introduce Efficient SpatialAST (ESpAST), a family of encoders guided by layer-wise task-affinity and cross-branch interaction analyses. Representational analysis reveals that sound event detection and distance prediction are most similar in early layers, whereas sound event detection and direction estimation remain comparatively dissimilar. We designed lightweight cross-branch mixers and observed sustained late-layer exchange between branches assigned distance and direction prediction. Together with greater late-layer representational overlap, these learned interactions motivate branch merging, yielding configurations that reduce computation while improving average multi-task performance. On SpatialSoundQA, ESpAST- achieves a average relative improvement across three tasks over a three-branch baseline model, while reducing encoder multiply–accumulate operations by . Evaluation with large language models further shows that reduced-computation encoders retain competitive spatial audio question-answering performance. Our results demonstrate how depth-dependent task relationships guide the allocation of shared and task-specific computation, improving the accuracy–computation trade-off of spatial audio encoders.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial audio encoding, multi-task learning, task affinity, efficient transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8P9oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how spatial audio encoders should allocate shared computation among sound event detection (SED), distance prediction (DIS), and direction-of-arrival estimation (DOA). It starts from DSpAST, whose three tasks share transformer parameters but execute transformer computation separately.

Critical strengths
  1. At approximately 47.8% of the three-branch computational budget, 1.5BM_DOA obtains a mean relative improvement of 7.89% ± 1.0%. Appendix Table 7 also supports the spatial-prediction gains in absolute terms: MAE decreases from 14.67 to 13.31 and MDE from 0.628 m to 0.530 m, although mAP decreases from 54.72 to 54.14. This is a meaningful encoder trade-off rather than an efficiency claim inferred solely from parameter count.

  2. The authors compare all three two-branch grouping configurations, three different depth combinations, and corresponding versions with and without a mixer, using four random seeds. The configuration with an independent DOA branch consistently achieves the highest average aggregation improvement across all test budgets, providing empirical support for its grouping design.

Critical weaknesses
  1. Comparisons focus on original Spatial-AST/DSpAST and ESpAST variants. They do not establish whether early branching followed by late merging outperforms conventional early sharing followed by late branching, or existing affinity-driven sharing principles, under the same input features, pretraining, FAMO, and comparable MACs.

  2. MTSL (ICML 2023) already merges structures using local task similarity. DTME-MTL (ICCV 2025) and TGLoRA (AISTATS 2026) study transformer token-space adaptation and task-related allocation of adapters, respectively. The manuscript cites these works without experimental comparisons.

  3. Comparisons mainly cover Spatial-AST/DSpAST and internal variants. Existing sharing architectures are not directly compared using the same backbone and similar MAC budgets, leaving the incremental contribution unclear.

Questions for authors
  1. Could you compare with existing sharing architectures using the same backbone, training settings, and similar MAC budgets?

  2. What is the performance difference between original DSpAST and the modified 3B_base under a common evaluation protocol?

  3. How does affinity determine task grouping and merging depth, and has this rationale been validated on independent samples or seeds?

    1. What are the performance, inference latency, and peak memory usage of a single checkpoint selected using a validation set?
4 comments

Thank you for the time and effort you devoted to reviewing our manuscript. Below, we provide our responses to the issues raised in the review.

Comparison with existing multi-task learning architectures

(Weaknesses 1, 3 and also Question 1)

  • Existing multi-task sharing architectures typically share early layers and introduce task-specific branches at intermediate layers, using layers with task-specific weights (MTSL [1]) or parameter-efficient fine-tuning PEFT modules such as SSF [2] (DTME-MTL [3]) and LoRA [4] (TGLoRA [5]), thereby forming a tree-like structure. However, such existing architectures are not directly applicable to our baseline (DSpAST [6]) for two reasons:

    1. Each downstream task (SED, DOA, and distance prediction) has a separate processing branch from the outset due to task-specific specialization of input features.
    2. Transformer parameters are shared across all task branches, reducing the number of learnable parameters.

    These properties make ESpAST structurally closer to an inverted-tree architecture, unlike prior MTL architectures that begin with shared layers and branch later.

  • Adapting these existing architectures to DSpAST [4] would require substantial architectural modifications and method-specific tuning to establish comparable training settings, constituting a separate research direction beyond the scope of our study.

Lack of comparison with cited prior multi-task learning works (Weakness 2)

  • As stated in L160-182 of our original manuscript, we had already tried applying gradient-cosine-similarity-based task grouping, as in DTME-MTL and TGLoRA, during the early development of our method. We had also reported the observed instability in Figure 6, showing substantial fluctuations in gradient similarities throughout training. This makes the choice of a representative checkpoint or temporal averaging window ambiguous. For example, extending the averaging window by just a few training steps reversed the sign of the observed mean similarity. Furthermore, their algorithms are not directly applicable for the reasons explained above.

Novelty beyond local task similarity (Weakness 2)

  • To the best of our knowledge, ESpAST is the first work to systematically investigate task affinity in sound event localization and detection (SELD) models through task-wise representational similarity analysis. After observing increased DIS-DOA similarity in later layers, we further investigated cross-branch interactions using our own mixer modules. These cross-branch interactions cannot be characterized by local task similarity alone. We developed efficient spatial audio encoders (ESpAST) by building upon existing research and introducing our own architectural designs and analyses. Our novelty lies in improving both predictive performance and computational efficiency in multi-task spatial audio encoding through analysis-guided architectural design.
  • Even before MTSL (ICML-2023), pioneering work had already explored architectural sharing using local task similarity (BMTN, BMVC-2020 [7]), and BMTN was also motivated by earlier observations of task affinity across different sets of tasks (Taskonomy, CVPR-2018 [8]).
  • Our original manuscript explicitly acknowledges the contributions of MTSL in several sections: the Introduction (L44-45), Related Work (L96), and Experiments (L224).

DSpAST under a common evaluation protocol (Question 2)

  • We reproduced DSpAST using the same random seeds used for ESpAST-3Bbase3\mathrm{B}_\text{base} and report the performance of both models under a common evaluation protocol in the table below. The reproduced results are comparable to those reported in the original DSpAST paper.
MetricDSpAST (reproduced)DSpAST (from [6])ESpAST-3Bbase3\mathrm{B}_{\mathrm{base}}
mAP (↑\uparrow)54.7954.5354.72
ER20 (↓\downarrow)20.9020.2820.59
MAE (↓\downarrow)15.0014.4414.67
DER (↓\downarrow)27.6128.0327.36
MDE (↓\downarrow)0.629-0.628

Question 3

  • Our analysis of task-wise affinity using intermediate representations and our affinity-guided task grouping were inspired by MTSL (L96). We observed consistent task-affinity trends across all four random seeds on the stage-3 training dataset and therefore reported the results averaged across seeds. We also observed the same qualitative trends on the evaluation split.

  • Task-wise affinity does not directly determine the merging depth. We evaluated different merging depths as part of our ablation study to compare the effects of:

    • Early merging: Less independent processing of task-specific representations and more shared computation.
    • Late merging: More independent processing of task-specific representations and less shared computation.
  • Our results suggest that maintaining a balance between task-specific and shared processing is beneficial, which aligns with a central motivation of multi-task learning. Specifically, the Δ‾MTL\overline{\Delta}_\text{MTL} of DOA-separated configurations varies non-monotonically with merging depth (1.25B<1.5B<1.75B>2B1.25\mathrm{B} < 1.5\mathrm{B} < 1.75\mathrm{B} > 2\mathrm{B}), both with and without mixers.

Question 4

  • SpatialSoundQA does not provide an explicit validation split for checkpoint selection. We systematically evaluated all tested combinations of task groupings, merging depths, and mixer settings, reporting both favorable and unfavorable results.
  • We report per-sample inference latency and peak memory usage measured on an Apple M4 GPU. Models without mixers or merging show similar peak memory usage regardless of the number of branches due to sequential branch-wise computation: all 12 transformer blocks are executed for one branch before processing the next branch.
Metric3BM3\mathrm{B}^\mathcal{M}3B3\mathrm{B}2BM2\mathrm{B}^\mathcal{M}2B2\mathrm{B}1.75BM1.75\mathrm{B}^\mathcal{M}1.75B1.75\mathrm{B}1.5BM1.5\mathrm{B}^\mathcal{M}1.5B1.5\mathrm{B}1.25BM1.25\mathrm{B}^\mathcal{M}1.25B1.25\mathrm{B}1B1\mathrm{B}
MAC (G)147.4146.598.498.182.282.070.169.957.957.849.7
Latency (ms)332311213208180176153151125124105
Peak memory (MB)379358360358364360364360363360354

Revision to the Manuscript

  • We revised Section 2.1 (Related Work: Multi-Task Learning) to clarify how task-specific representations can be obtained.
AuthorsOpenReview ↗

List of references used in the response to Reviewer 8P9o

[1] Gurulingan, N. K., Zonooz, B., & Arani, E. (2023, July). Multi-task structural learning using local task similarity induced neuron creation and removal. In International Conference on Machine Learning (pp. 12205-12223). PMLR.

[2] Lian, D., Zhou, D., Feng, J., & Wang, X. (2022). Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35, 109-123.

[3] Jeong, W., & Yoon, K. J. (2025, October). Resolving token-space gradient conflicts: Token space manipulation for transformer-based multi-task learning. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 2887-2897). IEEE.

[4] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR).

[5] Gangwar, N., Rangi, A., Deshmukh, R., Rahmanian, H., Dattatreya, Y. & Kani, N.. (2026). Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation . Proceedings of Machine Learning Research, 300:3916-3924

[6] Wilkinghoff, K., & Tan, Z. H. (2026, May). Dspast: Disentangled representations for spatial audio reasoning with large language models. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 14747-14751). IEEE.

[7] Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. (2019). Branched multi-task networks: deciding what layers to share. arXiv preprint arXiv:1904.02920, (BMVC 2020)

[8] Zamir, Amir R., et al. "Taskonomy: Disentangling task transfer learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018

Thank you for the detailed response. I am maintaining my score of 3.

Reviewer_9sxyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reduce computation in spatial audio encoders that jointly perform SED, DIS, and DOA. The authors propose ESpAST, which makes five changes. (1) Loss weighting: it adopts FAMO dynamic loss weighting as a training procedure, which stabilizes DOA. (2) Representational analysis: the authors measure layer-wise representational similarity between branches with linear CKA and orthogonal Procrustes similarity. They find that SED and DIS are most similar in early layers, SED and DOA are least similar, and DIS–DOA similarity increases across layers. (3) Task grouping: based on these findings, SED and DIS are grouped in one branch and DOA is kept separate. (4) Cross-branch mixers: zero-initialized low-rank mixers are added between branches. (5) Branch merging: branches are merged at block 3, 6, or 9 by averaging their audio tokens. All combinations of grouping, budget, and mixer/no-mixer are evaluated over 4 seeds on SpatialSoundQA, using MTL relative to a fixed-weight three-branch baseline. The mixer-enabled model is best overall. Finally, a frozen-encoder evaluation with SmolLM3-3B on SpatialSoundQA question answering shows that the authors’ approach remains competitive with DSpAST.

Critical strengths
  • The paper shows that roughly half of the computation of the DSpAST encoder, whose three branches triple the transformer computation, can be removed without losing performance (Table 7).
  • The paper presents a comprehensive and systematic controlled study. Every grouping is crossed with every merge depth, with and without mixers, across multiple seeds. The baseline is a careful reproduction of DSpAST (Table 7), and metrics are averaged over the final 15 epochs.
  • The paper reports interesting observations. For example, FAMO sharply stabilizes DOA in the single-branch model, and keeping DOA separate early is consistently the best grouping in mean MTL.
Critical weaknesses
  • The gains mostly come from FAMO and the mixers rather than from the computation-saving design. The MTL gain is measured against the fixed-weight baseline. Against the mixer-enabled three-branch model, every configuration trades accuracy for computation. The claim to “reduce computation while improving average multi-task performance” (L21) therefore conflates separate effects.
  • For mixer-free models, MTL jumps from 0.95% (2B DOA) to 7.22% (1.75B DOA) after merging (Table 2). The most plausible explanation is that averaging the branches’ tokens gives each task access to the other branch’s features. This is a crude form of the cross-branch exchange that mixers provide. The gain in mixer-free merged models therefore looks more like information exchange than efficiency.
  • The claimed “7.89% average relative improvement across three tasks” (L22, L80) needs per-task reporting. In Table 2, the distance term (MDE, +8–16%) drives most of the MTL gain. Meanwhile, mAP is negative for every merged DOA configuration except 1.75B DOA. For example, 1.5B DOA has mAP −1.05%, MAE +9.22%, and MDE +15.51%. Averaging relative changes across metrics with very different sensitivities (near-saturated mAP vs. sub-meter MDE) lets the distance task dominate. In addition, MTL is computed with the newly introduced MDE rather than the established DER; I would like to see it reported with the standard metrics as well.
  • At 1.25B, the three groupings score 1.60 ± 3.4, 0.70 ± 1.2, and 0.48 ± 1.2. For mixer-free models, 1.5B DOA vs. 1.5B SED is 5.89 ± 2.2 vs. 3.82 ± 0.4. The standard deviations overlap, and no significance tests are reported. The claim that DOA-separate is best “at every tested branch budget” (L65) is therefore not well established.
  • Efficiency is measured only in MACs and only for the encoder. Wall-clock latency, throughput, and memory are not reported.
Questions for authors
  • How are the dashed single-task curves in Fig. 2 computed? Is each task trained as a separate model, with similarity computed across models? If so, how much of the early-layer similarity comes from the shared AudioMAE initialization rather than from task affinity?
  • L154 first refers to a “3-branch 3B and also a single-branch 1B model.” L157 then states, “We denote the fixed-weight three-branch reference as 3B base and its FAMO-trained counterpart as 3B.” Does “3B” denote parameter count or number of branches on L154 and L157? The naming is confusing; I suggest using distinct names to avoid ambiguity.
7 comments

We appreciate the time and effort you devoted to reviewing our manuscript. Below, we provide our responses to the issues raised in the review.

W1 - Baseline selection and Sources of gains

  • The mixer-enabled three-branch model ESpAST-3BM3\mathrm{B}^\mathcal{M} is an accuracy-oriented configuration explored in our study, rather than the baseline model established for comparing our proposed model families against prior works.

  • Reproducing DSpAST [1] required substantial effort to stabilize training, and even our fixed-weight baseline (ESpAST-3Bbase3\mathrm{B}_\text{base}) incorporates the resulting adjustments. We subsequently found that dynamic loss weighting (FAMO [2]) further improved training stability and performance, particularly for ESpAST-1B1\mathrm{B}, which shares computation across all tasks. In contrast, ESpAST-3B3\mathrm{B}, which maintains separate task-processing branches, was already relatively stable under fixed loss weighting and benefited less from FAMO (Table 1). Therefore, we selected ESpAST-3Bbase3\mathrm{B}_\text{base} as our main baseline while also reporting results for ESpAST-3B3\mathrm{B} to distinguish optimization improvements from architectural contributions. Furthermore, several configurations achieve higher mean Δ‾MTL\overline{\Delta}_\text{MTL} than the three-branch model (ESpAST-3B3\mathrm{B}) trained with FAMO at lower inference costs, as described in L80-83 of our original manuscript (ESpAST-3B3\mathrm{B}: Δ‾MTL\overline{\Delta}_\text{MTL} = 2.06%, 146.5 GMAC vs. ESpAST-1.5BDOAM1.5\mathrm{B}_\text{DOA}^\mathcal{M}: Δ‾MTL\overline{\Delta}_\text{MTL} = 7.89%, 70.1 GMAC). Also, several of our models without mixers outperform ESpAST-3B3\mathrm{B} (e.g., ESpAST-1.5BDOA1.5\mathrm{B}_\text{DOA}: Δ‾MTL\overline{\Delta}_\text{MTL} = 5.89%, 69.9 GMAC). This demonstrates that our architectural design improves multi-task performance while reducing computation.

W2 - Information exchange and computational efficiency of branch merging

  • Information exchange and computational efficiency are complementary aspects of our design, not competing explanations. Branch merging reduces redundant transformer computations while allowing task representations to share information through token averaging. Specifically, merging reduces computation from 98.1 GMAC (ESpAST-2B2\mathrm{B}) to 82.0 GMAC (ESpAST-1.75B1.75\mathrm{B}).

W3a - Choice of MDE vs. DER for aggregated multi-task improvement

  • We agree that aggregate Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} alone is insufficient to fully describe task-wise performance trade-offs. However, we would like to clarify that using MDE instead of DER actually yields a more conservative aggregate improvement for most configurations in our experiments (26 out of 27). As requested, we provide both ΔDER\Delta_{\mathrm{DER}} and Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (DER) recomputed using mAP, MAE, and DER below. DER generally produces larger relative distance improvements than MDE, consequently increasing the contribution of distance prediction to Δ‾MTL\overline{\Delta}_{\mathrm{MTL}}. For example, for ESpAST-1.5BDOAM1.5\mathrm{B}_{\mathrm{DOA}}^{\mathcal{M}}, the relative distance improvement increases from 15.51% with MDE to 20.45% with DER, and Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} increases from 7.89% to 9.54%. We also note that DER counts predictions outside the error tolerance without distinguishing their error magnitudes, whereas MDE measures the average absolute distance error.

W3b - SED-spatial performance trade-offs and the contribution of distance prediction

  • Although distance prediction contributes the largest improvement over our baseline, this does not fully reflect the benefit of task-specific processing for DOA. The three-branch baseline already provides a separate processing path for DOA, whereas the single-branch baseline processes all tasks jointly. Comparing with ESpAST-1B1\mathrm{B} therefore highlights the angular-prediction benefits of introducing limited task-specific processing. ESpAST-1.25BDOAM1.25\mathrm{B}_\text{DOA}^{\mathcal{M}} improves ΔMAE\Delta_{\mathrm{MAE}} from -11.69% to -1.87% (+9.82 percentage points) and ΔMDE\Delta_{\mathrm{MDE}} from 4.82% to 12.24% (+7.42 percentage points) relative to ESpAST-1B1\mathrm{B}.
  • Regarding the different sensitivities of the task metrics, we acknowledge that most merged configurations show a decrease in mAP. However, we believe that preserving competitive semantic prediction performance while substantially improving spatial prediction and reducing computation is also valuable. Notably, ESpAST-1.75BDOAM1.75\mathrm{B}_\text{DOA}^{\mathcal{M}} retains mAP (+0.16%), and improves MAE by 9.35%, and MDE by 15.96%, with 43.9% fewer encoder MACs.
AuthorsOpenReview ↗

Table for W3a (Choice of MDE vs. DER for aggregated multi-task improvement)

Metric3BM3\mathrm{B}^{\mathcal{M}}3B3\mathrm{B}2BDOAM2\mathrm{B}_{\mathrm{DOA}}^{\mathcal{M}}2BSEDM2\mathrm{B}_{\mathrm{SED}}^{\mathcal{M}}2BDISM2\mathrm{B}_{\mathrm{DIS}}^{\mathcal{M}}2BDOA2\mathrm{B}_{\mathrm{DOA}}2BSED2\mathrm{B}_{\mathrm{SED}}2BDIS2\mathrm{B}_{\mathrm{DIS}}
ΔMDE\Delta_{\mathrm{MDE}}17.88%3.04%15.08%8.68%13.48%4.09%-10.03%1.50%
ΔDER\Delta_{\mathrm{DER}}23.47%3.56%18.98%10.86%16.72%6.56%-13.83%1.72%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (MDE)9.28%2.06%7.70%4.66%5.94%0.95%-2.15%-5.10%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (DER)11.15%2.23%9.00%5.39%7.02%1.77%-3.41%-5.02%
Metric1.75BDOAM1.75\mathrm{B}_{\mathrm{DOA}}^{\mathcal{M}}1.75BSEDM1.75\mathrm{B}_{\mathrm{SED}}^{\mathcal{M}}1.75BDISM1.75\mathrm{B}_{\mathrm{DIS}}^{\mathcal{M}}1.75BDOA1.75\mathrm{B}_{\mathrm{DOA}}1.75BSED1.75\mathrm{B}_{\mathrm{SED}}1.75BDIS1.75\mathrm{B}_{\mathrm{DIS}}1.5BDOAM1.5\mathrm{B}_{\mathrm{DOA}}^{\mathcal{M}}1.5BSEDM1.5\mathrm{B}_{\mathrm{SED}}^{\mathcal{M}}1.5BDISM1.5\mathrm{B}_{\mathrm{DIS}}^{\mathcal{M}}
ΔMDE\Delta_{\mathrm{MDE}}15.96%9.29%12.72%15.19%8.02%11.99%15.51%11.66%10.79%
ΔDER\Delta_{\mathrm{DER}}20.65%11.19%15.46%19.57%10.24%14.49%20.45%13.86%13.63%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (MDE)8.49%5.51%7.02%7.22%4.68%5.96%7.89%6.35%4.80%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (DER)10.05%6.15%7.93%8.68%5.42%6.79%9.54%7.08%5.75%
Metric1.5BDOA1.5\mathrm{B}_{\mathrm{DOA}}1.5BSED1.5\mathrm{B}_{\mathrm{SED}}1.5BDIS1.5\mathrm{B}_{\mathrm{DIS}}1.25BDOAM1.25\mathrm{B}_{\mathrm{DOA}}^{\mathcal{M}}1.25BSEDM1.25\mathrm{B}_{\mathrm{SED}}^{\mathcal{M}}1.25BDISM1.25\mathrm{B}_{\mathrm{DIS}}^{\mathcal{M}}1.25BDOA1.25\mathrm{B}_{\mathrm{DOA}}1.25BSED1.25\mathrm{B}_{\mathrm{SED}}1.25BDIS1.25\mathrm{B}_{\mathrm{DIS}}1B1\mathrm{B}
ΔMDE\Delta_{\mathrm{MDE}}14.12%8.31%9.85%12.24%10.81%8.84%10.87%9.19%8.21%4.82%
ΔDER\Delta_{\mathrm{DER}}18.62%10.62%12.50%16.25%14.09%11.56%14.38%12.10%10.70%5.24%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (MDE)5.89%3.82%2.64%2.81%1.98%1.73%1.60%0.70%0.48%-3.11%
Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} (DER)7.39%4.59%3.52%4.15%3.07%2.64%2.77%1.67%1.32%-2.97%

W4 - DOA separation and Δ‾MTL\overline{\Delta}_\text{MTL}

  • We reported Δ‾MTL\overline{\Delta}_\text{MTL} as a commonly used metric for evaluating aggregate multi-task performance. However, we recognize that our original explanation (L65), which focused mainly on this metric, did not sufficiently highlight our main findings regarding task-specific performance trade-offs across model configurations.

  • Here, we briefly summarize the rationale behind our architectural choices:

    • DOA exhibits lower representational affinity with other tasks, particularly SED, motivating a separate DOA branch. Distance prediction shows higher affinity with SED in early layers and with DOA in later layers.
    • Furthermore, the mixer experiments show sustained late-layer interactions between DIS and DOA.
    • These observations motivate jointly processing SED and DIS in early layers while keeping DOA separate, followed by merging their processing branches at later layers.
  • Our main finding is that different task groupings can substantially degrade individual task performance, even under the same computation budget. Keeping DOA separate does not necessarily yield the best DOA accuracy, but it provides a more favorable balance across tasks, particularly by avoiding substantial degradation in SED and distance prediction. In our four-seed analysis, grouping SED with DOA reduced mAP, while grouping DIS with DOA increased MDE, in all 32 matched comparisons for each pattern (4 seeds × 4 branch budgets × 2 mixer settings; see supplementary material). Table 7 of our manuscript provides supporting examples below, comparing DOA-separated configurations with alternative groupings (DOA-separated → alternative):

  • Without mixers:

    • Grouping SED and DOA:
      • mAP: 2BDIS2\mathrm{B}_\text{DIS} (54.68→51.48), 1.75BDIS1.75\mathrm{B}_\text{DIS} (54.62→53.59), 1.5BDIS1.5\mathrm{B}_\text{DIS} (54.10→52.91).
      • MAE: 2BDIS2\mathrm{B}_\text{DIS} (14.83→16.29).
    • Grouping DIS and DOA:
      • mAP: 1.5BSED1.5\mathrm{B}_\text{SED} (54.10→53.25).
      • MDE: 2BSED2\mathrm{B}_\text{SED} (0.601→0.690), 1.75BSED1.75\mathrm{B}_\text{SED} (0.532→0.577), 1.5BSED1.5\mathrm{B}_\text{SED} (0.538→0.575).
  • With mixers, branches exchange task-wise information, but similar task-specific degradation remains:

    • Grouping SED and DOA:
      • mAP: 2BDISM2\mathrm{B}_\text{DIS}^\mathcal{M} (54.91→53.47), 1.75BDISM1.75\mathrm{B}_\text{DIS}^\mathcal{M} (54.81→53.71), 1.5BDISM1.5\mathrm{B}_\text{DIS}^\mathcal{M} (54.14→53.23).
    • Grouping DIS and DOA:
      • MDE: 2BSEDM2\mathrm{B}_\text{SED}^\mathcal{M} (0.533→0.573), 1.75BSEDM1.75\mathrm{B}_\text{SED}^\mathcal{M} (0.527→0.569).

These comparisons illustrate why we favor DOA-separated configurations beyond the aggregate Δ‾MTL\overline{\Delta}_{\text{MTL}} ranking.

W5 - Measurements on real hardware

  • We chose multiply-accumulate counts (MACs) as our primary efficiency metric because they are device-agnostic and have been shown to correlate with measured efficiency on real hardware [3]. Here, we report our models' inference latency and peak memory usage, measured on Apple M4 hardware. Models without mixers or merging (1B, 2B and 3B) show similar peak memory usage regardless of the number of branches due to sequential branch-wise computation: all 12 transformer blocks are executed for one branch before processing the next branch.
Metric3BM3\mathrm{B}^\mathcal{M}3B3\mathrm{B}2BM2\mathrm{B}^\mathcal{M}2B2\mathrm{B}1.75BM1.75\mathrm{B}^\mathcal{M}1.75B1.75\mathrm{B}1.5BM1.5\mathrm{B}^\mathcal{M}1.5B1.5\mathrm{B}1.25BM1.25\mathrm{B}^\mathcal{M}1.25B1.25\mathrm{B}1B1\mathrm{B}
MAC (G)147.4146.598.498.182.282.070.169.957.957.849.7
Latency (ms)332311213208180176153151125124105
Peak memory (MB)379358360358364360364360363360354
Reviewer_azBJOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper demonstrates a new stereo spatial audio encoder (ESpAST). The main innovation is training a separate transform branch for each task (specifically: angle, distance and sound events). The transformers in the branches share the same parameters. The encoder is evaluated on a set of standard metrics like angular and distance error. Also, the encoder is paired with an LLM for evaluation on SpatialSoundQA

Critical strengths

As far as I understand, the model architecture and training approach is novel for the spatial audio encoders. I have not seen models with shared-weight transformer branches for each representation. Also, the cross-branch mixer is an interesting idea, and it seems to be novel I’ve only seen models with shared transformer blocks for all representations, producing one or more CLS tokens. I am not fully convinced that separate branches is a better approach, but it is interesting to see it explored here. Overall, It is not common for recent papers to look into the architecture of spatial audio encoders, as mostly standard models are used. I believe there is something to learn there, as often architectures are less than ideal

Critical weaknesses

Unclear contribution. The improvement over the current state-of-the-art or/and new functionality that the proposed method adds is not clear to me. Recent spatial audio encoders have moved beyond stereo (like Spatial-AST) to first-order-ambiosonics (FOA) such as Sci-Phi or directly decoding arbitrary microphone arrays (PhaseCoder, GI-DOAEnet). Other papers demonstrated novel functionality like detecting moving and overlapping sources. The model size is large (1B - 3B) for a spatial encoder due to three 12-layer transformers, which is order of magnitude larger than existing models such as Spatial-AST. Addition of SED head might not be very useful, as recent audio LLMs can potentially detect sound events using their existing mono audio encoders.

To show the improvement over state of the art, the authors should provide comparison to standard baseline models (such as OWL, Spatial-AST, SELDnet ) for accuracy, compute and size. The current evals only baseline with the authors own model (DSpAST). Similarly for LLM evaluation, baselines from other papers are missing. I think the author’s case would be stronger if they compared to existing baseline models. Furthermore, the authors only evaluated and trained on SpatialSoundQA, so there is no evidence that the model can generalize. Authors should evaluated on other datasets as well like STARSS23, which contains real-recorded data. STARSS23 is also a more established baseline for the encoder performance. Also, authors could evaluate the encoder on LOCATA and RSL2019 for localization performance.

Questions for authors

1B to 3B model size is large for spatial encoder. Did the authors try smaller models for examples with fewer transformer blocks ?

Did the authors do any ablations on the input features. (e.g, removing GCC-PHAT)

It is not clear why DSpAST was introduced, and why not use baselines from previous papers.

Why not evaluate on recorded datasets like STARS23 ?

5 comments

Thank you for taking the time and effort to review our manuscript. Before addressing the issues raised in the review, we would like to clarify two points that may have caused misunderstandings:

Clarification of Model Notation

  • L154-156 of our original manuscript provide the definition. For our proposed models, 1B and 3B denote single-branch and three-branch configurations, respectively, rather than parameter counts. We acknowledge that our naming convention could have been clearer, and we apologize for the confusion during the review.

Comparison with Baseline Models (Also Question 3)

  • We briefly outline the progression from prior works to our study:

    1. Spatial-AST [1]: A pioneering work demonstrating an encoder capable of both semantic and spatial audio encoding.
    2. DSpAST [2]: Introduces additional spatial features (GCC-PHAT and ILD) and task-wise input specialization with shared transformers, achieving high accuracy but requiring separated processing of the SED, DOA and distance branches.
    3. ESpAST (ours): Improves both prediction accuracy and computational efficiency by balancing task-specific and shared processing.
  • DSpAST is an independent prior work, not a model introduced by us, as stated in our original manuscript (introduction: L36–40, related works: L114–116, and also experiments: L146–147). Our experiments already include comparisons of our models with DSpAST and Spatial-AST as baselines (Table 7). To the best of our knowledge, these are the only publicly available models directly comparable under our training and evaluation setting. Furthermore, given that Spatial-AST is a single-branch model, ESpAST-1B can be regarded as another baseline model, an enhanced version of Spatial-AST evaluated under our common proposed configurations.


We address the following issues raised in the review:

Relevance of Binaural Audio Research

  • We acknowledge the importance of recent advances in FOA and arbitrary microphone array spatial audio modeling. However, binaural audio remains relevant to recent research on robotic spatial perception [3], audio-visual navigation [4], and resource-constrained wearable audio processing [5, 6].
  • Our work makes a complementary contribution: achieving improvements of both predictive performance and computational efficiency in multi-task spatial audio encoding with binaural inputs. Such improvements in the accuracy–computation trade-off are particularly relevant to resource-constrained applications. Rather than extending the supported recording formats, our contribution lies in determining when and how task-specific computation should be shared or kept separate and deriving efficient spatial audio encoders based on those analyses.

Use of an Additional SED Head

  • As illustrated in Figure 1, our audio LLM uses encoder output representations rather than predictions from the SED head, following the designs of DSpAST and Spatial-AST. Models derived from Spatial-AST (i.e., DSpAST and ours) jointly predict both semantics and spatial information with a single audio encoder. We did not introduce any additional SED head beyond those already present in Spatial-AST and DSpAST.

Size of Our Proposed Models (also Question 1)

  • First, we would like to clarify that ESpAST-1B and ESpAST-3B denote the number of task-processing branches, not model sizes in billions of parameters (L154–156). We also noted the reviewer's concern that our models are "an order of magnitude larger than existing models such as Spatial-AST." However, our largest model, ESpAST-3B$^{\mathcal M}$ (88M), has fewer than 3% more parameters than Spatial-AST due to shared transformer weights, rather than being an order of magnitude larger.
  • Other spatial audio understanding models also employ comparable or larger audio encoders. For example, OWL [7] uses approximately 85M parameters, while Sci-Phi [8] and PhaseCoder [9] with Gemma 3n use separate semantic and spatial encoders totaling approximately 442M and 686M parameters, respectively. Unlike the latter two approaches, ESpAST, like Spatial-AST and DSpAST, uses a unified ViT-B-based encoder to extract both semantic and spatial information.
  • We acknowledge that substantially smaller SELD and DOA-estimation models exist, but their task settings differ from ours. For example, SELDNet [10] has about 0.5M parameters and supports 11 classes from the DCASE 2019 dataset, while GI-DOAEnet [11] has about 2M parameters and estimates azimuth without semantic recognition.
  • However, our experiments on SpatialSoundQA require the model to recognize a much broader set of sound-event classes (355), supporting more general spatial audio understanding with LLMs. We believe it is reasonable to compare model sizes within similar model families and task settings; in this context, ESpAST’s parameter count is not unusually large.

Study of shallow transformer Models (Question 1)

  • We didn’t evaluate models with fewer transformer blocks. We consider reducing transformer depth a complementary research direction, alongside width compression, token reduction, and quantization methods developed for transformers. We believe this is a promising research direction, but systematically investigating and adapting existing depth-pruning algorithms would introduce another dimension of ablation studies, including comparisons with SOTA algorithms and additional hyperparameter tuning. Therefore, we consider depth reduction an interesting direction for future work, beyond the scope of the present study.

Input-Feature Ablation Study (Question 2)

  • GCC-PHAT and ILD are established input features in DSpAST, while IPD was introduced in Spatial-AST. Since our study focuses on efficient sharing of transformer computation rather than input feature selection, we retained these features without conducting feature addition or removal ablations. Nevertheless, we analyze how input-feature attention weights vary across branch configurations (Section 4.4 and Figure 5).

Evaluation on real-world recording (also Question 4)

  • STARSS23 is not directly compatible with our current input format and evaluation protocol. It provides FOA/microphone-array recordings with temporally varying and overlapping events; evaluating our static binaural encoder would require FOA-to-binaural conversion and a defined protocol for selecting stationary, single-source segments. RSL2019 [12] likewise uses a different, four-channel microphone-array configuration. We also acknowledge that LOCATA Task 1 [13] matches our stationary, single-source setting.
  • However, the specific HRTF used to render SpatialSoundQA is not documented in its paper or dataset description, making it difficult to assess and control HRTF mismatch in external evaluation. This is a non-negligible concern because HRTF mismatch can degrade learned binaural localization performance [14].
  • We follow Spatial-AST and DSpAST in using simulated spatial audio and acknowledge that real-world transfer remains to be established for ESpAST. Nevertheless, the newly published ST-AudioLM (EMNLP 2026) [15] provides promising empirical evidence: incorporating STARSS23-derived QA pairs amounting to approximately 5% of the synthetic dataset count at each curriculum stage improves spatial audio QA performance on real recordings. We view this as a promising direction for adapting ESpAST to real recordings, complementary to our investigation of efficient audio encoding.

Revisions to the Manuscript:

  • To clarify our model size, we added a comparison with Spatial-AST in the Introduction.
  • We plan to use a different notation (B → Br) that more explicitly distinguishes the number of branches from the parameter count. However, we have refrained from making this revision during the rebuttal period to avoid introducing further confusion.
AuthorsOpenReview ↗

List of references used in the response to Reviewer azBJ

[1] Zheng, Z., Peng, P., Ma, Z., Chen, X., Choi, E., & Harwath, D. (2024). BAT: Learning to Reason about Spatial Sounds with Large Language Models. International Conference on Machine Learning.

[2] Wilkinghoff, K., & Tan, Z. H. (2026, May). Dspast: Disentangled representations for spatial audio reasoning with large language models. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 14747-14751). IEEE.

[3] G. Liu et al. SAGENet: Binaural Echo-Based 3D Depth Estimation with Sparse Angular Queries and Refined Geometric Cues. IROS, 2025.

[4] Y. Zeng et al. Semantic Audio-Visual Navigation in Continuous Environments. CVPR, 2026.

[5] S. Oh et al. Fine-grained Soundscape Control for Augmented Hearing. MobiSys, 2026.

[6] X. Lu et al. A Lightweight Fourier-based Network for Binaural Speech Enhancement with Spatial Cue Preservation. ICASSP, 2026.

[7] Biswas, S., Khan, M., & Islam, B. (2026, April). Owl: Geometry-aware spatial reasoning for audio large language models. In International Conference on Learning Representations (Vol. 2026, pp. 20685-20710).

[8] Jiang, X., Gamper, H., & Braun, S. (2026). Sci-phi: A large language model spatial audio descriptor. IEEE Open Journal of Signal Processing.

[9] Dementyev, A., Zulfikar, W., Hersek, S., Getreuer, P., Kumar, A. & Kumar, V.. (2026). PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs. Proceedings of the 43rd International Conference on Machine Learning, 306:23672-23696

[10] Adavanne, S., Politis, A., Nikunen, J., & Virtanen, T. (2018). Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1), 34-48.

[11] Baek, M. S., Chang, J. H., & Cohen, I. (2025). Dnn-based geometry-invariant doa estimation with microphone positional encoding and complexity gradual training. IEEE Transactions on Audio, Speech and Language Processing, 33, 2360-2376.

[12] Sheelvant, R., Sharma, B., Madhavi, M., Das, R. K., Prasanna, S. R. M., & Li, H. (2019, October). RSL2019: A realistic speech localization corpus. In 2019 22nd Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA) (pp. 1-6). IEEE.

[13] Evers, C., Löllmann, H. W., Mellmann, H., Schmidt, A., Barfuss, H., Naylor, P. A., & Kellermann, W. (2020). The LOCATA challenge: Acoustic source localization and tracking. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 1620-1643.

[14] Phokhinanan, W., Obin, N., & Argentieri, S. (2024, April). Auditory cortex-inspired spectral attention modulation for binaural sound localization in HRTF mismatch. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 8656-8660). IEEE.

[15] Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin, Toshimitsu Uesaka, Takashi Shibuya, Kyeongyoon Lee, Tae-Hyun Oh, and Yuki Mitsufuji. 2026. Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 46568–46590, Budapest, Hungary. Association for Computational Linguistics.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Response: Summary of Revisions and Closing Remarks

We thank the reviewers for helping us improve the clarity of our manuscript. The following is a summary of our revisions:

Clearer model notation [azBJ, 9sxy, 8P9o]

  • We will revise the notation for our model families to avoid confusion about model size and distinguish our models more clearly from prior work.
  • We clarified our model’s parameter count in the introduction (similar to Spatial-AST).

Multi-task-learning [8P9o]

S2.1 - Related work

  • We added an explanation of how multi-task networks share or isolate task representations.

Clarification of our choice of mean distance error (MDE) over distance error rate (DER) for the aggregate improvement metric [9sxy]

S3.2 - Explanation of Equation 2 (Δ‾MTL\overline{\Delta}_{\mathrm{MTL}})

  • We clarified that MDE captures the magnitude of distance errors, whereas DER counts predictions outside a tolerance without distinguishing their error magnitudes. For the comparisons highlighted in our main results, using MDE also yields more conservative values of Δ‾MTL\overline{\Delta}_{\mathrm{MTL}} than using DER.

More concrete description of our main finding [9sxy]

S1 - Introduction, S4.3 - Sharing Later Computation Through Branch Merging, S6 - Conclusion and Discussion

  • We described our main finding in terms of individual-task performance as well as aggregate improvement: keeping DOA separate provides a more favorable balance across tasks by avoiding the substantial degradation in SED and distance prediction observed in other configurations. This is consistent with the inter-task relationships observed in our models.
  • To improve readability, we are also considering moving Table 7 (absolute performance metrics) to the main text, subject to the space and Latex constraints of the camera-ready version.

Clarification of the rationale for comparing merging depths [8P9o]

S4.3 - Sharing Later Computation Through Branch Merging

  • We clarified the motivation for comparing different merging depths: to examine how the balance between independent processing of task-specific representations and shared computation affects performance and computational cost.

Limitations and future directions for multi-task learning in spatial audio [azBJ]

S6 - Conclusion and Discussion

  • While our experiments rely on an established simulation-based spatial audio dataset, future work will investigate effective multi-task learning strategies for real-world sound event localization and detection.
  • Joint optimization of our proposed method with depth or width-pruning techniques would introduce additional complexity; however, we believe such combinations could offer promising directions for improving computational efficiency for edge-device deployment.

Addition of measurements on a real device [9sxy, 8P9o]

Appendix F

  • We added measurements of inference latency and peak memory usage on an Apple M4 GPU.

Closing Remarks

  • To the best of our knowledge, our work is the first sound event localization and detection (SELD) study to analyze how relationships between tasks evolve across model layers, achieving lower computational cost while maintaining or improving accuracy based on these findings.

  • We thank the reviewers once again for their valuable feedback. We hope our discussion will facilitate further research on SELD models based on the principles of multi-task learning, particularly regarding when, where, and how task representations should be shared or isolated to achieve better performance under computational constraints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.