acceptodds
Under review as a conference paper at ICLR 2027

AERIN: Acoustic Evidence Routing and Integration for Compact Speech Representation

Abstract

Different speech tasks depend on different properties of the same signal, yet frozen self-supervised encoders leave the organization of those properties largely to pretraining. Downstream systems can select or combine internal representations, but only after those representations have already been formed. We ask whether a reusable speech representation can instead be structured around explicit acoustic evidence while still learning how that evidence should be used. We introduce AERIN (Acoustic Evidence Routing and Integration Network), a raw-waveform encoder that organizes speech into Boundary, Envelope, Phonation, and Texture evidence streams and learns their contribution through fine-grained routing and contextual integration. Thus, the sources of evidence are specified by design, while their encoding and combination remain learned from data. AERIN is distilled from WavLM-base without downstream task labels, frozen, and evaluated across 11 clinical, phonetic, and affective tasks. It remains competitive with substantially larger pretrained encoders and with its WavLM-base teacher despite using 25.66 M parameters. Routing and stream-ablation analyzes further show that the learned representation is not governed by one globally dominant stream or a fixed mixture: routing is selective at individual frame-feature positions, and downstream sensitivity to the streams varies across tasks. These results support evidence-organized representation learning as a broadly transferable alternative to relying only on emergent layer representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Affective Computing, Speech Encoder, Knowledge Distillation, Self-Supervised Speech Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zCeFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AERIN, a 26M speech encoder trained with speech attribute factorization and knowledge distillation. It organizes waveform-derived information into 4 streams: Boundary, Envelope, Phonation, and Texture. These streams through competitive routing and contextualization with intra stream attention over the temporal axis and inter-stream attention at the same frame. It learns hierarchical read-outs through multi-layer knowledge distillation with WavLM-Base as the teacher. The frozen encoder is evaluated on 11 clinical, phonetic, and affective tasks from eight corpora, with an additional Speech Commands experiment. Reported results suggest competitive transfer with substantially fewer parameters than the teacher.

Critical strengths
  1. The proposed architecture is well-motivated. AERIN utilized factorised speech acoustic information, combining explicit amplitude, timing, and periodicity measurements with learned waveform features. This is a plausible inductive bias for clinical and paralinguistic applications. The design offers a concrete hypothesis about how acoustic evidence can be organized within a reusable encoder.

  2. The transfer performance is promising. The same frozen encoder remains competitive across heterogeneous tasks and several languages absent from its distillation data, despite of using much fewer model parameters. Separation between distillation and evaluation corpora strengthens the evidence for transfer, subject to the evaluation limitations below.

Critical weaknesses
  1. Important comparisons is missing. For example, SPEAR[1] is missing from both the related-work discussion and evaluation, leaving the comparison with another unified speech representation method incomplete.

  2. Explanations of the performance gains over WavLM-base is insufficient. WavLM-Base is the teacher model, and it's unusual that a KD-based method can exceed its own teacher. This was not explained enough through experiments separating acoustic stream organization from multi-layer distillation, representation regularization, additional training data, and read-out construction. An explanation on would help other researchers better understand the source of gains of the proposed method.

  3. The presentation makes the framework unnecessarily difficult to follow. Reconstructing the computation requires substantial cross-referencing between the main text and appendix. Figure 1 combines many stages, feedback paths, and auxiliary connections, with small labels that obscure the main information flow. Section 2.6 introduces the overall objective while deferring the mathematical definitions of its core components to Appendix D. These definitions should appear alongside the objective in the main text, with implementation details and derivations retained in the appendix. Clearer explanations of shared-stem conditioning, residual routing, and the read-out hierarchy are also needed.

[1] Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations". in Proc. ICML 2026

Questions for authors
  1. What explains AERIN’s improvements over WavLM-base? Please consider a teacher baseline with multi-layer aggregation. Can the effects of stream organization and representation regularization be isolated?

  2. Does AERIN’s relative advantage persist with a moderately stronger probe, such as a deeper MLP? Speech SSL model's representation tend to be highly engangled and could require more parameter to better digest. This would clarify whether the current small probe particularly benefits the organization of AERIN’s read-outs.

  3. Can AERIN’s frame-level representations support phoneme classification or speech recognition?

6 comments

We sincerely thank Reviewer zCeF for their thorough and insightful review and constructive feedback. We are grateful that the reviewer finds the architecture well motivated and its transfer performance promising. Below, we respond to each comment and question.

W1: SPEAR is missing from the related work and the evaluation

We thank the reviewer for pointing us to SPEAR. We added SPEAR-large (327M parameters; public checkpoint from Hugging Face) to the panel. It is evaluated under the same protocol as every other system, with all of its layers searched (Table R3.1).

Table R3.1. Scores of AERIN (retrained; details in response to Reviewer tnp2) and SPEAR-large, and their difference (AERIN − SPEAR). Bold: the 95% speaker-clustered bootstrap interval does not contain zero. †: Holm-adjusted p < 0.05 across the 10 corpora.

CorpusAERIN (25.7M)SPEAR-large (327M)AERIN − SPEAR [95% CI]
DAIC0.3030.394−0.091 [−0.231, +0.025]
MODMA0.7870.715+0.072 [−0.021, +0.170]
IPVS0.9630.952+0.010 [−0.024, +0.056]
NV-H/P0.8660.874−0.008 [−0.038, +0.024]
NV-SV0.9390.928+0.011 [−0.007, +0.028]
NV-GM0.9800.986−0.006 [−0.020, +0.006]
NV-LR0.9690.978−0.009 [−0.014, −0.004] †
TESS0.9030.914−0.011 [−0.023, +0.001]
RAV0.7050.760−0.055 [−0.089, −0.022] †
ShEMO0.6630.752−0.089 [−0.120, −0.061] †

SPEAR-large is the strongest system we evaluated. Its mean rank among the 16 systems we now evaluate (the 12 encoders of Table 1 (main paper), three distilled encoders of similar size, and SPEAR-large) is 4.15 (excluding SAND), compared with 6.05 for DPWavLM and 6.50 for AERIN. AERIN scores higher on 3 of 10 corpora but is not significantly better on any. SPEAR is significantly better on three corpora: NV-LR, RAVDESS, and ShEMO. On the other seven, including all four clinical corpora, the two systems cannot be distinguished. SPEAR has 12.7 times as many parameters as AERIN.

We will add SPEAR to Table 1 (main paper) and to the related work (Appendix A.1). SPEAR learns a unified representation of speech and general audio by masked prediction of fine-grained discrete tokens derived from speech and audio SSL teachers. Like AERIN, it learns from teacher models, but it pursues breadth through large-scale pre-training, whereas AERIN is a compact encoder that organizes its representation around explicit acoustic evidence.

W2: The gains over the WavLM-base teacher are not explained

We thank the reviewer for this important point. We agree that a student that outperforms its teacher needs an explanation. We tested each candidate source the reviewer listed, one at a time. Table R3.2 summarizes the results, and the paragraphs below give the details.

(a) Size of the gain. When the teacher's layers are also searched inside each fold, as for every system, AERIN scores higher than WavLM-base on 6 of 10 corpora. It is significantly higher on 3 (MODMA +0.150, NV-SV +0.017, RAVDESS +0.045) and significantly lower on 1 (NV-LR −0.004). The gain is real, but it is limited to a few corpora.

Table R3.2. Possible sources of AERIN's gain over WavLM-base, how we tested each one, and the result. Counts are over 10 corpora. For comparisons with WavLM-base, "significantly" means the 95% bootstrap interval does not contain zero; for 6-epoch training arms, "better / worse" counts resolved differences (see Table R3.3).

Possible sourceHow we tested itResult
How the teacher is readAERIN vs. WavLM-base read three ways: best single layer; layers 3, 7, 11 concatenated; learned mix of all 13 layersAERIN higher on 6 / 7 / 8 of 10; significantly higher on 3 / 2 / 2, significantly lower on 1 / 0 / 1
Temporal-token distillationRemovedWorse on 3 corpora
Multi-layer distillationFinal teacher layer onlyWorse on 3 corpora, better on 1
Choice of teacher layers and weightsLayers 4/8/12; uniform depth weightsNo resolved difference
Auxiliary distillation termsCosine alignment only (MSE and relational terms off)No resolved difference
Anti-collapse regularizer (Lgeo\mathcal{L}_{\text{geo}})RemovedWorse on 1 corpus
Cross-state decorrelation (Lsep\mathcal{L}_{\text{sep}})RemovedNo resolved difference
Read-out constructionAERIN chooses among 3 read-outs; the teacher among 13 layers plus 2 combined read-outsThe search favors the teacher, not AERIN

(b) Multi-layer aggregation of the teacher. We also read WavLM-base in two multi-layer ways: with layers 3, 7, and 11 concatenated (exactly the layers AERIN is distilled from), and with a SUPERB-style learned softmax over all 13 layers. AERIN scores higher on 7 and 8 of 10 corpora, respectively (significantly higher on 2 and 2; significantly lower on 0 and 1). Aggregating the teacher's layers does not close the gap. On some corpora, the aggregated read-outs are even below the teacher's best single layer (NV-H/P: 0.717 for the concatenation versus 0.858).

(c) Distillation targets and regularization. Six ablations each change one part of the objective (Table R3.3). The gains rest on two parts. Removing temporal-token distillation lowers NV-LR, RAVDESS, and ShEMO. Distilling from the final teacher layer only lowers NV-SV, RAVDESS, and ShEMO. The regularizers matter less: removing the anti-collapse term lowers ShEMO only, and removing cross-state decorrelation, the auxiliary distillation terms, or the depth weighting has no resolved effect. Each arm is compared with a reference trained on the same pipeline; the two 6-epoch references used across our responses do not differ on any corpus beyond noise.

Table R3.3. Change in score when one part of the distillation objective is changed (score of variant − score of reference). Bold: resolved, meaning the 95% speaker-clustered bootstrap interval does not contain zero, and the difference is larger than the seed gap (last column).

CorpusNo temporal-token KDCosine KD onlyUniform depth weightsFinal layer onlyNo anti-collapseNo decorrelationSeed gap
DAIC+0.091+0.091+0.030+0.182+0.152−0.0910.121
MODMA+0.044−0.003−0.066−0.051−0.066−0.0200.037
IPVS+0.006−0.007−0.006−0.017−0.013−0.0050.006
NV-H/P−0.003−0.026−0.012+0.049−0.027−0.0070.010
NV-SV−0.011+0.000−0.003−0.028−0.010+0.0060.016
NV-GM−0.006+0.002−0.000+0.002+0.002+0.0030.000
NV-LR−0.006−0.003+0.002−0.000+0.002−0.0030.001
TESS−0.012+0.003+0.002−0.002+0.004−0.0030.002
RAV−0.035+0.007+0.018−0.061−0.014−0.0120.001
ShEMO−0.020−0.012−0.002−0.041−0.027+0.0110.002
Resolved gain / loss0 / 30 / 00 / 01 / 30 / 10 / 0

W2 (d) Training data. AERIN's distillation mixture differs from WavLM-base's pre-training data. AERIN uses 539 h from nine corpora; about two-thirds of it is conversational, and it includes acted and spontaneous emotional speech from IEMOCAP, MELD, CREMA-D, and MSP-Podcast (Appendix B.1). WavLM-base was pre-trained on 960 h of read LibriSpeech. Each student above uses the same mixture, so the data cannot account for differences between students. It likely explains part of every student's gain over the teacher, especially on the affective corpora. We could not train a LibriSpeech-only control within the rebuttal period.

W2 (e) Read-out construction. AERIN chooses among three read-outs inside each fold, whereas WavLM-base chooses among 13 layers, and the two aggregated read-outs in (b) give the teacher the same multi-layer access. The read-out search therefore does not favor AERIN.

W3: The presentation is difficult to follow

We thank the reviewer for this feedback and agree that the framework is harder to follow than it should be. We will make the following changes in the revised manuscript:

  1. Overview. Open Section 2 with one paragraph that follows a 5-s chunk through the encoder and gives the input and output of each stage, so that the computation can be reconstructed from the main text alone.
  2. Consistency. Use one stream order, one name for each read-out and evaluation task, and one statement of the number of read-outs throughout the paper, and add a notation table to the appendix.
  3. Readability. Shorten long sentences and reduce cross-references between the main text and the appendix.

Que1: We thank the reviewer for this question. Details are in W2 and Table R3.2. With multi-layer teacher baselines, AERIN still scores higher than WavLM-base with layers 3, 7, and 11 concatenated (7 of 10 corpora; significantly higher on 2, lower on 0) and with a learned mix of all 13 layers (8 of 10; significantly higher on 2, lower on 1). For regularization, removing the anti-collapse term lowers one corpus and removing decorrelation lowers none, whereas removing temporal-token distillation or the multi-layer ladder lowers three each. To study the effect of stream organization, we need to retrain the models by removing 1 stream at a time, which was not possible for us in the rebuttal period; we will include this in the camera-ready version.

Que2: We thank the reviewer for this suggestion. Our reported probe already has one hidden layer (Appendix E.2: LayerNorm, Linear(3D, 128), GELU, dropout, Linear). We added two deeper MLP heads: one with two hidden layers (512 and 256 units) and one with three hidden layers (1024, 512, and 256 units). We evaluated seven systems with them under the same protocol: AERIN, DistilHuBERT, DPHuBERT, DPWavLM, HuBERT-base, WavLM-base, and WavLM-large. Each system keeps the read-out it selected with the reported probe, so probe depth is the only change (Table R3.4).

Table R3.4. AERIN's standing among the seven systems with probes of increasing depth (mean rank over 10 corpora; 1 is best).

ProbeAERIN mean rank [95% CI]AERIN's place (of 7)Best other system (mean rank)AERIN higher than WavLM-base
Reported (one hidden layer, 128)3.20 [2.70, 4.15]2DPWavLM (3.05)6 of 10
Two hidden layers (512, 256)3.60 [3.20, 4.30]3WavLM-large (2.75)6 of 10
Three hidden layers (1024, 512, 256)2.80 [2.40, 3.90]1 (tied)DPWavLM, WavLM-large (2.80)8 of 10

AERIN's overall ranking remains largely unchanged, placing second, third, and tied for first across the three evaluations, respectively, among seven systems. Increasing the head depth shifts individual scores in both directions across all systems. For AERIN, the scores on MODMA decrease from 0.787 to 0.674 and 0.677, whereas those on NV-SV increase from 0.939 to 0.944 and 0.941. With the three-hidden-layer evaluation head, AERIN outperforms DistilHuBERT on 9 of the 10 corpora and HuBERT-base on 9 of the 10 corpora. These results demonstrate that AERIN's relative advantage is not merely an artifact of using a shallow evaluation head. Investigating more complex downstream systems built upon AERIN is an interesting direction for future work.

Reviewer_eL6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AERIN, a compact speech encoder that organizes acoustic evidence into Boundary, Envelope, Phonation, and Texture streams, which are integrated via competitive routing and contextualization. Its hierarchical representations are distilled from multiple layers of WavLM-base without downstream task labels. Evaluations on 11 clinical, phonetic, and affective tasks show competitive performance with only 25.66M parameters. Stream interventions and routing analyses further examine how the model uses the different evidence sources.

Critical strengths
  • AERIN effectively combines explicit acoustic evidence with flexible representation learning, which can provide a promising inductive bias for capturing diverse speech properties. The use of explicit evidence also supports analysis of model behavior, offering interpretability that can be important to clinical applications.
  • Evaluation spans a diverse set of tasks, comprising clinical, phonetic, and affective tasks across five languages (including unseen one). Achieving competitive performance with a substantially fewer parameters than the teacher demonstrates the effectiveness of the compact speech representation.
Critical weaknesses
  • The architectural benefits of AERIN are not clearly validated. Comparisons with prior distilled encoders with the similar size and controlled ablations are needed to isolate the contributions of evidence organization and adaptive routing from the effect of distillation itself, which is quite powerful.
  • Statistical support for the overall comparisons is limited. Given the evaluation protocol limitations acknowledged in Sec. 3.1 and App. E.7, reporting uncertainty estimates would help accurately interpret the score differences and the resulting ranking.
Questions for authors
  • How does AERIN compare to distilled encoders, e.g., DistilHuBERT and DP-HuBERT/WavLM, under the same evaluation protocol? I understand that evidence organization is the authors' focus, but comparison with similarly sized, distilled models would clarify if explicit evidence organization improve representations or not.
  • How does AERIN perform on the full SUPERB benchmark? The current evaluation suite appears to sufficiently cover the paper's scope, but these results would facilitate comparison with prior work under standardized protocols.
  • How effective is learned routing compared to uniform or learned static gating? Controlled training ablations would help support the benefits of AERIN
  • How were WavLM layers 3, 7, and 11 selected for distillation? How sensitive is AERIN's performance to layer choices?
5 comments

We sincerely thank Reviewer eL6E for the thorough and insightful review and for providing valuable constructive feedback. We appreciate the reviewer's suggestions, which will help us further strengthen and improve our manuscript. While re-running the evaluation to add the uncertainty estimates the reviewer asked for, we found and corrected several problems in the submitted protocol; we describe them in full in W2. Below, we provide detailed responses to the reviewer's comments and questions.

W1: Architectural benefits are not validated against similarly sized distilled encoders or controlled ablations

Thank you for your valuable comment, which helped us improve the manuscript.

(a) We evaluated DistilHuBERT (23.5M parameters), DPHuBERT (23.6M) and DPWavLM (23.6M) under the same protocol as AERIN (25.7M), with all their layers searched (Table R2.1).

Table R2.1. Distilled encoders of comparable size under the corrected evaluation protocol (metrics as in Table 1; AERIN is the submitted checkpoint). Bold indicates the best result in each column.

SystemParams (M)DAICMODMAIPVSNV-H/PNV-SVNV-GMNV-LRTESSRAVShEMO
AERIN25.70.3030.7870.9630.8660.9390.9800.9690.9030.7050.663
DistilHuBERT23.50.4240.6620.9420.8630.9240.9740.9630.8980.6960.647
DPHuBERT23.60.4550.7530.9470.8520.9180.9670.9750.9230.6970.632
DPWavLM23.60.4550.7050.9530.8360.9280.9810.9740.9070.7210.683
WavLM-base (teacher)94.40.3330.6370.9470.8580.9220.9830.9740.9040.6600.637

AERIN is higher than DistilHuBERT on 9 of 10 corpora (significantly higher on 2, significantly lower on 0), than DPHuBERT on 7 (2 higher, 2 lower) and than DPWavLM on 4 (0 higher, 1 lower), where "significantly" means that the 95% paired bootstrap interval of the difference lies entirely above or below zero. Across all 16 systems evaluated on 10 datasets (excluding SAND), DPWavLM's mean rank is 6.05 and AERIN's 6.50. AERIN is therefore on par with the strongest distilled encoder of its size and ahead of DistilHuBERT, but not clearly better than both.

(b) Routing is isolated by three trained alternatives and three inference-time interventions; please refer to our response to Que3.

These results show that the evidence organization does not by itself improve accuracy over distillation: AERIN is comparable to the strongest distilled encoders of its size. What the evidence organization does provide is an encoder whose evidence pathways can be tested. Three of the four streams verifiably encode the evidence they are designed to encode, and the trained model's performance depends on its input-dependent routing (our response to Reviewer tnp2, W1, and W2). As the reviewer notes, this kind of analysis can be important for clinical applications. We will revise the paper to claim exactly this.

W2: Statistical support for the comparisons and the ranking is limited

We thank the reviewer for this important suggestion and agree that the differences in Table 1 cannot be interpreted without uncertainty estimates. We now report them for every score, every pairwise difference and the ranking itself.

(a) How uncertainty is estimated. Speakers are resampled 2,000 times (recordings, where recordings are the independent units; Appendix E.1), jointly for all systems, so that each resample re-scores every system on the same test units and differences between systems are paired. We report 95% percentile intervals for every per-corpus score, every pairwise difference and every system's mean rank, together with two-sided bootstrap p-values with Holm correction across the 10 corpora. SAND can be scored only by its challenge server, so it is excluded from these computations, which use the 10 remaining corpora.

(b) Uncertainty of the score differences. Table R2.3 counts, for each encoder of Table 1, the corpora on which AERIN is significantly higher or lower. Across the 110 AERIN-versus-encoder comparisons, 37 are significantly in AERIN's favour, 13 are significantly against it, and 60 cannot be distinguished from noise. Against its closest competitors, the picture is balanced (WavLM-large: 3 higher, 3 lower; WavLM-base+: 1 higher, 1 lower), whereas against several larger encoders AERIN is significantly higher more often than lower (for example, HuBERT-large, 3 versus 0, and emotion2vec+ large, 6 versus 2).

(c) Corpora affected by the limitations in Section 3.1 and Appendix E.7. The intervals are widest where these limitations apply, which makes them explicit in the comparison. On DAIC-WOZ (35 test participants), AERIN's F1 is 0.303 [0.044, 0.525] with an AUC of 0.630, so differences of this size between systems are not interpretable; we therefore report AUC beside F1 for every binary corpus. TESS contains only two actors, so its interval cannot reflect speaker variability, and its score remains a within-speaker estimate. The four NeuroVoz evaluations share speakers, so they are not independent evidence in the mean rank.

We will include the per-corpus intervals for every system, AUC for the binary corpora, mean-rank intervals, and all pairwise intervals in the appendix of the camera-ready version.

Que1: We thank the reviewer for this question. Please refer to W1(a) and Table R2.1. AERIN is higher than DistilHuBERT on 9 of 10 corpora, than DPHuBERT on 7, and than DPWavLM on 4; it is significantly higher on 2, 2, and 0 corpora and significantly lower on 0, 2, and 1, respectively. The ordering holds with deeper probes (please refer our response to Reviewer zCeF, Que 2): with a three-hidden-layer MLP probe, AERIN is higher than DistilHuBERT on 9 of 10 corpora (significantly higher on 5, lower on 0) and than DPWavLM on 6 (significantly higher on 0, lower on 1). AERIN is therefore comparable to the best similarly sized distilled encoders.

Que2: We thank the reviewer for this suggestion and agree that SUPERB results would ease comparison with prior work. However, AERIN's distillation framework supervises pooled chunk representations (at the recording level) and 16 temporal tokens per chunk, rather than individual frames. Consequently, AERIN is not designed to preserve fine-grained frame-level information. Therefore, AERIN may not match state-of-the-art models on phoneme classification and automatic speech recognition tasks. Within the rebuttal period, we ran the two SUPERB tasks that test frame-level content: phoneme recognition (PR) and Automatic speech recognition (ASR), which were trained for 15 epochs on a 25-hour subset of LibriSpeech train-clean-100. These are therefore not official SUPERB scores, but all systems share the same setup. AERIN achieves a phone error rate (PER) of 22.4% and a word error rate (WER) of 31.8% on the test-clean set, compared to 15.2% and 28.4% for DistilHuBERT, and 7.9% and 13.3% for WavLM-Base, respectively. For results related to the keyword spotting task, please refer to our response to reviewer tnp2 (W4).

Que3: Please refer to our response to Reviewer tnp2, W2(b).

Que4: The ladder was set from a per-layer probe of the frozen WavLM-base teacher on DAIC-WOZ: a logistic regression probe on participant-mean features for each layer. That probe ranked layers 3, 7, 8, and 11 highest at their depths; we kept layers 3, 7, and 11 and skipped layer 8 to distribute supervision across the teacher's depth. In that run, the layers were scored on the DAIC-WOZ development split, which is also the split we report, so labels of one reported evaluation split influenced this configuration choice, although no label entered any training gradient. The sensitivity results below indicate that AERIN does not depend critically on this choice.

Three 6-epoch arms change the ladder (Table R2.4): moving it to layers 4, 8, and 12 (DistilHuBERT's spacing; new targets computed from WavLM-base), using uniform depth weights, and distilling only from the final teacher layer.

Table R2.4. Change in score when the model is distilled with a different teacher-layer ladder (score of variant − score of reference). Bold: resolved, meaning the 95% speaker-clustered bootstrap interval does not contain zero, and the difference is larger than the seed gap.

CorpusLayers 4/8/12Uniform depth weightsFinal layer onlySeed gap
DAIC−0.091 [−0.288, +0.146]+0.030 [−0.203, +0.247]+0.182 [−0.013, +0.406]0.121
MODMA−0.019 [−0.117, +0.083]−0.066 [−0.170, +0.036]−0.051 [−0.188, +0.102]0.037
IPVS−0.004 [−0.032, +0.022]−0.006 [−0.020, +0.000]−0.017 [−0.040, +0.000]0.006
NV-H/P−0.020 [−0.053, +0.010]−0.012 [−0.042, +0.017]+0.049 [+0.003, +0.101]0.010
NV-SV+0.006 [−0.007, +0.018]−0.003 [−0.018, +0.011]−0.028 [−0.046, −0.013]0.016
NV-GM−0.002 [−0.008, +0.003]−0.000 [−0.006, +0.006]+0.002 [+0.000, +0.005]0.000
NV-LR−0.001 [−0.006, +0.004]+0.002 [−0.003, +0.008]−0.000 [−0.006, +0.005]0.001
TESS−0.002 [−0.011, +0.010]+0.002 [−0.012, +0.016]−0.002 [−0.011, +0.009]0.002
RAV−0.018 [−0.040, +0.004]+0.018 [−0.009, +0.046]−0.061 [−0.090, −0.033]0.001
ShEMO−0.011 [−0.029, +0.005]−0.002 [−0.024, +0.022]−0.041 [−0.062, −0.020]0.002

Moving the ladder to 4/8/12 gives no resolved difference on any corpus, although point estimates are lower on 9 of 10, so 3/7/11 may be slightly better. AERIN depends on distilling from several depths, but not critically on which three layers are used or on their weights, within the range tested.

Reviewer_tnp2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces AERIN, a compact speech encoder that integrates four acoustic evidence streams through learned routing. The model is distilled from WavLM-base and achieves competitive performance across diverse speech tasks with only about 26M parameters.

Critical strengths

This work presents a well-motivated approach to combining speech representations with different types of explicit acoustic evidence while retaining flexibility through learned routing and contextual integration. The proposed model is compact and achieves competitive performance across diverse clinical, phonetic, and affective tasks.

Critical weaknesses

My main concern is that, although the paper includes stream ablations and routing analyses, these experiments are insufficient to support the proposed design choices. The stream ablations only remove or retain certain streams at test time. Therefore, they show how the model depends on each stream but do not establish whether the four streams actually capture the acoustic evidence they are designed to represent. There are no targeted acoustic tasks to verify whether each stream captures its intended information. Although the paper compares learned routing with uniform routing, it only reports changes in representation similarity without evaluating the impact on downstream performance. Therefore, it remains unclear whether the proposed fine-grained routing outperforms simpler routing methods. The paper also lacks ablations of the contextualization modules. Since the main contribution of this work is its architectural design, more thorough ablation studies are important to justify the proposed design.

Additionally, the evaluation is limited to clinical, phonetic, and affective tasks, without general-purpose evaluation on benchmarks such as SUPERB.

Questions for authors

Can you verify whether each stream captures its intended acoustic evidence?

Can you provide additional ablations to justify the routing and contextual integration designs?

Can you evaluate AERIN on a broader range of tasks, such as those in SUPERB, to better assess its generalizability?

7 comments

We sincerely thank reviewer tnp2 for their comprehensive and thoughtful review and their valuable, constructive feedback. We retrained the models for the routing and contextualization ablation studies and report the results in Tables R1.4 and R1.5, using only 6 epochs because time constraints prevented us from running all experiments for the full 20 epochs. The results from the full 20-epoch experiments will be included in the revised version.

W1: Stream ablations do not show whether each stream captures its intended acoustic evidence

We sincerely thank the reviewer for this insightful and constructive feedback. We agree that removing or retaining streams at test time measures how the trained model depends on each stream, not what each stream encodes. We added three targeted tests. These tests use a retrained version of AERIN with the same configuration and seed (20 epochs, read at its final epoch, so no downstream label selects the checkpoint). On the panel, it does not differ significantly from the submitted model on any corpus.

(a) Controlled signals. We synthesized 5-s harmonic signals in which F0 (70 to 350 Hz), amplitude-modulation (AM) rate (1.5 to 12 Hz), pause ratio (0 to 0.5), and spectral tilt are drawn independently, and fitted cross-validated ridge regressions from each stream's private front end (its output before it is conditioned on the shared stem or mixed with any other stream) to each factor (Table R1.1). A shuffled-target control stays below R2=0.01R^2 = 0.01.

Table R1.1. Cross-validated R2R^2 scores for each private front end using controlled signals. Bold indicates the factor that the stream is designed to capture.

Front endF0 (log)AM ratePause ratioSpectral tilt
Phonation0.9970.7330.9940.441
Envelope0.5050.9441.0000.000
Boundary0.5990.9170.8730.000
Texture0.9890.5950.9960.990
Shared stem (context)0.9930.7300.9980.987

Phonation, Envelope, and Texture give the highest R2R^2 of the four front ends for the factor each is designed for, although on these clean harmonic signals F0 is also decodable from Texture (0.989). The amplitude-based streams are selective for spectral tilt: Envelope and Boundary carry no tilt information (R2R^2 = 0.000) and only partial F0 information (0.505 and 0.599). The Boundary stream does not behave as designed. Its front end tracks AM rate (0.917) but is the weakest of the four on pause ratio, a factor every front end decodes well on these signals.

(b) Real speech with independent targets. On RAVDESS speech (1,440 recordings) we measured voice properties with Praat (via parselmouth), independently of any computation inside AERIN, so the Phonation test is not autocorrelation scored against itself. Folds are grouped by speaker (Table R1.2).

Table R1.2. Cross-validated R2R^2 scores of Praat measurements from each private front end using RAVDESS speech. Bold indicates the measurements assigned to each stream in advance.

Front endF0 medianJitterHNRVoiced ratioIntensityShimmerSpectral centroid
Phonation0.810.430.500.530.550.340.28
Envelope0.340.190.240.390.990.260.17
Boundary0.26−0.17−0.040.030.84−0.190.12
Texture0.730.320.490.420.700.230.76

The pattern matches the controlled test. Phonation best predicts F0 and jitter and is level with Texture on HNR; Envelope best predicts intensity; Texture best predicts spectral centroid. Boundary is again the exception: it predicts the voiced-frame ratio worst of the four. Voiced ratio also reflects voicing, and these single-sentence recordings contain few pauses, so it is an imperfect target for Boundary; together with (a), however, it indicates that Boundary does not specialise as intended. Shimmer, which we had grouped with Envelope, is predicted slightly better by Phonation.

(c) Causal test. On the controlled signals, we lesioned one stream at a time and measured how much each factor's decodability falls from the read-outs ZlowZ_{\text{low}} and ZhighZ_{\text{high}}. The only sizeable loss is AM rate after removing Envelope (ΔR2\Delta R^2 = 0.048 and 0.066). Every other designed-factor loss is at most 0.008, and no off-design change exceeds 0.014. After shared-stem conditioning and contextualization, each factor is available from several streams, so the read-outs do not depend on any single stream. This agrees with Section 3.2, which notes that the ablations do not assign unique information to one stream.

Three of the four streams capture the evidence they were designed to represent, whereas the Boundary stream does not specialize as intended. We will add these tests to the paper, report the Boundary stream results, and revise its description accordingly.

W2: Routing is compared only by representation similarity, not downstream; is fine-grained routing better than simpler routing?

We appreciate the reviewer's insightful comment. We now evaluate routing downstream in two ways.

(a) Interventions on the trained model. We replaced the learned gates at inference by uniform gates (1/4 per stream), by each stream's mean gate (no input dependence), or by the model's own gates shuffled in time within each chunk, then re-encoded and re-scored every corpus (Table R1.3). Each cell gives the change in score followed by its 95% speaker-clustered bootstrap interval in brackets. If the whole interval lies below zero, the drop is significant (roughly p < 0.05); if the interval contains zero, the change cannot be distinguished from noise.

Table R1.3. Change in score when the learned gates of the trained AERIN are replaced at inference (score with intervention − score with learned gates). Bold: the 95% speaker-clustered bootstrap interval does not contain zero. †: Holm-adjusted p < 0.05 across the 10 corpora.

CorpusUniform 1/4Mean gate per streamGates shuffled in time
DAIC+0.121 [−0.097, +0.387]+0.091 [−0.126, +0.326]+0.242 [−0.003, +0.515]
MODMA+0.034 [−0.039, +0.109]+0.002 [−0.083, +0.080]+0.018 [−0.069, +0.111]
IPVS+0.000 [−0.019, +0.018]−0.018 [−0.042, +0.000]−0.048 [−0.091, −0.016] †
NV-H/P+0.029 [−0.013, +0.076]−0.012 [−0.051, +0.026]−0.039 [−0.080, −0.002]
NV-SV−0.017 [−0.036, +0.002]−0.020 [−0.036, −0.003]+0.000 [−0.011, +0.012]
NV-GM−0.021 [−0.041, −0.005]−0.010 [−0.021, +0.001]−0.009 [−0.020, −0.002]
NV-LR−0.004 [−0.010, +0.002]−0.008 [−0.016, −0.002] †−0.013 [−0.021, −0.007] †
TESS−0.014 [−0.030, +0.001]−0.016 [−0.032, +0.001]−0.020 [−0.040, +0.001]
RAV−0.034 [−0.061, −0.006]−0.045 [−0.068, −0.021] †−0.051 [−0.080, −0.018] †
ShEMO−0.055 [−0.078, −0.030] †−0.054 [−0.075, −0.033] †−0.058 [−0.078, −0.037] †
Intervals below / above zero3 / 04 / 06 / 0

No intervention significantly improves any corpus. Each significantly lowers performance (95% interval entirely below zero) on 3, 4 and 6 of the 10 corpora respectively, most consistently on RAVDESS and ShEMO. On DAIC-WOZ all three interventions raise the point estimate, but every interval contains zero (e.g. +0.242 [−0.003, +0.515] for shuffled gates), so these gains are not significant; DAIC-WOZ F1 rests on few positive test cases and is the noisiest corpus in the panel. The trained representation therefore uses the input-dependent, time-varying allocation, not only its average mixture. Because these interventions perturb a model that was trained with learned gates, they show this dependence rather than which routing design is better; (b) addresses that question.

W2 (b) Trained alternatives. To test whether fine-grained routing beats simpler routing when the model is trained with it, we trained three alternatives from scratch over 6 epochs and compared each with a 6-epoch reference trained identically: uniform gates; static learned gates (one learned weight per stream and feature, no input dependence); and independent sigmoid gates (the same gate networks without competition across streams, so capacity is matched). Table R1.4 shows the differences with the reported model.

Table R1.4. Change in score when the model is trained with a simpler routing instead of learned competitive routing (score of alternative − score of reference). Bold: resolved, meaning the 95% speaker-clustered bootstrap interval does not contain zero, and the difference is larger than the seed gap. Interval bounds that round to ±0.000 are treated as touching zero. Uniform and static arms have 22.5M parameters instead of 25.7M because removing the gate networks reduces the parameter count by 3.1 M.

CorpusUniform gatesStatic learned gatesIndependent sigmoidSeed gap
DAIC−0.061 [−0.221, +0.109]+0.000 [−0.180, +0.207]−0.121 [−0.330, +0.086]0.121
MODMA−0.082 [−0.243, +0.074]−0.031 [−0.152, +0.083]+0.023 [−0.031, +0.086]0.037
IPVS−0.005 [−0.033, +0.022]−0.013 [−0.041, +0.013]+0.000 [+0.000, +0.000]0.006
NV-H/P−0.019 [−0.055, +0.012]−0.014 [−0.047, +0.014]−0.059 [−0.107, −0.013]0.010
NV-SV+0.005 [−0.006, +0.016]+0.009 [−0.000, +0.018]−0.005 [−0.017, +0.006]0.016
NV-GM+0.003 [+0.000, +0.008]+0.003 [+0.000, +0.008]+0.003 [+0.000, +0.008]0.000
NV-LR−0.002 [−0.006, +0.002]−0.006 [−0.011, −0.002]−0.003 [−0.007, +0.001]0.001
TESS+0.001 [−0.011, +0.013]+0.004 [−0.009, +0.020]−0.000 [−0.013, +0.012]0.002
RAV−0.002 [−0.022, +0.018]−0.006 [−0.021, +0.010]−0.014 [−0.031, +0.001]0.001
ShEMO−0.015 [−0.034, +0.004]−0.018 [−0.035, −0.000]−0.006 [−0.022, +0.010]0.002
Resolved gain / loss0 / 00 / 10 / 1

No trained alternative improves any corpus beyond noise. Static gating loses on 1 corpus (NV-LR) and independent gating on 1 (NV-H/P), indicating that both input dependence and competition across streams contribute on some corpora. A model trained with uniform routing shows no resolved difference from the reference, although it is lower on 7 of 10 corpora. Fine-grained competitive routing is therefore never worse than these simpler designs beyond noise, but its trained advantage over uniform routing is not resolved at this schedule. In the revision we will replace the cosine-similarity statement in Section 3.3 with these downstream results and describe the effect at this size.

W3: No ablations of the contextualization modules

We thank the reviewer for this valuable suggestion. We trained three contextualization ablations from scratch on the same 6-epoch schedule (Table R1.5): removing the token Transformer, removing cross-stream attention from the axial mixer, and removing the measured attention prior from the token stage (capacity unchanged).

Table R1.5. Change in score when the model is trained without a contextualization module (score without module − score of reference). Bold: resolved (same rule as Table R1.4).

CorpusNo token Transformer (19.9M)No cross-stream attention (23.9M)No measured priorSeed gap
DAIC−0.061 [−0.280, +0.181]−0.152 [−0.289, −0.027]−0.091 [−0.220, +0.009]0.121
MODMA−0.019 [−0.124, +0.088]−0.098 [−0.230, +0.026]−0.042 [−0.140, +0.061]0.037
IPVS−0.016 [−0.053, +0.011]+0.007 [+0.000, +0.023]+0.007 [+0.000, +0.023]0.006
NV-H/P−0.021 [−0.063, +0.019]+0.008 [−0.035, +0.053]−0.029 [−0.064, +0.001]0.010
NV-SV+0.005 [−0.007, +0.020]+0.005 [−0.007, +0.016]−0.008 [−0.018, +0.001]0.016
NV-GM−0.002 [−0.010, +0.005]+0.005 [−0.006, +0.017]+0.002 [+0.000, +0.005]0.000
NV-LR−0.003 [−0.008, +0.002]−0.001 [−0.005, +0.003]−0.002 [−0.006, +0.002]0.001
TESS+0.010 [−0.001, +0.022]−0.004 [−0.015, +0.007]+0.003 [−0.007, +0.014]0.002
RAV−0.028 [−0.048, −0.008]−0.013 [−0.036, +0.008]−0.004 [−0.017, +0.009]0.001
ShEMO+0.012 [−0.006, +0.027]+0.004 [−0.016, +0.022]+0.002 [−0.013, +0.016]0.002
Resolved gain / loss0 / 10 / 10 / 0

No ablation improves any corpus beyond noise. Removing the token Transformer costs on RAVDESS; removing cross-stream attention costs on DAIC-WOZ; removing the measured prior yields no significant difference, although it is lower on 6 of 10 corpora. Each module therefore contributes to at most one corpus at this schedule. We will add these ablations to the paper and describe the contributions of the contextualization modules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.