- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes AERIN, a 26M speech encoder trained with speech attribute factorization and knowledge distillation. It organizes waveform-derived information into 4 streams: Boundary, Envelope, Phonation, and Texture. These streams through competitive routing and contextualization with intra stream attention over the temporal axis and inter-stream attention at the same frame. It learns hierarchical read-outs through multi-layer knowledge distillation with WavLM-Base as the teacher. The frozen encoder is evaluated on 11 clinical, phonetic, and affective tasks from eight corpora, with an additional Speech Commands experiment. Reported results suggest competitive transfer with substantially fewer parameters than the teacher.
Critical strengths
-
The proposed architecture is well-motivated. AERIN utilized factorised speech acoustic information, combining explicit amplitude, timing, and periodicity measurements with learned waveform features. This is a plausible inductive bias for clinical and paralinguistic applications. The design offers a concrete hypothesis about how acoustic evidence can be organized within a reusable encoder.
-
The transfer performance is promising. The same frozen encoder remains competitive across heterogeneous tasks and several languages absent from its distillation data, despite of using much fewer model parameters. Separation between distillation and evaluation corpora strengthens the evidence for transfer, subject to the evaluation limitations below.
Critical weaknesses
-
Important comparisons is missing. For example, SPEAR[1] is missing from both the related-work discussion and evaluation, leaving the comparison with another unified speech representation method incomplete.
-
Explanations of the performance gains over WavLM-base is insufficient. WavLM-Base is the teacher model, and it's unusual that a KD-based method can exceed its own teacher. This was not explained enough through experiments separating acoustic stream organization from multi-layer distillation, representation regularization, additional training data, and read-out construction. An explanation on would help other researchers better understand the source of gains of the proposed method.
-
The presentation makes the framework unnecessarily difficult to follow. Reconstructing the computation requires substantial cross-referencing between the main text and appendix. Figure 1 combines many stages, feedback paths, and auxiliary connections, with small labels that obscure the main information flow. Section 2.6 introduces the overall objective while deferring the mathematical definitions of its core components to Appendix D. These definitions should appear alongside the objective in the main text, with implementation details and derivations retained in the appendix. Clearer explanations of shared-stem conditioning, residual routing, and the read-out hierarchy are also needed.
[1] Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations". in Proc. ICML 2026
Questions for authors
-
What explains AERIN’s improvements over WavLM-base? Please consider a teacher baseline with multi-layer aggregation. Can the effects of stream organization and representation regularization be isolated?
-
Does AERIN’s relative advantage persist with a moderately stronger probe, such as a deeper MLP? Speech SSL model's representation tend to be highly engangled and could require more parameter to better digest. This would clarify whether the current small probe particularly benefits the organization of AERIN’s read-outs.
-
Can AERIN’s frame-level representations support phoneme classification or speech recognition?
6 comments
W2: The gains over the WavLM-base teacher are not explained
We thank the reviewer for this important point. We agree that a student that outperforms its teacher needs an explanation. We tested each candidate source the reviewer listed, one at a time. Table R3.2 summarizes the results, and the paragraphs below give the details.
(a) Size of the gain. When the teacher's layers are also searched inside each fold, as for every system, AERIN scores higher than WavLM-base on 6 of 10 corpora. It is significantly higher on 3 (MODMA +0.150, NV-SV +0.017, RAVDESS +0.045) and significantly lower on 1 (NV-LR −0.004). The gain is real, but it is limited to a few corpora.
Table R3.2. Possible sources of AERIN's gain over WavLM-base, how we tested each one, and the result. Counts are over 10 corpora. For comparisons with WavLM-base, "significantly" means the 95% bootstrap interval does not contain zero; for 6-epoch training arms, "better / worse" counts resolved differences (see Table R3.3).
| Possible source | How we tested it | Result |
|---|---|---|
| How the teacher is read | AERIN vs. WavLM-base read three ways: best single layer; layers 3, 7, 11 concatenated; learned mix of all 13 layers | AERIN higher on 6 / 7 / 8 of 10; significantly higher on 3 / 2 / 2, significantly lower on 1 / 0 / 1 |
| Temporal-token distillation | Removed | Worse on 3 corpora |
| Multi-layer distillation | Final teacher layer only | Worse on 3 corpora, better on 1 |
| Choice of teacher layers and weights | Layers 4/8/12; uniform depth weights | No resolved difference |
| Auxiliary distillation terms | Cosine alignment only (MSE and relational terms off) | No resolved difference |
| Anti-collapse regularizer () | Removed | Worse on 1 corpus |
| Cross-state decorrelation () | Removed | No resolved difference |
| Read-out construction | AERIN chooses among 3 read-outs; the teacher among 13 layers plus 2 combined read-outs | The search favors the teacher, not AERIN |
(b) Multi-layer aggregation of the teacher. We also read WavLM-base in two multi-layer ways: with layers 3, 7, and 11 concatenated (exactly the layers AERIN is distilled from), and with a SUPERB-style learned softmax over all 13 layers. AERIN scores higher on 7 and 8 of 10 corpora, respectively (significantly higher on 2 and 2; significantly lower on 0 and 1). Aggregating the teacher's layers does not close the gap. On some corpora, the aggregated read-outs are even below the teacher's best single layer (NV-H/P: 0.717 for the concatenation versus 0.858).
(c) Distillation targets and regularization. Six ablations each change one part of the objective (Table R3.3). The gains rest on two parts. Removing temporal-token distillation lowers NV-LR, RAVDESS, and ShEMO. Distilling from the final teacher layer only lowers NV-SV, RAVDESS, and ShEMO. The regularizers matter less: removing the anti-collapse term lowers ShEMO only, and removing cross-state decorrelation, the auxiliary distillation terms, or the depth weighting has no resolved effect. Each arm is compared with a reference trained on the same pipeline; the two 6-epoch references used across our responses do not differ on any corpus beyond noise.
Table R3.3. Change in score when one part of the distillation objective is changed (score of variant − score of reference). Bold: resolved, meaning the 95% speaker-clustered bootstrap interval does not contain zero, and the difference is larger than the seed gap (last column).
| Corpus | No temporal-token KD | Cosine KD only | Uniform depth weights | Final layer only | No anti-collapse | No decorrelation | Seed gap |
|---|---|---|---|---|---|---|---|
| DAIC | +0.091 | +0.091 | +0.030 | +0.182 | +0.152 | −0.091 | 0.121 |
| MODMA | +0.044 | −0.003 | −0.066 | −0.051 | −0.066 | −0.020 | 0.037 |
| IPVS | +0.006 | −0.007 | −0.006 | −0.017 | −0.013 | −0.005 | 0.006 |
| NV-H/P | −0.003 | −0.026 | −0.012 | +0.049 | −0.027 | −0.007 | 0.010 |
| NV-SV | −0.011 | +0.000 | −0.003 | −0.028 | −0.010 | +0.006 | 0.016 |
| NV-GM | −0.006 | +0.002 | −0.000 | +0.002 | +0.002 | +0.003 | 0.000 |
| NV-LR | −0.006 | −0.003 | +0.002 | −0.000 | +0.002 | −0.003 | 0.001 |
| TESS | −0.012 | +0.003 | +0.002 | −0.002 | +0.004 | −0.003 | 0.002 |
| RAV | −0.035 | +0.007 | +0.018 | −0.061 | −0.014 | −0.012 | 0.001 |
| ShEMO | −0.020 | −0.012 | −0.002 | −0.041 | −0.027 | +0.011 | 0.002 |
| Resolved gain / loss | 0 / 3 | 0 / 0 | 0 / 0 | 1 / 3 | 0 / 1 | 0 / 0 |
W2 (d) Training data. AERIN's distillation mixture differs from WavLM-base's pre-training data. AERIN uses 539 h from nine corpora; about two-thirds of it is conversational, and it includes acted and spontaneous emotional speech from IEMOCAP, MELD, CREMA-D, and MSP-Podcast (Appendix B.1). WavLM-base was pre-trained on 960 h of read LibriSpeech. Each student above uses the same mixture, so the data cannot account for differences between students. It likely explains part of every student's gain over the teacher, especially on the affective corpora. We could not train a LibriSpeech-only control within the rebuttal period.
W2 (e) Read-out construction. AERIN chooses among three read-outs inside each fold, whereas WavLM-base chooses among 13 layers, and the two aggregated read-outs in (b) give the teacher the same multi-layer access. The read-out search therefore does not favor AERIN.
W3: The presentation is difficult to follow
We thank the reviewer for this feedback and agree that the framework is harder to follow than it should be. We will make the following changes in the revised manuscript:
- Overview. Open Section 2 with one paragraph that follows a 5-s chunk through the encoder and gives the input and output of each stage, so that the computation can be reconstructed from the main text alone.
- Consistency. Use one stream order, one name for each read-out and evaluation task, and one statement of the number of read-outs throughout the paper, and add a notation table to the appendix.
- Readability. Shorten long sentences and reduce cross-references between the main text and the appendix.
Que1: We thank the reviewer for this question. Details are in W2 and Table R3.2. With multi-layer teacher baselines, AERIN still scores higher than WavLM-base with layers 3, 7, and 11 concatenated (7 of 10 corpora; significantly higher on 2, lower on 0) and with a learned mix of all 13 layers (8 of 10; significantly higher on 2, lower on 1). For regularization, removing the anti-collapse term lowers one corpus and removing decorrelation lowers none, whereas removing temporal-token distillation or the multi-layer ladder lowers three each. To study the effect of stream organization, we need to retrain the models by removing 1 stream at a time, which was not possible for us in the rebuttal period; we will include this in the camera-ready version.
Que2: We thank the reviewer for this suggestion. Our reported probe already has one hidden layer (Appendix E.2: LayerNorm, Linear(3D, 128), GELU, dropout, Linear). We added two deeper MLP heads: one with two hidden layers (512 and 256 units) and one with three hidden layers (1024, 512, and 256 units). We evaluated seven systems with them under the same protocol: AERIN, DistilHuBERT, DPHuBERT, DPWavLM, HuBERT-base, WavLM-base, and WavLM-large. Each system keeps the read-out it selected with the reported probe, so probe depth is the only change (Table R3.4).
Table R3.4. AERIN's standing among the seven systems with probes of increasing depth (mean rank over 10 corpora; 1 is best).
| Probe | AERIN mean rank [95% CI] | AERIN's place (of 7) | Best other system (mean rank) | AERIN higher than WavLM-base |
|---|---|---|---|---|
| Reported (one hidden layer, 128) | 3.20 [2.70, 4.15] | 2 | DPWavLM (3.05) | 6 of 10 |
| Two hidden layers (512, 256) | 3.60 [3.20, 4.30] | 3 | WavLM-large (2.75) | 6 of 10 |
| Three hidden layers (1024, 512, 256) | 2.80 [2.40, 3.90] | 1 (tied) | DPWavLM, WavLM-large (2.80) | 8 of 10 |
AERIN's overall ranking remains largely unchanged, placing second, third, and tied for first across the three evaluations, respectively, among seven systems. Increasing the head depth shifts individual scores in both directions across all systems. For AERIN, the scores on MODMA decrease from 0.787 to 0.674 and 0.677, whereas those on NV-SV increase from 0.939 to 0.944 and 0.941. With the three-hidden-layer evaluation head, AERIN outperforms DistilHuBERT on 9 of the 10 corpora and HuBERT-base on 9 of the 10 corpora. These results demonstrate that AERIN's relative advantage is not merely an artifact of using a shallow evaluation head. Investigating more complex downstream systems built upon AERIN is an interesting direction for future work.
We sincerely thank Reviewer zCeF for their thorough and insightful review and constructive feedback. We are grateful that the reviewer finds the architecture well motivated and its transfer performance promising. Below, we respond to each comment and question.
W1: SPEAR is missing from the related work and the evaluation
We thank the reviewer for pointing us to SPEAR. We added SPEAR-large (327M parameters; public checkpoint from Hugging Face) to the panel. It is evaluated under the same protocol as every other system, with all of its layers searched (Table R3.1).
Table R3.1. Scores of AERIN (retrained; details in response to Reviewer tnp2) and SPEAR-large, and their difference (AERIN − SPEAR). Bold: the 95% speaker-clustered bootstrap interval does not contain zero. †: Holm-adjusted p < 0.05 across the 10 corpora.
SPEAR-large is the strongest system we evaluated. Its mean rank among the 16 systems we now evaluate (the 12 encoders of Table 1 (main paper), three distilled encoders of similar size, and SPEAR-large) is 4.15 (excluding SAND), compared with 6.05 for DPWavLM and 6.50 for AERIN. AERIN scores higher on 3 of 10 corpora but is not significantly better on any. SPEAR is significantly better on three corpora: NV-LR, RAVDESS, and ShEMO. On the other seven, including all four clinical corpora, the two systems cannot be distinguished. SPEAR has 12.7 times as many parameters as AERIN.
We will add SPEAR to Table 1 (main paper) and to the related work (Appendix A.1). SPEAR learns a unified representation of speech and general audio by masked prediction of fine-grained discrete tokens derived from speech and audio SSL teachers. Like AERIN, it learns from teacher models, but it pursues breadth through large-scale pre-training, whereas AERIN is a compact encoder that organizes its representation around explicit acoustic evidence.