- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose AudioFM, an SSL audio representation model. AudioFM combines recording-level contrast between time and frequency-masked views with masked latent discrimination against full-view EMA-teacher targets. It additionally incorporates weaker supervision from neighboring temporal patches within a gaussian window. After pre-training on AudioSet, the model is evaluated through fine-tuning, frozen-encoder probing across different acoustic domains, sound event detection, cross-domain transfer, and scaling experiments. The model shows strong results for both frozen representation and fine-tuning setup.
Critical strengths
-
The paper performs a broad evaluation spanning over general audio, bioacoustics, medical acoustics, and speech emotion. This provides useful evidence of transfer beyond conventional audio classification. This is rarely done in prior audio SSL works.
-
AudioFM achieves consistent empirical improvements on a wide range of downstream tasks. AudioFM improves over AudioMosaic across several fine-tuning benchmarks and all reported detection metrics, with additional gains in cross-domain transfer. These results confirm the practical value of the learned representations.
Critical weaknesses
-
The methodological contribution is incremental. The view construction and recording-level contrast follow AudioMosaic. The distinctive additions are masked teacher-target discrimination with cross-recording contrastive learning and temporal-neighborhood supervision, but the ablations do not compare them with matched latent-regression or alternative contrastive objectives.
-
Some of the claims are not well-supported by the evaluations. For example, although cited, the HEAR evaluation is absent, leaving music coverage incomplete. The related-work discussion also omits SPEAR[1], a directly relevant unified speech/audio SSL framework evaluated on SUPERB and HEAR. Furthermore, in the introduction, the paper mentions "establishing the empirical scaling laws for audio pretraining". However, only three points per scaling axis were shown, which only shows preliminary trends rather than establish scaling laws. The small/large architectures and training budgets across configurations are also insufficiently specified.
[1]: Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations." in Proc. ICML 2026
Questions for authors
- How does MLD compare with latent regression and alternative negative sets?
- What are the additional memory and computation costs relative to AudioMosaic, and do the gains persist under matched compute?
- Can you provide HEAR results and discuss or compare with SPEAR, accounting for differences in data, model size, and teacher resources?
- Please specify all scaling configurations and training budgets, and moderate the “scaling laws” claim unless supported by a fitted and validated relationship.
9 comments
Author Response (Part 2/3)
Discussion and HEAR Comparison with SPEAR (W2 & Q3)
Thank you for pointing us to SPEAR [1], a strong recent unified SSL framework for speech and audio representations. We have added a discussion and citation in Section 2, Related Work. Regarding setup and resources, SPEAR-a Base is self-supervisedly pretrained on AudioSet (with an additional variant pretrained on a 13.1k-hour multi-corpus mix) by distilling quantized targets from an external frozen 1.2B Dasheng teacher [2] pretrained on 272k hours of audio, whereas AudioFM is pretrained from scratch on AudioSet with an online EMA self-teacher. Below, we benchmark AudioFM and AudioSet-pretrained ViT-Base baselines across 18 HEAR tasks [3] in Section 4.2, Table 4 and Appendix B.1, Figure 7 & Table 9, and further discuss our results alongside SPEAR:
- HEAR benchmark evaluation: Following the HEAR challenge and prior audio probing studies [3, 4, 5], alongside standard final-layer readout, we evaluate a multi-layer readout that combines averaged early- and late-layer representations while keeping the 768-d embedding dimension unchanged (please see details in
Appendix B.1). Domain and overall means (%) are summarized in the table below (full per-task breakdown inAppendix B.1, Table 9, bold: best within each readout block):
| Model | Teacher | Env | Speech | Music | Avg |
|---|---|---|---|---|---|
| SPEAR Paper Refs [1] | |||||
| Dasheng-Base (AS) | None | – | – | – | 70.43 |
| Dasheng 1.2B (AS) | None | – | – | – | 74.87 |
| SPEAR-a Base | Dasheng 1.2B (272kh) | 77.83 | 69.74 | 80.61 | 76.37 |
| Final-Layer Readout (768-d) | |||||
| AudioMAE | None | 74.27 | 46.47 | 76.26 | 65.89 |
| EAT | Self (EMA) | 77.94 | 56.78 | 77.36 | 70.63 |
| SSLAM | Self (EMA) | 79.00 | 57.25 | 75.97 | 70.40 |
| BAT | Self (EMA) | 79.97 | 58.59 | 80.69 | 73.16 |
| AudioMosaic | None | 85.28 | 61.31 | 71.71 | 71.26 |
| AudioFM (w/o pitch-shift) | Self (EMA) | 83.13 | 62.36 | 81.09 | 75.30 |
| AudioFM (Default) | Self (EMA) | 85.33 | 65.61 | 69.65 | 71.79 |
| Multi-Layer Readout (768-d) | |||||
| AudioMAE | None | 70.29 | 41.83 | 78.61 | 64.50 |
| EAT | Self (EMA) | 80.20 | 57.76 | 82.35 | 73.67 |
| SSLAM | Self (EMA) | 80.45 | 56.83 | 81.02 | 72.83 |
| BAT | Self (EMA) | 78.46 | 61.21 | 82.90 | 74.68 |
| AudioMosaic | None | 84.67 | 60.82 | 81.39 | 75.26 |
| AudioFM (Default) | Self (EMA) | 85.65 | 63.76 | 82.04 | 76.75 |
- Finding 1 - Strong overall & domain-wise HEAR performance: Under final-layer readout, default AudioFM already leads both environmental and speech domains among AudioSet baselines without external teachers (ranking 1st on 8 of 10 non-music tasks in the per-task breakdown in
Appendix B.1, Table 9). With multi-layer readout, default AudioFM's music score rises from 69.65% to 82.04%, achieving the highest overall average (76.75%). Compared with SPEAR-a Base (which excels on speech at 69.74%), AudioFM offers complementary strengths in environmental and music tasks, reaching 76.75% overall (higher than SPEAR-a's 76.37%) and closely approaching the 13.1k-hour SPEAR variant (77.0% inAppendix B.1, Table 9). - Finding 2 - Effect of pitch-shift augmentation on music tasks: Under final-layer readout, the lower overall average of AudioFM and AudioMosaic is driven by pitch-related music tasks (e.g., NSynth Pitch). By conducting layer-wise probing as shown in
Appendix B.1, Figure 7, we find that default AudioFM's music performance peaks in early layers and declines with depth, whereas speech and environmental performance peak in deeper layers. This may stem from pretraining augmentation: both AudioFM and AudioMosaic apply pitch-shift across positive views. Pulling pitch-shifted views together encourages invariance to small pitch changes [6], which benefits general audio recognition at the expense of fine-grained pitch cues in deeper layers. To further verify this trade-off, we additionally pretrained a controlled AudioFM (w/o pitch-shift) variant, which maintains strong music performance into deeper layers (also seeFigure 7), raising final-layer music to 81.09% and achieving the highest final-layer overall average (75.30%) as shown in the table above.
Author Response (Part 3/3)
Scaling Claim, Budgets, and Preliminary Data Scaling Exploration (W2 & Q4)
- Scaling claim & configuration budgets: In line with your feedback, we have softened our scaling claim throughout the revised manuscript to ensure a more objective characterization of our findings, and also discussed broader scaling directions in
Section 5, Limitations. All model architectures and 125k-step training budgets are specified in the compute table in Q2 above (Section 4.3;Appendix A.1, Table 6), with performance further plotted against pretraining chip-hours inAppendix B.2, Figure 8. - Preliminary larger-scale exploration: Going a step further, we no expand pretraining to a nearly 2× corpus (~2.89M segments, ~8.0k hours) by combining AudioSet with Freesound [7] and VGGSound [8] under the same AudioFM-B pipeline and step budget, with results reported in the table below (mAP in %; see data curation and preprocessing details in
Appendix B.5, Table 12):
| Pretraining Corpus | AS-20K LP | AS-20K FT | FSD50K LP | FSD50K FT |
|---|---|---|---|---|
| AudioSet (~4.7k h) | 35.0 | 42.3 | 58.6 | 71.4 |
| + Freesound & VGGSound (~8.0k h) | 35.1 | 42.5 | 61.3 | 72.9 |
| Gain | +0.1 | +0.2 | +2.7 | +1.5 |
- Gains with expanded dataset & ongoing efforts: Pretraining on the full (100%) expanded corpus brings clear FSD50K gains alongside modest improvements on AS-20K. Building on this result, we are currently running additional pretraining on 1%, 10%, and 50% subsets of this nearly 2× dataset to provide more representative data-scaling results as suggested, and hope to include them if compute and time allow.
Summary
In summary, the effectiveness of AudioFM is now validated across 12 main-paper benchmarks, SED, and HEAR tasks. Directly addressing each of your concerns, our (i) local-objective and negative-set ablations (Appendix B.4, Table 11), (ii) matched-compute and SED scaling analyses (Appendix A.1, Table 6; Appendix B.2, Figure 8), (iii) 18-task HEAR benchmark evaluation and layer-wise pitch-augmentation study (Section 4.2, Table 4; Appendix B.1, Figure 7 & Table 9), and (iv) refined scaling claims and ~8.0k-hour multi-source pretraining (Appendix B.5, Table 12) together confirm consistent gains across clip-level recognition, event detection, and transfer.
We again thank reviewer for constructive feedback, which has largely strengthen our work. We sincerely hope these clarifications and new experiments address your concerns, and we would be grateful if you would consider them in your evaluation. We are more than happy to address any further questions or concerns during the discussion period.
References
[1] Yang, X., et al. (2026). SPEAR: A unified SSL framework for learning speech and audio representations. ICML.
[2] Dinkel, H., et al. (2024). Scaling up masked audio encoder learning for general audio classification. Interspeech.
[3] Turian, J., et al. (2022). HEAR: Holistic evaluation of audio representations. PMLR.
[4] Wu, T.-Y., et al. (2022). The efficacy of self-supervised speech models for audio representations. PMLR.
[5] Rauch, L., et al. (2026). Unmute the patch tokens: Rethinking probing in multi-label audio classification. ICLR.
[6] McCallum, M. C., et al. (2024). On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations. ICASSP.
[7] Font, F., Roma, G., & Serra, X. (2013). Freesound technical demo. ACM MM.
[8] Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020). VGGSound: A large-scale audio-visual dataset. ICASSP.
Author Response (Part 1/3)
We sincerely thank the reviewer for the thorough and constructive evaluation. We are glad that the reviewer recognized the importance of our broad evaluation beyond conventional audio classification, as well as our consistent gains confirming the practical value of the learned representations.
We have carefully revised the manuscript according to your suggestion and uploaded a new version of the PDF, with all updated or newly added content highlighted in red. Please find our point-by-point responses to your questions below:
Thank you for raising these points. Holding the architecture, pipeline, and global objective fixed, we ablate local masked-patch objective designs in the table below (
Appendix B.4, Table 11):Findings:
To provide a fair comparison under the same hardware, we benchmark AudioFM against our controlled JAX reproduction of AudioMosaic (AudioMosaic-B*) using the exact same training pipeline and hyperparameters, with detailed compute specifications shown below (
Appendix A.1, Table 6) and downstream performance under matched compute budgets examined inAppendix B.2, Figure 8:Appendix B.2, Figure 8, we track frozen-encoder performance across pretraining checkpoints against TPU chip-hours on both AS-20K and AS-Strong SED tasks. Along these trajectories, AudioFM consistently outperforms AudioMosaic across matched pretraining compute budget: notably, where 79k steps of AudioFM-B approximately match the chip-hours of the full 125k-step AudioMosaic, AudioFM-B already leads on AS-20K LP (+0.3 mAP) and all SED metrics (+0.7 Frame mAP, +0.5 PSDS1, +1.6 PSDS2), and continues improving with additional compute whereas AudioMosaic starts to plateau.