acceptodds
Under review as a conference paper at ICLR 2027

AudioFM: Towards General Purpose Audio Representation Learning

Abstract

Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive objectives encourage agreement between pooled audio representations, but those objectives do not explicitly supervise time-frequency local representations. We introduce AudioFM, a self-supervised framework that combines recording-level contrast with patch-wise masked latent discrimination. Using visible acoustic context, a student predicts representations of masked patches and identifies their corresponding full-view teacher targets among candidates from other recordings. We further incorporate weaker supervision from eligible temporal neighbors to encourage local temporal consistency. Extensive evaluations show that AudioFM achieves state-of-the-art performance across diverse audio domains in linear probing, fine-tuning, and zero-shot transfer, demonstrating superior event detection, robust scene-level understanding, and strong domain generalizability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio, Self-supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_kpc5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose AudioFM, an SSL audio representation model. AudioFM combines recording-level contrast between time and frequency-masked views with masked latent discrimination against full-view EMA-teacher targets. It additionally incorporates weaker supervision from neighboring temporal patches within a gaussian window. After pre-training on AudioSet, the model is evaluated through fine-tuning, frozen-encoder probing across different acoustic domains, sound event detection, cross-domain transfer, and scaling experiments. The model shows strong results for both frozen representation and fine-tuning setup.

Critical strengths
  • The paper performs a broad evaluation spanning over general audio, bioacoustics, medical acoustics, and speech emotion. This provides useful evidence of transfer beyond conventional audio classification. This is rarely done in prior audio SSL works.

  • AudioFM achieves consistent empirical improvements on a wide range of downstream tasks. AudioFM improves over AudioMosaic across several fine-tuning benchmarks and all reported detection metrics, with additional gains in cross-domain transfer. These results confirm the practical value of the learned representations.

Critical weaknesses
  • The methodological contribution is incremental. The view construction and recording-level contrast follow AudioMosaic. The distinctive additions are masked teacher-target discrimination with cross-recording contrastive learning and temporal-neighborhood supervision, but the ablations do not compare them with matched latent-regression or alternative contrastive objectives.

  • Some of the claims are not well-supported by the evaluations. For example, although cited, the HEAR evaluation is absent, leaving music coverage incomplete. The related-work discussion also omits SPEAR[1], a directly relevant unified speech/audio SSL framework evaluated on SUPERB and HEAR. Furthermore, in the introduction, the paper mentions "establishing the empirical scaling laws for audio pretraining". However, only three points per scaling axis were shown, which only shows preliminary trends rather than establish scaling laws. The small/large architectures and training budgets across configurations are also insufficiently specified.

[1]: Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations." in Proc. ICML 2026

Questions for authors
  1. How does MLD compare with latent regression and alternative negative sets?
  2. What are the additional memory and computation costs relative to AudioMosaic, and do the gains persist under matched compute?
  3. Can you provide HEAR results and discuss or compare with SPEAR, accounting for differences in data, model size, and teacher resources?
  4. Please specify all scaling configurations and training budgets, and moderate the “scaling laws” claim unless supported by a fitted and validated relationship.
9 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

We sincerely thank the reviewer for the thorough and constructive evaluation. We are glad that the reviewer recognized the importance of our broad evaluation beyond conventional audio classification, as well as our consistent gains confirming the practical value of the learned representations.

We have carefully revised the manuscript according to your suggestion and uploaded a new version of the PDF, with all updated or newly added content highlighted in red. Please find our point-by-point responses to your questions below:


Methodological Contribution and Local Objective Ablation (W1 & Q1)

Thank you for raising these points. Holding the architecture, pipeline, and global objective fixed, we ablate local masked-patch objective designs in the table below (Appendix B.4, Table 11):

Local ObjectiveAS-20K LP (%)AS-20K FT (%)Frame mAP (%)PSDS1 (×100)PSDS2 (×100)
Global-only34.442.027.424.243.9
Latent Regression (MSE)34.341.927.924.644.3
MLD w/ Within-Recording Negatives30.441.120.917.933.5
MLD w/ Cross-Recording Negatives34.842.227.624.245.0
AudioFM: MLD + Loss Neighborhood35.042.328.524.945.9

Findings:

  • MLD vs. Latent Regression: MSE regression modestly lifts frame-level detection over global-only, but slightly lowers clip-level scores; cross-recording MLD improves clip-level scores and PSDS2, and our temporal neighborhood improves all metrics.
  • Cross- vs. Within-Recording Negatives: Same-recording negatives drop all metrics well below global-only. As general audio often has sustained events or backgrounds, same-clip patches may share acoustic content and act as false negatives, whereas cross-recording negatives reduce overlap.
  • Takeaway: Overall, standard local objectives do not readily transfer to general audio: MSE regression fails to improve clip-level recognition, and within-recording negatives severely degrade representations, whereas coupling cross-recording discrimination with our mask-aware temporal neighborhood unlocks simultaneous clip-level and frame-level sound event detection gains that hold consistently even under matched pretraining compute (please see response for Q2 below).

Pretraining Cost and Matched-Compute Comparison (Q2)

To provide a fair comparison under the same hardware, we benchmark AudioFM against our controlled JAX reproduction of AudioMosaic (AudioMosaic-B*) using the exact same training pipeline and hyperparameters, with detailed compute specifications shown below (Appendix A.1, Table 6) and downstream performance under matched compute budgets examined in Appendix B.2, Figure 8:

VariantEncoder (L/D/H)Encoder ParamsPeak Mem./ChipStep TimeChip-Hours (125k)
AudioMosaic-B*12/768/1285.7M14.1 GiB0.407 s904
AudioFM-S (ViT-S)12/384/621.6M14.5 GiB0.311 s691
AudioFM-B (ViT-B)12/768/1285.7M19.3 GiB0.649 s1,442
AudioFM-L (ViT-L)24/1024/16303.1M28.2 GiB1.864 s4,142
  • Consistent gains under matched compute: In Appendix B.2, Figure 8, we track frozen-encoder performance across pretraining checkpoints against TPU chip-hours on both AS-20K and AS-Strong SED tasks. Along these trajectories, AudioFM consistently outperforms AudioMosaic across matched pretraining compute budget: notably, where 79k steps of AudioFM-B approximately match the chip-hours of the full 125k-step AudioMosaic, AudioFM-B already leads on AS-20K LP (+0.3 mAP) and all SED metrics (+0.7 Frame mAP, +0.5 PSDS1, +1.6 PSDS2), and continues improving with additional compute whereas AudioMosaic starts to plateau.
  • No additional downstream cost: Alongside the performance gains above, although our approach adds +36.9% peak memory and +59.5% step time during pretraining against AudioMosaic (see the table above), downstream fine-tuning and inference introduce no extra cost as only the encoder is transferred.
AuthorsOpenReview ↗

Author Response (Part 2/3)

Discussion and HEAR Comparison with SPEAR (W2 & Q3)

Thank you for pointing us to SPEAR [1], a strong recent unified SSL framework for speech and audio representations. We have added a discussion and citation in Section 2, Related Work. Regarding setup and resources, SPEAR-a Base is self-supervisedly pretrained on AudioSet (with an additional variant pretrained on a 13.1k-hour multi-corpus mix) by distilling quantized targets from an external frozen 1.2B Dasheng teacher [2] pretrained on 272k hours of audio, whereas AudioFM is pretrained from scratch on AudioSet with an online EMA self-teacher. Below, we benchmark AudioFM and AudioSet-pretrained ViT-Base baselines across 18 HEAR tasks [3] in Section 4.2, Table 4 and Appendix B.1, Figure 7 & Table 9, and further discuss our results alongside SPEAR:

  • HEAR benchmark evaluation: Following the HEAR challenge and prior audio probing studies [3, 4, 5], alongside standard final-layer readout, we evaluate a multi-layer readout that combines averaged early- and late-layer representations while keeping the 768-d embedding dimension unchanged (please see details in Appendix B.1). Domain and overall means (%) are summarized in the table below (full per-task breakdown in Appendix B.1, Table 9, bold: best within each readout block):
ModelTeacherEnvSpeechMusicAvg
SPEAR Paper Refs [1]
Dasheng-Base (AS)None–––70.43
Dasheng 1.2B (AS)None–––74.87
SPEAR-a BaseDasheng 1.2B (272kh)77.8369.7480.6176.37
Final-Layer Readout (768-d)
AudioMAENone74.2746.4776.2665.89
EATSelf (EMA)77.9456.7877.3670.63
SSLAMSelf (EMA)79.0057.2575.9770.40
BATSelf (EMA)79.9758.5980.6973.16
AudioMosaicNone85.2861.3171.7171.26
AudioFM (w/o pitch-shift)Self (EMA)83.1362.3681.0975.30
AudioFM (Default)Self (EMA)85.3365.6169.6571.79
Multi-Layer Readout (768-d)
AudioMAENone70.2941.8378.6164.50
EATSelf (EMA)80.2057.7682.3573.67
SSLAMSelf (EMA)80.4556.8381.0272.83
BATSelf (EMA)78.4661.2182.9074.68
AudioMosaicNone84.6760.8281.3975.26
AudioFM (Default)Self (EMA)85.6563.7682.0476.75
  • Finding 1 - Strong overall & domain-wise HEAR performance: Under final-layer readout, default AudioFM already leads both environmental and speech domains among AudioSet baselines without external teachers (ranking 1st on 8 of 10 non-music tasks in the per-task breakdown in Appendix B.1, Table 9). With multi-layer readout, default AudioFM's music score rises from 69.65% to 82.04%, achieving the highest overall average (76.75%). Compared with SPEAR-a Base (which excels on speech at 69.74%), AudioFM offers complementary strengths in environmental and music tasks, reaching 76.75% overall (higher than SPEAR-a's 76.37%) and closely approaching the 13.1k-hour SPEAR variant (77.0% in Appendix B.1, Table 9).
  • Finding 2 - Effect of pitch-shift augmentation on music tasks: Under final-layer readout, the lower overall average of AudioFM and AudioMosaic is driven by pitch-related music tasks (e.g., NSynth Pitch). By conducting layer-wise probing as shown in Appendix B.1, Figure 7, we find that default AudioFM's music performance peaks in early layers and declines with depth, whereas speech and environmental performance peak in deeper layers. This may stem from pretraining augmentation: both AudioFM and AudioMosaic apply pitch-shift across positive views. Pulling pitch-shifted views together encourages invariance to small pitch changes [6], which benefits general audio recognition at the expense of fine-grained pitch cues in deeper layers. To further verify this trade-off, we additionally pretrained a controlled AudioFM (w/o pitch-shift) variant, which maintains strong music performance into deeper layers (also see Figure 7), raising final-layer music to 81.09% and achieving the highest final-layer overall average (75.30%) as shown in the table above.
AuthorsOpenReview ↗

Author Response (Part 3/3)

Scaling Claim, Budgets, and Preliminary Data Scaling Exploration (W2 & Q4)

  • Scaling claim & configuration budgets: In line with your feedback, we have softened our scaling claim throughout the revised manuscript to ensure a more objective characterization of our findings, and also discussed broader scaling directions in Section 5, Limitations. All model architectures and 125k-step training budgets are specified in the compute table in Q2 above (Section 4.3; Appendix A.1, Table 6), with performance further plotted against pretraining chip-hours in Appendix B.2, Figure 8.
  • Preliminary larger-scale exploration: Going a step further, we no expand pretraining to a nearly 2× corpus (~2.89M segments, ~8.0k hours) by combining AudioSet with Freesound [7] and VGGSound [8] under the same AudioFM-B pipeline and step budget, with results reported in the table below (mAP in %; see data curation and preprocessing details in Appendix B.5, Table 12):
Pretraining CorpusAS-20K LPAS-20K FTFSD50K LPFSD50K FT
AudioSet (~4.7k h)35.042.358.671.4
+ Freesound & VGGSound (~8.0k h)35.142.561.372.9
Gain+0.1+0.2+2.7+1.5
  • Gains with expanded dataset & ongoing efforts: Pretraining on the full (100%) expanded corpus brings clear FSD50K gains alongside modest improvements on AS-20K. Building on this result, we are currently running additional pretraining on 1%, 10%, and 50% subsets of this nearly 2× dataset to provide more representative data-scaling results as suggested, and hope to include them if compute and time allow.

Summary

In summary, the effectiveness of AudioFM is now validated across 12 main-paper benchmarks, SED, and HEAR tasks. Directly addressing each of your concerns, our (i) local-objective and negative-set ablations (Appendix B.4, Table 11), (ii) matched-compute and SED scaling analyses (Appendix A.1, Table 6; Appendix B.2, Figure 8), (iii) 18-task HEAR benchmark evaluation and layer-wise pitch-augmentation study (Section 4.2, Table 4; Appendix B.1, Figure 7 & Table 9), and (iv) refined scaling claims and ~8.0k-hour multi-source pretraining (Appendix B.5, Table 12) together confirm consistent gains across clip-level recognition, event detection, and transfer.

We again thank reviewer for constructive feedback, which has largely strengthen our work. We sincerely hope these clarifications and new experiments address your concerns, and we would be grateful if you would consider them in your evaluation. We are more than happy to address any further questions or concerns during the discussion period.

References

[1] Yang, X., et al. (2026). SPEAR: A unified SSL framework for learning speech and audio representations. ICML.

[2] Dinkel, H., et al. (2024). Scaling up masked audio encoder learning for general audio classification. Interspeech.

[3] Turian, J., et al. (2022). HEAR: Holistic evaluation of audio representations. PMLR.

[4] Wu, T.-Y., et al. (2022). The efficacy of self-supervised speech models for audio representations. PMLR.

[5] Rauch, L., et al. (2026). Unmute the patch tokens: Rethinking probing in multi-label audio classification. ICLR.

[6] McCallum, M. C., et al. (2024). On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations. ICASSP.

[7] Font, F., Roma, G., & Serra, X. (2013). Freesound technical demo. ACM MM.

[8] Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020). VGGSound: A large-scale audio-visual dataset. ICASSP.

Reviewer_kUdROpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AUDIOFM, a self-supervised framework for general-purpose audio representation learning. The method combines recording-level contrast and patch-wise masked latent discrimination, and innovatively introduces weak supervision from temporal neighborhoods to encourage local temporal consistency. Systematic evaluations via linear probing, fine-tuning, and zero-shot transfer are conducted across multiple domains, including general audio, bioacoustics, medical acoustics, and speech emotion. Experiments demonstrate that AUDIOFM achieves SOTA performance in event detection, scene-level understanding, and cross-domain generalization, validating the effectiveness of combining global alignment with local discriminative prediction.

Critical strengths

1.The evaluation covers not only general audio tasks, but also extends to bioacoustics, medical acoustics, and speech emotion recognition. This unified frozen-encoder linear probing protocol effectively isolates the pre-trained representations from the confounding effects of downstream fine-tuning strategies, providing highly compelling evidence for the generalization capability of audio foundation models. 2.The approach combines global contrast with local masked discrimination (MLD) and introduces weak supervision from temporal neighborhoods (temporal neighborhood extension). Ablation studies clearly demonstrate the effectiveness of a compact neighborhood (K=1) in improving frame-level event detection and classification tasks, avoiding the over-penalization of forcibly distinguishing acoustically similar regions within the same recording.

Critical weaknesses

1.Although the paper is dedicated to "General Purpose Audio Representation Learning," it notes that the AudioSet pre-training data used is approximately 1.68M clips (4,665 hours), which is even smaller than the 1.91M–2.0M clips used in some prior SSL works. As a foundation model, this relatively limited data scale may constrain the validity of extrapolating its claimed "scaling laws" to larger scales. 2.In the CirCor Murmur task, since the native audio is at 4kHz, the paper chooses to directly feed the 4kHz samples into the 16kHz frontend (a 4x rate shift) to match the pre-trained Mel filterbanks. Although this preserves the key frequency bands of heart murmurs, this non-standard approach lacks comparative ablations against standard resampling or feature extraction methods specifically optimized for low frequencies, leaving its generalizability and optimality questionable.

Questions for authors

1.Given the relatively limited current pre-training data size (1.68M clips), if the data scale were expanded by 2-3 times, would you expect the performance gain curve of AUDIOFM to maintain its current growth trend? Is there any preliminary experimental or theoretical evidence to support this? 2.In the CirCor task, directly feeding 4kHz audio into the 16kHz frontend is a clever engineering compromise. Have you attempted standard upsampling to 16kHz, or introducing multi-sample-rate support during the pre-training phase? How do the performance results of these two approaches compare to the current "4x rate shift"? 3.The introduction of a 2D Transformer predictor and dual-branch (global + local) objectives undoubtedly increases training complexity. Could you provide additional details on the specific percentage increase in peak memory usage and per-step training time for AUDIOFM compared to a purely global contrastive baseline (e.g., AudioMosaic)?

2 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We are deeply grateful for the reviewer's recommendation for clear acceptance and all the constructive suggestions. We are delighted that the reviewer recognizes our unified cross-domain evaluation and the effectiveness of our proposed framework. Below, please find our point-by-point responses to your questions, and we have also uploaded an updated manuscript according to your suggestions with all new content highlighted in red.


Scaling Claim and Preliminary Larger-Scale Data Exploration (W1 & Q1)

W1: Thank you for the suggestion. In line with your feedback, we have carefully refined our scaling claim throughout the revised manuscript to ensure a more objective characterization of our findings and also discussed broader scaling directions in Section 5, Limitations of the revised manuscript.

Q1: To further explore whether AudioFM maintains its growth trend at larger data scales, we conducted a preliminary pretraining exploration expanding our unlabeled corpus to 2.89M segments (~8.0k hours, 1.72×) by combining AudioSet with Freesound [1] and VGGSound [2] under the exact same AudioFM-B pipeline and number of pretraining steps, with downstream mAP (%) reported in the table below (full data curation and preprocessing details in Appendix B.5, Table 12):

Pretraining CorpusAS-20K LPAS-20K FTFSD50K LPFSD50K FT
AudioSet (1.68M, ~4.7k h)35.042.358.671.4
+ Freesound & VGGSound (2.89M, ~8.0k h)35.142.561.372.9
Gain+0.1+0.2+2.7+1.5

Findings:

  • Consistent improvements on the expanded corpus: Pretraining on the full (100%) expanded pool brings clear FSD50K gains (+2.7 mAP LP, +1.5 mAP FT) alongside slight improvements on AudioSet-20K under both protocols.
  • Ongoing efforts & future directions: To trace the performance gain curve across this nearly 2× corpus more systematically, we are currently pretraining additional models on 1%, 10%, and 50% subsets of this expanded pool, and hope to include them if compute and time allow. Further scaling to a 2–3× corpus, which requires careful curation of multi-dataset distributions across diverse downstream domains, remains a valuable direction for future work.

CirCor 4 kHz Frontend Ablation and Multi-Sample-Rate Pretraining (W2 & Q2)

W2: Thank you for your question. To adapt 4 kHz CirCor recordings to frozen 16 kHz encoders, we explore three frontend strategies in Appendix B.3, Figure 9 & Table 10: (i) 16 kHz resampling, which directly upsamples the audio to 16 kHz before applying the standard Mel frontend (up to 8 kHz); (ii) 16 kHz low-range Mel filterbank, which upsamples to 16 kHz and lowers the filterbank upper edge from 8 kHz to 2 kHz to match the cardiac band; and (iii) 4× native input (our setting in the main paper), which feeds the native 4 kHz samples directly into the standard frontend. In the table below (Appendix B.3, Table 10), we report patient-level Macro-F1 (%) under Bare probing across all six models:

Model16 kHz Resampling16 kHz Low-Range Mel4× Native InputBest
AudioMAE69.364.368.069.3
EAT43.538.728.443.5
SSLAM35.635.628.435.6
BAT53.656.152.256.1
AudioMosaic64.368.366.768.3
AudioFM (Ours)66.569.572.772.7

AudioFM improves steadily across the three frontends, ranking first under both 16 kHz low-range Mel filterbank and 4× native input. Moreover, since different pretrained encoders favor different frontends, we also report each model's highest score across the three options (Best column), where AudioFM still achieves the strongest overall performance (72.7%).

Q2: Beyond the 16 kHz upsampling comparison above, we follow the standard 16 kHz pretraining setup shared across baselines in this study, and leave multi-sample-rate pretraining as a very interesting and important future direction.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Training Complexity and Pretraining Cost vs. Global Contrastive Baseline (Q3)

Q3: Thank you for your question. To provide a fair comparison under the same hardware, we benchmark AudioFM against our controlled JAX reproduction of AudioMosaic (AudioMosaic-B*) using the exact same training pipeline and hyperparameters, with detailed compute specifications reported in Appendix A.1, Table 6, and further matched compute comparison in Appendix B.2, Figure 8:

  • Pretraining cost: Introducing the 2D predictor and local objective increases peak per-chip memory by +36.9% (14.1 → 19.3 GiB) and per-step training time by +59.5% (0.407 → 0.649 s) during pretraining, while downstream fine-tuning and inference introduce no additional cost as only the encoder is transferred.
  • Consistent gains under matched compute: In Appendix B.2, Figure 8, we further benchmark frozen encoders on two complementary tasks, AudioSet-20K (clip-level recognition) and AudioSet-Strong (sound event detection), and plot the pretraining trajectories of AudioFM and AudioMosaic across checkpoints against TPU chip-hours. Along these trajectories, AudioFM consistently outperforms AudioMosaic: notably, where 79k steps of AudioFM-B approximately match the chip-hours of the full 125k-step AudioMosaic, AudioFM-B leads on AudioSet-20K (+0.3 mAP) and all SED metrics (+0.7 Frame mAP, +0.5 PSDS1, +1.6 PSDS2), and continues improving with additional compute whereas AudioMosaic starts to plateau.

We again thank the reviewer for the valuable feedback! We hope our responses address your concerns and questions. Please feel free to let us know if you have any further questions.

References [1] Font, F., Roma, G., & Serra, X. (2013). Freesound technical demo. ACM MM. [2] Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020). VGGSound: A large-scale audio-visual dataset. ICASSP.

Reviewer_sNjdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of self-supervised learning to learn representations of diverse audio media that can transferred between a variety of audio domains. In contrast to previous work in this field, the paper considers diverse audio domains such as bio-medical applications, emotion recognition and sound event detection. They propose a new self-supervised learning methodology for audio that builds on and extends existing approaches in terms of the strategy for selecting targets for masked-patch prediction, and the method for discriminating targets from targets in other audio samples. Empirical evaluation on a variety of benchmark audio datasets from a diverse range of domains demonstrates that the proposed method has similar or better performance compared to recent state of the art methods.

Critical strengths

The paper provides a new approach for self-supervised learning of audio representations across a wider variety of audio application domains.

The proposed approach demonstrates comparable or better accuracy against several state of the art baselines on a diverse range of benchmark datasets.

Critical weaknesses

Some discussion of the computational costs for the proposed method would have been useful.

Questions for authors

Can you provide any assessment of the computation time required for your approach compared to the baseline methods that you reported on?

3 comments

We sincerely thank the reviewer for the invaluable evaluation, and for recognizing the value of our work in extending self-supervised audio representation learning to diverse, under-explored application domains. Below, we address your question regarding the computational cost of AudioFM relative to the baseline methods (W1 & Q1), with the updated manuscript uploaded and all additional experiments and discussion in red.


Pretraining Computational Cost and Matched-Compute Comparison (W1 & Q1)

Thank you for your question. In the revised manuscript, we have added comprehensive pretraining compute and memory records across all AudioFM model variants (Small, Base, and Large) in Appendix A.1, Table 6. Because AudioFM is implemented in JAX on Cloud TPU v4 whereas prior baselines in our study are officially implemented in PyTorch across varying NVIDIA GPU setups (with several not reporting pretraining duration), published training times and GPU-hours are not directly comparable with our approach across hardware platforms.

To go a step further and provide a fair assessment under the same hardware, we reproduced the strongest baseline (AudioMosaic-B*) in JAX and trained it using the exact same training pipeline and hyperparameters as AudioFM, with the resulting compute comparison summarized in the table below, and further examine downstream performance of both models under matched compute budgets in Appendix B.2, Figure 8.

VariantEncoder (L / D / H)Enc. / Trainable ParamsPeak Memory (per chip)Step TimeChip-Hours (125k steps)
AudioMosaic-B*12 / 768 / 1285.7M / 86.1M14.1 GiB0.407 s904
AudioFM-S (ViT-S)12 / 384 / 621.6M / 29.4M14.5 GiB0.311 s691
AudioFM-B (ViT-B)12 / 768 / 1285.7M / 94.5M19.3 GiB0.649 s1,442
AudioFM-L (ViT-L)24 / 1024 / 16303.1M / 312.4M28.2 GiB1.864 s4,142

Findings:

  • Consistent gains under matched compute: While incorporating local supervision introduces a per-step pretraining cost compared with AudioMosaic as shown above, we further examine downstream performance under equal compute budgets in Appendix B.2, Figure 8. Specifically, we benchmark frozen encoders on two complementary tasks, AudioSet-20K (clip-level recognition) and AudioSet-Strong (sound event detection), and plot the pretraining trajectories of AudioFM and AudioMosaic across checkpoints against pretraining chip-hours (alongside Small, Base, and Large scaling). Along these trajectories, AudioFM consistently outperforms AudioMosaic: notably, where 79k steps of AudioFM-B approximately match the chip-hours of the full 125k-step AudioMosaic, AudioFM-B leads on AudioSet-20K (+0.3 mAP) and all SED metrics (+0.7 Frame mAP, +0.5 PSDS1, +1.6 PSDS2), and continues improving with additional compute whereas AudioMosaic starts to plateau. Moreover, because only the encoder is transferred to downstream tasks, fine-tuning and inference introduce no additional cost.

We again thank the reviewer for the valuable feedback! We hope this addresses your question. Please feel free to let us know if you have any further concerns and we are more than happy to address them.

Reviewer_sNjdOpenReview ↗

Comment on authors' response

Thank you for providing this additional information. I believe it provides useful insights into the training cost of the approach. Overall, I consider that my original rating is appropriate.

Thank you very much for taking the time to review our manuscript and response! We truly appreciate your positive support and constructive feedback, which has strengthened both the clarity and completeness of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.