- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a dataset of marin mammal sounds (∼2.17M-hour unlabelled hydrophone pool ) and a benchmark protocol for evaluating audio encoders. The focus is on in domain self-supervised learning with controls including grid-tolerance coverage diagnostic, an annotation-geometry baseline, a published human band and a stated split rule. The evaluation includes speech and animal-audio encoders. The main findings are that bandwidth matters, but its value depends on the task. Also, timing tolerance changes apparent performance differences, and feature readout matters. No encoder seems to be consistently better across tasks.
Critical strengths
The dataset and benchmark are valuable for animal audio, especially the marine mammal community.
The empirical evaluation provides useful insights beyond ranking of encoders. Exposing bandwidth and timing as important factor in benchmark results is interesting. The representation quality depends on the downstream task and feature extraction choice. Validation-selected readout and linear probes can significantly improve accuracy. The performance depends on adaptation, training exposure and calibration.
Critical weaknesses
-
The paper addresses a relevant question for representation learning i.e. how preprocessing, temporal resolution and feature selection affect the evaluation of pretrained audio encoders? However, its broader implications are not sufficiently exposed. The experiments focus exclusively on marine-mammal tasks, and the presentation assumes familiarity with domain-specific datasets and terminology, making the main contribution harder to assess for a general ML audience. A clearer relevance to ICLR and broader significance needs to be supported by results on other datasets.
-
The focus is on animal audio encoders but including more general audio SSL models would make it more comprehensive. Not clear why BEATS or its extension, Dasheng, AudioMAE, NatureLM-audio were not included in the evaluation although some were mentioned as baselines.
-
The reported downstream experiments all concern marine-mammal audio. The paper evaluates encoders pretrained on speech and general audio, but does not test them on a non-animal benchmark. This would test how broadly the findings apply eg. by using the same controls and probe-selection across marine and non-animal tasks.
-
Generalization to unseen data is not clear, eg. without selections based on validation. Sample-rate experiments change several settings together so the claim about bandwidth is not supported by fully controlled training eg, for each bandwidth condition, same pretraining data, embedding grid etc.
The paper spills over 9 pages.
Questions for authors
Please address the issues raised in weaknesses 2-4.
Response to Reviewer MCDS
Thank you for your review. A clarification on the summary: we do not release a 2.17M-hour dataset. The 2.17M hours refer to public hydrophone archives. Our in-domain encoders were pretrained on 3.1k–3.5k-hour corpora drawn from them, and the labelled evaluation sets are listed in Appendix B (Table 7). The abstract now states this.
W1 and W3 (broader relevance and a non-animal benchmark). We added ESC-50 with the identical pipeline and validation selection, using official folds 1–3, 4 and 5 for training, validation and test. ESP-AVES2 (from the AVEX suite) reaches 94.25% accuracy, BEATs 93.75% and Perch 2.0 92.75%, while the marine animal2vec checkpoint reaches 49.25%, above only the log-mel control at 41.00%. For Whisper-tiny, the readout effect carries over outside marine audio: validation-selected readout raises it from 71.75% to 79.50%. The bandwidth control also runs on ESC-50: low-passing the input of Perch 2.0 and Bird-MAE at 4 kHz costs 4.00 and 5.25 points (exact McNemar p = 0.011 and 0.0015), while the band above 8 kHz adds 1.25 and 2.50 (p = 0.33 and 0.041), so, as on Watkins, most of the value lies below 8 kHz (Appendix Table 16). The two central variables, the embedding grid and the retained bandwidth, are properties of any audio encoder; marine audio is a domain where both vary widely and are rarely reported. Domain terms are now defined in the text: the embedding interval and the matching collar in Sec. 3.2, and time of arrival in Sec. 4.
W2 (general-audio SSL models). BEATs, AudioMAE and Dasheng are now evaluated on Watkins, HICEAS, Avacha and ESC-50, together with the bioacoustic ESP-AVES2 and Bird-MAE. No encoder leads everywhere. Dasheng is among four encoders within 4 test clips of each other on Watkins, AudioMAE leads HICEAS with AP 0.5737, Perch 2.0 heads a group of five encoders within one point on the recording-grouped call types, and ESP-AVES2 and BEATs lead ESC-50. We also evaluated NatureLM-audio's audio encoder, a BEATs model released in the AVEX suite, with the same probing protocol. It comes second on HICEAS (AP 0.5549) and on the recording-grouped call types (95.56 macro-F1, against 95.79 for Perch 2.0), and reaches 87.91% on Watkins and 84.50% on ESC-50.
W4 (generalisation without validation selection, and the bandwidth claim). All new classification comparisons select readout and regularisation on validation data only; the call types use nested recording-grouped cross-validation, and ESC-50 tests transfer outside marine audio. Generalisation to unseen recordings is tested most directly by a re-split of Watkins grouped by source tape (Appendix Table 13), which lowers every encoder from 85–93% to 36–62% accuracy. To isolate readout selection, Table 2 and the ESC-50 table also report each encoder's fixed native-final representation, with regularisation still selected on validation; the selected readout adds 12.7 points for Whisper-tiny, at most 3.2 for any other encoder, and costs the NatureLM-audio encoder 2.1.
On bandwidth, we agree that the original 8, 16 and 32 kHz rows change several factors at once, including the corpus: the three in-domain encoders were pretrained on different corpora (3,090, 3,542 and 3,371 h), not on one corpus resampled. We now present these rows as observational. We additionally evaluate bandwidth sensitivity on Watkins by low-pass filtering the 32 kHz inputs of Perch 2.0 and Bird-MAE, with encoder, input rate and embedding grid unchanged and the probe refit on the low-passed training clips (Appendix Table 16). Perch 2.0 gains monotonically, from 77.29% at a 2 kHz cutoff to 82.01% at 4 kHz and 89.97% with the full 16 kHz band; Bird-MAE is flat below 4 kHz and between 8 and 12 kHz. Low-passing at 4 kHz costs both encoders 7.7–8.0 points (exact McNemar test, p < 10⁻⁴). The band above 8 kHz adds less: 2.95 points for Perch 2.0 (p = 0.052) and 0.89 for Bird-MAE (p = 0.51). For killer-whale call types the band above 8 kHz adds 0.015 macro-F1, so above 8 kHz both tasks gain little; the task dependence shows in clicks, which lose 0.04–0.09 F1 without the band above 7.5 kHz and 0.325 without the band above 3.5 kHz. Together with the fixed click detector, this gives two controlled bandwidth results. We did not retrain the encoders per bandwidth. For the temporal grid we add a controlled ablation at a fixed encoder and detection head (Table 5, Sec. 5.3).
Length. You are right that the submitted main text ran onto a tenth page. The revision fits the 10-page limit that applies during the discussion; two field-result tables moved to the appendix.