acceptodds
Under review as a conference paper at ICLR 2027

MarineMammalBench: Sample Rate and Embedding Grid as Logged Variables in a Benchmark for Marine Mammal Self-Supervised Audio Encoders

Abstract

Self-supervised (SSL) audio encoders are now routinely reused far from the speech and general audio they were trained on. Two preprocessing choices then matter a great deal and are rarely reported: the input sample rate, and therefore the acoustic bandwidth the encoder retains, and the temporal grid on which it emits embeddings. We treat both as logged, swept experimental variables and study them in marine mammal passive acoustic monitoring, where the discriminative energy runs from killer-whale call types below 8 kHz to sperm-whale clicks above 20 kHz. We propose MarineMammalBench: a two-track protocol (in-domain SSL pretraining on a curated approximately 2.17M-hour unlabelled hydrophone pool; frozen-probe screening of public encoders under a matched protocol), a shared task ladder, and four mandatory controls, including a grid–tolerance coverage diagnostic. First-generation results give five findings. In a fixed 32 kHz sperm-whale click detector, restricting bandwidth to 3.5 kHz reduces event F1 from 0.7246 to 0.3999 at 20 ms while detector weights and the embedding grid remain unchanged. The historical 32–8 kHz F1 gap grows 2.8× when the matching collar tightens from 20 to 2.5 ms, so the temporal evaluation rule sets the size of the effect. For killer-whale call types the bandwidth gain is small (0.015 macro-F1 from the band above 8 kHz), so bandwidth has to be evaluated per task. On the official 31-class BEANS Watkins split, frozen marine animal2vec reaches 92.04% accuracy, ahead of Perch 2.0 (89.97%) and Whisper-tiny (87.91%); validation-selected readout raises Whisper accuracy from 75.22% at its final representation. On HICEAS minke-whale detection, AVES-bio instead leads the linear comparison with AP 0.5044, showing that encoder ranking depends on the task. On the prior system's presence test set, whether the model beats the human annotators depends on the metric and on the precision floor. The controls make bandwidth, temporal evaluation and uncertainty explicit parts of marine SSL benchmarking.

Primary area:
datasets and benchmarks
Keywords:
bioacoustics, marine mammals, audio representation learning, transfer learning, benchmark evaluation, acoustic event detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MCDSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a dataset of marin mammal sounds (∼2.17M-hour unlabelled hydrophone pool ) and a benchmark protocol for evaluating audio encoders. The focus is on in domain self-supervised learning with controls including grid-tolerance coverage diagnostic, an annotation-geometry baseline, a published human band and a stated split rule. The evaluation includes speech and animal-audio encoders. The main findings are that bandwidth matters, but its value depends on the task. Also, timing tolerance changes apparent performance differences, and feature readout matters. No encoder seems to be consistently better across tasks.

Critical strengths

The dataset and benchmark are valuable for animal audio, especially the marine mammal community.

The empirical evaluation provides useful insights beyond ranking of encoders. Exposing bandwidth and timing as important factor in benchmark results is interesting. The representation quality depends on the downstream task and feature extraction choice. Validation-selected readout and linear probes can significantly improve accuracy. The performance depends on adaptation, training exposure and calibration.

Critical weaknesses
  1. The paper addresses a relevant question for representation learning i.e. how preprocessing, temporal resolution and feature selection affect the evaluation of pretrained audio encoders? However, its broader implications are not sufficiently exposed. The experiments focus exclusively on marine-mammal tasks, and the presentation assumes familiarity with domain-specific datasets and terminology, making the main contribution harder to assess for a general ML audience. A clearer relevance to ICLR and broader significance needs to be supported by results on other datasets.

  2. The focus is on animal audio encoders but including more general audio SSL models would make it more comprehensive. Not clear why BEATS or its extension, Dasheng, AudioMAE, NatureLM-audio were not included in the evaluation although some were mentioned as baselines.

  3. The reported downstream experiments all concern marine-mammal audio. The paper evaluates encoders pretrained on speech and general audio, but does not test them on a non-animal benchmark. This would test how broadly the findings apply eg. by using the same controls and probe-selection across marine and non-animal tasks.

  4. Generalization to unseen data is not clear, eg. without selections based on validation. Sample-rate experiments change several settings together so the claim about bandwidth is not supported by fully controlled training eg, for each bandwidth condition, same pretraining data, embedding grid etc.

The paper spills over 9 pages.

Questions for authors

Please address the issues raised in weaknesses 2-4.

1 comment
AuthorsOpenReview ↗

Response to Reviewer MCDS

Thank you for your review. A clarification on the summary: we do not release a 2.17M-hour dataset. The 2.17M hours refer to public hydrophone archives. Our in-domain encoders were pretrained on 3.1k–3.5k-hour corpora drawn from them, and the labelled evaluation sets are listed in Appendix B (Table 7). The abstract now states this.

W1 and W3 (broader relevance and a non-animal benchmark). We added ESC-50 with the identical pipeline and validation selection, using official folds 1–3, 4 and 5 for training, validation and test. ESP-AVES2 (from the AVEX suite) reaches 94.25% accuracy, BEATs 93.75% and Perch 2.0 92.75%, while the marine animal2vec checkpoint reaches 49.25%, above only the log-mel control at 41.00%. For Whisper-tiny, the readout effect carries over outside marine audio: validation-selected readout raises it from 71.75% to 79.50%. The bandwidth control also runs on ESC-50: low-passing the input of Perch 2.0 and Bird-MAE at 4 kHz costs 4.00 and 5.25 points (exact McNemar p = 0.011 and 0.0015), while the band above 8 kHz adds 1.25 and 2.50 (p = 0.33 and 0.041), so, as on Watkins, most of the value lies below 8 kHz (Appendix Table 16). The two central variables, the embedding grid and the retained bandwidth, are properties of any audio encoder; marine audio is a domain where both vary widely and are rarely reported. Domain terms are now defined in the text: the embedding interval and the matching collar in Sec. 3.2, and time of arrival in Sec. 4.

W2 (general-audio SSL models). BEATs, AudioMAE and Dasheng are now evaluated on Watkins, HICEAS, Avacha and ESC-50, together with the bioacoustic ESP-AVES2 and Bird-MAE. No encoder leads everywhere. Dasheng is among four encoders within 4 test clips of each other on Watkins, AudioMAE leads HICEAS with AP 0.5737, Perch 2.0 heads a group of five encoders within one point on the recording-grouped call types, and ESP-AVES2 and BEATs lead ESC-50. We also evaluated NatureLM-audio's audio encoder, a BEATs model released in the AVEX suite, with the same probing protocol. It comes second on HICEAS (AP 0.5549) and on the recording-grouped call types (95.56 macro-F1, against 95.79 for Perch 2.0), and reaches 87.91% on Watkins and 84.50% on ESC-50.

W4 (generalisation without validation selection, and the bandwidth claim). All new classification comparisons select readout and regularisation on validation data only; the call types use nested recording-grouped cross-validation, and ESC-50 tests transfer outside marine audio. Generalisation to unseen recordings is tested most directly by a re-split of Watkins grouped by source tape (Appendix Table 13), which lowers every encoder from 85–93% to 36–62% accuracy. To isolate readout selection, Table 2 and the ESC-50 table also report each encoder's fixed native-final representation, with regularisation still selected on validation; the selected readout adds 12.7 points for Whisper-tiny, at most 3.2 for any other encoder, and costs the NatureLM-audio encoder 2.1.

On bandwidth, we agree that the original 8, 16 and 32 kHz rows change several factors at once, including the corpus: the three in-domain encoders were pretrained on different corpora (3,090, 3,542 and 3,371 h), not on one corpus resampled. We now present these rows as observational. We additionally evaluate bandwidth sensitivity on Watkins by low-pass filtering the 32 kHz inputs of Perch 2.0 and Bird-MAE, with encoder, input rate and embedding grid unchanged and the probe refit on the low-passed training clips (Appendix Table 16). Perch 2.0 gains monotonically, from 77.29% at a 2 kHz cutoff to 82.01% at 4 kHz and 89.97% with the full 16 kHz band; Bird-MAE is flat below 4 kHz and between 8 and 12 kHz. Low-passing at 4 kHz costs both encoders 7.7–8.0 points (exact McNemar test, p < 10⁻⁴). The band above 8 kHz adds less: 2.95 points for Perch 2.0 (p = 0.052) and 0.89 for Bird-MAE (p = 0.51). For killer-whale call types the band above 8 kHz adds 0.015 macro-F1, so above 8 kHz both tasks gain little; the task dependence shows in clicks, which lose 0.04–0.09 F1 without the band above 7.5 kHz and 0.325 without the band above 3.5 kHz. Together with the fixed click detector, this gives two controlled bandwidth results. We did not retrain the encoders per bandwidth. For the temporal grid we add a controlled ablation at a fixed encoder and detection head (Table 5, Sec. 5.3).

Length. You are right that the submitted main text ran onto a tenth page. The revision fits the 10-page limit that applies during the discussion; two field-result tables moved to the appendix.

Reviewer_UYS7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes MarineMammalBench, an evaluation protocol for marine bioacoustic encoders that makes sample rate / frequency bandwidth and embedding timing as variables in the experimental setup.

Critical strengths

The task itself is important, underwater bioacoustics deserves its own benchmark. The fixed-detector bandwidth experiment is useful.

Critical weaknesses
  • Comparison with state of the art: The paper is based largely on DORI training data + existing benchmarks datasets extracted from BEANS and DCLDE with a changes very specific and confusing protocol. Also maybe MetaPerch deserves to be compared on the current data. There are many missing models from existing state of the art (the avex suite, birdmae, birdnet).
  • The benchmark design: I find the design very confusing and somehow not common to how most of the machine learning audio benchmarks are. What is the need for two tracks? There will always be a mismatch in training data as new datasets appear. People will not respect this protocol More, the design requires grouped splits, validation-based selection and mandatory controls, but several central results do not meet these conditions. Table 3 retains checkpoint and layer selection on reported scores. Table 4 uses thresholds selected on held-out signal files. Appendix D acknowledges recording-group errors in the filtering task. These are different limitations and should not be treated as one general caveat. Test-selected thresholds affect the reported detector performance directly; imperfect grouping affects independence; historical selection affects model comparisons. A benchmark paper should provide at least one complete evaluation path where all its requirements are enforced. The way the design is presented does not help either with "runs" and "specified" it almost look like a design document for an LLM agent that runs the experiment, reports results while another agents builds the final table based on these flags "“Run” marks rungs with results in Sec. 5". This is not needed for the paper. I am not convinced that the task ladder is the best, I think basic detection (presence?) should be the easiest, followed by species, then call types (clicks should be included here). Also these should be somehow hierarchical rather than different tasks.
Questions for authors

-The GMWM experiment is evidence that an isolated-call classifier can fail on continuous recordings. It does not by itself show that the encoder is inferior: the head was trained on clips and evaluated under a different windowing and background distribution.

  • Recall at fixed precision is declared primary, but Table 4 omits it. Please report precision–recall curves, validation-selected operating points, and achieved test precision alongside recall. -The noise-only evaluation lasts approximately 0.72 hours. Observing zero false alarms over this interval is not evidence of negligible deployment false-alarm rates; the appendix appropriately notes an upper bound of roughly four events per hour.
  • The human comparison should also use matched operating constraints where possible. A model evaluated at precision 0.8 cannot be interpreted as matching an annotator operating at precision 0.99 simply because their recall is similar.
  • ICLR has unlimited pages for references so I don't understand why some papers are cited without their authors (Earth Species Project, DCLDE, Google) or et al authors, you can have a maximum number of authors and add "and others"
  • Confusing phrasing: (sometimes it feels like caveman was used with some LLM to compress the paper) "Frozen-feature probing follows the HEAR convention (Turian et al., 2022)." and the related work first paragraph. What are "rungs", why not "tiers" ? "Every experiment logs four quantities" -> why not "in each experiment we test different variables/factors/etc". I have been in the field for many years and this is the first time I come across this unconventional expressions in a ML paper.
  • The introduction motivates sperm-whale clicks above 20 kHz, but the fixed-detector experiment uses 32 kHz input and therefore retains at most 16 kHz. It demonstrates dependence on frequency content within that range. It does not test the value of ultrasonic content above 20 kHz.
  • Section 4 distinguishes the approximately 2.17 million hours available in archives from the roughly 3,000–3,500 hours used in the documented pretraining runs. That distinction should be equally clear in the abstract.
  • Can you provide a cleaned Watkins evaluation alongside the official split, removing cross-split duplicates and grouping examples by their source recording? Please document whether any evaluated checkpoint used the historical training pool identified in Appendix D.
  • Can you repair the canonical recording groups and rerun call-type/filtering comparisons under validation-only selection? I would prefer a smaller consistent comparison to the larger historical table.
2 comments
AuthorsOpenReview ↗

Response to Reviewer UYS7 (part 1 of 2)

Thank you for the detailed review. This comment addresses W1–W4; the next addresses Q1–Q9.

W1 (comparison with the state of the art). From DORI we use only its ONC presence test set and, for the label-provenance audit, its ecotype-label metadata; the other sets come from BEANS, DCLDE (annotation metadata only), the Avacha archive and the Dominica click set, and Appendix B (Table 7) now lists all of them with species, region and annotation counts. We added ESP-AVES2 from the AVEX suite (esp-aves2-sl-beats-all), Bird-MAE-Base, BirdNET v2.4 and the NatureLM-audio encoder, together with the general-audio BEATs, AudioMAE and Dasheng, all through the same frozen-feature probing protocol with layer and regularisation selected on validation data. On Watkins with 3 s input, ESP-AVES2 reaches 93.22% accuracy, Bird-MAE 92.63%, Dasheng 92.33%, animal2vec 92.04%, Perch 2.0 89.97% and BirdNET 88.20%. On HICEAS, AudioMAE leads with AP 0.5737, ahead of the NatureLM-audio encoder (0.5549), BEATs (0.5079), ESP-AVES2 (0.5065) and AVES (0.5044), while BirdNET reaches 0.3907. On the recording-grouped Avacha call types, Perch 2.0 (95.79 macro-F1), the NatureLM-audio encoder (95.56), Bird-MAE (95.29), ESP-AVES2 (94.89) and AudioMAE (94.88) lie within one point. The ESP-AVES2 model card lists the Watkins database (27 h) among its supervised training data, and the NatureLM-audio card lists Watkins in its training data, so both Watkins rows are flagged for exposure. MetaPerch is not included because its checkpoint is not yet public: the release page in the Perch repository states that access instructions will follow. Perch 2.0 is included in our main encoder comparisons, and the related work now cites MetaPerch. We now cite the AVEX paper with its authors (Miron et al., ICLR 2026); the earlier citation gave only the organisation name.

W2 (the two tracks, and central results that violate the protocol). We agree on both counts. The revision has a single evaluation protocol: public and in-domain encoders share Tables 2 and 3, Table 2 declares known exposure to the Watkins sources, and Tables 2 and 3 state each encoder's pretraining domain. New corpora will keep overlapping with benchmark data, so the protocol records exposure instead of forbidding it; ESP-AVES2 on Watkins and our own checkpoint show why this matters.

For a complete path we followed your suggestion of a smaller consistent comparison. Table 3 is replaced by a recording-grouped, validation-only comparison: outer 5-fold cross-validation grouped by recording (1,620 groups), with layer and regularisation strength chosen by inner grouped folds. Perch 2.0 has the highest pooled score on call type, call type with noise and filtering (95.79, 96.51 and 99.40 macro-F1), but five encoders lie within one point on call type with a fold SD of 2–4 points, so we read them as a top group. animal2vec trails every public SSL encoder (74.26). The historical table moves to the appendix.

We now state each limitation separately for each result. The official Watkins split uses validation-only selection but cannot be recording-grouped, because 333 of the 339 test clips share a tape prefix with training or validation clips, so we add a tape-grouped re-split alongside it (Q8); ESP-AVES2, the NatureLM-audio encoder and animal2vec carry exposure flags there. Table 3 is validation-only and nested, but its recording key is parsed from file names and is approximate, and its animal2vec row uses the historically selected checkpoint flagged in Table 2. In the click-detection table (Table 4), the F1 column and the recall at fixed precision both use thresholds scanned on the test files, and we label them as such. Its seven heads cannot be recalibrated on training files, as dense predictions were not kept. We therefore took the bandwidth-experiment head (Appendix G; same frozen 32 kHz encoder) and chose its thresholds on its 31 training files (there is no separate validation set). For this head, the training-selected threshold of 0.85 gives the same rounded test F1 of 0.725 at 20 ms as the previously used test-selected threshold of 0.84 (Sec. 5.3). The presence comparison uses nested cross-validation, but its folds are stratified rather than grouped by site.

W3 (terminology). We agree. "Run/specified", the Status column and "rungs" are removed, the former T6, T7 and T9 become deployment tasks D1–D3, described in one appendix paragraph, the former T8 retraining experiment moves under T5, and "every experiment logs four quantities" now reads "for every experiment we report four preprocessing variables".

W4 (the task ladder). Table 1 now runs from presence through species and call type to click detection. Clicks stay a separate event-detection task, because their labels are times of arrival rather than clip classes. A hierarchical formulation is a good direction; Sec. 3.3 mentions it without claiming results.

AuthorsOpenReview ↗

Response to Reviewer UYS7 (part 2 of 2)

(Part 2 of 2.) Q1 (GMWM). We agree that the experiment does not show the encoder to be inferior. Sec. 5.6 now states that the experiment cannot separate the head and the windowing from the embedding, and makes no claim about the encoder.

Q2 (recall at fixed precision, PR curves, operating points and false alarms). Table 4 now reports recall at precision of at least 0.5 and 0.8 at a 20 ms collar for all seven detectors, and at 2.5 ms for the five with a stored strict-collar scan, with PR curves in the appendix. These thresholds were scanned on test, so the values are descriptive. For the bandwidth-experiment head (Part 1, W2), a threshold chosen on training files for precision 0.8 gives test precision 0.757 and recall 0.572, missing the target. Five of the seven detectors raise no false alarm in 0.721 h of noise recorded in one night, a one-sided 95% upper bound of 4.2 per hour; this does not establish a deployment false-alarm rate.

Q3 (the human comparison at matched operating constraints). We now compare at each annotator's own precision. At annotator 3's precision of 0.51, Whisper-tiny recalls 0.969 against the annotator's 0.792; at annotator 2's 0.72 it recalls 0.698 against 0.805; at annotator 1's 0.99 it recalls 0.459 against 0.686. The model beats only the lowest-precision annotator, and "close to annotator 1" is removed. These comparisons are descriptive R@P values from out-of-fold PR curves, not validated operating points. Layer and pooling are now selected by nested cross-validation, which lowers WavLM from 0.790 to 0.763 AUROC. All presence values are recomputed with polyphase resampling; the submitted values are kept in Appendix Table 20.

Q4 (citations). References now list their authors, up to ten followed by "and others": the Earth Species Project entry becomes Miron et al. (AVEX, ICLR 2026), and every entry shortened to "et al." is expanded. The Google Research model card names no authors, so we cite it alongside Harrell (2024). The DCLDE entry now cites the dataset paper (Palmer et al., 2025). The propagated ecotype labels of Sec. 5.5 come from the DORI release, not DCLDE as submitted; the text is corrected.

Q5 (phrasing). The first paragraph of the related work and Sec. 3.2 are rewritten in plain language. The HEAR sentence now states the procedure, and "rungs" and the "four quantities" sentence are replaced (Part 1, W3).

Q6 (clicks above 20 kHz). We agree. The abstract now describes broadband clicks rather than clicks above 20 kHz, and Sec. 6 states that the 32 kHz detector retains at most 16 kHz, so content above 16 kHz is not tested.

Q7 (hours in the abstract). The abstract now states that the in-domain encoders are pretrained on 3.1k–3.5k-hour corpora drawn from a ~2.17M-hour unlabelled pool; Sec. 4 gives the three corpora (3,090, 3,542 and 3,371 h at 8, 16 and 32 kHz).

Q8 (a cleaned Watkins evaluation and the historical training pool). Removing nine conflicting-label test duplicates raises every row by 1.7–2.5 points and leaves the ranking of the pretrained encoders unchanged (Appendix Table 12). We also add a tape-grouped re-split (Appendix Table 13): 1,579 clips on 146 tapes, 5-fold cross-validation grouped by tape, with layer and regularisation chosen by inner grouped folds. Every encoder falls from 85–93% to 36–62% accuracy. Our animal2vec checkpoint, level with the leaders on the official split, falls to 36.3%, below the log-mel control at 37.6%, while Perch 2.0, the best encoder without declared Watkins exposure, reaches 56.8% (ESP-AVES2, trained with Watkins labels, 62.1%). Official-split accuracy therefore overstates transfer to unseen tapes.

For our checkpoint (BEST129k in Table 2), a check of every file path in its pretraining manifest (2,550,424 five-second segments) finds no Watkins, historical-pool or BEANS files; renamed or re-encoded copies cannot be ruled out. Separately, the training logs show that it was chosen from 104 checkpoints by a composite score that included Watkins test macro-F1, so its Watkins scores are test-informed. Table 2 marks it, and we make no ranking claim for it on the official split.

Only this checkpoint has had its manifest checked. The 16 and 8 kHz click encoders are different checkpoints, but they were also selected with a recorded Watkins macro-F1 (0.916 and 0.914), and the revision adds no evaluation of them beyond click detection (Appendix B). No such record exists for the 32 kHz click encoder or the AVES-HuBERT checkpoints.

Q9 (repaired recording groups and a validation-only rerun). The validation-only rerun is the new Table 3 (Part 1, W2). The canonical recording map is not ready: groups still come from file-name date–time tokens. The rerun applies the same parsed date–time grouping rule across call and noise clips; the key remains approximate, and the Table 3 caption says so.

Reviewer_m6vKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Using SSL audio encoders has become common practice in many bioacosutic workflows for data annotation and processing. However, for proper benchmarking and comparison of these models, it’s important to consider input sample rate, and the temporal grid of embedding outputs. This paper introduces a benchmark and training protocol that sweeps over the sample rate and resampling strategy and the temporal grid of the embeddings. The authors benchmark an SSL baseline ptrained on >2M hours of unlabelled audio and existing encoder baselines. Preliminary results show that the retained bandwidth effects click detection, the effect frequency of embeddings is variable, and call-type classification does not depend so much on bandwidth gains. They also show that the best encoder depends on the task, choosing how to extract features from each encoder using validation data can substantially improve performance.

Critical strengths
  • The authors present a strong motivation, and their framing is well-grounded in real world challenges in bioacoustics (variable sample rates, and the temporal grid of embeddings) with underwater PAM data. All metrics are defined precisely.
  • The positioning of this benchmark relative to existing benchmarks for generali bioacoustics and marine mammals is clear. The gap in the literature is well motivated.
  • The proposed benchmark task ladder in Table 1 is clear and intuitive way to discuss the varying levels of challenges with bioacoustic data, and the deployment-oriented evaluation around generatlization and cost is very practical.
  • The experiments reported in each section are very intuitive and well-motivated by the domain.
  • The authors provide very many interesting analyses. The inter-annotator agreement analysis in Appendix I and Section 5.5 are thoughtful.
Critical weaknesses
  • Include citations to support the claim that the temporal grid of the embeddings should be logged in the first place. This paper [1] seems like the most relevant.
  • Minor: Contribution (iv) is standard practice, and is arguable not a strong enough standalone contribution. Figure 1 is a bit misleading—the unlabelled PAM data and pretrained encoders likely shouldn’t be presented at the same level. This is a minor comment. Line ~155: Do you mean “T1-T3” instead of “T1-T4”? Tables 4-6 aren’t directly referenced in the main text.
  • The authors set up a intuitive benchmark task ladder in Table 1. In the presentation of results, it would make the narrative much clearer if the authors divided their subsections by each of these clearly defined tasks. Especially because there are many threads of exploration and many different variables to probe, it would make the resulting analysis more structured.
  • It is a step forward to log variables like sample rate while reporting results, especially in the space of marine bioacoustics. With so many variables changing across the models, it becomes hard to isolate the role of a single factor. For example, it seems Table 4 is showing results across input sampling rates, embedding intervals, and detection heads/archictures. It would help to have a controlled temporal-grid ablation with a fixed encoder, sample rate detection head, and vary only the temporal resolution.
  • The language around the presentation of the results is dense and difficult ot understand. There are also some numerical results that are not highlighted in the primary result tables, and therefore difficult to understand in relation to the other results. See L242-245 and L246-249 for results on the performance of a CNN trained on Watkins recordings without SSL pretraining, or the variant of selecting feature readouts for the Whisper model.

[1] Vengrovski, G., & Gardner, T. J. (2026). SongMAE: A bioacoustic encoder for birdsong. bioRxiv, 2026-08.

Questions for authors
  • What is the per-dataset species coverage and geographic range? It would be helpful to describe this where the information is available. How many annotations per dataset?
  • Does temporal resolution always affect performance? Does this depend on the task at hand, is it only relevant for detection?
  • Is it really meaningful to output 0 false alarms/hr on only 0.721 hours of noise (Table 4)?
  • What is actually the role of increasing exposure? Table 6 makes it seem mixed (see the frozen linear probe results)
3 comments
AuthorsOpenReview ↗

Response to Reviewer m6vK

Thank you for your positive and detailed review. One clarification on the summary: the in-domain encoders are pretrained on 3.1k–3.5k-hour corpora drawn from the ~2.17M-hour pool, not on the full pool. The abstract now says so.

W1 (a citation for logging the grid). SongMAE (Vengrovski & Gardner, 2026) is now cited in Sec. 3.2, where we define the embedding interval and note that it can be too coarse to resolve short units such as syllables.

W2 (contribution (iv), Figure 1, L155, Tables 4–6). Contribution (iv) is removed, as it describes standard practice, and Figure 1 is redrawn so that unlabelled PAM data and pretrained encoders are no longer at the same level. The T1–T4 slip at L155 is fixed, and Tables 4–6 are now referenced in the text.

W3 (results by task). Sec. 5 is reorganised by task: each subsection names the tasks it covers (T1–T5) and opens with its main finding.

W4 (a controlled grid ablation). As you suggested, we fixed the 32 kHz animal2vec encoder, the input rate and the trained BiLSTM detection head, and thinned only the post-encoder feature grid, from the native 1.25 ms to 2.5 ms and 5 ms (Table 5, Sec. 5.3). With the native threshold shared across grids, event F1 at a 20 ms collar falls from 0.725 to 0.342 and 0.007; the paired file-bootstrap differences are −0.38 [−0.59, −0.12] and −0.72 [−0.80, −0.62]. Most of this collapse is a shift in score calibration. Recalibrating the threshold per grid on training files, a secondary analysis, recovers much of it: F1 at a 20 ms collar becomes 0.581 on the 2.5 ms grid and 0.569 on the 5 ms grid. Even so, the 5 ms grid loses 0.12–0.16 F1 at a 20 ms collar, depending on the decoder's merge gap (0.569 to 0.604 against 0.725), and 0.29 at a 2.5 ms collar (0.662 to 0.374), even though every event time still lies within 2.5 ms of a point on the 5 ms grid. Varying the merge gap from 0 to 10 ms leaves the shared-threshold collapse unchanged (F1 0.007). The head is not retrained on the coarse grids, and thinning also changes the time step its recurrence sees, so the result measures the sensitivity of one detector rather than timing quantisation alone or an optimal grid.

W5 (dense language; CNN and readout results outside the tables). Table 2 now includes the supervised CNN (82.99 ± 2.07% at 3 s and 80.24 ± 2.99% at 5 s) and a final-layer column, so the Whisper readout effect (75.22% at the final layer against 87.91% with validation-selected readout) is visible in the table. The results text is rewritten around the per-task structure.

Q1 (species, geography and annotations). Appendix Table 7 now gives these per dataset, computed from the released manifests. Watkins covers 31 species (26 cetaceans and 5 pinnipeds) in 1,695 clips from a global archive. HICEAS covers the minke whale around the Hawaiian Islands in 2017, with 796 annotated call intervals. DORI's ONC set pools four species into presence labels, recorded in British Columbia in 2023; it has 385 five-minute files, 159 of them positive. Avacha has 22,452 clips (10,224 killer-whale calls of 11 types and 12,228 ambient-noise clips) from 1,620 recording groups in Kamchatka. Dominica has 28,249 annotated sperm-whale clicks. From the Northeast Pacific, the DCLDE killer-whale set has 50,927 ecotype annotations and DORI's release 15,407 ecotype labels. Compiling the table also showed that 2 of the 385 ONC files are in our 16 kHz pretraining manifest. No in-domain checkpoint is evaluated on that set, and the table declares the overlap.

Q2 (does temporal resolution always matter?). In our click-detection experiments it does; after per-grid recalibration the 5 ms grid loses more at the strict collar (0.29 against 0.16 F1). For clip classification the embeddings are pooled over time, and we did not test the grid separately, so we make no claim there.

Q3 (zero false alarms in 0.721 h). We agree that zero observed events are insufficient to estimate a reliable deployment rate. For the five detectors with no false alarm, Table 4 now reports a one-sided 95% upper bound of 4.2 events per hour.

Q4 (the role of exposure; submitted Table 6, now Appendix Table 17). The effect is an interaction of +7.10 F1 points (95% CI [3.05, 11.47]): more field presentations raise full fine-tuning (36.27 to 41.18) and leave frozen heads flat or lower (40.27 to 38.08; 39.74 for the best frozen head, selected on inner folds). At q = 60, the highest exposure in the table, full fine-tuning exceeds that head by only 1.43 points (CI [−1.92, 5.09]): exposure reverses the order of the frozen linear head and full fine-tuning, but no winner is established. The revision calls q field-training exposure, to keep it apart from pretraining exposure.

Thank you for your thoughtful and detailed response. I have reviewed the updated submission, and I believe that the edits meaningfully improved it. The organization of results by tasks made the sections much easier to follow, and the clarifications on the datasets helped as well. However, the writing of the results remains incredibly dense and difficult to parse. Additionally, the reporting of some of the results is still ambiguous. The temporal grid ablation still does not isolate the effect of temporal resolution alone. I stand by my original score and believe it is generous. This benchmark improves an important practical grounding for researchers in this space working with marine mammal bioacoustic data with ML methods. The authors should work to improve the presentation and clarity of their experiments in a revision.

AuthorsOpenReview ↗

Response to Reviewer m6vK

Thank you for reading the revision, and for your kind words about the task-based organisation and the dataset table. We agree on the grid ablation: the head is not retrained and thinning also changes the time step it sees, so it measures one detector's sensitivity rather than temporal resolution alone, as Sec. 5.3 now states. We take the point on density and will keep tightening the presentation of the results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for their detailed and constructive feedback. The revised paper strengthens the central empirical claim of MarineMammalBench: acoustic bandwidth, embedding-grid resolution and event-matching tolerance materially affect evaluation and should be controlled and reported, rather than treated as invisible implementation defaults. On a fixed 32 kHz sperm-whale detector, low-passing the input to 3.5 kHz reduces event F1 from 0.725 to 0.400 at a 20 ms collar, and a new ablation thins the post-encoder embedding grid from 1.25 to 5 ms, without retraining the head, which lowers F1 from 0.725 to 0.569 even after threshold recalibration. The separate 32–8 kHz comparison shows that tightening the collar from 20 to 2.5 ms enlarges the F1 gap 2.8×. The paper states the scope and confounds of each experiment explicitly.

The encoder comparison is now substantially broader. Table 2 (official BEANS tasks) adds general-audio BEATs, AudioMAE and Dasheng, plus ESP-AVES2, Bird-MAE, BirdNET and the audio encoder of NatureLM-audio. All but BirdNET also enter the recording-grouped killer-whale call-type comparison, and we add ESC-50 as a non-marine control. Table 3 replaces the historical screen with a nested comparison of ten encoders and a log-mel control on 22,452 Avacha clips from 1,620 approximate recording groups, using validation-only layer and regularisation selection. The previous screen is retained in the appendix for transparency.

We explicitly distinguish validation-selected probes, training-calibrated operating points and historical test-selected thresholds, with selection provenance disclosed in the relevant tables and appendix. Table 2 marks known Watkins training exposure and historical test-informed checkpoint selection; Tables 2 and 3 also report pretraining domains. Further experiments hold encoders fixed while low-pass filtering Watkins and ESC-50 inputs, complementing the click-detector bandwidth intervention. The four mandatory controls are retained (grid–tolerance coverage, annotation geometry, human band, split rule): on DCLDE ecotypes, a geometry-only predictor reaches 0.398 top-1 accuracy versus a 0.215 majority baseline. Table 4 adds recall at fixed precision and a Poisson false-alarm-rate bound; PR curves are provided in the appendix, and Table 5 adds the fixed-head temporal-grid experiment. The revision expands the dataset table, reorganises results by task, corrects citations and clarifies the task ladder; two field tables were moved to the appendix to meet the 10-page limit.

These additions substantively change what the benchmark can establish. On official BEANS Watkins, four encoders lie within four test clips (92.0–93.2% accuracy), but a tape-grouped re-split lowers scores to 36–62%, exposing a substantial source-generalisation gap. AudioMAE leads HICEAS in linear AP; five encoders led by Perch 2.0 lie within one point on recording-grouped killer-whale call types; ESP-AVES2 and BEATs lead on ESC-50. Whisper-tiny's readout gain on Watkins recurs on ESC-50 (71.75% to 79.50%). The matched-precision human comparison also replaces an overly broad performance claim with a narrower, descriptive result. Together, the controlled interventions, cross-task ranking changes and explicit exposure/split diagnostics show why we evaluate marine SSL transfer with this protocol rather than a single leaderboard.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.