- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
I apologize in advance if I deviate somewhat from the ICLR reviewing template. It's a bit too restrictive at times for what I want to say.
Paper Summary & Contributions
This paper relates to the creation of a new benchmark for bioacoustics and its use to train a new audio-language model for bioacoustics. Essentially, the paper falls under the category of "data-driven" deep learning. The authors introduce a large new benchmark (derived by existing benchmarks/datasets), recast it as an open-ended language generation benchmark, and fine-tune existing audio-language models to perform the tasks in the benchmark.
Validation
The validation in this paper is self-referrential. The authors construct a benchmark and train models on that benchmark. The lack of an external validation is a weakness of this paper (more below)
Decision & Overall Feedback
I am strongly leaning towards rejecting this paper. The reasons will be made more clear below. I say "leaning towards" and not "will reject" because there are some questions to the authors. I am also signalling to the AC that they might overrule my decision, if the other reviewers are strongly in favor of accepting it.
While there are enough minor weakneses that I will mention below, my mean reasons for rejection are two:
-
I disagree with enough choices made in the constructon of BEANS-next that I don't think it should be published like this
-
I consider ICLR to be the wrong venue for this paper. I consider this for two reasons: a) It is highly unlikely that this will be reviewed by biology experts, given they are not part of the reviewer pool. I don't think we should be releasing major benchmark papers without domain expert reviews. b) The page limit is simply too low for what the authors want to present. As I say below, this forced the authors to pack a lot of vital parts in the appendix (which technically I am not obligated to read, but I had to..)
I get that the authors want to maximize their exposure to the ML community by submitting to ICLR, and I think more exposure is good for the bioacoustic community (and most likely for the careers of the authors, so I sympathize with that), but they are missing on the most crucial community -- the biology/ecology experts that will be benefitting from these tools.
Reviewer self-evaluation
There's a multiple-choice box I have to click but I like to be more verbose on this so the authors have a chance to judge my expertise. I have been working on audio machine learning for the past 8 years. For about half of those years, I have worked on computational bioacoustics and ecoacoustics as a senior researcher at a public institution. I have thus had daily experience with the algorithms and data used here. Recently, I have also been working on AudioLLMs (like everyone else), but not directly applying them to computational bioacoustics.
I would thus consider myself very suited to evaluate this paper.
Critical strengths
Sensible goal
This is an interesting work that is a natural extension of ongoing threads in bioacoustic and general audio research. What I mean by that is that everyone is re-casting all previous benchmarks to fit the LLM paradigm and then training an audio-language model to perform the task. This work needs to be done for bioacoustics and the authors make the effort to do it.
This is a very worthy goal and weighs in favor of publishing this paper.
Sensible methodology (in broad strokes, not details)
This is how most bioacoustic researchers would go about creating the benchmark and models. The authors pooled the data from the right sources, they combined multiple tasks (I guess one could argue about the selection of tasks though), and trained standard models. They also release their code, which makes it a good entry point for future researchers.
Critical weaknesses
Presentation
I already phrased my concerns on presentation, but they are so strong I need to re-iterate them. The authors had to pack so much on 9 pages that a lot of the important stuff was relegated to the appendix:
-
The exact tasks -- what is a benchmark paper about if not about showing and discussing the tasks? Yes, I know, no way to do this on 9 ICLR pages, which is why I think the choice of venue is wrong...
-
The included/excluded species -- not that they would help "me", but domain experts would probably like to know. Again, that's why I recommend a different venue, so that these experts can judge whether the separation into training/held-out species is sensible
-
The reader is constantly referred to previous publications to get important information, such as where and how all benchmark datasets were recorded (immensely important in bioacoustics!), how the audio encoder was trained.
These (and other) issues made me see this paper as a hasty attempt to hit a deadline. That's understandable, we all do it, but I think it's really unsuited for a paper that aims to establish a standard benchmark in the field.
The most important thing is that the page limitation leaves no space whatsoever for any detailed discussion of a number of surprising results (see below). For instance, the authors repord CIDEr scores of 0.0 (!!) on captioning (Table 5) but don't even make a passing reference to that fact in their text.
Benchmark construction & Baselines
The most issue I could identify is the construction of the benchmark (and the lack of baselines).
The whole premise of the paper is that creating an audio-language ("captioning") benchmark covering multiple tasks is an important milestone for bioacoustics (Indeed, it is).
However, when going through the table of tasks, it is the first time that one sees any description of the actual tasks. I must say, I was somewhat disappointed at that point:
-
Most of the tasks rely on noisy Xeno-Canto/iNaturalist annotations; this is a standard approach in bioacoustics (and, generally, ML literature), but the authors should nevertheless have discussed the ambiguity of the annotations. Better yet, they should have done the hard work of verifying the reliability for a subset of them
-
Most of the tasks are cast as simple regression / multi-label classification tasks "dressed up" as linguistic tasks by turning them into a binary or multiple-choice question, using an LLM to do that work. I know this is standard practice in the AI literature at the moment, but I fail to see the point. I especially fail to see the point of using an AudioLLM to perform those tasks, when I simple audio encoder retrofitted with task-specific output heads could just as well do the job. This is a baseline that I would like to see, esp. for those tasks where baselines already exist in literature. The only "true" textual task I could identify was captioning (e.g. Table 5). But results there seem to be abysmally bad.
-
Not a benchmark issue per se, but the audio encoder is (presumably) BEATs, which has been pretrained on AudioSet-2M. AudioSet has a lot of animal sounds. Are the authors certain that held-out species do not appear during pre-training?
Weird benchmark numbers?
-
Are all scene results in Table 5 truly 0.0 except for the in-domain?
-
Mean f0 MAE is much lower for held-out species than for seen species. This makes absolutely no sense and is not explained in the text. Moreover, looking at previous work on the data used to create that F0 benchmark, the original authors used a different evaluation scheme, computing accuracy to the nearest semitone, and report a top accuracy of ca. 70%. That should map to match lower MAEs than reported here -- but of course it's impossible to compare since we don't even know if the same splits are used.
Questions for authors
-
Is there any explanation for the lower F0 MAE scores on held-out species vs seen?
-
Was NatureLM-audio T1 the only model that was in anyway finetuned for this benchmark?
-
Are there relevant, non-LLM baselines (e.g. a single audio encoder with output heads). What is the influence of the LLM decoder on performance?
3 comments
W3 (self-referential validation; noisy annotations). We agree that a benchmark built with the same pipeline as its training data needs checks from outside that pipeline. We point out that no model is trained on the benchmark, and we controlled for the ways in which the shared pipeline could still inflate the scores.
Shared text. Benchmark and training questions come from the same templates and LLM, so a model could in principle answer from text priors. We tested this with a classifier that sees only the question and the options, and found nine multiple-choice tasks where it was well above chance (96.5% on acoustic description matching, the task on which T1 scored 97.9%). We rebuilt eight of them (the behavior MCQ and the seven Tier 3 MCQs) with wrong options drawn from the correct answers of other items, so that the text no longer points to the answer. We removed description matching: a listening test showed that the task is ambiguous, because the field-note descriptions are incomplete and more than one option can fit a recording. On the rebuilt Tier 3 species tasks T3 scores 47–73% with the recording and 29–34% without it (chance 30–33%; App. G.3). We also re-evaluated with paraphrased prompts written by a different LLM, and observe minimal performance differences for the fine-tuned models (at most 6 points; App. G.8).
Shared recordings. We compared recording identifiers with all training files. About 3% of the taxon presence test items (163 of 4,940) are also in Tier 2 training files, with an effect of at most 0.2 points on T2. In Tier 3, 1,530 of the 1,903 items with a Xeno-canto identifier use recordings that are in Tier 2 training files; the fine-tuning of T3 did not use these files, and the Tier 3 questions rely on the event annotations of the source datasets, which are not in ROOTS. We point out that more than half of the tasks, including all of Tier 3, come from datasets that are not in ROOTS. Further, the sources that training and test data share are Xeno-canto and iNaturalist, which are crowd-sourced and contain a lot of variation in recording conditions, style, and text.
Checks outside our pipeline. The labels of f0, of the crow and otter call types and of DCASE do not come from our pipeline. Every model is now also evaluated on BirdSet and BEANS-Zero, and we added baselines that were trained by others (the pitch trackers pYIN and CREPE, and the audio-text models BioLingual and LAION-CLAP).
Annotation noise. We agree that the Xeno-canto and iNaturalist metadata is noisy: the focal label can miss background species, call type and behavior are free text from the recordist, and iNaturalist labels are community identifications. Still, these archives are the largest source of supervision that exists for bioacoustics, and the question is how much of the noise reaches the generated data. The generating LLM is instructed to decline when the recordist's notes do not support an item, and it rejects 27% of the generated items. To measure what remains, one of the authors listened to 125 items that passed the filter, next to the recordist's notes, and judged 8.0% unfavorable as training labels (App. G.4). We also argue that noise in the training labels works against the fine-tunes, not for them, on the tasks whose test labels come from experts. The paper now discusses these ambiguities. A verification of a subset of the benchmark labels by domain experts is named as a limitation, and the per-recording provenance is released so that it can be done.
W4 and Q3 (classification dressed as language; encoder with heads; influence of the decoder). We agree that taxon presence, call-type presence and f0 are classification or regression, and we expect an encoder with a trained head to match or beat a language model on them. We include them as basic checks of perception, not as our argument for the language interface. The argument is that one model answers questions for which it has no dedicated head. In Tier 3 and Tier 4 the question, the options or the labeled audio examples change from item to item (which species called first, which of these labeled recordings matches this one), and a head per task would need its own design and training data each time. Further, a language interface lets a practitioner pose a new question without training a model, which is one use case that motivates audio-language models in bioacoustics. On call-type matching for the crow and giant-otter datasets, neither of which is in ROOTS, T4 scores 0.72 and 0.65 where its base model is at chance, with the call types given only as labeled examples in the prompt.
To put the classification tasks in context, we added the baselines without a language model that need no training. The audio-text models BioLingual and LAION-CLAP, used zero-shot, reach 57–83% on taxon presence, against 88–96% for T2 (App. G.12). For f0, T1 is more accurate than the pitch trackers pYIN and CREPE on the taxa seen in training, and the trackers are more accurate on the held-out hyena (App. G.9). Training task-specific heads, and probing the encoder to isolate what the decoder adds, would warrant a study of their own. The paper names both as limitations, and we argue that the contribution still holds without them: the data and the benchmark for capabilities that current audio-language models lack.
W5 (are the held-out species absent from AudioSet pre-training?). All NatureLM-audio models start from the same encoder checkpoint, so any exposure is the same for the base model and the fine-tunes, and it does not explain the gains. For the encoder itself we cannot rule it out, but we argue it is unlikely to matter. The checkpoint we start from is the self-supervised BEATs model (iter3+), which is pretrained on AudioSet by masked prediction of acoustic tokens and is not fine-tuned for classification, so the encoder was not trained to name any species. Even with labels, AudioSet classes are coarse (bird, frog). AudioSet clips also come from YouTube and our held-out recordings from Xeno-canto and iNaturalist, so the same recording in both is unlikely. Further, the encoder is trained further on bioacoustic data in both stages of the base model, so the AudioSet pretraining is only its starting point. The limitations now state this.
W6 (zeros on captioning and scenes). The zeros are correct, and Section 5 now explains them (App. G.13). They have two causes.
Captions. AF-Next and Qwen3-Omni write generic descriptions that do not name the species or the call. To check that the zero is not an artifact of CIDEr, we added four further metrics, and under each of them their captions match the reference of another recording as well as their own. T3 names 25–31% of the reference species, against 0–4% for them.
Frequency range. The baselines do not produce the required list of frequency bands, so their answers cannot be scored. When given the format, the overlap of their band with the reference (intersection over union) is 0.17–0.39, against 0.49 for T3. One fixed band for all recordings scores 0.65, so we report this task as open for every model, T3 included.
Q1 (lower f0 MAE on held-out species; comparison with the original work). It is a scale effect. The held-out split is one species, the spotted hyena, with targets around 330 Hz, while the seen split has a median of 5175 Hz, so an error in Hz is smaller on the hyena for any method. For this reason the main tables now report the accuracy within a semitone, following Best et al. (2025), which does not depend on the scale: 71% on the seen taxa and 15% on the hyena, where the pitch trackers pYIN and CREPE are better (App. G.9). The paper now also narrows the f0 claim to the taxa seen in training. Our numbers and those of Best et al. are not directly comparable: they report per-frame accuracy with one model per taxon, and our model outputs the mean f0 of a call as text.
Q2 (was T1 the only fine-tuned model?). No model was fine-tuned on BEANS-Next. We trained NatureLM-audio v1.1 and, from it, T1–T4, each on one tier of ROOTS; the other models are evaluated as released. Due to computational constraints, we decided to pick the model with the most favorable bioacoustics performance to base our work on. Section 4.2 now states this.
We thank the reviewer for the thorough review. The revised paper now shows its tasks, datasets and held-out taxa in tables, and we ran experiments on each concern about the construction of the benchmark, which we describe below.
W1 (venue). We believe the paper is in scope: the ICLR call lists datasets and benchmarks, and comparable, impactful bioacoustic work has appeared here (BirdSet and NatureLM-audio, ICLR 2025). The paper is also addressed to the people who build audio-language models, and shows on which bioacoustic tasks current general models fail. Further, the problems it deals with are not specific to biology: how to generate audio-text supervision at scale where expert annotation is scarce, and how to control for text shortcuts in a benchmark built with an LLM. We agree that review at ICLR does not replace scrutiny by biologists, and we thus release the dataset card, the held-out taxa and the per-recording provenance so that domain experts can check them.
W2 (presentation). We agree that a benchmark paper has to show its tasks, and that a reader should not need the cited papers to interpret a result. However, we also think that for a large dataset paper, information needs to be balanced between the main text and appendix to keep the narrative of the paper clear and understandable. The main text now has a task table (what each task measures, number of items, metric, chance level, source of the labels). Appendix D lists the held-out taxa (200 species, 50 genera, 10 families) and has a table of the source datasets (taxa, origin of the labels, and the tasks built from each). Section 4.1 describes the audio encoder and which components were trained in each fine-tune.