acceptodds
Under review as a conference paper at ICLR 2027

BEANS-Next and ROOTS: Broadening Audio-Language Capabilities for Bioacoustics

Abstract

Bioacoustics and ethology encompass a wide range of audio understanding tasks, many of which stand to benefit from recent advances in large audio-language models. However, progress in the field has so far been assessed on a narrow set of tasks, primarily centered on label-centric biological category recognition, such as species and call-type classification. In this work, we introduce BEANS-Next, a benchmark grounded in a taxonomy of bioacoustics tasks spanning acoustic perception, biological category recognition, scene understanding, and in-context learning. Using BEANS-Next, we show that existing models exhibit limited performance beyond the task families emphasized by existing evaluations, constraining their usefulness for broader bioacoustic applications. To support progress on this broader task space, we also introduce ROOTS, a large-scale training resource built from expanded curated real-world data and previously underused behavioral and acoustic metadata, supplemented by audio-derived information and scalable synthetic generation where labeling is insufficient. We demonstrate that training on this dataset yields substantial progress across all task groups of BEANS-Next, moving audio-language models closer to their potential as general-purpose assistants for bioacoustics and ethology. To accelerate progress in the field, we open-source our benchmark, dataset, and data pipelines.

Primary area:
datasets and benchmarks
Keywords:
bioacoustics, audio-language models, benchmark, dataset, multimodal learning, ethology, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gwgKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

I apologize in advance if I deviate somewhat from the ICLR reviewing template. It's a bit too restrictive at times for what I want to say.

Paper Summary & Contributions

This paper relates to the creation of a new benchmark for bioacoustics and its use to train a new audio-language model for bioacoustics. Essentially, the paper falls under the category of "data-driven" deep learning. The authors introduce a large new benchmark (derived by existing benchmarks/datasets), recast it as an open-ended language generation benchmark, and fine-tune existing audio-language models to perform the tasks in the benchmark.

Validation

The validation in this paper is self-referrential. The authors construct a benchmark and train models on that benchmark. The lack of an external validation is a weakness of this paper (more below)

Decision & Overall Feedback

I am strongly leaning towards rejecting this paper. The reasons will be made more clear below. I say "leaning towards" and not "will reject" because there are some questions to the authors. I am also signalling to the AC that they might overrule my decision, if the other reviewers are strongly in favor of accepting it.

While there are enough minor weakneses that I will mention below, my mean reasons for rejection are two:

  1. I disagree with enough choices made in the constructon of BEANS-next that I don't think it should be published like this

  2. I consider ICLR to be the wrong venue for this paper. I consider this for two reasons: a) It is highly unlikely that this will be reviewed by biology experts, given they are not part of the reviewer pool. I don't think we should be releasing major benchmark papers without domain expert reviews. b) The page limit is simply too low for what the authors want to present. As I say below, this forced the authors to pack a lot of vital parts in the appendix (which technically I am not obligated to read, but I had to..)

I get that the authors want to maximize their exposure to the ML community by submitting to ICLR, and I think more exposure is good for the bioacoustic community (and most likely for the careers of the authors, so I sympathize with that), but they are missing on the most crucial community -- the biology/ecology experts that will be benefitting from these tools.

Reviewer self-evaluation

There's a multiple-choice box I have to click but I like to be more verbose on this so the authors have a chance to judge my expertise. I have been working on audio machine learning for the past 8 years. For about half of those years, I have worked on computational bioacoustics and ecoacoustics as a senior researcher at a public institution. I have thus had daily experience with the algorithms and data used here. Recently, I have also been working on AudioLLMs (like everyone else), but not directly applying them to computational bioacoustics.

I would thus consider myself very suited to evaluate this paper.

Critical strengths

Sensible goal

This is an interesting work that is a natural extension of ongoing threads in bioacoustic and general audio research. What I mean by that is that everyone is re-casting all previous benchmarks to fit the LLM paradigm and then training an audio-language model to perform the task. This work needs to be done for bioacoustics and the authors make the effort to do it.

This is a very worthy goal and weighs in favor of publishing this paper.

Sensible methodology (in broad strokes, not details)

This is how most bioacoustic researchers would go about creating the benchmark and models. The authors pooled the data from the right sources, they combined multiple tasks (I guess one could argue about the selection of tasks though), and trained standard models. They also release their code, which makes it a good entry point for future researchers.

Critical weaknesses

Presentation

I already phrased my concerns on presentation, but they are so strong I need to re-iterate them. The authors had to pack so much on 9 pages that a lot of the important stuff was relegated to the appendix:

  1. The exact tasks -- what is a benchmark paper about if not about showing and discussing the tasks? Yes, I know, no way to do this on 9 ICLR pages, which is why I think the choice of venue is wrong...

  2. The included/excluded species -- not that they would help "me", but domain experts would probably like to know. Again, that's why I recommend a different venue, so that these experts can judge whether the separation into training/held-out species is sensible

  3. The reader is constantly referred to previous publications to get important information, such as where and how all benchmark datasets were recorded (immensely important in bioacoustics!), how the audio encoder was trained.

These (and other) issues made me see this paper as a hasty attempt to hit a deadline. That's understandable, we all do it, but I think it's really unsuited for a paper that aims to establish a standard benchmark in the field.

The most important thing is that the page limitation leaves no space whatsoever for any detailed discussion of a number of surprising results (see below). For instance, the authors repord CIDEr scores of 0.0 (!!) on captioning (Table 5) but don't even make a passing reference to that fact in their text.

Benchmark construction & Baselines

The most issue I could identify is the construction of the benchmark (and the lack of baselines).

The whole premise of the paper is that creating an audio-language ("captioning") benchmark covering multiple tasks is an important milestone for bioacoustics (Indeed, it is).

However, when going through the table of tasks, it is the first time that one sees any description of the actual tasks. I must say, I was somewhat disappointed at that point:

  1. Most of the tasks rely on noisy Xeno-Canto/iNaturalist annotations; this is a standard approach in bioacoustics (and, generally, ML literature), but the authors should nevertheless have discussed the ambiguity of the annotations. Better yet, they should have done the hard work of verifying the reliability for a subset of them

  2. Most of the tasks are cast as simple regression / multi-label classification tasks "dressed up" as linguistic tasks by turning them into a binary or multiple-choice question, using an LLM to do that work. I know this is standard practice in the AI literature at the moment, but I fail to see the point. I especially fail to see the point of using an AudioLLM to perform those tasks, when I simple audio encoder retrofitted with task-specific output heads could just as well do the job. This is a baseline that I would like to see, esp. for those tasks where baselines already exist in literature. The only "true" textual task I could identify was captioning (e.g. Table 5). But results there seem to be abysmally bad.

  3. Not a benchmark issue per se, but the audio encoder is (presumably) BEATs, which has been pretrained on AudioSet-2M. AudioSet has a lot of animal sounds. Are the authors certain that held-out species do not appear during pre-training?

Weird benchmark numbers?

  1. Are all scene results in Table 5 truly 0.0 except for the in-domain?

  2. Mean f0 MAE is much lower for held-out species than for seen species. This makes absolutely no sense and is not explained in the text. Moreover, looking at previous work on the data used to create that F0 benchmark, the original authors used a different evaluation scheme, computing accuracy to the nearest semitone, and report a top accuracy of ca. 70%. That should map to match lower MAEs than reported here -- but of course it's impossible to compare since we don't even know if the same splits are used.

Questions for authors
  1. Is there any explanation for the lower F0 MAE scores on held-out species vs seen?

  2. Was NatureLM-audio T1 the only model that was in anyway finetuned for this benchmark?

  3. Are there relevant, non-LLM baselines (e.g. a single audio encoder with output heads). What is the influence of the LLM decoder on performance?

3 comments

We thank the reviewer for the thorough review. The revised paper now shows its tasks, datasets and held-out taxa in tables, and we ran experiments on each concern about the construction of the benchmark, which we describe below.

W1 (venue). We believe the paper is in scope: the ICLR call lists datasets and benchmarks, and comparable, impactful bioacoustic work has appeared here (BirdSet and NatureLM-audio, ICLR 2025). The paper is also addressed to the people who build audio-language models, and shows on which bioacoustic tasks current general models fail. Further, the problems it deals with are not specific to biology: how to generate audio-text supervision at scale where expert annotation is scarce, and how to control for text shortcuts in a benchmark built with an LLM. We agree that review at ICLR does not replace scrutiny by biologists, and we thus release the dataset card, the held-out taxa and the per-recording provenance so that domain experts can check them.

W2 (presentation). We agree that a benchmark paper has to show its tasks, and that a reader should not need the cited papers to interpret a result. However, we also think that for a large dataset paper, information needs to be balanced between the main text and appendix to keep the narrative of the paper clear and understandable. The main text now has a task table (what each task measures, number of items, metric, chance level, source of the labels). Appendix D lists the held-out taxa (200 species, 50 genera, 10 families) and has a table of the source datasets (taxa, origin of the labels, and the tasks built from each). Section 4.1 describes the audio encoder and which components were trained in each fine-tune.

W3 (self-referential validation; noisy annotations). We agree that a benchmark built with the same pipeline as its training data needs checks from outside that pipeline. We point out that no model is trained on the benchmark, and we controlled for the ways in which the shared pipeline could still inflate the scores.

Shared text. Benchmark and training questions come from the same templates and LLM, so a model could in principle answer from text priors. We tested this with a classifier that sees only the question and the options, and found nine multiple-choice tasks where it was well above chance (96.5% on acoustic description matching, the task on which T1 scored 97.9%). We rebuilt eight of them (the behavior MCQ and the seven Tier 3 MCQs) with wrong options drawn from the correct answers of other items, so that the text no longer points to the answer. We removed description matching: a listening test showed that the task is ambiguous, because the field-note descriptions are incomplete and more than one option can fit a recording. On the rebuilt Tier 3 species tasks T3 scores 47–73% with the recording and 29–34% without it (chance 30–33%; App. G.3). We also re-evaluated with paraphrased prompts written by a different LLM, and observe minimal performance differences for the fine-tuned models (at most 6 points; App. G.8).

Shared recordings. We compared recording identifiers with all training files. About 3% of the taxon presence test items (163 of 4,940) are also in Tier 2 training files, with an effect of at most 0.2 points on T2. In Tier 3, 1,530 of the 1,903 items with a Xeno-canto identifier use recordings that are in Tier 2 training files; the fine-tuning of T3 did not use these files, and the Tier 3 questions rely on the event annotations of the source datasets, which are not in ROOTS. We point out that more than half of the tasks, including all of Tier 3, come from datasets that are not in ROOTS. Further, the sources that training and test data share are Xeno-canto and iNaturalist, which are crowd-sourced and contain a lot of variation in recording conditions, style, and text.

Checks outside our pipeline. The labels of f0, of the crow and otter call types and of DCASE do not come from our pipeline. Every model is now also evaluated on BirdSet and BEANS-Zero, and we added baselines that were trained by others (the pitch trackers pYIN and CREPE, and the audio-text models BioLingual and LAION-CLAP).

Annotation noise. We agree that the Xeno-canto and iNaturalist metadata is noisy: the focal label can miss background species, call type and behavior are free text from the recordist, and iNaturalist labels are community identifications. Still, these archives are the largest source of supervision that exists for bioacoustics, and the question is how much of the noise reaches the generated data. The generating LLM is instructed to decline when the recordist's notes do not support an item, and it rejects 27% of the generated items. To measure what remains, one of the authors listened to 125 items that passed the filter, next to the recordist's notes, and judged 8.0% unfavorable as training labels (App. G.4). We also argue that noise in the training labels works against the fine-tunes, not for them, on the tasks whose test labels come from experts. The paper now discusses these ambiguities. A verification of a subset of the benchmark labels by domain experts is named as a limitation, and the per-recording provenance is released so that it can be done.

W4 and Q3 (classification dressed as language; encoder with heads; influence of the decoder). We agree that taxon presence, call-type presence and f0 are classification or regression, and we expect an encoder with a trained head to match or beat a language model on them. We include them as basic checks of perception, not as our argument for the language interface. The argument is that one model answers questions for which it has no dedicated head. In Tier 3 and Tier 4 the question, the options or the labeled audio examples change from item to item (which species called first, which of these labeled recordings matches this one), and a head per task would need its own design and training data each time. Further, a language interface lets a practitioner pose a new question without training a model, which is one use case that motivates audio-language models in bioacoustics. On call-type matching for the crow and giant-otter datasets, neither of which is in ROOTS, T4 scores 0.72 and 0.65 where its base model is at chance, with the call types given only as labeled examples in the prompt.

To put the classification tasks in context, we added the baselines without a language model that need no training. The audio-text models BioLingual and LAION-CLAP, used zero-shot, reach 57–83% on taxon presence, against 88–96% for T2 (App. G.12). For f0, T1 is more accurate than the pitch trackers pYIN and CREPE on the taxa seen in training, and the trackers are more accurate on the held-out hyena (App. G.9). Training task-specific heads, and probing the encoder to isolate what the decoder adds, would warrant a study of their own. The paper names both as limitations, and we argue that the contribution still holds without them: the data and the benchmark for capabilities that current audio-language models lack.

W5 (are the held-out species absent from AudioSet pre-training?). All NatureLM-audio models start from the same encoder checkpoint, so any exposure is the same for the base model and the fine-tunes, and it does not explain the gains. For the encoder itself we cannot rule it out, but we argue it is unlikely to matter. The checkpoint we start from is the self-supervised BEATs model (iter3+), which is pretrained on AudioSet by masked prediction of acoustic tokens and is not fine-tuned for classification, so the encoder was not trained to name any species. Even with labels, AudioSet classes are coarse (bird, frog). AudioSet clips also come from YouTube and our held-out recordings from Xeno-canto and iNaturalist, so the same recording in both is unlikely. Further, the encoder is trained further on bioacoustic data in both stages of the base model, so the AudioSet pretraining is only its starting point. The limitations now state this.

W6 (zeros on captioning and scenes). The zeros are correct, and Section 5 now explains them (App. G.13). They have two causes.

Captions. AF-Next and Qwen3-Omni write generic descriptions that do not name the species or the call. To check that the zero is not an artifact of CIDEr, we added four further metrics, and under each of them their captions match the reference of another recording as well as their own. T3 names 25–31% of the reference species, against 0–4% for them.

Frequency range. The baselines do not produce the required list of frequency bands, so their answers cannot be scored. When given the format, the overlap of their band with the reference (intersection over union) is 0.17–0.39, against 0.49 for T3. One fixed band for all recordings scores 0.65, so we report this task as open for every model, T3 included.

Q1 (lower f0 MAE on held-out species; comparison with the original work). It is a scale effect. The held-out split is one species, the spotted hyena, with targets around 330 Hz, while the seen split has a median of 5175 Hz, so an error in Hz is smaller on the hyena for any method. For this reason the main tables now report the accuracy within a semitone, following Best et al. (2025), which does not depend on the scale: 71% on the seen taxa and 15% on the hyena, where the pitch trackers pYIN and CREPE are better (App. G.9). The paper now also narrows the f0 claim to the taxa seen in training. Our numbers and those of Best et al. are not directly comparable: they report per-frame accuracy with one model per taxon, and our model outputs the mean f0 of a call as text.

Q2 (was T1 the only fine-tuned model?). No model was fine-tuned on BEANS-Next. We trained NatureLM-audio v1.1 and, from it, T1–T4, each on one tier of ROOTS; the other models are evaluated as released. Due to computational constraints, we decided to pick the model with the most favorable bioacoustics performance to base our work on. Section 4.2 now states this.

Reviewer_XTqkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BEANS-Next, a benchmark spanning acoustic perception, biological category recognition, scene understanding, alongside ROOTS, a training resource containing approximately 44 million audio-language pairs built from curated metadata, LLM-generated text, audio-derived annotations, and synthetic recordings. The authors evaluate general-purpose and bioacoustic audio-language models and train separate tier-specific models from a shared NatureLM-audio checkpoint. Results show substantial reported gains in acoustic-description matching, biological recognition, several scene-reasoning tasks, and few-shot matching, while counting, captioning, and expert crow-description matching remain challenging.

Critical strengths
  1. BEANS-Next broadens evaluation in bioacoustic research. Beyond species recognition, it tests acoustic measurement, event counting, temporal relationships, and matching against audio examples.
  2. ROOTS provides promising evidence that richer supervision can improve capabilities beyond species identification. Starting from a shared checkpoint, tier-specific training yields substantial reported improvements in behavior recognition, call-type matching, and few-shot detection on real recordings
Critical weaknesses
  1. The evidence for transferable acoustic measurement is incomplete. The held-out fundamental-frequency evaluation covers only spotted hyenas, so improvement on this split provides limited evidence of generalization across unseen data.
  2. The experiments do not isolate which aspects of ROOTS account the improvements. Using a shared architecture and starting checkpoint is a useful control, but the fine-tuning comparisons change the supervision, training mixture, and amount of training together
  3. The main results and appendix controls give substantially different scores for the same models and tasks, without adequately explaining why. For example, NatureLM-audio performs well above zero on species-by-order identification in the main table but scores approximately zero in the appendix’s real-audio condition
Questions for authors
  1. Does the fine-tuned model outperform a pitch-estimation methods, and does its performance deteriorate when the audio is mixed with noise?
  2. How do you define a single vocalization in the counting tasks? Is the definition consistent across synthetic training data and real evaluation recordings, particularly for individual notes, complete phrases, and overlapping calls?
  3. In the synthetic call-type tasks, how often are negative examples different calls from the same species? Can the model otherwise solve these tasks by distinguishing species rather than call types?
1 comment

We thank the reviewer for the comments, which led us to test the f0 claim on a second group of taxa and to compare T1 with pitch trackers.

W1 (held-out f0 covers only spotted hyenas). We agree. We built a second held-out group by retraining T1 with and without the f0 labels of three taxa (canids, little owls, Réunion grey white-eyes; 473 test calls), and added a pitch-shift control (App. G.10). Both give the same picture as the hyena: on taxa without f0 labels T1 finds the frequency region of the call but not its exact pitch. With the labels T1 is within a semitone on 81.8% of the calls; without them on 7.4%, with 80.5% of the answers within an octave. On pitch-shifted clips its answer follows 86–87% of the shift on the seen taxa and 71–83% on the hyena. Section 5.1 now states that a coarse, audio-driven estimate of the frequency region transfers and fine pitch does not, and no longer speaks of a transferable mapping.

W2 (the experiments do not isolate which aspects of ROOTS account for the gains). The comparison separates the three factors that the reviewer names as far as our claim needs.

Amount of training. The base model was trained to the end of its schedule (1.5 million steps, as stated in the appendix on training), so the base model is not undertrained. Additional training alone also does not produce the gains: every fine-tune received it, but only the one trained on the data of a tier improves that tier (Table G1; f0 on seen taxa: 253 Hz for T1, against 4291–4618 Hz for T2, T4 and the base).

Supervision. In a retraining with the same recipe and number of steps, removing the f0 labels of three taxa takes T1 from 81.8% to 7.4% within one semitone on these taxa, and leaves the other taxa nearly unchanged (66.6% and 62.9%; App. G.10). The gain follows the labels.

Mixture. Our claim is at the level of the tier: the data of a tier, taken as a whole, yields the capability of that tier. The paper now says this and no longer assigns an ability to a single data source. Separating the sources within a tier needs one retraining per source, which we name as future work.

W3 (main results and appendix controls disagree). We traced this to a parser version difference, and reran the controls with the newest one (App. G.11, E.6). For example, NatureLM-audio v1.0 starts every answer with a timestamp, which the main-table scorer ignored and the control scorer rejected. This produced the zeros that were pointed out. 

Q1 (pitch trackers and noise). We ran two standard pitch trackers, pYIN and CREPE, on the same clips, clean and with added white noise (App. G.9). On the taxa seen in training T1 is more accurate (71% within one semitone, against 39% for pYIN and at most 44% with per-species settings). On the held-out hyena the trackers are more accurate (42% and 35%, against 15% for T1). With noise the error of T1 on the seen taxa rises gradually, from 253 Hz to 319 Hz at 10 dB and 1064 Hz at −5 dB, where the trackers are at 1595 Hz or more; on the hyena CREPE is more robust. Section 5.1 limits the comparison to off-the-shelf trackers on the taxa seen in training.

Q2 (definition of a vocalization in the counting tasks). The benchmark follows the expert annotation: in the evaluation recordings a vocalization is one annotated event in the selection table of the source dataset, so whether a note or a phrase counts follows the convention of that dataset. The synthetic training data has no such annotation, so there a vocalization is one activity window of the source-separated audio (Hoffman et al., 2025), which can hold more than one note. The counting tasks therefore test whether a model trained on the automatic definition transfers to expert annotation. A difference between the two definitions works against T3, not for it, and T3 still gives the most exact vocalization counts of all models (26% exact, next model 19%). Appendices B.3 and D.1 now acknowledge both definitions.

Q3 (same-species negatives in the synthetic call-type tasks). Rarely. In a sample of the synthetic training items, 1 of 3,476 wrong options is another call type of the same species. This is why the evaluation result is informative. The crow and giant-otter tasks contain one species each, with expert labels, and neither dataset is in ROOTS, so they cannot be solved by telling species apart. T4 scores 0.72 and 0.65 on them, against 0.24 and 0.20 for its base model (chance 0.25), and 0.09 and 0.14 when the query is replaced by another call type. Training on matching across species therefore transfers to call types within a species that the model never saw. Section 3.4 now describes how the synthetic negatives are drawn.

Reviewer_kgwXOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper contributes BEANS-Next, a comprehensive benchmark for bioacoustic audio-language models organized around a four-tier taxonomy, and ROOTS, a 44M-pair training corpus aligned to the same taxonomy. It also presents experiments showing that fine-tuning a shared base model on each tier of ROOTS substantially improves the corresponding tier of BEANS-Next.

Critical strengths
  1. The four-tier taxonomy is a valuable contribution and grounded in actual practitioner needs. 

  2. Data engineering of ROOTS is carefully designed and well structured. It draws on underused behavioral and acoustic metadata and uses synthetic audio pipelines to produce the strongly labeled timing, counting, and ordering supervision that public archives currently lack.

3. The paper is honest about its limitations and failures (counting regression and low-SNR Gibbons gains) and includes text-only and noise controls which makes the results more reliable.

Critical weaknesses
  1. No joint model has been trained, but the paper claims to contribute towards general-purpose assistants.

  2. Benchmark and training data share templates and text-generation machinery.

  3. It lacks quantitative quality audit of the LLM-generated supervision (judge agreement with humans, rejection rates, estimated error rate).

  4. The text-only and Gaussian-noise controls are only run on the baselines (AF-Next, Qwen3-Omni, NatureLM v1.0), not on the T1–T4 fine-tunes that the paper is actually about. It is unclear if the tier gains come from audio understanding or from learned output-format/text priors.

Questions for authors
  1. Can you report the full cross-tier matrix along with the established benchmark scores for tier 1 through tier 4? and do the tier fine-tunes preserve species classification?

  2. Could you add general LALMs given in-distribution exemplars or explicit output format instructions on a subset of tasks?

  3. MAE for f0, SNR, and counting is computed over parsed answers only, with parse rates that range from 2% to 100%. Could you recompute the comparisons with unparseable answers imputed at worst-case error?

  4. Do the T1–T4 gains survive the text-only and Gaussian-noise controls? If a tier model retains much of its score on noise or text-only input, that would weaken the audio-capability interpretation.

  5. What explains the ESC-50 regression from 0.820 to 0.360 in v1.1, and does it affect the tier models built on top of it?

2 comments

We thank the reviewer for the constructive comments. The controls and the audit requested in this review are now in the paper. On the main tasks of each tier the gains of the fine-tunes disappear without the audio, showing the importance of the audio and audio tasks, and where a task still was partially solvable without audio, we rebuilt it,

W1 (no joint model). The claim of the paper is about the data: every fine-tune starts from the same base model and adds only the training data of one tier, which tests directly what that data contributes. Training a joint model comes with several other considerations that would warrant an additional paper. We thus adjust the abstract, the introduction and the contributions to say this, and no longer suggest a general-purpose assistant. As an alternative to a joint model, we weight-merged the four fine-tunes without retraining (App. G.1). The merged model keeps most of Tier 2 (bird presence 80.5% vs. 92.2% for T2) and matches T3 on the open-ended Tier 3 species tasks, but it does not keep the in-context matching of Tier 4 and f0 estimation, so joint training is still needed for these.

W2 (benchmark and training data share templates and generation pipeline). This is correct, and Section 3.2 says so. We controlled for the three ways in which it could inflate the scores.

The answer can be guessed from text priors. We found this in nine multiple-choice tasks, where a classifier that sees only the options was well above chance. We rebuilt eight of them (the behavior MCQ and the seven Tier 3 MCQs) with wrong options drawn from the correct answers of other items. The classifier is then at chance on all but one, the Tier 3 referring task (41.6%, chance 25%), and we report this value as the level that a model has to beat. On the rebuilt Tier 3 species tasks T3 scores 47–73% with the recording and 29–34% without it (chance 30–33%). We removed the ninth task, acoustic description matching (Desc. MCQ in Table 3 of the submitted version): on its rebuilt version T1 was at chance, and a listening test showed that the task is ambiguous, because the field-note descriptions are incomplete and more than one option can fit a recording (App. G.3).

The model has heard the recordings. We compared recording identifiers with all training files (App. G.8). About 3% of the taxon presence test items are also in Tier 2 training files, which changes the T2 scores by at most 0.2 points. We point out that more than half of the tasks, including all of Tier 3, come from datasets that are not in ROOTS. Further, most of the shared data is from Xeno-canto and iNaturalist, which are crowd-sourced and contain a lot of variation in recording conditions, style, and text. We thus argue the source data differences between train and test, including the source text, contribute to some vocabulary variations in the resulting audio-text pairs.

The model has learned the phrasing. The templates contain instructions and examples on variations of the prompt, that lead to large prompt diversity in the final data. Still, it can be argued that their distribution is shared between train and test (with the caveat of different source data leading to different text data, mentioned above). Thus, we re-evaluated one task per tier with three paraphrased prompts that do not occur in ROOTS, written by a different LLM (Claude Fable 5.2) from the one that generated ROOTS (Gemma 4). We observe minimal performance differences for the finetuned models, with the largest loss being 6 points on bird presence, and the Tier 3 and Tier 4 scores move by at most 2 points (App. G.8). 

W3 (no quantitative audit of the generated supervision). We ran a human audit and report the three quantities (App. G.4). Section 3.2 now describes the filter as it was run: the LLM judge was used to refine the generation prompts, and in the final runs the generating LLM declines to write an item when the recordist's notes do not support it, followed by format and keyword checks.

Rejection rate: the filter rejects 27% of the 10.67M generated items, mostly for the behavior MCQ (70% of the recordings).

Error rate: one of the authors listened to 125 randomly drawn items that passed the filter, next to the recordist's notes, and judged 8.0% unfavorable as training labels (95% interval 4.4–14.1%).

Agreement with a human: the same rater judged a second set that included rejected recordings, without seeing the decision of the filter. Rater and filter agree on 91% of the SNR category MCQ items (Cohen's kappa 0.81) and on 30% of the behavior MCQ items, where the filter rejects recordings labeled only "song" or "call".

W4 and Q4 (do the gains of T1–T4 survive the text-only and noise controls?). We ran three controls (Gaussian noise, a different audio, and no audio, which is the text-only control) on T1–T4, the base model and the baselines, on the test items of the main tables (App. G.2). The gains survive on f0, taxon presence, the open-ended Tier 3 tasks and the in-context matching of Tier 4: without the recording the scores fall to chance or to the level of the base model. The nine multiple-choice tasks of Weakness 2 did not pass these controls, which is why we rebuilt them.

Q1 (cross-tier matrix; is species classification preserved?). Every model is now evaluated on every tier, on BirdSet and on BEANS-Zero (Table G1). The fine-tunes are specialists: each is the best NatureLM-audio model on its own tier (f0 on seen taxa: 253 Hz for T1 against 2568–4618 Hz for the others). The one clear gain outside the own tier is T4 on taxon presence (amphibians 89.6% against 53.1% for the base). T2–T4 keep most of the BirdSet accuracy of the base model (41–47% against 50.4%); T1 does not (2.7%), and Section 5 says so.

Q2 (general models with exemplars or format instructions). We ran both for AF-Next and Qwen3-Omni (App. G.5).

f0: the gap is not a format penalty. With a format instruction both models answer with a bare number on 96–100% of the items, and their error on seen taxa stays above 3300 Hz, against 253 Hz for T1. With three example recordings and their true values before each query it stays above 2100 Hz. On the held-out hyena the baselines with these examples reach the level of T1 (123 and 86 Hz against 125 Hz), which agrees with our revised claim that fine pitch does not transfer to taxa without f0 labels.

In-context matching (Tier 4): the gap was a format penalty. With a letter-only instruction Qwen3-Omni reaches 0.66–0.72, the same as T4 (0.65–0.72). Table 7 has this row, and Section 5.4 states the result of T4 as a gain over its base model.

Q3 (MAE with unparseable answers imputed at worst case). We agree that MAE over the parsed answers only is not comparable between models. With worst-case imputation (App. G.6), T1 remains first on f0 (257 Hz on seen taxa, against 3753 Hz for the next model) and on SNR; T3 is first on the vocalization count and third on the species count. The main tables now report accuracy within a tolerance, following Best et al., where an answer without a number counts as wrong: T1 is within one semitone on 71% of the seen-taxa calls and on 15% for the held-out species (other models: at most 2% and 3%), and within 3 dB on 40% of the 298 SNR items (others: at most 9%). T3 gives the most exact vocalization counts (26%, next model 19%) and is level with the best model on the number of species (38%).

Q5 (ESC-50 regression of v1.1). The difference is in the non-animal classes, where v1.1 handles the long list of options in the prompt less well than v1.0 (App. G.7). On a rerun of the 400 items, v1.0 obtains 81.8% and v1.1 43.0%; v1.1 keeps 87.5% on the animal sounds, the domain of the paper. With 5 options instead of 50, its accuracy on the non-animal groups rises to 59–84% (chance 20%), and the order of the options has no effect. The tier models inherit this: T2 (40.2%) stays near its base and the other three are lower.

Comments on the paper

1 comment

We thank the three reviewers for the detailed and constructive reviews. We ran the experiments they requested. All new results are in Appendix G of the revised paper along with changes throughout the paper, and each reply points to the section.

What the new experiments show. The gains of the fine-tunes disappear when the recording is replaced by noise or removed: on f0 for the taxa seen in training (71% within one semitone, better than off-the-shelf pitch trackers), on the presence tasks of Tier 2 (88–96%, chance level with noise), on the species tasks of Tier 3 (47–73%, against 29–34% without audio) and on the in-context matching of Tier 4 (from chance to 0.65–0.72).

We comment further on the matter of sharing templates between train and test on individual reviews, but overall we also conducted an experiment paraphrasing the questions in BEANS-Next, and found minimal performance difference for our models.

What we changed. Nine multiple-choice tasks could be answered from the text of their options; we rebuilt eight and removed one that was deemed to have too noisy source data. For regression tasks, due to low parse rates, we also now present accuracy-within-tolerance metrics. After new results, we narrowed the f0 estimation generalization claim, and noted that Qwen3-Omni can perform on par with the T4 model on in-context matching once prompted with the answer format.

What we could not do in the rebuttal period. A jointly trained model, ROOTS factors ablations within a tier, and retraining a different LLM decoder. The paper states these as limitations, though for each review we argue that our contribution still holds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.