acceptodds
Withdrawn from ICLR 2027

Acoustic-to-KV Regression for Bioacoustic Recognition in Speech Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Animal-sound recognition requires distinguishing call types, callers, and species from acoustic evidence. Speech language models can encode useful distinctions between animal sounds yet fail to select the correct labels. On marmoset calls, a classifier fitted to frozen Qwen2.5-Omni features outperforms audio in-context learning given the same labelled recordings. We introduce Acoustic-to-KV Regression (AKR) to help the model use its acoustic features when forming its own answer. The key idea is to learn from labelled recordings how to adjust the model’s internal states to favor the correct answer. Negative gradients of the correct-answer loss provide correction targets for audio-token key/value (K/V) states. A low-rank ridge regressor then learns to predict these targets from acoustic features. For each new recording, AKR adds the predicted correction to its K/V states before the original decoder generates a label, without backbone updates or query-time backpropagation. Across five recording-group evaluations of 1-kHz marmoset recognition, using support clips that become misclassified after filtering, AKR increases mean accuracy from 12.35% to 32.47% and macro-F1 from 5.92% to 24.54%, outperforming native generation and a fixed correction in every evaluation. Held-out analyses further show that acoustic features predict these correction targets more accurately than fixed-mean and shuffled-pairing controls. These results demonstrate that predicting internal corrections from acoustic features can improve a frozen model’s own recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech language models, bioacoustic recognition, few-shot learning, spectral decision drift, inference-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7PoROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides an alternative to fine-tuning and in-context learning for bioacoustic classification in a single particular case: marmoset vocalization classification with 2 other datasets used in experimentation: dogs and watkins from the larger suite BEANS.

Critical strengths

Strong method to steer an audio LLM towards a particular task and dataset without fine-tuning or complex in-context learning. I found the experiments concerening predicting gradients from acoustic features very interesting proving that the model learns relationships between these features and the direction of the steering gradients

Critical weaknesses

The dataset suite is somehow limited. There are multiple datasets in BEANS and there are other bioacoustic datasets (BirdSet has a real drift between train and test) but somehow only 2 from BEANS were chosen in addition to the main one on which probably the main development and experiments were conducted. It looks somehow cherry-picked but it might be that the results were not ready for the deadline.

The paper has some readability problems, I find many paragraphs very compressed and hard to grasp, unusual terms which have more established alternatives, bad captions (if you present results mention the dataset for which the result is presented) .

The main evaluation issue is that the paper uses different sometimes not consistent experimental design for different experiments which makes it hard when actually trying to understand if the proofs support the claims. It would be better to have some clear experiments across multiple datasets (use all datasets for all main experiments), without doubtful conditions like the LP-experiments and a series of ablations (including the many kinds of drifts).

Questions for authors

The protocols are somehow confusing, tbh I have never seen this experiment design pre-LLM era in AI audio related papers. These sound like ablation or different condition testing. Comparators/evaluators are just "metrics" in the ML idiom. Why use "Exploratory diagnostics" and not just "Results". Going into the actual text I found the same issue with multiple paragraph, it makes it a bit hard to read, like a compressed version of a longer, more comprehensive explanation. Why use these terms when there are established ones? I hope I am wrong but I guess the experiment design and the way it's described for readers are largely LLM-driven?

I am not convinced by the low pass filtering experiments: a classifier may decide to focus less on high frequency content and for particular datasets like dogs there is not much above 8KHz. For marmosets may indeed matter but then low pass may introduce biases e.g. different classes have consistently higher frequency than the others.

It's not clear why that one particular model was used. NatureLM-audio is a bioacoustic specialized model so I wonder if the results generalize to it.

The main alternative to AKR is ICR and the paper lacks a main AKR-versus-ICL comparison across the dataset. In addition, ridge seems far better than AKR. AKR itself improves the internal representations only in the LP ablations. Not sure if this is enough.

Reviewer_h8E8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Acoustic-to-KV Regression (AKR), a method for improving bioacoustic recognition in frozen speech-language models. It learns to predict corrections to transformer key/value states from acoustic features using gradients derived from labeled support recordings. The method is primarily evaluated on marmoset call-type recognition using Qwen2.5-Omni, with additional analyses involving BEANS Dogs and Watkins. The authors report improvements over native generation in their main experiment, although supervised classifiers using frozen acoustic representations achieve substantially higher recognition performance.

Critical strengths

An interesting method for adapting frozen speech-language models: The paper proposes a method for predicting input-dependent corrections to internal K/V states without updating backbone parameters. Its gradient-prediction experiments and intervention controls provide evidence that acoustic features can predict useful correction directions. The approach demonstrates that such interventions can improve the original model's predictions under certain experimental conditions.

Critical weaknesses

1. Insufficient evidence for the practical advantage of K/V interventions: The paper does not adequately establish when gradient-based K/V interventions are preferable to directly classifying frozen acoustic representations. Although the authors include frozen Qwen ridge classifiers and AVES-bio baselines, conventional acoustic features such as MFCCs and GeMAPS, as well as general-purpose pretrained audio representations such as BEATs, are not evaluated under matched low-resource conditions. More importantly, the frozen Qwen ridge classifier substantially outperforms AKR under the matched-support fixed-setting extension (Table 3). Therefore, the practical advantage of the proposed intervention over simpler alternatives remains insufficiently established.

2. Limited evidence of generalization beyond the main experimental setting: The principal recognition gains are demonstrated on MarmAudio recordings low-pass filtered at 1 kHz using support recordings specifically selected for filtering-induced recognition failures. The additional datasets are primarily used for complementary analyses rather than comparable AKR recognition evaluations. Furthermore, the fixed-setting extension shows an accuracy improvement in only one of four model/input conditions, and this improvement is not accompanied by a macro-F1 gain. The experiments also remain restricted to the Qwen2.5-Omni model family. Together, these limitations weaken the evidence that AKR provides robust improvements across bioacoustic tasks, support-selection strategies, input conditions, and model architectures.

Questions for authors

1. Baseline comparisons and practical motivation: Given the substantially higher performance of frozen ridge classifiers in Table 3, what practical advantage does AKR provide over directly classifying acoustic representations? Could the authors report a matched-support frozen Qwen classifier under the main error-enriched recognition protocol, together with relevant alternative acoustic representations such as MFCCs, GeMAPS, or BEATs?

2. Generalization across tasks and architectures: The principal AKR recognition gains are demonstrated on MarmAudio under error-enriched support selection. Although BEANS Dogs is used for intervention controls and BEANS Watkins for additional recognition diagnostics, the paper does not establish comparable AKR recognition improvements across these tasks. Could the authors clarify why AKR was not evaluated more extensively on these datasets or larger bioacoustic collections such as DogSpeak? Furthermore, the fixed-setting extension improves accuracy in only one of four model/input conditions, and all AKR evaluations use Qwen2.5-Omni variants. What evidence supports the applicability of AKR beyond this architecture family and the particular support-selection conditions responsible for the main improvements?

Reviewer_tbdROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AKR, Acoustic-to-KV Regression, for improving bioacoustic recognition in speech language models. The method is motivated by the observation that representations of speech language models on animal sound support linear classification of the animal sound, but the model cannot generate the correct label. It fits a regressor that maps acoustic features to the gradient-derived correction. At test time it adds the predicted correction to the audio tokens' k/v states, so that the unchanged model can produce a better answer. The authors evaluated AKR on MarmAudio call-type recognition and showed that accuracies rise.

Critical strengths
  1. The observation that leads the paper is interesting. The fact that a linearly decodable information inside an LM is not used by the model to produce its own answer is a practical problem to solve.
  2. The method as far as I understand seems cheap to use. Fitting seems to require just one backward pass and a closed-form ridge fit.
Critical weaknesses
  1. Why the method uses the model's decoder part is not supported by the results. A simple ridge classifier consistently beats AKR by a wide margin (Table 3).
  2. The evaluation seems particularly thin. The accuracy gain is reported on a small set of (20) clips that are picked especially because the model gets them right on full-band audio and wrong after 1-kHz filtering. This seems to be a subset that is especially suitable for the method. On other setups like Table 3, the gain is a lot smaller. Table 1 is also missing basic baselines like a ridge classifier on the features.
Questions for authors
  1. How were the hyperparameters chosen? Can the authors add more details to help understand or reproduce the results? For example section 3.1 says corrections are added at “selected” K/V layers. What are these layers? How were they selected?
  2. What's the advantage of using the model's original decoder than a simple ridge classifier on top of the features? and how does AKR compare with ridge for the Table 1 examples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.