acceptodds
Under review as a conference paper at ICLR 2027

MusicListenBench: Can Audio LLMs Hear the Difference Yet?

Abstract

Audio large language models (LLMs) are typically evaluated on song genre, mood, and instrumentation. However, their ability to recognize the same melody across two short clips remains untested. We introduce MusicListenBench, a benchmark comprising paired listening tests for melody, harmony, timbre, and tempo. We generate all audio from symbolic music, ensuring precise ground-truth answers. Each test item comes with two variants. A FLIP variant changes the music, so the answer must change. A STAY variant changes only the sound, such as room echo or hiss, so the answer must stay the same. The two variants have opposite answers, so a model that ignores the audio scores 50%, whatever letter it prefers. Seven of eight open audio LLMs score within 4 points of this floor, and three of them give the same letter to 88% to 99% of items. Of four commercial models, both GPT-Audio models are at chance, and the best, Gemini 2.5 Pro, reaches 74.2%, 12.8 points below human listeners (87.0%). Post-training with GRPO on 10k synthetic clean pairs raises Qwen2.5-Omni from 61.2% to 99.0% on held-out clean pairs, 11.5 points above our listeners, and the gain transfers to a task left out of training. Yet on FLIP/STAY the trained model only reaches the level of our listeners (86.0%). It now misses almost no change (1.9%, against 24.7% for Gemini 2.5 Pro), yet it errs on STAY pairs as often as Gemini does (26.1% against 26.9%). On timbre and rhythm it still changes its answer on 20.8% and 16.8% of pairs where only the sound changed, while our listeners made at most one error on each of these tasks. Nine in ten of these errors come from one change, a quiet background hiss. Training taught the model to ignore EQ changes it never saw (1.4% errors), but not hiss or transposition. Models learn to hear a difference much faster than they learn which differences matter. We release (anonymous) data, code [https://anonymous.4open.science/r/MusicListenBench-F38B/], and an open leaderboard [https://musiclistenbench-review.pages.dev].

Primary area:
datasets and benchmarks
Keywords:
Music Understanding, Audio LLM Benchmark, Music Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vMnsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a benchmark that tests the abilities of audio LLMs to recognize the same musical components, in terms of melody, harmony, timbre, and tempo. When comprising positive pairs, the authors changed sound itself rather than musical components, such as reverberation. Using the introduced benchmarks, the authors tested open weights as well as commercial audio LLMs, and made an open leaderboard. Their additional contributions include post-training an open audio LLM and analyze the results.

Critical strengths
  1. Human baseline is a strong comparison point.
  2. A benchmark to test the ability to discern musical components, including melody and rhythm, is timely and important.
  3. Open leaderboard and data is a contribution to the research community.
  4. The authors tried to reduce overfitting using held-out test conditions (e.g., EQ).
Critical weaknesses
  1. Training and evaluation have been conducted on synthetic music while validating using a small human-curated dataset seems necessary and doable.

  2. Whether the distribution of training and evaluation data reflects real-world music data is questionable. Held-out test conditions resemble training conditions and it is difficult to claim generalizability.

  3. Timbre is highly multi-dimensional but the authors experimented with limited sets of instruments.

Because of the data quality issues, the paper provides limited insights.

Questions for authors
  1. Did the authors try building a small human-curated dataset consisting of pairs from real-world dataset?

  2. Can timbre be tested on different levels, using synthesizers, for example?

2 comments

I apologize for my tardy response given the importance of the period. I appreciate the authors' effort to address my concerns.

The authors introduced two additional test sets based on real recordings as well as synth pairs to assess the ability or discerning timbre.

However, I'm not still convinced whether why the authors need to choose limited parameters (brightness and attack) to explain LLM's ability. Limited choice of parameters make it easy for models to overfit to benchmark when typical synthesizers offer hundreds of configurations to change.

Based on the authors' overall contributions, however, I'm inclined to raise my score.

AuthorsOpenReview ↗

Response to Official Comment by Reviewer vMns

Dear Reviewer vMns,

Thank you for your reply, and for your kind words about our revisions. We appreciate it.

We agree that brightness and attack cover only a small part of what a synthesizer can change. We chose these two because attack time and spectral centroid are two of the main dimensions found in a study of synthesized timbres [1], and we vary each in graded steps (cutoff 1/3 to 2 octaves higher, attack 20 to 200 ms against 5 ms).

On what the benchmark is for: it tests a necessary condition, whether a model can tell that two short clips are the same. Our main answer to the title question is "not yet" (Sec. 1). A narrow test is enough to show such a failure, but it cannot certify broad timbre ability, and we do not use it that way. The paper shows the risk you describe: after training, Qwen2.5-Omni scores 99.6% on the four-instrument task but detects only 10% of violin vs viola changes (App. P). This is why every test item comes with flip/stay variants and held-out test conditions, so that a high score on clean items alone does not count as success.

Our timbre results also do not rest on the synth parameters. Brightness and attack are two of the three new sub-tasks in App. P. The third uses close pairs of sampled instruments from the test soundfont, and the real-recordings set adds seven instruments from Medley-solos-DB. As App. P notes, a model may judge a brighter version of one synthesizer to be the same instrument, so the close pairs are the clearer test.

On overfitting: the rendered test set uses a different soundfont and different pitch ranges from training, the real-recordings set is used for evaluation only, and the leaderboard lists models trained on our training split separately from zero-shot models. Because the rendered items come from a generator, we can also draw a fresh test set with new seeds if the public one becomes stale.

We have added this limit to App. P (p. 40 end - p. 41 beginning), which now reads: "Brightness and attack cover only a small part of what a synthesizer can change, so these two sub-tasks are a first step toward graded timbre tests, not a full test of timbre."

We hope this addresses your remaining concern, and we would be grateful if your updated assessment could be reflected in the rating. Thank you again for your time.

Best regards,

The authors

[1] McAdams et al., "Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes", Psychological Research 58(3), 1995.

Reviewer_NZqhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a synthetic music-related benchmark for paired judgments of melody, harmony, timbre, and tempo, comprising 10,000 training and 3,000 evaluation items. The goal is to test whether current Audio LLM can really discover the difference between music. Its flip/stay variants test Audio LLM's responses to relevant changes and robustness to irrelevant changes. Experiments cover twelve audio LLMs, human listeners, and GRPO on four models. The best trained model reaches 99.0% clean accuracy but 86.0% flip/stay accuracy, exposing substantial residual errors under nuisance transformations.

Critical strengths
  1. The paper studies a useful and underexplored problem: whether audio LLMs can detect small musical differences. The transformations provide useful tests of both sensitivity to musical changes and robustness to irrelevant sound changes.
  2. The experiments cover 12 audio LLMs and post-training on four models. The results consistently show that improvements on clean pairs do not fully transfer to transformed audio.
  3. The generator, training split, evaluation code, and leaderboard support reproducibility. Pitch-range separation, soundfont changes, and checks for simple shortcuts strengthen the evaluation. Soundfont overlap is also clearly disclosed.
Critical weaknesses
  1. FFR does not track answer changes. A model can be wrong on both clean and stay pairs without changing its answer. A high FFR therefore does not necessarily show a failure to ignore sound changes. Further illustration or improvement should be considered on this defect of metrics.
  2. The training comparison is incomplete. GRPO uses only clean pairs, although transformed versions are available. Without an augmentation baseline, it is unclear how much of the reported robustness gap could be fixed by including these examples in training.
  3. Pitch edits may introduce a shortcut. Changing one note may make a melody or chord sound unusual. The current controls do not rule out that models detect this unusual sound instead of comparing the two clips.
Questions for authors
  1. Could pitch edits make a clip sound unusual, allowing the model to answer without comparing the two clips?
  2. The melody task contains only four notes in a 2.5-second clip. Do the model rankings and post-training conclusions persist for 5- and 10-second melodies containing more notes but only one altered note, two altered notes, etc. Will the duration and degree of change affect your final conclusion?
  3. Why use only clean pairs for GRPO? Would data augmentation improve robustness to unseen sound changes?
Reviewer_hzTpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MusicListenBench, a benchmark of paired same/different (and faster/slower) listening tests on melody, harmony, timbre and rhythm, rendered from symbolic music so that labels are exact. Each test item has a flip variant (the musical attribute changes) and a stay variant (only the sound changes: EQ, room echo, hiss, transposition). Together they yield a 50% floor for audio-agnostic models and separate miss rate from false-flip rate. The authors evaluate 8 open and 4 commercial audio LLMs against 12 human listeners, and post-train four open models with GRPO. Training raises clean accuracy to 99% but leaves false flips largely unchanged, with hiss and transposition as the main failure sources.

Critical strengths

Clear motivation. The paper asks whether audio LLMs actually perceive musical attributes rather than relying on language priors or answer bias. This is a timely and well-posed question.

Well-designed flip/stay protocol. Pairing each item with an answer-changing and an answer-preserving variant cancels letter bias and separates two failure modes that a single accuracy score conflates. This is shown convincingly by the letter-bias analysis (e.g., Qwen2-Audio answers “B” on 98.6% of items).

Exact, controllable ground truth. Rendering from symbolic music gives precise labels, graded difficulty, and interpretable per-level analyses (Fig. 7, Appendix I).

Broad evaluation. The paper covers 12 models spanning open and commercial systems and different architectures.

Strong reproducibility. The authors release the generator, scoring script, training settings and a public leaderboard, which should make the benchmark easy to adopt and extend.

Critical weaknesses

Simplified musical tasks. Melody is a four-note motif and harmony mostly a moved third. These tasks are well suited to isolating perception, but they cover a narrow slice of musical understanding compared with MIR-grounded benchmarks such as Pitchbench and CMI-Bench. The paper would benefit from stating more explicitly which level of musical competence the benchmark targets.

Limited ecological validity. All audio is MIDI + SoundFont, single-instrument and unmixed. Results may partly reflect the synthetic distribution, and robustness at the audio-encoder/tokenizer level to real recordings and polyphonic mixtures is untested. The authors acknowledge this, but some validation on real or mixed audio, even small-scale, would considerably strengthen the claims.

Thin human reference. There are 12 participants and 288 answers (24 clean and 48 flip/stay answers per task). Wilson intervals and a Fisher test are reported, which is appreciated. However, no inter-rater agreement is given, and accuracy is not broken down by musical experience, although that information was collected and per-person accuracy ranges from 66.7% to 95.8%. Since the human score (87.0%) serves as the target in the conclusion, its stability matters.

Missing training baselines and ablations. GRPO is used as a probe rather than proposed as a method, which is reasonable. Still, without an SFT (or DPO) baseline it is unclear whether the miss/false-flip asymmetry is specific to RL or to the clean-only reward. The paper also does not train on stay-type pairs or ablate perturbation types and training strategies, even though the training split already contains unused sound-only variants (Table 5). Such an ablation would directly test the paper’s central explanation.

Interpretation of the 99% clean accuracy. The authors themselves show that clean accuracy overstates listening ability. The near-ceiling score may therefore reflect learned similarity comparison on synthetic audio rather than musical concepts. The abstract’s framing (“11.5 points above our listeners”) could be toned down, and the mechanism behind the gain deserves clarification.

Limited analysis of failure mechanisms. The hiss and transposition findings are the most interesting results, but they remain descriptive. A representation-level analysis would turn the observation into an explanation, for example probing whether audio-encoder embeddings shift more under hiss than under EQ. A hiss-level sweep, already noted by the authors, would complement it.

Two closely related works appear to be missing:

[1] Dujardin, Milan Liessens, et al. "PitchBench: Measuring Pitch Hearing in Audio-Language Models." arXiv preprint arXiv:2605.26176 (2026).

[2] Ma, Yinghao, et al. "Cmi-bench: A comprehensive benchmark for evaluating music instruction following." arXiv preprint arXiv:2506.12285 (2025). o

Questions for authors

Does false-flip behaviour under hiss correlate with changes in audio-encoder representations?

How does the human score vary with musical experience, and what is the inter-rater agreement?

Would GRPO (or SFT) on stay-type pairs reduce the false-flip rate, and does this invariance transfer to unseen perturbations?

Are the clean-accuracy gains after GRPO significant per task? Some per-task gains are small (e.g., rhythm for most runs)

How to understand the 99% accuracy?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle follow-up regarding our response and new experiments

Dear Reviewer NZqh,

As the discussion period is nearing its end, we wanted to gently follow up on our recent response to your review. We have added the new experimental controls you requested to check if edited clips sound unusual on their own, evaluated the models on longer melodies, and ran the ablations with sound-only pairs in training. We also have updated the manuscript with newer numbers and explanations.

We would be very glad to hear your thoughts on these new results and whether they help alleviate your initial concerns. Please let us know if there are any remaining questions we can address before the period concludes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.