acceptodds
Under review as a conference paper at ICLR 2027

BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning

Abstract

Music understanding is a complex task that often requires reasoning over both structural and semantic elements of audio. We introduce BASS, designed to evaluate music understanding and reasoning in audio language models across four broad categories: structural segmentation, lyric transcription, musicological analysis, and artist collaboration. BASS comprises 2658 questions spanning 12 tasks, 1993 unique songs and covering over 138 hours of music from a wide range of genres and tracks, crafted to assess musicological knowledge and reasoning in real-world scenarios. We evaluate 14 open-source and frontier multimodal LMs, finding that even state-of-the-art models struggle on higher-level reasoning tasks such as structural segmentation and artist collaboration, while performing best on lyric transcription. Our analysis reveals that current models leverage linguistic priors effectively but remain limited in reasoning over musical structure, vocal, and musicological attributes. BASS provides an evaluation framework with widespread applications in music recommendation and search and has the potential to guide the development of audio LMs.

Primary area:
datasets and benchmarks
Keywords:
Long audio reasoning, multimodal evaluation, large audio language models, Music reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GrV6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BASS, a benchmark for testing whether audio language models can reason over complete musical recordings rather than short excerpts. It covers 2,658 questions from 1,993 songs and approximately 138 hours of music, organized into 12 tasks spanning structural segmentation, structure-aware lyric transcription, musicological analysis, and multi-artist collaboration. The benchmark’s main value is its emphasis on long-range temporal and compositional reasoning, with a evaluation of 14 open-source models. The results are informative but also reveal a substantial capability gap: models perform relatively better on lyric transcription, where linguistic or memorized priors may help, while struggling with musical structure, artist attribution, timing, and higher-level musicological reasoning. However, BASS relies substantially on existing annotations, and the paper leaves unresolved questions about overlap with baseline training data and the fairness of the evaluation. In addition, several newer SOTA models are not included. Thus, while BASS is promising, its claims about the current capability of audio language models should be supported by stronger contamination analyses and more timely baselines.

Critical strengths
  1. The paper addresses an important and timely gap: many existing music benchmarks emphasize short excerpts, captions, or shallow recognition, while practical applications such as retrieval, recommendation, and metadata generation often require reasoning over complete recordings and their long-range structure.

  2. BASS provides unusually broad and complementary task coverage. Its combination of temporal segmentation, structure-aware lyric transcription, musicological comparison/ranking, and multi-artist attribution evaluates capabilities that go substantially beyond standard audio captioning or generic music question answering.

  3. Several tasks require genuine compositional reasoning over time rather than local acoustic recognition. In particular, artist duration, artist localization, section-conditioned attribution, and structure-conditioned transcription require models to integrate information across repeated, non-contiguous, and hierarchically organized parts of a song.

  4. The empirical analysis is diagnostically valuable and reasonably transparent. The paper examines performance across musical sections, genres, languages, vocal styles, metadata conditions, reasoning modes, and audio preprocessing settings, providing useful evidence about current model limitations. Its song-based bootstrap procedure, raw results, task prompts, and detailed appendix statistics also make the findings easier to inspect and build upon.

Critical weaknesses
  1. The benchmark is substantially inherited from existing annotations. Although the questions are newly phrased, their ground truth is derived from Harmonix, Genius, MGPHot, and CoSoD. The paper therefore needs stronger evidence that BASS provides independent evaluation rather than merely a new natural-language descriptions to known datasets.

  2. Data contamination, label reliability, and external validity are not sufficiently established. The data of BASS may have appeared in baselines' pretraining. The paper should analyze the impact of data contamination as much as possible. In the era of LLMs, although it is impossible to guarantee that evaluation data has not been used for training, we can strive to ensure the fairness of evaluation.

  3. The baseline models may already be outdated. The evaluation omits newer systems such as Gemini 3.1 Pro, Gemini 3.8 Flash, Qwen3.5-Omni, and Gemma 4, so the reported performance gap may not reflect the current state of audio language models. Including more SOTA models would make the paper’s claims more timely and informative.

Questions for authors
  1. Could the authors provide a systematic analysis of potential data contamination, particularly overlap between BASS and the training data of the evaluated baseline models, to ensure a fair comparison?

  2. Could the authors include more recent SOTA models, such as Gemini 3.1 Pro, Gemini 3.8 Flash, Qwen3.5-Omni, and Gemma 4, so that the BASS benchmark remains timely and its conclusions better reflect the current state of audio language models?

2 comments

We thank Reviewer GrV6 for taking the time to review our paper and for recognizing the important gap BASS addresses in evaluating understanding of complete musical recordings. We appreciate the reviewer’s recognition of its complementary task coverage, emphasis on compositional reasoning over time, and diagnostic analyses of current Audio LM capabilities.

Use of existing annotations.

We do not agree that building on existing expert annotations is, in itself, a reason to reject a benchmark. BASS is not presented as a new annotation collection or as an attempt to introduce new musicological concepts. Rather, it transforms reliable, expert-annotated music resources into twelve natural-language evaluation tasks that require Audio LMs to reason over complete recordings. Importantly, these tasks go beyond simply reproducing the original source annotations. For example:

  • Structural Lyrics Transcription requires models to identify musical sections and associate the corresponding lyrics with each section, going beyond flat lyric transcription.
  • Musicological Analysis converts continuous expert ratings into selection, comparison, attribution, and ranking questions about musical attributes, rather than asking models to reproduce the original ratings.
  • Artist Collaboration evaluates artist counting, temporal localization, accumulated performance duration, and attribution of vocal delivery and artist role, including across non-contiguous sections.

Reusing expert-annotated resources is an established and reasonable approach to benchmark construction. The relevant question is whether the resulting benchmark evaluates capabilities that existing evaluations do not jointly assess. BASS does so through its full-song setting, complementary task formulations, and diagnostic analyses. Its contribution lies in this task design, scope, and unified evaluation of Audio LMs—not in creating another set of source annotations.

Potential data contamination

We agree that exposure to the source recordings or annotations during model training cannot be ruled out. However, the training data of the proprietary models are not publicly accessible, so we cannot systematically determine whether particular songs or source datasets appeared in their pretraining corpora. We do not believe that the possibility of unobserved overlap, which applies to many benchmarks built from publicly available material, is sufficient grounds to reject BASS.

We have already conducted a metadata-only diagnostic in which the audio is replaced with the artist name and song title. This experiment shows that metadata can improve lyric transcription, consistent with models leveraging memorized artist–song associations. On the other tasks, performance either decreases or remains similar to the audio condition. We do not interpret this experiment as ruling out contamination; rather, it demonstrates that metadata-based priors can benefit some tasks but do not substitute for audio on tasks that require musical and temporal information.

We will make this distinction explicit and temper any language that could be read as claiming that contamination has been ruled out. We will also clearly disclose the source datasets and recommend that future evaluations report known overlap with their training data.

More recent models.

We agree that evaluating newer models would provide an updated picture of model capabilities. However, BASS already evaluates 14 open-source and frontier multimodal LMs across 2,658 questions, 1,993 songs, and over 138 hours of music. Beyond the main benchmark results, we also conduct extensive diagnostic analyses examining the effects of ASR transcripts, source separation, step-by-step reasoning, metadata, genre, and language. Extending this evaluation to every newly released model would require rerunning a substantial long-form audio evaluation and its associated analyses, incurring considerable computational and API costs. This is particularly relevant for newer systems such as Qwen3.5-Omni, Gemini 3.8 which is not available as an open-weight checkpoint and requires paid API inference.

We believe the 14 models evaluated already provide a substantial comparison across open-source and proprietary systems. While additional baselines would be valuable, continually evaluating every newly released model is not a practical prerequisite for establishing a benchmark. BASS is intended as a reusable evaluation framework: its released questions, annotations, and evaluation protocol allow future work to assess newer models under the same conditions. Our empirical claims are therefore scoped to the models evaluated in this study. The absence of additional, more recent baselines may limit the recency of the leaderboard, but it does not undermine the benchmark's task design, ground-truth construction, or value. We do not believe that evaluating these additional models is a reason to reject the benchmark.

I maintain that the experimental setup should be strengthened in two ways:

  1. At least 1–2 SOTA proprietary models should be added to the experiments, even if paid API access is required.
  2. Among open-source models, Gemma 4 should be used instead of Gemma 3, and SALMONN-2 (audio-visual model) should also be included in the evaluation.
Reviewer_MLurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BASS, a benchmark designed to assess long-form music understanding in audio language models. The benchmark comprises 2,658 questions spanning 12 tasks, including musical structure analysis, lyrics transcription, musicological attribute recognition, and artist collaboration. The authors evaluate 14 audio language models and provide further analyses involving source separation, ASR-assisted transcription, and reasoning modes. The experimental results highlight substantial limitations in current models, particularly in understanding musical structure and reasoning over extended temporal contexts.

Critical strengths

The paper addresses several important yet relatively underexplored aspects of music understanding, especially long-range structural reasoning and artist collaboration. The broad coverage of evaluation tasks, together with experiments on a diverse set of audio language models, offers a useful overview of current model capabilities. The supplementary analyses also provide additional insights into the challenges these models face.

Critical weaknesses
  1. Limited novelty. BASS mainly reformulates existing MIR datasets into QA tasks. The tasks themselves are not new, and it is unclear what additional insights the benchmark offers over existing evaluations.

  2. The genre imbalance limits generalizability, while multilingual transcription may primarily test ASR rather than music understanding. The aggregated score, single-run evaluation, and externally hosted audio also raise concerns about validity and reproducibility.

Questions for authors
  1. How do the authors distinguish music-specific understanding from general language proficiency and ASR capabilities in the multilingual evaluation?

  2. To what extent are the aggregate scores and model rankings affected by alternative task weightings or variability across repeated evaluation runs?

  3. Could the authors explain the apparent discrepancy between Tables 1 and 7? For instance, SALMONN-7B achieves a normalized PGD score of 37.26 in Table 1, whereas Table 7 reports a raw accuracy of only 1.36. These values do not appear to align with the normalization formula described in the paper.

2 comments

We thank Reviewer MLur for taking the time to review our paper and for recognizing that BASS addresses important yet underexplored aspects of music understanding, particularly long-range structural reasoning and artist collaboration. We also appreciate the recognition of the benchmark's broad task coverage and diagnostic analyses.

Novelty and use of existing annotations.

BASS does not claim to introduce new musicological annotations or new definitions of musical concepts. Its contribution is to formulate new natural-language tasks that require Audio LMs to reason over complete recordings, using reliable expert-annotated resources as ground truth.

Importantly, BASS does not simply ask models to reproduce the original source labels. For example:

  • Structural lyric transcription requires identifying musical sections and associating the corresponding lyrics with each section; this goes beyond flat lyric transcription.
  • Musicological Analysis converts continuous expert ratings into comparison, selection, attribution, and ranking questions over musical attributes, rather than asking models to predict the original ratings.
  • Artist Collaboration evaluates artist counting, temporal localization, accumulated performance duration, and attribution of vocal delivery and artist role, including across non-contiguous sections.

The contribution is the benchmark's task design, scope, and diagnostic evaluation of Audio LMs, not the creation of another set of source annotations.

Genre imbalance and multilingual evaluation.

There is no single genre distribution that represents real-world music: genre prevalence varies across regions, platforms, and music catalogs, and is not uniformly distributed. Requiring equal representation across genres would therefore not necessarily make a benchmark more representative. BASS covers eight genres, with Pop (30.3%) and Hip Hop (25.6%) being the most represented. This distribution reflects the coverage of the underlying annotated datasets. We acknowledge that this limits generalization to underrepresented genres, but it does not invalidate the evaluation of the musical capabilities covered by BASS. We also report genre-specific performance to examine how models perform across the included genres.

Regarding multilingual evaluation, we do not claim to isolate music-specific understanding from language proficiency or ASR capabilities. Rather, our lyric transcription tasks evaluate a combination of these capabilities: models must not only recognize lyrics but also identify musical sections and align the lyrics with the corresponding sections of a complete recording. Furthermore, our multilingual evaluation also includes Artist Counting, which does not require lyric transcription. Our results are mixed: performance declines on full-song lyric transcription but improves on section-level transcription and artist counting. These results show how Audio LMs handle different music-understanding tasks across languages, rather than as an isolated measure of music-specific reasoning.

Aggregate scores and evaluation variability.

The aggregate score is intended as a compact summary across the twelve tasks, not as a replacement for per-task evaluation or a universal measure of music understanding. We report the individual task scores precisely because performance varies considerably across capabilities.

We also examined an alternative aggregation that assigns equal weight to each of the four task categories, rather than weighting categories in proportion to their number of tasks. The ordering of the three leading models—Gemini 2.5 Pro, Qwen3-Omni, and Step-Audio-R1—remains unchanged. Some lower-ranked models exchange positions, reinforcing why the individual task results should remain the primary basis for interpreting model capabilities.

Regarding variability, we use greedy decoding for model evaluations. For multiple-choice musicological questions, we use majority voting over four runs to account for answer-option ordering. The ten paraphrased prompts are distributed across examples; they are not ten repeated evaluations of each example. Our reported 95% confidence intervals use song-level clustered bootstrap resampling and therefore quantify uncertainty from song sampling, not run-to-run decoding variability. We will clarify this distinction rather than imply that the intervals capture every source of evaluation variability.

PGD discrepancy between Tables 1 and 7.

The reviewer is correct that the SALMONN-7B PGD score is inconsistent between the two tables. The raw accuracy is 1.36%. Since PGD asks the model to select an unordered pair from eight attributes, random guessing corresponds to one of the ((82)=28)(\binom{8}{2}=28) possible pairs, giving (R=1/28)(R=1/28). Applying the normalization defined in the paper gives:

\frac{1.36-3.57}{100-3.57}\times 100 = -2.29\%.$$ We will correct it and recompute SALMONN-7B's aggregate score from the underlying task scores.

Thank you for the detailed response. I have carefully considered the authors' clarifications and will maintain my score.

Reviewer_gCcWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BASS, a benchmark for evaluating music understanding and reasoning in audio language models. BASS consists of 12 tasks across four categories. The authors evaluate 14 open-source and frontier audio language models and show that even the best-performing model achieves only 26% average performance.

Critical strengths

A key strength is that the paper leverages four existing datasets with music annotations and reformulates them into 12 natural-language reasoning tasks, providing a unified framework for evaluating full-song understanding in Audio LMs.

Critical weaknesses
  • Although the questions were reviewed by music experts to ensure that they can be answered from the audio alone, no human performance as upper bound is reported. Human performance would help confirm that these questions can actually be answered reliably from the audio.
  • The paper presents several interesting findings. It would be helpful to provide a clearer summary of the key findings, especially the new insights obtained through BASS beyond what has already been reported in prior work.
Questions for authors

Minor typos/editing errors remain, e.g., “from the from” (L.111), “on musicological analysis:” (L.148), and “Gemma 3 Gemma” (L.227).

1 comment

We thank Reviewer gCcW for taking the time to review our paper and for recognizing the value of BASS as a unified framework for evaluating full-song understanding in Audio LMs through natural-language reasoning tasks.

“Human performance would help confirm that these questions can actually be answered reliably from the audio”

  • A human performance baseline would provide a useful reference, but its absence does not make the questions unanswerable or invalidate their ground truth. BASS is built on expert-annotated datasets, and all questions were written by the authors with guidance from music-expert co-authors. Answers were derived from the source annotations using the rules described in the paper. Music-expert co-authors reviewed every question for ambiguity, answerability from the audio alone, and consistency with its source annotation; questions failing any check were revised or removed. While a human-performance estimate would be a useful additional comparison, we do not believe its absence undermines the benchmark's validity or constitutes grounds for rejecting the work.

“Key findings and insights.” BASS investigates several aspects of long-form music understanding that, to our knowledge, have not been systematically studied in existing music benchmarks. Our findings are relevant to music practitioners, developers of music AI systems, and researchers, as they highlight the capabilities current models can support, where they remain unreliable, and how common modeling choices affect their performance. Section 4 is organized around research questions concerning musical structure and lyrics, artist collaboration, and musicological understanding, alongside cross-cutting analyses of preprocessing, reasoning modes, metadata, genre, and multilinguality.

Our main findings are:

  • Structural and lyrical understanding. Understanding how a song is organized and how lyrics relate to its sections is fundamental to music perception. We find that models perform better at segmenting an entire song than locating a specific section type, with an average difference of 7.63 IoU points. However, performance declines for later sections, with frequent confusion between verses and choruses. We also find that adding Whisper transcripts reduces structural lyric transcription performance for three of the four tested models. We suspect this is partly due to transcription errors being propagated to the downstream model, as well as the lack of structural information in the flat transcript. Interestingly, providing only artist and song metadata can improve lyric transcription performance. This suggests that models can exploit memorized song–lyric associations but struggle to ground lyrics in the audio and align them with the song's structure.
  • Artist collaboration. Identifying who performs, when they perform, how long they perform, and how they perform is important for understanding multi-artist music which is quite prevalent in the real world. We find that artist duration and localization are particularly challenging, with all models achieving below 10% accuracy on both tasks. Using diarization tools improves Gemini 2.5 Pro's artist localization accuracy from 3.12% to 11.48%, but reduces artist attribution accuracy from 56.66% to 18.09%. This suggests that diarization helps identify when a performer enters but does not necessarily establish who is performing or their vocal delivery.
  • Musicological understanding. Recognizing and comparing musical attributes such as instrumentation, rhythm, and vocal characteristics is central to music understanding. While models perform relatively well at identifying a dominant attribute, they struggle to compare multiple attributes and rank their prominence. Sonority and Rhythm are particularly challenging across models. Performance also varies across genres, with models performing relatively better on Country and Rock than Pop.
  • Music specialization and modeling interventions. Music-specialized training does not necessarily translate to stronger long-form performance: Music Flamingo (12.50 average) scores below the general-purpose Qwen3-Omni (23.61). While previous studies have reported benefits from vocal source separation for certain music-understanding tasks, we find that replacing mixed recordings with isolated vocal stems generally degrades performance across BASS task categories. Similarly, enabling step-by-step reasoning for Qwen3-Omni improves six of twelve tasks, including Artist Attribution (+24.53 points) and Gene Attribution (+17.97), but reduces Full Lyrics Transcription performance (−12.77).. Our multilingual analysis also reveals different trends across tasks: performance declines on full-song lyric transcription but improves on section-level transcription and artist counting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.