- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents BASS, a benchmark for testing whether audio language models can reason over complete musical recordings rather than short excerpts. It covers 2,658 questions from 1,993 songs and approximately 138 hours of music, organized into 12 tasks spanning structural segmentation, structure-aware lyric transcription, musicological analysis, and multi-artist collaboration. The benchmark’s main value is its emphasis on long-range temporal and compositional reasoning, with a evaluation of 14 open-source models. The results are informative but also reveal a substantial capability gap: models perform relatively better on lyric transcription, where linguistic or memorized priors may help, while struggling with musical structure, artist attribution, timing, and higher-level musicological reasoning. However, BASS relies substantially on existing annotations, and the paper leaves unresolved questions about overlap with baseline training data and the fairness of the evaluation. In addition, several newer SOTA models are not included. Thus, while BASS is promising, its claims about the current capability of audio language models should be supported by stronger contamination analyses and more timely baselines.
Critical strengths
-
The paper addresses an important and timely gap: many existing music benchmarks emphasize short excerpts, captions, or shallow recognition, while practical applications such as retrieval, recommendation, and metadata generation often require reasoning over complete recordings and their long-range structure.
-
BASS provides unusually broad and complementary task coverage. Its combination of temporal segmentation, structure-aware lyric transcription, musicological comparison/ranking, and multi-artist attribution evaluates capabilities that go substantially beyond standard audio captioning or generic music question answering.
-
Several tasks require genuine compositional reasoning over time rather than local acoustic recognition. In particular, artist duration, artist localization, section-conditioned attribution, and structure-conditioned transcription require models to integrate information across repeated, non-contiguous, and hierarchically organized parts of a song.
-
The empirical analysis is diagnostically valuable and reasonably transparent. The paper examines performance across musical sections, genres, languages, vocal styles, metadata conditions, reasoning modes, and audio preprocessing settings, providing useful evidence about current model limitations. Its song-based bootstrap procedure, raw results, task prompts, and detailed appendix statistics also make the findings easier to inspect and build upon.
Critical weaknesses
-
The benchmark is substantially inherited from existing annotations. Although the questions are newly phrased, their ground truth is derived from Harmonix, Genius, MGPHot, and CoSoD. The paper therefore needs stronger evidence that BASS provides independent evaluation rather than merely a new natural-language descriptions to known datasets.
-
Data contamination, label reliability, and external validity are not sufficiently established. The data of BASS may have appeared in baselines' pretraining. The paper should analyze the impact of data contamination as much as possible. In the era of LLMs, although it is impossible to guarantee that evaluation data has not been used for training, we can strive to ensure the fairness of evaluation.
-
The baseline models may already be outdated. The evaluation omits newer systems such as Gemini 3.1 Pro, Gemini 3.8 Flash, Qwen3.5-Omni, and Gemma 4, so the reported performance gap may not reflect the current state of audio language models. Including more SOTA models would make the paper’s claims more timely and informative.
Questions for authors
-
Could the authors provide a systematic analysis of potential data contamination, particularly overlap between BASS and the training data of the evaluated baseline models, to ensure a fair comparison?
-
Could the authors include more recent SOTA models, such as Gemini 3.1 Pro, Gemini 3.8 Flash, Qwen3.5-Omni, and Gemma 4, so that the BASS benchmark remains timely and its conclusions better reflect the current state of audio language models?
2 comments
I maintain that the experimental setup should be strengthened in two ways:
- At least 1–2 SOTA proprietary models should be added to the experiments, even if paid API access is required.
- Among open-source models, Gemma 4 should be used instead of Gemma 3, and SALMONN-2 (audio-visual model) should also be included in the evaluation.
We thank Reviewer GrV6 for taking the time to review our paper and for recognizing the important gap BASS addresses in evaluating understanding of complete musical recordings. We appreciate the reviewer’s recognition of its complementary task coverage, emphasis on compositional reasoning over time, and diagnostic analyses of current Audio LM capabilities.
Use of existing annotations.
We do not agree that building on existing expert annotations is, in itself, a reason to reject a benchmark. BASS is not presented as a new annotation collection or as an attempt to introduce new musicological concepts. Rather, it transforms reliable, expert-annotated music resources into twelve natural-language evaluation tasks that require Audio LMs to reason over complete recordings. Importantly, these tasks go beyond simply reproducing the original source annotations. For example:
Reusing expert-annotated resources is an established and reasonable approach to benchmark construction. The relevant question is whether the resulting benchmark evaluates capabilities that existing evaluations do not jointly assess. BASS does so through its full-song setting, complementary task formulations, and diagnostic analyses. Its contribution lies in this task design, scope, and unified evaluation of Audio LMs—not in creating another set of source annotations.
Potential data contamination
We agree that exposure to the source recordings or annotations during model training cannot be ruled out. However, the training data of the proprietary models are not publicly accessible, so we cannot systematically determine whether particular songs or source datasets appeared in their pretraining corpora. We do not believe that the possibility of unobserved overlap, which applies to many benchmarks built from publicly available material, is sufficient grounds to reject BASS.
We have already conducted a metadata-only diagnostic in which the audio is replaced with the artist name and song title. This experiment shows that metadata can improve lyric transcription, consistent with models leveraging memorized artist–song associations. On the other tasks, performance either decreases or remains similar to the audio condition. We do not interpret this experiment as ruling out contamination; rather, it demonstrates that metadata-based priors can benefit some tasks but do not substitute for audio on tasks that require musical and temporal information.
We will make this distinction explicit and temper any language that could be read as claiming that contamination has been ruled out. We will also clearly disclose the source datasets and recommend that future evaluations report known overlap with their training data.
More recent models.
We agree that evaluating newer models would provide an updated picture of model capabilities. However, BASS already evaluates 14 open-source and frontier multimodal LMs across 2,658 questions, 1,993 songs, and over 138 hours of music. Beyond the main benchmark results, we also conduct extensive diagnostic analyses examining the effects of ASR transcripts, source separation, step-by-step reasoning, metadata, genre, and language. Extending this evaluation to every newly released model would require rerunning a substantial long-form audio evaluation and its associated analyses, incurring considerable computational and API costs. This is particularly relevant for newer systems such as Qwen3.5-Omni, Gemini 3.8 which is not available as an open-weight checkpoint and requires paid API inference.
We believe the 14 models evaluated already provide a substantial comparison across open-source and proprietary systems. While additional baselines would be valuable, continually evaluating every newly released model is not a practical prerequisite for establishing a benchmark. BASS is intended as a reusable evaluation framework: its released questions, annotations, and evaluation protocol allow future work to assess newer models under the same conditions. Our empirical claims are therefore scoped to the models evaluated in this study. The absence of additional, more recent baselines may limit the recency of the leaderboard, but it does not undermine the benchmark's task design, ground-truth construction, or value. We do not believe that evaluating these additional models is a reason to reject the benchmark.