acceptodds
Under review as a conference paper at ICLR 2027

PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Abstract

Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. On PianoBench, none of the open-weight audio-language models we evaluate significantly outperform a no-audio control, while proprietary frontier models show mixed gains from audio. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between reasoning about musical events in symbolic form and perceiving them from audio. To investigate this further, we mid-train Qwen3.5-9B-Base on our interleaved dataset to improve use of music transcripts without chain of thought, raising accuracy on PianoBench from 42.8% to 56% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Supplement: https://anon834910.github.io/anon-dataset-demo/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
music, benchmark, dataset, piano, understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BrR6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PianoBench, a benchmark of 250 questions evaluating musical perception and reasoning across audio and MIDI-text representations, and PianoTalk, a dataset containing over 20,000 hours of instructional music transcriptions. Experiments reveal a performance gap between audio-based and symbolic music understanding. The authors further demonstrate that interleaved speech-MIDI training improves symbolic music reasoning compared to separated training.

Critical strengths
  1. The paper introduces PianoTalk, a large-scale music-language dataset comprising over 20,000 hours of interleaved speech and piano transcriptions from instructional videos.
  2. The evaluation covers diverse open-weight and proprietary models, revealing interesting gaps in musical understanding.
Critical weaknesses
  1. PianoBench exhibits substantial language and answer-choice biases, with no-music accuracy ranging from 41.2% to 63.6%, considerably above the 25% random baseline. This suggests that many questions can be answered without actually understanding the musical content. Although the authors report improvements over the no-music control, these biases still undermine the benchmark's validity as a measure of genuine musical understanding.

  2. The paper attributes the performance gap between audio and MIDI-text inputs primarily to limitations in musical perception. However, these input formats differ substantially in representation and information accessibility, making it difficult to isolate perception from other factors. Consequently, the observed performance differences do not conclusively establish that audio perception is the primary bottleneck.

  3. Although the paper identifies musical perception from audio as a major limitation of existing models, the proposed interleaved training approach is evaluated exclusively on MIDI-text inputs. None of the trained Qwen3.5-9B models are tested with audio inputs. Therefore, the experiments demonstrate improvements in symbolic music reasoning but do not establish whether these improvements transfer to actual audio-based music understanding.

Questions for authors

See weaknesses above

Reviewer_riJEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PianoTalk and PianoBench, studying the perception and reasoning abilities of language models regarding music. PianoTalk is derived from a large-scale collection of piano teaching videos. The authors used source separation, speech recognition, and automatic piano transcription to convert the instructional videos into interleaved explanatory text and MIDI note sequences. The dataset contains 138,212 videos, with segments of at least one modality totaling over 26,000 hours. PianoBench selected 162 videos from the test data to create 250 manually reviewed music multiple-choice questions. Each question was tested using original piano audio, synthesized piano audio, and MIDI-text input, with a control condition where no music was provided. The experiments show that many models have significantly higher accuracy when using MIDI-text compared to real audio, indicating that the symbolic music reasoning ability of the models has not yet fully translated into direct audio understanding ability. The author further uses PianoTalk for continuous pre-training of Qwen3.5-9B-Base, followed by instruction fine-tuning. The results show that the model retaining the interleaved structure of explanatory text and music segments achieved an accuracy of 56.0%, while the model separating the two types of content only reached 42.4%. The core contribution of the paper is the proposal of a data construction method that uses natural teaching videos to study music comprehension abilities, and through experiments, it demonstrates the importance of music representation forms and the text-music interleaved training structure.

Critical strengths

1: By using different input forms of the same musical content, a clearer distinction can be made between music perception and music reasoning abilities. An important design feature of PianoBench is that the same musical segment can be represented as raw audio, standardized synthesized audio, and MIDI-text. This design can help analyze whether the model's errors stem from understanding the sound signal or from reasoning about known musical events. Additionally, it is also important not to provide control conditions for the music, as the multiple-choice questions may contain linguistic information that can directly infer the answers. Table 1 of the paper shows that the MIDI-text accuracy of some models is significantly higher than the audio accuracy; for example, the MIDI-text accuracy of Gemini 3.1 Pro is 89.2%, while the original piano audio is 80.4%. This experimental design is more diagnostically valuable than simply reporting music question-answering accuracy.

2: The training experiment with interleaved teaching data provides meaningful research findings. The author not only constructed the dataset but also studied how to use it to improve the model's understanding of symbolic music. Especially, the interleaved training and separated training use the same speech and MIDI content, comparing by adjusting the arrangement and organization methods. Table 2 shows that the MIDI-text accuracy of the interleaved model reached 56.0%, significantly higher than the 42.4% of the separated model. This result supports a valuable research hypothesis: preserving the temporal relationship between explanation and demonstration in natural teaching processes helps the model learn the connections between musical events and language descriptions. Compared to simply increasing the amount of music text data, this research direction has a certain methodological inspiration.

Critical weaknesses

1: The benchmark has a noticeable answer prior, affecting its effectiveness as a tool for assessing musical perception ability. Basis: Section 5.2, Table 1. Each question in ianoBench includes four options, with a 25% accuracy rate for random guessing. However, in Table 1, the accuracy rate has already reached 41.2%–63.6% without any control experiments involving music. This means that many questions can be answered correctly through language knowledge, option distribution, or question phrasing, without the need to actually understand the music clips. Although the authors have already recognized this issue and used the performance gain relative to a no-music control condition to explain the experimental results, this approach can only mitigate the problem and cannot completely eliminate the bias inherent in the dataset itself. For example, even if a model has some understanding of audio, its final performance may still be influenced by the difficulty of the questions, the design of the options, and language priors. In addition, the Benchmark has only 250 questions in total. For fine-grained comparisons across multiple models and input modalities, such a sample size may render smaller performance differences insufficiently stable. Therefore, I believe the current benchmark is more suitable as a preliminary diagnostic tool for music understanding capabilities, and is not yet sufficient as a basis for highly reliable model capability rankings.

2: Automatic MIDI transcription errors may affect the fair comparison between audio and symbolic input, and the evidence of training effectiveness is currently limited. The comparison between audio and MIDI-text is based on an important premise: both inputs should express fundamentally consistent musical content. But MIDI-text is generated by an automatic piano transcription system, so it inevitably contains errors in pitch, rhythm, note duration, and polyphonic separation. The author mentions in Section 5.3 that among the errors in GPT 6 Astra, five involved transcription errors that were not detected during the manual review phase, and these samples were still retained in the final evaluation. This indicates that the current data review process cannot fully ensure semantic consistency between different modalities. For a paper with the core conclusion of "the gap between music perception and reasoning," this point is very important. On the other hand, interleaved training is currently validated mainly through a Qwen3.5-9B model and a 250-question benchmark. Although the gap between 56.0% and 42.4% is significant, it remains unclear whether this benefit can be consistently applied to other models, other music datasets, or more complex music reasoning tasks.

Questions for authors

Question 1: Can we further reduce the prior for non-music answers in the Benchmark? Currently, multiple models achieve an accuracy of 50% or even above 60% when music is not provided. It is recommended to construct a stricter subset, filtering out items that are easily guessed in the no-music control experiments across multiple different models, and to re-report the performance differences between audio and MIDI-text. If the main conclusions still hold for this subset, it will significantly enhance the credibility of the benchmark.

Question 2: Is it possible to provide a fully manually verified MIDI alignment subset? Since the core conclusions of the paper rely on the comparison of different music representations, can the authors perform note-level or at least music-semantic-level manual verification on a high-quality subset? At the same time, we hope the authors will report whether the performance gap between audio and MIDI-text changes after removing samples with transcription errors.

Question 3: Can the advantages of interleaved training be replicated in other music evaluation tasks? Currently, the main improvement in Table 2 is validated on PianoBench. Can we further use existing symbolic music understanding datasets or independently constructed artificial music question-answering test sets to verify whether interleaved training improves general music reasoning ability, rather than primarily adapting to the data distribution and question format of PianoBench?

Question 4: Can you increase the number of random seeds and the scale of the training data for the ablation experiments? Interleaved training improved from 42.4% to 56.0%, but the current results lack sufficient statistical stability analysis. It is recommended to at least report the mean and standard deviation of multiple trainings, and further compare the effects of different data scales and different interleaving granularities. This helps determine whether the performance improvement indeed comes from the temporal relationship between text and music, rather than from single training differences or specific sequence construction methods.

Question 5: Does the improvement from interleaved training come from temporal alignment information, or from the co-occurrence of the two modalities in the same context? Current separate training splits speech and music segments into different documents, while interleaved training keeps them in the same document. It is recommended to add a control experiment that keeps speech and music within the same document but randomly shuffles the correspondence or order of the segments. This experiment will more directly verify the contribution of the time correspondence itself.

Reviewer_M456OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds PianoTalk, a dataset of 138,212 instructional piano videos (more than 20,000 hours) converted into interleaved, timestamped speech transcripts and piano MIDI transcriptions, the latter serialized as a compact text format ("MIDI-text"). The pipeline uses an LLM-guided crawl, piano classification, source separation, automatic music transcription and ASR, with channel-disjoint train/validation/test splits. From the test split it constructs PianoBench, 250 multiple-choice questions over 2–20 s passages from 162 videos. Questions are generated by an LLM from the teacher's speech and coarse note counts, distractors are ranked by a text-only model, and an author verifies the items by listening. Each question is posed with the source-separated stem, a Pianoteq rendering of the transcription, MIDI-text, and no music. Open-weight audio LMs gain little over the no-music control; proprietary models gain more, but every frontier audio model scores higher with MIDI-text than with audio, and the best text result is 93.2% (GPT 6 Astra).

Critical strengths
  1. A new, scalable source of aligned supervision. Instructional videos place commentary next to the passage it describes; the paper turns this into 20,000+ hours of interleaved text with a documented pipeline, channel-disjoint splits and a measured interleaving score (Appendix A.3).
  2. Matched representations with a reported no-music control. Every model is evaluated on the same items with stem audio, synthesized audio, MIDI-text and no music, so gains are read within model rather than against an assumed chance level.
  3. A controlled mid-training experiment. Interleaved and separated conditions use identical chunks and token counts (2,987M vs 2,988M), with speech-only and SFT-only controls and an instruction-tuning stage that contains no music data.
Critical weaknesses
  1. The benchmark is small and no uncertainty is reported.
  2. Verification by one author leaves known label errors in the results. Of GPT 6 Astra's 17 errors, five involve transcription errors that verification missed (Table 4), and these items remain in every reported accuracy. For the MIDI-text condition the transcription is the input, so those items are wrong by construction. A second, independent musician should verify all items, agreement should be reported, and results should be given with the flagged items removed.
  3. The perception-versus-reasoning framing is too clean. No-music accuracy is 41–64% against 25% chance, so substantial text cues remain despite distractor filtering with a single 12B scorer. Some question types (chord quality, note or voice counts) become nearly arithmetic once pitches are listed explicitly, so a MIDI-text advantage need not reflect "reasoning" in the sense the paper intends.
  4. The mid-training evaluation is in-distribution and narrow. PianoBench questions are generated from the same kind of teacher commentary that the interleaved condition pairs with MIDI, so the gain may reflect learning this corpus's commentary-to-pattern vocabulary rather than transferable symbolic reasoning. The result should be checked on an external symbolic-music benchmark and on general benchmarks to measure capability retention after mid-training. Each condition is a single run.
  5. Ground truth derives from speech that may not match the clip. One annotator judged audibility. The paper should report how many generated questions were rejected and for which reasons, as an estimate of residual error in the retained set.
Questions for authors
  1. Can you report video-clustered 95% intervals for Table 1 and Table 2, and paired tests for the key comparisons?
  2. Can a second, independent annotator with musical training verify all 250 items, with agreement reported? What are the results with the flagged items excluded?
  3. How do the perception and reasoning gaps break down by question category, and what is the no-music accuracy per category?
  4. Does the mid-trained model improve on an external symbolic-music benchmark, and how does it perform on general benchmarks (e.g., MMLU) relative to Base-SFT?
  5. Of the candidate questions generated, how many were rejected during verification, and for what reasons?
  6. Stem and Pianoteq accuracies are close for most models. Does this indicate that audio models are not limited by source-separation artifacts, or that both conditions are limited by the same perceptual bottleneck?
  7. What licensing or terms-of-service analysis supports releasing speech transcripts and MIDI derived from 138k YouTube videos? Is there an opt-out mechanism for channel owners?

Comments on the paper

1 comment

Thank you for the response. I maintain my rating of 2 because the evidence for training benefits remains limited, with gains shown only on PianoBench using MIDI-text inputs. Given the benchmark’s focus on musical perception, I consider validation of these gains with audio inputs necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.