- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PianoBench, a benchmark of 250 questions evaluating musical perception and reasoning across audio and MIDI-text representations, and PianoTalk, a dataset containing over 20,000 hours of instructional music transcriptions. Experiments reveal a performance gap between audio-based and symbolic music understanding. The authors further demonstrate that interleaved speech-MIDI training improves symbolic music reasoning compared to separated training.
Critical strengths
- The paper introduces PianoTalk, a large-scale music-language dataset comprising over 20,000 hours of interleaved speech and piano transcriptions from instructional videos.
- The evaluation covers diverse open-weight and proprietary models, revealing interesting gaps in musical understanding.
Critical weaknesses
-
PianoBench exhibits substantial language and answer-choice biases, with no-music accuracy ranging from 41.2% to 63.6%, considerably above the 25% random baseline. This suggests that many questions can be answered without actually understanding the musical content. Although the authors report improvements over the no-music control, these biases still undermine the benchmark's validity as a measure of genuine musical understanding.
-
The paper attributes the performance gap between audio and MIDI-text inputs primarily to limitations in musical perception. However, these input formats differ substantially in representation and information accessibility, making it difficult to isolate perception from other factors. Consequently, the observed performance differences do not conclusively establish that audio perception is the primary bottleneck.
-
Although the paper identifies musical perception from audio as a major limitation of existing models, the proposed interleaved training approach is evaluated exclusively on MIDI-text inputs. None of the trained Qwen3.5-9B models are tested with audio inputs. Therefore, the experiments demonstrate improvements in symbolic music reasoning but do not establish whether these improvements transfer to actual audio-based music understanding.
Questions for authors
See weaknesses above
Thank you for the response. I maintain my rating of 2 because the evidence for training benefits remains limited, with gains shown only on PianoBench using MIDI-text inputs. Given the benchmark’s focus on musical perception, I consider validation of these gains with audio inputs necessary.