1. Lack of demonstrated relevance to real-world musical practice.
This is a fundamental concern about the benchmark's construct validity: verifying the answers to a collection of formally defined questions does not establish that the collection provides a valid or scientifically meaningful operationalization of the broader score-reading and reasoning capabilities suggested by the paper's title. The benchmark predominantly evaluates elementary notation retrieval and deterministic operations over very short musical contexts (a median of four measures), while the information actually needed to answer most questions appears substantially smaller. On examining the examples provided in the paper, I find the questions overly mechanical and narrowly specified, with little indication of how they could be useful in human musical practice. The paper does not establish why these particular questions are meaningful or useful in realistic musical activities.
While I appreciate the use of ABRSM music-theory examinations as motivation, the claimed grounding is unconvincing. Actual ABRSM assessments require students to read, construct, compare, and interpret conventional graphical staff notation, often involving broader musical context. Only a narrow subset overlaps with ScoreQA even at the level of question topics, and there is no demonstrated equivalence in the capabilities being assessed. ScoreQA instead operates on ABC text, where note names and register are explicitly encoded, and uses format-specific constructs such as V:1 and numerical offsets, which bear little to no resemblance to the modality and question formats of the ABRSM examinations used as motivation.
The paper provides no human evaluation of whether its benchmark questions are relevant to musical practice. The Independent Human Review in Appendix A.3 assesses whether the questions can be answered correctly, not whether they are meaningful, useful, or representative of actual musical activities. The authors do not establish why these are the right questions to ask.
2. Lack of relevance to contemporary AI capabilities and future research progress.
Even if the questions are accepted as legitimate tests of foundational ABC notation processing, the benchmark is already nearly saturated by frontier LLMs upon introduction. The evaluated commercial models achieve 92.8%–99.9% accuracy, with the strongest reaching 99.6% complete-quartet correctness.
This raises serious questions about what unresolved research challenge the benchmark identifies and how it could meaningfully track future advances. The main empirical finding appears to be that modern frontier LLMs are already highly proficient at these elementary ABC notation operations.
The experimental results also leave the broader scientific takeaway unclear. The open-weight models perform substantially worse, but they are evaluated under a different inference protocol, including restricted next-token scoring rather than generated answers, and different prompting and reasoning settings. Consequently, the large performance gap between the open-weight and API models cannot be attributed solely to differences in model capabilities. The adaptation experiments demonstrate that smaller models can improve substantially on ScoreQA's own task distribution, but do not establish whether these improvements transfer to broader musical competencies or meaningful downstream tasks. It remains unclear what important scientific conclusion these experiments support beyond characterizing model performance and learnability on the benchmark's narrowly defined operations.
A potentially more meaningful contribution would be to use the proposed framework as a source of automatically verifiable training signals and demonstrate that models trained on ScoreQA improve on independent benchmarks with clearer relevance to real-world musical tasks. Such transfer would provide evidence of the framework's scientific utility without necessarily requiring human validation of ScoreQA's questions as meaningful standalone evaluation tasks. However, the current paper provides no evidence of such transfer, leaving the demonstrated benefit limited to improving performance on ScoreQA itself, a benchmark already saturated by contemporary frontier LLMs.
Dear reviewers,
As the discussion period approached the end, please take a moment to read the authors’ responses and share your thoughts on any remaining points of discussion. If the discussion has changed your assessment, please update your review accordingly.
Thank you for your time and contributions!