acceptodds
Under review as a conference paper at ICLR 2027

ScoreQA: A Score-Grounded Benchmark for Reading and Reasoning over Symbolic Scores

Abstract

Evaluating score-reading ability requires questions whose answers are determined by the supplied notation and explicit musical conventions. ScoreQA assesses defined Reading and Reasoning abilities on short ABC 2.1 score windows. Reading identifies events, notation attributes, and timing; Reasoning compares or transforms this information. A MusicXML-based oracle verifies answers and their evidence in ABC. Each quartet fixes one question and four choices across four valid windows supporting different answers. ARJQ describes item, paired-score, and complete-quartet correctness, together with obsolete-answer retention. Seven open-weight backbones evaluated with plain-text five-shot restricted next-token scoring achieve no complete Reasoning quartets. In a separate five-shot API evaluation with generated answer letters, five models achieve 92.8% to 99.9% Accuracy and 76.8% to 99.6% Quartet, including 72.0% to 100.0% Reasoning Quartet. Results are reported separately for the two prompting and inference settings. Supervised adaptation provides a supplementary use of the training split, raising Qwen3.5-9B’s Reading and Reasoning Quartet to 84.0% and 44.0%. ScoreQA provides a verifiable test of specified score-reading and analysis abilities, for which complete correctness is an intended outcome.

Primary area:
datasets and benchmarks
Keywords:
Music Understanding, Symbolic Music, Benchmark Datasets, Model Evaluation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zLKBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ScoreQA evaluates foundational reading and reasoning over symbolic scores using ABC notation and MusicXML-based reference answers. Questions are organized into quartets sharing the same question and choices across four scores with different correct answers. The paper evaluates item and group correctness, score-dependent answer changes, and supervised adaptation under separate open-weight and API evaluation protocols.

Critical strengths
  1. Programmatic answer verification and explicit musical conventions reduce ambiguity and make reference answers traceable to the supplied notation.

  2. The quartet design controls question-only shortcuts and tests whether models consistently distinguish scores supporting different answers.

Critical weaknesses
  1. Four API models exceed 97% overall accuracy, and the strongest reaches 99.9% overall accuracy and 100% Reasoning Accuracy and Quartet. Although useful for foundational assessment, the benchmark offers little remaining headroom for the strongest evaluated system. Its attainment-oriented positioning does not sufficiently establish lasting value for frontier-model research.

  2. Tasks mainly involve local pitch comparisons, interval calculations, and transposition over short windows. This limits coverage of broader musical reasoning, including harmony, voice leading, and structural analysis. Pooling all reasoning operations into one category further reduces diagnostic granularity.

Questions for authors
  1. What substantive research questions can the current benchmark still help investigate once the strongest evaluated system nearly saturates its metrics?

  2. Could you provide the complete Reasoning operation inventory and per-operation test counts and results, to clarify task diversity and remaining capability gaps?

Reviewer_cAphOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ScoreQA introduces a multiple-choice question-answering benchmark over short musical scores represented in ABC notation. The provided scores contain a median of four measures, and the questions involve basic notation retrieval and musical calculations, with answers verified using a MusicXML-based oracle. The main empirical finding is that frontier models achieve near-perfect accuracy on these surface-level questions over short musical excerpts, suggesting that they are highly proficient at the tested ABC notation operations.

The paper does not provide a human review of the benchmark questions themselves in terms of their relevance to human musical practice. Although Appendix A.3 contains a section titled Independent Human Review, this evaluates whether the questions have identifiable, correct answers. Two reviewers with musical backgrounds answered 96 questions, showing high agreement with the reference answers. They were not asked to assess the relevance, usefulness, or authenticity of the questions as musical tasks.

The authors instead motivate the benchmark through limited topical overlap with elementary ABRSM music-theory curricula, without independently establishing whether the resulting ABC-based questions correspond to actual ABRSM assessments or meaningful musical practice.

Critical strengths

The paper presents a rigorous experimental evaluation, including confidence intervals and careful answer-verification procedures, which strengthens confidence in the reported results. I also appreciate the authors' effort to motivate the benchmark's topic selection through an established human music-theory examination framework (ABRSM), rather than relying solely on arbitrary task definitions. Although I find the connection insufficiently established at the level of actual benchmark questions, the attempt to provide a domain-grounded motivation is a positive aspect of the work.

Critical weaknesses

1. Lack of demonstrated relevance to real-world musical practice.

This is a fundamental concern about the benchmark's construct validity: verifying the answers to a collection of formally defined questions does not establish that the collection provides a valid or scientifically meaningful operationalization of the broader score-reading and reasoning capabilities suggested by the paper's title. The benchmark predominantly evaluates elementary notation retrieval and deterministic operations over very short musical contexts (a median of four measures), while the information actually needed to answer most questions appears substantially smaller. On examining the examples provided in the paper, I find the questions overly mechanical and narrowly specified, with little indication of how they could be useful in human musical practice. The paper does not establish why these particular questions are meaningful or useful in realistic musical activities.

While I appreciate the use of ABRSM music-theory examinations as motivation, the claimed grounding is unconvincing. Actual ABRSM assessments require students to read, construct, compare, and interpret conventional graphical staff notation, often involving broader musical context. Only a narrow subset overlaps with ScoreQA even at the level of question topics, and there is no demonstrated equivalence in the capabilities being assessed. ScoreQA instead operates on ABC text, where note names and register are explicitly encoded, and uses format-specific constructs such as V:1 and numerical offsets, which bear little to no resemblance to the modality and question formats of the ABRSM examinations used as motivation.

The paper provides no human evaluation of whether its benchmark questions are relevant to musical practice. The Independent Human Review in Appendix A.3 assesses whether the questions can be answered correctly, not whether they are meaningful, useful, or representative of actual musical activities. The authors do not establish why these are the right questions to ask.

2. Lack of relevance to contemporary AI capabilities and future research progress.

Even if the questions are accepted as legitimate tests of foundational ABC notation processing, the benchmark is already nearly saturated by frontier LLMs upon introduction. The evaluated commercial models achieve 92.8%–99.9% accuracy, with the strongest reaching 99.6% complete-quartet correctness.

This raises serious questions about what unresolved research challenge the benchmark identifies and how it could meaningfully track future advances. The main empirical finding appears to be that modern frontier LLMs are already highly proficient at these elementary ABC notation operations.

The experimental results also leave the broader scientific takeaway unclear. The open-weight models perform substantially worse, but they are evaluated under a different inference protocol, including restricted next-token scoring rather than generated answers, and different prompting and reasoning settings. Consequently, the large performance gap between the open-weight and API models cannot be attributed solely to differences in model capabilities. The adaptation experiments demonstrate that smaller models can improve substantially on ScoreQA's own task distribution, but do not establish whether these improvements transfer to broader musical competencies or meaningful downstream tasks. It remains unclear what important scientific conclusion these experiments support beyond characterizing model performance and learnability on the benchmark's narrowly defined operations.

A potentially more meaningful contribution would be to use the proposed framework as a source of automatically verifiable training signals and demonstrate that models trained on ScoreQA improve on independent benchmarks with clearer relevance to real-world musical tasks. Such transfer would provide evidence of the framework's scientific utility without necessarily requiring human validation of ScoreQA's questions as meaningful standalone evaluation tasks. However, the current paper provides no evidence of such transfer, leaving the demonstrated benefit limited to improving performance on ScoreQA itself, a benchmark already saturated by contemporary frontier LLMs.

Questions for authors
  1. Representation and modality. How would the results change if the same musical content were presented as graphical staff notation, MusicXML, or kern rather than ABC? What evidence establishes that ScoreQA measures transferable musical understanding rather than proficiency in parsing a particular textual notation format?

  2. Relevance to musical practice. What evidence establishes that the benchmark questions reflect meaningful information needs or competencies in real-world human or agentic musical practice? In particular, how do the questions correspond to actual ABRSM examination tasks in terms of assessed capabilities and representation, beyond topical overlap? Has any human evaluation assessed the relevance of the questions themselves, rather than merely verifying their answers?

  3. Role of musical context. How much of the supplied musical context is actually necessary to answer the questions? Have the authors evaluated whether performance depends on integrating information across the score, rather than retrieving a few localized symbols? How would the findings change for questions requiring reasoning over longer musical passages?

  4. Relevance to future AI progress. Given the near-perfect performance of frontier LLMs, what important unresolved capability does ScoreQA measure, and what evidence supports its usefulness for tracking future progress or predicting performance on meaningful downstream musical tasks?

  5. Scientific contribution of the quartet construction. What is the incremental scientific value of constructing these specific fixed-question quartets? Appendix B.3 shows that actual Quartet performance is generally close to random regrouping baselines, with differences varying in direction across models. What distinctive model capability or failure mode does the proposed quartet construction reveal that cannot be captured by simpler item-level evaluations? What empirical evidence justifies presenting this construction as a principal contribution of a scientific paper?

  6. External transfer of ScoreQA adaptation. Appendix C.5 reports evaluations on external subsets of SSMR-Bench and MusICA-MetaBench using only the unadapted Qwen3.5-9B backbone. Why are the corresponding results for the ScoreQA-adapted model not reported? Do the substantial improvements observed after ScoreQA adaptation transfer to these external subsets? More importantly, can the authors demonstrate improvements on independent musical benchmarks involving tasks beyond those closely overlapping with ScoreQA's own operations? Such evidence would provide more convincing support for the framework's scientific utility beyond ABC parsing fluency.

Reviewer_tCAaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScoreQA, a benchmark for symbolic music score reading and reasoning, featuring automatically generated questions with verifiable answers computed by a MusicXML-based oracle and a score-grounded quartet design evaluated using the proposed ARJQ metrics. Experiments across open-weight and API models reveal substantial performance differences across evaluation settings, while supervised LoRA adaptation significantly improves score-reading and reasoning accuracy.

Critical strengths
  • The benchmark is carefully constructed, with deterministic MusicXML-based answer verification, consistency checks between MusicXML and ABC notation, and source-disjoint data splits. These efforts help ensure reliable ground-truth answers and reduce potential data leakage.

  • The score-grounded quartet design provides a more fine-grained evaluation than standard item-level accuracy by examining correctness across different scores sharing the same question and answer choices. The additional error analysis and supervised adaptation experiments further demonstrate the benchmark's usefulness for diagnosing and improving symbolic music understanding.

Critical weaknesses
  • Existing music benchmarks already use programmatically generated questions and verifiable answers, while the quartet design is closely related to contrast-set evaluation. Moreover, Appendix B.3 shows that Quartet scores are generally close to what would be expected from item-level correctness. It remains unclear what additional model capabilities these metrics reveal beyond standard accuracy.

  • The experimental comparisons are difficult to interpret because the evaluation protocols differ substantially. Open-weight models use restricted next-token scoring without chat templates or generated reasoning, whereas API models use generated answers with different prompting and reasoning settings. This makes it difficult to determine whether the large performance gaps reflect actual model capabilities or evaluation choices. Additionally, the supervised adaptation results are limited to the same task distribution and templates, leaving broader generalization insufficiently validated.

Questions for authors
  • Could the authors clarify what new diagnostic information ARJQ provides beyond standard accuracy and existing contrast-set evaluations? In particular, Appendix B.3 shows that actual Quartet scores are generally close to the fixed-correctness regrouping reference. Also, since the original and replacement predictions come from independent calls, how should Retention be interpreted as a measure of evidence dependence rather than answer-label bias?

  • Could the authors provide a more controlled comparison using consistent prompting and inference settings? For example, evaluating the same open-weight models with chat templates, direct answer generation, and restricted next-token scoring would help clarify whether the reported performance gaps are caused by model capabilities or evaluation protocols.

  • Could the authors evaluate the ScoreQA-adapted models on external benchmarks such as SSMR-Bench? This would help establish whether the large improvements from LoRA reflect transferable musical reasoning abilities or primarily learning the task templates and operations used in ScoreQA.

Comments on the paper

1 comment

Dear reviewers,

As the discussion period approached the end, please take a moment to read the authors’ responses and share your thoughts on any remaining points of discussion. If the discussion has changed your assessment, please update your review accordingly.

Thank you for your time and contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.