acceptodds
Under review as a conference paper at ICLR 2027

Symbolic Music Reasoning Needs Checkers That Survive Rewording: What a High Score Covers

Abstract

On SSMR-Bench, a template-generated benchmark of questions about sheet music in ABC notation, our hand-written music21 checker, developed on the test items, answers 70% of items at 99.8% precision. Prove-then-route systems for symbolic music reasoning build on this: the checker answers the questions it can derive and a language model answers the rest. However, a high score cannot tell whether the checker computes music theory or merely recognizes the benchmark's question templates. Our key idea is that a task-preserving rewording separates the two: a checker that computes should keep its contribution, while one that recognizes wording should lose it as the model's own accuracy stays put. Our audit therefore measures the checker's matched contribution, the system's accuracy minus the same model's alone, before and after such rewording; a normalizer that maps reworded questions back onto the known templates tests whether a loss lies in how the checker reads the question. A checker that Claude Opus 4.8 wrote with execution feedback answers 93.4% of questions at 99.67% precision, yet rephrasing only the question line removes most of its contribution (from +21 to +3–4 points over a Qwen2.5-14B model distilled from DeepSeek-R1 traces) while the model's own accuracy barely moves. Most of that gain sits on outputs the distilled model truncates, yet on the outputs it finishes the smaller gain still falls (both post hoc). Paraphrases by the checker's own writer model also cut its coverage, and mapping them back onto the templates restores every checker answer: under rewording, what breaks is the checker's reading of the question, not its music computation. In logic the failure is worse: on ProofWriter, equivalent rule rewrites make a checker built by the same loop answer wrongly on most items through a silent default “Unknown” (post hoc), pulling the system below the model alone; the published Logic-LM design, run with a Qwen translator, makes the same errors, though its precision holds under rewording. Primary tests were pre-registered with frozen programs, several after the results that motivated them. Where a checker settles part of a task, a prove-then-route result should therefore report its coverage, precision and matched contribution under rewording.

Primary area:
datasets and benchmarks
Keywords:
symbolic music reasoning, benchmark auditing, program-aided reasoning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ggkaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work study Prove-then-route systems for symbolic music reasoning. To tell the contributions of the checker and the LLMs, they proposed task-preserving rewording and evaluate the checker’s matched contribution, the system’s accuracy minus the same model’s alone. They found that rewording the questions degrades the checker’s contributions significantly, for both accuracy and coverage. Normalizing the questions can restore the checker’s performance.

Critical strengths

The problem studied is clearly defined and the proposed method can clearly separate the contributions of the LLMs and the checker. They also find the weakness of SSMR-Bench.

Critical weaknesses
  1. The experiment has critical flaws. In the paper, the checker is frozen and the questions is reworded. Based on this setting, the performances dropped a lot. However, When the questions are reworded, the checker should also be updated accordingly. In such case, I believe that the checker can still answer the questions.

  2. For ProofWriter experiment, the default for the checker is “Unknown” while “Unknown” is also a possible answer. This is not a reasonable checker in such scenario. Abstaining on it is like a bug fix instead of a critical improvement.

  3. It is good that they found that the options alone determine all 352 key-identification items. However, this should be regarded as the design flaw of the benchmark. Since SSMR-Bench is not widely used (based on citation) and the paper was rejected, I don’t think it is significant contribution.

Questions for authors

SSMR-Bench is not a widely used benchmark, why do you want to study on it?

1 comment

Thank you for the review. Changes are marked in green in the revised PDF. On your main point you were right, and we ran the experiment you asked for, twice more.

W1: "When the questions are reworded, the checker should also be updated."

You were largely right: an updated checker keeps its gain on most rewordings its writer never saw, more than we had predicted. What the experiment also shows is that the original score does not say how much an update still loses. The first and second updates look the same on the original questions (85% to 86% answered, 99.6% correct). On Phi-4's free rewording one lost 10 points and the other under 2, and both still lost 25 to 30 on Phi-4's everyday-words set. Published systems report only the first kind of score.

CheckerAdds over the model alone, original wordingLost on four sets by Llama and Phi-4Lost on three later sets by Mistral and OLMo
Original4311, 25, 34, 542, 16, 12
First update (in the submission)383, 10, 30, 1not run
Second update39under 2, under 2, 25, under 243, 3, 0
Third update35seen by its writer25, 2, 0

Points, with Qwen2.5-14B (Appendix A.7). Third column, in order: Llama free, Phi-4 free, Phi-4 everyday words in place of music terms, Llama context first; five rewrites that changed the task are left out. Fourth column: everyday words, formal register, a learner's casual request. The second update is the run our fixed rule picked among three (the other two answer only 66% and 48% of the original questions); the third is one run.

  • A gap remains. The third update, whose writer had seen Phi-4's everyday-words set, still lost 25 points on everyday words by a new author, and it answers fewer original questions (75%; we had predicted at least 85%).
  • Why we first tested the checker frozen. A score is reported for one fixed program, and a deployed checker meets wordings its writer never saw. Nothing in the original checker's score asked for an update.
  • It is not one program. A second writer model (Opus 5.5; a third, Sonnet 4.6, produced no program), given our procedure and only the original wording, wrote a checker that scores like ours (94.5% answered, 99.7% correct). It lost 10 points or more on 8 of 11 reworded sets, as predicted (our checker: on 10 of 11; it loses less than ours on most).

W1 (continued): are everyday-words questions still the same task?

There are two such sets. On Phi-4's, Qwen2.5-14B alone scores 53% against 54% on the original wording, and a model shown the nine templates maps 99% of the questions back to their template, after which the checker's gain returns; Claude Opus alone loses about 7 points there, so that set is somewhat harder. On Mistral's, a second model and we checked every rewrite before scoring (for example, "chord" becomes "group of notes" and "key" becomes "sound set"; we noted three of the eight rewrites as loose and kept them), and Qwen2.5-14B alone gains 2 points. The checkers fail on both because they pick what to compute from the music terms.

W2: "Defaulting to Unknown is not reasonable; abstaining is a bug fix."

Agreed on both. The paper now calls the default a bug, treats logic as a case study, and no longer lists the fix as a contribution. Logic-LM's released programs have the same silent default: all 121 of their wrong answers on the original questions come from it (unlike ours, Logic-LM keeps its precision under rewording). With the bug fixed our checker gave no wrong answer on 900 new problems, but its gain still falls from 22 points to 2 under rewritten rules and to 0 when the question sentence is reworded: the fix removes the wrong answers, not the dependence on wording.

W3 and Q1: "The shortcut is the benchmark's flaw; SSMR-Bench is not widely used. Why study it?"

We agree on both, and the benchmark is not our contribution. The project began on it as music-reasoning work; our checker with a 14B model reached 97%, level with Claude Opus alone, and the test came from asking what that score was worth. What carries beyond it is the test. On methods others built it found:

  • PAL on GSM8K (two models): kept its gain of about 7 points under GSM-Plus's paraphrases. When GSM-Plus inserts irrelevant sentences (distraction, not rewording; the answers are unchanged) it fell to about 4 points below the model alone, which PAL's GSM8K score does not show.
  • LATM's released tools: the overall gain held; one tool's gain fell from 17 to 5 points under one rewording (we had predicted no such drop). Music checkers built by LATM's tool maker lost 0 to 16 points.
  • CSyMR's released music agent: its gain (about 5 points) is too small to be sure of on 126 questions; one rewording run outside our pre-set rule showed no sign of a loss.

Losses above 20 points are only in checkers written by our procedure, and the revised abstract says so.

Reviewer_DXZdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies checker-policy systems on symbolic music reasoning and deductive logic benchmarks. The authors propose an audit that measures the checker's contribution to the benchmark performance before and after rewording the question. The authors observe a general decrease of the checker contribution on both benchmarks after rewording. Then, two remedies are proposed: one based on normalizing reworded questions back onto the benchmark's original templates before the checker sees them, and robustness-aware re-synthesis.

Critical strengths

None

Critical weaknesses
  1. The motivation of the problem is unclear. It is not clear why reasoning for music is needed, what problems can it solve, and what methods have been attempted by prior works. Additionally, the work never introduces the concept of symbolic music reasoning, or ABC notation. Additionally, the work never motivates the need for checker-policy harnesses.

  2. The Section 3 statement that "the music audit as a whole was not registered in advance" suggests that the evaluation was focused on the music domain after seeing the results. This suggests that the focus on music has been selected after seeing the results. If that is the case, then this decision induces selective bias in the results. Otherwise, the authors should clarify their methodology further.

  3. The related work section contains no references on reasoning for music. If the authors claim there are no prior works that consider reasoning for music, this should be made explicitly clear. However, this is not the case, due to the existence of ChatMusician, CSyMR [1,2]. No prior work focused on music application has been tested under rewording.

  4. It is unclear why we need those checker-policy systems, with checkers generated by an LLM, instead of the LLM itself. State-of-the-art language models come with harnesses which should provide strong robustness to rewording. These checkers are lexical, as shown in Figure 1 ("if 'name of the interval' in q"), so it is not unexpected that they are brittle under rewording. The stated normalizer knows the nine question templates in advance and is not tested on unseen rewording structures, putting doubt into the generalizability of the method.

  5. The paper presents two benchmarks, one of which is not music-specific, one policy family at two sizes plus one Llama model, and one checker architecture for music. This evaluation is too narrow to support any kind of general claim. A similar weakness holds for the different rephrasing strategies.

References
[1] Yuan, Ruibin, et al. "Chatmusician: Understanding and generating music intrinsically with llm." Findings of the Association for Computational Linguistics: ACL 2024. 2024.

[2] Wang, Boyang, et al. "CSyMR: Benchmarking compositional music information retrieval in symbolic music reasoning." Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2026.

Questions for authors
  1. Table 4 reports that the score of Opus 4.8 alone on SSMR-Bench is 96.2%. What is the motivation for using the checker-policy systems in the first place when LLMs alone show such high capabilities?

  2. Why were no music-specific prior systems, such as CSyMR’s LLM-plus-music21 agent or the SSMR-Bench authors’ GRPO model, evaluated under the same rewording?

  3. Was the music domain fixed before the rewording audit was designed?

1 comment

Thank you for the direct questions. Changes are marked in green in the revised PDF. On two main points you are right: a program that matches question wording is brittle, and on this benchmark Claude Opus alone is the better choice. We do not recommend our checker (a program Opus wrote); it is what we test.

W4: "A lexical checker is expected to break."

Agreed, and we expected it too: before the first rewording test we predicted its coverage would at least halve. But we did not write the wording match (Opus did, from 40 worked examples and feedback on training questions), and the score gave no warning: with Qwen2.5-14B the checker scored 97%, level with Opus alone at 96%. The normalizer needs the templates, so the paper now calls it a diagnostic, not a fix.

What was not expected is what the test shows about the score. Such systems report one score, on the benchmark's wording, and it ranked two checkers the wrong way:

Points added over Qwen2.5-14B aloneOriginal wordingAfter rewording (nine sets)
Our checker436 to 22 on five sets by Opus; 18, 10 on two by Phi-4; 31, 38 on two by Llama
Checker built by LATM's published tool maker (a model reads each question)2924 to 27; 28, 25; 26, 27

The lower-scoring checker ends up adding more on 7 of these 9 sets, and on 1 of 3 later ones (Appendix A.8 and the run logs in the supplement; no interval computed). Nor can it tell apart checkers that then behave differently:

  • Three runs of LATM's tool maker add 29, 29 and 33 points on the original wording. On Llama's context-first set they lost 3, 12 and 0 (we had predicted the runs would agree), and on a later everyday-words set by Mistral the first lost 16 (we had predicted no clear loss).
  • A second writer model (Opus 5.5, same family; one run), given our procedure and only the original wording, wrote a checker that scores like ours (94.5% of questions answered, 99.7% correct). It lost 10 points or more on 8 of 11 reworded sets (our checker: on 10 of 11; it loses less than ours on most). We had predicted at least 6. A third writer, Sonnet 4.6, produced no program.

Programs others released mostly held (LATM's tools; PAL on GSM8K under GSM-Plus's paraphrases). Losses above 20 points are only in checkers written by our procedure, and the revised abstract says so.

Q1: "Why use a checker when Opus alone scores 96%?"

On this benchmark we would not. A checker saves cost only over many passes (ours took about 709,000 Opus output tokens to write, then answers 93% of questions with no model call; Opus alone spends about 284,000 per pass), and only while questions keep the benchmark's wording: on the 11 reworded sets our system is 4 to 37 points behind Opus. So the paper offers a test, not a checker; the title means that a checker, where one is used, must be shown to survive rewording.

Q2 and W3: "Why not CSyMR's agent or SSMR-Bench's GRPO model?"

We had wrongly believed CSyMR's agent code was unreleased, and rebuilt it. It is public, and we have now run it as released. (Our rebuild had scored about 6 points lower; we had predicted within 5.) Its published gain of about 9 points used GPT-4.1-mini, which we cannot run (no OpenAI access); with Qwen2.5-14B it scored 54% against 49% for the model alone, and with Claude Haiku 4.5 63% against 58%. Gains of about 5 points cannot be told from zero on 126 questions, so our pre-fixed rule said not to run the rewording test. Because you asked, we ran one anyway: on a Llama-written rewording the agent scored 67% and Haiku alone 56%, so we see no sign that it loses its gain. We did not run the GRPO model: we found no released weights for it. The closest we ran, a Qwen2.5-14B fine-tuned on a larger model's answers to the training questions (not GRPO), was stable: 76% on the original wording, 74% to 77% on seven reworded sets.

W5: "The evaluation is too narrow."

Agreed. It now has six answering models from four families on the five Opus-written sets (rewording removes 11 to 46 points of what our checker adds; each model alone moves by at most 3), seven more sets written by Llama, Phi-4, Mistral and OLMo, and four ways of building a music checker (ours, its updates, a second writer's, LATM's tool maker).

W2 and Q3: "Was music chosen after seeing the results?"

No. The project began in May 2026 on symbolic music (SSMR-Bench from 22 May). The first rewording test was pre-registered on 23 September and the logic test on 24 September. Some choices did follow results, as Section 3 says: the rewording test came after the checker's high score, and most tests were registered after the results that motivated them. Three studies ran past their registered time limits, each under an amendment made before that step (Appendix A).

W1: "Motivation and background are missing."

You are right. The introduction now explains the task, ABC notation and why programs are paired with models; related work adds seven music-reasoning papers.

Reviewer_AcK9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits systems where a frozen program answers first and a language model handles abstentions. It measures coverage, precision, and the program's contribution over the same model alone under rewording. On SSMR-Bench, question paraphrases sharply reduce the synthesized checker's coverage; normalization to known templates restores its answers. On ProofWriter, equivalent rule rewrites expose incorrect default Unknown answers that can make the system worse than its fallback. The paper also examines option shortcuts, robustness-aware synthesis, and abstention on Unknown.

Critical strengths
  1. The matched fallback comparison and normalization control make the music result more informative than a drop in system accuracy alone. The coverage collapse, identified dispatch triggers, and restored checker outputs support a specific interface failure. The authors also show that truncation accounts for much of the headline gain over the distilled policy and report the smaller effects on completed outputs. This makes the remaining claim easier to assess.

  2. The audit identifies actionable failures beyond abstention: options alone solve the key-identification subset, and silent Unknown commits can override correct policy answers. The fresh-theory evaluation of abstaining on Unknown strengthens the latter diagnosis while showing its coverage cost. Together these concrete findings make me lean toward acceptance as an empirical audit, despite limited methodological novelty.

Critical weaknesses
  1. The evidence has limited reach beyond the audited implementations. The largest collapses concern programs built within this study; Logic-LM retains precision under rewording. Prior work already compares autoformalisation and informal reasoning under perturbations and separates execution failures from semantic errors [1]. The distinctive contribution is therefore the paired contribution analysis and the specific failure diagnoses. The known-template normalizer establishes where these checkers fail, but does not establish robustness to an unknown interface. These limits reduce the broader significance of the proposed audit.

  2. One end-to-end paraphrase condition does not preserve the task. Set 3 adds “above” to note-completion questions, invalidating some original labels, while normalization removes that constraint. Appendix Y.1 establishes checker-only collapse and restoration after excluding this template, but Appendix Y.3 still reports end-to-end contributions on the full set. Those numbers mix interface restoration with a task change and do not fully support the reported five-set conclusion. The other four sets remain useful evidence.

[1] Fabian Hoppe, Filip Ilievski, and Jan-Christoph Kalo. "Investigating the Robustness of Deductive Reasoning with Large Language Models." ECAI 2025. arXiv.

Questions for authors
  1. Using the existing outputs, what are the original, raw, and normalized matched contributions for set 3 after excluding the note-completion template? Do the corresponding conclusions hold for all three policies?

  2. For the distilled policy on the same-model paraphrases, how much of the contribution loss remains on items completed in both conditions? (This would clarify whether the truncation sensitivity found for the two initial rephrasings also applies to this broader wording test.)

1 comment

Thank you for the careful review. Changes are marked in green in the revised PDF. "Gain" below means the accuracy of checker plus model minus the accuracy of the same model alone, in points.

W1: "The largest collapses concern programs built within this study."

You are right, and the revised abstract now says it: losses above 20 points occur only in checkers written by our procedure. Every music checker here was built in this study; what we varied is how (our procedure with two writer models, and LATM's published tool maker). Programs that others released mostly kept their gain under paraphrase. The test still matters because the original score did not tell us which checker would hold up:

  1. It reverses a ranking. On the original wording our checker adds 43 points and one built by LATM's tool maker (Cai et al., ICLR 2024), in which a model reads each question, adds 29. After rewording, the LATM-built checker adds more than ours on 8 of the 12 sets both were run on: all five Opus-written sets (24 to 27 points against 6 to 22), both Phi-4 sets (28 against 18, and 25 against 10) and Mistral's everyday words (11 against 0). Ours stays ahead on the two Llama sets and two later ones. This is one tool-making run, with no interval computed (Appendix A.8). Our stated prior was that the LATM-built checker would keep its gain on at least 3 of the 4 Llama and Phi-4 sets; it did on 2.
  2. It separates runs of a published method. Two repeats of the tool making (a third was killed by the host) add 29 and 33 points on the original wording, like the first. One lost under 4 points on all four sets; the other lost 4, 5, 6 and 12 (on the last set ours loses 5). We had predicted the runs would agree. On three later sets by Mistral and OLMo the first run lost 16, 5 and 4 points, and on the first its answers fell from 97% to 82% correct; we had predicted it would lose 5 or more on none.
  3. The loss is not one program's. We reran our procedure with the writer changed (Claude Opus 5.5 for 4.8; no system prompt), on the original wording only. Its checker scores like ours (94.5% of questions answered against 93.4%; both 99.7% correct) and lost 10 points or more on 8 of 11 reworded sets; we had predicted at least 6. It loses less than ours on most sets. A third writer, Sonnet 4.6, produced no program, so this rests on one further writer of the same family, in one run.
  4. Programs we did not write. With two small models, PAL on GSM8K kept its gain of 7 points under GSM-Plus's paraphrases and fell 4 points below the model alone when GSM-Plus inserts irrelevant sentences (distraction, not rewording; the answers are unchanged). LATM's released tools kept their overall gain; one tool's gain fell from 17 to 5 points under one rewording, where we had predicted no such drop.

Q1: set 3 without the note-completion questions. You are right that set 3 changes the task for those questions. With them removed, no set-3 conclusion changes:

ModelGain, original wordingSet 3: mapped back to templates / read directly
Untuned Qwen2.5-14B41.541.6 / 14.9
Untuned Qwen2.5-7B50.450.1 / 17.3
Distilled 14B21.120.1 / 9.4

Each model alone moves by 1 point or less. One result outside set 3 does change: the distilled model's loss now reaches 10 points on 3 of the 5 sets instead of 2 (Appendix A.3; this re-scoring was chosen after your review).

Q2: cut-off outputs. Yes, it applies here too. 71% to 94% of the distilled model's loss sits on outputs cut off at the length limit; on questions it finishes under both wordings the loss is 0.4 to 5 points. The abstract no longer leads with this model. The untuned 14B model cuts off at most 5 of 1,600 outputs and loses 20 to 37 points.

W2 in the new rewording sets. We now read every new rewrite before scoring. Of 34 in the Llama and Phi-4 sets, 5 changed the task; comparisons between checkers leave them out (with them ours loses 7, 25, 33 and 16). The losses that survive updating are on two everyday-words sets. On the one Mistral wrote, Qwen2.5-14B alone gains 2 points. On the one Phi-4 wrote, Qwen2.5-14B alone moves from 54.0% to 52.8%, and a model shown the templates maps 99% of the questions back to their template, after which the gain returns; Claude Opus alone does lose about 7 points on that set.

Hoppe et al. and the normalizer. Section 2 now cites it. They compare autoformalisation with informal reasoning under perturbation; we hold one program fixed and score it against the same model alone on the same questions. We agree the normalizer shows where a checker fails and is no defence against unknown wording; the paper now calls it a diagnostic. Our closest test of unknown wording is updating the checker: the second update kept its gain on 3 of 4 unseen sets (we had predicted a loss on all 4), and a third still lost 25 points on everyday words by a new author (Appendix A.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.