- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work study Prove-then-route systems for symbolic music reasoning. To tell the contributions of the checker and the LLMs, they proposed task-preserving rewording and evaluate the checker’s matched contribution, the system’s accuracy minus the same model’s alone. They found that rewording the questions degrades the checker’s contributions significantly, for both accuracy and coverage. Normalizing the questions can restore the checker’s performance.
Critical strengths
The problem studied is clearly defined and the proposed method can clearly separate the contributions of the LLMs and the checker. They also find the weakness of SSMR-Bench.
Critical weaknesses
-
The experiment has critical flaws. In the paper, the checker is frozen and the questions is reworded. Based on this setting, the performances dropped a lot. However, When the questions are reworded, the checker should also be updated accordingly. In such case, I believe that the checker can still answer the questions.
-
For ProofWriter experiment, the default for the checker is “Unknown” while “Unknown” is also a possible answer. This is not a reasonable checker in such scenario. Abstaining on it is like a bug fix instead of a critical improvement.
-
It is good that they found that the options alone determine all 352 key-identification items. However, this should be regarded as the design flaw of the benchmark. Since SSMR-Bench is not widely used (based on citation) and the paper was rejected, I don’t think it is significant contribution.
Questions for authors
SSMR-Bench is not a widely used benchmark, why do you want to study on it?
Thank you for the review. Changes are marked in green in the revised PDF. On your main point you were right, and we ran the experiment you asked for, twice more.
W1: "When the questions are reworded, the checker should also be updated."
You were largely right: an updated checker keeps its gain on most rewordings its writer never saw, more than we had predicted. What the experiment also shows is that the original score does not say how much an update still loses. The first and second updates look the same on the original questions (85% to 86% answered, 99.6% correct). On Phi-4's free rewording one lost 10 points and the other under 2, and both still lost 25 to 30 on Phi-4's everyday-words set. Published systems report only the first kind of score.
Points, with Qwen2.5-14B (Appendix A.7). Third column, in order: Llama free, Phi-4 free, Phi-4 everyday words in place of music terms, Llama context first; five rewrites that changed the task are left out. Fourth column: everyday words, formal register, a learner's casual request. The second update is the run our fixed rule picked among three (the other two answer only 66% and 48% of the original questions); the third is one run.
W1 (continued): are everyday-words questions still the same task?
There are two such sets. On Phi-4's, Qwen2.5-14B alone scores 53% against 54% on the original wording, and a model shown the nine templates maps 99% of the questions back to their template, after which the checker's gain returns; Claude Opus alone loses about 7 points there, so that set is somewhat harder. On Mistral's, a second model and we checked every rewrite before scoring (for example, "chord" becomes "group of notes" and "key" becomes "sound set"; we noted three of the eight rewrites as loose and kept them), and Qwen2.5-14B alone gains 2 points. The checkers fail on both because they pick what to compute from the music terms.
W2: "Defaulting to Unknown is not reasonable; abstaining is a bug fix."
Agreed on both. The paper now calls the default a bug, treats logic as a case study, and no longer lists the fix as a contribution. Logic-LM's released programs have the same silent default: all 121 of their wrong answers on the original questions come from it (unlike ours, Logic-LM keeps its precision under rewording). With the bug fixed our checker gave no wrong answer on 900 new problems, but its gain still falls from 22 points to 2 under rewritten rules and to 0 when the question sentence is reworded: the fix removes the wrong answers, not the dependence on wording.
W3 and Q1: "The shortcut is the benchmark's flaw; SSMR-Bench is not widely used. Why study it?"
We agree on both, and the benchmark is not our contribution. The project began on it as music-reasoning work; our checker with a 14B model reached 97%, level with Claude Opus alone, and the test came from asking what that score was worth. What carries beyond it is the test. On methods others built it found:
Losses above 20 points are only in checkers written by our procedure, and the revised abstract says so.