- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce a benchmark that tests the abilities of audio LLMs to recognize the same musical components, in terms of melody, harmony, timbre, and tempo. When comprising positive pairs, the authors changed sound itself rather than musical components, such as reverberation. Using the introduced benchmarks, the authors tested open weights as well as commercial audio LLMs, and made an open leaderboard. Their additional contributions include post-training an open audio LLM and analyze the results.
Critical strengths
- Human baseline is a strong comparison point.
- A benchmark to test the ability to discern musical components, including melody and rhythm, is timely and important.
- Open leaderboard and data is a contribution to the research community.
- The authors tried to reduce overfitting using held-out test conditions (e.g., EQ).
Critical weaknesses
-
Training and evaluation have been conducted on synthetic music while validating using a small human-curated dataset seems necessary and doable.
-
Whether the distribution of training and evaluation data reflects real-world music data is questionable. Held-out test conditions resemble training conditions and it is difficult to claim generalizability.
-
Timbre is highly multi-dimensional but the authors experimented with limited sets of instruments.
Because of the data quality issues, the paper provides limited insights.
Questions for authors
-
Did the authors try building a small human-curated dataset consisting of pairs from real-world dataset?
-
Can timbre be tested on different levels, using synthesizers, for example?
2 comments
Response to Official Comment by Reviewer vMns
Dear Reviewer vMns,
Thank you for your reply, and for your kind words about our revisions. We appreciate it.
We agree that brightness and attack cover only a small part of what a synthesizer can change. We chose these two because attack time and spectral centroid are two of the main dimensions found in a study of synthesized timbres [1], and we vary each in graded steps (cutoff 1/3 to 2 octaves higher, attack 20 to 200 ms against 5 ms).
On what the benchmark is for: it tests a necessary condition, whether a model can tell that two short clips are the same. Our main answer to the title question is "not yet" (Sec. 1). A narrow test is enough to show such a failure, but it cannot certify broad timbre ability, and we do not use it that way. The paper shows the risk you describe: after training, Qwen2.5-Omni scores 99.6% on the four-instrument task but detects only 10% of violin vs viola changes (App. P). This is why every test item comes with flip/stay variants and held-out test conditions, so that a high score on clean items alone does not count as success.
Our timbre results also do not rest on the synth parameters. Brightness and attack are two of the three new sub-tasks in App. P. The third uses close pairs of sampled instruments from the test soundfont, and the real-recordings set adds seven instruments from Medley-solos-DB. As App. P notes, a model may judge a brighter version of one synthesizer to be the same instrument, so the close pairs are the clearer test.
On overfitting: the rendered test set uses a different soundfont and different pitch ranges from training, the real-recordings set is used for evaluation only, and the leaderboard lists models trained on our training split separately from zero-shot models. Because the rendered items come from a generator, we can also draw a fresh test set with new seeds if the public one becomes stale.
We have added this limit to App. P (p. 40 end - p. 41 beginning), which now reads: "Brightness and attack cover only a small part of what a synthesizer can change, so these two sub-tasks are a first step toward graded timbre tests, not a full test of timbre."
We hope this addresses your remaining concern, and we would be grateful if your updated assessment could be reflected in the rating. Thank you again for your time.
Best regards,
The authors
[1] McAdams et al., "Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes", Psychological Research 58(3), 1995.
I apologize for my tardy response given the importance of the period. I appreciate the authors' effort to address my concerns.
The authors introduced two additional test sets based on real recordings as well as synth pairs to assess the ability or discerning timbre.
However, I'm not still convinced whether why the authors need to choose limited parameters (brightness and attack) to explain LLM's ability. Limited choice of parameters make it easy for models to overfit to benchmark when typical synthesizers offer hundreds of configurations to change.
Based on the authors' overall contributions, however, I'm inclined to raise my score.