- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
I feel very sorry about this paper, because it had a great potential but ended up with an improper focus on the post-training part.
First, I really enjoyed reading this paper. This is the most well-written paper in my review batch for this version of ICLR. The motivation is clear: low-resource folk music are important but lack comprehensive evaluation and dedicated performance optimization in today's MLLM community.
Second, I think this paper successfully combined the academic value (see the sentence above) and an attempt to learn trending techniques (e.g., REPA, latent reasoning, reinforcement learning) as well as engineering tricks (FlashAttention, gradient checkpointing). The paper is really well-written, the explanation of all these techniques are clear, accessible, without any over-complication that is common in nowadays Machine Learning community.
Nevertheless, the post-training ended up with either marginal performance improvement or performance degradation compared to the simplest baseline. It is a pity that the main text spent so many lines to introduce post-training stuffs, which made the focus and contribution of this paper unclear after looking at table1.
I totally believe the authors can re-format and re-purpose the paper by spending more lines on the benchmark, and only mention post-training stuffs in the appendix or in the last page. However, this level of modification is often considered "major revision" in academia.
Although I feel sorry, I can only recommend a "weak reject".
Critical strengths
- Solid and interesting research topic
- Benchmark contribution
- The clear explanation on the post-training and resource-saving techniques should be a reference for similar papers: we don't need over complication.
Critical weaknesses
- Wrong focus:
- The paper spend too much space to talk about post-training, but the post-training itself was not successful in table 1.
Questions for authors
I cannot access the demo page url, could you fix the access or upload the .zip file of your html repository (if it is html)?
5 comments
Similar to BtXo, I would like to maintain my original rating.
- Thank you for fixing the demo page!
Regarding the balance across the benchmark, UniVerseSet, and Post-training
I totally understand the authors' attempt to present overall contributions (evaluation, tuning data, engineering tricks, and post-training recipe) to the topic of low-resource music understanding.
I appreciate the discussions between authors and Reviewer BtXo, as Reviewer BtXo mentioned more detailed concerns that I have.
However, I still believe the contributions beyond benchmark/evaluation (i.e., the contribution of UniVerseSet and post-training) are too limited for a paper of 10 pages.
- The authors mentioned "Option-only accuracy cannot reveal whether a model relies on musical reasoning or superficial option-level cues."
- This is still confusing to me. Because if the accuracy can be improved by non-musical hints, this is a problem of the benchmark design
- Obviously, it should be totally fine to allow LLMs to include non-musical hints into their thinking process.
- The authors also mentioned "Our training explicitly targets Thinking Mode because we consider a correct answer supported by explicit musical reasoning a stronger indication of understanding than option selection alone."
- If the CoT (i.e., explicit musical reasoning) itself has values beyond accuracy, CoT is no more a middle product but one of the final outputs.
- I would expect to see some evaluations (e.g., human evaluation to indicate if this helped humans to enhance the understanding to the music), to more explicitly explain the value of this thinking mode.
The above two examples express my mixed feelings regarding the current manuscript: the benchmark is valuable, but the tuning set and post-training have limited value. This paper raised good research topics and questions, and it tried to tackle them, but I don't think the progress is sufficient for a 10-page paper
We sincerely thank the reviewer for the continued engagement. We would like to clarify the following points and respectfully seek more specific guidance.
1. Regarding the paper's focus and contributions.
In the initial review, the reviewer acknowledged the "benchmark contribution" and stated, "I have no substantive concerns about the paper's soundness." The main suggestion was to "spend more lines on the benchmark" and move post-training details to the appendix or final page. We have carefully considered this feedback. However, we are still somewhat unclear about what substantive contributions are considered insufficient beyond the benchmark itself.
We would greatly appreciate more concrete suggestions: Beyond constructing benchmarks, what additional research directions or contributions would be necessary to meaningfully advance low-resource music understanding? In particular, what aspects of UniVerseBench, UniVerseSet, or their associated experiments would the reviewer suggest improving or exploring further? Such guidance would be valuable not only for this work but also for the broader research community.
2. Regarding Option-only evaluation and musical reasoning.
The reviewer states that "if the accuracy can be improved by non-musical hints, this is a problem of the benchmark design."
Our point is not that models must exclude all non-musical context, nor that Option-only evaluation is invalid. Rather, a correct option alone cannot establish whether the model has correctly interpreted the underlying musical evidence, as the same answer might be obtained through superficial textual cues.
As already discussed in §3.1.3, we have explicitly addressed this concern by refining question stems and answer options to minimize the possibility of guessing from textual cues. If the reviewer considers the benchmark problematic simply because such shortcuts cannot be completely eliminated, we would appreciate concrete suggestions on how this could be further addressed in practice. We believe that acknowledging the limitations of Option-only evaluation and introducing complementary reasoning-based assessment is a constructive approach, rather than assuming these limitations can be entirely eliminated through question design alone.
3. Regarding the value and evaluation of CoT.
The reviewer suggests that "CoT is no more a middle product but one of the final outputs" and requests further evaluation of its value.
We agree that reasoning quality deserves evaluation, which is why §4.2.3 and Appendix C.2 already provide analyses of models' CoT responses.
Finally, we sincerely appreciate the reviewer's time and engagement. We respectfully invite the AC to independently assess the manuscript's contributions and the evidence provided in our responses, and to consider the broader value of this work to low-resource music understanding.
We sincerely thank the reviewer for the encouraging assessment and thoughtful suggestions. Below, we address the demo accessibility issue and clarify the complementary roles of our benchmark and post-training components.
Q1: Accessibility of the demo page
We apologize for the inconvenience. A server-side issue caused the temporary interruption and has now been resolved. The demo is accessible again:
Demo: http://143.89.224.8:8790/
We have prepared the complete benchmark resources and a detailed README for open release. The demo includes selected recordings, corresponding scores, textual materials, and a model inference interface for exploring model capabilities on UniVerse. These resources support accessibility, reproducibility, and practical use.
Q2: Arrangement of benchmark and training strategies sections
We fully understand the reviewer’s concern. During the development of this work, we carefully considered whether to focus solely on benchmark construction or also explore post-training. We chose the latter because many previous efforts on low-resource music understanding have largely stopped at benchmark construction, with limited attention to adapting models to the specific characteristics of these musical traditions.
Our goal is not simply to introduce another benchmark, but to explore how LALMs can move beyond generic music understanding toward meaningful engagement with culturally diverse music. We therefore investigate mainstream, model-driven multimodal post-training without explicitly incorporating region-specific expertise. Although the improvements are incremental, they reveal the limitations of generic self-play data-driven approaches and the importance of musicological expertise and human interaction. Given the underrepresentation of MIR in broader audio understanding research, we envision developing human-in-the-loop data flywheels to continuously improve low-resource audio benchmarks and training data through collaboration between models and domain experts.
UniVerseBench and UniVerseSet are complementary contributions: the former provides an evaluation resource, while the latter explores model-driven learning and adaptation. Their motivation, design, construction pipeline, and key validation procedures are presented in the main paper, with implementation details and error analyses in the appendix. We believe the current manuscript already provides a substantive account of both contributions, while remaining open to targeted editorial refinements to better highlight their relationship without substantially restructuring the work.
Ultimately, we envision UniVerse not merely as a static benchmark, but as a starting point for integrating underrepresented musical traditions into modern AI systems, enabling richer music education, performance, and creation.
We sincerely appreciate the reviewer's thoughtful feedback. We hope these clarifications convey the motivation behind our design choices and demonstrate the complementary value of the benchmark and training components. We respectfully invite the reviewer to reconsider the overall assessment of our work.