acceptodds
Under review as a conference paper at ICLR 2027

UniVerse: Benchmarking and Enhancing LALMs on Culturally Inclusive Low-resource Music Understanding

Abstract

Recent advances in large audio-language models (LALMs) have unlocked impressive capabilities in music understanding and reasoning. However, current LALMs exhibit severe performance degradation when applied to diverse global musical traditions, particularly low-resource folk music rooted in rich cultural contexts. Due to significant data imbalance and the absence of dedicated evaluation protocols, existing models often fail to capture the nuanced structural, acoustic, and stylistic characteristics of local genres. To bridge this gap, we present UniVerse, a comprehensive, reproducible framework for culturally-aware low-resource music understanding. UniVerse consists of two core components: (1) UniVerseBench, an expert-guided benchmark featuring 5,042 meticulously curated Q&A pairs across 38 distinct cultural entities to evaluate fine-grained musical comprehension; and (2) UniVerseSet, a fully automated multi-turn dialogue dataset synthesized to facilitate multimodal alignment. Leveraging UniVerse, we present the first systematic study investigating multimodal imbalance learning strategies across both Dense and Mixture-of-Experts (MoE) LALM architectures. Extensive experiments demonstrate that our automated curation paired with latent representation alignment yields significant gains on dense backbones (+14.9 pp), while standard SFT remains a resilient baseline on sparse MoE architectures. Nevertheless, diagnostic evaluations reveal a remaining gap between surface-level textual alignment and deep acoustic comprehension, highlighting key challenges of building general LALMs for culturally inclusive music. Our anonymous demo page is available here.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Low-resource Music, Music Understanding, Large Audio Language Models, Cross-cultural

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2hJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

I feel very sorry about this paper, because it had a great potential but ended up with an improper focus on the post-training part.

First, I really enjoyed reading this paper. This is the most well-written paper in my review batch for this version of ICLR. The motivation is clear: low-resource folk music are important but lack comprehensive evaluation and dedicated performance optimization in today's MLLM community.

Second, I think this paper successfully combined the academic value (see the sentence above) and an attempt to learn trending techniques (e.g., REPA, latent reasoning, reinforcement learning) as well as engineering tricks (FlashAttention, gradient checkpointing). The paper is really well-written, the explanation of all these techniques are clear, accessible, without any over-complication that is common in nowadays Machine Learning community.

Nevertheless, the post-training ended up with either marginal performance improvement or performance degradation compared to the simplest baseline. It is a pity that the main text spent so many lines to introduce post-training stuffs, which made the focus and contribution of this paper unclear after looking at table1.

I totally believe the authors can re-format and re-purpose the paper by spending more lines on the benchmark, and only mention post-training stuffs in the appendix or in the last page. However, this level of modification is often considered "major revision" in academia.

Although I feel sorry, I can only recommend a "weak reject".

Critical strengths
  • Solid and interesting research topic
  • Benchmark contribution
  • The clear explanation on the post-training and resource-saving techniques should be a reference for similar papers: we don't need over complication.
Critical weaknesses
  • Wrong focus:
    • The paper spend too much space to talk about post-training, but the post-training itself was not successful in table 1.
Questions for authors

I cannot access the demo page url, could you fix the access or upload the .zip file of your html repository (if it is html)?

5 comments

We sincerely thank the reviewer for the encouraging assessment and thoughtful suggestions. Below, we address the demo accessibility issue and clarify the complementary roles of our benchmark and post-training components.

Q1: Accessibility of the demo page

We apologize for the inconvenience. A server-side issue caused the temporary interruption and has now been resolved. The demo is accessible again:

Demo: http://143.89.224.8:8790/

We have prepared the complete benchmark resources and a detailed README for open release. The demo includes selected recordings, corresponding scores, textual materials, and a model inference interface for exploring model capabilities on UniVerse. These resources support accessibility, reproducibility, and practical use.

Q2: Arrangement of benchmark and training strategies sections

We fully understand the reviewer’s concern. During the development of this work, we carefully considered whether to focus solely on benchmark construction or also explore post-training. We chose the latter because many previous efforts on low-resource music understanding have largely stopped at benchmark construction, with limited attention to adapting models to the specific characteristics of these musical traditions.

Our goal is not simply to introduce another benchmark, but to explore how LALMs can move beyond generic music understanding toward meaningful engagement with culturally diverse music. We therefore investigate mainstream, model-driven multimodal post-training without explicitly incorporating region-specific expertise. Although the improvements are incremental, they reveal the limitations of generic self-play data-driven approaches and the importance of musicological expertise and human interaction. Given the underrepresentation of MIR in broader audio understanding research, we envision developing human-in-the-loop data flywheels to continuously improve low-resource audio benchmarks and training data through collaboration between models and domain experts.

UniVerseBench and UniVerseSet are complementary contributions: the former provides an evaluation resource, while the latter explores model-driven learning and adaptation. Their motivation, design, construction pipeline, and key validation procedures are presented in the main paper, with implementation details and error analyses in the appendix. We believe the current manuscript already provides a substantive account of both contributions, while remaining open to targeted editorial refinements to better highlight their relationship without substantially restructuring the work.

Ultimately, we envision UniVerse not merely as a static benchmark, but as a starting point for integrating underrepresented musical traditions into modern AI systems, enabling richer music education, performance, and creation.

We sincerely appreciate the reviewer's thoughtful feedback. We hope these clarifications convey the motivation behind our design choices and demonstrate the complementary value of the benchmark and training components. We respectfully invite the reviewer to reconsider the overall assessment of our work.

Reviewer_2hJGOpenReview ↗

Similar to BtXo, I would like to maintain my original rating.

  • Thank you for fixing the demo page!

Regarding the balance across the benchmark, UniVerseSet, and Post-training

I totally understand the authors' attempt to present overall contributions (evaluation, tuning data, engineering tricks, and post-training recipe) to the topic of low-resource music understanding.

I appreciate the discussions between authors and Reviewer BtXo, as Reviewer BtXo mentioned more detailed concerns that I have.

However, I still believe the contributions beyond benchmark/evaluation (i.e., the contribution of UniVerseSet and post-training) are too limited for a paper of 10 pages.

  1. The authors mentioned "Option-only accuracy cannot reveal whether a model relies on musical reasoning or superficial option-level cues."
    • This is still confusing to me. Because if the accuracy can be improved by non-musical hints, this is a problem of the benchmark design
    • Obviously, it should be totally fine to allow LLMs to include non-musical hints into their thinking process.
  2. The authors also mentioned "Our training explicitly targets Thinking Mode because we consider a correct answer supported by explicit musical reasoning a stronger indication of understanding than option selection alone."
    • If the CoT (i.e., explicit musical reasoning) itself has values beyond accuracy, CoT is no more a middle product but one of the final outputs.
    • I would expect to see some evaluations (e.g., human evaluation to indicate if this helped humans to enhance the understanding to the music), to more explicitly explain the value of this thinking mode.

The above two examples express my mixed feelings regarding the current manuscript: the benchmark is valuable, but the tuning set and post-training have limited value. This paper raised good research topics and questions, and it tried to tackle them, but I don't think the progress is sufficient for a 10-page paper

We sincerely thank the reviewer for the continued engagement. We would like to clarify the following points and respectfully seek more specific guidance.

1. Regarding the paper's focus and contributions.

In the initial review, the reviewer acknowledged the "benchmark contribution" and stated, "I have no substantive concerns about the paper's soundness." The main suggestion was to "spend more lines on the benchmark" and move post-training details to the appendix or final page. We have carefully considered this feedback. However, we are still somewhat unclear about what substantive contributions are considered insufficient beyond the benchmark itself.

We would greatly appreciate more concrete suggestions: Beyond constructing benchmarks, what additional research directions or contributions would be necessary to meaningfully advance low-resource music understanding? In particular, what aspects of UniVerseBench, UniVerseSet, or their associated experiments would the reviewer suggest improving or exploring further? Such guidance would be valuable not only for this work but also for the broader research community.

2. Regarding Option-only evaluation and musical reasoning.

The reviewer states that "if the accuracy can be improved by non-musical hints, this is a problem of the benchmark design."

Our point is not that models must exclude all non-musical context, nor that Option-only evaluation is invalid. Rather, a correct option alone cannot establish whether the model has correctly interpreted the underlying musical evidence, as the same answer might be obtained through superficial textual cues.

As already discussed in §3.1.3, we have explicitly addressed this concern by refining question stems and answer options to minimize the possibility of guessing from textual cues. If the reviewer considers the benchmark problematic simply because such shortcuts cannot be completely eliminated, we would appreciate concrete suggestions on how this could be further addressed in practice. We believe that acknowledging the limitations of Option-only evaluation and introducing complementary reasoning-based assessment is a constructive approach, rather than assuming these limitations can be entirely eliminated through question design alone.

3. Regarding the value and evaluation of CoT.

The reviewer suggests that "CoT is no more a middle product but one of the final outputs" and requests further evaluation of its value.

We agree that reasoning quality deserves evaluation, which is why §4.2.3 and Appendix C.2 already provide analyses of models' CoT responses.

Finally, we sincerely appreciate the reviewer's time and engagement. We respectfully invite the AC to independently assess the manuscript's contributions and the evidence provided in our responses, and to consider the broader value of this work to low-resource music understanding.

Reviewer_BtXoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UNIVERSE, a framework for evaluating and training Large Audio-Language Models (LALMs) on culturally diverse, low-resource folk music.

Critical strengths
  1. Curating 372 traditional audio tracks with manual ABC transcriptions and ethnomusicologist-audited QA pairs addresses a real gap in multicultural music evaluation.
  2. Testing SFT, DPO, and REPA across both dense and MoE audio-language models provides useful insights for the community.
Critical weaknesses
  1. The paper argues that current audio models fail on non-Western scales, additive rhythms, and regional instruments, and that Western staff/ABC notation cannot capture folk nuances. Yet UNIVERSESET is built entirely by running existing automated models and asking a text-only LLM to pretend it is hearing the audio from those text transcripts. A text LLM cannot recover fine-grained acoustic traits that the upstream captioner and ABC transcriber missed.
  2. UNIVERSESET is 83.2% Spanish, English, and Portuguese, while many low-resource languages have zero or near-zero training dialogues.
  3. Post-training improvements (+14.9 pp on Qwen2.5 and +5.9 pp on Qwen3) only appear in strict "thinking mode". In standard direct-option mode, post-training yields negligible gains (+0.3 pp) or performance drops (-1.7 pp), and the best post-trained thinking model remains worse than the untuned base model in option-only mode.
Questions for authors

Why does post-training fail to improve direct-option accuracy, leaving post-trained thinking models performing below untuned base models in option-only mode?

5 comments

We sincerely thank the reviewer for the constructive feedback. We address the three concerns below.

1. Reliability and acoustic grounding of UniVerseSet

We would like to clarify a potential misunderstanding of our data construction pipeline. The text-only LLM is not used to infer acoustic properties or verify extracted information against the original audio. Audio captions and lyrics are obtained using multimodal models, while ABC notation is generated by an automatic transcription model (Sec. 3.2.1). The text-only LLM is used solely to check consistency among these textual representations (Appendix A.2).

We acknowledge that this process cannot guarantee the acoustic correctness of upstream outputs, and do not claim UNIVERSESET as expert-verified acoustic ground truth. Rather, UniVerseBench evaluates culturally grounded understanding of real audio, while UniVerseSet explores model-driven multimodal weak supervision (Secs. 3.1–3.2). Our results demonstrate its potential while highlighting the need for expert-in-the-loop verification (Sec. 5).

2. Imbalanced language distribution

We deliberately preserve the naturally observed data distribution rather than artificially balancing languages and musical traditions. Such long-tailed distributions reflect the actual scarcity of music resources and are precisely what make low-resource music understanding challenging (Sec. 3.2.1). Artificially equalizing these distributions would obscure rather than address this problem.

We acknowledge insufficient coverage of some low-resource languages. Importantly, UniVerseBench and UniVerseSet are constructed through fully independent workflows, without sharing samples, annotations, or intermediate knowledge (Sec. 3, opening paragraph). Furthermore, as reported in our paper, training gains are not consistently correlated with data quantity (Sec. 4.3; Fig. 4; Table 4), suggesting that supervision quality and relevance also play important roles beyond sample counts alone.

3. Thinking vs. Option-only performance

The original models already exhibit substantially lower Thinking Mode accuracy than Option-only accuracy (Qwen2.5: 33.9% vs. 46.4%; Qwen3: 47.5% vs. 59.7%; Table 1). This gap predates our post-training.

Our training explicitly targets Thinking Mode because we consider a correct answer supported by explicit musical reasoning a stronger indication of understanding than option selection alone. Option-only accuracy cannot reveal whether a model relies on musical reasoning or superficial option-level cues. Accordingly, our CoT-based training aims to improve reasoning-oriented performance rather than merely optimize answer selection (Secs. 3.2.1 and 4.2.1).

Post-training improves Thinking Mode accuracy by +14.9 pp for Qwen2.5 and +5.9 pp for Qwen3. The dense Qwen2.5 model also maintains comparable Option-only accuracy (+0.3 pp), although Qwen3 shows a slight decrease (-1.7 pp; Table 1). Moreover, prior studies show that CoT does not universally improve multiple-choice performance: Ma et al. (2025) report potential reasoning-related confusion in difficult audio tasks, while Turpin et al. (2023) demonstrate that CoT explanations may rationalize incorrect predictions. Thus, longer reasoning does not necessarily yield more reliable answers. Related reasoning–label disconnect and answer–option mismatch are discussed in Sec. 4.2.3 and Appendix C.2.

We sincerely appreciate the reviewer’s feedback and hope these clarifications address the concerns raised. We also respectfully invite the AC and PC to consider the clarifications above when assessing the overall discussion, particularly regarding our actual data construction pipeline and the intended objectives of post-training. Some concerns may arise from differences in interpreting these design choices, and we hope the work can be evaluated in light of its complete methodology, evidence, and contributions. We respectfully invite the reviewer to reconsider the overall assessment.

References

  • Ma et al. (2025). Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model.
  • Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.

Thank you for the response. But my concerns remain.

  1. The response says the text-only LLM is only used to check that the captions, lyrics and scores agree with each other . But the paper shows it does much more. Sec. 3.2.1 (step 3) says the same model, Qwen3-Next-80B-A3B-Instruct, writes all the training dialogues, with both speakers “modeled as audio-only listeners”. App. A.3.3 tells it to write its reasoning as if it is listening to the music, and not to mention the text sources it actually read. So the training reasoning comes from a model that never heard the audio, but it is written as if it did.

  2. Table 4 lists 12 benchmark languages with 0 training utterances and Korean with 10, while Chinese and Korean tracks are about half of the benchmark (Fig. 2a). The authors say the benchmark and training set were built separately. That rules out leakage, but I did not raise leakage. The weak link between gains and data volume also cuts the other way. In thinking mode, Qwen2.5 gains +25.3 pp on Danish (0 utterances) and +46.0 pp on Romanian (76 utterances). Gains this large without any matching data suggest a generic effect, for example learning the thinking output format, rather than culture-specific learning. Could the authors report invalid or unparseable output rates in thinking mode on the full benchmark, before and after post-training?

  3. I understand that the paper focuses on thinking mode. I also agree that thinking mode was already weaker before post-training. My question is what post-training actually adds. For Qwen3, the post-trained model in thinking mode (53.4) still scores 6.3 pp below the original model answering with options only (59.7). After post-training, the options-only score also drops on PR (65.0 to 57.5) and CA (61.6 to 52.6). The authors say training targets thinking mode. But App. A.3.1 says half of the academic training items use Direct Answer mode, which is meant for benchmark evaluation. So the training does include direct answers, and the options-only score still does not improve.

  4. Relation to Reviewer 2hJG. Reviewer 2hJG makes a related point: in Table 1, post-training gives marginal gains or drops compared with the simplest baseline. In their reply to 2hJG, the authors call the improvements incremental and say they “reveal the limitations of generic self-play data-driven approaches.” I agree with that reading. But it does not match the reply to my review, which presents the post-training as effective. If the main finding is that generic synthetic data has limits, the paper should be framed around that.

I see value in the benchmark itself, but I don’t think the paper is ready for ICLR in its current form. I keep my score.

We sincerely thank the reviewer for the follow-up comments. We would like to further clarify several aspects of our methodology and experimental findings.

1. Clarification of the dialogue generation and validation pipeline

Our previous response addressed the specific concern about acoustic grounding during cross-modal validation, rather than denying the role of Qwen3-Next in dialogue synthesis. As described in §3.2.1, Qwen3-Next is used both to check consistency among independently extracted captions, lyrics, and ABC notation, and to generate multi-turn training dialogues based on validated text materials. In the latter stage, as specified in Appendix A.3.3, both speakers are assigned audio-listening personas to simulate natural listening interactions. This persona design does not imply that the text-only LLM independently perceives audio or discovers acoustic evidence; its generated reasoning is based on the extracted annotations.

Using text-only LLMs to sythesize training data for advancing audio understanding is not an unprecedented approach. For example, Music Flamingo (§3.3) also employs a text-only LLM (gpt-oss-120b) to generate music-related CoT from structured metadata. In other words, if existing audio LLMs already possessed sufficiently reliable musical reasoning to replace text-only LLMs in generating training data, how would we explain their substantially weaker performance in Thinking Mode on music benchmarks?

2. Zero-shot language gains and output validity

Our correlation analysis (§4.2.2, Appendix B.3.2, Table 6) similarly shows that training volume is not significantly positively correlated with performance gains across languages. This does not imply that data quantity is irrelevant, but rather that it is not the sole determining factor.

Regarding output validity, our reported Overall Accuracy already uses the full evaluation set as its denominator, counting invalid or unparseable outputs as incorrect (Appendix C.1.2). We additionally examined accuracy among parseable outputs:

  • Qwen2.5: 40.91% → 48.83% (LR + REPA, Phase B).
  • Qwen3: 53.22% → 57.75% (SFT w/ thinking).

These results show that improvements persist even when considering only parseable outputs, suggesting that they cannot be attributed solely to improved output formatting.

3. Direct Answer vs. Direct-option

We believe the reviewer has conflated two distinct concepts in our framework. Direct Answer refers to training dialogues without explicit CoT, whereas Direct-option refers to an evaluation setting that requires selecting an answer without generating reasoning traces. We provide both Direct Answer and CoT training samples below.

  • Direct Answer (without CoT): Directly provides the natural-language answer about the harp's arpeggios, the bagpipe drone, and their supportive relationship with the vocal line.
"direct_answer_no_cot": {
"messages": [
{
"role": "user",
"content": "<audio>How is the vocal line structured in relation to the instrumental accompaniment?"
},
{
"role": "assistant",
"content": "The vocal line is framed by a sparse, texturally restrained accompaniment: ..."
}
]
},
  • With CoT: Includes an explicit <think> reasoning trace before providing the identical final answer.
"with_cot": {
"messages": [
 {
    "role": "user",
    "content": "<audio>How is the vocal line structured in relation to the instrumental accompaniment?"
  },
  {
    "role": "assistant",
    "content": "<think>\nThe vocal line is not the dominant structural force but is framed and supported by a minimal, texturally consistent instrumental backdrop. The harp functions as ...\n</think>\n\nThe vocal line is framed by a sparse, texturally restrained accompaniment: ..."
      }
    ]
  }

4. Consistency of our interpretation of post-training

We respectfully believe that our responses are consistent. Throughout the manuscript and rebuttal, we have described post-training as a systematic investigation of different strategies, rather than proposing a universally effective solution to low-resource music understanding. As stated in our Introduction (§1) and Conclusion (§5), the experiments reveal both measurable improvements and the limitations of generic model-driven learning. Our response to Reviewer 2hJG described these gains as incremental because we want to reclarify our motivation for the training strategies section. Reporting improvements on specific metrics and acknowledging the limitations of these approaches are not contradictory; they are complementary findings of our investigation.

We sincerely thank the reviewer for the continued engagement. We respectfully invite the AC and PC to consider our clarifications and evidence in their overall assessment of the paper.

Reference

Ghosh, Sreyan, et al. "Music flamingo: Scaling music understanding in audio language models." International Conference on Learning Representations. Vol. 2026. 2026.

Reviewer_kxpsOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper contributes a benchmark and a dataset to improve music understanding and reasoning for many music traditions from around the world that have been overlooked in past music understanding efforts. Without paying special attention to these traditions, LALMs tend to ignore and/or misunderstand music, because the datasets and benchmarks used to build them don't include enough detail for these lower resource music traditions. The authors show that using this dataset, they can get significantly improved music understanding and reasoning.

Critical strengths

This dataset and benchmark are very welcome to everyone trying to make better models for music understanding and reasoning. The problem is clearly explained, and the solution is sensible and useful.

Critical weaknesses

I didn't like the dense vs. MoE models distinctions in this paper. As someone involved in training both dense and MoE models, I'm aware of the challenges one faces. It can be the case that we need a different data-mixing strategy or a different curriculum for dense vs MoE models. However, I felt like this is orthogonal to the main contribution of the paper, and I think this paper doesn't need to have an opinion on dense models vs MoE models, because I trust the community will be able to use the contributions of this paper for all models. I just felt this discussion was an artifact of the training methodology the authors used and isn't that important - or perhaps the authors are making claims about dense vs MoE that aren't fully supported.

Questions for authors

I feel AI for resource constrained music traditions is still quite nascent. This paper focuses on traditions from different parts of the world, but I think there is likely significant more work to be done on different musical traditions, even within one region of the world. I'd love to hear from the authors what things they want to focus on next.

1 comment

We sincerely thank the reviewer for the positive assessment of our work and for recognizing the importance of culturally inclusive music understanding. We are particularly encouraged by the reviewer’s appreciation of the benchmark, the clarity of our motivation, and the potential value of this resource to the broader music understanding and reasoning community.

Regarding the dense vs. MoE discussion, we appreciate the reviewer’s insightful observation. We agree that differences between these architectures may be influenced by various factors, including data-mixing strategies, training curricula, and architecture-specific optimization. Since our experiments involve representative models with different architectures and training configurations, rather than a controlled comparison isolating architectural differences, the results should not be interpreted as general conclusions about the relative advantages of dense and MoE models. Our intention is to explore how different LALM architectures respond to culturally diverse music supervision, rather than to advocate for either paradigm. We will accordingly narrow our claims of separating dense and MoE architecture and streamline this discussion to keep the focus on UniVerse.

Regarding future research directions, we fully agree that culturally diverse music understanding remains an emerging research area, with substantial diversity both across and within geographical regions. We envision two promising directions:

1. Greater depth within specific musical traditions: We believe deeper exploration of individual musical traditions is as important as expanding geographical coverage. Although UniVerseBench already includes examples such as Indian ragas, many region-specific practices remain underexplored. We hope to investigate these traditions through collaboration with ethnomusicologists and local musicians, while making UniVerse's human–AI interaction workflows and tools adaptable and accessible to people with expertise in their own musical cultures, enabling them to actively lead the documentation and digitization of their traditions.

2. Human-centered music understanding and creation: We aim to move beyond static music knowledge retrieval toward interactive human–AI systems that support culturally grounded music learning, understanding, performance, and co-creation. Beyond generating music that respects specific cultural traditions, these systems should inspire performers, creators, and anyone interested in music, fostering new forms of musical exploration and creativity.

We also believe that research on culturally diverse music can contribute to broader low-resource audio understanding by offering insights into how models handle underrepresented acoustic characteristics and domain-specific knowledge. Ultimately, our goal is to foster closer collaboration between AI systems and human musical expertise, enabling more inclusive and meaningful music technologies. We sincerely appreciate the reviewer’s constructive suggestions, which have helped us further clarify the scope and future potential of UniVerse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.