acceptodds
Under review as a conference paper at ICLR 2027

MUUNE: Towards Fine-Grained Full-Song Music Retrieval with Multimodal Large Language Models

Abstract

Music retrieval requires correspondence between complete songs and diverse linguistic expressions, together with recognition of multiple levels of musical attributes, including style, instrumentation, vocals, and music theory. Existing methods make substantial progress in short-clip alignment, but preserving full-song detail, understanding complex queries, and constructing high-quality supervision remain key challenges. We present MUUNE (MUsic UNderstanding Embedding), which combines music–text data construction, music knowledge adaptation, and unified representation learning with a multimodal large language model (MLLM). Audio language models and music information retrieval tools provide structured metadata for multi-view retrieval supervision and atomic counterfactual evaluation. MUUNE adapts a pretrained audio language model into a Music Reasoner and learns shared music–text representations supporting inputs of up to 300 seconds. MUUNE-EH with Music-Meta-of-Thought (MMOT) exceeds all compared baselines on all seven primary retrieval tasks, reaching 44.48% mean R@10 and 93.93% mean counterfactual accuracy, exceeding CLaMP3 by 23.30 and 15.37 percentage points, respectively. Relative to MuQ-MuLan adapted with the same contrastive data, the gains are 16.09 and 7.70 percentage points. Controlled ablations show that music knowledge adaptation and an independent embedding head improve retrieval. With the model and queries fixed, extending audio from 30 to 300 seconds improves the retrieval mean by 10.95 points. The benefit of explicit music analysis depends on the representation design.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Music Information Retrieval, Audio–Text Representation Learning, Multimodal Large Language Models, Music Understanding Embeddings

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xzS2OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MUUNE, which combines music-text data construction, music knowledge adaptation and unified representation supporting audio of up to 300 seconds learning with a multimodal large language model. In the evaluation, authors construct a scalable multi-view supervision pipeline and atomic counterfactual evaluations with public-data extensions.

Critical strengths
  • The paper proposes a well-designed data construction pipeline that generates Multi-View Retrieval Text from audio using both Audio Language Models and MIR tools, enabling the creation of diverse texts (views) as well as counterfactual evaluation data.

  • The proposed full-song encoding approach, which represents an entire song by encoding the song within one music-adapted language-model sequence, enabling retrieval based on full-song music understanding rather than short audio excerpts. The experiments further demonstrate the effectiveness of this design, showing substantial gains when extending the input length from 30 seconds to 300 seconds.

Critical weaknesses
  • It remains unclear whether the gains come from music-specific reasoning learned by the Music Reasoner or simply from adapting an MLLM into a retrieval model. For example, a comparison with a baseline trained as an MLLM retriever without the reasoning component would help more clearly isolate the contribution of the Music Reasoner.

  • The relationship between the newly constructed training/evaluation datasets and the proposed method is not entirely clear.

Questions for authors

If my understanding is correct, in Table 1, SDD orig. and JMC use the original text captions, whereas the other datasets are based on the proposed Multi-View text generation pipeline. Since the Multi-View texts are constructed using the same methodology as that used for creating the training data (e.g., CL-Subset and CL-Full), there may be some concern that the evaluation setting is more favorable to the proposed approach. To avoid potential misunderstanding, it would be helpful to explicitly indicate which evaluation datasets are derived from the same Multi-View construction pipeline, for example through a marker or note in the table.

1 comment

Thank you for your positive assessment of our work and for suggesting clearer annotations of the evaluation-text sources. We clarify below the relationship between our data pipeline and evaluation, and the contribution of music knowledge adaptation.

1. Evaluation-text sources and the motivation for our benchmarks

Your understanding is broadly correct, but the evaluation sets have different text sources. SDD_orig uses the original human-written descriptions, and JMC-30s uses the official captions. JMC-fullsong retains the original captioning prompt but regenerates descriptions from longer audio. Music4All-US10K preserves the original tags and constructs additional text views from these tags and associated attributes. In-house, SDD-MV, and MTG-Test-MV use the MusicMeta multi-view pipeline. We will explicitly annotate these distinctions in Table 1 and the appendix.

Public resources for full-song music understanding and retrieval remain underdeveloped, particularly when queries must specify multiple fine-grained musical attributes. SDD and JMC provide valuable starting points, but additional evaluation resources are needed to assess richer song-level conditions. Developing a scalable data-construction pipeline and multi-view, multi-attribute retrieval benchmarks from public audio is therefore an important contribution of this paper, rather than simply a means of evaluating our model.

Our evaluation is also not restricted to texts generated by the MusicMeta pipeline. Alongside SDD_orig, the JMC settings, and the original-tag-based Music4All-US10K evaluation, Table 2 includes independent public benchmarks covering short-clip retrieval and music-theoretic conditions. These evaluations examine performance across different text sources, granularities, and protocols, rather than relying exclusively on the multi-view benchmarks.

2. The data pipeline provides useful supervision beyond MUUNE

First, its benefits transfer across architectures. Table 1 adapts CLAP and MuQ-MuLan using the same CL-Subset and CL-Full contrastive data. Both models achieve higher overall retrieval scores than their public-weight counterparts. This shows that the supervision is useful for conventional dual encoders as well as our MLLM-based framework, and is not specific to the MUUNE architecture.

Second, the benefits extend to independent public benchmarks. Table 2 evaluates data-adapted models on MusicCaps-test, FGMCaps, and MusicBench A/B. In particular, MuQ-MuLan improves on all four tasks after CL-Full adaptation. This provides evidence that the training data support transfer beyond evaluation texts constructed by the same pipeline.

Third, the annotations undergo human quality assessment. Tables 14 and 15 report audits of structured tags, music-theoretic attributes, and multi-view queries. The audits provide direct evidence that the pipeline can deliver rich attribute coverage with low query hallucination rates. Thus, its usefulness is supported by both downstream experiments and annotation-quality assessment, rather than assumed from the generation procedure alone.

Together, these results establish that our pipeline supplies quality-audited supervision that is useful across architectures and evaluation settings. Furthermore, CLAP and MuQ-MuLan remain below MUUNE in overall retrieval performance even after adaptation with the same CL-Full contrastive data. This supports the effectiveness of combining music knowledge adaptation with unified representation learning, beyond the contribution of the contrastive dataset alone.

3. Music knowledge adaptation is distinguished from direct MLLM retrieval

Regarding whether the gains come from the Music Reasoner or merely from converting an MLLM into a retriever, Table 3(a) already includes MLLM baselines trained directly for retrieval without our music-metadata SFT.

The Baseline provides the most direct controlled comparison: Baseline, SFT-Subset, and SFT-Full share the same backbone initialization, eight-layer EH, and CL-Subset contrastive data. Their mean retrieval scores increase from 33.63% to 37.92% and 42.42%, respectively. The improvement therefore demonstrates the contribution of music knowledge adaptation beyond direct retrieval training. Qwen3 AuT and MOSS-Music provide supplementary comparisons under different initialization conditions.

Importantly, music knowledge adaptation and inference-time MMOT generation are distinct. The former incorporates musical knowledge into model parameters; the latter expresses that knowledge as explicit analysis text. The SFT comparison uses direct encoding without MMOT, so its gains do not depend on generating an intermediate reasoning sequence at inference. The continued improvement from SFT-Subset to SFT-Full further demonstrates that scaling music-specific supervision strengthens retrieval representations. This is precisely why music knowledge adaptation is a central component of MUUNE.

Reviewer_KCNmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MUUNE, a fine-grained music retrieval method based on a multimodal large language model, aimed at retrieving longer or complete song content based on natural language queries that include multiple music attributes. MUUNE consists of three main stages. First, the author uses audio language models and traditional music information retrieval tools to extract information such as musical style, instruments, vocals, rhythm, tonality, chords, and emotions from songs, constructing a unified MusicMeta representation and generating various retrieval texts and counterfactual test samples. Secondly, the author fine-tunes the pre-trained audio language model through supervised learning in the music domain to adapt it as a Music Reasoner. The model is capable of generating structured music analysis, which the author refers to as Music-Meta-of-Thought (MMOT). Finally, the authors explore two methods for sharing embedding spaces between music and text: EOS-LoRA and Independent Embedding Head (EH), and use contrastive learning to optimize retrieval representations. The experiments used up to 2.3 million songs and approximately 140,000 hours of audio for training, and were tested on seven major retrieval tasks and two counterfactual attribute recognition tasks. Table 1 shows that the average R@10 for MUUNE-EH combined with MMOT is 44.48%, and the counterfactual accuracy is 93.93%, surpassing public baselines such as CLaMP3.

Critical strengths

Combining music domain knowledge adaptation with shared embedding learning has achieved significant performance improvements across multiple retrieval tasks. Existing music retrieval systems often rely on direct contrastive learning between audio and text, but their ability to understand complex musical attributes may be limited.

MUUNE first uses structured music attributes to adapt the model to the domain, and then learns a shared embedding space. This design allows the model to not only learn the correspondence between audio and text but also to grasp more explicit musical concepts. The paper provides a controlled experiment of music SFT in Table 3(a). Under the same CL-Subset setting, the average retrieval score of the model without domain adaptation is 33.63%, which increases to 42.42% after SFT-Full. This indicates that the model's domain knowledge adaptation makes a substantial contribution to the retrieval task. Additionally, the authors retrained the traditional CLAP and MuQ-MuLan on the same comparative data, providing more substantial evidence for assessing the role of data and model structure than directly comparing public weights.

Critical weaknesses

1: The scale and source of the training data differ significantly from the public baseline, and the current experiments cannot fully isolate the contribution of the model architecture itself. MUUNE uses large-scale commercial music data from the authors' internal music platform, with the complete training configuration including approximately 2.3 million songs and 140,000 hours of audio. The paper does indeed provide the CLAP and MuQ-MuLan baselines after data adaptation, which is commendable. However, there are still several important factors that have not been fully controlled:

  • The pre-training data and domain adaptation levels of different models vary.
  • MUUNE, in addition to contrastive training, has undergone additional music knowledge SFT.
  • Different models have different mechanisms for processing long audio.
  • The audio encoding length, trainable parameters, and inference computation load are not entirely consistent.

Therefore, MUUNE's advantages in the seven tasks may stem from the combined effects of multiple factors, rather than primarily from the independent Embedding Head or the unified MLLM representation architecture. Moreover, internal commercial data cannot be directly verified by external researchers, further increasing the difficulty of reproducibility and data distribution assessment. The authors need to more clearly distinguish the performance improvements brought by domain-specific training data, long audio inputs, and representation architecture.

2: The actual contribution of MMOT is relatively weak, which does not align well with the paper's emphasis on explicit musical reasoning. MMOT is one of the key designs introduced in the paper, which allows the model to first generate structured music analysis and then use the analysis information for embedding construction. But the main experimental results in Table 1 show that for the best-performing Embedding Head architecture:

  • Without using MMOT: average R@10 is 44.34%.
  • Using MMOT: average R@10 is 44.48%.

The improvement is only 0.14 percentage points. The counterfactual accuracy also only increased from 93.58% to 93.93%. The authors have already acknowledged that these changes do not have clear statistical significance and have pointed out in the conclusion that direct EH encoding remains attractive. This means that, under the primary model configuration, explicit music analysis has not been proven to be a significant source of performance. Although MMOT shows more significant benefits under the EOS-LoRA architecture, its necessity for the paper's optimal model is currently not sufficient. Therefore, the paper needs to more clearly answer: Is explicit music analysis a key methodological contribution, or is it an optional module whose effectiveness depends on the specific architecture?

Questions for authors

Question 1: Can the roles of long audio processing, domain SFT, and Embedding Head be further separated? The current system simultaneously includes multiple important designs. It is recommended to provide a unified ablation matrix, controlling for domain adaptation, embedding head, audio length, and MMOT under as similar training data and computational budget as possible. Especially, we would like to understand how much benefit MUUNE still has compared to a strong baseline trained on the same data when the audio length is standardized to 30 seconds or 120 seconds.

Question 2: Does MMOT really provide useful information beyond the audio itself? In Table 1, EH using MMOT only improves by 0.14 percentage points. Can you provide grouped experiments based on different musical attributes, such as rhythm, tonality, instrumentation, vocals, and overall musical structure? If MMOT primarily enhances a few complex queries that require explicit music theory analysis, then the authors should clearly define this scope of application, rather than relying mainly on average scores.

Question 3: Does the performance improvement for longer audio indeed come from modeling cross-temporal relationships? Table 3(b) demonstrates that longer audio is beneficial for retrieval, but it does not alone prove that the model understands cross-temporal structures. For example, a 300-second input might just contain more local information available for matching, which does not necessarily mean the model has learned the structural relationships between different segments. It is recommended to add inputs of equal length but with the segments shuffled, or to use average pooling of multiple short segments for comparison. If MUUNE still has an advantage under these controlled conditions, it would more strongly support the conclusion of long-term music understanding.

4 comments

Thank you for recognizing our methodology and data-adapted comparisons. We address component contributions, MMOT, and long-audio modeling below.

  1. Data, domain adaptation, and representation design (Weakness 1; Question 1)

MUUNE contributes an integrated framework, rather than attributing all gains to EH alone. Our experiments separately examine its components:

Data: Table 1 adapts CLAP and MuQ-MuLan with the same contrastive data. With CL-Full, MuQ-MuLan reaches 28.39% mean retrieval, versus 44.34% for MUUNE-EH without MMOT, supporting the framework beyond contrastive data alone.

Domain adaptation: Table 3(a) fixes backbone initialization, the eight-layer EH, and CL-Subset data. Mean retrieval increases from 33.63% without music SFT to 37.92% with SFT-Subset and 42.42% with SFT-Full.

Representation: EOS-LoRA and EH each have approximately 403M trainable parameters. Without MMOT, EH improves mean retrieval by 5.24 percentage points, supporting independent representation optimization at comparable adaptation capacity.

Audio coverage: Table 3(b) fixes weights, queries, candidates, and scoring. Extending inference audio from 30 to 300 seconds improves mean retrieval by 10.95 points without retraining.

A unified matrix would further examine interactions; existing ablations already provide component-level evidence. We will consolidate their control conditions. The duration ablation assesses coverage within MUUNE, not cross-model performance at equal durations. Matching contrastive data also does not equalize pretraining, SFT, or inference costs.

Copyright prevents full release of training audio. We are working toward releasing code and weights and pursuing internal legal approvals. We will also provide construction protocols and public evaluation resources.

  1. MMOT and its scope (Weakness 2; Question 2)

Music SFT incorporates domain knowledge into parameters; MMOT generates explicit analysis at inference. Table 3(a) uses direct encoding without MMOT, so modest MMOT gains do not diminish the demonstrated contribution of music adaptation.

MMOT is optional and architecture-dependent, not essential for strong retrieval. We believe its modest gains here reflect that musical attributes are already present in audio. Music-adapted EH can extract them directly, leaving limited marginal gain from expressing them as text. MMOT nevertheless offers:

(1) Explicit attributes alongside implicit representations: inspectable structured outputs reusable in downstream tasks.

(2) Diagnostic support: generated analysis allows inspection of attribute predictions. Table 3(c) shows that mismatched metadata substantially degrades a model trained with MMOT, demonstrating its use of musical semantics.

Attribute-grouped analysis would further clarify this scope. Our additional exploration finds a 5.92-percentage-point R@5 gain in instruction-guided retrieval, such as finding songs with instrumentation similar to a reference song. We also observe benefits in identifying users who may enjoy a song through audio understanding. These tasks require instruction-specific selection and organization of information, allowing fuller use of MLLM reasoning. We plan to detail these results in future work.

  1. Longer audio and temporal structure (Question 3)

Following your suggestion, we compared original and shuffled segment orders at equal audio durations. Seven-task mean retrieval changes as follows:

120 seconds: 43.70% to 42.33% (-1.37 percentage points). 300 seconds: 44.49% to 42.68% (-1.81 percentage points).

The modest decreases suggest that current retrieval primarily depends on musical attributes and their coverage, rather than strict temporal structure. Our queries mainly describe song-level style, vocals, mood, tempo, and instrumentation, not their temporal locations. Shuffling preserves most matching cues despite changing their order.

Our metadata analysis indicates that many global attributes are identifiable within 120 seconds; later gains arise more from instrumentation and arrangement. Longer audio therefore benefits retrieval through broader coverage, without requiring strong order sensitivity. Full-song retrieval here means using more complete song information within a unified representation, not a claim to understand all temporal relationships. We plan to explore chorus-specific instrument queries, full-song lyric retrieval, and segment-level captioning to assess temporal localization more directly.

Thank you again for your careful review. If further clarifications would help you reassess our work and consider a higher score, please let us know. We would be very grateful.

Dear Reviewer, Thank you again for your thoughtful review. We would like to kindly follow up on our response, which includes an additional segment-shuffling experiment addressing your question about temporal structure. We would greatly appreciate any further feedback on whether our clarifications and new results address your concerns. If any points remain unresolved, we would be happy to provide further clarification. Thank you for your time and consideration.

Thank you for the response and the shuffled-segment experiment. The clarified ablations address my main concern on attribution, and I am raising my score to 3.

Please reflect the following in the final version: present MMOT as optional; state that long-audio gains come from coverage rather than temporal modelling; and report that the largest gains fall on the three tasks whose queries follow the training pipeline.

Thank you for considering our rebuttal and additional experiments, and for updating your assessment. We are glad that the clarified ablations addressed your main concern about attribution. We will incorporate your constructive feedback into the final version to improve the paper’s clarity and precision. Thank you again for your time and response. We wish you all the best in your research and future endeavors.

Reviewer_MsWYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MUUNE, which is a framework for full-song fine-grained music retrieval using MLLMs. Their approach consists of a multi-view data construction pipeline (multi-view music-text pairs), music-specific SFT, and contrastive representation learning. The paper shows two embedding techniques: EOS-LoRA as well as EH (Independent embedding head). The paper also shows experiments to investigate the usefulness of explicit musical analysis using MMOT (Music-Meta-of-Thought), which is similar to Chain-of-Thought (CoT) in spirit. Experiments by the authors show that MUUNE-EH with MMOT scores 44.48% mean Retrieval@10 accuracy and 93.93% counterfactual accuracy, which is better than the baselines compared. Authors also do more ablations to show the benefit of their music knowledge adaptation stage, longer audio inputs as well as independent embedding learning.

Critical strengths
  1. I liked the overall methodology of the paper. Especially their music meta of thought and their dataset construction framework and their comparisons between EOS-LoRa and the independent embedding head techniques. I also liked the use of the multi-view queries and atomic counterfactuals, as they actually help in evaluating whether the embeddings capture specific musical attributes or not.

  2. I feel the paper has detailed experiments to support the work, and the ablation studies also give useful insights. For example the controlled audio duration experiment helps show that there's a 10.9% improvement when extending the input from 30 seconds to 300 seconds. This is an interesting finding, which tells us longer audios do help performance and helps quantify the improvement.

Critical weaknesses
  1. The usefulness of MMOT (Music-Meta-of-Thought) is not very strongly supported by the results. Table 1 shows that adding MMOT to the EH (independent embedding head) improves performance by only 0.14%, which is almost negligible. And adding MMOT to the EOS-LoRA is also 1.8% improvement only.

  2. The paper states that it supports full-song fine-grained retrieval. But Table 3 shows that when we increase the duration of the audio from 120 seconds to 300 seconds, the mean retrieval performance changes from 43.45% to 44.33%. So the benefits of longer audio inputs are not so clear beyond 120 seconds. It is not clear how effectively and how well the model captures information distributed throughout longer songs.

Questions for authors
  1. As I said, Table 3 shows a 0.88% (which is 43.45% to 44.33%) improvement when extending audio from 120 to 300 seconds. So I would like the authors to clarify or conduct some experiment to see if this is due to diminishing useful information present in later segments of a song or is it a limitation of the model in its ability to capture long-range musical information?
3 comments

Thank you for your positive assessment of our work. We would like to further clarify the meaning of fine-grained and full-song retrieval, the role of MMOT, and the benefits of longer audio inputs.

Fine-grained and full-song retrieval. Compared with conventional music captions, our retrieval texts cover a richer and more detailed set of musical attributes, including style, vocals, mood and listening context, melody, tempo, and instrumentation. For example, style labels are further refined into substyles, while instrumentation and related attributes are described using more detailed labels. “Full-song” also refers to a distinction in the encoding approach: rather than encoding short audio segments—typically around 10 seconds—and pooling their representations, as in CLAP-style methods, MUUNE encodes long audio within a unified MLLM sequence. This preserves more complete song context and captures fine-grained information that may be absent from individual short segments.

The role of MMOT. We attribute the modest improvement from MMOT in the current retrieval task primarily to the fact that musical metadata, such as style, are already present in the audio signal. After music knowledge adaptation, the model can extract this information directly from audio, leaving relatively limited marginal information gain from explicitly generating it as text for caption-based music retrieval. Nevertheless, the value of MMOT extends beyond improvements in the average retrieval score in two respects:

  1. Providing both explicit musical attributes and implicit retrieval representations. The structured attribute outputs are inspectable and can also be used directly in other downstream tasks.
  2. Supporting diagnosis of music understanding and retrieval representations. The generated analysis allows us to inspect whether the model identifies the correct musical attributes. For example, Table 3(c) shows that, for a representation model trained with MMOT, replacing matched metadata with mismatched metadata substantially degrades retrieval performance, demonstrating that the model uses these musical semantics.

We have also explored the role of explicit textual reasoning in additional tasks. In instruction-guided attribute-similarity retrieval—for example, “find songs with instrumentation similar to this song”—MMOT yields a 5.92-percentage-point improvement in R@5. We also observe substantial benefits in a recommendation task that uses audio understanding to identify users who may enjoy a given song. These tasks require selecting and organizing musical information according to specific instructions, allowing the MLLM’s reasoning capabilities to provide additional benefits. We plan to present these results in detail in future work.

The gains from 120 to 300 seconds. Our metadata analysis indicates that attributes such as style, vocals, mood and listening context, melody, and tempo can typically be identified within the first 120 seconds. Extending the input to 300 seconds provides additional benefits primarily through instrumentation and arrangement information appearing later in the song. Thus, the diminishing returns in the current song-level attribute retrieval setting are related to how different attributes are distributed throughout a song; they do not imply that later audio contains no valuable information.

Motivated by this observation, we plan to introduce tasks that place greater demands on temporal localization and full-song coverage, such as “find songs that feature a saxophone in the chorus,” lyric retrieval across the entire song, and segment-level music captioning. These tasks will more directly exploit information from different parts of a song and further strengthen fine-grained full-song music understanding.

Reviewer_MsWYOpenReview ↗

Response to Author's rebuttal

I thank the authors for their detailed rebuttal, and for the clarifications provided by them. It addresses my concerns. I will maintain my rating at 3.

Thank you for your thoughtful review and for taking the time to consider our rebuttal. We are glad that our clarifications addressed your concerns, and we sincerely appreciate your constructive feedback. We wish you all the best in your research and future endeavors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.