acceptodds
Under review as a conference paper at ICLR 2027

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Abstract

Symbolic models make musical composition explicit, while audio models generate complete recordings with composition largely implicit. We introduce YuE2, which unifies symbolic and audio music generation in one model at frontier song quality. Through symbolic planning, the model writes an editable melody-and-chord score, expands it into semantic music tokens, and renders full-song audio. The score specifies musical structure while leaving performance details to audio generation. Automatically constructed score and semantic targets let the model learn this generation process from recordings without pre-existing aligned scores. On WildSongBench, YuE2 scores 6.73 on SongBench (Wu et al., 2026) Global Avg with a two-candidate budget, exceeding all evaluated public baselines. YuE2 (best-of-8) achieves 6.96, the highest observed mean among all evaluated systems. In expert listening, YuE2 (best-of-8) receives 57.3% of overall preferences versus 30.5% for Suno v4.5 (Team Suno, 2025), and 40.4% versus 39.9% for Suno v5 (Suno, 2025). Within the same checkpoint, symbolic planning improves overall quality and musicality: 49.3% of overall judgments favor planning versus 34.6% without it. The generated audio follows the particular planned melody and harmony; score edits change the requested content while largely preserving unedited musical material. Without cover-specific training, the same checkpoint generates zero-shot covers from full scores of 948 unseen works, surpassing both evaluated cover systems on all eight work-identity retrieval metrics. This score-to-audio mapping also enables agentic music editing. An external language-model agent turns user feedback into score, style, and lyric revisions, which YuE2 renders as complete songs. Audio examples: https://yue2-0001.github.io.

Primary area:
generative models
Keywords:
music generation, symbolic music generation, audio generation, mixture-of-transformers, flow matching, multimodal generative models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NfizOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces YuE2, which generates an editable melody-and-chord score, semantic music tokens, and acoustic latents to produce complete songs. It combines autoregressive discrete prediction and non-autoregressive acoustic generation in a unified architecture. The paper reports expert-listening comparisons of symbolic planning and unified versus separate generation, and demonstrates editing and cover generation through the same score-to-audio interface.

The direction is promising, but I believe the main paper requires major revisions before I can confidently assess the methodological contribution. My principal concerns are the explanation of the core architecture, the motivation for its design choices, and the specification of the comparison baseline. A more focused presentation should clarify the gaps in existing approaches and how the proposed method addresses them. On balance, I recommend weak reject in the current form.

Critical strengths
  1. Explicit score planning offers an interesting route to controllable and editable full-song generation.
  2. The expert-listening study provides useful evidence about perceived quality. In particular, the within-checkpoint comparison with and without planning controls several important factors and supports a benefit from planning in the evaluated setting.
  3. The planned public release of model weights and WildSongBench, together with open-source evaluation code, would be a meaningful contribution to the research community and support further study and reproducibility.
Critical weaknesses

1. Lack of clarity, clutter, and excessive breadth

I found the paper really difficult to follow. In its current form, it reads more like a technical report than a clearly developed research paper: it presents numerous components, benchmark results, and applications without giving the reader a sufficiently clear account of the central method and the reasoning behind it. Generation, transcription, representation learning, planning, editing, covers, and agentic editing compete for attention in a short main text. This breadth makes it difficult to distinguish the essential contribution from supporting infrastructure and demonstrations.

For example, Section 2.2 introduces MERT2, SheetSage2, and a separate tokenizer branch, then moves immediately to transcription benchmark performance. Their role in producing training targets is stated, but the construction and dependencies of this supervision pipeline are too compressed to understand adequately. This is a substantive presentation problem because it impedes assessment of how the proposed system is trained and which components account for its contribution.

2. Insufficient detail about the method, problem, and context in prior work

Figure 2 labels a central operation as “hybrid self-attention,” but the main text does not explicitly define that operation. Section 2.2 describes causal attention for discrete positions and attention to conditioning tokens and other acoustic positions for acoustic generation. However, translating these statements into a concrete layer computation requires substantial reconstruction by the reader. An explicit attention mask and an explanation of how the AR and NAR projections and experts operate together are needed to understand this core architectural component.

The introduction states the high-level goal of composing an editable score and rendering a complete recording. However, the specific limitations of existing approaches, the motivation for the proposed architecture, and the rationale for the intermediate semantic representation are not developed sufficiently. The explicit related-work discussion appears near the end of the paper, rather than providing context for the method as it is introduced. Consequently, I find it difficult to identify precisely what scientific insight the proposed combination establishes beyond the capabilities of the resulting system.

3. Important comparisons are insufficiently specified or difficult to interpret

The unified-versus-separate comparison in Section 3.2 reports comparable total parameter counts (3.58B versus 3.4B), shared training-data volume, matched AR token budgets, batch sizes and update counts, and 900,000 DiT updates. These are useful controls. However, the main text does not specify how parameters are partitioned between discrete prediction and acoustic generation in either system, or adequately describe the separate baseline's architecture and conditioning interface. Comparable total parameter counts alone do not establish comparable capacity for each stage. The information given is insufficient to assess whether the preference difference is attributable to unified modeling or to baseline architecture, capacity allocation, or optimization.

Questions for authors
  1. Can you explicitly define the hybrid attention operation, including the attention mask and the computation performed by the AR and NAR experts within a layer?
  2. How are parameters allocated to discrete prediction and acoustic generation in YuE2 and the separate baseline? What motivates the baseline's allocation, and what evidence establishes that it is adequately optimized?
  3. Can you explain the supervision pipeline and the necessity of its main components in the main text, and position the proposed method against existing approaches before presenting the experiments?
  4. How should the best-of-eight results be interpreted relative to baselines with a smaller candidate budget and a different selection rule? Can you provide a comparison under matched candidate budgets and selection criteria?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Nfiz (1/4): clarity and hybrid attention (W1, W2, Q1)

W1. Clarity and breadth.

Thank you for the careful review and for recognizing the value of explicit score planning and of the within-checkpoint comparison. Below we answer each weakness and question, including the explicit definitions you requested.

The central question of this work is whether explicit symbolic planning can improve the quality of full-song generation while providing an editable interface for composition. YuE2 addresses this by jointly modeling the score, semantic tokens, and acoustic latents, as formalized in Eq. (1) (Sec. 2.1, p. 3). The score connects these two goals: it serves as a generation plan and as the interface through which users can inspect and modify melody and harmony.

The components and evaluations have distinct roles in this framework. MERT2 and SheetSage2 construct offline training targets from recordings without aligned scores. Creation, editing and covering then use the same learned score-to-audio mapping, as stated in Sec. 2.3 (p. 4): the score is generated, revised or transcribed, respectively. The generation experiments measure song quality; editing tests whether changes to the score produce the intended musical changes; covering tests the mapping on unseen compositions. Agentic editing demonstrates how an external agent can operate this same score interface.

The within-checkpoint comparison directly tests the role of symbolic planning. As reported in Sec. 3.3 (p. 6), it holds model weights, prompts, lyrics, the two-candidate budget, selection rule and decoder fixed. Experts prefer planning for overall quality (49.3% versus 34.6%, p=0.0070) and musicality (45.0% versus 29.4%, p=0.0141). These results provide empirical support for explicit score planning as a modeling choice for full-song generation.

W2 and Q1. Hybrid attention.

By “hybrid self-attention,” we mean attention over discrete and acoustic positions with type-specific projections and different visibility patterns. As described in Sec. 2.2 (pp. 3–4) and Fig. 2 (p. 3), discrete positions use the AR expert and acoustic positions use the NAR expert. Each expert has its own normalization, Q/K/V/output projections and feed-forward network. Their projected queries, keys and values participate in one attention computation: discrete queries follow a causal mask, while acoustic queries can attend to the completed discrete prefix and all acoustic positions of the same song.

Let DD denote valid discrete positions (text/lyrics, score and semantic tokens), AA valid acoustic positions, and bib_i the song ID of position ii in a packed sequence. Expert routing is fixed by position type: e(i)=ARe(i)=\mathrm{AR} for i∈Di\in D, and e(i)=NARe(i)=\mathrm{NAR} for i∈Ai\in A. The additive attention mask is

0,&b_i=b_j,\quad i,j\in D,\quad j\le i,\\\\ 0,&b_i=b_j,\quad i\in A,\quad j\in D\cup A,\\\\ -\infty,&\text{otherwise}. \end{cases}$$ For a layer input $h_i$, the computation is as follows. For readability, layer and head indices, grouped-query head mapping, RoPE and per-head Q/K normalization are suppressed; the attention output $a_i$ includes head concatenation before the output projection. All normalization, projection and feed-forward parameters carry the indicated expert index. $$\begin{aligned} \tilde h_i &= \mathcal N_{\mathrm{attn}}^{e(i)}(h_i),\\\\ q_i &= W_Q^{e(i)}\tilde h_i,\qquad k_i = W_K^{e(i)}\tilde h_i,\qquad v_i = W_V^{e(i)}\tilde h_i,\\\\ \alpha_{ij} &= \operatorname{softmax}_{j}\\!\left(\frac{q_i^\top k_j}{\sqrt{d_h}}+M_{ij}\right), \qquad a_i = \operatorname{Concat}_{\mathrm{heads}}\\!\left(\sum_j\alpha_{ij}v_j\right),\\\\ h'_i &= h_i + W_O^{e(i)}a_i,\\\\ h_i^{\mathrm{next}} &= h'_i + \operatorname{FFN}^{e(i)}\\!\left(\mathcal N_{\mathrm{ffn}}^{e(i)}(h'_i)\right). \end{aligned}$$ Here $\mathcal N$ denotes expert-specific RMSNorm and $d_h$ the head dimension. An acoustic query can therefore read discrete keys/values produced by the AR expert and acoustic keys/values produced by the NAR expert. The output projection and feed-forward network use the query position's expert. Discrete queries cannot read acoustic targets, and different packed songs cannot attend to one another. The implementation has 28 layers, 16 query heads, eight KV heads and head dimension 128; the discrete head predicts tokens, while the acoustic head predicts flow velocity for all frames in parallel.
AuthorsOpenReview ↗

Response to Reviewer Nfiz (2/4): positioning, supervision and the semantic intermediate (W2, Q3)

W2 and Q3. Positioning, supervision pipeline and the semantic intermediate.

The problem stated in the Introduction is to connect an editable composition with a complete audio realization. Relative to symbolic-only generation, the target includes the finished recording. Relative to generation in which composition remains implicit in audio tokens, the score exposes melody, harmony and form for inspection and intervention. Recent planning stages use audio–text embeddings (MusiCoT) or global attributes such as tempo, key and structure (ACE-Step 1.5) rather than a readable score. Controllable models (Music ControlNet, JASCO) render controls supplied by the user rather than composing them, and Seed-Music handles lead sheets through a separate symbolic pipeline. YuE2 composes a readable full-song melody-and-chord score and renders it within one model, which also makes a within-checkpoint test of planning possible. The unified architecture models the score, semantic sequence and acoustic latents through the factorization in Eq. (1) (Sec. 2.1, p. 3), allowing the acoustic stage to condition directly on text, score and semantic tokens.

The supervision pipeline is constructed offline from recordings (Sec. 2.2, p. 4; Appendix C.1, pp. 18–19). Two separately adapted branches share the same pretrained MERT2 foundation. The full-song, bidirectional branch provides features to SheetSage2-AR, which transcribes the symbolic score ss. The semantic-tokenizer branch uses causal self-attention and a 32,768-entry quantizer to produce 25-Hz tokens cc. A separately trained VAE encodes the same recording into continuous acoustic latents zz. These three targets supervise YuE2's score, semantic and acoustic streams. During text-to-song generation, YuE2 generates s,c,zs,c,z itself; MERT2 and SheetSage2 are not run.

The semantic intermediate is a practical design choice supported by prior work and our own experiments. Qwen-Music, HeartMuLa and LeVo provide successful precedents for autoregressive modeling of semantically rich discrete music representations followed by acoustic rendering. Direct audio-latent diffusion is also possible, as in ACE-Step 1.0 and DiffRhythm; as a reference for training scale, ACE-Step 1.0 reports 460,000 pretraining updates and 240,000 fine-tuning updates (Appendix A, Table 4). In an internal study, we trained two generation tasks. Both first generate a score from text. One then predicts semantic tokens before generating VAE latents, while the other generates the latents directly from text and score. We evaluated both tasks at the same 36K checkpoint on 30 vocal prompts with matched seeds and identical acoustic inference and decoding settings. Median ASR-based phoneme error rate was 3.12% with semantic tokens and 46.30% on the direct route. We therefore use semantic tokens to improve lyric fidelity within our training budget and retain the score as an editable compositional plan.

AuthorsOpenReview ↗

Response to Reviewer Nfiz (3/4): parameter allocation and the separate baseline (W3, Q2)

W3 and Q2. Parameter allocation, baseline design and optimization.

Capacity allocation and baseline design. YuE2 allocates 1.409B parameters to the autoregressive Transformer blocks that predict score and semantic tokens, and 1.409B to the non-autoregressive blocks that generate acoustic latents. The token embedding and untied output head add 756.55M, and the latent projections, timestep MLP and final normalization add 4.99M, for 3.580B in total (table below). The separate baseline follows the LM+DiT design of Qwen-Music. Its LM uses the same Qwen3-1.7B configuration as YuE2's autoregressive expert (28 layers, width 2048, FFN 6144, 16 query and 8 KV heads) and predicts the same score and semantic tokens. Sec. 3.2 names this LM by its Qwen3-1.7B backbone; with the same 184,704-token vocabulary and untied output head as YuE2, it has 2.166B parameters, exactly matching YuE2's discrete stage. Its 1.7B DiT follows the Qwen-Music renderer: the LM's semantic tokens enter frame-aligned with the noisy VAE latents, text and lyrics enter through cross-attention in every block, and the diffusion timestep enters through AdaLN. The baseline therefore totals about 3.87B parameters, versus 3.58B for YuE2. The DiT receives the score through the semantic tokens that the LM generates from it, together with separately encoded text and lyrics. In YuE2, the acoustic expert attends directly to text, score and semantic hidden states at every layer, and both experts are trained jointly.

ComponentParameters
AR Transformer blocks, including their norms1,409,408,000
NAR Transformer blocks, including their norms1,409,408,000
Input embedding and untied output head756,547,584
Latent projections, timestep MLP and final norm4,988,992
Total3,580,352,576

The table reports trainable parameters in the YuE2 generator. Each Transformer-block count includes its normalization layers.

Baseline training and optimization. Both systems use the same melody-and-chord planning, semantic tokenizer and acoustic VAE, and we matched training-data volume across them. For discrete prediction, the baseline LM and YuE2's AR expert used the same token budget, batch size and number of updates. The separate acoustic DiT received 900,000 updates (Sec. 3.2 and Appendix D.1). For context, LeVo trained each of its token-conditioned acoustic DiTs for 300,000 steps (its Appendix B.2.1), and ACE-Step 1.0 used 700,000 steps in total, as noted above. The baseline was competitive on text alignment in the blind listening test: of 206 judgments, 69 favored the baseline, 70 favored YuE2 and 67 were ties.

Results. Each system generated two candidates for each of the same 192 prompts, and the one with the lower phoneme error rate (PER) was evaluated. In blinded listening tests, 53.4% of overall-quality judgments (111 of 208) favored YuE2 and 35.6% favored the separate baseline (p = 0.0084). For audio quality, the shares were 48.5% and 29.6% (p = 0.000306). The remaining judgments were ties, and both tests accounted for clustering by evaluator and song. Because selection used only PER, we could assess the selected outputs separately with SongEval: across 192 paired prompts, mean SongEval was 4.270 for YuE2 and 4.109 for the baseline, a paired difference of +0.162 (95% confidence interval clustered by prompt [0.116, 0.208], Holm-adjusted p = 4.17 × 10⁻¹⁰ across nine metrics). Expert preferences and an automatic metric excluded from selection therefore agree, against a baseline with more parameters in total than YuE2.

Reviewer_8pzuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a symbolic planning method YuE2, that learns to predict music score of a song to aid final audio song generation from text prompt. Their symbolic planning converts a melody to chord from score and thus learn score-to-audio mapping with better quality. Two separate QKV transformers are trained for AR and NAR respectively, where AR associates prompt y with score s then with semantic token c, while NAR associates noisy zt to acoustic latent z. Finally the VAE converts the acoustic latents into generated audio. It reaches higher performance in terms of full-song frontier quality against many sota baseline models including SUNO v4.5, SUNO v5.0.

Critical strengths

The paper proposes a novel idea to incorporate music score as intermediate representation in music generation that can not only support better music generation but also have the merits for subsequent user or agent revision on the generated music.

The experimental evaluations are rigorous and comprised of various recent public released models such as ACE-step1.5, HeartMula, etc. and proprietary ones such as SUNO v5, v6.and Mureka.In terms of song quality it rank highest in SongBench. They adopt substantial computational power to train the complex model by combining text prompt, music score, semantic token embedding as well as audio generation from

Critical weaknesses

In section 2.2, the paper writes”SheetSage2-AR (hereafter SheetSage2) uses MERT2’s full-context representations to recover melody, chords, rhythm, and form as a readable score.” However, in Figure 2 shows SheetSage2 and MERT2 are separate processes.

In Figure 3, the comparative experiments show the system outperforms the baseline models at music quality, however not the overall performance if we ignore SUN 4.5 comparison. The guiding by music score seems to improve music quality but restrict the text alignment of the music generation. The experiments seem to show some contradictory results on different set-up comparisons.

Questions for authors

How is the efficiency of music generation of the model against other baseline models, if users have only light weight facility?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8pzu (1/2): model summary, Figures 2–3 and text alignment

Thank you for recognizing the value of score-based generation and editing and the breadth of our evaluation. YuE2 jointly plans melody and chords in an ABC lead sheet. Its AR and NAR experts are jointly trained within one 28-layer backbone, with expert-specific normalization, QKV/output projections and feed-forward networks connected by a shared attention computation (Sec. 2.2; see also our response to Reviewer Nfiz, 1/4).

W1. MERT2 and SheetSage2 in Figure 2.

The two offline-target boxes in Figure 2 come from two branches derived from MERT2 (Appendix C.1). The full-song branch, MERT2-FS, is adapted using bidirectional context on complete recordings and serves as the encoder inside SheetSage2-AR. The semantic-tokenizer branch uses causal attention and a 32,768-entry quantizer to produce 25-Hz semantic tokens. Thus, the “SheetSage2” box includes its MERT2-FS encoder, while “MERT2 + CVQ” denotes the semantic tokenizer. We will state this explicitly in the Figure 2 caption. Both branches prepare training targets offline. During text-to-song generation, YuE2 predicts the score and semantic tokens itself. For cover generation, SheetSage2 transcribes the reference recording.

W2. Figure 3, text alignment and the different comparisons.

The six systems in Figure 3 are commercial flagship products, Mureka 9 and five Suno versions, rather than research baselines; among publicly released models, YuE2 ranks first on SongBench (Table 1a). Excluding Suno v4.5, YuE2 with eight candidates has nearly equal wins and losses in overall quality against Mureka 9 (83 vs. 87), Suno v5 (86 vs. 85) and Suno v6 Wild (87 vs. 98). Suno v5.5 is narrowly preferred (44.2%, p = 0.052), and only Suno v6 is significantly preferred (36.1%; Table 19). On audio quality, YuE2 with eight candidates is significantly preferred over Mureka 9 (65.8%), Suno v4.5 (68.1%), Suno v5 (56.1%) and Suno v6 Wild (60.6%), and no commercial system is significantly preferred over YuE2. A 3.58B model thus matches or exceeds four of these six commercial flagships in overall quality and exceeds most of them in audio quality.

Score guidance does not restrict text alignment. The planning comparison holds the YuE2 checkpoint, prompts, lyrics, decoder, two-candidate budget and lower-PER selection fixed (Sec. 3.3). With planning, YuE2 first generates a melody-and-chord score; without it, YuE2 generates semantic tokens and audio directly from the prompt and lyrics. In both cases the semantic and acoustic stages condition directly on the original text (Eq. 1), so the score adds a musical plan rather than replacing the text condition. Experts preferred the planned songs for overall quality (49.3% vs. 34.6%, p = 0.0070) and musicality (45.0% vs. 29.4%, p = 0.0141; Figure 4). For text alignment they also favored planning, by 78 judgments to 55 with 78 ties (Table 19).

Automatic text-alignment metrics point the same way: under the standard two-candidate protocol, YuE2 with planning ranks first among publicly released models on MuLan, AllMusicCaps and Q3O (Table 1a).

These results are consistent. Each comparison isolates a different factor, and each favors YuE2's design. Within one model, planning improves overall quality and musicality without lowering text alignment (Figure 4, Table 19). With planning in both systems, experts prefer the unified backbone over a separate LM+DiT on five of six criteria (overall 58.9%, p = 0.0084), with text alignment balanced at 70 vs. 69 votes and 67 ties (Figure 6, Appendix D.1). Against commercial products, which differ in training data and inference pipelines, YuE2 leads in audio quality and matches or exceeds most of them overall (Figure 3). None of these comparisons shows planning trading text alignment for quality.

AuthorsOpenReview ↗

Response to Reviewer 8pzu (2/2): efficiency on a consumer GPU against public baselines

Q. Efficiency on lightweight hardware, compared with baselines.

On a single consumer RTX 4090 (24 GB), YuE2 is the fastest of the four models we measured and the only one that runs with all modules resident on the GPU. ACE-Step 1.5, HeartMuLa 3B and LeVo 2 run out of memory in that setting and fit only with their documented memory-saving modes. Each model generated 20 songs from the same ten prompts (five Chinese, five English, two seeds each); we excluded one warmup song per model and timed generation from text to the final waveform. RTF is generation time divided by song duration.

ModelMean seconds/songMean RTF ↓Median song durationPeak allocated memoryConfiguration
YuE274.40.340217 s8.99 GiBAll modules resident on GPU
ACE-Step 1.5, XL-SFT + 4B LM85.20.415206 s18.09 GiBLM, VAE and text-encoder CPU offload
HeartMuLa 3B118.90.763157 s12.93 GiBModules loaded on demand
LeVo 2218.21.259177 s18.66 GiBOfficial low-memory mode with layer offload

YuE2 has the lowest RTF on all 20 prompt–seed pairs against every baseline; the geometric means of the paired baseline-to-YuE2 RTF ratios are 1.22 (ACE-Step 1.5), 2.25 (HeartMuLa) and 3.71 (LeVo 2). Its peak allocated memory, 8.99 GiB, is under half that of ACE-Step 1.5 and LeVo 2. Score planning takes 13.4 s of YuE2's 74.4 s per song (1,902 tokens on average).

Settings: YuE2 used BF16 generation weights, FP32 VAE decoding and 32 midpoint ODE steps; ACE-Step used its PyTorch LM backend and 50 diffusion steps; HeartMuLa used its lazy-loading mode, and its module reloads (3.85 s per song on average) are included; LeVo 2 used its official low-memory path (CFG 1.5, temperature 0.9, top-k 50). Timings exclude one-time model initialization, file saving and LeVo 2's one-time switch from token generation to decoding.

Reviewer_R8W6OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

YuE2 generates songs by first generating a symbolic musical score, conditioned on the music style attributes and lyrics. Generates melody and chords conditioned on that, which are further used to generate a latent variable which is decoded into a song.

Critical strengths
  1. The approach of planning in the minimal ABC symbolic space and generating music conditioned on those leads to output with high musicality.
  2. Due to the symbolic nature of the representation, editing capabilities are tested through intervention.
  3. The ablation is promising 49.3% versus 34.6% for with and without planning.
Critical weaknesses
  1. The bo8 comparison is made without appropriate comparison of compute costs with other systems. Also, the improvement in performance due to this is low.
  2. SheetSage2 is used to get training annotations and is also used to evaluate adherence.
  3. Songbench musicality score is used to filter out outputs in bo8 setting, and is also used as a comparison metric in Table 1.
Questions for authors
  1. Have the authors performed an inference-time compute scaling experiment? It would be interesting to understand the variation of musicality score with varying planning compute budgets?
  2. Do you think doing planning with other types of symbolic representations like Midi would be helpful? In what ways? What are the trade offs of midi vs ABC?
2 comments
AuthorsOpenReview ↗

Response to Reviewer R8W6 (1/2): best-of-8 compute and SheetSage2 circularity

Thank you for recognizing the musicality benefit of ABC planning, the intervention-based editing tests and the within-checkpoint planning ablation. The ABC score itself contains the melody and chords; conditioned on it and the text, YuE2 predicts semantic tokens and then acoustic latents, which the VAE decodes into audio (Eq. 1).

W1. Compute cost of best-of-8, and the size of its gain.

YuE2's lead over public models is established at a matched budget: every system generates two candidates per prompt and keeps the one with the lower phoneme error rate (PER) ("Bo2"; Sec. 3.1, App. E.3). With Bo2, YuE2 has the highest SongBench Global Avg among public systems (6.73, Table 1a) and exceeds every public baseline on both indices of Fig. 1.

Best-of-8 (Bo8) uses four times as many candidates. Measured on one RTX 4090 with the same prompts (response to Reviewer 8pzu, 2/2), the generation cost of each setting is:

SettingCandidatesRTF per candidateGPU seconds per second of audio
YuE2 (Bo2)20.3400.68
ACE-Step 1.520.4150.83
HeartMuLa 3B20.7631.53
LeVo 221.2592.52
YuE2 (Bo8)80.3402.72

GPU time per second of audio is the number of candidates times the real-time factor (RTF). Candidate scoring is excluded, and the three baselines run in the memory-saving modes they need on 24 GB. At the matched budget YuE2 is the cheapest of the four models, and its eight candidates cost close to LeVo 2's two. With eight candidates each and the same selector, YuE2 exceeds LeVo 2 on SongEval (+0.134, 95% CI 0.042–0.226; eight-prompt pilot, response to Reviewer Nfiz, 4/4), and each YuE2 candidate is generated 3.7 times faster. The proprietary systems do not disclose model size or internal sampling, so their compute cannot be measured; Suno returns two songs per request and Mureka's API defaults to two, which is why Bo2 is our standard setting.

None of our controlled claims depends on Bo8: the comparison with public models and the planning and architecture ablations all use Bo2. Bo8 adds a small but significant gain on SongEval, which the selector does not use (4.263 → 4.296; see W3). In blind expert listening against the six proprietary systems, the two settings receive similar preferences (Table 19, averages over the six systems): 48.0% and 48.4% for overall quality, and 56.6% and 58.9% for audio quality, which favors YuE2 in both settings (p = 0.0105 and 1.7 × 10⁻⁶).

W2. SheetSage2 provides both training annotations and adherence evaluation.

Our adherence results do not rest on raw SheetSage2 output:

  1. Experts correct the transcriptions. In the score–audio agreement study (Sec. 3.4) and the editing study (Sec. 3.5), music experts check and correct every SheetSage2 transcription before any metric is computed.
  2. All conditions share the transcriber. Table 2 compares each recording with its own score and with two controls transcribed the same way: the recording of a different score for the same prompt and lyrics, and a recording generated without a score. A transcriber bias shared by the conditions would raise all three, yet only the corresponding pairs agree: melody similarity is 0.946, against 0.216 and 0.206, and chord similarity is 0.925, against 0.247 and 0.190. In every evaluable case, the correct score–audio pairing is more similar than the swapped one (Sec. 3.4).
  3. Evaluators independent of SheetSage2 confirm the effect of the score. In the editing study, madmom estimates tempo (Sec. 3.5). In zero-shot covers, two version-identification models measure whether the source work is preserved: with the full score, CLEWS and Discogs-VINet mAP are 0.647 and 0.288; without the score, they fall to 0.006 and 0.004, with the same lyrics and target style (Table 4).

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer R8W6 (2/2): selection reuse, compute scaling and ABC vs. MIDI

W3. SongBench Musicality is used both to select Bo8 outputs and to evaluate them.

This reuse applies only to YuE2 (Bo8), whose selection rule is stated in the Table 1 caption and App. E.3. YuE2 (Bo2) selects by lyric PER alone, so its SongBench and Q3O scores involve no selection reuse; it is the YuE2 row of Table 1a and appears next to Bo8 in Table 1b. With Bo2, YuE2 has the highest SongBench Global Avg among public systems (6.73), exceeds 5 of the 7 proprietary systems on the same metric, has higher AudioBox PQ than every proprietary system (8.260; best proprietary 8.254), and has the highest Q3O of all evaluated systems other than YuE2 (Bo8) (4.682).

For Bo8, SongBench Musicality (and hence Global Avg), Q3O and PER take part in selection, so part of its SongBench gain (6.73 → 6.96) comes from the selection itself. We therefore measure the Bo8 gain with evaluators outside the selector: SongEval Musicality 4.215 → 4.251, SongEval Avg 4.263 → 4.296 (paired over the 192 prompts: +0.034, 95% CI 0.011–0.056, p = 0.004) and AudioBox PQ 8.260 → 8.271, while style alignment changes little (MuLan 0.507 → 0.505, AMCaps 0.405 → 0.398). Expert listening does not use SongBench either: listeners compare anonymized recordings in randomized order (App. E.7), and Table 19 reports both settings (see W1).

Q1. Inference-time compute scaling. We ran two analyses with the same checkpoint.

Number of candidates. From the eight planned candidates per prompt behind Table 1, we formed subsets of N = 1, 2, 4 and 8 on all 192 WildSongBench prompts and applied each selector. Bold cells reproduce the two YuE2 settings of Table 1.

NSongBench Mus., PER selectorSongEval Avg, PER selectorSongBench Mus., Bo8 selectorSongEval Avg, Bo8 selector
15.8984.2615.8984.261
25.9084.2636.0554.274
45.9034.2546.1684.285
85.8884.2496.2674.296

What extra candidates improve depends on the selector. With PER selection, lyric accuracy improves (corpus PER 0.115 → 0.054) while musicality stays flat on both evaluators. With the Bo8 selector, SongBench Musicality, a selection criterion, rises steadily (5.898 → 6.267), and SongEval, which the selector does not use, rises from 4.261 to 4.296. Under this selector, going from N = 2 to N = 8 adds +0.022 on SongEval (95% CI −0.002 to 0.046); the Bo2-to-Bo8 gain in W3 (+0.034) also includes the change of selector.

Planning compute at a fixed output budget. With eight outputs per prompt, we varied how many distinct scores are planned: 8 plans × 1 rendering, 4 × 2, 2 × 4 and 1 × 8 (eight prompts, 256 recordings, every output scored without selection). Mean SongEval Musicality was 4.266, 4.238, 4.251 and 4.275, and SongEval Avg 4.311, 4.290, 4.299 and 4.326; no pairwise difference is significant after Holm correction (smallest adjusted p: 0.14 for Musicality, 0.64 for Avg). Planning is about a fifth of a candidate's cost, so reusing one plan for all eight renderings cut generation time per prompt from 558 s to 476 s (−15%) on an RTX 4090, without a measurable loss in this pilot.

Q2. ABC vs. MIDI as the planning representation. MIDI would help when the plan must control multiple tracks or performance, such as velocities and exact note timing. We plan in ABC for three reasons:

  • Compression. ABC sequences are about 38% as long as MIDI-based encodings (ChatMusician; Yuan et al., 2024). A YuE2 score averages about 1,900 tokens per song, against about 5,500 semantic tokens (RTX 4090 benchmark).
  • Native to language models. ABC is a plain string, so YuE2 uses the existing text BPE tokenizer without new vocabulary (Sec. 2.1); MIDI tokenizations such as REMI need their own.
  • Text-native input and output. Frontier LLMs already read and write ABC; over 90% of GPT-4's ABC outputs parse correctly (ChatMusician). An external agent can therefore inspect and revise the plan directly, as in our agentic editing (Sec. 3.7), at a low token cost.

The trade-off is control granularity: ABC fixes compositional decisions and leaves velocities, precise timing and full multi-track arrangement to the audio model (Sec. 2.1). A MIDI-like layer for these would add finer control at the cost of longer sequences and a separate vocabulary.

Comments on the paper

1 comment
~Tongxi_Wang1OpenReview ↗

On the Broader Significance and Community Impact of YuE2

As a researcher working in a related area, I would like to highlight the value of YuE2's open-source contribution. High-quality, openly available music generation models remain important resources for the community, and YuE2 provides a solid foundation for researchers to build upon, reproduce results, and explore new ideas. I believe this kind of substantial open-source effort deserves recognition, as it can benefit the broader research community well beyond the paper itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.