- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces YuE2, which generates an editable melody-and-chord score, semantic music tokens, and acoustic latents to produce complete songs. It combines autoregressive discrete prediction and non-autoregressive acoustic generation in a unified architecture. The paper reports expert-listening comparisons of symbolic planning and unified versus separate generation, and demonstrates editing and cover generation through the same score-to-audio interface.
The direction is promising, but I believe the main paper requires major revisions before I can confidently assess the methodological contribution. My principal concerns are the explanation of the core architecture, the motivation for its design choices, and the specification of the comparison baseline. A more focused presentation should clarify the gaps in existing approaches and how the proposed method addresses them. On balance, I recommend weak reject in the current form.
Critical strengths
- Explicit score planning offers an interesting route to controllable and editable full-song generation.
- The expert-listening study provides useful evidence about perceived quality. In particular, the within-checkpoint comparison with and without planning controls several important factors and supports a benefit from planning in the evaluated setting.
- The planned public release of model weights and WildSongBench, together with open-source evaluation code, would be a meaningful contribution to the research community and support further study and reproducibility.
Critical weaknesses
1. Lack of clarity, clutter, and excessive breadth
I found the paper really difficult to follow. In its current form, it reads more like a technical report than a clearly developed research paper: it presents numerous components, benchmark results, and applications without giving the reader a sufficiently clear account of the central method and the reasoning behind it. Generation, transcription, representation learning, planning, editing, covers, and agentic editing compete for attention in a short main text. This breadth makes it difficult to distinguish the essential contribution from supporting infrastructure and demonstrations.
For example, Section 2.2 introduces MERT2, SheetSage2, and a separate tokenizer branch, then moves immediately to transcription benchmark performance. Their role in producing training targets is stated, but the construction and dependencies of this supervision pipeline are too compressed to understand adequately. This is a substantive presentation problem because it impedes assessment of how the proposed system is trained and which components account for its contribution.
2. Insufficient detail about the method, problem, and context in prior work
Figure 2 labels a central operation as “hybrid self-attention,” but the main text does not explicitly define that operation. Section 2.2 describes causal attention for discrete positions and attention to conditioning tokens and other acoustic positions for acoustic generation. However, translating these statements into a concrete layer computation requires substantial reconstruction by the reader. An explicit attention mask and an explanation of how the AR and NAR projections and experts operate together are needed to understand this core architectural component.
The introduction states the high-level goal of composing an editable score and rendering a complete recording. However, the specific limitations of existing approaches, the motivation for the proposed architecture, and the rationale for the intermediate semantic representation are not developed sufficiently. The explicit related-work discussion appears near the end of the paper, rather than providing context for the method as it is introduced. Consequently, I find it difficult to identify precisely what scientific insight the proposed combination establishes beyond the capabilities of the resulting system.
3. Important comparisons are insufficiently specified or difficult to interpret
The unified-versus-separate comparison in Section 3.2 reports comparable total parameter counts (3.58B versus 3.4B), shared training-data volume, matched AR token budgets, batch sizes and update counts, and 900,000 DiT updates. These are useful controls. However, the main text does not specify how parameters are partitioned between discrete prediction and acoustic generation in either system, or adequately describe the separate baseline's architecture and conditioning interface. Comparable total parameter counts alone do not establish comparable capacity for each stage. The information given is insufficient to assess whether the preference difference is attributable to unified modeling or to baseline architecture, capacity allocation, or optimization.
Questions for authors
- Can you explicitly define the hybrid attention operation, including the attention mask and the computation performed by the AR and NAR experts within a layer?
- How are parameters allocated to discrete prediction and acoustic generation in YuE2 and the separate baseline? What motivates the baseline's allocation, and what evidence establishes that it is adequately optimized?
- Can you explain the supervision pipeline and the necessity of its main components in the main text, and position the proposed method against existing approaches before presenting the experiments?
- How should the best-of-eight results be interpreted relative to baselines with a smaller candidate budget and a different selection rule? Can you provide a comparison under matched candidate budgets and selection criteria?
6 comments
Response to Reviewer Nfiz (2/4): positioning, supervision and the semantic intermediate (W2, Q3)
W2 and Q3. Positioning, supervision pipeline and the semantic intermediate.
The problem stated in the Introduction is to connect an editable composition with a complete audio realization. Relative to symbolic-only generation, the target includes the finished recording. Relative to generation in which composition remains implicit in audio tokens, the score exposes melody, harmony and form for inspection and intervention. Recent planning stages use audio–text embeddings (MusiCoT) or global attributes such as tempo, key and structure (ACE-Step 1.5) rather than a readable score. Controllable models (Music ControlNet, JASCO) render controls supplied by the user rather than composing them, and Seed-Music handles lead sheets through a separate symbolic pipeline. YuE2 composes a readable full-song melody-and-chord score and renders it within one model, which also makes a within-checkpoint test of planning possible. The unified architecture models the score, semantic sequence and acoustic latents through the factorization in Eq. (1) (Sec. 2.1, p. 3), allowing the acoustic stage to condition directly on text, score and semantic tokens.
The supervision pipeline is constructed offline from recordings (Sec. 2.2, p. 4; Appendix C.1, pp. 18–19). Two separately adapted branches share the same pretrained MERT2 foundation. The full-song, bidirectional branch provides features to SheetSage2-AR, which transcribes the symbolic score . The semantic-tokenizer branch uses causal self-attention and a 32,768-entry quantizer to produce 25-Hz tokens . A separately trained VAE encodes the same recording into continuous acoustic latents . These three targets supervise YuE2's score, semantic and acoustic streams. During text-to-song generation, YuE2 generates itself; MERT2 and SheetSage2 are not run.
The semantic intermediate is a practical design choice supported by prior work and our own experiments. Qwen-Music, HeartMuLa and LeVo provide successful precedents for autoregressive modeling of semantically rich discrete music representations followed by acoustic rendering. Direct audio-latent diffusion is also possible, as in ACE-Step 1.0 and DiffRhythm; as a reference for training scale, ACE-Step 1.0 reports 460,000 pretraining updates and 240,000 fine-tuning updates (Appendix A, Table 4). In an internal study, we trained two generation tasks. Both first generate a score from text. One then predicts semantic tokens before generating VAE latents, while the other generates the latents directly from text and score. We evaluated both tasks at the same 36K checkpoint on 30 vocal prompts with matched seeds and identical acoustic inference and decoding settings. Median ASR-based phoneme error rate was 3.12% with semantic tokens and 46.30% on the direct route. We therefore use semantic tokens to improve lyric fidelity within our training budget and retain the score as an editable compositional plan.
Response to Reviewer Nfiz (3/4): parameter allocation and the separate baseline (W3, Q2)
W3 and Q2. Parameter allocation, baseline design and optimization.
Capacity allocation and baseline design. YuE2 allocates 1.409B parameters to the autoregressive Transformer blocks that predict score and semantic tokens, and 1.409B to the non-autoregressive blocks that generate acoustic latents. The token embedding and untied output head add 756.55M, and the latent projections, timestep MLP and final normalization add 4.99M, for 3.580B in total (table below). The separate baseline follows the LM+DiT design of Qwen-Music. Its LM uses the same Qwen3-1.7B configuration as YuE2's autoregressive expert (28 layers, width 2048, FFN 6144, 16 query and 8 KV heads) and predicts the same score and semantic tokens. Sec. 3.2 names this LM by its Qwen3-1.7B backbone; with the same 184,704-token vocabulary and untied output head as YuE2, it has 2.166B parameters, exactly matching YuE2's discrete stage. Its 1.7B DiT follows the Qwen-Music renderer: the LM's semantic tokens enter frame-aligned with the noisy VAE latents, text and lyrics enter through cross-attention in every block, and the diffusion timestep enters through AdaLN. The baseline therefore totals about 3.87B parameters, versus 3.58B for YuE2. The DiT receives the score through the semantic tokens that the LM generates from it, together with separately encoded text and lyrics. In YuE2, the acoustic expert attends directly to text, score and semantic hidden states at every layer, and both experts are trained jointly.
| Component | Parameters |
|---|---|
| AR Transformer blocks, including their norms | 1,409,408,000 |
| NAR Transformer blocks, including their norms | 1,409,408,000 |
| Input embedding and untied output head | 756,547,584 |
| Latent projections, timestep MLP and final norm | 4,988,992 |
| Total | 3,580,352,576 |
The table reports trainable parameters in the YuE2 generator. Each Transformer-block count includes its normalization layers.
Baseline training and optimization. Both systems use the same melody-and-chord planning, semantic tokenizer and acoustic VAE, and we matched training-data volume across them. For discrete prediction, the baseline LM and YuE2's AR expert used the same token budget, batch size and number of updates. The separate acoustic DiT received 900,000 updates (Sec. 3.2 and Appendix D.1). For context, LeVo trained each of its token-conditioned acoustic DiTs for 300,000 steps (its Appendix B.2.1), and ACE-Step 1.0 used 700,000 steps in total, as noted above. The baseline was competitive on text alignment in the blind listening test: of 206 judgments, 69 favored the baseline, 70 favored YuE2 and 67 were ties.
Results. Each system generated two candidates for each of the same 192 prompts, and the one with the lower phoneme error rate (PER) was evaluated. In blinded listening tests, 53.4% of overall-quality judgments (111 of 208) favored YuE2 and 35.6% favored the separate baseline (p = 0.0084). For audio quality, the shares were 48.5% and 29.6% (p = 0.000306). The remaining judgments were ties, and both tests accounted for clustering by evaluator and song. Because selection used only PER, we could assess the selected outputs separately with SongEval: across 192 paired prompts, mean SongEval was 4.270 for YuE2 and 4.109 for the baseline, a paired difference of +0.162 (95% confidence interval clustered by prompt [0.116, 0.208], Holm-adjusted p = 4.17 × 10⁻¹⁰ across nine metrics). Expert preferences and an automatic metric excluded from selection therefore agree, against a baseline with more parameters in total than YuE2.
Response to Reviewer Nfiz (1/4): clarity and hybrid attention (W1, W2, Q1)
W1. Clarity and breadth.
Thank you for the careful review and for recognizing the value of explicit score planning and of the within-checkpoint comparison. Below we answer each weakness and question, including the explicit definitions you requested.
The central question of this work is whether explicit symbolic planning can improve the quality of full-song generation while providing an editable interface for composition. YuE2 addresses this by jointly modeling the score, semantic tokens, and acoustic latents, as formalized in Eq. (1) (Sec. 2.1, p. 3). The score connects these two goals: it serves as a generation plan and as the interface through which users can inspect and modify melody and harmony.
The components and evaluations have distinct roles in this framework. MERT2 and SheetSage2 construct offline training targets from recordings without aligned scores. Creation, editing and covering then use the same learned score-to-audio mapping, as stated in Sec. 2.3 (p. 4): the score is generated, revised or transcribed, respectively. The generation experiments measure song quality; editing tests whether changes to the score produce the intended musical changes; covering tests the mapping on unseen compositions. Agentic editing demonstrates how an external agent can operate this same score interface.
The within-checkpoint comparison directly tests the role of symbolic planning. As reported in Sec. 3.3 (p. 6), it holds model weights, prompts, lyrics, the two-candidate budget, selection rule and decoder fixed. Experts prefer planning for overall quality (49.3% versus 34.6%, p=0.0070) and musicality (45.0% versus 29.4%, p=0.0141). These results provide empirical support for explicit score planning as a modeling choice for full-song generation.
W2 and Q1. Hybrid attention.
By “hybrid self-attention,” we mean attention over discrete and acoustic positions with type-specific projections and different visibility patterns. As described in Sec. 2.2 (pp. 3–4) and Fig. 2 (p. 3), discrete positions use the AR expert and acoustic positions use the NAR expert. Each expert has its own normalization, Q/K/V/output projections and feed-forward network. Their projected queries, keys and values participate in one attention computation: discrete queries follow a causal mask, while acoustic queries can attend to the completed discrete prefix and all acoustic positions of the same song.
Let D denote valid discrete positions (text/lyrics, score and semantic tokens), A valid acoustic positions, and bi the song ID of position i in a packed sequence. Expert routing is fixed by position type: e(i)=AR for i∈D, and e(i)=NAR for i∈A. The additive attention mask is
0,&b_i=b_j,\quad i,j\in D,\quad j\le i,\\\\ 0,&b_i=b_j,\quad i\in A,\quad j\in D\cup A,\\\\ -\infty,&\text{otherwise}. \end{cases}$$ For a layer input $h_i$, the computation is as follows. For readability, layer and head indices, grouped-query head mapping, RoPE and per-head Q/K normalization are suppressed; the attention output $a_i$ includes head concatenation before the output projection. All normalization, projection and feed-forward parameters carry the indicated expert index. $$\begin{aligned} \tilde h_i &= \mathcal N_{\mathrm{attn}}^{e(i)}(h_i),\\\\ q_i &= W_Q^{e(i)}\tilde h_i,\qquad k_i = W_K^{e(i)}\tilde h_i,\qquad v_i = W_V^{e(i)}\tilde h_i,\\\\ \alpha_{ij} &= \operatorname{softmax}_{j}\\!\left(\frac{q_i^\top k_j}{\sqrt{d_h}}+M_{ij}\right), \qquad a_i = \operatorname{Concat}_{\mathrm{heads}}\\!\left(\sum_j\alpha_{ij}v_j\right),\\\\ h'_i &= h_i + W_O^{e(i)}a_i,\\\\ h_i^{\mathrm{next}} &= h'_i + \operatorname{FFN}^{e(i)}\\!\left(\mathcal N_{\mathrm{ffn}}^{e(i)}(h'_i)\right). \end{aligned}$$ Here $\mathcal N$ denotes expert-specific RMSNorm and $d_h$ the head dimension. An acoustic query can therefore read discrete keys/values produced by the AR expert and acoustic keys/values produced by the NAR expert. The output projection and feed-forward network use the query position's expert. Discrete queries cannot read acoustic targets, and different packed songs cannot attend to one another. The implementation has 28 layers, 16 query heads, eight KV heads and head dimension 128; the discrete head predicts tokens, while the acoustic head predicts flow velocity for all frames in parallel.