- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a joint discrete and continuous framework for song generation from lyrics and a style description. An autoregressive language model predicts semantic tokens using VQ tokenizer. A semantic flow model then generates continuous features. It uses two conditions: the full token sequence and MLP readouts of the language model’s hidden states. A separately trained SSL-based variational encoder defines the target space for these features. A frozen acoustic diffusion model then renders the features into audio. The language model and the flow model train jointly with token cross-entropy and flow matching. The authors compare one discrete baseline and five joint configurations on 192 prompts. They also report representation probes and reconstruction tests. Relative to the discrete baseline, different proposed groups can outperform the discrete baseline.
Critical strengths
- The overall design of the architecture is clear and easy to follow.
- The experiment covers multiple different evaluation metrics that covers aesthetics, production quality, style alignment, lyric intelligibility as well as subjective reviews.
- The code and demo is provided to review on the generated pieces.
Critical weaknesses
- The paper contains both discrete and continuous for song and music generation, however, the motivation of the paper on joint training is not very convincing and there has no ablation study nor deeper analysis on which part contributes most to the performance. Besides, the paper mentioned two limitations (1) distinctions lost at code book assignment; (2) separate optimized stages prevent the objective to update language models. More analysis and experiments to demonstrate some of these would help strengthen the paper.
- The objective function of SSL-VAE and joint training lack more rigorous justifications. For SSL-VAE loss, there are multiple terms in the objective function and each with different weights, it is not clear how each loss is balanced and contributed for training. For joint training objectives, there is no further sensitivity study to demonstrate the model performance under different loss combinations. More experiments on loss-weight sensitivity, training stability, and error propagation would help validate the proposed design.
- It seems different dimensions of embeddings have different performance as shown in both table 1 and table 2. Increasing the dimension does not necessarily improve the performance. The paper provides limited analysis to explain this difference but rather listing the facts. More in-depth explanation on the phenomenon and analysis, as well as controlled experiments would help explain these results and justify the representation design.
- No baselines are included in any of the experiments and thus making it hard to judge the performance of the model compared to SOTA, except a LeVo2 in Figure 3.
Questions for authors
- What is the expert agreement for the listening test?
- What are the specific set ups to process and generate the datasets used for training? What are the dataset statistics?
6 comments
Response to UjFn (2/3): Loss design and representation dimensionality
UjFn-2 | Roles and normalization of the losses (W2)
We first distinguish the two training stages: the SSL-VAE losses learn the continuous target space; the SSL-VAE is then frozen while the LM, readout, and Semantic DiT are jointly trained. The representation-learning losses and the CE/FM generation objectives therefore optimize different components.
The SSL-VAE objective is:
=2\mathcal L_{\mathrm{CTC}} +\mathcal L_{\mathrm{mel}} +\mathcal L_{\mathrm{chr}} +10^{-5}\mathcal L_{\mathrm{KL}} +\mathcal L_{\mathrm{align}}.$$ CTC constrains lyric content; Mel/Chroma reconstruction constrains spectral and harmonic information; alignment encourages agreement with a frozen SSL teacher; and KL regularizes the Gaussian posterior. The coefficients are **CTC: 2; Mel: 1; Chroma: 1; KL: 10⁻⁵; alignment: 1**. Each Mel/Chroma loss combines spectral convergence with an element-wise mean magnitude L1 error. These objectives differ in units and normalization, so equal coefficients do not imply equal numerical scales or gradient contributions. Joint training uses **L_CE + L_FM, with coefficient 1 for each term**. CE supervises audio tokens and BOS/EOS boundaries, excluding the text prompt. FM is normalized over valid frames and feature dimensions and updates the LM through the readout. The acoustic renderer remains frozen during this stage. At the same time, we very much appreciate the insightful question you raise. SSL-VAE representation learning is, in fact, a problem that we are strongly motivated to continue studying and resolving. The difficulty is that semantic and reconstruction supervision can conflict, while we want a representation that both captures semantics well and supports high-quality reconstruction. Achieving both is challenging. We are currently conducting extensive related experiments to find a better balance in this trade-off. We also frankly acknowledge that SSL-VAE is not yet a particularly elegant solution, and we hope that further research will lead to a more elegant representation design. **Existing manuscript evidence:** The SSL-VAE losses and normalization are defined in Appendix A (p. 14, line 718); the joint objective and valid-frame normalization are in Appendix B (p. 15, line 782). These definitions were already in the original manuscript. **New evidence** is provided by the controlled experiments in Sec. 4.3 (p. 8, line 378). **UjFn-3 | Representation dimensionality and generation performance (W3)** **Information recoverable from reference features and the quality of generated songs are different evaluation targets.** The standard Joint-STD configurations can be compared directly: | Dimension | micro-PER (%) ↓ | SongEval ↑ | Genre probe accuracy ↑ | MOS quality ↑ | |---|---:|---:|---:|---:| | 48D | **14.47** | **4.269** | 0.472 | **4.14** | | 96D | 17.25 | 4.222 | 0.510 | 3.38 | | 128D | 18.86 | 4.145 | **0.541** | 3.76 | Probes use features extracted from reference audio and measure the information available to downstream tasks. Free generation additionally requires the LM to produce a token plan from lyrics and style, the semantic flow to predict continuous features, and the renderer to synthesize audio. Richer reference features must therefore also be predicted and rendered accurately; better probes do not automatically imply better generation. Reference-conditioned reconstruction provides another view. Under the ground-truth acoustic-latent reconstruction protocol in Table 5, 48D has CTC/CER of **1.331/25.93%**, versus **1.465/28.62%** for 128D. The higher-dimensional representation's genre-probe advantage therefore also does not translate into an advantage on these reconstruction metrics. This connects three levels of evaluation: information in reference features, reconstruction under reference conditions, and free generation. A plausible explanation is that higher-dimensional targets preserve more information while also making prediction and rendering more difficult. As discussed above, SSL-VAE training remains difficult to characterize systematically. Our current experiments do not yet establish a systematic relationship in this respect, but we are sincerely interested in pursuing a deeper investigation. At this stage, we hope to report these results and trade-offs faithfully and provide useful reference points for the community. **Existing manuscript evidence:** Generation results are in Table 1 (p. 7, line 336); probes and reconstruction are in Tables 4 and 5 (p. 9, starting at lines 433 and 458). The "Semantic capacity and generation behavior" paragraph in Sec. 5 (p. 10, line 494) discusses these dimensionality results.Response to UjFn (3/3): External baselines, expert agreement, and data
UjFn-4 | Original external baselines (W4)
The original Table 1 includes the Discrete internal baseline, while Joint-LeVo and Joint-SB are internal configurations. To address the limited external comparison, we added a common-protocol evaluation of the original LeVo 2 and SongBloom checkpoints.
All four systems use the same 192 prompts and the manuscript's duration/retry retention rules and scoring procedure. Songs are retained by duration and retry rules. We run Qwen3-ASR four times on each retained song, choose the lowest-error transcript, and compute micro-PER from pooled phoneme edit and reference-phoneme counts.
| System | micro-PER (%) ↓ | SongEval ↑ | SongBench ↑ | AudioBox PQ ↑ | MuLan ↑ | MOS quality ↑ | MOS musicality ↑ |
|---|---|---|---|---|---|---|---|
| Full-48D | 14.47 | 4.269 | 6.282 | 8.300 | 0.514 | 4.14 | 3.26 |
| Discrete | 20.26 | 4.152 | 6.217 | 8.282 | 0.487 | 3.14 | 2.76 |
| LeVo 2 | 28.73 | 4.132 | 6.277 | 8.361 | 0.394 | 3.28 | 2.64 |
| SongBloom | 21.29 | 3.580 | 4.257 | 8.154 | 0.214 | 2.72 | 2.38 |
Full-48D has lower micro-PER and higher MuLan, SongEval, and both MOS scores. SongBench is close to LeVo 2, while LeVo 2 has higher AudioBox PQ. The external models differ in training corpora, scale, and pretraining methods, so these results provide a system-level quality comparison. Attribution of the method's contributions is supported by the internal controls above.
Revisions: Sec. 4.4 (p. 8, line 407; Table 3 at line 414) adds the external evaluation. Figure 3 (p. 7, caption at line 366) uses the new measurements. Sec. 4.1 (p. 6, line 311) clarifies song retention and ASR transcript selection.
UjFn-5 | Expert agreement (Q1)
We supplement the listening study of five music experts, ten matched song groups, and 1–5 ratings with Krippendorff's alpha: 0.77 for sound quality, with 95% CI [0.62, 0.83], and 0.63 for musicality, with 95% CI [0.42, 0.73]. Quality/musicality MOS for Full, Stop-grad, and w/o readout is 4.14/3.26, 3.78/3.18, and 3.66/3.12, respectively. We report agreement alongside MOS.
Revisions: Sec. 4.1 (p. 7, line 324) and Appendix E (p. 16, line 863) add agreement statistics. Ablation listening results are in Table 2 (p. 8, line 384).
UjFn-6 | Dataset scale, coverage, and processing (Q2)
The tokenizer, LM, Semantic DiT, and Acoustic DiT training stages in this work use the same deduplicated dataset. It contains 19,521.8 million semantic tokens at 25 Hz, corresponding to approximately 216,909 hours of audio. Chinese, English, Spanish, Japanese, and Korean account for 47.44%, 41.29%, 7.91%, 2.11%, and 1.25%, respectively. The 24 genre categories are:
| Genre label | Share | Genre label | Share |
|---|---|---|---|
| Dance | 7.68% | Country | 3.73% |
| Rap | 7.23% | Jazz | 3.73% |
| Classical | 6.72% | British | 3.51% |
| World music | 6.67% | Post-rock | 3.36% |
| Gufeng | 6.30% | Latin | 3.30% |
| Metal | 5.66% | Electronic | 2.72% |
| Blues | 5.60% | Easy listening | 2.48% |
| Punk | 5.44% | Reggae | 2.34% |
| New Age | 4.80% | Rock | 2.02% |
| Pop | 4.52% | Bossa Nova | 1.88% |
| Other | 3.99% | Folk | 1.69% |
| Ethnic music | 3.76% | R&B | 0.85% |
Language and genre distributions are normalized separately by their respective record/label frequencies. Genre percentages sum to 99.98% after rounding to two decimal places. The VQ and SSL-VAE representations use 24-kHz audio and produce temporally aligned 25-Hz sequences. Reference-audio encoders are used only during feature extraction.
Revisions: The Data paragraph in Sec. 4.1 (p. 6, line 307) describes the shared corpus and its scale. Appendix D (p. 16, line 834; Table D.1 at line 844) reports the language and full genre distributions. Existing manuscript evidence: The 24-kHz/25-Hz representation settings are in Appendix A (p. 14, line 704).
Response to UjFn (1/3): Motivation and controlled ablations
Thank you for your careful and thorough review and constructive comments, and for recognizing the architectural clarity, broad evaluation, and code and demos. To address the motivation for joint training and attribution of its benefits, we added interface controls with fixed representations, renderer, and training budget, as well as planner replacement under the original Discrete renderer. We first explain how this evidence supports the method, then address losses, representation dimensionality, external models, expert agreement, and data.
All page and line references below refer to the revised manuscript. Revisions are marked in blue for ease of reference. Previously reported results and definitions are identified below as "Existing manuscript evidence."
UjFn-1 | Motivation and attribution (W1)
Discrete token planning and continuous semantic generation can be jointly optimized through a differentiable interface. The two design choices address the two limitations you identified: an independently learned continuous target space allows generation beyond finite codebook lookups, while a differentiable readout establishes a semantic FM-to-LM gradient path so that the generation objective can update the planner without differentiating through discrete token sampling.
The continuous target s comes from a separately trained SSL-VAE, while readout c comes from the LM's predictive states; they serve different roles. In the existing MARBLE results, continuous 48D reference features outperform VQ lookups on five of six probes, providing complementary evidence about information preservation. The contribution of joint training is examined directly by the following controls.
The three 48D variants use the same continuous representation, initialization, training data, training budget, and acoustic renderer on the same 192 prompts. Stop-grad retains the readout but blocks FM-to-LM gradients; the readout and Semantic DiT continue to learn, while cross-entropy updates the LM. The w/o readout variant uses only token conditioning to generate the same continuous targets.
Full reduces micro-PER by 2.07 percentage points relative to Stop-grad, raises SongEval by 0.040, and improves quality/musicality MOS from 3.78/3.18 to 4.14/3.26. Stop-grad reduces micro-PER by 0.92 percentage points relative to w/o readout, with similar SongEval. These comparisons test FM updates to the LM and readout conditioning, respectively. Stop-grad has higher SongBench and AudioBox PQ, so we emphasize the improvements in lyric fidelity, SongEval, and listening scores.
We also fix the original Discrete renderer and codebook interface, supply no readout, and replace only the token planner. Full-joint LM achieves 18.74% micro-PER, versus 20.26% for the original LM and 20.72% for CE-only LM; their respective SongEval scores are 4.184, 4.152, and 4.164. Thus, the benefits also appear in discrete outputs under the original renderer, and the CE-only control further separates joint optimization from additional cross-entropy training.
Revisions: The introduction (p. 1, line 43) connects the two motivations to the design; Sec. 3.2 (p. 5, line 216) clarifies the interface and gradient path. Sec. 4.3 (p. 8, line 378; Table 2 at line 384) adds the interface ablations, with planner replacement results in the paragraph starting at line 398. Existing manuscript evidence: MARBLE probes are in Table 4 (p. 9, line 433).