acceptodds
Under review as a conference paper at ICLR 2027

High-Fidelity Song Generation with Joint Discrete and Continuous Modeling

Abstract

Generating a full song from lyrics and a style description requires long-range musical planning without sacrificing vocal and instrumental detail. Low-rate discrete semantic tokens make this tractable, but quantization constrains the information available to acoustic rendering, while separately optimized stages prevent downstream generation objectives from shaping the planner. We propose joint discrete and continuous modeling for high-fidelity song generation. A VQ tokenizer supplies tokens for autoregressive planning, and a separately trained SSL-derived variational encoder supplies continuous semantic targets. Conditioned on the completed token plan and differentiable readouts of the planner’s predictive states, a full-sequence flow model generates semantic features for acoustic rendering. Its flow-matching gradients reach the language model through the readouts, without differentiating through token sampling; the standard pathway keeps continuous features out of autoregressive feedback. On 192 shared prompts, its 48D configuration improves SongEval from 4.152 to 4.269 and reduces micro-PER from 20.26% to 14.47% relative to a discrete baseline, while receiving substantially higher expert listening scores. We further analyze the system through downstream semantic probes and conditioned reconstruction.

Primary area:
generative models
Keywords:
song generation, music generation, joint discrete and continuous modeling, semantic representation learning, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UjFnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a joint discrete and continuous framework for song generation from lyrics and a style description. An autoregressive language model predicts semantic tokens using VQ tokenizer. A semantic flow model then generates continuous features. It uses two conditions: the full token sequence and MLP readouts of the language model’s hidden states. A separately trained SSL-based variational encoder defines the target space for these features. A frozen acoustic diffusion model then renders the features into audio. The language model and the flow model train jointly with token cross-entropy and flow matching. The authors compare one discrete baseline and five joint configurations on 192 prompts. They also report representation probes and reconstruction tests. Relative to the discrete baseline, different proposed groups can outperform the discrete baseline.

Critical strengths
  1. The overall design of the architecture is clear and easy to follow.
  2. The experiment covers multiple different evaluation metrics that covers aesthetics, production quality, style alignment, lyric intelligibility as well as subjective reviews.
  3. The code and demo is provided to review on the generated pieces.
Critical weaknesses
  1. The paper contains both discrete and continuous for song and music generation, however, the motivation of the paper on joint training is not very convincing and there has no ablation study nor deeper analysis on which part contributes most to the performance. Besides, the paper mentioned two limitations (1) distinctions lost at code book assignment; (2) separate optimized stages prevent the objective to update language models. More analysis and experiments to demonstrate some of these would help strengthen the paper.
  2. The objective function of SSL-VAE and joint training lack more rigorous justifications. For SSL-VAE loss, there are multiple terms in the objective function and each with different weights, it is not clear how each loss is balanced and contributed for training. For joint training objectives, there is no further sensitivity study to demonstrate the model performance under different loss combinations. More experiments on loss-weight sensitivity, training stability, and error propagation would help validate the proposed design.
  3. It seems different dimensions of embeddings have different performance as shown in both table 1 and table 2. Increasing the dimension does not necessarily improve the performance. The paper provides limited analysis to explain this difference but rather listing the facts. More in-depth explanation on the phenomenon and analysis, as well as controlled experiments would help explain these results and justify the representation design.
  4. No baselines are included in any of the experiments and thus making it hard to judge the performance of the model compared to SOTA, except a LeVo2 in Figure 3.
Questions for authors
  1. What is the expert agreement for the listening test?
  2. What are the specific set ups to process and generate the datasets used for training? What are the dataset statistics?
6 comments
AuthorsOpenReview ↗

Response to UjFn (1/3): Motivation and controlled ablations

Thank you for your careful and thorough review and constructive comments, and for recognizing the architectural clarity, broad evaluation, and code and demos. To address the motivation for joint training and attribution of its benefits, we added interface controls with fixed representations, renderer, and training budget, as well as planner replacement under the original Discrete renderer. We first explain how this evidence supports the method, then address losses, representation dimensionality, external models, expert agreement, and data.

All page and line references below refer to the revised manuscript. Revisions are marked in blue for ease of reference. Previously reported results and definitions are identified below as "Existing manuscript evidence."

UjFn-1 | Motivation and attribution (W1)

Discrete token planning and continuous semantic generation can be jointly optimized through a differentiable interface. The two design choices address the two limitations you identified: an independently learned continuous target space allows generation beyond finite codebook lookups, while a differentiable readout establishes a semantic FM-to-LM gradient path so that the generation objective can update the planner without differentiating through discrete token sampling.

The continuous target s comes from a separately trained SSL-VAE, while readout c comes from the LM's predictive states; they serve different roles. In the existing MARBLE results, continuous 48D reference features outperform VQ lookups on five of six probes, providing complementary evidence about information preservation. The contribution of joint training is examined directly by the following controls.

The three 48D variants use the same continuous representation, initialization, training data, training budget, and acoustic renderer on the same 192 prompts. Stop-grad retains the readout but blocks FM-to-LM gradients; the readout and Semantic DiT continue to learn, while cross-entropy updates the LM. The w/o readout variant uses only token conditioning to generate the same continuous targets.

Configurationmicro-PER (%) ↓SongEval ↑SongBench ↑AudioBox PQ ↑MuLan ↑MOS quality ↑MOS musicality ↑
Full14.474.2696.2828.3000.5144.143.26
Stop-grad16.544.2296.2948.3500.5093.783.18
w/o readout17.464.2306.2708.3240.5133.663.12

Full reduces micro-PER by 2.07 percentage points relative to Stop-grad, raises SongEval by 0.040, and improves quality/musicality MOS from 3.78/3.18 to 4.14/3.26. Stop-grad reduces micro-PER by 0.92 percentage points relative to w/o readout, with similar SongEval. These comparisons test FM updates to the LM and readout conditioning, respectively. Stop-grad has higher SongBench and AudioBox PQ, so we emphasize the improvements in lyric fidelity, SongEval, and listening scores.

We also fix the original Discrete renderer and codebook interface, supply no readout, and replace only the token planner. Full-joint LM achieves 18.74% micro-PER, versus 20.26% for the original LM and 20.72% for CE-only LM; their respective SongEval scores are 4.184, 4.152, and 4.164. Thus, the benefits also appear in discrete outputs under the original renderer, and the CE-only control further separates joint optimization from additional cross-entropy training.

Revisions: The introduction (p. 1, line 43) connects the two motivations to the design; Sec. 3.2 (p. 5, line 216) clarifies the interface and gradient path. Sec. 4.3 (p. 8, line 378; Table 2 at line 384) adds the interface ablations, with planner replacement results in the paragraph starting at line 398. Existing manuscript evidence: MARBLE probes are in Table 4 (p. 9, line 433).

AuthorsOpenReview ↗

Response to UjFn (2/3): Loss design and representation dimensionality

UjFn-2 | Roles and normalization of the losses (W2)

We first distinguish the two training stages: the SSL-VAE losses learn the continuous target space; the SSL-VAE is then frozen while the LM, readout, and Semantic DiT are jointly trained. The representation-learning losses and the CE/FM generation objectives therefore optimize different components.

The SSL-VAE objective is:

=2\mathcal L_{\mathrm{CTC}} +\mathcal L_{\mathrm{mel}} +\mathcal L_{\mathrm{chr}} +10^{-5}\mathcal L_{\mathrm{KL}} +\mathcal L_{\mathrm{align}}.$$ CTC constrains lyric content; Mel/Chroma reconstruction constrains spectral and harmonic information; alignment encourages agreement with a frozen SSL teacher; and KL regularizes the Gaussian posterior. The coefficients are **CTC: 2; Mel: 1; Chroma: 1; KL: 10⁻⁵; alignment: 1**. Each Mel/Chroma loss combines spectral convergence with an element-wise mean magnitude L1 error. These objectives differ in units and normalization, so equal coefficients do not imply equal numerical scales or gradient contributions. Joint training uses **L_CE + L_FM, with coefficient 1 for each term**. CE supervises audio tokens and BOS/EOS boundaries, excluding the text prompt. FM is normalized over valid frames and feature dimensions and updates the LM through the readout. The acoustic renderer remains frozen during this stage. At the same time, we very much appreciate the insightful question you raise. SSL-VAE representation learning is, in fact, a problem that we are strongly motivated to continue studying and resolving. The difficulty is that semantic and reconstruction supervision can conflict, while we want a representation that both captures semantics well and supports high-quality reconstruction. Achieving both is challenging. We are currently conducting extensive related experiments to find a better balance in this trade-off. We also frankly acknowledge that SSL-VAE is not yet a particularly elegant solution, and we hope that further research will lead to a more elegant representation design. **Existing manuscript evidence:** The SSL-VAE losses and normalization are defined in Appendix A (p. 14, line 718); the joint objective and valid-frame normalization are in Appendix B (p. 15, line 782). These definitions were already in the original manuscript. **New evidence** is provided by the controlled experiments in Sec. 4.3 (p. 8, line 378). **UjFn-3 | Representation dimensionality and generation performance (W3)** **Information recoverable from reference features and the quality of generated songs are different evaluation targets.** The standard Joint-STD configurations can be compared directly: | Dimension | micro-PER (%) ↓ | SongEval ↑ | Genre probe accuracy ↑ | MOS quality ↑ | |---|---:|---:|---:|---:| | 48D | **14.47** | **4.269** | 0.472 | **4.14** | | 96D | 17.25 | 4.222 | 0.510 | 3.38 | | 128D | 18.86 | 4.145 | **0.541** | 3.76 | Probes use features extracted from reference audio and measure the information available to downstream tasks. Free generation additionally requires the LM to produce a token plan from lyrics and style, the semantic flow to predict continuous features, and the renderer to synthesize audio. Richer reference features must therefore also be predicted and rendered accurately; better probes do not automatically imply better generation. Reference-conditioned reconstruction provides another view. Under the ground-truth acoustic-latent reconstruction protocol in Table 5, 48D has CTC/CER of **1.331/25.93%**, versus **1.465/28.62%** for 128D. The higher-dimensional representation's genre-probe advantage therefore also does not translate into an advantage on these reconstruction metrics. This connects three levels of evaluation: information in reference features, reconstruction under reference conditions, and free generation. A plausible explanation is that higher-dimensional targets preserve more information while also making prediction and rendering more difficult. As discussed above, SSL-VAE training remains difficult to characterize systematically. Our current experiments do not yet establish a systematic relationship in this respect, but we are sincerely interested in pursuing a deeper investigation. At this stage, we hope to report these results and trade-offs faithfully and provide useful reference points for the community. **Existing manuscript evidence:** Generation results are in Table 1 (p. 7, line 336); probes and reconstruction are in Tables 4 and 5 (p. 9, starting at lines 433 and 458). The "Semantic capacity and generation behavior" paragraph in Sec. 5 (p. 10, line 494) discusses these dimensionality results.
AuthorsOpenReview ↗

Response to UjFn (3/3): External baselines, expert agreement, and data

UjFn-4 | Original external baselines (W4)

The original Table 1 includes the Discrete internal baseline, while Joint-LeVo and Joint-SB are internal configurations. To address the limited external comparison, we added a common-protocol evaluation of the original LeVo 2 and SongBloom checkpoints.

All four systems use the same 192 prompts and the manuscript's duration/retry retention rules and scoring procedure. Songs are retained by duration and retry rules. We run Qwen3-ASR four times on each retained song, choose the lowest-error transcript, and compute micro-PER from pooled phoneme edit and reference-phoneme counts.

Systemmicro-PER (%) ↓SongEval ↑SongBench ↑AudioBox PQ ↑MuLan ↑MOS quality ↑MOS musicality ↑
Full-48D14.474.2696.2828.3000.5144.143.26
Discrete20.264.1526.2178.2820.4873.142.76
LeVo 228.734.1326.2778.3610.3943.282.64
SongBloom21.293.5804.2578.1540.2142.722.38

Full-48D has lower micro-PER and higher MuLan, SongEval, and both MOS scores. SongBench is close to LeVo 2, while LeVo 2 has higher AudioBox PQ. The external models differ in training corpora, scale, and pretraining methods, so these results provide a system-level quality comparison. Attribution of the method's contributions is supported by the internal controls above.

Revisions: Sec. 4.4 (p. 8, line 407; Table 3 at line 414) adds the external evaluation. Figure 3 (p. 7, caption at line 366) uses the new measurements. Sec. 4.1 (p. 6, line 311) clarifies song retention and ASR transcript selection.

UjFn-5 | Expert agreement (Q1)

We supplement the listening study of five music experts, ten matched song groups, and 1–5 ratings with Krippendorff's alpha: 0.77 for sound quality, with 95% CI [0.62, 0.83], and 0.63 for musicality, with 95% CI [0.42, 0.73]. Quality/musicality MOS for Full, Stop-grad, and w/o readout is 4.14/3.26, 3.78/3.18, and 3.66/3.12, respectively. We report agreement alongside MOS.

Revisions: Sec. 4.1 (p. 7, line 324) and Appendix E (p. 16, line 863) add agreement statistics. Ablation listening results are in Table 2 (p. 8, line 384).

UjFn-6 | Dataset scale, coverage, and processing (Q2)

The tokenizer, LM, Semantic DiT, and Acoustic DiT training stages in this work use the same deduplicated dataset. It contains 19,521.8 million semantic tokens at 25 Hz, corresponding to approximately 216,909 hours of audio. Chinese, English, Spanish, Japanese, and Korean account for 47.44%, 41.29%, 7.91%, 2.11%, and 1.25%, respectively. The 24 genre categories are:

Genre labelShareGenre labelShare
Dance7.68%Country3.73%
Rap7.23%Jazz3.73%
Classical6.72%British3.51%
World music6.67%Post-rock3.36%
Gufeng6.30%Latin3.30%
Metal5.66%Electronic2.72%
Blues5.60%Easy listening2.48%
Punk5.44%Reggae2.34%
New Age4.80%Rock2.02%
Pop4.52%Bossa Nova1.88%
Other3.99%Folk1.69%
Ethnic music3.76%R&B0.85%

Language and genre distributions are normalized separately by their respective record/label frequencies. Genre percentages sum to 99.98% after rounding to two decimal places. The VQ and SSL-VAE representations use 24-kHz audio and produce temporally aligned 25-Hz sequences. Reference-audio encoders are used only during feature extraction.

Revisions: The Data paragraph in Sec. 4.1 (p. 6, line 307) describes the shared corpus and its scale. Appendix D (p. 16, line 834; Table D.1 at line 844) reports the language and full genre distributions. Existing manuscript evidence: The 24-kHz/25-Hz representation settings are in Appendix A (p. 14, line 704).

Reviewer_FifcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses two limitations in two-stage full-song generation: quantization loss in 25-Hz single-codebook VQ tokens and the lack of end-to-end gradient feedback from the acoustic renderer to the autoregressive (AR) planner. The authors keep discrete VQ tokens as the sole AR feedback in a Qwen2.5-1.5B LM, while routing a 2-layer MLP readout ct=g(ht)c_t = g(h_t) of the LM's causal states (alongside the discrete token sequence) to a full-sequence 1D Semantic DiT that predicts continuous SSL-VAE features. During joint training (LCE+LFML_{\text{CE}} + L_{\text{FM}}), flow-matching gradients update the LM through ctc_t without differentiating through hard token sampling. Evaluated on 192 WildSongBench prompts, the 48D system reduces micro-PER from 20.26% to 14.47% and improves SongEval (4.152 to 4.269) and expert MOS over the discrete baseline.

Critical strengths
  • Pragmatic architecture for minute-scale audio: Keeping the AR history strictly discrete while passing continuous states outward to a bidirectional semantic flow avoids the long-horizon rollout drift typically seen in continuous or interleaved AR-diffusion models.
  • Transparent reporting and detailed system-level ablations: The authors provide thorough ablations across semantic bottleneck widths (48D/96D/128D) and generation paradigms (Joint-STD, Joint-LeVo, Joint-SB), backed by MARBLE linear probes and reference-conditioned reconstruction diagnostics, and report the trade-offs across all configurations transparently.
Critical weaknesses
  • Overly complex pipeline with too many extractors and cascaded stages: Training the standard system requires four separate feature extractors/teachers (1024D SSL encoder, VQ tokenizer, SSL-VAE, and LeVo-VAE) and three multi-stage generative models (LM, a 28-block Semantic DiT, and a 24-block Acoustic DiT). At inference, cascading two full DiTs (64 flow/diffusion steps total) is heavy, yet the simplified single-flow variant (Joint-LeVo-64D) drops to 3.04 in sound-quality MOS (below Discrete's 3.14).
  • Unclear effect of LFML_{\text{FM}} on the LM planner: Joint training fine-tunes a converged LM with a small learning rate (10−510^{-5}), making it unclear whether LFML_{\text{FM}} improves the discrete token plan dd itself or merely teaches the LM to pass auxiliary features into ctc_t. And what happens under a stop_gradient ablation on ctc_t?
Questions for authors
  1. If you feed the discrete tokens from the jointly trained Joint-STD-48D LM (without ctc_t) into the baseline Discrete renderer, does it outperform the original pretrained LM?
  2. What is the exact scale of the "in-house synthetic music data" (total hours, number of songs, and language breakdown) used to train the tokenizers, LM, and DiT stages?
  3. Was Figure 2 generated by an AI image model? The colors and line weights look noticeably uneven—please consider redrawing it as a clean vector diagram.
4 comments
AuthorsOpenReview ↗

Response to Fifc (1/3): Discrete planning and stop-gradient controls

Thank you for your careful and thorough review and constructive comments, and for recognizing the architecture and evaluation at multiple levels. Your question about whether the benefits exist only in the continuous readout helped us identify what must be tested directly: the discrete token plan produced by the jointly trained LM. We added planner replacement under the original Discrete renderer and a Stop-grad control, and clarified the roles of feature extraction and generation, the training data, and inference cost.

All page and line references below refer to the revised manuscript. Revisions are marked in blue for ease of reference.

Fifc-1 | Does the discrete token plan improve? (W2 / Q1)

Yes. With the original Discrete renderer and codebook interface fixed, and without supplying readout condition c to the renderer, discrete tokens from the jointly trained Joint-STD-48D LM still improve the results. Following your suggestion, we replace only the token planner and include a CE-only LM continued with cross-entropy as a control.

Token planner onlymicro-PER (%) ↓SongEval ↑MOS quality ↑MOS musicality ↑
Original LM20.264.1523.142.76
CE-only LM20.724.1643.082.82
Full-joint LM18.744.1843.222.88

Full-joint LM reduces micro-PER by 1.52 percentage points relative to the original LM and 1.98 percentage points relative to CE-only LM, while also achieving higher SongEval and both MOS scores. Because the renderer receives only codebook lookups of discrete tokens in this experiment, the results directly show benefits through a discrete rendering interface that does not use c. The CE-only control further separates joint optimization from additional cross-entropy training.

Revision: The planner replacement paragraph in Sec. 4.3 (p. 8, starting at line 398) adds this comparison under the fixed renderer and its full results.

Fifc-2 | Stop-gradient control and joint gradients (W2)

Full, Stop-grad, and w/o readout use the same 48D continuous representation, initialization, training data, training budget, and acoustic renderer, and are evaluated on the same 192 prompts. Stop-grad retains the readout condition but blocks FM-to-LM gradients; the readout and Semantic DiT continue to learn, while cross-entropy still updates the LM. The w/o readout variant uses only token conditioning while generating features in the same continuous target space.

Configurationmicro-PER (%) ↓SongEval ↑SongBench ↑AudioBox PQ ↑MuLan ↑MOS quality ↑MOS musicality ↑
Full14.474.2696.2828.3000.5144.143.26
Stop-grad16.544.2296.2948.3500.5093.783.18
w/o readout17.464.2306.2708.3240.5133.663.12

Relative to Stop-grad, Full reduces micro-PER by 2.07 percentage points, increases SongEval by 0.040, and raises quality/musicality MOS from 3.78/3.18 to 4.14/3.26. This tests the effect of FM updates to the LM. Stop-grad reduces micro-PER by 0.92 percentage points relative to w/o readout, with similar SongEval, testing the readout condition when FM-to-LM gradients are blocked. Stop-grad has higher SongBench and AudioBox PQ, and the complete table preserves these trade-offs. Together with planner replacement, these controls examine both the full generation interface and discrete outputs under a fixed renderer.

We also clarify the source of the joint gradient: the LM receives gradients from semantic flow matching through the readout. The acoustic renderer remains frozen during this stage; gradients from a final audio reconstruction loss are not propagated through the acoustic renderer to the LM.

Revisions: Sec. 4.3 (p. 8, line 378; Table 2 at line 384) adds the interface ablations. Sec. 3.2 (p. 5, line 216) clarifies the three sequences and the FM-to-LM gradient path.

AuthorsOpenReview ↗

Response to Fifc (2/3): Feature extraction and generation cost

Fifc-3 | Feature extraction, generation complexity, and efficiency (W1)

The audio encoders are used only during feature extraction and do not run when generating songs. The VQ tokenizer and SSL-VAE provide discrete tokens and continuous semantic targets for training, respectively. The SSL encoder/teacher serves representation learning or probing, and the LeVo-VAE encoder provides acoustic latent targets. Standard generation runs LM/readout → Semantic DiT → acoustic renderer → waveform decoder; the waveform decoder has a different role from the encoder used for feature extraction. Joint training updates the LM, readout, and Semantic DiT while keeping the representation encoders and acoustic renderer frozen.

Your comments on runtime prompted us to identify overhead from forced CUDA synchronization and PCIe transfers at every step in the hidden-state readout path. After the fix, Joint-STD-48D's RTF is corrected from 1.879 to 1.573. Discrete has an RTF of 1.511 under the same timing conditions, so the main model adds approximately 4.1% inference time. The corresponding micro-PER is 14.47% versus 20.26%, SongEval is 4.269 versus 4.152, and sound-quality MOS is 4.14 versus 3.14.

The lower sound quality of Joint-LeVo-64D that you highlighted is indeed present: its MOS is 3.04, below Discrete's 3.14 and the standard 48D pathway's 4.14. This single-flow configuration has a current RTF of 1.565, only 0.008 below the standard 48D model's 1.573. Thus, in the current complete-system comparison, removing a generation stage does not improve sound quality, while runtime remains close to that of the standard pathway. This is an important empirical reason for retaining the standard continuous semantic generation pathway.

Most of the main model's runtime comes from AR planning:

SystemToken planningSemantic flowAcoustic rendering
Discrete98.3%0.00%1.69%
Joint-STD-48D98.0%0.38%1.62%

The standard setting uses 32 full-sequence semantic-flow updates and 32 acoustic-rendering updates per acoustic window. For the same 150-second song with Joint-STD-48D, reducing these from 32/32 to 8/16 steps lowers the combined time of the latter two stages from 4.89 to 2.49 seconds, saving 2.40 seconds. With other overheads fixed, this corresponds to an RTF reduction of 0.016. When AR planning dominates runtime, reducing later sampling steps has a smaller effect on end-to-end speed.

Revisions: Sec. 3.2 (p. 5, line 250) clarifies that encoders are used only for feature extraction and identifies the modules used at generation time. Table 1 (p. 7, line 336) corrects RTF. Sec. 4.5 (p. 8, line 426), Table E.1 (p. 17, line 868), and Table E.2 (p. 17, line 883) add stage timing and sampling-step analysis.

AuthorsOpenReview ↗

Response to Fifc (3/3): Training data and vector figures

Fifc-4 | Dataset scale and use across training stages (Q2)

The tokenizer, LM, Semantic DiT, and Acoustic DiT training stages in this work use the same deduplicated dataset. It contains 19,521.8 million semantic tokens at 25 Hz, corresponding to approximately 216,909 hours of audio.

The language distribution is 47.44% Chinese, 41.29% English, 7.91% Spanish, 2.11% Japanese, and 1.25% Korean. Genre labels cover 24 categories:

Genre labelShareGenre labelShare
Dance7.68%Country3.73%
Rap7.23%Jazz3.73%
Classical6.72%British3.51%
World music6.67%Post-rock3.36%
Gufeng6.30%Latin3.30%
Metal5.66%Electronic2.72%
Blues5.60%Easy listening2.48%
Punk5.44%Reggae2.34%
New Age4.80%Rock2.02%
Pop4.52%Bossa Nova1.88%
Other3.99%Folk1.69%
Ethnic music3.76%R&B0.85%

Language and genre percentages are normalized separately by their respective record/label frequencies. Genre percentages sum to 99.98% after rounding to two decimal places. These distributions describe the language and style coverage of the shared training data.

Revisions: The Data paragraph in Sec. 4.1 (p. 6, line 307) describes the shared training corpus and its scale. Appendix D (p. 16, line 834; Table D.1 at line 844) reports the shared use across stages, language percentages, and complete genre distribution.

Fifc-5 | How Figure 2 was made and its vector redraw (Q3)

AI assistance was used to adjust the layout of the original figure. Thank you for pointing out the inconsistent colors and line weights. We have redrawn Figures 1–2 according to the actual model architecture, standardized their colors, line weights, fonts, and symbols, and replaced the originals with vector PDFs so that lines and text remain sharp when enlarged.

The redrawn Figure 2 distinguishes discrete token d, readout condition c, and continuous semantic feature s. It shows the token cross-entropy and semantic flow-matching training paths, including the connection through which FM gradients update the LM via the readout. The caption also states that the representation encoders and acoustic renderer remain frozen, aligning the diagram with the training procedure.

Revisions: Figures 1–2 have been replaced with vector diagrams (pp. 3–4, captions starting at lines 148 and 178), and their captions have been updated.

AuthorsOpenReview ↗

Follow-up on Discussion

Dear reviewer Fifc, thank you again for your time.

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its deadline. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns.

Thanks very much for your consideration.

Reviewer_RQPGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a jointly trained framework for generating complete songs from lyrics and style descriptions, combining discrete-token planning with continuous semantic generation. Its main contribution is a differentiable readout of the language model’s hidden states, which allows the semantic flow-matching loss to update the planner. Experiments compare a discrete baseline with five proposed configurations varying representation dimensionality and generation architecture. Evaluation uses 192 shared WildSongBench prompts to measure song quality, style alignment, phoneme error rate, and generation time, supplemented by five experts rating ten matched song groups for sound quality and musicality. MARBLE representation probes and source-conditioned reconstruction on 100 in-house songs provide additional validation of representation utility and information preservation. The representative 48D model improves SongEval from 4.152 to 4.269 and reduces phoneme error rate from 20.26% to 14.47%, while also improving expert sound-quality ratings. However, these results demonstrate system-level improvements; further controlled ablations are needed to isolate the contributions of joint training, continuous representations, and the acoustic renderer.

Critical strengths
  1. The representative 48D model improves both lyric fidelity and perceived sound quality, reducing PER from 20.26% to 14.47% and increasing expert sound-quality ratings from 3.14 to 4.14. These improvements are consistent with gains in automatic quality metrics, although their statistical reliability requires further assessment.

  2. The evaluation covers multiple levels, combining automatic and expert assessments of generated songs with MARBLE representation probes and source-conditioned reconstruction on 100 songs. This provides evidence beyond final generation scores and helps examine which musical information the intermediate representations preserve.

  3. The configuration comparisons reveal useful trade-offs: increasing representation dimensionality does not consistently improve generation, and the strongest configuration on automatic quality metrics differs from the one that performs best on lyric fidelity and expert ratings. The authors explicitly acknowledge these distinctions and the limits of their analysis, avoiding claims of universal superiority or definitive causal explanations.

Critical weaknesses
  1. The paper lacks controlled ablations that isolate its central contribution, as joint training, the continuous representation, and the acoustic renderer differ between the baseline and the proposed system. Consequently, it is difficult to determine whether the improvements result from propagating the flow-matching loss to the language model or from changes to the representation and renderer.

  2. The comparisons primarily involve an in-house discrete baseline and internal variants, limiting the evidence for competitiveness against existing external systems. Joint-LeVo and Joint-SB are internal configurations rather than the original LeVo and SongBloom models, while the LeVo 2 reference in Figure 3 follows a different candidate-selection protocol.

  3. The quality improvements come with additional computational cost, and the standard pipeline must complete the entire token plan before running the semantic flow. Under the reported evaluation conditions, every joint configuration has a higher RTF than the discrete baseline, and the authors acknowledge that the experiments do not establish a computational advantage.

Questions for authors
  1. To isolate the effect of joint training, could the authors compare the full model with a variant that retains the readout but blocks flow-matching gradients to the LM, keeping the representation, renderer, data, and training budget fixed? A token-conditioned variant without the readout would further distinguish the value of continuous conditioning from the benefit of updating the planner.

  2. How does the proposed method perform against LeVo 2 or SongBloom under a common evaluation protocol covering prompts, duration requirements, candidate selection, and ASR scoring? This is particularly relevant to Figure 3, where the LeVo 2 reference follows a different candidate-selection procedure.

  3. Please provide a breakdown of generation time across token planning, semantic flow, and acoustic rendering, with hardware and output duration held constant. Do the quality gains persist at comparable runtime or computational cost, for example after adjusting the number of sampling steps?

6 comments
AuthorsOpenReview ↗

Response to RQPG (1/3): Joint-training and planner controls

Thank you for your careful and thorough review and constructive comments, and for recognizing the results on lyric fidelity, expert listening evaluations, and evaluation at multiple levels. Your suggestions helped us strengthen the experimental evidence and clarify the method's contributions and computational cost. We added controlled experiments, re-evaluated the original external models under a common protocol, and analyzed generation time by stage. We address your points below.

All page and line references below refer to the revised manuscript. Revisions are marked in blue for ease of reference.

RQPG-1 | Joint training and the readout (W1 / Q1)

Following your suggestion, we compare Full, Stop-grad, and w/o readout with the same 48D continuous representation, initialization, training data, training budget, and acoustic renderer on the same 192 prompts. Full uses both token and readout conditions, with FM gradients updating the LM through the readout. Stop-grad retains the readout condition but blocks FM gradients to the LM: the readout and Semantic DiT continue to learn, while the LM is updated by cross-entropy. The w/o readout variant uses only token conditioning to generate the same continuous targets.

Configurationmicro-PER (%) ↓SongEval ↑SongBench ↑AudioBox PQ ↑MuLan ↑MOS quality ↑MOS musicality ↑
Full14.474.2696.2828.3000.5144.143.26
Stop-grad16.544.2296.2948.3500.5093.783.18
w/o readout17.464.2306.2708.3240.5133.663.12

Relative to Stop-grad, Full reduces micro-PER by 2.07 percentage points, increases SongEval by 0.040, and improves quality/musicality MOS from 3.78/3.18 to 4.14/3.26. This directly tests the effect of FM updates to the LM while holding the continuous representation and acoustic renderer fixed. Stop-grad reduces micro-PER by 0.92 percentage points relative to w/o readout, with similar SongEval, showing that readout conditioning itself helps lyric fidelity even when FM-to-LM gradients are blocked. Stop-grad has higher SongBench and AudioBox PQ; the main benefits of joint training appear in lyric fidelity, SongEval, and expert ratings.

We also keep the original Discrete renderer and codebook interface fixed, supply no readout to the renderer, and replace only the token planner. Full-joint LM achieves 18.74% micro-PER, versus 20.26% for the original LM and 20.72% for an LM continued with cross-entropy alone. Their respective SongEval scores are 4.184, 4.152, and 4.164, and quality/musicality MOS scores are 3.22/2.88, 3.14/2.76, and 3.08/2.82. Thus, the benefits also appear in discrete token outputs under the original renderer. The CE-only control further separates joint optimization from additional cross-entropy training.

Revisions: Sec. 4.3 (p. 8, starting at line 378; Table 2 at line 384) adds the controlled experiments, with planner replacement results starting at line 398. Sec. 3.2 (p. 5, line 216) clarifies discrete tokens, readout conditions, continuous targets, and the FM-to-LM gradient path.

AuthorsOpenReview ↗

Response to RQPG (2/3): External comparisons under a common protocol

RQPG-2 | Original external models and a common evaluation protocol (W2 / Q2)

We re-evaluated the original LeVo 2 and SongBloom checkpoints. The new comparison includes Joint-STD-48D, Discrete, LeVo 2, and SongBloom, using the same 192 WildSongBench prompts, duration and retry rules for song retention, and scoring procedure. Joint-LeVo and Joint-SB remain explicitly identified as internal configurations.

The standard procedure retries token sequences shorter than 30 seconds in a fixed seed order, retaining the first result that meets the duration requirement, or the longest result if none does. Each retained waveform is transcribed four times with Qwen3-ASR. We select the lowest-error transcript and compute micro-PER as the total phoneme edits divided by the total reference phonemes across songs. The selection among four ASR transcripts concerns the same retained song; songs themselves are retained by the duration and retry rules. All quality, style, and lyric metrics refer to the songs retained for each system.

Systemmicro-PER (%) ↓SongEval ↑SongBench ↑AudioBox PQ ↑MuLan ↑MOS quality ↑MOS musicality ↑
Full-48D14.474.2696.2828.3000.5144.143.26
Discrete20.264.1526.2178.2820.4873.142.76
LeVo 228.734.1326.2778.3610.3943.282.64
SongBloom21.293.5804.2578.1540.2142.722.38

Full-48D has lower micro-PER and higher MuLan, SongEval, and both MOS scores. Its SongBench is close to LeVo 2, while LeVo 2 has higher AudioBox PQ. This common-protocol evaluation directly addresses external competitiveness. The external models differ in training corpora, scale, and pretraining methods, and our paper also places particular emphasis on comparisons among configurations of the proposed method.

Revisions: Sec. 4.1 (p. 6, line 311) clarifies song retention, ASR transcript selection, and micro-PER aggregation. Sec. 4.4 (p. 8, line 407; Table 3 at line 414) reports the external evaluation. Figure 3 (p. 7, caption at line 366) replaces the external reference points with these re-evaluated LeVo 2 and SongBloom results.

AuthorsOpenReview ↗

Response to RQPG (3/3): Runtime corrections and expert agreement

RQPG-3 | Computational cost, corrected RTF, and sampling steps (W3 / Q3)

Your suggestion to profile individual stages helped us identify overhead in the hidden-state readout path: forced CUDA synchronization and PCIe transfers at every autoregressive step interrupted the inference pipeline. The original timing included this implementation overhead. After fixing the path, we repeated the measurements under the same timing conditions and corrected Table 1:

SystemOriginally reported RTFCurrent RTF
Discrete1.5111.511
Joint-STD-48D1.8791.573
Joint-STD-96D1.7791.542
Joint-STD-128D1.6821.517
Joint-LeVo-64D1.7281.565
Joint-SB-48D2.4342.372

Joint-STD-48D has an RTF of 1.573, approximately 4.1% above Discrete's 1.511. Under this standard configuration, micro-PER is 14.47% versus 20.26%, SongEval is 4.269 versus 4.152, and quality/musicality MOS is 4.14/3.26 versus 3.14/2.76. This quantifies the additional inference time associated with the main model's quality gains. Joint-STD-128D has an RTF of 1.517, close to Discrete, but exhibits different quality trade-offs from 48D; we therefore report each configuration separately.

The stage time shares for the main model and baseline are:

SystemToken planningSemantic flowAcoustic rendering
Discrete98.3%0.00%1.69%
Joint-STD-48D98.0%0.38%1.62%

Table E.2 compares two sampling settings using Joint-STD-48D and the same 150-second song. Reducing semantic-flow/acoustic-rendering steps from 32/32 to 8/16 lowers their combined time from 4.89 to 2.49 seconds, saving 2.40 seconds. With other overheads fixed, this reduces RTF by 0.016. Acoustic updates are counted per rendering window. Since token planning accounts for approximately 98.0% of the total time, reducing subsequent sampling steps has a smaller effect on end-to-end latency.

Revisions: Table 1 (p. 7, line 336) corrects RTF. Sec. 4.5 (p. 8, line 426), Table E.1 (p. 17, line 868), and Table E.2 (p. 17, line 883) add stage timing and sampling-step results.

RQPG-4 | Expert agreement (regarding Strength 1)

We supplement the existing listening study of five music experts, ten matched song groups, and 1–5 ratings with Krippendorff's alpha: 0.77 for sound quality, with 95% CI [0.62, 0.83], and 0.63 for musicality, with 95% CI [0.42, 0.73].

Revisions: Sec. 4.1 (p. 7, agreement results at line 324) and Appendix E (p. 16, line 863) add these statistics. Tables 2 and 3 (p. 8, starting at lines 384 and 414) report all automatic metrics alongside both MOS scores.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper 39247 Rebuttal Summary

We thank the reviewers for their constructive feedback. We summarize the new experiments and analyses in the revised manuscript below.

Reviewer Scores and Discussion Outcomes

ReviewerOverall RatingConfidenceOutcome After Rebuttal
RQPG32Confirmed that we addressed their questions and declined to raise the rating further.
Fifc33No follow-up comment during the discussion.
UjFn2 → 32Confirmed that we addressed their concerns, raised their rating, and asked us to include the additional experimental details and analysis in the paper.

Reviewer RQPG

The reviewer asked for flow-matching gradient and readout controls, external comparisons under a common protocol, and generation timings.

  • Controlled ablations. We compared Full, Stop-grad, and w/o readout with the same 48D representation, initialization, training data, training budget, and acoustic renderer. We also compared the planners with the original discrete renderer fixed. §3.2, §4.3, Table 2.
  • External comparisons. We re-evaluated the original LeVo 2 and SongBloom checkpoints using the same prompts, duration and retry rules, and scoring. We clarified ASR transcript selection and micro-PER aggregation. §4.1, §4.4, Table 3, Figure 3.
  • Generation cost. Profiling exposed forced CUDA synchronization and PCIe transfers in the readout path. We fixed the implementation, remeasured runtime, and added stage timings and a sampling-step comparison. §4.5, Table 1, Appendix E.

Reviewer Fifc

The reviewer asked about discrete planning, inference components, data use across stages, and how Figure 2 was produced.

  • Discrete token planning. With the original discrete renderer and codebook interface fixed, the jointly trained planner improved lyric fidelity over the original and cross-entropy-only planners. §4.3.
  • Architecture and inference cost. We clarified which modules extract reference-audio features and which run during generation. We reported corrected real-time factors and stage timings. §3.2, §4.5, Table 1, Appendix E.
  • Training data and figures. We reported the shared corpus’s language and genre distributions and redrew Figures 1–2 as vector diagrams of the architecture and training paths. §4.1, Appendix D, Figures 1–2.

Reviewer UjFn

The reviewer asked for evidence supporting joint training, analysis of losses and representation dimensionality, external comparisons, expert agreement, and data statistics.

  • Motivation and attribution. We clarified how the continuous target space and differentiable readout address two distinct limitations of discrete planning. The new ablations and fixed-renderer planner comparison test readout conditioning and flow-matching gradients to the planner. Introduction, §3.2, §4.3, Table 2.
  • Loss design and representation dimensionality. In the rebuttal, we explained the existing SSL-VAE and joint-training losses and their normalization. We used existing generation, probe, and reconstruction results to discuss why better reference-feature probe scores can coexist with weaker free-generation results. Appendices A–B, Tables 1, 4–5, §5.
  • Evaluation and data. The revised manuscript includes the requested external comparisons, expert agreement statistics, and the scale and composition of the shared corpus. §4.1, §4.4, Table 3, Figure 3, Appendices D–E.

Key Results in the Revised Manuscript

RevisionMain ResultLocation
Joint-training and readout ablationsFull, Stop-grad, and w/o readout had micro-PERs of 14.47%, 16.54%, and 17.46%, respectively. Sound-quality MOS was 4.14 for Full and 3.78 for Stop-grad.§4.3, Table 2
Fixed-renderer planner comparisonThe jointly trained, original, and cross-entropy-only planners had micro-PERs of 18.74%, 20.26%, and 20.72%, respectively.§4.3
External comparisonsOn the same 192 WildSongBench prompts, micro-PER was 14.47% for Joint-STD-48D, 28.73% for LeVo 2, and 21.29% for SongBloom. Joint-STD-48D scored higher than both on SongEval and expert MOS. LeVo 2 scored higher on AudioBox PQ.§4.1, §4.4, Table 3, Figure 3
Runtime correction and profilingJoint-STD-48D’s real-time factor was corrected from 1.879 to 1.573, versus 1.511 for the discrete baseline. Token planning accounted for 98.0% of its generation time.§4.5, Table 1, Appendix E
Expert agreementKrippendorff’s α\alpha was 0.77 for sound quality (95% CI [0.62, 0.83]) and 0.63 for musicality (95% CI [0.42, 0.73]).§4.1, Appendix E
Training corpus and diagramsDocumented approximately 216,909 hours of shared training data and its language and genre distributions. Redrew the overview and architecture as vector diagrams.§4.1, Appendix D, Figures 1–2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.