acceptodds
Under review as a conference paper at ICLR 2027

SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling

Abstract

Recent song generation systems can synthesize realistic audio, yet generating complete songs remains challenging for two reasons. First, explicit song-level arrangement planning remains limited in existing methods, so models often need to organize overall arrangement development while generating low-level audio details. This often leads to incoherence in arrangements, such as weak section transitions and limited dynamic progression. Second, coarse modeling of different musical parts obscures their distinct roles and interactions, limiting arrangement richness of generated songs. In this paper, we present SketchSong, a hierarchical song generation framework that addresses these issues through song-level sketch planning and fine-grained multi-track modeling. Along the temporal dimension, SketchSong first predicts a compact sequence of high-level sketch tokens derived from compressed audio representations, and then generates audio tokens conditioned on these sketches. This coarse-to-fine process gives the model an explicit arrangement plan before detailed audio generation. Along the track dimension, SketchSong explicitly models four tracks, i.e., vocals, bass, drums and other instruments. This enables the model to capture the roles and interactions of different musical parts more precisely. Experiments on song generation benchmarks show that SketchSong consistently outperforms our baseline on both objective metrics and human listening tests.

Primary area:
generative models
Keywords:
Music Generation, Song Generation, Sketch Planning, Multi-Track Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U9qzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SketchSong, a hierarchical song generation framework built on the two-stage autoregressive paradigm of LeVo. Along the temporal dimension, the first-stage model predicts a low-rate sequence of discrete sketch tokens (MuQ-MuLan RVQ token) before generating mixed audio tokens, providing an explicit song-level plan. Along the track dimension, the second stage extends LeVo's vocal/accompaniment formulation to four explicit tracks (vocals, bass, drums, other instruments). Experiments on a 1M-song corpus show consistent improvements over a retrained LeVo baseline on objective metrics and listening tests, and ablations indicate the two components contribute complementarily.

Critical strengths
  1. The two designs are well motivated, and the evidence matches the claims unusually well. Predicting a compact MuQ-MuLan-based sketch before the audio tokens is a sensible way to give the model a song-level plan, and the ablations line up with the motivation: sketch planning mainly helps structure, musicality and memorability, while four-track modeling mainly helps production quality and instrumental richness.

  2. The sketch replacement experiment (Table 4) is probably the most convincing part of the paper. Fixing the prompt and swapping in a ballad or rock sketch shifts the output style consistently across 20 diverse prompts, which shows the sketch is a real control interface rather than an incidental gain.

Besides, the evaluation practice is honest and the paper is easy to read. The authors retrain LeVo on their own data and setup for a fair comparison while also reporting the released checkpoint, which is more rigorous than most song generation submissions I have seen. They are also candid about what the system does not do (no post-training, shared accompaniment codec, naive remixing), which makes the positive results more trustworthy.

Critical weaknesses
  1. The framework is largely an incremental combination on top of LeVo. Using MuLan-type embeddings as a low-rate planning sequence ahead of audio token generation was already established by MusiCoT (CLAP as musical CoT). The paper cites it but does not articulate what is methodologically different beyond swapping the feature extractor.

  2. The multi-track contribution is undermined by its decoding path. No dedicated codecs are trained for bass, drums, or other instruments. All three streams are decoded by LeVo's accompaniment codec, which was trained on mixed accompaniment and is therefore used out of distribution on isolated stems, and then remixed with a fixed ratio.

Also, there is no comparison with commercial systems (e.g., Suno v6) or stronger recent baselines (e.g., LeVo 2). Additionally, the reported PER metric is really poor.

Questions for authors
  1. What exactly distinguishes the sketch planning here from MusiCoT's musical CoT, in terms of representation, granularity, and training schedule? Please state this explicitly in the paper.

  2. Could you add comparisons with stronger baselines, e.g., LeVo 2, and commercial systems such as Suno-v6 if feasible? In addition, please consider including newer evaluation models such as SongBench in the objective metrics.

  3. Have you trained or fine-tuned dedicated codecs for the bass, drums, and other tracks, even at a small scale? Relatedly, is there any evidence that reusing LeVo's two-stem codecs for four-track decoding is adequate, for example codec reconstruction metrics?

Other minor issues:

  1. The paper states it uses MuQ-MuLan but cites Huang et al. (2022), the original MuLan. These are different models (MuQ-MuLan, DOI: 10.1109/TASLPRO.2025.3602320). Which one was actually used, both for sketch extraction and for the MuQ-T/MuQ-A metrics? Please clarify and fix the citation.
  2. Why is PER so much higher than the released LeVo checkpoint for both the baseline and SketchSong? A brief diagnosis would be appreciated.

I find the framework clean and the sketch controllability analysis genuinely interesting, but the novelty over MusiCoT and the comparison set are not sufficient as presented. If these points are addressed convincingly in the rebuttal, I am willing to raise my score.

Reviewer_bDfrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SketchSong extends LeVo's two-stage autoregressive song generator in two ways.

  1. Sketch planning. MuQ-MuLan features are extracted on 5 s windows (0.2 Hz) and quantized with an RVQ (four quantizers, codebook size 4,096). The first-stage LM predicts these "sketch" tokens before the mixed audio tokens. It is trained first on sketches alone, then jointly. At inference the sketch can be predicted or supplied by the user.
  2. Four stems. The second-stage LM predicts four Demucs stems (vocals, bass, drums, other) instead of LeVo's vocal/accompaniment pair, conditioned on frozen first-stage hidden states. Bass, drums and other are decoded with LeVo's accompaniment codec and remixed at 1/3 amplitude.

Training uses 1M songs (about 54k h) with automatically produced lyrics, structure labels, stems and captions. Evaluation conditions come from 200 validation songs (100 Chinese, 100 English), scored with FAD, MuQ-T, PER, Audiobox aesthetics and SongEval. A 15-listener MOS study (5 experts, 10 amateurs) rates overall quality, structure clarity, arrangement development, instrumental richness and vocal quality for the internal variants. Against LeVo retrained on the same data, the full model improves FAD 3.73 → 3.06, PER 32.3 → 27.8, MOS overall 3.60 → 3.68 and instrumental richness 3.56 → 3.84. Released YuE, DiffRhythm 2, ACE-Step 1.5 and LeVo are also scored. Appendices add audio-to-song results and a sketch-substitution controllability study.

Critical strengths
  • LeVo is retrained on the same data, and all four variants (baseline / +sketch / +multi-track / both) are reported. The full model improves all 12 automatic metrics over that baseline (Table 1), and the MOS study covers the same four variants. This factorial comparison is useful.
  • Replacing the sketch shifts the output toward the reference across 20 prompts (MuQ-MuLan similarity 0.643 → 0.828 for the ballad sketch, 0.643 → 0.770 for rock; Table 4). This is a practical feature.
Critical weaknesses
  1. The evidence that sketches act as an arrangement plan, rather than a global style condition, is thin.

    • Human ratings only. The structure evidence is the MOS ratings of structure clarity and arrangement development (sketch-only: SSC 3.76 → 3.88, AD 3.62 → 3.90) plus one qualitative spectrogram example. These means come from 15 listeners with no CIs or tests. There is no objective structure measure, such as agreement between generated section boundaries and the input structure tags, repetition/self-similarity, or per-section energy contours. There is also no control separating temporally varying sketches from a global style condition.
    • Mixed objective ablations. SongEval coherence decreases slightly with sketches (3.52 → 3.49) and rises with multi-track (3.88). Sketches also raise production complexity more than multi-track does (5.80 → 6.11 vs 5.98), even though Sec. 4.2 associates multi-track with production complexity. Multi-track's PQ gain (8.35 → 8.43) and the subjective ratings do support parts of the proposed division of benefits.
    • Fig. 4 (right). The y-axis offset is 1e−7 + 0.999999, so the curves vary by only about 2.5 × 10⁻⁷ around one. That is consistent with a normalization or numerical-precision artifact. The legend reads "Ref sketch (soulful ballad)" / "Ref sketch (rock and roll)", while A.4 describes the energy of the generated songs. As presented, the panel does not substantiate "substantially higher energy" in the generated audio.
    • Shared metric family. Controllability (A.4/A.5) is measured with MuQ-MuLan, the same feature family the sketch is built from, so it shows that the conditioning is followed rather than that planning happens. More generally, MuQ-T/MuQ-A and the five SongEval dimensions (whose evaluator is built on MuQ) share a representation family with the sketch. The four Audiobox scores, PER and the MOS ratings are the main complementary evidence, and the FAD embedding is unspecified.
    • MOS overclaim. Sec. 4.3 calls OVL +0.08 and instrumental richness +0.28 "a clear perceptual benefit" without reporting uncertainty.
  2. Novelty relative to the closest work is thin, and the baseline is weak.

    • Same recipe as prior work. SongBloom (NeurIPS 2025) generates full songs from quantized MuQ "sketch" tokens, interleaved with acoustic generation. MusiCoT predicts RVQ-tokenized CLAP "musical thoughts" before the audio tokens. SketchSong is the same recipe: quantized MuQ-MuLan sketches at 0.2 Hz, predicted first. Sec. 2 discusses both and frames the distinction as combining planning with four-track refinement, but gives no experimental comparison and no account of what the sketch component adds beyond them. The four-stem stage is Demucs pseudo-labels inside LeVo's existing mixed-then-track design. MusicGen-Stem (Rouard et al., ICASSP 2025), which generates bass/drums/other stems with stem editing, is not discussed.
    • Weak retrained baseline. The retrained LeVo has PER 32.3 on this evaluation, 23.0 points above the released checkpoint (9.3). The paper attributes this to training data and post-training. For scale, LeVo's own paper reports PER 7.2 with DPO and 10.6 without it. The gap is not explained, and it is unclear whether the gains would hold over a stronger baseline.
    • FAD reference set unspecified. Evaluation conditions are drawn from the authors' validation set, and the FAD reference pool is not specified, so a distributional advantage for models trained on this corpus cannot be ruled out.
    • External systems not in MOS. None of the external systems appear in the MOS study.

Smaller points:

  • The PDF does not give the source or license of the 1M-song corpus, and has no ethics or reproducibility statement (only an AI-use statement). It also does not give LM parameter counts, the value of λ_sk, the evaluation duration/cropping protocol, or the FAD embedding; A.1 refers to LeVo and to supplementary material.
  • MuQ-MuLan is attributed to Huang et al. (2022), which is Google's MuLan. MuQ-MuLan was introduced in Zhu et al. (2025), "MuQ: Self-supervised music representation learning with Mel residual vector quantization".
  • PER is 0.35–0.41 in Table 3 but 9.3–45.16 in Table 1. Which unit is it?
  • Audio-to-song (Table 3): the full model is worse than the baseline on 6 of 12 metrics (FAD 2.53 → 2.65, coherence 3.32 → 3.25).
  • Multi-track alone hurts FAD (3.73 → 3.82); the authors attribute this to the shared codec and remixing. There is no quantitative per-stem evaluation and no stem-editing use case. Fig. 3 inspects re-separated mixtures from a single example, not the decoded stems. Broadband energy in a bass stem is not by itself "physically implausible", since bass has upper harmonics.
Questions for authors
  1. (Could change my score.) Can you report an objective structure measure, such as boundary F-measure between generated sections (from your All-In-One analyzer) and the input structure tags, for the baseline and +sketch?
  2. (Could change my score.) Under the same training, how do shuffled sketches and a time-constant (pooled) sketch compare? If constant sketches perform similarly, that would weaken the case that temporally varying plans are needed and point to global style conditioning instead.
  3. (Clarification.) What exactly is plotted in Fig. 4 (right): reference sketches or generated songs? How is "energy" defined, and why is the range about 10⁻⁷?
  4. (Clarification.) Please give MOS confidence intervals, the number of unique items across listeners, and the playback length.
  5. (Clarification.) Can you explain the retrained LeVo's PER of 32.3? Is training compute matched across the four variants? What is the FAD reference set?
  6. (Clarification.) Where does the 1M-song corpus come from, and under what terms is it used?
Reviewer_t7FmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SketchSong adds a low-rate sketch sequence and four-track refinement to a LeVo-style song generator. Sketch tokens quantize MuQ-MuLan features from five-second windows, and the complete sketch is predicted before mixed audio. A second language model uses first-stage hidden states to generate vocals, bass, drums, and other instruments. Comparisons with a retrained LeVo baseline show improvements in several text-to-song metrics. Reference substitution and audio-to-song experiments examine control through the sketches and the trade-offs introduced by multitrack generation.

Critical strengths
  • Retraining LeVo on the same data makes the internal comparison more informative than comparisons against released systems alone. In Table 1, SketchSong reduces FAD from 3.73 to 3.06 and PER from 32.3 to 27.8. Separate sketch and multitrack ablations help identify which measured properties change with each component.
  • Supplying a sketch before audio generation creates an interface for reference-based arrangement control. Appendix A.5 extends Figure 4 with substitution experiments over 20 prompts. Increased similarity to the selected reference supports an effect on generated style, although the experiment does not isolate temporal structure.
Critical weaknesses
  • Temporal planning is not isolated from additional style conditioning in the current evaluation. Sections 3.2 and A.5 use MuQ-MuLan representations to construct the sketches and measure their effect, so reference substitution primarily establishes control within that representation. Ordered section development requires evidence beyond aggregate style similarity. Table 2 supplies relevant listening judgments, with overall quality changing from 3.60 to 3.68, but omits uncertainty, distinct-song counts, and repeated-rating allocation. Reporting those details from the existing study would establish the precision of the observed changes. An order-disrupting sketch control in a revision would test whether the sequence itself drives the claimed planning benefit.
  • Combining both components does not improve every dimension reported in Tables 2 and 3. Relative to sketch-only generation, the full system raises instrumental richness from 3.62 to 3.84 while reducing structure clarity, arrangement development, and vocal quality. Audio-to-song FAD is also worse at 2.65, versus 2.53 for the baseline and 2.34 for sketch-only generation. Decoder sharing and remixing may explain these regressions, but their effects are not isolated experimentally. I would narrow the conclusion's claim of consistent improvement to the supported dimensions and identify the decoder explanation as a hypothesis.
  • Predicting a complete low-rate sketch first is not compared against the closest compatible sketch-based design. Section 2 already discusses SongBloom, while Appendix A.1 leaves the total training budgets across ablations unclear. Matched budgets and a controlled comparison against an existing sketch design would distinguish the contribution of full-sequence planning from adding a semantic intermediate representation.
Questions for authors

How many distinct songs and ratings per song contribute to Table 2? Were identities hidden, presentation order randomized, and the same cases rated across methods? Can existing ratings support intervals that account for repeated listeners and songs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.