acceptodds
Withdrawn from ICLR 2027

SMILE: Section-Aware Music-Driven Lyric Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Music-driven lyric generation seeks to create lyrics that are semantically coherent across a song and aligned with its musical phrasing and section structure. A song's sections, such as verses, choruses, and bridges, organize where lyrical ideas develop, recur, and change, making them an effective interface between musical structure and lyric generation. However, existing approaches often rely on externally supplied lyric conditions or extract musical cues without inferring sections to organize the complete lyrics. To connect these levels of organization, we introduce SMILE (Section-aware Music-drIven Lyric gEneration), a framework that generates full-song lyrics from a complete recording and aligned notes without user-provided themes, lyric templates, or drafts. First, a Music Section Analyzer identifies section boundaries and functions. Within these sections, a Lyric Line Planner assigns notes to lines and predicts their target syllable counts, providing a common plan for lyrical form and local musical conditioning. A Section Lyric Generator then follows this plan to produce all passage drafts, using the corresponding notes and shared semantic context from the recording. Finally, a Cross-Section Lyric Refiner uses the complete draft and optional musical relationships to coordinate passage development and repetition while preserving the line plans. In the evaluated section-level comparisons, SMILE outperforms state-of-the-art lyric-generation baselines in music-lyric association, syllable-count accuracy, and section-format adherence.

Primary area:
generative models
Keywords:
Music-driven Lyric Generation, Multimodal Learning, Generative Models, Music Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_8GpXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper clarifies that existing approaches generate lyrics without inferring sections to organize the complete lyrics, so the authors propose SMILE, which can generate lyrics considering both music passages and external content. SMILE first infers the structure from the whole music, then analyzes the structure and generates lyrics temporally aligned with the music syllables, with paragraph refinement before output. Experimental results show that some metrics are improved using SMILE than other baselines.

Critical strengths
  1. define a new pipeline that generates lyrics considering both music and external supplied content;
  2. propose an algorithm to distinguish sections only from the music itself, and propose a planning process before lyrics generation with a global refinement after generation. The pipeline sounds complete and useful;
  3. some metrics are improved using this pipeline.
Critical weaknesses
  1. It seems that this work only proposes a pipeline with some tricks in algorithm, but without improvement in model structure;
  2. I am not sure whether the proposed algorithm can make sense, for example, in MSA, can the probability be correctly inferred only with hidden state input? Is it more reasonable to use attention network which can take the contextual relationship into consideration?
  3. The experimental results highly depend on oracle and reference contents, it cannot confirm the general generation ability of the model;
  4. The evaluation set is tiny for objective experiment and there is no human evaluation to support the conclusion.
  5. The ablation study seems that "time + line" cannot outperform "time only" and "line only" in most metrics,.
  6. I am not sure whether numbers like "[0.898, 6.319]" in Line 395 and "[−14.58, 39.58]" in Line 431 refer to the 95%CI (because no explanation), if yes, the average of these two numbers seems not equal to the number in front of it, and the interval is too wide which undermines the credibility
Questions for authors
  1. Is there any improvement in the model structure? or just define a new pipeline with some tricks in algorithm?
  2. Can SMILE generate lyrics without reference paragraph, line intervals, syllable counts, etc.? And how is the performance?
  3. "These sources may be combined within one method." are stated in Introduction, so how can SMILE generates lyrics from both external prompts and the original music? Is there any cross-attention module or other adaptors?
  4. Is there any universality evaluation of using more samples not from the same distribution or any subjective evaluation results?
Reviewer_YUtqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SMILE is a four-stage pipeline for writing full-song lyrics from a recording and time-aligned notes, with no user-supplied theme, template or draft. (1) A Music Section Analyzer (note/bar encoder + temporal network) predicts section boundaries and then section functions, using a contextual BiGRU readout. (2) A Lyric Line Planner assigns notes to lyric lines by constrained BIO decoding and predicts a target syllable count for each line. (3) A Section Lyric Generator (a 36-layer LM with audio Q-Former context and gated cross-attention to note/bar tokens) writes each section. It uses a line-dependent attention bias and a ranking loss against note-mismatched negatives. (4) A Cross-Section Lyric Refiner (LoRA planner + editor, constraint controller, external content review) revises the drafts at song level while keeping the line plans fixed. Main evidence (Table 1): 64 English sections (one per song, TriDataset), generated with reference-derived (Oracle) section function, line count, per-line syllable counts and line intervals, before refinement. SMILE reaches 88.9% exact syllable match and 96.9% section format, and the highest CLaMP3 score. Baselines are SongMASS, SongComposer, Back-S, a local adaptation of You et al. (2025) and MASS→REFFLY. Component studies cover section-function readout, count-aware BIO decoding, editor reconstruction of masked sections (8 songs / 41 lines), attention-bias interventions and a music-swap intervention.

Critical strengths
  1. The decomposition is well motivated. Musical sections provide a sensible interface between structure analysis, line planning, local generation, and song-level revision. If demonstrated end to end, this would address a real gap between line-level music conditioning and coherent full-song lyric generation.
  2. The reporting is unusually candid. Failures remain in denominators, completion and evaluator eligibility are separated, paired bootstrap intervals are reported, and the appendices distinguish executed experiments from design settings. That transparency makes the evidentiary limits assessable.
Critical weaknesses
  1. The central full-song claim is not evaluated. Table 1 tests one English section per song using Oracle section function, line count, syllable counts, and reference line intervals. Those intervals determine note routing and the time bias, which is disabled in automatic inference; refinement is also off. No experiment runs predicted MSA -> LLP -> SLG -> CSLR end to end, and no experiment measures full-song coherence or cross-section consistency. The appendices indicate that important parts are not yet an established evaluated system:
  • Relation-enabled section analysis is not established for the reported run.
  • The planner study evaluates BIO segmentation without a syllable head or downstream generation; its count-aware decoder helps only variant C and hurts A/B count error.
  • Refinement hyperparameters are described as design settings, and the content-review interface is not instantiated as a reproducible component. The available component results are also weak for the claimed pipeline: function readout obtains 84.31% accuracy but only 26.30% macro-F1 against a 74.39% always-verse diagnostic; variant C line planning reaches 19.99% exact line count and 22.24% internal-boundary F1; and the editor study covers only eight songs and 41 lines, with an interval of [-14.58, 39.58] points for its advantage. Oracle-plan generation is therefore likely to overstate automatic performance.
  1. The comparison is not like-for-like, and the paper does not establish grounding in the specific music.
  • SMILE receives reference syllable counts and line timing. SongMASS and SongComposer receive line windows but no counts, while Back-S receives counts but no music.
  • Several rows are not functioning reproductions: the You et al. adaptation uses randomly initialized query tokens and completes 0/64; MASS->REFFLY completes 0/64; SongComposer completes 22/64. The format advantage is therefore confounded by privileged Oracle structure and baseline failures.
  • CLaMP3 scores are small in absolute value, and the only reported paired main-table contrast is against Back-S, which has no music input.
  • The paper's own interventions do not show reliable dependence on the matched music: donor-music NLL changes by 0.0141 [-0.0034, 0.0320], and the CLaMP3 matching contrast is 0.0034 [-0.0059, 0.0141]. Bias interventions change only 7/64 outputs and their exact-score intervals include zero. Together, these results do not support the strong "music-driven" and end-to-end full-song claims. There is also no human evaluation of lyric quality, singability, or music fit. Presentation is generally understandable, but many experimental labels are defined only in the appendices, and the name SMILE is easily confused with the main S2MILE baseline.
Questions for authors

Clarifications that could change my assessment:

  1. Please report the main generation comparison under automatic inference: predicted section functions, predicted line assignments and syllable counts, and no reference timing bias. How much do Exact, Format, completion, and CLaMP3 change?
  2. Is CSLR trained and runnable end to end? What supplies target revision plans, what implements the external content-review interface, and what full-song evidence shows improved coherence or refrain consistency?
  3. What evidence shows dependence on the specific music beyond Oracle format controls? Is SMILE significantly better than the music-conditioned baselines on a common paired set, including SongMASS?
  4. Can every baseline receive comparable structural information, including per-line counts, and can the You et al. baseline be evaluated with trained query parameters?
  5. How does the section analyzer compare with established systems on standard boundary and function metrics, and what explains the very low macro-F1? Future-work suggestion: add a human listener study of lyric quality, singability, and music fit after a genuine end-to-end run.
Reviewer_LUvbOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SMILE, which takes aligned MIDI and audio as input and generates section-aware lyrics. It uses a cascaded approach, starting from section analysis, to planning and drafting, then final refinement. It addresses the problem of existing lyrics generation systems' issue where the lyrics generated do not follow music semantics or require additional structure information. In section-level comparisons with reference lyrics plans, SMILE outperforms prior systems in syllable-count accuracy, section-format adherence and music–lyric association (CLaMP3).

Critical strengths
  1. The problem is well-motivated and not fully addressed by existing systems, as musical lyrics need to align with the music.
  2. The cascaded system provides interpretable intermediate outputs that users can adjust, and the model design (i.e., plan then refine) is justifiable.
Critical weaknesses
  1. The cascaded system might be prone to error propagation, for instance, what if the section boundary predictions are wrong? This paper lacks comparisons with end-to-end baselines (i.e., a model that receives audio and MIDI and predicts lyrics directly), which makes the cascaded system less convincing. Some ablation studies are also lacking, for instance, the custom attention implementation in Equation 5. An ablated model should be trained without the extra bias term and compared with the proposed lyrics generator.

  2. Although SMILE outperforms existing systems objectively, the paper misses subjective listening studies, and existing objective metrics do not reflect whether the lyrics are coherent with the music. An AB preference test between different systems will be beneficial. I would also recommend including supplementary listening files to showcase the results.

Other suggestions:

  1. The use of terminology is non-standard, e.g., line, passage. It might be better to define them first (e.g., lyric lines, musical phrases, MIDI notes). Also in Section 3.2 it is confusing whether MIDI or audio is used. I would recommend clearly defining these notations.
  2. Section 4.3 has no supporting results in the main paper, although the authors referenced the Appendix. The metrics used (e.g., function readout, and boundary swap) are also not explained clearly. I would recommend either moving this section to the Appendix or moving the results table to the main paper.
Questions for authors
  1. TriDataset and Popular Hooks are not cited, are these public datasets?
  2. In Table 1, You et.al and Mass-->REFFLY scored 0% on formatting. Does this mean the metric is only suited for SMILE? What causes these dramatic differences?
  3. How would users express their creative intents (e.g., the user wants to write lyrics about nature)? Although the paper's main contribution is apart from human-AI co-creation, it would be nice to think about real-world use cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.