acceptodds
Withdrawn from ICLR 2027

CLASVS: Continuous-Latent Autoregression for Length-Changing Lyric Edits from a Reference Singing Performance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reference-conditioned lyric editing changes words within a sung phrase while preserving its melody, singer identity, and naturalness. When edits change the syllable count, target lyrics must override source-correlated cues while retaining the reference phrasing. We introduce CLASVS, a continuous-latent autoregressive editor with learned stopping. Its State–Control–Transition (SCT) history-access contract keeps target lyrics and reference melody available throughout generation, returns causal semantic feedback from generated latent patches to the planner, and confines direct predecessor acoustics to local synthesis. Progressive State–Control Grounding (PSCG) learns these routes through semantic pretraining, control perturbations, and staged reconstruction without paired lyric-edit recordings or score annotations. Across two Mandarin benchmarks, CLASVS improves deletion and insertion over continuous-NAR YingMusic+ and all four edit operations over discrete-AR Vevo2. Blinded paired listening favors its naturalness and target-lyric intelligibility, complementing objective measurements of melody and singer preservation. Matched routing and training comparisons support the access design and grounding recipe. Together, these results connect reliable lyric-count editing with the quality of the finished singing, using performances and transcripts for training. Audio demos are available at https://anonymous.4open.science/w/CLASVS-65B7/index.html.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
singing voice synthesis, length-changing lyric editing, melody-preserving lyric editing, continuous-latent autoregression, reference-conditioned generation, paired-edit-free learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fVVWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reference-conditioned singing lyric editing (including edits that change syllable counts) while preserving the reference melody and singer timbre. It proposes CLASVS, a continuous-latent autoregressive model with learned stopping. For the architecture, it uses State-Control-Transition (SCT) routing: the planner takes target lyrics and reference melody as input and gets semantic summaries of generated latent patches as feedback, while a local flow-matching generator handles the acoustic generation using the previous acoustic patch. For training, it uses Progressive State-Control Grounding (PSCG), which combines semantic pretraining, control perturbations, and staged training without needing paired lyric-edit data or score annotations. On two Mandarin benchmarks across four edit operations, the model outperforms Vevo2 across the board and beats YingMusic+ on deletion and insertion (but YingMusic+ is still stronger on substitution). The authors also include objective metrics for melody and singer preservation, listening tests, and ablation studies.

Critical strengths
  • Gains in length-changing lyric editing (deletion and insertion). CLASVS outperforms YingMusic+ on deletion and insertion across both Mandarin benchmarks, and also wins the A/B listening tests on naturalness and lyric intelligibility (Table 2, Fig. 3). This matters because insertion and deletion are usually the hardest parts of singing lyric editing—changing syllable counts easily breaks the melody or causes artifacts. Seeing consistent gains in both objective editing metrics and human listening tests shows the method actually works well in practice for variable-length edits.
  • Thorough and well-controlled ablations. The authors do a good job isolating the effects of their routing and training designs (Table 3, Appendix A, Table 10). Specifically, the local vs. global acoustic access comparison keeps the rest of the model and training setups fixed, and the ablations account for the extra training budget from semantic pretraining, alongside checks on multiple seeds, freezing, perturbations, and stage ordering.
Critical weaknesses
  1. Limited novelty over related continuous-AR models. The core idea of SCT (feeding semantic summaries back into the LLM + a separate flow head for acoustics) was already used in prior work like dots.tts. While CLASVS adds melody conditioning and limits the flow head's history to the preceding patch, the paper does not clearly separate what is actually new here from existing architectures.

  2. PSCG is mostly a combination of well-known tricks (semantic pretraining, freezing, input perturbations, and staged training). Also, the routing ablation in Table 10 only tests whether raw acoustic history should go into the planner, without testing the effect of limiting the flow head to just one preceding patch.

Questions for authors
  1. Could the authors clarify what they consider the main methodological contributions of SCT and PSCG beyond this prior work(e.g. dots.tts)? Also, since Table 10 only tests acoustic input to the planner, do you have a matched comparison of one-patch vs. longer acoustic history inside Flow-DiT with all other settings fixed?

  2. In State Grounding, the feedback encoder is pretrained with both CTC and Mel reconstruction. Why include Mel reconstruction if the goal is semantic/phonetic progress tracking? Is there a comparison with CTC-only pretraining under a matched budget?

  3. What singing genres and syllable rates are covered in the benchmarks, and do the deletion/insertion gains hold for fast singing or rap-like vocals? Using the existing evaluation outputs, could you also show how lyric accuracy and melody preservation vary with the percentage of syllables added or deleted?

  4. Why choose a 100 ms patch size? Has its impact on editing accuracy been compared against shorter patches (especially for dense lyrics or fast articulation) to see the trade-off between feedback frequency and generation cost?

Reviewer_BAkSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a system for cover generation: users can change words within a sung phrase without altering its melody, singer identity, and naturalness. This is done using, according to the naming of the authors, a "State–Control–Transition (SCT) history access contract" and a "Progressive State–Control Grounding" (PSCG). Experiments are conducted on Mandarin datasets.

Critical strengths

Supporting the editing of word counts is an interesting contribution.

Critical weaknesses

The paper is hard to read, lacking flow and filled with unnecassary jargon. The writing of the methodology sections is in particular obscure, making it hard to judge the relevance of the methods and their relations with existing ones.

Moreover, the proposed system seems to be unnecassarily complicated. Simpler alternatives should have been considered and empirically evaluated as baselines. For example, given a singing phrase, we may apply singing transcription models to get the lyrics, melody notes, and note-lyrics associations. Once these are given, editing word counts seems trivial in some cases, because we can just change the note-lyrics associations.

Questions for authors

See the "Critical Weaknesses" section.

Reviewer_8eumOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses length-changing lyric editing given a reference singing clip and target lyrics without score annotations, where the words of a sung phrase are changed while its melody and singer identity are preserved. It proposes CLASVS, a continuous-latent AR editor. It uses Qwen3-0.6B planner with target lyrics and reference melody tokens to drive a Flow-DiT generating short-time audio latent patches, and a learned stop head determines the output length. Its State–Control–Transition (SCT) design returns features from a frozen semantic encoder of the generated audio to the planner, while the previous acoustic patch stays within the local Flow-DiT. The model is trained by reconstructing performances from their transcripts without paired edit data, using semantic-encoder pretraining, predecessor dropout and melody masking, and a three-stage curriculum (PSCG). On a new 320-edit Mandarin benchmark, the Mandarin portion of LyricEditBench, and a singer-held-out set, CLASVS improves deletion and insertion over YingMusic+ and all four operations over Vevo2, while YingMusic+ remains better on substitution.

Critical strengths
  1. The paper addresses an important and practical problem. Length-changing lyric edits require redistributing a different number of syllables over the original melody and rhythm while keeping the result natural, which makes insertion and deletion harder than substitution.
  2. The paper explicitly formulates which generated history should inform global lyric planning and which should only support local synthesis, and proposes a clear design that routes semantic feedback to the planner and keeps predecessor acoustics local.
Critical weaknesses
  1. This paper argues that reference-conditioned editors are at risk of source-lyric leakage and motivates the routing design for global/local history accordingly. However, source reversion (SrcRev) and target-preference margins are reported only for CLASVS. So the severity of leakage in existing work (e.g. Vevo2 and YingMusic+) and improvements brought by CLASVS remain unclear.
  2. The paper states that lyric progress and stopping are central for length-changing edits, and presents learning when to stop as an advantage of continuous-latent AR. However, the evaluation prefers stopping at the reference duration, as reference-duration deviation is lower-is-better, and premature/delayed stops are defined as endpoint deviations beyond ±200 ms from the reference. So an insertion that appropriately lengthens the phrase may be counted as a delayed stop. The operation-resolved reference timing is about 1% for all four operations (Table 15A), which suggests that the stop head largely reproduces the reference duration rather than adapting it to the edit. Also, stopping statistics are not reported for Vevo2, which is also an AR method.
Questions for authors
  1. What are the SrcRev and target-preference margins of Vevo2 and YingMusic+ on both benchmarks?
  2. Table 2A shows CLASVS outperforms YingMusic+ on Del and Ins, but underperforms on Sub operations on objective metrics. However, in Table 2C, listeners report better number on all three dimensions. I am wondering if the listeners can tell the quality difference between various operations (e.g. FSub and Del). Could the paper report per-operation MOS, especially L-MOS, to show whether listeners perceive this difference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.