- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes , a zero-shot singing style transfer framework based on a structured lyrics–melody bottleneck. It transcribes full-mix audio and supplied lyrics into timed notes, note–lyric assignments, and melisma groups, then generates singing conditioned on this symbolic score and a target-singer reference. Experiments on Mandarin singing report improved transcription, style similarity, naturalness, intelligibility, and predicted quality over the evaluated baselines.
Critical strengths
-
The structured bottleneck provides a well-motivated way to reduce source-style leakage from frame-level F0 and linguistic features while retaining lyrics and melody.
-
The framework unifies SVS and SVC through a shared score representation and achieves strong transcription and subjective generation results, especially in style similarity and naturalness.
Critical weaknesses
-
The generation-side evidence lacks controlled ablations. It is therefore difficult to determine whether the gains arise from the structured bottleneck, reference representations, model/data scale, or other system differences.
-
has substantially worse source-relative FFE than several SVC baselines. The paper’s explanation that this reflects intended style variation is plausible, but the current analysis does not separate such variation from transcription or generation errors.
Questions for authors
-
Do the improvements from Joint decoding over lyrics-conditioned Raw decoding translate into better downstream SVC results when the generator and reference are fixed?
-
Could the authors report an inference-time ablation of reference codec tokens and the ECAPA speaker embedding: both, reference tokens only, speaker embedding only, and neither? This would clarify whether reference tokens transfer expression beyond singer identity.
-
Could the authors decompose the higher SVC FFE into coarse note/voicing or timing errors versus within-note variations such as vibrato and pitch transitions? This distinction would materially affect whether the FFE degradation should be interpreted as style transfer or reduced melody fidelity.
4 comments
Response to Reviewer 9A4S (Part II)
3. Interpretation of source-relative FFE
Thank you for raising this question. We used FFE as an objective measure of melody fidelity because few metrics separate the underlying melody from differences in how it is sung. We therefore also asked listeners to rate singing accuracy. Separating melody from singing style is one aim of this work, and we believe our transcriber can help future studies recover a note-level representation of melody across different performances.
To understand what the higher source-relative FFE means for melody accuracy, we compared the F0 of each system’s output with the official HSD MIDI annotations. We applied the same procedure to the original vocal. For each reference MIDI note, we retain the central 60% of its duration and identify frames with a finite, positive F0. We count the note as correct only if at least half of the frames in that window have valid F0 estimates and the median of those estimates, converted to MIDI pitch and rounded to the nearest semitone, exactly matches the reference pitch. We compute accuracy over all reference notes in each test pair and average the resulting accuracies equally across pairs.
| System | Quantized F0 note accuracy (%) ↑ |
|---|---|
| Original vocal | 73.347 |
| SoulX-Melody | 72.889 |
| Vevo2-Style | 54.510 |
| SoulX-SVC | 72.660 |
| Vevo2-FM | 70.242 |
| S³ | 77.624 |
S³ achieved the highest note accuracy. Even the original vocal matched the reference MIDI on only 73.347% of notes under this criterion. This illustrates how a sung pitch contour can differ from the written score. It is possible that expressive pitch variation contributes to that difference.
SoulX-SVC had roughly one-tenth the source-relative FFE of S³, yet S³ matched the reference MIDI on more notes (77.624% vs. 72.660%). On this note-level measure, S³ preserves the melody at least as accurately as SoulX-SVC. The listening results are consistent with this finding, with mean singing accuracy of 4.220 for S³ and 4.106 for SoulX-SVC. The examples on our demo page also let listeners judge melody accuracy directly. Audio examples.
Conclusion
Thank you again for your thoughtful suggestions. We hope the additional experiments and explanations address your concerns, and we would be happy to answer any further questions.
References
- Du, Z., et al. (2025). CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training.
- Hu, H., et al. (2026). Qwen3-TTS Technical Report.
Thank you for the detailed response. The Raw+SOFA vs. Joint comparison and note-level F0 analysis are useful and clarify the downstream benefit and FFE interpretation. I encourage the authors to include these results and evaluation details in the final manuscript. The reference-conditioning ablation remains incomplete, and overall I will keep my current score unchanged.
Thank you for the feedback. We will include the Raw + SOFA vs. Joint comparison and the note-level F0 analysis, along with their evaluation details, in the revised manuscript.
Response to Reviewer 9A4S (Part I)
Thank you for your thoughtful review. We address your questions below with additional experiments and clarifications.
1. Downstream SVC results with S³ Raw + SOFA and Joint decoding
Thank you for suggesting this comparison. To test how the choice of transcriber affects conversion, we kept the S³ generator, test pairs, and target-singer references fixed. In the Raw setting, we used the MIDI notes from S³ Raw and applied SOFA to assign the supplied lyrics to those notes. In the Joint setting, our transcriber recovered the notes and their lyric assignments together. We evaluated the native 24 kHz outputs in both settings using the same baselines as in our SVC experiment.
SVC objective scores
SVC listening-test means (1–5)
Listeners gave the two settings nearly identical mean ratings for timbre and style similarity. With Joint decoding, CER fell from 3.186% to 1.510%, and listeners rated the singing higher for both naturalness (4.082 vs. 3.541) and accuracy (4.220 vs. 3.983). These ratings may be related, since an inaccurately sung passage can also sound less natural. The gains in transcription thus carry over to the singing produced by the same generator.
2. Contributions of reference codec tokens and the ECAPA speaker embedding
Thank you for suggesting this ablation. Comparing both inputs, reference tokens alone, the speaker embedding alone, and neither input would help us understand how each contributes to reproducing the reference singer’s timbre and style. During training, the model received both inputs in most examples. We applied a small amount of dropout independently to each input, so the model had limited training with either input alone. With only one input, the model can produce less accurate singing as well. This makes an inference-only ablation of the original checkpoint difficult to interpret. A drop in listener ratings could reflect both the loss of timbre or style cues and the model’s limited experience singing with a single input. Such a comparison would therefore tell us little about what each representation contributes to timbre and style. We are fine-tuning the model to work reliably with either input alone, but we may not have the results in time to report them during the rebuttal period.
We can share one observation from our early experiments. The reference tokens seemed to convey fine details of the singer’s style, including their accent and how they pronounced particular syllables. The speaker embedding seemed especially important for keeping the voice consistent throughout a song. When we used reference tokens alone, the opening lines could closely match the reference in both timbre and style, but the voice gradually drifted away from the reference as the song progressed. We heard this even when the model had access to the preceding audio it had generated. One possible explanation is that small deviations in timbre accumulate during autoregressive generation.
Our main contribution is the structured symbolic bottleneck and the transcriber that recovers it from a recording. For voice conditioning, we followed an approach used successfully for voice cloning in CosyVoice 1, 2, and 3 and in the in-context voice-cloning mode of Qwen3-TTS Base, which combine reference speech tokens with a speaker embedding (Du et al., 2025; Hu et al., 2026). The proposed ablation would clarify the roles of these inputs in S³. The transcription results evaluate how accurately the bottleneck is recovered, and the conversion results show how well it works with both reference inputs present. Thank you again for the suggestion. We will complete the ablation as soon as possible and include the results in the final manuscript.