acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Singing Style Transfer via a Structured Lyrics-Melody Bottleneck

Abstract

Zero-shot singing style transfer aims to reproduce a reference singer’s vocal identity and expressive delivery while preserving a song’s lyrics and melody. Existing conversion systems can retain unwanted traces of the original singer's style: the pitch trajectories and frame-level features used to preserve the song also encode how it was performed. We introduce , a score-conditioned, singer-prompted singing framework built around a structured lyrics–melody bottleneck. This symbolic score represents timed notes, their lyric assignments, and groups of notes sung on the same syllable (melismas), separating score-level constraints from acoustic realization. Joint lyrics–melody transcription recovers this structure from full-mix audio (vocals with accompaniment) and supplied lyrics, allowing lyric evidence to guide note selection. A reference-conditioned generator explicitly encodes this structure and uses reference audio to guide pronunciation timing and vocal expression. The framework supports both synthesis from supplied scores and conversion from recovered scores without singer-specific fine-tuning. Experiments show higher mean style-similarity and naturalness ratings for both tasks, alongside improved note transcription, note–lyric assignment, conversion intelligibility, and predicted singing quality. Audio examples are available at https://scube2027iclr.github.io/demo/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
singing voice synthesis, singing voice conversion, zero-shot singing, singing voice transcription, lyrics–melody alignment, music information retrieval, audio generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9A4sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S3S^3, a zero-shot singing style transfer framework based on a structured lyrics–melody bottleneck. It transcribes full-mix audio and supplied lyrics into timed notes, note–lyric assignments, and melisma groups, then generates singing conditioned on this symbolic score and a target-singer reference. Experiments on Mandarin singing report improved transcription, style similarity, naturalness, intelligibility, and predicted quality over the evaluated baselines.

Critical strengths
  1. The structured bottleneck provides a well-motivated way to reduce source-style leakage from frame-level F0 and linguistic features while retaining lyrics and melody.

  2. The framework unifies SVS and SVC through a shared score representation and achieves strong transcription and subjective generation results, especially in style similarity and naturalness.

Critical weaknesses
  1. The generation-side evidence lacks controlled ablations. It is therefore difficult to determine whether the gains arise from the structured bottleneck, reference representations, model/data scale, or other system differences.

  2. S3S^3 has substantially worse source-relative FFE than several SVC baselines. The paper’s explanation that this reflects intended style variation is plausible, but the current analysis does not separate such variation from transcription or generation errors.

Questions for authors
  1. Do the improvements from Joint decoding over lyrics-conditioned Raw decoding translate into better downstream SVC results when the generator and reference are fixed?

  2. Could the authors report an inference-time ablation of reference codec tokens and the ECAPA speaker embedding: both, reference tokens only, speaker embedding only, and neither? This would clarify whether reference tokens transfer expression beyond singer identity.

  3. Could the authors decompose the higher SVC FFE into coarse note/voicing or timing errors versus within-note variations such as vibrato and pitch transitions? This distinction would materially affect whether the FFE degradation should be interpreted as style transfer or reduced melody fidelity.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9A4S (Part I)

Thank you for your thoughtful review. We address your questions below with additional experiments and clarifications.

1. Downstream SVC results with S³ Raw + SOFA and Joint decoding

Thank you for suggesting this comparison. To test how the choice of transcriber affects conversion, we kept the S³ generator, test pairs, and target-singer references fixed. In the Raw setting, we used the MIDI notes from S³ Raw and applied SOFA to assign the supplied lyrics to those notes. In the Joint setting, our transcriber recovered the notes and their lyric assignments together. We evaluated the native 24 kHz outputs in both settings using the same baselines as in our SVC experiment.

SVC objective scores

SystemWavLM ↑FFE (%) ↓CER (%) ↓SingMOS ↑
SoulX-Melody0.8662.4073.3893.981
Vevo2-Style0.88827.6814.6963.992
SoulX-SVC0.8751.9922.1404.112
Vevo2-FM0.86411.1253.6224.011
S³ (S³ Raw + SOFA)0.88720.5543.1864.101
S³ (Joint, 24 kHz)0.88720.3981.5104.125

SVC listening-test means (1–5)

SystemTimbre ↑Style ↑Naturalness ↑Singing accuracy ↑
SoulX-Melody2.7212.3842.6402.835
Vevo2-Style2.3842.0351.9301.624
SoulX-SVC3.0002.7093.2794.106
Vevo2-FM2.8602.6553.3143.506
S³ (S³ Raw + SOFA)3.7203.6813.5413.983
S³ (Joint, 24 kHz)3.7193.6834.0824.220

Listeners gave the two settings nearly identical mean ratings for timbre and style similarity. With Joint decoding, CER fell from 3.186% to 1.510%, and listeners rated the singing higher for both naturalness (4.082 vs. 3.541) and accuracy (4.220 vs. 3.983). These ratings may be related, since an inaccurately sung passage can also sound less natural. The gains in transcription thus carry over to the singing produced by the same generator.

2. Contributions of reference codec tokens and the ECAPA speaker embedding

Thank you for suggesting this ablation. Comparing both inputs, reference tokens alone, the speaker embedding alone, and neither input would help us understand how each contributes to reproducing the reference singer’s timbre and style. During training, the model received both inputs in most examples. We applied a small amount of dropout independently to each input, so the model had limited training with either input alone. With only one input, the model can produce less accurate singing as well. This makes an inference-only ablation of the original checkpoint difficult to interpret. A drop in listener ratings could reflect both the loss of timbre or style cues and the model’s limited experience singing with a single input. Such a comparison would therefore tell us little about what each representation contributes to timbre and style. We are fine-tuning the model to work reliably with either input alone, but we may not have the results in time to report them during the rebuttal period.

We can share one observation from our early experiments. The reference tokens seemed to convey fine details of the singer’s style, including their accent and how they pronounced particular syllables. The speaker embedding seemed especially important for keeping the voice consistent throughout a song. When we used reference tokens alone, the opening lines could closely match the reference in both timbre and style, but the voice gradually drifted away from the reference as the song progressed. We heard this even when the model had access to the preceding audio it had generated. One possible explanation is that small deviations in timbre accumulate during autoregressive generation.

Our main contribution is the structured symbolic bottleneck and the transcriber that recovers it from a recording. For voice conditioning, we followed an approach used successfully for voice cloning in CosyVoice 1, 2, and 3 and in the in-context voice-cloning mode of Qwen3-TTS Base, which combine reference speech tokens with a speaker embedding (Du et al., 2025; Hu et al., 2026). The proposed ablation would clarify the roles of these inputs in S³. The transcription results evaluate how accurately the bottleneck is recovered, and the conversion results show how well it works with both reference inputs present. Thank you again for the suggestion. We will complete the ablation as soon as possible and include the results in the final manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9A4S (Part II)

3. Interpretation of source-relative FFE

Thank you for raising this question. We used FFE as an objective measure of melody fidelity because few metrics separate the underlying melody from differences in how it is sung. We therefore also asked listeners to rate singing accuracy. Separating melody from singing style is one aim of this work, and we believe our transcriber can help future studies recover a note-level representation of melody across different performances.

To understand what the higher source-relative FFE means for melody accuracy, we compared the F0 of each system’s output with the official HSD MIDI annotations. We applied the same procedure to the original vocal. For each reference MIDI note, we retain the central 60% of its duration and identify frames with a finite, positive F0. We count the note as correct only if at least half of the frames in that window have valid F0 estimates and the median of those estimates, converted to MIDI pitch and rounded to the nearest semitone, exactly matches the reference pitch. We compute accuracy over all reference notes in each test pair and average the resulting accuracies equally across pairs.

SystemQuantized F0 note accuracy (%) ↑
Original vocal73.347
SoulX-Melody72.889
Vevo2-Style54.510
SoulX-SVC72.660
Vevo2-FM70.242
S³77.624

S³ achieved the highest note accuracy. Even the original vocal matched the reference MIDI on only 73.347% of notes under this criterion. This illustrates how a sung pitch contour can differ from the written score. It is possible that expressive pitch variation contributes to that difference.

SoulX-SVC had roughly one-tenth the source-relative FFE of S³, yet S³ matched the reference MIDI on more notes (77.624% vs. 72.660%). On this note-level measure, S³ preserves the melody at least as accurately as SoulX-SVC. The listening results are consistent with this finding, with mean singing accuracy of 4.220 for S³ and 4.106 for SoulX-SVC. The examples on our demo page also let listeners judge melody accuracy directly. Audio examples.

Conclusion

Thank you again for your thoughtful suggestions. We hope the additional experiments and explanations address your concerns, and we would be happy to answer any further questions.

References

Thank you for the detailed response. The Raw+SOFA vs. Joint comparison and note-level F0 analysis are useful and clarify the downstream benefit and FFE interpretation. I encourage the authors to include these results and evaluation details in the final manuscript. The reference-conditioning ablation remains incomplete, and overall I will keep my current score unchanged.

Thank you for the feedback. We will include the Raw + SOFA vs. Joint comparison and the note-level F0 analysis, along with their evaluation details, in the revised manuscript.

Reviewer_ndewOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main contribution of this paper is a lyrics-informed singing transcription system that transcribes singing audio into a symbolic score specifying three elements: timed note pitch, pitch-lyrics association, and melismas (groups of notes sung on the same syllable). With this transcribed symbolic score, the paper presents a zero-shot singing voice conversion (SVC) system that leads to higher mean style-similarity and naturalness ratings compared to existing SVC systems (that use frame-level conditions instead of a symbolic score as a condition).

The introduction nicely introduces the problem: existing SVC systems use frame-level conditions such as f0 and linguistic features that carry style-related information, leaving room to improve the content-style disentanglement.

Figure 2 nicely illustrates the proposed system. On the left hand side is the singing transcription model, and on the right side is the singing generation model that can support both singing voice synthesis (SVS) and SVC.

There are three main experiments. Experiment 1 compares the proposed singing transcription model with existing ones. Experiment 2 focuses on SVS and uses groundtruth MIDI as input (so it's unrelated to the transcription models), to validate the effectiveness of the singing generation model itself. Experiment 3 combines the transcription model and singing generation model to realize SVC and empirically compares the proposed SVC system with existing SVC systems. For the purpose of this paper, Experiments 1 and 3 are more important, while Experiment 2 is like a sanity check.

All three experiments show positive results, so the authors claim that the proposed methods are effective.

I think the paper is in general interesting. Although the authors did not make the claim explicitly, it might be the first model that combines singing transcription with SVS to realize SVC. As such, the realization of this high-level idea is important.

There is, however, room to improve the experimental design of the paper. In its current form, the experiments do not clearly demonstrate the strength of the proposed design, as detailed in the "Critical Weaknesses" section.

My impression is that this could make a good AAAI paper, but I am not sure whether it meets the bar of ICLR.

I hope the authors may address these concerns in the rebuttal period. Without further input from the authors, my current stand is weak reject.


After reading the authors' response, I have updated my score from 2 to 3.

Critical strengths
  • Could be the first system that combine singing transcription and singing generation to realize SVC.
  • The proposed joint lyrics-melody transcription model seems new and effective.
  • Could be the first model for singing voice transcription that takes melismas into account. Failing to consider melismas, as in the case of previous methods, seems to be a critical issue.
  • Nice implementation of the system and a good set of three experiments providing preliminary evidence of the success of the design.
Critical weaknesses
  • The writing can be further improved: If I were the authors, I would use more space to describe the proposed "audio->midi->audio" pipeline, and to articulate more clearly the importance of the proposed singing transcription model. SVS uses "midi->audio" and existing SVC uses "audio->something->audio", both of which are different from what the authors present here. The key is a successful "audio->midi" part that supplies the singing generation model with sufficient information.

  • The most critical issue, I think, is about the experiments, especially regarding Experiment 3 (the one on SVC). While the result presented is positive, a few important comparisons are missing, so as a reviewer I cannot gather sufficient information to judge the effectiveness of the system design.

  • First, if the core idea is the combination of "singing transcription" and "singing generation" to realize SVC, then besides the combination of the proposed transcription model and the proposed singing generation model, the authors should also have considered using the combination of "existing singing transcription" models (e.g., that provide pitch and pitch-lyrics association) with the proposed singing generation model in the SVC experiments. Otherwise, we cannot judge the relevance of the proposed singing transcription model: what if the use of existing singing transcription models here already provides decent results? Even if existing transcription models do not consider melismas, this might not matter so much---we don't know unless the authors demonstrate it.

  • Second, if I understand correctly, the baseline systems/models considered in either Experiments 2 or 3 use different training data than the proposed system. Moreover, the proposed system seems to be much bigger (in terms of model size) than the evaluated baselines. As such, it's unclear whether the performance gain (e.g., the better SVS result than Soulx-singer) is due to model size, training data size, or something else.

  • Third, the assumption that the lyrics are pre-provided is fair but also quite strong. Not all the evaluated baselines assume the availability of the lyrics. This should have been at least mentioned more clearly.

  • Finally, reproducibility can be an issue, since the authors did not promise to share code, checkpoints and data.

Questions for authors

See the "Critical Weaknesses" section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ndew (Part I)

Thank you for providing this thoughtful review. Your questions helped us see where the paper needed stronger evidence and a clearer explanation. Below we address the experiments and the changes we will make to the text.

1. How the proposed pipeline relates to existing SVS and SVC systems (answering Weakness 1)

Thank you for this profound insight. Combining transcription with SVS has been a relatively uncommon method in SVC. Recovering a correct MIDI sequence is a challenging task. Moreover, even when the MIDI sequence is correct, one practical difficulty is that a generator can sing the right pitches and still sing the wrong syllable on each note. Our contribution is a structure that preserves this relationship and a lyrics-informed transcriber that can recover it from a full mix. On HSD57, it assigns the correct pronunciation to 99.01% of notes matched in onset and pitch. We will introduce the full conversion pipeline at the point where the Introduction moves from the limitations of acoustic features to our proposed bottleneck. We will replace the last two sentences of the third paragraph (beginning “To address this entanglement”) with the following passage.

We approach conversion as synthesis from a transcribed score (Figure 1). In score-based SVS, the score is supplied. In SVC, our transcriber recovers it from the source recording and the written lyrics. This score forms a structured lyrics–melody bottleneck. It records each note’s pitch and timing, the syllable sung on it, and groups of notes that share a syllable (melismas). The generator uses this structure to preserve the song’s words and melody, while the target-singer reference guides within-note pitch modulation, phonation, and pronunciation timing.

We will also revise the opening sentence of the next paragraph, which explains why we decode notes and lyrics jointly.

To recover a score that a generator can sing from, the transcriber must determine where each new syllable begins in the melody.

2. SVC with an existing transcriber and the S³ generator (answering Weakness 2.1)

Thank you for suggesting this comparison. Using SOFA to assign lyrics to the notes from existing MIDI transcriptors such as GAME gives a direct comparison with our joint transcriber. Reviewer Vqcn raised a related question about applying SOFA to the notes from S³ Raw. We therefore evaluated both alternatives.

  1. GAME + SOFA. We used GAME, the strongest external MIDI transcription baseline in our evaluation, to transcribe the notes and SOFA to assign the lyrics to those notes.
  2. S³ Raw + SOFA. We used only the MIDI notes from S³ Raw and obtained the note–lyric assignments with SOFA.

We evaluated both as alternative SVC configurations, using the same S³ generator, dataset, and baselines as in our SVC experiment.

SVC objective scores

SystemWavLM ↑FFE (%) ↓CER (%) ↓SingMOS ↑
SoulX-Melody0.8662.4073.3893.981
Vevo2-Style0.88827.6814.6963.992
SoulX-SVC0.8751.9922.1404.112
Vevo2-FM0.86411.1253.6224.011
S³ (GAME + SOFA)0.88819.8863.1944.098
S³ (S³ Raw + SOFA)0.88720.5543.1864.101
S³ (Joint)0.88720.4331.7074.121

SVC listening-test means (1–5)

SystemTimbre ↑Style ↑Naturalness ↑Singing accuracy ↑
SoulX-Melody2.7212.3842.6402.835
Vevo2-Style2.3842.0351.9301.624
SoulX-SVC3.0002.7093.2794.106
Vevo2-FM2.8602.6553.3143.506
S³ (GAME + SOFA)3.7253.5593.5243.876
S³ (S³ Raw + SOFA)3.7203.6813.5413.983
S³ (Joint)3.7333.6864.1744.212

Listeners rated Joint higher for both naturalness and singing accuracy. S³ Raw + SOFA scored slightly higher than GAME + SOFA on both measures. Even with GAME + SOFA, our generator scored higher than every external SVC baseline for timbre similarity, style similarity, and naturalness. This supports the use of the structured symbolic bottleneck for SVC, while the lower CER and higher naturalness and accuracy ratings show the benefit of our joint transcriber.

AuthorsOpenReview ↗

Response to Reviewer ndew (Part II)

3. Differences in training data and model size across systems (answering Weakness 2.2)

Reviewer Vqcn raised the same question, which we discuss in point 4 of our response to them. At submission, we could find few baselines for these tasks that came close to S³ in either training hours or model size. SoulX-Singer gives us a direct comparison in training hours. It was trained on about 42,000 hours of singing, including roughly 20,000 hours of Mandarin, about as much Mandarin singing as our generator saw (Qian et al., 2026). Its full training corpus has not been released, so we could not train both systems on the same recordings. In SVS, where all systems received the same MIDI, lyrics, and singer reference, listeners rated S³ higher than SoulX-Score for both style similarity (4.13 vs. 3.36) and naturalness (4.47 vs. 3.40). In SVC, S³ also scored higher than SoulX-SVC on both measures (3.69 vs. 2.71 and 4.17 vs. 3.28, respectively).

For transcription, SheetSage-Jukebox uses features from the first 53 layers of the 5B-parameter Jukebox model (Donahue et al., 2022). On HSD57, both transcribers received the same full mix without lyrics. Our 746.5M-parameter transcriber achieved 88.88% onset-and-pitch F1 versus 57.52% for SheetSage-Jukebox, with a note error rate of 52.94% versus 89.13% (Table 1).

The gains hold at both stages of S³. Our transcriber recovers the melody more accurately than a baseline built on large-scale music pretraining. With comparable amounts of training data, our score-based system earns higher listener ratings for both style similarity and naturalness. A completely fair comparison is hard to achieve, but the above results demonstrate the proposed design is competitive compared to baselines with a similar training scale.

4. Availability of lyrics to each system (answering Weakness 2.3)

Thank you for pointing this out. Several of our SVC baselines work from audio alone, so we agree that readers should know which systems receive the lyrics. Conditioning on written lyrics also has precedent in melody-guided singing generation and SVC systems (Zheng et al., 2025; Zhang et al., 2025a), so we adopt this setting. We will state our assumption in the experimental setup and identify the inputs for each system in §4.4 and Appendix C.3.

SVS (§4.3). Every system receives the same ground-truth MIDI, lyrics, and target-singer reference. SoulX-Score, TechSinger, and TCSinger also receive phoneme timestamps. S³ uses the note–syllable assignments in the supplied score.

SVC (§4.4). S³ receives the original mix and the verified source lyrics. All four baselines receive the separated source vocal, and all systems use the same target-singer reference. Vevo2-Style also receives verified lyrics for both the source and the reference. SoulX-Melody, SoulX-SVC, and Vevo2-FM run without external lyrics. SoulX-Melody infers timed phonemes, SoulX-SVC extracts Whisper features, and Vevo2-FM extracts content–style tokens from the source vocal.

5. Reproducibility (answering Weakness 2.4)

Thank you for raising this. Reviewer Vqcn also asked about reproducibility. We will release our evaluation pipeline, the processed test data, and every test output from S³ and all baselines. Researchers can use these materials to reproduce the objective scores and benchmark new systems on the same examples. We are still discussing the release of the full codebase and model weights. We also plan to share a dataset of scores annotated by the S³ transcriber so the community can train and evaluate models on the same symbolic structure.

AuthorsOpenReview ↗

Response to Reviewer ndew (Part III)

Conclusion

Overall, we believe this contribution merits publication at ICLR. S³ shows that combining transcription with SVS is a practical way to perform singing voice conversion. We designed a symbolic bottleneck that preserves the melody, the timing of each note, and the way syllables span notes. Our lyrics-informed transcriber recovers this structure directly from a full mix and written lyrics, so the generator can sing the piece in a new voice without a manually aligned score.

This idea of transcribing a recording and using the score to generate a new performance is also emerging in music generation. YuE2, a powerful music generation model released around the time of our submission, transcribes a song into an ABC melody score and uses it to generate a cover (Yuan et al., 2026). Its cover pipeline supplies the melody and lyrics separately, leaving the generator to infer which syllable belongs to each note. In our preliminary tests with YuE2, we repeatedly heard syllables land on different notes from those in the original. Our bottleneck addresses this problem directly by telling the generator which syllable belongs to each note and when to carry that syllable across several notes, demonstrating the advantages of the proposed design.

Thank you again for your candid and constructive suggestions. We hope these results and explanations address your questions, and we would welcome any further feedback.

References

Reviewer_VqcnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper presents S3S^3, a zero-shot singing style transfer framework for singing voice synthesis (SVS) and conversion (SVC) that replaces source frame-level F0F_0 and phoneme timestamps with a structured symbolic bottleneck (timed MIDI notes, syllable assignments, and melisma groups) via joint full-mix transcription and reference-conditioned codec generation. S3S^3 achieves strong perceptual quality, which is very impressive in this area; however, the evaluation is undermined by an undocumented 24 kHz→48 kHz24\text{ kHz} \rightarrow 48\text{ kHz} audio super-resolution post-processing step applied exclusively to S3S^3 outputs (major problem), as well as missing ablations to separate architectural contributions from massive model and proprietary data scale (minor problems).

Strengths

  • Strong Perceptual Quality: S3S^3 delivers clear perceptual gains in timbre similarity, style similarity, naturalness, and character error rate (CER) over SVS and SVC baselines in listening and objective tests (Page 8, Tables 3–4; Page 9, Tables 5–6).
  • Effective Symbolic Bottleneck: Replacing source frame-level F0F_0 and acoustic timestamps with a 25 Hz25\text{ Hz} MIDI grid, melisma continuation tokens (⟨CONT⟩\langle\text{CONT}\rangle), and unaligned syllable tokens decouples musical structure from reference-guided articulation timing and pitch modulation (Page 6, Lines 280–286; Page 15, Lines 756–768).
  • Unified Full-Mix Transcription: The joint segmental decoder recovers note boundaries, pitches, and slur-onset melisma groups directly from full-mix audio and lyrics (Page 4, Lines 182–186; Page 5, Lines 235–269).

Weaknesses

  • [Major] Undocumented 48 kHz48\text{ kHz} Audio Super-Resolution Post-Processing and Unfair Evaluation: Page 6, Lines 304–306 state: "The codec decoder reconstructs 24 kHz24\text{ kHz} singing. We place decoded segments on the score timeline, crossfade them, and apply audio super-resolution to obtain the final 48 kHz48\text{ kHz} waveform" (repeated on Page 16, Lines 811–813). Furthermore, Page 18, Lines 962–963 explicitly state: "Both objective and subjective evaluations of S3S^3 use the final waveform after audio super-resolution." However, the paper provides zero architectural or training details for this super-resolution module, and no equivalent 48 kHz48\text{ kHz} upsampler is applied to the evaluated baselines (SoulX-Singer, Vevo2, TechSinger, TCSinger). Upsampling from 24 kHz24\text{ kHz} to 48 kHz48\text{ kHz} adds high-frequency harmonics and vocal airiness that directly inflate subjective ratings of naturalness, timbre, and breathiness-related style similarity, as well as objective SingMOS scores. The authors must specify what this super-resolution model is and report objective and subjective evaluations of S3S^3 on its direct 24 kHz24\text{ kHz} codec outputs without super-resolution.
  • [Minor] Entanglement of Structural Novelty with Data and Model Scale: S3S^3 uses a 746.5M746.5\text{M}-parameter transcriber trained on 2,5002,500 hours of pretraining data plus 8080 hours of manually annotated data, and a 1.43B1.43\text{B}-parameter Qwen3 generator trained on 20,00020,000 hours of separated dry vocals (Page 6, Lines 310–313; Page 14, Tables 7–8), while only promising to release "selected evaluation pipelines" rather than weights or data (Page 10, Lines 501–503: "We plan to release selected evaluation pipelines to support independent evaluation and comparison"). In Table 1 (Page 7, Lines 360–369), $S^3$ Raw, no lyrics (88.88%88.88\% COnP F1) and $S^3$ Joint (88.91%88.91\% COnP F1) perform nearly identically while outperforming GAME (79.18%79.18\%), yet Table 2 (Page 8, Lines 386–393) compares $S^3$ Joint (87.71%87.71\% Joint F1) against SOFA + GAME (74.66%74.66\%) without reporting SOFA + $S^3$ Raw. Similarly, Tables 3–6 (Pages 8–9) lack internal generator ablations on the same 1.43B1.43\text{B} backbone and 20,00020,000-hour dataset, making it unclear how much of the gain comes from the proposed architecture versus data and model scale.
  • [Minor] Mandarin-Only Evaluation: Page 6, Lines 313–315 ("We represent Mandarin syllables using toneless pinyin, mapping each complete syllable to one pronunciation token qjq_j") and Page 21, Lines 1086–1088 ("Training and evaluation focus on Mandarin singing... its performance beyond Mandarin has not been systematically evaluated") confirm that S3S^3 is only tested on Mandarin, leaving its effectiveness on stress-timed languages with complex consonant clusters and polysyllabic melismas unverified.
Critical strengths
  • Strong Perceptual Quality: S3S^3 delivers clear perceptual gains in timbre similarity, style similarity, naturalness, and character error rate (CER) over SVS and SVC baselines in listening and objective tests (Page 8, Tables 3–4; Page 9, Tables 5–6).
  • Effective Symbolic Bottleneck: Replacing source frame-level F0F_0 and acoustic timestamps with a 25 Hz25\text{ Hz} MIDI grid, melisma continuation tokens (⟨CONT⟩\langle\text{CONT}\rangle), and unaligned syllable tokens decouples musical structure from reference-guided articulation timing and pitch modulation (Page 6, Lines 280–286; Page 15, Lines 756–768).
  • Unified Full-Mix Transcription: The joint segmental decoder recovers note boundaries, pitches, and slur-onset melisma groups directly from full-mix audio and lyrics (Page 4, Lines 182–186; Page 5, Lines 235–269).
Critical weaknesses
  • [Major] Undocumented 48 kHz48\text{ kHz} Audio Super-Resolution Post-Processing and Unfair Evaluation: Page 6, Lines 304–306 state: "The codec decoder reconstructs 24 kHz24\text{ kHz} singing. We place decoded segments on the score timeline, crossfade them, and apply audio super-resolution to obtain the final 48 kHz48\text{ kHz} waveform" (repeated on Page 16, Lines 811–813). Furthermore, Page 18, Lines 962–963 explicitly state: "Both objective and subjective evaluations of S3S^3 use the final waveform after audio super-resolution." However, the paper provides zero architectural or training details for this super-resolution module, and no equivalent 48 kHz48\text{ kHz} upsampler is applied to the evaluated baselines (SoulX-Singer, Vevo2, TechSinger, TCSinger). Upsampling from 24 kHz24\text{ kHz} to 48 kHz48\text{ kHz} adds high-frequency harmonics and vocal airiness that directly inflate subjective ratings of naturalness, timbre, and breathiness-related style similarity, as well as objective SingMOS scores. The authors must specify what this super-resolution model is and report objective and subjective evaluations of S3S^3 on its direct 24 kHz24\text{ kHz} codec outputs without super-resolution.
  • [Minor] Entanglement of Structural Novelty with Data and Model Scale: S3S^3 uses a 746.5M746.5\text{M}-parameter transcriber trained on 2,5002,500 hours of pretraining data plus 8080 hours of manually annotated data, and a 1.43B1.43\text{B}-parameter Qwen3 generator trained on 20,00020,000 hours of separated dry vocals (Page 6, Lines 310–313; Page 14, Tables 7–8), while only promising to release "selected evaluation pipelines" rather than weights or data (Page 10, Lines 501–503: "We plan to release selected evaluation pipelines to support independent evaluation and comparison"). In Table 1 (Page 7, Lines 360–369), $S^3$ Raw, no lyrics (88.88%88.88\% COnP F1) and $S^3$ Joint (88.91%88.91\% COnP F1) perform nearly identically while outperforming GAME (79.18%79.18\%), yet Table 2 (Page 8, Lines 386–393) compares $S^3$ Joint (87.71%87.71\% Joint F1) against SOFA + GAME (74.66%74.66\%) without reporting SOFA + $S^3$ Raw. Similarly, Tables 3–6 (Pages 8–9) lack internal generator ablations on the same 1.43B1.43\text{B} backbone and 20,00020,000-hour dataset, making it unclear how much of the gain comes from the proposed architecture versus data and model scale.
  • [Minor] Mandarin-Only Evaluation: Page 6, Lines 313–315 ("We represent Mandarin syllables using toneless pinyin, mapping each complete syllable to one pronunciation token qjq_j") and Page 21, Lines 1086–1088 ("Training and evaluation focus on Mandarin singing... its performance beyond Mandarin has not been systematically evaluated") confirm that S3S^3 is only tested on Mandarin, leaving its effectiveness on stress-timed languages with complex consonant clusters and polysyllabic melismas unverified.
Questions for authors
  • What is the super-resolution model?
  • Do you plan to open-source the model weights or data?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Vqcn (Part I)

Thank you for your thoughtful review. We address your concerns below with additional experiments and clarifications.

1. Audio super-resolution and evaluation at 24 kHz (answering Weakness 1 and Question 1)

Thank you for raising this point. To produce the 48 kHz output, we applied FLowHigh, a single-step flow-matching model for audio super-resolution (Yun et al., 2025), using the official pretrained weights without modification or fine-tuning. During development, we heard little difference in the singing before and after super-resolution due to its limited performance. Given that impression, we compared each system’s final end-to-end output without normalizing sampling rate.

We agree that the fairest way to assess our architecture is to compare its native output with the baselines at the same sampling rate. We therefore repeated the objective and listening evaluations for both SVS and SVC using the native 24 kHz output of S³. The tables below show the results alongside the original 48 kHz evaluation.

SVS objective scores

SystemWavLM ↑FFE (%) ↓CER (%) ↓SingMOS ↑
SoulX-Score0.90831.9051.5334.046
TechSinger0.75731.5812.5253.516
TCSinger0.82028.3593.1743.258
S³ (24 kHz, native)0.90125.5361.6134.078
S³ (48 kHz, FLowHigh)0.90025.5621.5294.070

SVS listening-test means (1–5)

SystemTimbre ↑Style ↑Naturalness ↑Singing accuracy ↑
SoulX-Score3.6213.3603.3953.954
TechSinger1.7501.8412.2952.618
TCSinger1.8511.7241.6672.148
S³ (24 kHz, native)4.0634.1054.2024.459
S³ (48 kHz, FLowHigh)4.0574.1264.4714.455

SVC objective scores

SystemWavLM ↑FFE (%) ↓CER (%) ↓SingMOS ↑
SoulX-Melody0.8662.4073.3893.981
Vevo2-Style0.88827.6814.6963.992
SoulX-SVC0.8751.9922.1404.112
Vevo2-FM0.86411.1253.6224.011
S³ (24 kHz, native)0.88720.3981.5104.125
S³ (48 kHz, FLowHigh)0.88720.4331.7074.121

SVC listening-test means (1–5)

SystemTimbre ↑Style ↑Naturalness ↑Singing accuracy ↑
SoulX-Melody2.7212.3842.6402.835
Vevo2-Style2.3842.0351.9301.624
SoulX-SVC3.0002.7093.2794.106
Vevo2-FM2.8602.6553.3143.506
S³ (24 kHz, native)3.7193.6834.0824.220
S³ (48 kHz, FLowHigh)3.7333.6864.1744.212

Across both tasks, the objective scores and listening ratings show little consistent change with super-resolution. Small differences may reflect variation in generation or evaluation. Naturalness shows a modest upward trend at 48 kHz in both tasks. Even at its native 24 kHz, S³ outscored every external SVC baseline on all four listening dimensions. The gains in timbre similarity, style similarity, naturalness, and singing accuracy are already present in the output of our conversion system before super-resolution.

2. Concerns about releasing model weights and data (answering Weakness 2 and Question 2)

We will release our evaluation pipeline, the processed test data, and every test output from S³ and all baselines. Researchers can use these materials to reproduce the objective scores and benchmark new systems on the same examples. We are still discussing the details of releasing the full codebase and model weights. We also plan to share a dataset of scores annotated by the S³ transcriber so the community can train and evaluate models on the same symbolic structure.

3. SOFA + S³ Raw for note–lyric assignment (answering Weakness 2)

Thank you for suggesting this comparison. Using SOFA to assign lyrics to the notes from S³ Raw gives a direct comparison with our joint transcriber. Reviewer ndew raised a related question about using an existing MIDI transcriber with our generator. We therefore evaluated both alternatives.

  1. GAME + SOFA. We used GAME, the strongest external MIDI transcription baseline in our evaluation, to transcribe the notes and SOFA to assign the lyrics to those notes.
  2. S³ Raw + SOFA. We used only the MIDI notes from S³ Raw and obtained the note–lyric assignments with SOFA.

We evaluated both as alternative SVC configurations, using the same S³ generator, dataset, and baselines as in our SVC experiment.

AuthorsOpenReview ↗

Response to Reviewer Vqcn (Part II)

SVC objective scores

SystemWavLM ↑FFE (%) ↓CER (%) ↓SingMOS ↑
SoulX-Melody0.8662.4073.3893.981
Vevo2-Style0.88827.6814.6963.992
SoulX-SVC0.8751.9922.1404.112
Vevo2-FM0.86411.1253.6224.011
S³ (GAME + SOFA)0.88819.8863.1944.098
S³ (S³ Raw + SOFA)0.88720.5543.1864.101
S³ (Joint, 24 kHz)0.88720.3981.5104.125

SVC listening-test means (1–5)

SystemTimbre ↑Style ↑Naturalness ↑Singing accuracy ↑
SoulX-Melody2.7212.3842.6402.835
Vevo2-Style2.3842.0351.9301.624
SoulX-SVC3.0002.7093.2794.106
Vevo2-FM2.8602.6553.3143.506
S³ (GAME + SOFA)3.7253.5593.5243.876
S³ (S³ Raw + SOFA)3.7203.6813.5413.983
S³ (Joint, 24 kHz)3.7193.6834.0824.220

Listeners rated Joint higher for both naturalness and singing accuracy. S³ Raw + SOFA scored slightly higher than GAME + SOFA on both measures. These ratings may be related, since an inaccurately sung passage can also sound less natural. Even with GAME + SOFA, our generator scored higher than every external SVC baseline for timbre similarity, style similarity, and naturalness. This supports the use of the structured symbolic bottleneck for SVC, while the lower CER and higher naturalness and accuracy ratings show the benefit of our joint transcriber.

4. Contributions of the proposed design (answering Weakness 2)

S³ outperformed SoulX-SVC on both style similarity and naturalness, even though the two systems use comparable amounts of Mandarin singing data. Our generator used about 20,000 hours of Mandarin singing. SoulX-Singer, the base model for SoulX-SVC, was trained on about 42,000 hours of singing, including roughly 20,000 hours in Mandarin (Qian et al., 2026). On a five-point scale, listeners rated S³ nearly a full point higher for style similarity (3.69 vs. 2.71) and naturalness (4.17 vs. 3.28).

We pretrained our 746.5M-parameter transcriber on about 2,500 hours of Mandarin songs and fine-tuned it on a separate 80-hour manually annotated set. At submission, it was the largest dedicated singing-score transcriber we knew of. SheetSage-Jukebox also benefits from large-scale pretraining through features extracted from the first 53 layers of the 5B-parameter Jukebox model (Donahue et al., 2022). On HSD57, with both systems given the full mix and no lyrics, S³ achieved 88.88% onset-and-pitch F1 versus 57.52% for SheetSage-Jukebox, and a note error rate of 52.94% versus 89.13% (Table 1).

Our transcriber recovers the melody more accurately than a baseline built on large-scale music pretraining. Listeners also rate the full system higher than SoulX-SVC for style similarity and naturalness, even though SoulX-Singer used as much Mandarin singing data and more singing data overall. A completely fair comparison is hard to achieve, but the above results demonstrate the proposed design is competitive compared to baselines with a similar training scale.

5. Evaluation beyond Mandarin (answering Weakness 3)

We trained and evaluated the submitted model on Mandarin singing only due to limited training resources. The symbolic structure itself can represent lyrics in multiple languages. It records each note’s pitch and timing and links it to a lyric syllable. When a singer holds one syllable across several notes, those notes share the same lyric position.

Our multilingual model is in development. The accompanying demo (English lyrics: Let me fly, I am proud to fly up high...) uses the same symbolic structure to sing passages that mix Mandarin and English, showing the proposed structure is capable of presenting other languages. Demo Link

Conclusion

Thank you again for carefully reviewing our paper and for all the listed questions. We hope the additional experiments and clarifications address your concerns. We will update the experiment and appendix section of the PDF manuscript with the new ablation results. We believe these results give our work a solid empirical foundation and show that the structured symbolic bottleneck enables high-quality transfer of vocal timbre and singing style while preserving the structure of the source song. It would be a great pleasure if you could reconsider the assessment in light of these results. We would be happy to clarify any other aspect of the paper.

AuthorsOpenReview ↗

Follow-up on Previous Comments

Thank you again for your time and thoughtful review. As the discussion period is drawing to a close, we would greatly appreciate the opportunity to further clarify and improve our work before the deadline.

Could you kindly let us know if our previous response, including the sampling rate, adequately addresses your concern? We look forward to your feedback and would be more than happy to address any remaining questions you may have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the AC

Dear Area Chair,

Thank you for handling our submission (#56018). As the discussion period concludes, we summarize our core contributions, reviewer-acknowledged strengths, new rebuttal results, and current review status.

Context and Reviewer-Acknowledged Strengths

S³ is a zero-shot singing style transfer framework built around a structured lyrics–melody bottleneck (timed MIDI notes, note–lyric alignments, and melisma groups) linking joint full-mix transcription with reference-conditioned generation. Replacing frame-level source F0 and acoustic timestamps with an explicit symbolic score prevents source-style leakage and unifies SVS and SVC without singer-specific fine-tuning.

We believe S³ establishes a new paradigm for singing voice modeling: by bridging full-mix transcription and reference-conditioned generation through an interpretable symbolic score bottleneck, it enables true zero-shot singing style transfer from real-world music while unifying SVS and SVC. Backed by decisive gains across multiple benchmarks, we firmly believe this principled framework offers lasting value and an open foundation for the ICLR community.

We thank all three reviewers (9A4s, ndew, Vqcn) for recognizing its novelty and effectiveness, notably:

  1. Score-Bottleneck Paradigm (9A4s, ndew, Vqcn): Replacing frame-level acoustic conditions with a symbolic score decouples musical content from expressive style, realizing an "audio -> symbolic score -> audio" zero-shot SVC pipeline with explicit melisma modeling.
  2. Joint Full-Mix Transcription (ndew, Vqcn): Our lyrics-conditioned segmental decoder recovers note boundaries, pitches, and slur-onset melisma groups directly from full-mix audio and unaligned lyrics (99.01% note–lyric accuracy on HSD57).
  3. Strong Perceptual Quality (9A4s, ndew, Vqcn): S³ achieves large gains in timbre/style similarity, naturalness, and intelligibility across SVS and SVC.

Summary of New Rebuttal Contributions and Experiments

  1. Native 24 kHz SVS & SVC Evaluations (Vqcn): To rule out confounding effects from 48 kHz FLowHigh super-resolution, we repeated all objective metrics and listener evaluations on S³'s native 24 kHz outputs. At the same 24 kHz rate as the baselines, S³ still outperforms every external baseline across all four perceptual dimensions—achieving 4.082 vs. 3.279 in naturalness and 3.683 vs. 2.709 in style similarity over SoulX-SVC in SVC (and 4.202 vs. 3.395 in naturalness and 4.105 vs. 3.360 in style similarity over SoulX-Score in SVS).
  2. Downstream SVC Transcriber Ablations (Vqcn, ndew, 9A4s): Fixing the generator and references, we replaced our joint transcriber with GAME + SOFA and S³ Raw + SOFA. Even with GAME + SOFA, our generator beats all external SVC baselines in timbre (3.725), style (3.559), and naturalness (3.524), validating the symbolic bottleneck itself. Using S³ Joint further halves CER and raises naturalness and singing accuracy, proving the benefit of joint decoding.
  3. Source-Relative FFE vs. Melody Fidelity (9A4s): Reviewer 9A4s noted S³'s higher source-relative FFE (20.398%) and asked if it reflected style transfer or melody errors. Since frame-level F0 naturally diverges when adopting a target singer's vibrato and transitions, we evaluated score-level pitch agreement against ground-truth HSD MIDI notes via strict exact-semitone matching. Under this strict metric, even the original human vocal matches the MIDI score on 73.347% of notes due to natural ornamentation, whereas S³ achieves the highest agreement at 77.624%—surpassing SoulX-SVC (72.660%) and the original vocal. Combined with S³'s top human-rated singing accuracy (4.220/5.0), this confirms that higher source-relative FFE reflects target-singer expressiveness rather than melody errors.
  4. Training Scale, Transparency, Reproducibility & Cross-Lingual Demo (Vqcn, ndew): We showed strong gains over comparable-scale baselines (SoulX-Singer's ~20k-hour Mandarin corpus and SheetSage-Jukebox's 5B-parameter pretraining), documented lyric/input availability across all systems (§4.4 & Appendix C.3), committed to releasing our evaluation code, processed test data, system outputs, and annotated scores, and added a Mandarin–English demo (Vqcn_5.wav) showing cross-lingual extensibility.

Current Review Status

Reviewers 9A4s and ndew have both acknowledged our rebuttal explanations and new experiments, resulting in two positive ratings. While Reviewer Vqcn has not yet had the opportunity to reply during the discussion period, we hope our newly added native 24 kHz benchmarks, downstream transcriber ablations, and clarifications have thoroughly addressed all of their concerns.

We will incorporate all new results and clarifications into the revised manuscript. Thank you again for your time and consideration.

Best regards,
The Authors of Submission 56018

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.