acceptodds
Under review as a conference paper at ICLR 2027

ZeroOpera: Low-Resource Peking Opera Singing Voice Synthesis with Zero-Shot Timbre Transfer

Abstract

Recent singing voice synthesis (SVS) systems have achieved strong performance, but Peking Opera synthesis remains difficult because it requires domain-specific pronunciation rules and has only limited annotated data. We propose ZeroOpera, a low-resource Peking Opera SVS framework that supports zero-shot timbre transfer from an acoustic prompt. Instead of fine-tuning the foundation model, ZeroOpera keeps it frozen and uses its prompt-conditioned mel-spectrogram as an acoustic prior. A trainable decoder reconstructs regions that require opera-specific pronunciations and enhances the full spectrogram. We further introduce Technique-driven Phonological Modulation (TPM). TPM converts lexicon-derived phonological category labels into continuous feature-level controls. By separating Standard Mandarin pinyin from opera-specific pronunciation categories, TPM can handle pinyin–category combinations that are not observed during adaptation. This design does not require an expanded pinyin vocabulary or manually aligned opera-specific phoneme annotations. For evaluation, we develop an automatic G2P pipeline and a Peking Opera ASR model that recognizes pinyin syllables. On target passages from a Peking Opera singer excluded from opera-domain adaptation, ZeroOpera reduces Phoneme Error Rate by 19.4% relative to the frozen foundation model. It also improves timbre-similarity MOS by 0.43 when using unseen utterances from singers included in adaptation, and by 0.30 when using cross-domain prompts from 50 pop singers excluded from opera-domain adaptation. These results show that ZeroOpera improves pronunciation and operatic expressiveness while preserving the timbre specified by the prompt under low-resource domain adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Singing Voice Synthesis, Peking Opera Synthesis, Low-Resource

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zLFnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts a frozen, pop-oriented zero-shot SVS model (SoulX-Singer) to Peking Opera using 7.58 h of adaptation data. The frozen model produces a prompt-conditioned mel-spectrogram. Frames belonging to characters that a lexicon tags as Jian, Tuan or Shang-kou-zi are masked out.

A separately trained 143M-parameter flow-matching decoder then regenerates the full mel from the masked prior. The decoder is conditioned on Standard-Mandarin pinyin. Two training tricks are added:

  • Stochastic routing. With probability 0.5 the target is the opera ground truth with TPM on; otherwise the target is Mel_pop itself with TPM off.
  • Region-aware weighting. The flow-matching loss weights masked frames 5x.

The authors also build a Pinyin-syllable Peking Opera ASR model (fine-tuned Qwen3-ASR-0.6B) and use it to compute PER.

Critical strengths
  1. Well-motivated, under-served problem.
  2. Expert listeners. 15 raters with professional Peking Opera experience, blinded and randomized presentation, and the interface is shown.
  3. Practical, cheap adaptation. The foundation model is untouched and priors are cached, so adaptation costs about 6 h on 8 GPUs
Critical weaknesses
  1. No demo page. It becomes very hard to judge the model based on numbers alone.
  2. The ablations do not isolate where the gains come from. Every variant in Table 2 keeps a 143M opera-trained decoder that regenerates the full spectrogram. Gains in AQ, OF and even TS could therefore come mainly from a generic opera-domain post-net or enhancer trained for 300k steps, not from phonology-aware masking or TPM. Two missing controls:
  • (a) Same decoder, no category information at all: no TPM and either no mask or a random mask with the same masked-frame ratio.
  • (b) An inference-time "all labels = Standard" condition evaluated quantitatively. Appendix E.2 shows only a single spectrogram.
Questions for authors
  1. Please report WavLM SIM (with CIs) for SoulX-Singer and ZeroOpera under both prompt conditions at matched target F0, overall and restricted to masked frames.
3 comments
AuthorsOpenReview ↗

We release an anonymous repo!

To enhance the credibility of our work, we have made our inference code and model weights publicly available via an anonymous link: https://anonymous.4open.science/r/ZeroOpera-7E5E/. We hope this gesture fosters a more favorable impression of our work.

I thank the authors for their uploading the demo page and providing additional controls, and I have improved my ratings based on the revisions. 1 question remains: In my listening to the provided demos, ZeroOpera sounds broadly comparable to SoulX-Singer in overall perceptual quality, and the audible advantage is not clear to me. Appreicate your explanation.

AuthorsOpenReview ↗

Clarifying the audible differences from SoulX-Singer

Thank you for updating your assessment and for taking the time to listen to our demos.

The main improvement we would like to clarify is opera-specific pronunciation. A syllable can sound clear and remain understandable while using a Mandarin pronunciation instead of the required Peking Opera pronunciation. SoulX-Singer already follows the melody well; ZeroOpera builds on this strength to improve these pronunciation details while preserving prompt timbre.

For example, Appendix A.1 illustrates “箭” pronounced as zian rather than jian, and “白” as bo rather than bai. To hear the relevant differences in the demos, the highlighted syllables can be compared across the ground-truth recording, SoulX-Singer, and ZeroOpera. Appendix E.3, Figure 7 also shows the system comparison, including melodic contours and ASR-recognized pronunciations.

We tested pronunciation correctness directly with three Peking Opera practitioners in a blinded evaluation under opera prompts. The results below were already reported in Section 5 (“Independent rule correctness”) of our response to Reviewer 9o1p. Of 197 syllable tokens, 11 uncertain cases were excluded, leaving 186.

CategoryFrozen SoulX-SingerZeroOpera
Jian12%85.2% (115/135)
Tuan15%88.0% (103/117)
Shang-kou8%82.0% (251/306)

Each eligible syllable received three expert judgments. Percentages show the proportion of judgments rated correct; parentheses give correct/total judgments. SoulX-Singer percentages are rounded as previously reported.

We also measured overall listening quality separately. In Table 1, 15 listeners with professional Peking Opera experience gave higher average scores under unseen pop-singer prompts: Audio Quality increased from 3.23 to 3.57, and Opera Flavor from 3.22 to 3.61. This supports an average improvement, while allowing that individual demo excerpts may sound comparable.

Reviewer_9o1pOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ZeroOpera tackles low-resource Peking Opera singing voice synthesis via zero-shot timbre transfer. It freezes a pretrained singing model and uses its prompt-conditioned mel-spectrogram as an acoustic prior. A decoder is trained to reconstruct opera-specific pronunciation regions and enhance overall output. Key contributions: phonology-aware acoustic inpainting; Technique-driven Phonological Modulation (TPM), which separates standard Mandarin pinyin from opera pronunciation categories; and an evaluation framework with an annotated opera corpus, pronunciation lexicon, and opera-specific ASR. Validation uses 7.58 hours of adaptation data, 44 target recordings from one held-out opera singer, and both seen-singer opera prompts and unseen prompts from 50 pop singers. Comparisons, component ablations, ASR-based pronunciation metrics, pitch metrics, and listening tests with 15 experienced evaluators show improvements over frozen SoulX-Singer. These include a 19.4% relative PER reduction under opera prompts and 0.30 higher timbre-similarity MOS under unseen pop prompts. An unseen pinyin–category analysis gives preliminary evidence of compositional generalization. Evaluation remains limited to one opera tradition and one held-out target singer.

Critical strengths

1, Technical novelty

The overall design is reasonably motivated. Retaining the base model and performing targeted repair on the output acoustic representation is a sensible adaptation path.

2, Experimental validation

The author is honest in his explanation of the experimental boundaries. The paper clearly distinguishes between "new prompt recordings of seen singers" and "zero-shot prompts of unseen singers", and acknowledges limitations such as single genre, ASR error, and mismatch in adaptation baseline budget. The involvement of professional Peking Opera reviewers in listening tests, as well as the isolation of Singer 3 from adaptation and ASR training, are all valuable experimental designs.

Critical weaknesses

1, Technical limitations

The evidence does not fully support the proposed combinatorial generalization mechanism. TPM applies category conditioned FiLM, but the discrete concatenation baseline also represents pinyin and category separately. Their comparison therefore does not isolate the benefit of factorization. Moreover, Appendix B.4 does not guarantee that both constituent factors of every OOV combination were individually observed during adaptation. Table 3 may consequently mix compositional generalization with unseen-pinyin generalization. This weakens the paper’s principal claim of a transferable methodological insight beyond domain-specific conditioning.

2, Experimental gaps

a, Adaptation comparisons do not isolate the architectural contribution. ZeroOpera is trained for 300,000 steps, versus 30,000 for full fine-tuning and 7,500 for LoRA, with substantially different trainable capacities. Equivalent access to phonological-category information is also not established. Although these differences are acknowledged, the results cannot adequately distinguish architectural benefits from training and supervision differences. Furthermore, the adaptation comparisons and component ablations in Table 2 only use seen singer opera prompts, leaving the components’ contribution to the headline unseen singer capability insufficiently tested.

b, Opera-specific pronunciation accuracy lacks independent validation. The pronunciation and OOV conclusions rely heavily on a custom ASR whose PER on real recordings is already 0.200. The listening protocol explicitly assesses intelligibility rather than pronunciation-rule correctness for PC, while OF combines pronunciation with several stylistic attributes. Figure 7 also derives its correctness labels from the same ASR. Targeted expert judgments are needed to substantiate the central pronunciation claim.

3, Clarity

Inconsistent evaluation reporting undermines confidence in the central results. For the same held-out recordings, Table 3 reports GT IV/OOV-category PERs of 0.318/0.322, whereas Table 10 reports Jian/Tuan/Shang-kou PERs of 0.265/0.218/0.282. Under the described pooled-error calculation, these partitions of the same special-category phonemes cannot yield compatible totals. Different aggregation procedures could explain the discrepancy, but none is specified. This requires clarification and reproducible error counts before the key generalization results can be trusted.

4, Missing related works

RDSinger already uses a pretrained model's mel-spectrogram as a reference. It selectively corrupts problematic regions and increases their loss weights. This substantially overlaps with ZeroOpera's acoustic-prior, masking, and region-weighting rationale. TPM also directly follows FiLM's feature-wise affine conditioning, which is not cited. These omissions leave the methodological increment insufficiently established. The paper must show what its phonological conditioning and routing add beyond these designs.

Questions for authors

1, Relative to RDSinger’s generated-spectrogram conditioning, selective corruption, and regional loss weighting, and FiLM’s affine conditioning, what is the specific methodological advance of ZeroOpera? Which existing experiment most directly demonstrates the value of that advance?

2, How many OOV examples contain pinyin and category factors that were each individually observed during adaptation, and how many are unseen only because of tone distinctions? Since discrete concatenation also separates the two factors, what evidence attributes TPM’s advantage specifically to compositional generalization, and were the conditioning implementations matched in capacity, initialization, and routing?

3, Is there independent evidence that the custom ASR reliably distinguishes correct opera pronunciation from intelligible Mandarin substitutions? Do any existing expert annotations directly assess pronunciation-rule correctness, given that PC excludes this criterion and OF combines multiple attributes?

4, How can the GT category PERs in Table 3 (IV/OOV: 0.318/0.322) be reconciled with those in Table 10 (Jian/Tuan/Shang-kou: 0.265/0.218/0.282)? Do the tables use different samples, denominators, or aggregation procedures? Please provide the underlying error counts and aggregation definitions.

1 comment
AuthorsOpenReview ↗

We release an anonymous repo!

To enhance the credibility of our work, we have made our inference code and model weights publicly available via an anonymous link: https://anonymous.4open.science/r/ZeroOpera-7E5E/. We hope this gesture fosters a more favorable impression of our work.

Reviewer_tdWROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Paper describes an approach to singing voice synthesis (SVS) by modifying an off-the-shelf SVC system with a post-editing step that introduces genre-specific local modifications for which only little training data is avaliable. The genre in question is Peking Opera (PO), the base system is Chinese pop-singing synthesis. The post-editing model learns to effect the archaic and idiosyncratic pronunciations typical for PO. The post-editing is done by identifying and masking the speech regions with non-standard syllables and passing them to an inpainting decoder. All TTS decoders are based on flow-matching TTS.

The approach is seemingly novel and could be modified for other speech-genre variants for which training data is scarce.

Critical strengths
  • Approach is novel
  • Validation largely convincing, compared favorably with alternative methods
  • Interesting application case (PO signing generation)
Critical weaknesses
  • The paper mentions, and the appendix details, the baselines based on SoulX-Singer and standard adaptation methods (FFT, LoRA), but the main result comparison tables (1 and 3) only give results for the baseline SoulX-Singer.
  • The Prior work section only mentions other SVS approaches, but doesn't mention the more general problem of modifying a standard language TTS for a region language variant, which is essentially the same problem (modulo the singing aspect). There is work on data-constrained language variant TTS modeling that should be compared as well.
  • Since ICLR is not a speech conference the paper should avoid using abbreviations like MOS or algorithms like flow-matching without giving at least a brief definition or summary of the underlying modeling. The fundamental phenomena that make PO different from standard Mandarin (the pronunciation differences) should be characterized briefly in section 3 so it is made clear what aspect of the speech model needs to be adapted (instead of relying on the reader searching through appendix).
  • Figure 2 (system diagram) should include how the inputs (pitch, type, phoneme, tech type) are derived from the raw lyrics. Even the text is not very explicit about that and you have gather that basic information by collecting bits of information throughout the paper.
Questions for authors

Can you address the weaknesses listed above?

1 comment
AuthorsOpenReview ↗

We release an anonymous repo!

To enhance the credibility of our work, we have made our inference code and model weights publicly available via an anonymous link: https://anonymous.4open.science/r/ZeroOpera-7E5E/. We hope this gesture fosters a more favorable impression of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.