- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In continual unlearning for zero-shot TTS (ZS-TTS), previously forgotten speakers can re-emerge whenever a new speaker is unlearned. This paper aims to address this issue through two approaches:
- Saliency Map: Identifies and selectively updates only the parameters necessary for unlearning based on their importance, while leaving other parameters unchanged to prevent previously forgotten speakers from re-emerging.
- Orthogonal Projection: Modifies the gradient subspace through orthogonal projection to prevent the re-emergence of previously forgotten speakers.
Critical strengths
- Compared to Student-Guided Unlearning (SGU), Teacher-Guided Unlearning (TGU), Update Normalization (UN), and Selective Fine-tuning (SelFT), the proposed method demonstrates superior performance in both knowledge retention and successful forgetting.
- The proposed method requires significantly fewer computational steps and less execution time than Compared to Student-Guided Unlearning (SGU), Teacher-Guided Unlearning (TGU), Update Normalization (UN), and Selective Fine-tuning (SelFT).
- The ablation studies demonstrate that each component of CORTIS contributes meaningfully to its overall effectiveness.
Critical weaknesses
- All experiments are conducted using a single TTS model, Voicebox, leaving the generalizability of the proposed method to other model architectures insufficiently explored.
- The thresholds used to evaluate the method based on S-R and S-f_i are derived from only 200 pairs generated using the LibriSpeech test-clean subset. This relatively small sample size raises concerns about the reliability and generalizability of the thresholds.
- Orthogonal projection-based unlearning has already been widely explored in various research fields, potentially limiting the methodological novelty of this component.
Questions for authors
- Can the authors demonstrate whether the proposed method maintains its effectiveness when extended to other TTS models and datasets?
- Given that the thresholds were derived from a relatively small sample of 200 pairs from LibriSpeech test-clean, could the authors investigate whether similar results can be obtained by deriving the thresholds from a larger and more diverse dataset?
4 comments
W2, Q2. Larger Dataset for Threshold Computation.
We appreciate the reviewer for pointing out additional evaluation to strengthen our paper. As our evaluation set is LibriSpeech for remain set and LibriHeavy for forget set, we stick to calibrating on in-domain LibriSpeech but extend calibration to 2,000 pairs. Note that LibriSpeech and LibriHeavy are both derived from LibriVox public-domain audiobook recordings. Through this, we derive the forget bound as 0.34 and retain bound as 0.40.
| Verifier (dataset) | diff.-spk mean | forget bound | same-spk mean | retain bound |
|---|---|---|---|---|
| WavLM (200 pair) | ||||
| WavLM (2,000 pair) |
With larger samples, the bounds become looser. Therefore, every forget and retain success remain the same. The increase in dataset pairs for calibration does not make any changes on Table 1 results, keeping the results consistent with our claims. We will update Appx. F to include the 2,000-pair calibration but keep the stricter bounds throughout.
Additionally, please refer to results with second verifier (Appx. N.2) for diversity of evaluation.
Thank you to the authors for conducting the additional experiments. Most of my concerns have been resolved. Good luck!
Dear Reviewer zVdr
Thank you for your confirmation in support of the paper.
To the best of our knowledge, this paper is the first to study Speaker Identity Unlearning under specifically continual scenarios, to closely resemble real-world unlearning requests (multiple and continuous) and privacy rights (necessity of non-retention rules). We tested whether existing ideas in Speaker Identity Unlearning, and continual unlearning from other domains can be directly applied to Continual Speaker Identity Unlearning. Then, proposed a simple, effective method to solve the challenge : CORTIS.
We sincerely hope this work provides a strong foundation for future works in Speaker Identity Unlearning, to reach the ultimate goal of providing protection to everyone's voice. Please do not hesitate to leave extra questions, comments on this thread.
We sincerely appreciate Reviewer zVdr's time in providing valuable feedback for our paper. We are grateful that the reviewer sees CORTIS as a successful mechanism in terms of performance and also computation, with each component contributing.
W1, Q1. Extended Backbone, Evaluation
We agree the paper could be strengthened with extended experiment and evaluation. Our choice in experimental settings follows previous work that defined Speaker Identity Unlearning [1]. However, for thorough evaluation a method should be validated for generalizability. We would like to mention that CORTIS operates in parameter space rather than on architecture-specific representations. It requires (i) per-parameter gradients of a forget and remain objective (Eq.3) and (ii) gradient snapshots for subspace construction. Neither depends on the generative formulation.
We would also like to present updates on a new backbone model, NaturalSpeech 2 (NS2) [2] which was trained on a different dataset, LibriTTS [3]. Unlike VoiceBox (flow matching, mel features, masked-infilling training) [4], NS2 is a latent diffusion model with neural-codec (RVQ) latents and speech-prompt conditioning with duration/pitch predictors. We implement the baseline TGU[1], whose denoising objective becomes random-speaker teacher targets; CORTIS applies without modification, as it operates at the parameter level.
NS2 is pretrained on LibriTTS with learning rate 1e-4 per the Amphion configuration [5]. Unless specified otherwise, all settings follow NS2's defaults. For unlearning, random speakers are selected (speaker ids: 2836, 696, 7505, 6209), each unlearned for 1K steps with 100 warmup steps at learning rate 1e-4. Other configurations match our implementations on the VoiceBox backbone. Below, we present continual unlearning results through four sequential requests. Due to the discussion window, we could not unlearn to full convergence, which may affect absolute WER levels.
The same failure pattern emerges on the new backbone. TGU's prior-speaker similarities climb as requests accumulate: S- f2 reverts above the threshold (0.180→0.344>τF=0.32), and S- f1 trends toward it (0.051→0.284). CORTIS holds every prior speaker flat (S- fi≤0.110 throughout) and recovers retention by Request 4 (S-R 0.552 vs. 0.556). TGU also degrades forget-output intelligibility (W-F up to 46.5 vs. 8.5 pretrained), which CORTIS keeps lower. By Request 4, CORTIS outperforms TGU, the existing SOTA baseline, on every metric. This shows that catastrophic re-learning is not specific to the flow-matching backbone, and that CORTIS transfers to a structurally different generative family without modification.
This experiment, on a different model (NaturalSpeech2) and a different dataset (LibriTTS), supports the generality of CORTIS. We thank the reviewer for strengthening our work and will include the full results in the revised paper. For now, we include above results in Appx. N.3 in the revised paper.
[1] Kim et al., "Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech", ICML 2025
[2] Shen et al., "Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers", ICLR 2024
[3] Zen et al., "LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech", Interspeech 2019
[4] Le et al., "Voicebox: Text-guided multilingual universal speech generation at scale", NeurIPS 2023.
[5] Zhang et al, "Amphion: An open-source audio, music and speech generation toolkit", IEEE SLT 2024