acceptodds
Under review as a conference paper at ICLR 2027

Continual Speaker Identity Unlearning with Minimal Interference

22 views

Abstract

Machine unlearning removes designated concepts or memorized data from pre-trained models. Recent work has made strong progress on speaker identity unlearning in zero-shot text-to-speech (ZS-TTS). Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present CORTIS, the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. Across ZS-TTS models, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unlearning, speech, voice privacy, privacy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zVdrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In continual unlearning for zero-shot TTS (ZS-TTS), previously forgotten speakers can re-emerge whenever a new speaker is unlearned. This paper aims to address this issue through two approaches:

  1. Saliency Map: Identifies and selectively updates only the parameters necessary for unlearning based on their importance, while leaving other parameters unchanged to prevent previously forgotten speakers from re-emerging.
  2. Orthogonal Projection: Modifies the gradient subspace through orthogonal projection to prevent the re-emergence of previously forgotten speakers.
Critical strengths
  • Compared to Student-Guided Unlearning (SGU), Teacher-Guided Unlearning (TGU), Update Normalization (UN), and Selective Fine-tuning (SelFT), the proposed method demonstrates superior performance in both knowledge retention and successful forgetting.
  • The proposed method requires significantly fewer computational steps and less execution time than Compared to Student-Guided Unlearning (SGU), Teacher-Guided Unlearning (TGU), Update Normalization (UN), and Selective Fine-tuning (SelFT).
  • The ablation studies demonstrate that each component of CORTIS contributes meaningfully to its overall effectiveness.
Critical weaknesses
  • All experiments are conducted using a single TTS model, Voicebox, leaving the generalizability of the proposed method to other model architectures insufficiently explored.
  • The thresholds used to evaluate the method based on S-R and S-f_i are derived from only 200 pairs generated using the LibriSpeech test-clean subset. This relatively small sample size raises concerns about the reliability and generalizability of the thresholds.
  • Orthogonal projection-based unlearning has already been widely explored in various research fields, potentially limiting the methodological novelty of this component.
Questions for authors
  • Can the authors demonstrate whether the proposed method maintains its effectiveness when extended to other TTS models and datasets?
  • Given that the thresholds were derived from a relatively small sample of 200 pairs from LibriSpeech test-clean, could the authors investigate whether similar results can be obtained by deriving the thresholds from a larger and more diverse dataset?
4 comments

We sincerely appreciate Reviewer zVdr's time in providing valuable feedback for our paper. We are grateful that the reviewer sees CORTIS as a successful mechanism in terms of performance and also computation, with each component contributing.


W1, Q1. Extended Backbone, Evaluation

We agree the paper could be strengthened with extended experiment and evaluation. Our choice in experimental settings follows previous work that defined Speaker Identity Unlearning [1]. However, for thorough evaluation a method should be validated for generalizability. We would like to mention that CORTIS operates in parameter space rather than on architecture-specific representations. It requires (i) per-parameter gradients of a forget and remain objective (Eq.3) and (ii) gradient snapshots for subspace construction. Neither depends on the generative formulation.

We would also like to present updates on a new backbone model, NaturalSpeech 2 (NS2) [2] which was trained on a different dataset, LibriTTS [3]. Unlike VoiceBox (flow matching, mel features, masked-infilling training) [4], NS2 is a latent diffusion model with neural-codec (RVQ) latents and speech-prompt conditioning with duration/pitch predictors. We implement the baseline TGU[1], whose denoising objective becomes random-speaker teacher targets; CORTIS applies without modification, as it operates at the parameter level.

NS2 is pretrained on LibriTTS with learning rate 1e-4 per the Amphion configuration [5]. Unless specified otherwise, all settings follow NS2's defaults. For unlearning, random speakers are selected (speaker ids: 2836, 696, 7505, 6209), each unlearned for 1K steps with 100 warmup steps at learning rate 1e-4. Other configurations match our implementations on the VoiceBox backbone. Below, we present continual unlearning results through four sequential requests. Due to the discussion window, we could not unlearn to full convergence, which may affect absolute WER levels.

MethodReqW-R ↓\downarrowS-R ↑\uparrowW-FS- f1f_1 ↓\downarrowS- f2f_2 ↓\downarrowS- f3f_3 ↓\downarrowS- f4f_4 ↓\downarrow
NS2 (pretrained)–9.09.00.5560.5568.58.50.6450.6450.4290.4290.7070.7070.3550.355
TGU111.311.30.4860.48646.546.50.0510.051–––
TGU28.38.30.4930.49321.221.20.2160.2160.1800.180––
TGU37.47.40.5170.51719.119.10.1730.1730.2700.2700.0260.026–
TGU47.77.70.5320.53226.626.60.2840.2840.3440.3440.0090.0090.0770.077
CORTIS111.011.00.4950.49522.822.80.0580.058–––
CORTIS29.99.90.5100.51017.917.90.0830.0830.1020.102––
CORTIS38.58.50.5100.51020.020.00.1050.1050.1140.114−0.062-0.062–
CORTIS47.67.60.5520.55222.922.90.1100.1100.1200.120−0.048-0.048−0.022-0.022

The same failure pattern emerges on the new backbone. TGU's prior-speaker similarities climb as requests accumulate: S- f2f_2 reverts above the threshold (0.180→0.344>τF=0.320.180 \rightarrow 0.344 > \tau_F = 0.32), and S- f1f_1 trends toward it (0.051→0.2840.051 \rightarrow 0.284). CORTIS holds every prior speaker flat (S- fi≤0.110f_i \leq 0.110 throughout) and recovers retention by Request 4 (S-R 0.5520.552 vs. 0.5560.556). TGU also degrades forget-output intelligibility (W-F up to 46.546.5 vs. 8.58.5 pretrained), which CORTIS keeps lower. By Request 4, CORTIS outperforms TGU, the existing SOTA baseline, on every metric. This shows that catastrophic re-learning is not specific to the flow-matching backbone, and that CORTIS transfers to a structurally different generative family without modification.

This experiment, on a different model (NaturalSpeech2) and a different dataset (LibriTTS), supports the generality of CORTIS. We thank the reviewer for strengthening our work and will include the full results in the revised paper. For now, we include above results in Appx. N.3 in the revised paper.

[1] Kim et al., "Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech", ICML 2025

[2] Shen et al., "Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers", ICLR 2024

[3] Zen et al., "LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech", Interspeech 2019

[4] Le et al., "Voicebox: Text-guided multilingual universal speech generation at scale", NeurIPS 2023.

[5] Zhang et al, "Amphion: An open-source audio, music and speech generation toolkit", IEEE SLT 2024

W2, Q2. Larger Dataset for Threshold Computation.

We appreciate the reviewer for pointing out additional evaluation to strengthen our paper. As our evaluation set is LibriSpeech for remain set and LibriHeavy for forget set, we stick to calibrating on in-domain LibriSpeech but extend calibration to 2,000 pairs. Note that LibriSpeech and LibriHeavy are both derived from LibriVox public-domain audiobook recordings. Through this, we derive the forget bound as 0.34 and retain bound as 0.40.

Verifier (dataset)diff.-spk meanforget boundsame-spk meanretain bound
WavLM (200 pair)0.100.100.320.320.710.710.460.46
WavLM (2,000 pair)0.090.090.340.340.720.720.400.40

With larger samples, the bounds become looser. Therefore, every forget and retain success remain the same. The increase in dataset pairs for calibration does not make any changes on Table 1 results, keeping the results consistent with our claims. We will update Appx. F to include the 2,000-pair calibration but keep the stricter bounds throughout.

Additionally, please refer to results with second verifier (Appx. N.2) for diversity of evaluation.

Thank you to the authors for conducting the additional experiments. Most of my concerns have been resolved. Good luck!

Dear Reviewer zVdr

Thank you for your confirmation in support of the paper.

To the best of our knowledge, this paper is the first to study Speaker Identity Unlearning under specifically continual scenarios, to closely resemble real-world unlearning requests (multiple and continuous) and privacy rights (necessity of non-retention rules). We tested whether existing ideas in Speaker Identity Unlearning, and continual unlearning from other domains can be directly applied to Continual Speaker Identity Unlearning. Then, proposed a simple, effective method to solve the challenge : CORTIS.

We sincerely hope this work provides a strong foundation for future works in Speaker Identity Unlearning, to reach the ultimate goal of providing protection to everyone's voice. Please do not hesitate to leave extra questions, comments on this thread.

Reviewer_CeKLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CORTIS, a framework for continual speaker identity unlearning in zero-shot text-to-speech. It combines Fisher-information-based parameter masking with orthogonal projection to preserve prior unlearning updates, without requiring access to previously unlearned speaker data. Experiments show that CORTIS effectively removes new speaker identities while maintaining the forgetting of previously targeted speaker.

Critical strengths
  1. The paper formalizes a well-motivated problem: continual speaker identity unlearning for zero-shot TTS under realistic constraints that requests arrive sequentially, and forget data is deleted after each one. The “data retention paradox” clearly explains why simply re-running one-shot unlearning on all accumulated forget data conflicts with the right to be forgotten.
  2. Clear identification of a TTS-specific failure mode. “Catastrophic re-learning” is a useful observation. Because speaker identity is a zero-shot ability rather than memorized content, the retain loss alone revives previously suppressed speakers. This is shown consistently across Table 1, the step-matched comparison (Table 8), and the acoustically similar speakers (Table 13).
  3. Consistent gains on the main metric. CORTIS keeps earlier speakers suppressed (S-f ≤ 0.172 after request 3). TGU, UN and SelFT let them return to 0.43–0.64, a gap far larger than the reported standard deviations.
Critical weaknesses
  1. Limited methodological novelty. Both mechanisms adapt existing techniques. The Fisher-ratio saliency in Eq. 3 is closely related to Selective Synaptic Dampening (Foster et al., AAAI 2024, not cited) and SalUn. The projection follows OGD/GPM directly. The main new element is adding prior forget Fishers to the saliency denominator. The paper also doesn’t explain how CORTIS differs from projection- or orthogonality-based continual LLM unlearning methods it cites (e.g., Gao et al., 2025), and none of them are included as baselines.

  2. CORTIS is evaluated on at most 5 requests, and all baselines, ablations, order-robustness and similar-speaker experiments stop at 3. Request 1 is trained with 10× the steps and learning rate of later requests, which may bias that weighting. Results at 10–20 requests, with baselines, are needed to support the "continual" claim.

  3. Main results rest on a single order and a single training run. Table 1 uses one request order and one training run per method; the three seeds vary only generation. Continual-learning results are known to depend on order. The S-R differences between CORTIS and TGU/UN (0.557 vs 0.582/0.580) are comparable to the variation CORTIS itself shows across orders in Table 11 (0.520–0.557). Main results should be reported as mean ± std over several orders, and ideally training seeds, for all methods.

  4. Limited order-robustness study (App. N.1). All three orders start with speaker 1166, so request 1, which uses a different and much longer schedule, is never varied. Order C changes the speaker set (9437 replaces 3912), so the only pure reordering is swapping 7199 and 3912. Only 3 requests are used.

  5. Mask-disjointness claim (App. H) lacks a reference point. Two independent random 30% masks have an expected Jaccard of 0.09/0.51 ≈ 0.18. The observed values (0.049–0.197, mean about 0.12) are only moderately below this. The f1–f5 pair (0.197), where the prior-forget term should have the strongest effect, is above it. “Largely disjoint” is therefore not well supported.

  6. Unverifiable headline number. The “75% average reduction” in the introduction cannot be checked, because original-model similarities for f4 and f5 are not reported. Computing it from f1–f3 gives about 80%.

Questions for authors
  1. Is the orthogonal projection applied to the full parameter vector or re-masked afterwards?
  2. Are gradient snapshots collected within the mask MjM_j or over all parameters?
  3. Can you report Table 1 as mean ± std over several random request orders, including orders that do not start with speaker 1166, for all methods?
  4. Please report original-model similarities for f4 and f5 and clarify how the 75% reduction is computed.
  5. Please report the Jaccard overlap for random masks and for the “w/o prior-forget” variant in App. H.
  6. Were the forget speakers part of the pretraining data? Does unlearning behave differently for speakers seen in training versus unseen speakers cloned zero-shot?
8 comments

We express our gratitude to Reviewer CeKL's in providing valuable and insightful feedback for our paper. Thank you for helping us further refine and strengthen the paper.


W1. Method Novelty and Related Works, Baselines

We thank the reviewer for the suggestions. In the below, we restate our contribution. Additionally, we would like to explain why other domain's unlearning baselines are not the best comparisons for Speaker Identity Unlearning. Then, we clarify that we have already partially adopted reviewer's suggested baselines in modified form for speech domain, and also provide additional baselines and how CORTIS differs.

Novelty of CORTIS. We agree that Fisher-based saliency and gradient-subspace projection are well established tools. Our contribution is in the problem formulation and the specific combination that solves it. (i) Catastrophic re-learning as failure mode that no prior speech work adresses, which Reviewer PeDt also noted as a strength. (ii) The prior-forget term in Eq.3 to disentangle speakers in weight-space as the reviewer has identified. This term specifically is load-bearing (Table 3). Removing this simple, specific term returns all prior speakers. (iii) a fixed-rank, energy-weighted merged basis (Sec.4.2) that keeps projection cost and memory constant in the request count and stores no per-speaker artifact. GPM and OGD grow their basis per task and retain per-task bases; neither property is admissible under (C2) or practical for an unbounded request stream.

Limitations of other-domain Baselines. We would like to clarify that unlearning is specifically different in Speaker Identity Unlearning domain. In Zero-Shot Text-to-Speech, models take three inputs : an audio prompt, a transcript, and frame-aligned phonemes. Each sample entangles speaker identity with content, prosody, and accent, so LLM/image unlearning methods do not transfer without substantial modification. For the continual unlearning methods cited in the paper [1-3], we agree with the reviewer on stating differences explicitly. One of the suggested baselines is already in the paper - SelFT in Table 1 is a modified speech adaptation of SalUn[4], using gradient-magnitude saliency on forget data. We now state this explicitly in the main paper.

We implemented two additional baselines : SSD [5], and ALKN [3]. We use each method’s default configuration and replace only the token-level loss with the backbone’s flow-matching loss. Further adaptation would amount to designing a new method, which is outside the scope. We report the first unlearn request only (f1f_1), and commit to include full baselines for camera-ready executed under continual scenario.

W-R ↓\downarrowS-R ↑\uparrowW-F ↓\downarrowS- f1f_1 ↓\downarrow
Original2.12.10.6490.6492.62.60.7210.721
SSD ( α=10\alpha=10 ) [5]3.03.00.5370.5372.92.90.4840.484
SSD ( α=5\alpha=5 ) [5]11.411.40.3590.35910.810.80.2070.207
ALKN [3]49.649.60.2850.28587.587.50.1130.113
CORTIS (Ours)2.92.90.6020.6022.62.60.1620.162

Both fail even on a single request.

  • SSD [5] at default α=10\alpha = 10, 0.6% of parameters are dampened and the forget speaker remains clonable (S-f 0.48). At α=5\alpha =5 , model seems to forget (S-f 0.21) but S-R falls to 0.36 and WER rises about 5x. We have included this baseline in related works as well.
  • ALKN [3] finetuning on one speaker does not yield a speaker-specific task vector, because the update is entangled with the spoken content. Negating it destroys intelligibility (W-R 49.6, W-F 87.5).

We have not implemented O3 [1] and FIT [2]. O3 keeps the base model frozen and places unlearning in a LoRA adapter and an inference-time OOD gate. This OOD gate and retaining the original model weights violate our RTBF formulation (Sec.~3.1; our response to Reviewer PeDt, W1/Q1). FIT's redundancy filter and loss router are specific to text streams and likelihood losses, and its remaining component reduces to SelFT (SalUn).

In conclusion, CORTIS is a weight-editing method for continual unlearning that requires neither the original weights nor prior forget speakers’ data at later requests (unlike O3, which retains the original model). It redirects the forget speaker toward a teacher-defined target instead of negating a fine-tuning drift (unlike ALKN), and localizes each update per parameter, with prior forget Fishers in denominator so that successive speakers land in disjoint parameters (unlike SSD, which uses forget-to-retain ratio to dampen weights once, and FIT, which localizes at the layer level with a fixed family of likelihood losses). Every update is then projected against a fixed-rank basis of prior requests. The individual mechanisms are known, but their combination is what lets CORTIS forget a new speaker without re-learning previous ones.

[1] Gao et al., "On large language model continual unlearning", ICLR 2025

[2] Xu et al., "Fit: Defying catastrophic forgetting in continual llm unlearning", arXiv 2026

[3] Wuerkaixi et al., "Adaptive localization of knowledge negation for continual llm unlearning", ICML 2025

[4] Fan et al., "SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation" ICLR 2024

[5] Foster et al., "Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening", AAAI 2024

W2-1. Evaluation - Step size at request 1.

We clarify the step size at Request 1 below, by explaining why schedule for Request 1 differs, clarifying that step schedule at Request 1 is shared among baselines, and evidences this does not bias unlearning.

The longer schedule at Request 1 does not drive its durability, and the schedule is because CORTIS is not being fully applied. At i=1i=1, no prior subspace exists (no prior forget speakers), so projection is inactive and the Request 1 is ordinary one-shot unlearning (Appx. C). CORTIS’s continual mechanism takes full effect from Request 2 onward. The Request 1 schedule is therefore, a property of the base objective, not of CORTIS. Its step count is therefore orthogonal to CORTIS’s contribution.

The 10K-step, 5e-5 schedule is from TGU's one-shot schedule from [6], and every method in Table 1 uses the same step schedule at Request 1 (Appendix C, E). The baselines keep 10K steps at every request, while CORTIS drops to 1K from Request 2 onward. If anything, the budget favors the baselines.

We claim this does not explain durability with following evidences:

(i) TGU, and all other baselines, trained with the identical Request-1 schedule, still reverts f1f_1 (0.164$\to$0.612 at Request 2).

(ii) f2f_2 and f3f_3 are unlearned with only 1K steps and remain forgotten under CORTIS fully applied (0.148, 0.124 at Request 3), so durability does not require the long schedule.

(iii) This does not bias weighting. Fisher (Eq. 3) FfiF_{f_i} is computed at θi−1\theta_{i-1} before the unlearning update, so its magnitude does not depend on the request’s step count.

(iv) Merged basis (Sec. 4.2) snapshots are collected every 150 steps at Request 1, and every 15 steps afterward (Request 2+), so each request contributes the same number of snapshots (~66) to the energy-weighted merge.

[6] Kim et al. "Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech", ICML 2025.


W2-2. Evaluation - Longer Unlearning Sequences

We agree that longer sequences strengthen the continual claim. The current evidence extends to 5 requests. The per-request cost of CORTIS is constant, since the merged basis has fixed rank and the max-Fisher is a single tensor (Sec. 4.2). Each additional request costs ~3.5 h of training plus up to ~10 h of VoiceBox inference and evaluation, roughly half a GPU-day. Within the discussion period, we are prioritizing the experiments that test whether CORTIS’s mechanism holds in terms of validity (request order, W3) over extensions. For the camera-ready version, we will extend CORTIS to a ten-request stream under multiple random orders.


W3, W4, Q3. Multiple Random Request Orderss

We agree that the paper would benefit from orders that do not start with speaker 1166. Appx. N.1 currently varies the order, but all three orders begin with 1166.

Experiments. We are running a new set of random ordered unlearning of the fixed speaker set {1166,7199,3912}\{1166, 7199, 3912\}, each starting from a different speaker for CORTIS. We will report Table 1's CORTIS as mean ±\pm std over orders in this thread and in the revised Appx N.1. We will extend this to all baselines, but they cannot be completed within the discussion window. Each baseline request takes approximately 35~40 hours per request (100+ hours per baseline at 1 seed). Covering 4 baselines x 3 orders x 3 requests is therefore far beyond discussion period. We plan to report CORTIS first, and then expand to all baselines in the revised version.

S-R comparison in Table 11 (now Table 12) We would like to clarify on interpretation of the S-R comparison the reviewer raises. We do not claim CORTIS wins on S-R. The gap to TGU and UN at Request 3 (0.557 vs. 0.582 and 0.580) is between methods that all sit well above the retain bound τR\tau_R. The continual claim requires keeping prior-speaker similarity low ( < τF\tau_F), where the gap is an order of magnitude larger. S- f1f_1 is 0.172 for CORTIS vs. 0.603 (TGU) and 0.638 (UN) at Request 3, compared with an across-order range of only 0.128–0.172 for CORTIS.

Reviewer_PeDtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies speaker identity unlearning in zero-shot TTS when deletion requests arrive one at a time and each requester's data must be deleted after processing. It shows that existing methods (TGU and simple continual-unlearning regularizers) fail in this setting: after a new speaker is removed, earlier removed speakers become clonable again ("catastrophic re-learning"). CORTIS combines a Fisher-based parameter mask, which avoids parameters important for retained speakers or earlier deletions, with GPM-style orthogonal projection against the directions used for earlier deletions. On VoiceBox with up to five sequential requests, all removed speakers stay below the forgetting threshold, with moderate utility loss.

Critical strengths
  1. The sequential setting with deletion of forget data is realistic and well motivated, and the failure of existing methods is shown very clearly (e.g., TGU's first speaker jumps back from 0.16 to 0.61 similarity). This is a useful observation for the community.

  2. The method is simple and the ablations are clean. Removing either the projection or the prior-forget term in the mask brings the relapse back, and the cost is much lower than re-running unlearning on all accumulated data.

Critical weaknesses
  1. The framing hides what is actually going on. As the paper itself notes, a model never trained on speaker f can still clone f, so "unlearning" here means learning an exception rule ("if the prompt sounds like f, do not copy it"). The relapse is then standard catastrophic forgetting of that rule, which is why standard continual-learning tools (GPM) fix it. This also raises the question of why the weights need to be modified at all: an input-side speaker-verification filter would block listed voices without any forgetting problem. The weight-editing approach only makes sense if the model weights are released openly, and the paper does not argue this.

  2. The evaluation is small and not adversarial. It uses three to five speakers, one main request order, one backbone, and one speaker-similarity model. It does not test whether a removed voice comes back with light fine-tuning, different prompts, or a different speaker verifier. The strongest reference, TGU re-run on the cumulative forget set, appears only in the cost table, not in the results table. The method also keeps statistics derived from deleted data (the running Fisher max and the subspace), which is in some tension with the non-retention constraint.

Questions for authors
  1. Why is weight editing preferable to an input-side speaker filter? Is the intended deployment open-weight release?

  2. Does a removed speaker come back after light fine-tuning on a small amount of their audio, or under a different speaker verifier?

  3. How does CORTIS behave with tens of requests, given the fixed-rank merged subspace?

  4. Please add cumulative TGU to Table 1 as a reference point.

7 comments

We thank the reviewer for the careful review and acknowledging strengths in our paper's observations and experiments for speaker identity unlearning domain. We address the concerns pointed out by the reviewer below.


W1, Q1. Necessity of Weight Editing Approach

We agree with the characterization: Sec.~3.2 states that speaker identity in ZS-TTS is a zero-shot capability rather than memorized content, so unlearning is a suppression problem in the sense of [1], and catastrophic re-learning is the loss of that suppression. This is the premise of the paper. Also, we claim that for two reasons Speaker Identity Unlearning must live in the weights regardless of deployment.

(i) Non-retention of the capability to generate the forget speaker. With only a speaker filter, the ZS-TTS provider still holds a model (weights, θ0\theta_{0} ) capable of reproducing the forgotten identity. This does not fully guarantee the forget speaker that their voice will not be generated. As long as the weights exist, the filter can be disabled, misconfigured, or bypassed, thus model being misused. The EDPB Opinion 28/2024 [2] assesses whether personal data can be extracted from, or obtained through, the model itself, considering all means reasonably likely to be used, including unintended disclosure of the model. Since ZS-TTS reproduces identity from any prompts, we adopt this strict capability-level non-retention. Thus, adding a filter under black-box api setting does not guarantee that a model is free of risk. We formulate continual speaker identity unlearning so that, when a forget speaker submits an RTBF request against a ZS-TTS system, the deployer removes the capability to produce that speaker's voice without needing to retain their voice, or any means to generate their voice.

(ii) Non-retention of data. A speaker-verification requires the provider to store a speaker-identifiable representation of every forgotten speaker (an identity embedding or reference audio) and to compare every incoming prompt against it for the lifetime. This is exactly the quantity that constraint (C2) in Sec. 3.1 excludes, and that Appendix O identifies as personal data under GDPR Art. 4, 9, 17 [3] and EDPB Opinion 28/2024 [2]. 'Using a filter' therefore is not a solution. Instead it converts an erasure request into data retention and continuous processing of the requester's personal data. Existing filtering approaches which require storing forget-speaker identities are not a compliant substitute.

Revision and deployment. We agree that we should state the deployment setting explicitly. Weight-level suppression is helpful whenever the weights need to be open. However, it is not sufficient alone due to relearning and finetuning attacks.

Overall, we think weight-level suppression is not only needed for open weights, but also closed api settings. Point (i) especially applies equally behind a closed API. In current ZS-TTS domain, several state-of-the-art ZS-TTS models (VoiceBox, VALL-E) have been withheld from release due to misuse risk [4,5]. We believe a weight-level mechanism is a prerequisite for any release path, open or restricted. We do not object to a provider additionally running a filter as defense in depth, but it does not remove the need for the weights themselves to stop reproducing the identity. We have added this discussion to Sec. 3, specifying which designs this framing excludes and why.

[1] A. F. Cooper et al., "Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research," NeurIPS 2025.

[2] European Data Protection Board, Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, adopted 17 Dec. 2024, para. 43.

[3] Regulation (EU) 2016/679 (GDPR), Art. 4(1), 4(14), 9(1), 17(1).

[4] Le et al., "Voicebox: Text-guided multilingual universal speech generation at scale", NeurIPS 2023.

[5] Chen et al., "Neural codec language models are zero-shot text to speech synthesizers", IEEE/ACM TASLP 2025.

W2-1, (Q2-1). Light fine-tuning.

Vulnerability to finetuning (relearning) attacks is not a design failure of CORTIS or any particular unlearn method in the ZS-TTS domain. It is inherent to the domain, and it is the same mechanism behind catastrophic re-learning. When retain-only finetuning take place, the mechanism analysis of Sec. 3.2 applies directly. The retain loss provides no supervision on the suppressed identities, and catastrophic re-learning demonstrates that unconstrained retain-only updates revive them.

However, due to this domain characteristic of ZS-TTS, building a method immune against relearning attacks is impractical. Speaker identity in ZS-TTS is not memorized content but a zero-shot capability (can clone any voice, without direct training on it), so even a model with zero exposure/train on speaker ff, including retrained from scratch, regains the ability to clone f. An attacker with the data (not specifically including f), code, and compute to finetune a ZS-TTS model can then simply pretrain an unrestricted one from scratch, which no unlearning method could ever have touched.

Model immunization (hardening weights against specific fine-tuning directions) inherits the same bypass, and from a service provider's view it adds a rights tension of its own, since a speaker who revokes consent may later wish to opt back in. We view this as an open policy question. [1] argues that unlearning methods must be evaluated against the specifics of the setting and domain they are deployed in, rather than by general-purpose criteria. ZS-TTS is one such setting: the standard diagnostic, re-emerge under fine-tuning, does not carry its usual meaning there.


W2-2, (Q2-2). Evaluation with Different Speaker Verifier.

We thank the reviewer for suggesting additional evaluation to support the robustness of CORTIS. In the initial paper, we used WavLM-TDCNN [6] as the speaker verification model (following evaluation of backbone Voicebox [4] ). Here, we evaluate the continual unlearning baselines on SpeechBrain ECAPA-TDNN [7]. Both speaker verification models are of few SOTA models [8]

Because the two verifiers operate on different cosine scales, we first recalibrated the forget/retain bounds of Appx. E with ECAPA-TDNN on 2,000 LibriSpeech test-clean dataset pair following same protocol.

Verifierdiff.-spk meanforget boundsame-spk meanretain bound
ECAPA-TDNN0.0810.0810.2950.2950.6810.6810.3400.340

Using a different speaker verification model does not bring the forget speaker back. We color successful forget (under the forget bound) as green, and successful retain (over the retain bound) as orange. The evaluation protocol follows Table 1, with only the speaker verification model varied.

MethodS-R ↑\uparrow (WavLM)S- f1f_1 ↓\downarrow (WavLM)S- f2f_2 ↓\downarrow (WavLM)S- f3f_3 ↓\downarrow (WavLM)S-R ↑\uparrow (ECAPA)S- f1f_1 ↓\downarrow (ECAPA)S- f2f_2 ↓\downarrow (ECAPA)S- f3f_3 ↓\downarrow (ECAPA)
UN0.580\color{orange}0.5800.6380.6380.5550.5550.106\color{green}0.1060.598\color{orange}0.5980.5840.5840.5880.5880.010\color{green}0.010
SelFT0.548\color{orange}0.5480.5530.5530.4340.4340.110\color{green}0.1100.605\color{orange}0.6050.4980.4980.5120.5120.031\color{green}0.031
CORTIS0.557\color{orange}0.5570.172\color{green}0.1720.148\color{green}0.1480.124\color{green}0.1240.575\color{orange}0.5750.092\color{green}0.0920.139\color{green}0.1390.053\color{green}0.053

Results under ECAPA are consistent with WavLM. After the 3rd unlearn request, every CORTIS forget speaker stays well below the forget bound, whereas UN and SelFT revive f1f_1 and f2f_2 to 0.50--0.59. S-R remains above the retain bound as well.

[6] Chen et al., Wavlm: Large-scale self-supervised pre-training for full stack speech processing, IEEE J-STSP 2022

[7] Desplanques et al, ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification, Interspeech 2020

[8] Baali et al, SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions, EMNLP 2025 findings

W2-3, (Q4). Cumulative TGU as baseline.

We agree that cumulative TGU is a strong upper-bound reference and will report it. We cannot place it in Table 1 as a row, because Table 1 is restricted to methods that satisfy the constraints of Sec. 3.1. Cumulative TGU must retain every prior forget set DFi−1\mathcal{D}^{\mathcal{F}_{i-1}} and the pre-unlearning checkpoint, which violates (C2). We will add it in Appx. I with other non-retention violating baselines with a pointer from main text.

However during the rebuttal phase, cumulative TGU at requests 2,3 requires 20K and 30K steps on VoiceBox. This is roughly 58 and 87.5 hours of training respectively, about 6 days of training in total before evaluation (appx. 10h eval per request). We will attempt to complete the three-request result within the discussion period and report it as soon as it is available; if it does not finish in time, we commit to including it in the final version.


W2-4. Clarification on Different Prompts.

We would like to clarify that forgetting is evaluated on multiple different prompts. Following prior work [9], each forget speaker contributes average 20 minutes of audio (Sec. 5.1), which we split into disjoint unlearning and evaluation pools. The evaluation pool contains average of 232 audio prompts per speaker (across all 5 speakers).

Forgetting therefore already holds across multiple prompts the model never saw during the unlearning update. We have stated the split and the evaluation prompt count explicitly in Sec.5.3.

[9] Kim et al. "Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech", ICML 2025.


W2-5. Maxed Fisher, Subspace and Non-Retention Constraint.

We clarify that CORTIS does not store any personal data that can be used as biometric information (speaker identity, audio files, etc). As the reviewer has mentioned, only the max-Fisher tensor which is only one scalar per parameter and the merged basis that holds update directions are kept. Thus, there is no identity content retained throughout CORTIS. Each of these retained statistics are many-to-one entangled through element-wise max and aggregate.

Additionally, we show that even on per-speaker Fisher level (before maximum is taken as in CORTIS setting) that each maps are not correlated with speaker's voice. For seven speakers (the five forget speakers and the two remain speakers closest in voice to f1f_1), we compared the similarity of each pair's Fisher maps (Spearman correlation over one fixed subset of 2M parameters, randomly sampled) with the pair's voice similarity (cosine between mean WavLM-TDCNN embeddings). Over the 21 speaker pairs we find no significant relationship (ρ=−0.04\rho=-0.04, p=0.85p=0.85). The speaker closest in voice to f1f_1 (similarity 0.460.46) is farther from it in Fisher space (0.790.79) than the least similar speaker (−0.10-0.10; 0.860.86). We have included the results as a figure in Appx. O with Fig.7.

Comments on the paper

1 comment

We thank the AC for managing the review process and all three reviewers for their careful and constructive feedback.

To the best of our knowledge, this paper is the first to formalize and study speaker identity unlearning specifically under continual scenarios, with efforts to closely resemble real-world unlearning requests (multiple and sequential) and privacy rights (the necessity of non-retention). We tested whether existing ideas from speaker identity unlearning and from continual unlearning in other domains can be applied directly to this setting, and then proposed a simple, effective method for it: CORTIS.

We are grateful that the reviewers recognized these points as strengths:

  • A realistic, well-motivated problem. The sequential setting with deletion of forget data, and the data retention paradox that follows from it (PeDt S1 CeKL S1).
  • Catastrophic re-learning as a useful observation. A failure mode specific to ZS-TTS, shown clearly and consistently across experiments (PeDt S1 CeKL S2).
  • A simple and effective method. Consistent gains in both forgetting and retention, clean ablations in which each component contributes, and much lower cost than the baselines (PeDt S2 CeKL S3 zVdr S1–S3).

The reviewers' suggestions let us strengthen each of these during the discussion period. Below we summarize the updates made during the discussion period. In the revised manuscript, all changes are colored in blue.

  • Second backbone and longer sequences. CORTIS is applied without modification to NaturalSpeech 2 (latent diffusion, pretrained on LibriTTS). TGU re-learns a prior speaker by Request 4 (S-f2f_2 = 0.344 > τF\tau_F), while CORTIS keeps every forget speaker at or below 0.161 through ten sequential requests with retention unchanged (S-R 0.555 vs. 0.556 pretrained) (PeDt W2-6 & Q3 CeKL W2-2 zVdr W1 & Q1).
  • Request-order robustness. CORTIS is run on all six permutations of the three-speaker set. Across 6 orders × 3 continual unlearn requests, the highest forget similarity is 0.198 and the lowest S-R is 0.529. We report mean ± std over orders (CeKL W3, W4 & Q3 PeDt W2).
  • Out-of-domain forget speakers. On three LibriSpeech test-other speakers never seen in pretraining, every forgotten speaker stays at or below 0.145 after three requests, with S-R above 0.52 (CeKL Q6).
  • Evaluation reliability. Results hold under a second speaker verifier (ECAPA-TDNN) with recalibrated bounds. Recalibrating the WavLM bounds on 2,000 pairs instead of 200 only loosens them, so no result in Table 1 changes. We also state the disjoint unlearning/evaluation split (about 232 held-out prompts per speaker) (PeDt W2-2, W2-4 & Q2 zVdr W2 & Q2).
  • Additional baselines. We add SSD and ALKN at the first request; both fail to forget without collapsing utility. We clarify that SelFT (already implemented) is our speech adaptation of SalUn/FIT, and explain how CORTIS differs from O3, FIT, GPM and OGD (CeKL W1 zVdr W3).
  • Mask and retained-statistics analysis. CORTIS masks overlap nearly 3× less than the same saliency without the prior-forget term (Jaccard 0.120 vs. 0.330). Per-speaker Fisher maps show no significant relationship with voice similarity (ρ=−0.04\rho = -0.04, p=0.85p = 0.85), supporting that the retained aggregates carry no voice information (CeKL W5 & Q5 PeDt W2-5).
  • Clarified statements. We clarified the deployment setting and why filters are excluded, the Request-1 schedule, where projection and gradient snapshots are applied, and the headline number, with original similarities for f4f_4 and f5f_5 reported (PeDt W1 & Q1 CeKL W2-1, W6, Q1, Q2 & Q4).

We sincerely hope this work provides a strong foundation for future work in speaker identity unlearning, toward the ultimate goal of protecting everyone's voice. We have left individual comments addressing each reviewer's questions in detail, and we welcome any further questions or comments in this thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.