acceptodds
Under review as a conference paper at ICLR 2027

ESIAN: Training-Free Continual Speaker Identity Erasure for Zero-Shot TTS

Abstract

Zero-shot text-to-speech (ZS-TTS) models can clone an arbitrary voice from only a few seconds of reference audio, even when the speaker was absent from the training data. When a speaker requests that their voice no longer be cloned, the provider therefore needs to address this request regardless of whether that speaker's recordings were used for training. We study erasing target-speaker identity during generation while preserving voice cloning for other speakers. Finetuning-based unlearning approaches update the model's weights for every removal request, a cost that becomes impractical as requests accumulate. Inference-time alternatives avoid weight updates but assume that the incoming prompt is already known to belong to a target speaker. As new requests continue to arrive, maintaining erasure for previously enrolled speakers poses an additional challenge. We propose ESIAN (Erasing Speaker Identity via Adaptive gatiNg), a training-free framework that treats an erasure request as a single enrollment step. From several minutes of enrollment recordings, ESIAN derives a closed-form eraser together with an intrinsic gate that determines, for every incoming prompt, how strongly the eraser should be applied. Our framework enables continual speaker identity erasure without updating the backbone weights or revisiting the utterances of previous requests. Experiments demonstrate that ESIAN achieves erasure comparable to finetuning-based unlearning while preserving voice-cloning quality for the remaining speakers, and maintains low average target-speaker similarity as the enrolled set grows to a hundred speakers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Zero-Shot Text-to-Speech, Speaker Identity Erasure, Voice Cloning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yAvnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ESIAN addresses continual speaker identity suppression in zero-shot TTS without updating backbone weights. It combines a projection on reference-audio conditioning features with an intrinsic gate that identifies registered speakers using backbone activations. Experiments on VoiceBox show substantial target-speaker suppression, preservation of non-target cloning, and stable average performance through 100 registered speakers.

Critical strengths
  1. Relevant deployment problem and practical constraints: The paper addresses the fact that excluding a speaker from training does not prevent zero-shot voice cloning.

  2. An integrated approach without an additional speaker-verification model: Using backbone activations for target identification removes the assumption that target membership is supplied externally. The combination of internal signatures, synthetic-negative calibration, and inference-time intervention is a useful system design.

  3. Substantial empirical analysis and transparent reporting: The evaluation includes non-target preservation, text accuracy, a second speaker-similarity evaluator, and speakers absent from training. The appendices explicitly report detection failures, individual changes during continual enrollment, and limitations of the cost measurements.

Critical weaknesses
  1. The necessity of the projection is unclear: Once the gate identifies a target speaker, why not simply discard the reference-audio conditioning and generate speech from text? The proposed subspace captures most conditioning energy for both target and non-target speakers, so speaker selectivity primarily comes from the gate. Moreover, Table 9 reports nearly identical target similarity for the top-40 projection and removal of the full conditioning span (0.179 versus 0.180). The latter lacks a complete preservation and quality evaluation.

  2. Reliability as the registered population grows remains unresolved: The projection rank stays fixed, but the gate must distinguish incoming prompts against an expanding target set. This can increase opportunities for false matches and complicate the trade-off between missed targets and incorrectly suppressed non-target speakers.

Questions for authors

All questions are addressed in the Weaknesses section.

6 comments

We thank the reviewer for the careful review and for acknowledging the deployment relevance of the problem and the integrated design without an additional speaker-verification model. We address the two weaknesses below.

W1. Necessity of the projection, and the evaluation of full-span removal

ESIAN separates the intervention strength γ\gamma on the reference conditioning from the intervention scope rr. As the reviewer points out, speaker selection is done by the gate, and Sec. 4 of the paper also states that the eraser is not speaker-selective. Unlike discarding the reference conditioning of every detected request, γ\gamma adjusts the strength per request according to the gate score. In addition, the top-rr projection is designed to suppress only as many dominant directions as target suppression needs and to keep the remaining components. At the same level of suppression, removing fewer directions is preferable, because the cost of removing directions that suppression does not need can depend on the backbone and on the metric. We compared the role of each element: voice preservation for remaining speakers against a binary gate, and target output quality against full-span strength scaling.

First, we checked whether a continuous strength reduces the loss of speaker similarity on false-positive remaining requests. Of the 800 remaining pairs under balanced K=10K=10, we regenerated the 36 pairs (5 speakers) in the transition range 0.01<γ<0.990.01<\gamma<0.99 and compared the deployed soft gate with a binary gate at the same threshold, under the same projection and generation settings.

γ\gammaPairs / speakersOriginal SIMSoft gateBinary gate
0.5≤γ<0.990.5\le\gamma<0.99 (false positive)17 / 40.6760.3550.186
0.01<γ<0.50.01<\gamma<0.519 / 50.6690.6590.669

On the false-positive pairs in the transition range, the binary gate intervenes at full strength and mean SIM drops to 0.186, whereas the soft gate adjusts the strength per request and mean SIM is 0.355. For requests from remaining speakers in the evaluated transition range, a continuous strength can therefore preserve more speaker similarity than a binary gate.

Next, we completed the preservation and quality evaluation of full-span on VoiceBox that the reviewer found missing. Below, we present results on 200 target pairs and 800 remaining pairs under the balanced K=10K=10 setting of Table 9, using the same stored gate.

SettingSIM-FWER-F (%)UTMOS-FSIM-RWER-R (%)UTMOS-R
Top-400.1795.744.1270.6474.514.063
Full-span0.1805.864.1300.6514.164.066

On VoiceBox, top-40 reaches nearly the same target suppression, remaining-speaker preservation and output quality as full-span.

Finally, the cost of reducing all directions becomes visible on NaturalSpeech2 (setup in our response to Reviewer 7VKU, Q4). For 10 target speakers (200 pairs) and identical generation settings, we adjusted the strength of full-span (r=128r=128) so that its target SIM matched that of top-20 and top-30, generated with that strength, and compared WER and UTMOS.

Settingγ\gammaSIM-FWER (%)UTMOS
Top-2010.164510.001.671
Full-span0.7210.166011.551.459
Top-3010.135011.571.521
Full-span0.8140.136712.711.379

At a similar target SIM, both top-20 and top-30 give lower mean WER and higher mean UTMOS than full-span strength scaling. With 95% speaker-bootstrap confidence intervals, the UTMOS difference (top-rr minus full-span) is +0.213+0.213 [+0.159, +0.270][+0.159,\,+0.270] for top-20 and +0.142+0.142 [+0.108, +0.173][+0.108,\,+0.173] for top-30, and the WER difference is −1.56-1.56 pp [−3.13, +0.03][-3.13,\,+0.03] and −1.14-1.14 pp [−2.10, −0.15][-2.10,\,-0.15]. That is, on this backbone, suppressing only the dominant directions preserves the quality of the target output better than reducing the strength of the whole conditioning. We have added the full-span evaluation on VoiceBox and the NaturalSpeech2 comparison to the appendix of the revised manuscript (Tables 16 and 18). Changes in the revised manuscript are highlighted in blue.

W2. Reliability as the registered population grows

ESIAN recalibrates τ\tau at every enrollment. When a new speaker is enrolled, whitening and centroids are updated, the synthetic negatives are rescored against the current enrollment set, and τ\tau is reset to meet the target budget α\alpha. Each update therefore reflects the change in the score distribution caused by the larger enrollment set. With this recalibration, the false-positive rate averaged over enrollment orders stays below 2% and leak below 5% up to K=100K=100.

In addition, α\alpha controls the trade-off between missed targets and false positives. α\alpha is the target false-positive budget for the event that a remaining speaker is flagged as a target at least once in MM generation requests. When preservation of remaining speakers has priority, α\alpha can be lowered to reduce false positives.

We first measured the actual effect of a growing enrollment set. The sequential-enrollment experiment in the paper (Table 2) uses a single enrollment order, so to separate the effect of the order we increased KK from 10 to 100 under 10 random enrollment orders of the 100 speakers. Keeping α=0.15\alpha=0.15 and M=20M=20, we evaluated 20 requests per target speaker and 800 requests from the 40 remaining speakers. Below, we present means over the 10 enrollment orders at representative KK.

KKLeak, per request (%)FP, per request (%)
104.00.81
302.00.91
503.00.96
704.11.13
1004.51.75

We do observe that both false positives and leak tend to increase as the number of enrolled speakers grows. However, at every evaluated KK the order-averaged false-positive rate stays below 2% and leak below 5%.

We also checked the speaker-level false positives that correspond to the definition of α\alpha. Under the fixed enrollment order of the paper, the fraction of the 40 remaining speakers who are flagged at least once in their 20 requests is 10.0%, 12.5%, 5.0%, 5.0% and 5.0% at K=10,20,30,50,100K=10, 20, 30, 50, 100, respectively. At each evaluated enrollment stage this fraction is within the target budget α=15%\alpha=15\%.

Finally, we checked how α\alpha moves the trade-off. Below are the results at K=100K=100 when α\alpha is lowered from 0.15 to 0.10.

α\alphaLeak, per request (%)FP, per request (%)
0.154.51.75
0.109.40.625

Lowering α\alpha reduces the false-positive rate from 1.75% to 0.625%, and target leak rises from 4.5% to 9.4% accordingly. These measurements cover K≤100K\leq100.

Thank you for the response. It addresses some of my concerns, but I still see little difference between the proposed method and simply using a gate. I therefore remain unconvinced of the paper’s novelty.

Reviewer_niGzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses continual speaker-identity erasure for zero-shot TTS: as speakers request, one after another, that their voice no longer be cloned, the provider should suppress cloning of the accumulated target set while preserving cloning for everyone else, without updating weights, without an external detector/oracle, and without revisiting earlier enrollment audio. The proposed method ESIAN is built on VoiceBox and has two components: closed-form eraser and intrinsic gate.

ESIAN is evaluated on LibriHeavy and LibriSpeech speakers under standard and continual erasure settings (up to 100 speakers), with additional out-of-domain tests and ablations. It achieves SIM-F of 0.196/0.141 (micro/macro), comparable to TGU (0.171/0.148) and SGU (0.158/0.148), while largely preserving non-target speaker similarity (0.604 vs. 0.615). Enrollment takes approximately 2.5 seconds per speaker, compared with 5–17 hours of finetuning for the baselines.

Critical strengths

The paper provides a thorough and transparent evaluation, including micro/macro analysis, per-speaker tracking, calibration validation, out-of-domain testing, additional quality metrics, and bootstrap confidence intervals. The authors also openly report negative findings, strengthening the credibility and reproducibility of the empirical results.

Critical weaknesses

The conditioning path is only 80-dimensional (the mel dimension). The top-40 subspace already holds about 99.3% of every speaker's energy, and projecting out the full span gives the same SIM-F as top-40. When γ≈1\gamma\approx1, ESIAN therefore removes almost all prompt information, not a speaker-specific component. Proposition 1 follows directly from commutativity of matrix addition and adds little. In effect, ESIAN is "detect the target speaker from backbone activations, then drop the prompt." This is functionally close to the external-verifier filtering baseline, except that it returns prompt-free-like speech instead of refusing. The gate itself is a standard speaker-verification pipeline (mean/std pooling, whitening, cosine to centroids, AS-norm) applied to backbone features. Its main advantage over an off-the-shelf verifier is avoiding a dependency, since ECAPA gating performs on par. This is consequential because the paper's framing (closed-form eraser, continual equivalence guarantee) overstates the methodological novelty.

Questions for authors

N/A

1 comment

We thank the reviewer for the careful reading and for recognizing the transparency of our evaluation, including the negative findings. Below we address the concerns the reviewer has pointed out.

W1-1. Top-40 versus full-span on the 80-dimensional conditioning path

The rank rr is a configurable setting of the eraser, and full-span removal is the setting where the selected directions cover the whole conditioning span (r=80r=80 on VoiceBox). ESIAN is designed to suppress only as many dominant directions as target suppression needs and to keep the remaining components. At the same level of suppression, removing fewer directions is preferable, because the cost of removing directions that suppression does not need can depend on the backbone and on the metric. The gate sets the strength γ\gamma, and the projection applies that strength to the selected directions. Speaker selection is done by the gate, and Sec. 4 of the paper also states that the eraser is not speaker-selective.

Accordingly, we use r=40r=40 on VoiceBox. As the reviewer points out, top-40 holds most of the conditioning energy (about 99.3%), so top-40 and full-span reach a similar level of suppression (Table 9, SIM-F 0.179 vs. 0.180).

The cost of reducing all directions becomes visible on NaturalSpeech2 (setup in our response to Reviewer 7VKU, Q4). For 10 target speakers (200 pairs) and identical generation settings, we adjusted the full-span strength so that its target SIM matched that of top-20 and top-30, generated with that strength, and compared WER and UTMOS.

Settingγ\gammaSIM-FWER (%)UTMOS
Top-2010.164510.001.671
Full-span0.7210.166011.551.459
Top-3010.135011.571.521
Full-span0.8140.136712.711.379

At a similar target SIM, both top-20 and top-30 give lower mean WER and higher mean UTMOS than full-span strength scaling. With 95% speaker-bootstrap confidence intervals, the UTMOS difference (top-rr minus full-span) is +0.213+0.213 [+0.159, +0.270][+0.159,\,+0.270] for top-20 and +0.142+0.142 [+0.108, +0.173][+0.108,\,+0.173] for top-30, and the WER difference is −1.56-1.56 pp [−3.13, +0.03][-3.13,\,+0.03] and −1.14-1.14 pp [−2.10, −0.15][-2.10,\,-0.15]. That is, on this backbone, suppressing only the dominant directions preserves the quality of the target output better than reducing the strength of the whole conditioning.

W1-2. Proposition 1

We agree that Proposition 1 follows directly from the properties of matrix addition. Its role is to state that, because the stored state is a sum of per-speaker statistics, a new request can be added without reusing the enrollment audio of earlier requests. In the revised manuscript we have demoted it to a remark and revised the sentence on sequential and batch equivalence in the contribution list.

W1-3. The gate and ECAPA gating

We agree with the reviewer that the components of the gate are standard speaker-verification tools and that it performs on par with ECAPA gating on clean prompts (Table 3, Table 10). As the reviewer says, the advantage of the intrinsic gate is that it removes the dependency on an external verification model.

This independence from an external verifier can be extended toward robustness against attacks. A gate built on a public verifier can be optimized against directly, because the attacker holds the same weights, whereas a gate that reads the backbone's internal activations cannot be accessed from outside the service. To check this possibility, we extended the submitted setting, which uses only clean enrollment audio. To handle perturbed prompts, we added centroids computed from noise-augmented enrollment audio and combined the scores of the two centroids according to the condition of the input prompt. The backbone, the eraser and the synthetic-negative calibration are the same as in the submission. Under this setting, for 40 target pairs at K=10K=10, we evaluated an attack that applies PGD to the reference waveform to minimize its similarity to the speaker's own clean recording under the public ECAPA (L∞≤ϵL_\infty\le\epsilon, no queries to the service).

Target SIM of the outputNo protectionIntrinsic gateECAPA gate
Clean prompt0.6120.1970.180
Random noise (±0.005\pm0.005)0.4310.1820.176
ECAPA-PGD (ϵ=0.002\epsilon=0.002)0.4760.2770.476
ECAPA-PGD (ϵ=0.005\epsilon=0.005)0.3790.2800.379

Under the attack aimed at ECAPA, the output of the ECAPA gate is the same as the unprotected output, while the intrinsic gate keeps a lower target SIM.

W1-4. Framing

We have revised the framing in the manuscript accordingly, and changes are highlighted in blue. Sec. 4.1 now states the design of the eraser and the distinction between intervention strength and scope.

Reviewer_7VKUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies target-speaker identity erasure in zero-shot text-to-speech (TTS), aiming to suppress the reproduction of designated voices while preserving voice-cloning capabilities for other speakers. To address the challenges of high computational cost and reliance on prior identification of target-speaker prompts, this work proposes ESIAN, a training-free framework combining closed-form conditioning-feature projection with an intrinsic gate calibrated on synthetic negatives to adapt erasure strength for each prompt. Experiments involving 10 target speakers and 40 remaining speakers evaluate speaker similarity and word error rate, while continual-erasure experiments assess sequential enrolment of up to 100 target speakers.

Critical strengths
  1. The proposed method is practically meaningful to save the calculation cost.

  2. The intrinsic gate, calibrated using synthetic negatives, automatically detects target speakers and adjusts the erasure strength without requiring an external speaker verifier.

Critical weaknesses
  1. The proposed projection’s advantage over uniformly reducing reference-audio conditioning remains unclear. The directions selected in Section 4.1 also capture nearly all remaining-speaker conditioning energy, and Table 8 reports lower SIM-F with comparable SIM-R for full-span removal.

  2. The relationship between calibration and preservation quality remains unclear. The false-positive budget α\alpha in Section 4.2 concerns threshold crossings, whereas Equation 7 can also apply partial erasure below the threshold. The effect of these interventions on remaining-speaker identity remains unclear.

  3. Evaluation is limited to VoiceBox, leaving ESIAN’s applicability to other conditioning architectures unverified.

Questions for authors
  1. With gate scores and generation settings fixed, what advantage does the learned projection provide over uniformly reducing, given the full-span results in Table 8?

  2. How does false-positive budget α\alpha relate to actual voice preservation under the soft gate, particularly for remaining speaker prompts near the threshold?

  3. Does ESIAN assume that enrolment recordings are correctly grouped by speaker? How sensitive are the gate and eraser to enrolment sets containing mislabelled recordings or speech from multiple speakers?

  4. Could the authors evaluate ESIAN on another zero-shot TTS backbone and clarify which architectural assumptions are required for its projection and intrinsic gating?

5 comments

We thank the reviewer for the careful review and for acknowledging the practical value of ESIAN's low cost and of an intrinsic gate that is calibrated without an external speaker verifier. We address the questions raised by the reviewer below.

W1, Q1. Advantage of the projection over uniformly reducing the conditioning

The rank rr is a configurable setting of the eraser, and the full-span removal that the reviewer compares against is the setting where the selected directions cover the whole conditioning span (r=80r=80 on VoiceBox). Applying γ\gamma to it is the same as uniformly reducing the reference conditioning. ESIAN is designed to suppress only as many dominant directions as target suppression needs and to keep the remaining components. At the same level of suppression, removing fewer directions is preferable, because the cost of removing directions that suppression does not need can depend on the backbone and on the metric. The gate sets the strength γ\gamma, and the projection applies that strength to the selected directions.

Accordingly, we use r=40r=40 on VoiceBox. In the rank sweep of Table 8, r=20r=20 leaves target similarity higher (SIM-F 0.231 vs. 0.196 under the main protocol), while r=40r=40 reaches a level of suppression similar to larger ranks. Under the balanced K=10K=10 protocol of Table 9 with the same gate, SIM-F is 0.179 for top-40 and 0.180 for full-span, and under the main protocol of Table 8 full-span is lower by 0.013.

The cost of reducing all directions becomes visible on NaturalSpeech2 (setup in Q4). For 10 target speakers (200 pairs) and identical generation settings, we adjusted the full-span strength so that its target SIM matched that of top-20 and top-30, generated with that strength, and compared WER and UTMOS.

Settingγ\gammaSIM-FWER (%)UTMOS
Top-2010.164510.001.671
Full-span0.7210.166011.551.459
Top-3010.135011.571.521
Full-span0.8140.136712.711.379

At a similar target SIM, both top-20 and top-30 give lower mean WER and higher mean UTMOS than full-span strength scaling. With 95% speaker-bootstrap confidence intervals, the UTMOS difference (top-rr minus full-span) is +0.213+0.213 [+0.159, +0.270][+0.159,\,+0.270] for top-20 and +0.142+0.142 [+0.108, +0.173][+0.108,\,+0.173] for top-30, and the WER difference is −1.56-1.56 pp [−3.13, +0.03][-3.13,\,+0.03] and −1.14-1.14 pp [−2.10, −0.15][-2.10,\,-0.15]. That is, on this backbone, suppressing only the dominant directions preserves the quality of the target output better than reducing the strength of the whole conditioning.

W2, Q2. False-positive budget and actual voice preservation under the soft gate

The false-positive budget α\alpha is a target set by the provider, defined on the event that a remaining speaker is flagged as a target at least once in MM generation requests. How much voice is preserved on a given request depends on the intervention strength γ\gamma that the request receives. To examine this, we compared Original and ESIAN on the same remaining requests with the same generation seed (K=10K=10, 40 remaining speakers, 800 pairs).

γ\gamma on remaining requestsRequests (%)Original SIM-RESIAN SIM-R
γ<0.01\gamma < 0.01760 (95%)0.6570.657
0.01≤γ<0.250.01 \leq \gamma < 0.2516 (2.0%)0.6850.686
0.25≤γ<0.50.25 \leq \gamma < 0.53 (0.4%)0.6700.635
0.5≤γ<0.750.5 \le \gamma < 0.756 (0.75%)0.6600.506
γ≥0.75\gamma \ge 0.7515 (1.9%)0.6860.171

Below the threshold, the partial intervention of Eq. 7 is rare and its effect is small. For the 97% of requests with γ<0.25\gamma<0.25, mean SIM is the same as Original. The 3 pairs with 0.25≤γ<0.50.25\le\gamma<0.5 drop from 0.670 to 0.635, and their SIMs after intervention (0.669, 0.621, 0.615) are all within the 10th to 90th percentile range [0.538, 0.765][0.538,\,0.765] of the per-request SIM of Original on the 800 remaining pairs.

For the 21 requests that cross the threshold, the loss of speaker similarity is larger (0.660 to 0.506, and 0.686 to 0.171). These come from 4 of the 40 remaining speakers, so the fraction of speakers flagged at least once in their 20 requests is 10.0%, within the target budget α=15%\alpha=15\%. In other words, α\alpha does not bound how much similarity is lost on a false positive. It is a target budget for speaker-level false-positive events.

We ran the same analysis for K=20K=20 to 100100 on the range 0.25≤γ<0.50.25\le\gamma<0.5 (same 40 remaining speakers, percentages out of 800 pairs).

KKRequests in range (%)Original SIM-RESIAN SIM-R
204 (0.5%)0.6610.598
301 (0.125%)0.7570.595
500––
1000––

At every KK, the requests in this range stay at or below 0.5% of all requests.

Q3. Enrollment sets with mislabelled recordings or speech from multiple speakers

Speaker identity erasure assumes that the enrollment recordings are correctly grouped by the speaker who requested erasure. Because the eraser is not speaker-selective (Sec. 4), sensitivity to mislabelled recordings is a question about the gate. With 10% of each enrollment set mislabelled, target detection and false positives on remaining speakers stay stable, and target leak rises as the mislabelled fraction grows to 50%.

To measure this, we replaced 10%, 25% and 50% of each target speaker's 40 enrollment utterances with utterances from another speaker and recomputed the centroids (K=10K=10 and 5050, 5 seeds that vary the mixed-in speaker and the replaced utterances). Leak is measured on 20 held-out utterances per target speaker, and the false-positive rate on the 40 remaining speakers (800 pairs), who are unrelated to the mislabelled recordings. Rows with mislabelled recordings report mean ±\pm standard deviation over the seeds, and the 0% rows are the paper's setting.

KKMislabelled (%)Leak (%)FP on remaining (%)
1003.52.6
10103.9±1.33.9 \pm 1.32.2±0.42.2 \pm 0.4
10254.3±1.74.3 \pm 1.71.9±0.61.9 \pm 0.6
10507.9±3.97.9 \pm 3.91.2±0.81.2 \pm 0.8
5002.91.75
50103.8±0.73.8 \pm 0.71.7±0.41.7 \pm 0.4
50255.4±0.75.4 \pm 0.71.3±0.41.3 \pm 0.4
50509.1±1.59.1 \pm 1.51.6±0.51.6 \pm 0.5

With 10% mislabelled, the mean increase in target leak is 0.4 pp at K=10K=10 and 0.9 pp at K=50K=50, and the false-positive rate of the unrelated remaining speakers does not increase. At 50%, target leak rises to 7.9% and 9.1%, respectively.

A speaker whose recordings are wrongly included has their voice characteristics reflected in the enrollment set, and can therefore be recognized as a target. Correct enrollment sets are thus needed in particular to protect such wrongly included non-target speakers.

W3, Q4. Another backbone and the architectural assumptions

ESIAN requires (i) a linear input projection where the contribution of the reference can be projected out, and (ii) internal activations from which speaker identity can be read. NaturalSpeech2 [1] meets both conditions, and ESIAN suppresses target speakers and preserves remaining speakers on it. We used Amphion's [2] LibriTTS-based reimplementation and its public weights. Unlike VoiceBox, a mel-based flow-matching model, NaturalSpeech2 is a diffusion model over neural-codec latents and conditions on the reference through a prompt encoder. In this implementation, the codec latents enter through a linear layer (i), and the internal signatures of the prompt encoder separate target from non-target prompts with AUC 0.997 (ii).

The evaluation uses the balanced K=10K=10 protocol with 200 target pairs and 800 remaining pairs. NaturalSpeech2 does not support generation without a reference, so we used 400 utterances generated by VoiceBox without a reference as the synthetic-negative bank and set the threshold at its 99.25% quantile. Target leak is 1.0%, and the speaker-level false-positive rate is 5.0% (2/40).

SettingSIM-FSIM-RWER-F / WER-R (%)UTMOS-F / UTMOS-R
Original0.4300.3567.00 / 7.532.47 / 2.94
ESIAN (r=50r=50)0.0880.35213.48 / 7.601.51 / 2.93
ESIAN (r=30r=30)0.1350.35211.57 / 7.621.52 / 2.93
ESIAN (r=20r=20)0.1650.35310.00 / 7.571.67 / 2.93

At all three ranks target SIM decreases, while remaining SIM and quality stay close to Original. On this backbone, rr sets the trade-off between target suppression and the quality of the target output. From r=20r=20 to r=50r=50, target SIM goes from 0.165 to 0.088 and target WER from 10.00% to 13.48% (Original 7.00%). We have added the NaturalSpeech2 results to the appendix of the revised manuscript (Tables 17 and 18). Changes in the revised manuscript are highlighted in blue.

[1] Shen et al., "NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers," ICLR 2024.

[2] Zhang et al., "Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit," IEEE SLT 2024.

We would like to share one additional result for W3 and Q4. We trained TGU [3] on NaturalSpeech2 for the same 10 target speakers (1,000 steps, a single run with its default settings) and evaluated it under the same protocol.

SettingSIM-FSIM-RWER-F / WER-R (%)UTMOS-F / UTMOS-R
Original0.4300.3567.00 / 7.532.47 / 2.94
TGU0.1740.28911.00 / 7.551.77 / 2.66
ESIAN (r=20r=20)0.1650.35310.00 / 7.571.67 / 2.93

At a target SIM close to ESIAN (r=20r=20), TGU also raises target WER, and its remaining SIM and UTMOS fall below Original. We would be glad to answer any remaining questions.

[3] Kim et al., "Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech," ICML 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.