- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ESIAN addresses continual speaker identity suppression in zero-shot TTS without updating backbone weights. It combines a projection on reference-audio conditioning features with an intrinsic gate that identifies registered speakers using backbone activations. Experiments on VoiceBox show substantial target-speaker suppression, preservation of non-target cloning, and stable average performance through 100 registered speakers.
Critical strengths
-
Relevant deployment problem and practical constraints: The paper addresses the fact that excluding a speaker from training does not prevent zero-shot voice cloning.
-
An integrated approach without an additional speaker-verification model: Using backbone activations for target identification removes the assumption that target membership is supplied externally. The combination of internal signatures, synthetic-negative calibration, and inference-time intervention is a useful system design.
-
Substantial empirical analysis and transparent reporting: The evaluation includes non-target preservation, text accuracy, a second speaker-similarity evaluator, and speakers absent from training. The appendices explicitly report detection failures, individual changes during continual enrollment, and limitations of the cost measurements.
Critical weaknesses
-
The necessity of the projection is unclear: Once the gate identifies a target speaker, why not simply discard the reference-audio conditioning and generate speech from text? The proposed subspace captures most conditioning energy for both target and non-target speakers, so speaker selectivity primarily comes from the gate. Moreover, Table 9 reports nearly identical target similarity for the top-40 projection and removal of the full conditioning span (0.179 versus 0.180). The latter lacks a complete preservation and quality evaluation.
-
Reliability as the registered population grows remains unresolved: The projection rank stays fixed, but the gate must distinguish incoming prompts against an expanding target set. This can increase opportunities for false matches and complicate the trade-off between missed targets and incorrectly suppressed non-target speakers.
Questions for authors
All questions are addressed in the Weaknesses section.
6 comments
W2. Reliability as the registered population grows
ESIAN recalibrates at every enrollment. When a new speaker is enrolled, whitening and centroids are updated, the synthetic negatives are rescored against the current enrollment set, and is reset to meet the target budget . Each update therefore reflects the change in the score distribution caused by the larger enrollment set. With this recalibration, the false-positive rate averaged over enrollment orders stays below 2% and leak below 5% up to .
In addition, controls the trade-off between missed targets and false positives. is the target false-positive budget for the event that a remaining speaker is flagged as a target at least once in generation requests. When preservation of remaining speakers has priority, can be lowered to reduce false positives.
We first measured the actual effect of a growing enrollment set. The sequential-enrollment experiment in the paper (Table 2) uses a single enrollment order, so to separate the effect of the order we increased from 10 to 100 under 10 random enrollment orders of the 100 speakers. Keeping and , we evaluated 20 requests per target speaker and 800 requests from the 40 remaining speakers. Below, we present means over the 10 enrollment orders at representative .
| Leak, per request (%) | FP, per request (%) | |
|---|---|---|
| 10 | 4.0 | 0.81 |
| 30 | 2.0 | 0.91 |
| 50 | 3.0 | 0.96 |
| 70 | 4.1 | 1.13 |
| 100 | 4.5 | 1.75 |
We do observe that both false positives and leak tend to increase as the number of enrolled speakers grows. However, at every evaluated the order-averaged false-positive rate stays below 2% and leak below 5%.
We also checked the speaker-level false positives that correspond to the definition of . Under the fixed enrollment order of the paper, the fraction of the 40 remaining speakers who are flagged at least once in their 20 requests is 10.0%, 12.5%, 5.0%, 5.0% and 5.0% at , respectively. At each evaluated enrollment stage this fraction is within the target budget .
Finally, we checked how moves the trade-off. Below are the results at when is lowered from 0.15 to 0.10.
| Leak, per request (%) | FP, per request (%) | |
|---|---|---|
| 0.15 | 4.5 | 1.75 |
| 0.10 | 9.4 | 0.625 |
Lowering reduces the false-positive rate from 1.75% to 0.625%, and target leak rises from 4.5% to 9.4% accordingly. These measurements cover .
Thank you for the response. It addresses some of my concerns, but I still see little difference between the proposed method and simply using a gate. I therefore remain unconvinced of the paper’s novelty.
We thank the reviewer for the careful review and for acknowledging the deployment relevance of the problem and the integrated design without an additional speaker-verification model. We address the two weaknesses below.
W1. Necessity of the projection, and the evaluation of full-span removal
ESIAN separates the intervention strength γ on the reference conditioning from the intervention scope r. As the reviewer points out, speaker selection is done by the gate, and Sec. 4 of the paper also states that the eraser is not speaker-selective. Unlike discarding the reference conditioning of every detected request, γ adjusts the strength per request according to the gate score. In addition, the top-r projection is designed to suppress only as many dominant directions as target suppression needs and to keep the remaining components. At the same level of suppression, removing fewer directions is preferable, because the cost of removing directions that suppression does not need can depend on the backbone and on the metric. We compared the role of each element: voice preservation for remaining speakers against a binary gate, and target output quality against full-span strength scaling.
First, we checked whether a continuous strength reduces the loss of speaker similarity on false-positive remaining requests. Of the 800 remaining pairs under balanced K=10, we regenerated the 36 pairs (5 speakers) in the transition range 0.01<γ<0.99 and compared the deployed soft gate with a binary gate at the same threshold, under the same projection and generation settings.
On the false-positive pairs in the transition range, the binary gate intervenes at full strength and mean SIM drops to 0.186, whereas the soft gate adjusts the strength per request and mean SIM is 0.355. For requests from remaining speakers in the evaluated transition range, a continuous strength can therefore preserve more speaker similarity than a binary gate.
Next, we completed the preservation and quality evaluation of full-span on VoiceBox that the reviewer found missing. Below, we present results on 200 target pairs and 800 remaining pairs under the balanced K=10 setting of Table 9, using the same stored gate.
On VoiceBox, top-40 reaches nearly the same target suppression, remaining-speaker preservation and output quality as full-span.
Finally, the cost of reducing all directions becomes visible on NaturalSpeech2 (setup in our response to Reviewer 7VKU, Q4). For 10 target speakers (200 pairs) and identical generation settings, we adjusted the strength of full-span (r=128) so that its target SIM matched that of top-20 and top-30, generated with that strength, and compared WER and UTMOS.
At a similar target SIM, both top-20 and top-30 give lower mean WER and higher mean UTMOS than full-span strength scaling. With 95% speaker-bootstrap confidence intervals, the UTMOS difference (top-r minus full-span) is +0.213 [+0.159,+0.270] for top-20 and +0.142 [+0.108,+0.173] for top-30, and the WER difference is −1.56 pp [−3.13,+0.03] and −1.14 pp [−2.10,−0.15]. That is, on this backbone, suppressing only the dominant directions preserves the quality of the target output better than reducing the strength of the whole conditioning. We have added the full-span evaluation on VoiceBox and the NaturalSpeech2 comparison to the appendix of the revised manuscript (Tables 16 and 18). Changes in the revised manuscript are highlighted in blue.