- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that speaker embeddings trained for verification are widely misused to measure perceptual voice similarity, even though the two objectives are misaligned. The authors empirically show that verification-trained embeddings barely correlate with human similarity ratings on different-speaker pairs, and theoretically prove that the optimal geometry under the verification objective collapses intra-speaker variation and spreads speakers uniformly, while remaining invariant to perceptual similarity rankings. They propose Simpson, which adds a ranking objective on top of verification, with ordering targets from a frozen self-supervised backbone and an EMA teacher, requiring no human similarity labels, plus an optional preference stage with a small set of human labels. Simpson reaches a Spearman correlation of 0.79 with human ratings on different-speaker pairs and is validated on TTS evaluation and GRPO reward tasks.
Critical strengths
- The paper first shows why the verification objective fails for perceptual similarity, then designs the ranking objective accordingly.
- It proves the verification-optimal geometry is invariant to perceptual similarity rankings, explaining why the near-zero correlation is structural, not model-specific.
Critical weaknesses
- Figure 2 shows that human ratings for the same speaker are inconsistent, yet the paper does not specify which dimensions raters scored on or report inter-rater agreement. If the rating criteria are vague, the "human perception" baseline against which the 0.79 correlation is measured is questionable; moreover, representing similarity with a single score implicitly assumes a one-dimensional, rankable quantity, which contradicts the multi-dimensionality suggested by the inconsistent ratings.
- The ranking targets are defined by a frozen self-supervised backbone and an EMA teacher, and teacher bias on unseen accents or ages may propagate to the student model, with limited discussion of mitigation. Meanwhile, the clear male-female separation in the right panel of Figure 3 is inconsistent with the expectation of "learning continuous perceptual similarity," possibly indicating that the ranking objective has introduced coarse-grained category signals such as gender, which the paper does not explain.
Questions for authors
Figure 2 shows inconsistent human ratings for the same speaker, Figure 3 shows a clear male-female separation, and the ranking objective depends on teacher models. Is the human "similarity" being rated a single overall impression or a mix of timbre, prosody, and pronunciation dimensions? If raters weigh dimensions differently, is single-score benchmarking itself limited? Does the gender boundary mean the model learned coarse category signals rather than continuous perceptual similarity, and have you checked the relationship between the gender direction and the similarity ranking direction?
5 comments
[W4] Teacher bias on unseen accents or ages.
Thank you for this valuable observation. We acknowledge that Simpson, like other speaker embedders, may carry biases on unseen accents or ages. Since these biases often arise from the training data, mitigating them would require retraining from the backbone on large-scale diverse speech corpora, which is beyond the scope of this work. The base Simpson training objective does not require human similarity labels, reducing the annotation burden of extending training to more diverse data. We have added this discussion to the future work in Appendix D.
[W5, Q2] Male-female separation and the gender direction.
We are especially grateful for this question about what Simpson learns. Figure 3 is a toy example of Corollary 1, which bounds the distance between group centroids when the bound in Proposition 1 is attained. For the two groups, we chose gender, a salient distinction in VoxSim ratings (see below), and we added an order within each group. In the right panel, the points within each group also lie in order along an arc, so the separation comes with a continuous order rather than in place of it. In Simpson, gender separation also coexists with within-gender perceptual ranking, as the following analyses show.
Human ratings strongly reflect gender. Among the different-speaker pairs of VoxSim test, 85% of cross-gender pairs receive the lowest human rating ("definitely different speakers"), against 6% of same-gender pairs. VoxSim-ft, fine-tuned on these ratings, separates the genders as well (centroid gap 0.90, Figure 8). In Simpson, trained without any gender label, this separation lies along the first principal component of the speaker space (57% of the variance, correlation with gender 0.96), which we call the gender direction.
Gender direction and similarity ranking. We have checked this relationship on the different-speaker pairs of VoxSim test (SRCC). We either removed the gender direction from Simpson's embeddings or scored each pair by the product of the two embeddings' components along that direction.
| Similarity computed from | All different-speaker pairs | Same-gender different-speaker pairs |
|---|---|---|
| Full embedding (Simpson) | 0.785 | 0.49 |
| Gender direction removed | 0.11 | 0.38 |
| Gender direction only | 0.76 | 0.26 |
| WavLM(Large)-ECAPA, full embedding (Table 19) | 0.061 | 0.115 |
Within the same gender, the SRCC falls from 0.49 to 0.38 after the gender direction is removed and remains 0.26 with the gender direction alone, whereas a binary male/female label would give all such pairs the same score. These results show that Simpson captures both a strong gender-associated component and perceptual ranking within gender groups.
Controlling the gender direction. Prompted by the reviewer's question, we also tested whether the influence of gender can be changed after training. To adjust the contribution of this gender-associated direction, we scaled the component of every embedding along it by a factor and left the rest unchanged. The direction is the first principal component, so no gender label is needed; a direction estimated from 35 other VoxCeleb1 speakers gives similar results (cosine 0.99). We report the SRCC on each test set.
| Variance along the gender direction | VoxSim test (full / diff / same) | Same-gender different-speaker pairs | VCC2018 | VCC2020 | TTSDS2 | |
|---|---|---|---|---|---|---|
| 1 (Simpson) | 57% | 0.822 / 0.785 / 0.617 | 0.49 | 0.547 | 0.739 | 0.415 |
| 0.5 | 27% | 0.806 / 0.597 / 0.607 | 0.44 | 0.520 | 0.732 | 0.400 |
| 0.35 | 15% | 0.789 / 0.404 / 0.601 | 0.41 | 0.498 | 0.728 | 0.395 |
| 0 | 0% | 0.754 / 0.111 / 0.593 | 0.38 | 0.458 | 0.722 | 0.388 |
As decreases to 0, the largest decrease occurs in the VoxSim different-speaker SRCC, consistent with the strong association between gender and human ratings. We have added these experimental results and analyses to Appendix C.13.
We thank the reviewer once more for the careful reading and the questions, which have made our paper clearer and stronger. In particular, the new result above came from the reviewer's excellent question on the gender direction. We hope our responses have addressed the reviewer's concerns and would welcome any further questions.
We thank the reviewer again for the careful review and insightful questions. We have revised the paper following the reviewer’s comments, with all changes shown in red. In particular, we have added the rating criteria and the inter-rater agreement of VoxSim to Appendix A.2 and the analysis of the gender direction to Appendix C.13. As the discussion period ends soon, we would sincerely appreciate any feedback on whether our responses have addressed the reviewer’s concerns, and we would be happy to answer any remaining questions.
We thank the reviewer for the careful review and insightful questions. We have added new analyses. We have uploaded a revised paper, with all changes shown in red, and address each point below.
[W1, W2] Rating criteria and inter-rater agreement of VoxSim.Thank you for raising this important issue. We would like to clarify that we use the publicly released human ratings of VoxSim [1]. The dataset was built specifically to train and evaluate models that predict perceived voice similarity, and its ratings were collected under explicit criteria and quality control. In short, raters assessed the voice characteristics of the main speaker on a 6-point scale ranging from “definitely different speakers” to “definitely the same speaker,” regardless of content, language, or acoustic environment. Please refer to Section 2 of the VoxSim paper [1] for more details.
As the reviewer suggested, we have measured the inter-rater agreement on the released training labels. The split-half reliability of the mean rating is 0.90 (Spearman–Brown corrected), supporting the consistency of the aggregated human ratings. We have added the rating criteria and this discussion to Appendix A.2.
In addition, we respectfully note that Figure 2 shows variation across different same-speaker utterance pairs, which does not by itself establish disagreement among human raters evaluating the same pair.
[1] VoxSim: A perceptual voice similarity dataset
[W3, Q1] Single-score similarity.Thank you for this thoughtful question. VoxSim raters gave one score per pair for the voice as a whole, regardless of content, language, and acoustic environment ([1], Section 2). We therefore understand the VoxSim rating as a single overall impression of the voice. Such a score can summarize a judgment based on multiple voice characteristics without assuming that perceptual similarity is intrinsically one-dimensional.
To examine which acoustic characteristics are associated with these ratings, we have correlated the mean rating of each pair with the closeness of its two clips in gender and in the 14 acoustic markers of [6] on the VoxSim training split (table below). Among the tested acoustic markers, mean pitch (0.45) and the harmonics-to-noise ratio (HNR, 0.39) show the strongest correlations with human ratings. Within same-gender pairs, these correlations remain positive at 0.23 and 0.15, respectively.
Across the 14 markers, the correlations of Simpson are highly consistent with those of the raters (Pearson r = 0.997). Following [6], we have also predicted each marker from the embeddings of speakers unseen by the regressor. For every one of the 14 markers, the r² of Simpson is higher than those of ECAPA-TDNN and WavLM(Large)-ECAPA, with a mean of 0.70 against 0.25 and 0.19, respectively, so each dimension remains recoverable from the Simpson embedding.
We would respectfully note that an overall similarity score is commonly used in speech evaluation, both as the cosine between speaker embeddings (SIM) [2–5] and as the similarity MOS (SMOS) of listening tests. A single similarity score is used in all four benchmarks of our evaluation (VoxSim, VCC2018, VCC2020, and TTSDS2).
As the reviewer suggested, separating similarity into its dimensions is a promising direction. We thank the reviewer for this important question.
[1] VoxSim: A perceptual voice similarity dataset
[2] Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
[3] Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
[4] FlowTSE: Target Speaker Extraction with Flow Matching
[5] FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching
[6] Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis