acceptodds
Under review as a conference paper at ICLR 2027

Beyond Speaker Verification: Speaker Embeddings for Perceptual Similarity

Abstract

Speaker embedders are trained to verify whether two utterances come from the same speaker. However, the same embedders increasingly serve as evaluation metrics, reward models, and data filters for a different purpose: estimating how similar two voices sound to human listeners. We first show that verification-trained embedders do not reliably preserve perceptual similarity rankings, and that this limitation stems from the verification objective rather than from any particular model. For a speaker embedder widely used in text-to-speech (TTS) evaluation, scores on different-speaker pairs exhibit almost no correlation with human ratings. Our theoretical analysis examines the optimal embedding geometry under the verification objective, which collapses intra-speaker variation and spreads speakers uniformly. We prove that this geometry is invariant to inter-speaker perceptual similarity rankings and show empirically that existing speaker embedders closely approach it. Building on this analysis, we propose Simpson, a speaker embedder that measures Similarity like a person. Simpson augments the verification objective with a ranking objective to learn perceptual similarity rankings without human similarity labels. The ordering targets are derived from a frozen self-supervised backbone and Simpson's own exponential moving average (EMA) teacher. An optional preference stage further refines the model using a small set of human similarity labels. Across natural, converted, and synthetic speech, Simpson achieves the highest correlation with human ratings among the publicly available embedders evaluated, reaching a Spearman rank correlation of 0.79 on different-speaker pairs where verification-trained embedders show near-zero correlation. We further demonstrate Simpson's practical effectiveness in downstream tasks, both as an evaluation metric for TTS and as a reward function for group relative policy optimization (GRPO).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speaker similarity, speaker embedding, perceptual evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eBiKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that speaker embeddings trained for verification are widely misused to measure perceptual voice similarity, even though the two objectives are misaligned. The authors empirically show that verification-trained embeddings barely correlate with human similarity ratings on different-speaker pairs, and theoretically prove that the optimal geometry under the verification objective collapses intra-speaker variation and spreads speakers uniformly, while remaining invariant to perceptual similarity rankings. They propose Simpson, which adds a ranking objective on top of verification, with ordering targets from a frozen self-supervised backbone and an EMA teacher, requiring no human similarity labels, plus an optional preference stage with a small set of human labels. Simpson reaches a Spearman correlation of 0.79 with human ratings on different-speaker pairs and is validated on TTS evaluation and GRPO reward tasks.

Critical strengths
  1. The paper first shows why the verification objective fails for perceptual similarity, then designs the ranking objective accordingly.
  2. It proves the verification-optimal geometry is invariant to perceptual similarity rankings, explaining why the near-zero correlation is structural, not model-specific.
Critical weaknesses
  1. Figure 2 shows that human ratings for the same speaker are inconsistent, yet the paper does not specify which dimensions raters scored on or report inter-rater agreement. If the rating criteria are vague, the "human perception" baseline against which the 0.79 correlation is measured is questionable; moreover, representing similarity with a single score implicitly assumes a one-dimensional, rankable quantity, which contradicts the multi-dimensionality suggested by the inconsistent ratings.
  2. The ranking targets are defined by a frozen self-supervised backbone and an EMA teacher, and teacher bias on unseen accents or ages may propagate to the student model, with limited discussion of mitigation. Meanwhile, the clear male-female separation in the right panel of Figure 3 is inconsistent with the expectation of "learning continuous perceptual similarity," possibly indicating that the ranking objective has introduced coarse-grained category signals such as gender, which the paper does not explain.
Questions for authors

Figure 2 shows inconsistent human ratings for the same speaker, Figure 3 shows a clear male-female separation, and the ranking objective depends on teacher models. Is the human "similarity" being rated a single overall impression or a mix of timbre, prosody, and pronunciation dimensions? If raters weigh dimensions differently, is single-score benchmarking itself limited? Does the gender boundary mean the model learned coarse category signals rather than continuous perceptual similarity, and have you checked the relationship between the gender direction and the similarity ranking direction?

5 comments

We thank the reviewer for the careful review and insightful questions. We have added new analyses. We have uploaded a revised paper, with all changes shown in red, and address each point below.

[W1, W2] Rating criteria and inter-rater agreement of VoxSim.

Thank you for raising this important issue. We would like to clarify that we use the publicly released human ratings of VoxSim [1]. The dataset was built specifically to train and evaluate models that predict perceived voice similarity, and its ratings were collected under explicit criteria and quality control. In short, raters assessed the voice characteristics of the main speaker on a 6-point scale ranging from “definitely different speakers” to “definitely the same speaker,” regardless of content, language, or acoustic environment. Please refer to Section 2 of the VoxSim paper [1] for more details.

As the reviewer suggested, we have measured the inter-rater agreement on the released training labels. The split-half reliability of the mean rating is 0.90 (Spearman–Brown corrected), supporting the consistency of the aggregated human ratings. We have added the rating criteria and this discussion to Appendix A.2.

In addition, we respectfully note that Figure 2 shows variation across different same-speaker utterance pairs, which does not by itself establish disagreement among human raters evaluating the same pair.

[1] VoxSim: A perceptual voice similarity dataset

[W3, Q1] Single-score similarity.

Thank you for this thoughtful question. VoxSim raters gave one score per pair for the voice as a whole, regardless of content, language, and acoustic environment ([1], Section 2). We therefore understand the VoxSim rating as a single overall impression of the voice. Such a score can summarize a judgment based on multiple voice characteristics without assuming that perceptual similarity is intrinsically one-dimensional.

To examine which acoustic characteristics are associated with these ratings, we have correlated the mean rating of each pair with the closeness of its two clips in gender and in the 14 acoustic markers of [6] on the VoxSim training split (table below). Among the tested acoustic markers, mean pitch (0.45) and the harmonics-to-noise ratio (HNR, 0.39) show the strongest correlations with human ratings. Within same-gender pairs, these correlations remain positive at 0.23 and 0.15, respectively.

Spearman with closenessRatersSimpsonECAPA-TDNNWavLM(Large)-ECAPA
Same gender0.810.820.000.02
Mean pitch0.450.570.020.03
HNR0.390.480.010.02
α-ratio (mean)0.060.110.040.03
α-ratio (std)0.100.140.030.03
Shimmer0.070.100.000.00
Pitch variation0.080.080.020.01
Loudness (mean)0.010.020.030.02
Loudness (std)0.000.020.000.00
Voiced segment length (mean)0.020.040.010.01
Voiced segment length (std)0.000.020.000.01
Unvoiced segment length (mean)0.020.030.00−0.01
Unvoiced segment length (std)0.020.03−0.01−0.01
Speech rate0.000.020.000.00
Utterance duration0.00−0.02−0.01−0.01

Across the 14 markers, the correlations of Simpson are highly consistent with those of the raters (Pearson r = 0.997). Following [6], we have also predicted each marker from the embeddings of speakers unseen by the regressor. For every one of the 14 markers, the r² of Simpson is higher than those of ECAPA-TDNN and WavLM(Large)-ECAPA, with a mean of 0.70 against 0.25 and 0.19, respectively, so each dimension remains recoverable from the Simpson embedding.

We would respectfully note that an overall similarity score is commonly used in speech evaluation, both as the cosine between speaker embeddings (SIM) [2–5] and as the similarity MOS (SMOS) of listening tests. A single similarity score is used in all four benchmarks of our evaluation (VoxSim, VCC2018, VCC2020, and TTSDS2).

As the reviewer suggested, separating similarity into its dimensions is a promising direction. We thank the reviewer for this important question.

[1] VoxSim: A perceptual voice similarity dataset

[2] Seed-TTS: A Family of High-Quality Versatile Speech Generation Models

[3] Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

[4] FlowTSE: Target Speaker Extraction with Flow Matching

[5] FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching

[6] Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis

[W4] Teacher bias on unseen accents or ages.

Thank you for this valuable observation. We acknowledge that Simpson, like other speaker embedders, may carry biases on unseen accents or ages. Since these biases often arise from the training data, mitigating them would require retraining from the backbone on large-scale diverse speech corpora, which is beyond the scope of this work. The base Simpson training objective does not require human similarity labels, reducing the annotation burden of extending training to more diverse data. We have added this discussion to the future work in Appendix D.

[W5, Q2] Male-female separation and the gender direction.

We are especially grateful for this question about what Simpson learns. Figure 3 is a toy example of Corollary 1, which bounds the distance between group centroids when the bound in Proposition 1 is attained. For the two groups, we chose gender, a salient distinction in VoxSim ratings (see below), and we added an order within each group. In the right panel, the points within each group also lie in order along an arc, so the separation comes with a continuous order rather than in place of it. In Simpson, gender separation also coexists with within-gender perceptual ranking, as the following analyses show.

Human ratings strongly reflect gender. Among the different-speaker pairs of VoxSim test, 85% of cross-gender pairs receive the lowest human rating ("definitely different speakers"), against 6% of same-gender pairs. VoxSim-ft, fine-tuned on these ratings, separates the genders as well (centroid gap 0.90, Figure 8). In Simpson, trained without any gender label, this separation lies along the first principal component of the speaker space (57% of the variance, correlation with gender 0.96), which we call the gender direction.

Gender direction and similarity ranking. We have checked this relationship on the different-speaker pairs of VoxSim test (SRCC). We either removed the gender direction from Simpson's embeddings or scored each pair by the product of the two embeddings' components along that direction.

Similarity computed fromAll different-speaker pairsSame-gender different-speaker pairs
Full embedding (Simpson)0.7850.49
Gender direction removed0.110.38
Gender direction only0.760.26
WavLM(Large)-ECAPA, full embedding (Table 19)0.0610.115

Within the same gender, the SRCC falls from 0.49 to 0.38 after the gender direction is removed and remains 0.26 with the gender direction alone, whereas a binary male/female label would give all such pairs the same score. These results show that Simpson captures both a strong gender-associated component and perceptual ranking within gender groups.

Controlling the gender direction. Prompted by the reviewer's question, we also tested whether the influence of gender can be changed after training. To adjust the contribution of this gender-associated direction, we scaled the component of every embedding along it by a factor γ\gamma and left the rest unchanged. The direction is the first principal component, so no gender label is needed; a direction estimated from 35 other VoxCeleb1 speakers gives similar results (cosine 0.99). We report the SRCC on each test set.

γ\gammaVariance along the gender directionVoxSim test (full / diff / same)Same-gender different-speaker pairsVCC2018VCC2020TTSDS2
1 (Simpson)57%0.822 / 0.785 / 0.6170.490.5470.7390.415
0.527%0.806 / 0.597 / 0.6070.440.5200.7320.400
0.3515%0.789 / 0.404 / 0.6010.410.4980.7280.395
00%0.754 / 0.111 / 0.5930.380.4580.7220.388

As γ\gamma decreases to 0, the largest decrease occurs in the VoxSim different-speaker SRCC, consistent with the strong association between gender and human ratings. We have added these experimental results and analyses to Appendix C.13.


We thank the reviewer once more for the careful reading and the questions, which have made our paper clearer and stronger. In particular, the new result above came from the reviewer's excellent question on the gender direction. We hope our responses have addressed the reviewer's concerns and would welcome any further questions.

We thank the reviewer again for the careful review and insightful questions. We have revised the paper following the reviewer’s comments, with all changes shown in red. In particular, we have added the rating criteria and the inter-rater agreement of VoxSim to Appendix A.2 and the analysis of the gender direction to Appendix C.13. As the discussion period ends soon, we would sincerely appreciate any feedback on whether our responses have addressed the reviewer’s concerns, and we would be happy to answer any remaining questions.

Reviewer_3kzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a speaker embedding model suitable for ranking the perceptual similarity of speakers. By analyzing the properties of the speaker verification loss—which promotes the uniform spreading of embeddings on a hypersphere—the authors argue that verification-trained embedders are fundamentally suboptimal for evaluating speaker similarity. To address this, the proposed model (Simpson) integrates a ranking loss based on proxy similarity rankings extracted via a self-supervised student-teacher framework. The model can be optionally fine-tuned using Direct Preference Optimization (DPO) with human labels to further align with actual perceptual similarity. Experimental results verify that Simpson outperforms baselines trained on verification objectives and matches the performance of supervised models across natural speech, voice conversion, and TTS evaluations.

Critical strengths
  • The motivation for adopting rank-based objectives is well-justified through a geometric analysis of speaker verification losses, showing how they enforce hyperspherical uniformity and collapse intra-speaker variations.

  • The effectiveness of the proposed rank-based objective is rigorously validated through both objective metrics and insightful visualizations comparing embedding geometries.

  • A key conceptual contribution is demonstrating that label-free proxy ranking (using frozen backbones and EMA teachers) achieves the vast majority of performance gains, reducing reliance on scarce human similarity labels.

Critical weaknesses
  • Asymmetry in Proxy Similarities (Section 3.3)\textbf{Asymmetry in Proxy Similarities (Section 3.3)}: While ablation (Table 11) empirically justifies using utterance-level proxies for same-speaker pairs and speaker-centroid proxies for different-speaker pairs, a more thorough analytical justification for this asymmetry would strengthen the methodology section.

  • Architectural Ambiguity\textbf{Architectural Ambiguity}: The term “backbone” requires a slightly more explicit definition regarding the exact boundary between the truncated WavLM layers, LoRA adaptations, and downstream readout modules.

  • Loss Term Mechanics (Section 3.3)\textbf{Loss Term Mechanics (Section 3.3)}: The claim that the different-speaker term prevents even spreading and the same-speaker term prevents collapse warrants a deeper analytical or mathematical linking to the verification optimum geometry described in Section 2.3.

  • Interpretation of Baselines (Table 1)\textbf{Interpretation of Baselines (Table 1)}: While the "hypothetical perfect verifier" illustrates how much correlation stems purely from inter-class separation, its inclusion could be better framed to explicitly highlight the limitation of standard verification EER metrics.

Questions for authors
  • Architectural Exploration of Layer Aggregation\textbf{Architectural Exploration of Layer Aggregation}: The paper notes that pre-IN statistics are gathered from layers 0–3, while ECAPA takes the output of the final truncated layer. Given the layer-wise correlation drop shown in Figure 6, how does performance change if a learnable weighted sum of all layers (including deeper ones) is fed into the ECAPA head, and how does this compare to raw WavLM layer outputs?

  • Training Stability\textbf{Training Stability}: Given the reliance on an EMA teacher and self-generated proxy similarities, how sensitive is the training stability to the choice of the warmup schedule for proxy weights (alpha_d and alpha_s) defined in Table 6?

4 comments

We thank the reviewer for the thoughtful review and helpful comments. We have uploaded a revised paper, with all changes shown in red, and respond to each comment below.

[W1] Asymmetry in Proxy Similarities (Section 3.3).

Thank you for this constructive suggestion. The two ranking terms require different information. Same-speaker ranking must distinguish utterance pairs from the same speaker. A shared speaker centroid would provide no ordering among these pairs, so we use utterance-level proxies for this term.

For different-speaker pairs, we use speaker centroids to reduce dependence on individual utterances. On VoxCeleb2 dev, we computed two sets of frozen-backbone similarity scores for the same set of speaker pairs using disjoint recordings. The correlation between these estimates increases from 0.42 with single utterances to 0.81 with four-utterance centroids. The target ablation provides complementary evidence: replacing centroid-level different-speaker proxies with utterance-level proxies decreases the VoxSim different-speaker SRCC from 0.785 to 0.380 (Table 13; Table 11 in the submission).

[W2] Architectural Ambiguity.

Thank you for pointing this out, and we apologize for the ambiguity. We have clarified the component boundaries in Section 3.1. The student backbone consists of the WavLM-Large CNN feature encoder and its first three transformer layers. The pretrained weights remain frozen; only the LoRA parameters are trainable within the backbone.

The trainable readout comprises the ECAPA head, which receives the instance-normalized output of the third transformer layer, and the statistics side path, which projects the pre-normalization means and standard deviations from layers 0–3. Layer 0 denotes the CNN output.

[W3] Loss Term Mechanics (Section 3.3).

Thank you for this insightful suggestion. A deeper analytical link between the two ranking terms and the verification optimum geometry of Section 2.3 would make the motivation of our loss design clearer. We will investigate this link further.

[W4] Interpretation of Baselines (Table 1).

We are grateful for the reviewer's valuable suggestion, as the limitation of EER is the central message of our paper. The hypothetical perfect verifier in Table 1 assigns one constant score to same-speaker pairs and a lower constant score to different-speaker pairs. It achieves an EER of 0% but provides no ranking within either set. The full SRCC of 0.698 comes only from separating the two sets. We have clarified this in Section 4.2: an embedder can reach a perfect EER without ordering any pair within either set.

[Q1] Architectural Exploration of Layer Aggregation.

Thank you for this question. We trained two variants that feed a learnable softmax-weighted sum of layers to the ECAPA head, with everything else unchanged. The first combines layers 0–3 of the truncated backbone, and the second combines all 25 hidden states of the untruncated WavLM-Large. For the two variants, we report VoxSim test SRCC (full / different speaker / same speaker) at the same epoch as Simpson, averaged over five seeds.

VoxSim test SRCC (full / diff / same)
Frozen WavLM, layer 3 (raw output)0.710 / 0.417 / 0.585
Frozen WavLM, last layer (raw output)0.57 / 0.23 / 0.36
Simpson (layer 3)0.822 / 0.785 / 0.617
Weighted sum of layers 0–30.821 / 0.779 / 0.613
Weighted sum of all layers0.818 / 0.771 / 0.618

With layers 0–3, the weights concentrate on layer 3 (0.95), and the performance matches Simpson. With all layers, the full SRCC stays similar, but the different-speaker SRCC drops to 0.771 and the VCC2020 SRCC drops from 0.739 to 0.718; the backbone also grows from 51M to 317M parameters. Deeper layers therefore add cost without improving the ranking. We have added these results to Appendix C.12.

[Q2] Training Stability.

We appreciate the reviewer's attention to training stability. We evaluated sensitivity to the proxy-weight schedule using three alternatives and compared them with the original schedule of Table 6.

Warmup / ramp of αd\alpha_{\mathrm{d}}, αs\alpha_{\mathrm{s}} (steps)VoxSim test SRCC (full / diff / same)VCC2020 SRCCTTSDS2 SRCC
None0.825 / 0.783 / 0.6190.7090.355
4k / 16k0.836 / 0.787 / 0.6500.7230.413
8k / 16k (Table 6, Simpson)0.822 / 0.785 / 0.6170.7390.415
8k / 32k0.834 / 0.785 / 0.6420.7270.417

With a warmup, the different-speaker SRCC varies by at most 0.002 across schedules, and the TTSDS2 SRCC by at most 0.004. Without a warmup, the different-speaker SRCC stays similar, but the SRCCs on synthetic speech drop. We believe the warmup is needed because early student-derived targets may be less reliable than the frozen-backbone targets. Mixing them from the beginning would make early training less stable.

Thank you for the thorough and constructive review. The reviewer's suggestions have helped us clarify the method and strengthen the experimental evaluation. We hope our responses have resolved the reviewer's concerns and would be happy to answer any further questions.

Thank you for clearly addressing the points I raised. I have no further questions regarding these items, and I look forward to seeing the analysis results for [W3] in your future work. Accordingly, I will maintain my original score.

We sincerely thank the reviewer for the prompt response and for the valuable comments throughout the review process. We will further investigate the link between the ranking terms and the verification optimum.

Reviewer_JxDmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem where embedding-based speaker similarity has repeatedly been reported to diverge from human ratings of speaker similarity. For example, an embedder with a lower verification equal error rate (EER) can show a lower correlation with human similarity ratings than an embedder with a higher EER.

The authors first show that high full-set correlations on VoxSim are largely driven by separating same-speaker from different-speaker pairs, whereas correlation on different-speaker pairs alone is near zero for standard verification embedders. They argue that the optimal geometry of the verification objective, where utterances of the same speaker collapse to a point and different speakers spread evenly on the hypersphere, is invariant to perceptual similarity ranking. Motivated by this analysis, the authors propose Simpson, a speaker embedder that augments the verification loss with a pairwise ranking loss to order utterance pairs by perceptual similarity without human labels. Its ranking targets come from a frozen self-supervised backbone combined with Simpson’s own exponential moving average (EMA) teacher, followed by an optional preference stage using a small set of human labels. Across human-rated test sets of natural, converted, and synthetic speech, Simpson achieves the strongest correlation with human judgments among evaluated public embedders (reaching 0.79 SRCC on VoxSim different-speaker pairs), and the authors validate it downstream as a TTS evaluation metric and as a reward model for GRPO.

Critical strengths
  1. The paper provides rigorous diagnostic evaluation results that are clear and practical. Splitting evaluation by pair type, incorporating the perfect verifier as a baseline, and running the same-gender (gender-controlled) analysis provide strong evidence that standard SIM metrics rank voices of different speakers poorly. This conclusion holds even when a weakened version of the theory is considered, and thus should influence the use of speaker embedders in the TTS and RL communities. This is the primary motivation for accepting this paper.

  2. The proposed solution Simpson's performance is backed by large and consistent gains without labels, and validated on downstream tasks. Simpson provides improvements to different-speaker SRCC by around 0.7 on VoxSim, having narrow CIs with low variance between random seeds (±0.005). It also improves SRCC and LCC on all subsets of VCC, even cross-lingual ones not seen during training. GRPO experiments show reduced length-based reward hacking by reducing the length gap from 1.51 s to 0.92 s and over-length output generation from 13.3% to 6.1% without a length penalty.

Critical weaknesses

Theory and causal statements are overstated. The paper asserts that the verification objective causes the failure and that its optimal geometry precludes perceptual ranking. However, the propositions only indicate that the objective is indifferent to perceptual ordering, and achievement of the Prop. 1 bound at the optimum is assumed rather than proven. Furthermore, the scale invariance of Spearman's measure refutes the 1/√d concentration argument. The paper's own baselines (Fast ResNet-34 at 0.452, InfoNCE-only Simpson at 0.462) also suggest that the objective is not all there is to it. Because ICLR considers the theory to be a major contribution, this gap is important.

Vague label-free training protocol and missing baselines that would isolate the gain. If the truncation layer, proxy model, and loss weights were selected with VoxSim test labels, the headline test numbers are overly optimistic, and the phrase "no human labels" is misleading. The main tables also lack the most straightforward baseline approaches: a) pre-trained WavLM-Large layer-3 cosine (on utterance- and centroid-level) for all experiments; b) post-hoc evaluation metrics applied to pre-trained verification embeddings (for example, whitening, LDA/PLDA, or a bilinear metric trained on the same proxy targets); c) existing similarity prediction approaches such as SVSNet.

In their absence, it is not possible to measure how much of the gain in Simpson's method comes from the ranking criterion and self-distillation rather than from distilling structure already present in the low layers of WavLM, or from the fact that verification embeddings include the information and only cosine scoring obscures it.

Questions for authors
  1. From which split were Fig. 6 and Table 11 derived? Were the truncation layer, the choice of centroid vs. utterance proxy, λ_d/λ_s, the α schedule, and the choice of checkpoints selected based on VoxSim test labels? If yes, could you please share the results with all those parameters chosen on the VoxSim train split or held-out data?

  2. Do you have a proof, or a citation of one, that CE/AAM-softmax/InfoNCE with N ≫ d achieves the Prop. 1 bound at the optimum? Also, since Spearman correlation is invariant to scale, why couldn't a tight frame still encode a perceptual ordering within small cosine variations when the cosine RMS is ≈ 1/√d?

  3. Please share frozen WavLM-L layer-3 cosine (utterance and centroid) on VoxSim, VCC, and TTSDS2. Please also share the ablative result where only frozen-backbone proxies are used (α_d = α_s = 0) to isolate the contributions of the EMA teacher and student centroid bank.

  4. Can a learned linear or bilinear metric on frozen ECAPA / WavLM-ECAPA embeddings, trained with your proxy targets or the DPO label budget, recover different-speaker ranking? This test checks whether the information is lost or merely not expressed in cosine.

4 comments

We sincerely thank the reviewer for the thorough and constructive review. The comments have led to new baselines and ablations and to a more precise statement of our theory. We have uploaded a revised paper, with changes presented in red.

[W1] Theory and causal statements are overstated.

Thank you for this careful reading. We agree that our theoretical analysis does not directly establish that verification training necessarily removes perceptual ordering. We have revised the expressions in the abstract and Section 2.3 to clarify the scope of our geometric analysis; as the reviewer pointed out, the verification objective is indifferent to a perceptual ordering, and this indifference does not imply that verification-trained models cannot preserve the perceptual order.

We’d like to note that our ablation empirically supports the benefit of explicit ranking losses. Adding the ranking objective increases different-speaker SRCC from 0.462 to 0.785, supporting the role of the training objective in shaping alignment with human perceptual judgments.

[W2, Q2] Attainment of the Prop. 1 bound and the scale invariance of Spearman's measure.

Thank you for raising these important points.

Attainment of the bound. We would like to clarify that our paper does not claim that CE, AAM-softmax, or InfoNCE provably attains the bound when N≫dN \gg d. Proposition 1 gives a lower bound on the mean squared cosine and characterizes its equality condition; Corollary 1 is conditional on that equality. We have revised the paper to describe our theoretical analysis as an idealized geometric reference, rather than a proven optimum of the verification losses. Figure 3 illustrates this geometry in a simplified setting. For trained verification embedders, the centroid cosine RMS values lie at 1.16 to 1.87× the bound (Figure 5 and Table 10). These empirical observations characterize their proximity to the bound without establishing exact attainment.

Scale invariance of Spearman's correlation. We agree that small cosine variations can represent the perceptual ordering. Proposition 1 provides a lower bound on the mean squared cosine, but because Spearman correlation is invariant to positive rescaling, cosine concentration alone does not imply low SRCC. The central point of our analysis is that the verification objective does not explicitly enforce perceptual ordering, regardless of the scale of the cosine similarities. We thank the reviewer for pointing this out, and we have made Section 2.3 more precise in response.

[W3, Q1] Vague label-free training protocol.

Thank you for raising this methodological point. The VoxSim test split was not used for any model-selection decision. We used the VoxSim training split, and this split shares no speaker or utterance with the test split. On the training split, the different-speaker correlation of a frozen WavLM-Large peaks at layer 3 (0.423, against 0.414 at layer 2 and 0.401 at layer 4), supporting our truncation choice. Figure 6 and Table 13 (Table 11 in the submission) report the VoxSim test split results for comparison with the main results. We have clarified this protocol in Appendices A.5 and B.1 and marked the split in the captions of Figure 6 and Table 13.

The term “label-free” refers specifically to the absence of human similarity labels as supervision in the base Simpson model’s training objective. As mentioned in the manuscript, the optional DPO stage additionally uses these labels for preference-based refinement. The test split is reserved exclusively for evaluation.

[W4, Q3] Isolating the gain: frozen WavLM-L layer-3 cosine and frozen-proxy-only training.

Thank you for this valuable suggestion.

VoxSim (full / diff / same)VCC2018VCC2020TTSDS2
Frozen WavLM-L layer 3, utterance-level0.710 / 0.417 / 0.5850.3530.6100.414
Frozen WavLM-L layer 3, centroid-level– / 0.718 / ––––
Simpson0.822 / 0.785 / 0.6170.5470.7390.415

We report centroid-level results for different-speaker VoxSim pairs. Same-speaker pairs share the same centroid and therefore yield constant scores, making their correlation undefined. Simpson improves over frozen utterance-level features on VoxSim and both VCC benchmarks. We have added this baseline to Appendix B.1.

Training with frozen-backbone proxies only. We compared training with frozen proxies alone against training with mixed proxies that incorporate the student centroid bank and EMA teacher. The main advantage of the student centroid bank and the EMA teacher appears for the realistic case in which each speaker has only a few recordings and the frozen-backbone centroids are therefore unreliable. To examine this effect, we built the frozen different-speaker centroids from one, two, or four utterances per speaker and trained Simpson with and without the two components.

Utterances per centroidαd\alpha_{\mathrm{d}} / αs\alpha_{\mathrm{s}}VoxSim (full / diff / same)VCC2018VCC2020TTSDS2
10 / 00.677 / 0.554 / 0.4450.4270.5610.039
10.5 / 0.80.824 / 0.791 / 0.6180.5400.7140.367
20 / 00.744 / 0.614 / 0.4810.4070.4520.121
20.5 / 0.80.827 / 0.787 / 0.6250.5390.7460.428
40 / 00.827 / 0.783 / 0.6250.5370.7240.415
40.5 / 0.80.828 / 0.792 / 0.6270.5450.7460.422

With one or two utterances per speaker, mixed proxies improve all six reported correlations. With four utterances, mixed proxies still improve all six correlations, while the additional gains are smaller. These results suggest that the student centroid bank and the EMA teacher reduce sensitivity to the number of utterances used to estimate the frozen centroids and make training more robust.

[W5, Q4] Post-hoc metrics on pre-trained verification embeddings.

Thank you for suggesting this comparison. We evaluated whether post-hoc metrics can recover perceptual ranking from frozen verification embeddings. We applied each scoring method to the released ECAPA-TDNN and WavLM(Large)-ECAPA embeddings. Whitening, LDA, and PLDA were fitted on VoxCeleb2 dev, the training data of Simpson. The linear metric cos⁡(Ax,Ay)\cos(Ax, Ay) and the bilinear metric x⊤Wyx^\top W y were trained with our ranking loss, either on our proxy targets or on the VoxSim training labels. We report VoxSim test SRCC (full / different speaker / same speaker).

ECAPA-TDNN (full / diff / same)WavLM(Large)-ECAPA (full / diff / same)
Cosine (as released)0.751 / −0.010 / 0.5270.736 / 0.061 / 0.472
Whitening0.750 / −0.037 / 0.5240.755 / 0.109 / 0.517
LDA (150 dims)0.749 / −0.026 / 0.5170.744 / 0.063 / 0.487
PLDA0.746 / −0.005 / 0.5050.745 / 0.078 / 0.488
Bilinear metric, proxy targets0.733 / 0.032 / 0.4710.736 / 0.080 / 0.468
Linear metric, proxy targets0.757 / 0.030 / 0.5370.748 / 0.116 / 0.497
Linear metric, VoxSim train labels0.755 / 0.220 / 0.5050.750 / 0.347 / 0.470
Bilinear metric, VoxSim train labels0.713 / 0.076 / 0.4060.700 / 0.210 / 0.360
Simpson0.822 / 0.785 / 0.617

The tested methods do not close the different-speaker ranking gap. Whitening, LDA, and PLDA leave it near zero, and our proxy targets raise it to at most 0.116. Even the human-supervised metric learning (DPO label budget) improves the best result to 0.347, against 0.785 for Simpson. These results suggest that the performance gap with Simpson cannot be explained by cosine scoring alone. We have added this table to Appendix B.8.

[W6] Existing similarity prediction approaches such as SVSNet.

We appreciate this suggestion. We’d like to first note that no trained SVSNet checkpoint is publicly available. For the fairest comparison, we cite the SVSNet results of the VoxSim paper [1], where SVSNet is trained on the VoxSim training set, the same data set that our DPO stage uses.

ModelHuman labelsVoxSim test: SRCC / LCC / MSE / ACCVCC2018 test: LCC
SVSNet (VoxSim paper)VCC2018 train–0.575
SVSNet (VoxSim paper)VoxSim train0.753 / 0.758 / 1.384 / 39.70.509
Simpsonnone0.822 / 0.827 / 0.998 / 46.60.555
Simpson + DPOVoxSim train0.833 / 0.838 / 0.923 / 48.10.575

Using the same data set, Simpson + DPO surpasses SVSNet on both test sets. On VCC2018, SVSNet trained on VoxSim falls to an LCC of 0.509, whereas SVSNet trained on VCC2018 reaches 0.575. Simpson + DPO matches SVSNet trained on VCC2018 without any VCC2018 label. We have added this comparison to Appendix C.10.

[1] VoxSim: A perceptual voice similarity dataset


We are grateful to the reviewer for the time and care taken with our paper, and for the concrete suggestions that improved it. We hope our responses and the new baselines have resolved the reviewer's concerns. We would be glad to discuss any remaining questions.

We thank the reviewer again for the thorough and constructive review. We have revised the paper following the reviewer’s comments, with all changes shown in red. In particular, we have revised the theoretical and causal statements in the abstract and Section 2.3, and added the frozen WavLM-L baseline, the post-hoc back-ends, and the comparison with SVSNet to Appendices B.1, B.8, and C.10, respectively. As the discussion period ends soon, we would sincerely appreciate any feedback on whether our responses have addressed the reviewer’s concerns, and we would be happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.