Beyond Speaker Verification: Speaker Embeddings for Perceptual Similarity

Speaker embedders are trained to verify whether two utterances come from the same speaker. However, the same embedders increasingly serve as evaluation metrics, reward models, and data filters for a different purpose: estimating how similar two voices sound to human listeners. We first show that verification-trained embedders do not reliably preserve perceptual similarity rankings, and that this limitation stems from the verification objective rather than from any particular model. For a speaker embedder widely used in text-to-speech (TTS) evaluation, scores on different-speaker pairs exhibit almost no correlation with human ratings. Our theoretical analysis examines the optimal embedding geometry under the verification objective, which collapses intra-speaker variation and spreads speakers uniformly. We prove that this geometry is invariant to inter-speaker perceptual similarity rankings and show empirically that existing speaker embedders closely approach it. Building on this analysis, we propose Simpson, a speaker embedder that measures Similarity like a person. Simpson augments the verification objective with a ranking objective to learn perceptual similarity rankings without human similarity labels. The ordering targets are derived from a frozen self-supervised backbone and Simpson's own exponential moving average (EMA) teacher. An optional preference stage further refines the model using a small set of human similarity labels. Across natural, converted, and synthetic speech, Simpson achieves the highest correlation with human ratings among the publicly available embedders evaluated, reaching a Spearman rank correlation of 0.79 on different-speaker pairs where verification-trained embedders show near-zero correlation. We further demonstrate Simpson's practical effectiveness in downstream tasks, both as an evaluation metric for TTS and as a reward function for group relative policy optimization (GRPO).