When Matching Is Not Selection: Learning Criterion-Dependent Similarity in Speech Representations

Speech carries multiple attributes such as speaker identity, emotion, and linguistic content, and different retrieval tasks require judging similarity based on different attributes. Existing instruction-conditioned methods can already condition speech representations or retrieval behavior on different criteria, but correct matching on the target attribute does not mean that candidate ranking is still determined by the current criterion when attributes compete. This paper distinguishes attribute matching from attribute selection, and reveals a consistent matching-selection gap across multiple instruction-conditioned representation models through attribute competition and criterion reversal evaluations. Conventional criterion-wise training does not explicitly organize the joint relevance status of the same candidate across criteria. We propose Joint-State Relevance Learning (JSRL), which structures training supervision using the four joint relevance states of a candidate under a criterion pair and performs criterion-wise ranking over the resulting relevance partitions, while allowing all criteria to share a single candidate index. On three corpora not used in training, JSRL outperforms the baselines on attribute-conflict ranking, criterion reversal, and multi-candidate retrieval. The learned criterion-dependent ranking behaviors can also be recombined for the speaker-content pair, which is not used in criterion-pair training, and transfer to the speaker/style instruction retrieval tasks constructed in INSPIRE Synthetic. These results show that organizing supervision around joint relevance states can improve how reliably speech similarity follows the current criterion.