acceptodds
Under review as a conference paper at ICLR 2027

When Matching Is Not Selection: Learning Criterion-Dependent Similarity in Speech Representations

Abstract

Speech carries multiple attributes such as speaker identity, emotion, and linguistic content, and different retrieval tasks require judging similarity based on different attributes. Existing instruction-conditioned methods can already condition speech representations or retrieval behavior on different criteria, but correct matching on the target attribute does not mean that candidate ranking is still determined by the current criterion when attributes compete. This paper distinguishes attribute matching from attribute selection, and reveals a consistent matching-selection gap across multiple instruction-conditioned representation models through attribute competition and criterion reversal evaluations. Conventional criterion-wise training does not explicitly organize the joint relevance status of the same candidate across criteria. We propose Joint-State Relevance Learning (JSRL), which structures training supervision using the four joint relevance states of a candidate under a criterion pair and performs criterion-wise ranking over the resulting relevance partitions, while allowing all criteria to share a single candidate index. On three corpora not used in training, JSRL outperforms the baselines on attribute-conflict ranking, criterion reversal, and multi-candidate retrieval. The learned criterion-dependent ranking behaviors can also be recombined for the speaker-content pair, which is not used in criterion-pair training, and transfer to the speaker/style instruction retrieval tasks constructed in INSPIRE Synthetic. These results show that organizing supervision around joint relevance states can improve how reliably speech similarity follows the current criterion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Selective Speech Representation, Instruction-Aware Speech Retrieval, Attribute Selection, Attribute Competition, Criterion-Dependent Similarity Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w3awOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a fine-tuning technique (Joint-State Relevance Learning - JSRL) to help speech retrieval models (models trained to match a pair of utterances based on a given criterion) to be less affected by co-occurent similarities in the given utterances. Tested on MOSS-Audio-4B-Instruct, compared to the original implementation as well as a set of Speech-aware LLMs and SSL models, on diverse set of Speech Emotion Recognition datasets, they show that most retrieval models choose by similarity, thus being easily confunded by co-occuring properties of a speech utterance. Their implementation show almost no lasting impact by co-occuring proprties of the speech.

Critical strengths

The augmentation technique is quite straightforward, well detailed, with obvious results.

Critical weaknesses

None

Questions for authors

With the test set being a composite of 2 acted and 1 naturalistic speech emotion recognition datasets, some differences mayb be expected in behaviour. Could you add details on the balancing operations that were performed between those sets, or provide dataset-specific results?

2 comments

We sincerely thank you for your time and positive assessment. We have uploaded a revised PDF, with text changes highlighted in blue.

Q: Balancing across datasets and dataset-specific results.

We balance the three datasets at the aggregation stage. Each metric is first computed within each dataset, weighting all queries equally, and then averaged with equal weight over the three datasets, so each dataset contributes equally to the aggregate regardless of its size (App. A.7). On individual datasets, absolute scores differ, but JSRL achieves the highest Conflict and SwitchExact scores for all three criterion pairs on every dataset (Appendix Tables 17–19). In multi-candidate retrieval, JSRL also achieves the highest mAP, NDCG@5, and Recall@5 on all three datasets, outperforming the strongest baseline, WAVE (Dynamic), in mAP by 1.6 (CREMA-D), 7.6 (MSP-IMPROV), and 3.2 (RAVDESS) percentage points. These results have been added to Appendix E.2 (Table 20).

Reviewer_w3awOpenReview ↗

Thank you for the per-dataset additions

I aknowledge the per-dataset additions in the appendix, which ease the issue with the blind dataset mixing during the evaluation, and thank the authors for their answer.

Reviewer_vjDaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies instruction-conditioned similarity learning for speech representations and distinguishes attribute matching from attribute selection. The authors examine the matching–selection gap in existing models through three attribute-competition conditions—Aligned, Balanced, and Conflict—and the SwitchExact metric, which requires the same candidate pair to be ranked correctly in opposite directions when the retrieval criterion changes. To address this gap, the authors propose Joint-State Relevance Learning (JSRL), which organizes candidates into four joint states according to their relevance under two retrieval criteria. This structure provides ranking supervision covering both relevance switching and relevance persistence. On three corpora not used in the paper’s retrieval training, JSRL improves attribute-conflict ranking, criterion reversal, and multi-candidate retrieval. The authors also evaluate the untrained speaker–content criterion pairing and transfer to INSPIRE Synthetic, and provide training-budget-matched ablations, multiple random seeds, and cluster-bootstrap analyses.

Critical strengths
  1. The problem formulation is valuable, and the evaluation design reveals capability differences that conventional matching metrics can overlook. Through attribute conflicts and bidirectional ranking requirements on the same candidate pairs, the authors distinguish correct target-attribute matching from ranking genuinely governed by the target criterion. This provides a useful perspective for evaluating instruction-following behavior in speech retrieval.
  2. The empirical evidence is reasonably comprehensive. The authors evaluate the method on multiple held-out corpora and provide a training-budget-matched continuation baseline, ablations of relevance switching and persistence, multiple random seeds, and paired statistical analyses. These evaluations strengthen the credibility of the main experimental conclusions.
Critical weaknesses
  1. The existing controls do not sufficiently isolate the contribution of four-state supervision organization itself. Because the training branches differ in their objectives and coverage of supervised relationships, it remains difficult to determine whether the gains arise from the four-state organization or from more extensive positive–negative comparisons and differences in their weighting. The external baselines also do not receive task-specific adaptation using the same data and training budget. Clarifying these controls and comparing conventional criterion-wise sampling with four-state stratified sampling under a fixed loss, data budget, and positive/negative counts would strengthen the attribution of the gains.
  2. The evaluation primarily covers a limited set of discrete attributes and controlled candidate constructions, leaving the scope of generalization uncertain. Content relevance is defined by sentence identity, while multi-candidate retrieval and its pool-size extensions maintain fixed joint-state proportions. Both criteria in the untrained speaker–content pairing have already been learned, and the INSPIRE evaluation covers only two single-attribute tasks closely related to the training criteria. The results therefore support improvements in structured speaker/emotion/content retrieval, but provide more limited evidence for broader instruction-conditioned similarity learning, which limits the overall significance of the work.
Questions for authors
  1. Do Stage 2 continuation, the switching-only variant, and full JSRL use exactly the same training units and candidate compositions? Please clarify their respective losses, sample counts for each joint state, effective numbers of positive–negative comparisons, and comparison weights. Do you have results that hold the loss formulation and comparison budget fixed while varying only conventional sampling versus four-state organization?
  2. How do Stage 2 and Stage 2 continuation perform on the two INSPIRE tasks? These results would help determine whether the transfer gains primarily arise from adaptation to the three retrieval attributes or specifically from Stage 3 joint-state supervision.
  3. Please provide the number of training steps or epochs for each stage, the temperature parameter (\tau), dataset and attribute sampling proportions, candidate counts for each joint state, handling of insufficient candidates, and checkpoint-selection rules. In particular, how are content labels obtained and the third-attribute mismatch constraint implemented for each training corpus?
  4. Does JSRL retain its advantage over the strongest baseline when the joint-state proportions change or when candidates are evaluated without fixed-proportion sampling on the existing test corpora? This would more directly assess the dependence of the current conclusions on candidate construction than adding new tasks.
3 comments

We sincerely thank you for your careful review and detailed suggestions. We have uploaded a revised PDF, with text changes highlighted in blue.


W1 & Q1: Attribution of the gains

With the loss, positive/negative counts, and number of comparisons held fixed, four-state stratified sampling outperforms conventional criterion-wise sampling (CW) on criterion reversal, recombination on the untrained criterion pair, and multi-candidate retrieval:

MetricJSRL − CW [95% CI]
S–E SwitchExact+2.52 [2.09, 2.95]
E–C SwitchExact+2.99 [2.84, 3.13]
S–C SwitchExact (untrained pair)+2.13 [1.80, 2.59]
S/E/C mAP+2.12 [1.96, 2.25]

For each criterion, CW independently samples 4 positives and 4 negatives uniformly from the relevant and irrelevant sets, whereas JSRL samples candidates stratified by the four joint states and shares them between the two criteria. Both use the same corpora, criterion sampling schedule, and number of training steps. Because CW samples independently for each criterion, matching the number of comparisons (as requested in your Q1) means that CW encodes twice as many candidates per unit as JSRL. The table reports seed 0, with CIs from a paired cluster bootstrap. Across three training seeds, all four differences are positive, with cross-seed standard deviations of 0.29–0.58 (Appendix E.4, Table 23).

JSRL and the switching-only variant (denoted JSRL w/o persistence in the paper) use the same joint-state units (8 candidates per unit, about 2 per state) and the same loss formulation and weighting scheme (Eq. 4). The only difference is that the latter masks comparisons involving D11 and D00. Per unit and criterion, JSRL therefore has 16 comparisons and the switching-only variant about 4.4–4.9. Stage 2 continuation instead keeps the attribute-competition units and hard-negative InfoNCE of Stage 2, with 127–158 comparisons. We have revised Sec. 4.4 to note the difference in comparison counts between JSRL and the switching-only variant and to point to the CW control above. The per-state sample counts, numbers of comparisons, and weights of each branch are given in Appendix Table 24.

The external baselines are evaluated with their public weights, primarily to characterize the matching–selection gap of existing instruction-conditioned models (Sec. 4.2). The contribution of the training scheme itself is measured by Stage 2 continuation and CW, which use the same backbone, data, and number of training steps as JSRL. The Limitations section states this.


W2 & Q4: Candidate construction and scope of generalization

Without fixed-proportion sampling, JSRL retains its advantage over the strongest baseline, WAVE (Dynamic). We performed full-corpus ranking on the three existing test corpora, in which each query is ranked against all other utterances in the same corpus, so the proportions of the joint states vary naturally across queries. JSRL exceeds WAVE (Dynamic) in S/E/C mAP by 3.45 [2.70, 4.22], and the difference is positive on each of the three corpora (+2.49 to +5.17). The protocol and results are reported in Appendix E.5 (Table 25). In addition, the attribute competition and criterion reversal evaluations themselves use all candidates in each joint state (App. A.2) and do not depend on fixed proportions. JSRL achieves the highest Conflict and SwitchExact scores in these evaluations (Sec. 4.2).

Regarding the scope of evaluation, our work studies whether retrieval ranking can be determined by the current criterion when multiple attributes coexist. To obtain controlled and interpretable evidence, both the method and the experiments focus on speaker, emotion, and content, three attributes with discrete labels (Sec. 3.1). Content relevance is defined by sentence identity (Sec. 4.1), and under this definition the joint states can be assigned unambiguously. The S–C pair tests whether the learned criteria can be recombined in a pairing absent from criterion-pair training (Sec. 4.5). INSPIRE tests the transfer of the learned criteria to speaker and speaking-style retrieval under independently constructed data, candidate pools, and instructions (Sec. 4.6). We have stated this scope in the final sentence of the abstract (adding “in speaker, emotion, and content retrieval”) and described it further in the new Limitations section.

Q2: Stage-wise results on INSPIRE

Both sources you mention contribute to the transfer gains. Training on the three attributes (Stages 1–2) yields part of the transfer, and Stage 3 brings a further improvement on top of this. The CW control from W1 and Stage 2 continuation, both trained for the same number of steps as JSRL, each produce small changes, which indicates that the four-state organization makes a clear contribution to this further improvement. The results are as follows (Recall, %):

ModelSpeaker R@10R@50R@100Style R@10R@50R@100
Stage 27.7512.7517.006.5012.2515.25
Stage 2 continuation7.7513.5017.757.2512.7516.00
CW7.5014.0018.007.2512.5016.50
JSRL11.7520.0025.2510.7517.5022.25

Relative to Stage 2, Stage 2 continuation and CW change each Recall value by at most 1.25, whereas JSRL exceeds both by 3.50–7.50. We have added the first three rows to Table 15.


Q3: Training details

The training settings for each stage are summarized below:

SettingStage 1Stage 2Stage 3 (JSRL)
Training steps500600700
Temperature τ0.010.070.07
Criterion and data samplingC : S : E = 3 : 4 : 6; C entirely from ESD; E equally from ESD and MSP-Podcast; S from ESD, EARS, LibriTTS-R, and MSP-PodcastS–E : E–C = 1 : 1, with 8 competition units per pair at each stepSingle-criterion and joint-state steps alternate at 1 : 1; single-criterion steps sample S/E/C equally; joint-state steps use S–E : E–C = 1 : 1, with S–E units drawn equally from ESD and MSP-Podcast and E–C units entirely from ESD

Candidate counts and checkpoint selection. Each Stage 3 joint-state unit contains 8 candidates, with 4 positives and 4 negatives under each criterion and about 2 candidates per joint state on average. All candidates are sampled without replacement. For ESD, rejection sampling is used to meet the per-state quotas. For MSP-Podcast, only queries with enough candidates are used. All stages use the checkpoint at the end of training (Sec. 4.1).

Content labels and the third-attribute mismatch constraint. Content supervision comes only from ESD, whose labels are the sentence identities of its official parallel text. EARS and LibriTTS-R are used only for speaker supervision in Stage 1. Stage 3 joint-state units require each candidate to differ from the query in the third attribute, i.e., S–E units use candidates with different content and E–C units use candidates from different speakers. For ESD, this is determined by sentence and speaker identities. MSP-Podcast is used only in S–E units, where content identity is determined from its provided transcripts after normalization.

Training details are given in Appendix B.

Thank you again for your detailed review. In our response and the revised PDF, we added the criterion-wise sampling control with the loss, positive/negative counts, and number of comparisons held fixed (Appendix E.4), full-corpus ranking without fixed-proportion sampling (Appendix E.5), the stage-wise INSPIRE results (Table 15), and the training details (Appendix B). We would be grateful to know whether these address your concerns about the attribution of the gains and the scope of generalization. We are happy to clarify any remaining points before the discussion closes.

Reviewer_u5asOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes "attribute matching" from "attribute selection" in speech retrieval: existing instruction-conditioned methods can make representations match a target attribute, but when multiple attributes compete, candidate ranking may no longer be determined by the current criterion. Through attribute competition and criterion reversal evaluations, the authors reveal a consistent "matching–selection gap" across multiple instruction-conditioned representation models. To address this, they propose Joint-State Relevance Learning (JSRL), which structures training supervision using the four joint relevance states of a candidate under a criterion pair (relevant under both criteria, irrelevant under both, or relevant under only one of the two criteria), and performs criterion-wise ranking over the resulting relevance partitions while allowing all criteria to share a single candidate index. On three corpora not used in training, JSRL outperforms the baselines on attribute-conflict ranking, criterion reversal, and multi-candidate retrieval. The learned criterion-dependent ranking behaviors can also be recombined for the speaker–content pair, which is not used in criterion-pair training, and transfer to the speaker/style instruction retrieval tasks constructed in INSPIRE Synthetic.

Critical strengths
  1. The distinction between "matching ≠ selection" captures a real blind spot in instruction-conditioned retrieval, and the attribute-competition and criterion-reversal evaluations directly target this blind spot, offering more explanatory power than simply reporting retrieval metrics.
  2. Structuring supervision with the four joint relevance states under a criterion pair, rather than training each criterion independently, explicitly teaches the model the relative position of the same candidate under different criteria, providing a mechanistic explanation for cross-criterion-pair recombination and transfer.
Critical weaknesses
  1. The reliability of determining the four states decreases in order across speaker (discrete, clear categories), emotion (categorical but with mixed and intensity issues), and content (continuous, strongly semantics-dependent). If content-related state annotation is the noisiest, while JSRL's advantage is attributed to learning a transferable criterion-dependent ranking structure, it is difficult to rule out that part of the gain comes from fitting unstable labels.

  2. Figure 3's legend cannot be matched one-to-one with the data lines, undermining readability.

Questions for authors

For content, where relevance is continuous, how is the boundary between "relevant under only one criterion" and "relevant under both" adjudicated, and how does mislabeling affect ranking learning and cross-criterion-pair recombination?

2 comments

We sincerely thank you for your careful review and constructive comments. We have uploaded a revised PDF, with text changes highlighted in blue.


Q: How content relevance is adjudicated.

We determine content relevance by sentence identity (Sec. 4.1), treating two utterances as relevant if they correspond to the same sentence and as irrelevant otherwise. Under this definition, content labels are binary, so whether a candidate is relevant under only one criterion or under both follows directly from its labels under the two criteria, without a threshold. Content supervision in JSRL training comes from ESD and is likewise labeled by the sentence identity of each utterance's script. We have made this explicit in Appendix B.2 and restated the definition in the new Limitations section of the main text (Sec. 5).


W1 & Q: Label reliability and the effect of mislabeling.

The two analyses we added indicate that JSRL's gains are difficult to attribute to fitting incorrect content labels. In the label audit, we used ASR to transcribe all utterances that carry content supervision in JSRL training, as well as the three evaluation corpora, compared the transcriptions with the corresponding scripts, and manually reviewed the flagged utterances. The audit found no mismatch between the utterances and their scripts. As for emotion labels, we retain only those utterances in the evaluation corpora CREMA-D and MSP-IMPROV whose listener-perceived emotion agrees with the actor-intended emotion (App. A.1).

To further examine how sensitive the results are to content label errors, we also performed noise injection, swapping the relevant and irrelevant labels in about 10% and 20% of the content training units in JSRL training and retraining with all other settings unchanged. With about 10% and 20% of the units flipped, E–C SwitchExact (ranking learning) is 70.65 and 68.13, compared with 70.80 without noise, and S–C SwitchExact on the untrained pair (cross-criterion-pair recombination) is 72.80 and 70.96, compared with 72.88. Under 10% noise both metrics remain essentially unchanged, and under 20% noise they decrease by 2.67 and 1.92 percentage points, respectively. The results are insensitive to label errors at about the 10% level, so even if a small number of label errors were missed by the audit, they would not be enough to explain JSRL's gains. Both analyses are reported in Appendix E.6 (Table 26).


W2: Legend of Fig. 3.

We have redrawn Fig. 3 so that the legend entries can be matched one-to-one with the data lines. Each model has its own color and marker shape, and each model–interface configuration is listed separately in the legend. Static/Fixed interfaces are drawn with solid lines and filled markers, and Dynamic interfaces with dashed lines and open markers, as indicated in the legend. We also made the top-row panels taller, narrowed the y-axis range, and slightly offset the markers of different configurations along the x-axis to reduce overlap between curves and markers. The per-corpus Conflict and SwitchExact values underlying the figure are given in Appendix Tables 17–19.

Thank you again for your review. In our response and the revised PDF, we clarified how content relevance is adjudicated (Sec. 4.1, Appendix B.2), added a label audit and a noise-injection analysis (Appendix E.6), and redrew Fig. 3. We would be grateful to know whether these address your concerns, and we are happy to clarify any remaining points before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

We thank all three reviewers for their reviews. Reviewer u5as noted that distinguishing matching from selection captures a real blind spot in instruction-conditioned retrieval and that joint-state supervision provides a mechanistic explanation for cross-criterion-pair recombination. Reviewer vjDa found the problem formulation valuable and noted that the evaluation design reveals capability differences that conventional matching metrics can overlook, with reasonably comprehensive empirical evidence. Reviewer w3aw found the method straightforward and well detailed. Below we summarize, for each concern, what we added or clarified and where it appears in the revised PDF (text changes highlighted in blue).


Reviewer u5as

  1. How content relevance is adjudicated (Q). Content relevance is determined by sentence identity, so content labels are binary and whether a candidate is relevant under one or both criteria follows directly from its labels (Sec. 4.1, App. B.2, Sec. 5).
  2. Label reliability and the effect of mislabeling (W1, Q). An ASR-based label audit found no mismatch between utterances and their scripts. In noise injection, flipping about 10% of the content training units leaves E–C and S–C SwitchExact essentially unchanged, and flipping about 20% lowers them by less than 3 percentage points (App. E.6, Table 26).
  3. Legend of Fig. 3 (W2). We redrew Fig. 3 so that each legend entry matches one data line.

Reviewer vjDa

  1. Attribution of the gains (W1, Q1). With the loss, positive/negative counts, and number of comparisons held fixed, four-state stratified sampling outperforms conventional criterion-wise sampling (CW) by 2.12–2.99 percentage points on SwitchExact for all three criterion pairs and on S/E/C mAP. All 95% CIs exclude zero, and all four differences are positive across three training seeds (App. E.4, Table 23). The supervision composition of each branch is given in App. Table 24, and Sec. 4.4 now notes the difference in comparison counts. Sec. 5 states that the external baselines are evaluated with their public weights.
  2. Candidate construction and scope of generalization (W2, Q4). In full-corpus ranking without fixed-proportion sampling, JSRL exceeds the strongest baseline, WAVE (Dynamic), by 3.45 [2.70, 4.22] in S/E/C mAP, and the difference is positive on each of the three corpora (App. E.5, Table 25). The final sentence of the abstract now states that the conclusions concern speaker, emotion, and content retrieval, and the new Limitations section (Sec. 5) describes this scope further.
  3. Stage-wise results on INSPIRE (Q2). Training on the three attributes (Stages 1–2) yields part of the transfer. Relative to Stage 2, Stage 2 continuation and CW, both trained for the same number of steps as JSRL, change each Recall value by at most 1.25, whereas JSRL exceeds both by 3.50–7.50 (App. D.4, Table 15).
  4. Training details (Q3). The requested settings are given in App. B (Table 7, App. B.2).

Reviewer w3aw

  1. Balancing across datasets and dataset-specific results (Q). Each metric is averaged with equal weight over the three test corpora (App. A.7). JSRL achieves the highest Conflict and SwitchExact scores for all three criterion pairs on every corpus (App. Tables 17–19) and the highest multi-candidate retrieval mAP, NDCG@5, and Recall@5 on every corpus (App. E.2, Table 20). Reviewer w3aw noted that these per-dataset additions "ease the issue with the blind dataset mixing during the evaluation".

We would be happy to answer any further questions from the reviewers before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.