acceptodds
Under review as a conference paper at ICLR 2027

SLS: Structured Local Supervision for Fine-Grained Speech-Text Alignment

Abstract

Fine-grained speech-text alignment, such as understanding subtle shifts in speaking rate or vocal delivery, requires audio-text models to assign high similarity to accurate descriptions, while distinguishing them from inaccurate ones. We introduce Structured Local Supervision (SLS), a framework that adapts pretrained audio-text models using structured captions with SigLIP and supervised contrastive learning. Accurate captions encourage invariance to paraphrasing and partial descriptions, while inaccurate captions define targeted attribute contrasts and guide in-batch audio-negative mining. Across three backbone families, SLS substantially improves speech retrieval, classification, and controlled fine-grained ranking. Notably, SLS fine-grained alignment extends beyond retrieval, offering a lightweight, zero-shot proxy for instruction-conditioned text-to-speech evaluation. SLS rivals multi-billion-parameter audio LLMs in correlation with human adherence ratings. The findings highlight SLS as both a general recipe for attribute-sensitive representations and an efficient, scalable foundation for generative speech evaluation and alignment objectives.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-text alignment, contrastive learning, text-to-speech evaluation, tts instruction-adherence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TkwhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structured Local Supervision (SLS), a fine-tuning framework for improving fine-grained speech-text alignment in pretrained dual-encoder audio-text models. SLS uses an LLM to construct multi-granularity accurate captions for each speech sample, including paraphrases and partial positives that omit some attributes, together with targeted inaccurate captions that flip exactly one speaking-style or acoustic attribute. Training combines a global multi-positive and multi-negative SigLIP loss with bidirectional supervised contrastive learning (SupCon). For text-to-audio SupCon, the inaccurate captions of the anchor recording are used as queries to mine the top (NA−)(N_A^-) most similar real recordings in the current batch as hard audio negatives. Across five models from three backbone families (VoiceCLAP, M2D-CLAP, and PEAV), SLS substantially improves speech retrieval, classification, and fine-grained ranking over the pretrained weights. A listening study with 150 participants further shows that SLS-adapted lightweight dual encoders (0.1B to 0.4B parameters) transfer zero-shot to the evaluation of instruction-conditioned TTS. Their correlation with human adherence ratings exceeds that of 7B to 30B open-source audio LLMs and matches Gemini 3.7 Flash.

Critical strengths

SLS is highly practical as a lightweight evaluator of expressive TTS. Judging whether instruction-conditioned speech follows a nuanced style prompt usually requires an expensive listening test or a multi-billion-parameter audio LLM. The paper shows that an SLS-adapted dual encoder with only 0.11B to 0.39B parameters reaches a macro Spearman correlation of 0.402 to 0.403 with human ratings on InstructTTSEval. This far exceeds a 30B open-source audio LLM such as Qwen3-Omni (0.159) and matches the closed model Gemini 3.7 Flash (0.402). Because a dual encoder supports precomputed embeddings and fast dot-product scoring, the result offers an efficient and differentiable tool for large-scale automatic TTS evaluation and, potentially, for reward modeling.

The experimental validation is broad, and the statistical checks are solid. The method is tested on five models from three backbone families and compared with more than 20 baselines on retrieval, four families of speech classification, and fine-grained ranking. The paper reports multi-seed variance, a CompA-style paired evaluation of same-speaker contrasts, and a perceptual study of 1,800 human ratings with utterance-clustered bootstrap confidence intervals and Holm-adjusted p-values. These results support the full training recipe.

Critical weaknesses

The marginal gain of bidirectional SupCon over standard SigLIP with structured captions is very small. Table 2 (row 4 versus row 7) and the TTS results in Table 6 leave open how much of the reported improvement comes from bidirectional SupCon and audio mining, and how much comes from the LLM-synthesized structured captions themselves. Multi-positive and multi-negative SigLIP, trained only on the synthetic accurate and inaccurate captions (row 4), already raises ranking PairAcc to 87.9%, 87.8%, and 88.7% on the three backbones. Adding the paper’s core algorithmic piece, bidirectional SupCon with in-batch audio mining (row 7), increases PairAcc by only 0.2, 0.7, and 0.7 points, while lowering retrieval on two backbones (VoiceCLAP S by 0.4 points and PEAV S by 1.9 points). In Table 6, M2D-CLAP trained with SigLIP and structured captions alone correlates better with humans (0.371) than the full SLS objective (0.347). About 90% of the gain therefore comes from the synthetic caption construction rather than from the proposed local contrastive loss.

Global retrieval drops sharply, and the ranking benchmark may fit the wording preferences of the same LLM family. Relative to standard SigLIP fine-tuning on the same training corpus (Table 2, row 2), structured captions plus SLS (row 7) lower ParaSpeechCaps speech-retrieval Recall@5 by 11.2 to 13.3 points on all three models. PEAV S, for example, falls from 85.0% to 73.8%. General audio retrieval on AudioCaps and Clotho in Table 9 also declines broadly. The paper mentions this only as a trade-off and does not explain why local supervision damages instance-level retrieval to this degree. Separately, the structured training captions and the ten-candidate ranking test are both generated by the same model family, Gemini 3.1 Flash-Lite or Gemini 3.5 Flash, under the same prompt rules. The adapted models may therefore fit Gemini’s lexical preferences and antonym substitutions. Gemini 3.7 Flash reaching 99.8% PairAcc on this ranking benchmark in Table 11 is consistent with that same-family match.

Questions for authors
  1. What does bidirectional SupCon add beyond the constructed captions? Row 4 of Table 2, SigLIP with accurate and inaccurate captions, already accounts for most of the ranking gain of full SLS in row 7. Classification still rises by 1.3 to 4.5 points from row 4 to row 7, and on M2D-CLAP in Table 6 the structured-caption SigLIP run outperforms full SLS. In which settings do bidirectional SupCon and audio mining give a statistically significant advantage over multi-positive and multi-negative SigLIP trained on the structured captions alone?

  2. Why does global retrieval degrade? In Table 2, why do accurate and inaccurate captions together with the local losses (rows 3, 4, and 7) reduce ParaSpeechCaps Recall@5 by 11 to 13 points relative to standard SigLIP fine-tuning (row 2), and why does retrieval also drop on AudioCaps and Clotho in Table 9? Please explain, theoretically or experimentally, why stronger local caption invariance and discrimination harms instance-level retrieval.

  3. Can the authors measure same-family generation bias in the ranking benchmark? The positive and negative training captions and the ten-candidate test set are generated by Gemini 3.1 Flash-Lite or Gemini 3.5 Flash. Please regenerate the test candidates with a model from another family, such as GPT-4o or Qwen, either to show that this shared preference does not inflate the scores or to quantify the effect. If accuracy drops substantially under captions from another model family, the ranking benchmark should be rebuilt.

  4. How acoustically pure are the mined audio negatives in a small mixed batch? Under the default global batch size (B=80), with 17 heterogeneous speech, environmental-sound, and music datasets, what fraction of the top (NA−=8N_A^-=8) recordings (Mi)(\mathcal{M}_i) selected by Equation 1 actually share the anchor’s speaker or acoustic background and differ only in the targeted attribute? How many are unrelated recordings?

  5. If the authors address these concerns with a sufficient analysis or with targeted experiments in the rebuttal, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

Response to reviewer Tkwh's initial review (part 1)

We thank the reviewer for the constructive feedback and would like to address the questions raised.


Q1. Gains beyond structured-caption SigLIP.

Structured captions account for most of the mean PairAcc gain (Table 2). We compare full SLS with SigLIP using the same accurate and inaccurate captions (structured-caption SigLIP) across three matched training seeds to assess the added bidirectional SupCon and audio mining.

Values below are SLS minus structured-caption SigLIP, as mean ± SD in percentage points. Bold marks Holm-adjusted p<0.05p<0.05. Mean PairAcc averages caption and audio ranking, while StrictSep requires every positive to outrank every negative.

BackboneMacro F1Mean PairAccCaption StrictSepAudio StrictSep
VoiceCLAP S+1.48 ± 0.18+0.16 ± 0.35+4.09 ± 0.46+0.84 ± 0.29
M2D-CLAP+4.08 ± 0.36+0.78 ± 0.14+3.32 ± 0.89+3.97 ± 1.29
PEAV S+1.75 ± 0.43+0.43 ± 0.26+0.50 ± 0.76+2.83 ± 1.58

Classification improves in every seed and is significant on all three backbones. The added ranking benefit is clearest in StrictSep: four of six comparisons are significant, with gains in every seed. Mean PairAcc is significant only for M2D-CLAP.

We directly compare agreement with human TTS adherence ratings on the 20-source listening set. The table reports macro Spearman correlation (ρ\rho) and Holm-adjusted pp-values for SLS versus structured-caption SigLIP.

BackboneStructured-caption SigLIP ρ\rhoFull SLS ρ\rhoHolm pp
VoiceCLAP S0.3670.4030.402
M2D-CLAP0.3710.3470.556
PEAV S0.3420.4020.239

None of the direct differences is statistically significant. M2D-CLAP's decrease is concentrated in acoustic-parameter specification (APS), partly offset by better role-play correlation. Ranking gains therefore do not transfer uniformly to human agreement. Still, Table 6 supports SLS's use as a lightweight evaluator, with significant gains over pretrained M2D-CLAP and PEAV S without training on human ratings.


Q2. Retrieval versus attribute discrimination.

Most speech-retrieval loss occurs when structured captions are introduced, before adding the local losses. Table 2 reports mean Recall@5 (%) over both retrieval directions. Columns add components cumulatively.

BackboneStandard SigLIP+ Accurate captions+ Inaccurate captions+ Local losses (SLS)
VoiceCLAP S70.361.557.457.0
M2D-CLAP75.166.457.762.1
PEAV S85.078.675.773.8

Across three seeds, the added local losses significantly improve M2D-CLAP retrieval and lower PEAV S retrieval, while the VoiceCLAP S change is not significant.

Moreover, instruction-style training captions differ from the descriptive captions used for ParaSpeechCaps retrieval. Removing these instruction captions recovers much of VoiceCLAP S's retrieval loss:

MethodRecall@5 with instructions (%)Recall@5 without instructions (%)
Structured-caption SigLIP57.469.0
Full SLS57.067.4

This suggests that the difference between training instructions and retrieval descriptions explains much of the loss. The caption supervision that supports broader style understanding can come at a cost to retrieving a specific recording.

General-audio retrieval also depends on the backbone and dataset. The component comparison below reports mean Recall@5 (%) over both directions.

BackboneDatasetStandard SigLIPStructured-caption SigLIPFull SLS
VoiceCLAP SAudioCaps17.116.116.2
VoiceCLAP SClotho3.74.04.3
M2D-CLAPAudioCaps76.875.468.2
M2D-CLAPClotho35.836.431.5
PEAV SAudioCaps79.377.576.2
PEAV SClotho49.947.548.5

Local losses reduce both M2D-CLAP scores, while VoiceCLAP S improves slightly, and PEAV S moves in the opposite direction across datasets. The speech-focused training mixture (Table 8) may contribute to this trade-off. General-audio retrieval is not consistently preserved and is beyond the scope of this paper.

AuthorsOpenReview ↗

Response to reviewer Tkwh's initial review (part 2)

Q3. LLM generator-family bias.

We regenerated the ten-candidate ParaSpeechCaps caption-ranking test using GPT-6 Luna with the same generation template. Adapted-model scores average three training seeds. Cells show Gemini → GPT scores (%).

BackboneMethodPairAccStrictSep
VoiceCLAP SPretrained70.8 → 66.23.2 → 1.7
VoiceCLAP SStructured-caption SigLIP96.8 → 95.566.6 → 63.3
VoiceCLAP SFull SLS96.8 → 95.570.6 → 65.6
M2D-CLAPPretrained56.4 → 57.00.2 → 0.5
M2D-CLAPStructured-caption SigLIP96.3 → 94.465.9 → 59.5
M2D-CLAPFull SLS96.5 → 94.969.2 → 63.6
PEAV SPretrained65.1 → 66.01.8 → 2.0
PEAV SStructured-caption SigLIP96.9 → 95.772.5 → 68.5
PEAV SFull SLS97.3 → 95.973.0 → 68.5

SLS remains well above pretraining with GPT-generated captions, although performance drops. Its StrictSep advantage over structured-caption SigLIP persists for VoiceCLAP S and M2D-CLAP, while PEAV S is approximately tied. This supports transfer across generators while showing some sensitivity to the candidate captions. Table 13 provides complementary evidence: all five SLS models improve on manually reviewed same-speaker contrasts.


Q4. Composition of mined audio negatives.

We asked two independent reviewers to audit 24 anchors and 72 mined audio pairs from SLS-PEAV S in random batches of 80 recordings.

The reviewers checked speaker identity, audible background, the changed attribute in one selected inaccurate caption, and whether recordings were unrelated. Rates below summarize both reviewers' judgments.

Property of the mined recordingRate
Same speaker as the anchor0.0%
Similar audible background78.9%
Matches the changed attribute value35.9%
Unrelated / substantially different scene6.3%
Incompatible with the anchor's full caption99.2%

The audited negatives usually share similar backgrounds, but none was confirmed as a same-speaker, single-attribute contrast. Also, 35.9% of the mined recordings actually match the changed attribute in the text queries.

Thanks for running the extra experiments and detailed replies.

The GPT-6 test and the instruction-caption ablation clear up my concerns on Q2 and Q3.

However, the Q1 and Q4 results reinforce my earlier concern. Full SLS helps on Macro F1 and StrictSep, but most of the gain still comes from the structured captions, and the human TTS correlation difference isn't statistically significant. Also, the audit shows audio mining doesn't really find same-speaker, single-attribute pairs (0% same-speaker, 35.9% target-attribute match)—it mostly grabs negatives with similar backgrounds.

Overall I still see this as a useful empirical paper, so I'll keep my rating at Weak Accept.

Thank you for taking the time to review the additional experiments. We're glad the GPT-6 test and the instruction-caption ablation addressed your concerns. We also appreciate your feedback on Q1 and Q4.


In the audit, we mine negatives within batches randomly sampled from our training mix (which combines several datasets). So, for a given anchor, the batch may contain few or no other recordings from the same speaker.

We also follow the ParaSpeechCaps audio preparation, which applies VoiceFixer noise removal to the VoxCeleb recordings. This may partly account for the similar audible backgrounds observed in the audit.

Reference: https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#221-voxceleb

Reviewer_Qkx4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SLS adapts pretrained audio-text encoders using accurate captions, including paraphrases and valid partial descriptions, together with captions containing targeted attribute contradictions. The inaccurate captions also query an in-batch miner to select audio negatives. Training combines SigLIP with bidirectional supervised contrastive losses. Experiments cover three backbone families, speech retrieval/classification/ranking, controlled same-speaker contrasts, and transfer to TTS instruction-adherence scoring against human judgments. The main contribution is a reusable supervision recipe for making embedding similarities more sensitive to fine-grained speech attributes.

Critical strengths
  1. The paper develops a useful adaptation recipe for attribute-sensitive speech-text matching. Treating paraphrases and valid partial descriptions as accurate while distinguishing explicit contradictions is an appropriate goal. SLS combines these relations with audio mining and adapts pretrained encoders without changing the downstream scoring interface. The practical contribution is this transferable supervision recipe, rather than multi-positive learning or prompt-adherence scoring in isolation.

  2. The empirical study supports utility across model families and tasks. Three backbone families, controlled same-speaker contrasts, three seeds, mining alternatives, and transfer to TTS adherence give substantial evidence. The human study includes 1,800 ratings, and the paper appropriately reports utterance-clustered uncertainty; the underlying content comprises only 20 source utterances, which limits breadth. Voice and PE_S reach approximately 0.40 macro Spearman correlation, supporting a useful inexpensive scoring option without establishing equivalence to human judgment. These practical results are the main reason for my positive recommendation.

Critical weaknesses
  1. The shared audio-negative set need not preserve the validity of partial descriptions. Equation 1 constructs one mined-audio set per anchor, and Equation 3 uses it as negatives for every accurate caption. Consider a source described as “a woman speaks quickly,” its valid partial description “a woman speaks,” and an inaccurate caption “a woman speaks slowly.” A recording matching the inaccurate caption can still satisfy the partial description. Excluding the anchor and known positive-caption matches cannot exhaust unannotated semantic compatibility. The authors acknowledge the difficulty of discovering additional positives; the remaining issue is a tension between source-instance discrimination and semantic adherence, not an assertion that they overlooked false negatives altogether. SPAM's style-key positives explicitly recognize many-to-many matching. The logical counterexample establishes that the objective permits such false negatives, not their empirical prevalence. This matters when the score is interpreted as adherence of arbitrary generated speech to a partial prompt.

  2. The incremental contribution of mining and the added losses is less uniform than the headline gains suggest. Structured-caption SigLIP already accounts for a substantial fraction of the benefit. For Voice in Table 2, ranking rises from 73.0 with ordinary SigLIP to 87.9 with structured-caption SigLIP and only to 88.1 with full SLS, while retrieval falls from 70.3 to 57.4 to 57.0. In Table 6, full SLS improves TTS macro Spearman over structured-caption SigLIP for Voice (0.367 to 0.403) and PE_S (0.342 to 0.402), but lowers it for M2D (0.371 to 0.347). The mining operation also averages inaccurate-caption dot products, exactly equivalent to retrieval against their mean embedding; this is a useful heuristic, not a guarantee that each retrieved recording contradicts each accurate caption. Potential cancellation is not itself a measured failure. Given the precedents in CLSP for multi-granular contrastive learning, SPAM for adherence scoring, and Learning Audio Concepts from Counterfactual Natural Language for counterfactual caption learning, the contribution should distinguish the value of structured captions from the additional, task-dependent benefit of mining and the contrastive terms.

Questions for authors
  1. Clarification of relation semantics: Beyond known-positive exclusion, are caption-specific compatibility masks used? How is the full/partial-caption counterexample handled? If text-to-audio training intentionally uses source identity as a proxy, what limits should users place on interpreting the resulting score as prompt adherence?

  2. Interpretation of existing ablations: What do the current ablations establish about the added value of mining and the contrastive terms over structured-caption SigLIP? Please explain the M2D TTS decrease and the retrieval/attribute-ranking trade-off, and clarify whether the mean-embedding miner is intended to retrieve certified contradictions or simply difficult nearby recordings. This question asks for decomposition of existing evidence rather than more benchmark coverage.

1 comment
AuthorsOpenReview ↗

Response to reviewer Qkx4's initial review

We thank the reviewer for the constructive feedback and would like to address the questions raised.


Q1. Relation semantics and partial descriptions.

No caption-specific compatibility mask is used. Eq. 3 shares mined negatives across all accurate captions and uses the paired recording as the positive. The counterexample is valid: a recording can contradict the full caption while satisfying a partial description.

To verify this, we asked two independent reviewers to audit 24 anchors and 72 mined audio pairs from SLS-PEAV S in random batches of 80 recordings.

A recording matches a description only if all stated attributes hold, while omitted attributes are ignored (partial description). Rates below summarize both reviewers' judgments.

Mined recording judgedFull captionPartial description
Matches0.0%2.9%
Does not match99.2%96.4%
Unclear0.8%0.8%

Most mined recordings were incompatible with both descriptions in this sample, while partial-caption matches were uncommon (2.9%). For partial prompts, the score can therefore reflect source identity as well as semantic adherence. Its downstream support comes from agreement with human ratings on the studied prompts.


Q2. What the ablations establish.

Q2-1. Structured captions account for most of the mean PairAcc gain (Table 2). We compare full SLS with SigLIP using the same accurate and inaccurate captions (structured-caption SigLIP) across three matched training seeds to assess the added bidirectional SupCon and audio mining.

Values in the following table are SLS minus structured-caption SigLIP, as mean ± SD in percentage points. Bold marks Holm-adjusted p<0.05p<0.05. Mean PairAcc averages caption and audio ranking, while StrictSep requires every positive to outrank every negative.

BackboneMacro F1Mean PairAccCaption StrictSepAudio StrictSep
VoiceCLAP S+1.48 ± 0.18+0.16 ± 0.35+4.09 ± 0.46+0.84 ± 0.29
M2D-CLAP+4.08 ± 0.36+0.78 ± 0.14+3.32 ± 0.89+3.97 ± 1.29
PEAV S+1.75 ± 0.43+0.43 ± 0.26+0.50 ± 0.76+2.83 ± 1.58

Classification improves in every seed and is significant on all three backbones. The added ranking benefit is clearest in StrictSep: four of six comparisons are significant, with gains in every seed. Mean PairAcc is significant only for M2D-CLAP.

Q2-2. Mining. Eq. 1's average similarity equals similarity to the mean inaccurate-caption embedding. It selects difficult nearby recordings and does not guarantee contradiction of every accurate caption. The comparisons in Tables 3 and 15 support its practical value: Accurate- and inaccurate-caption queries give close, mixed results (Table 15). Similarity-based mining improves retrieval, classification, and audio ranking over random mining, with a small caption-ranking decrease (Table 3).

Q2-3. Most speech-retrieval loss occurs when structured captions are introduced, before adding the local losses. Instruction-style training captions differ from the descriptive captions used for ParaSpeechCaps retrieval. Removing these instruction captions recovers much of VoiceCLAP S's retrieval loss:

MethodRecall@5 with instructions (%)Recall@5 without instructions (%)
Structured-caption SigLIP57.469.0
Full SLS57.067.4

This suggests that the difference between training instructions and retrieval descriptions explains much of the loss. The caption supervision that supports broader style understanding can come at a cost to retrieving a specific recording.

Q2-4. TTS transfer.

We directly compare agreement with human TTS adherence ratings on the 20-source listening set. The table reports macro Spearman correlation (ρ\rho) and Holm-adjusted pp-values for SLS versus structured-caption SigLIP.

BackboneStructured-caption SigLIP ρ\rhoFull SLS ρ\rhoHolm pp
VoiceCLAP S0.3670.4030.402
M2D-CLAP0.3710.3470.556
PEAV S0.3420.4020.239

None of the direct differences are statistically significant. M2D-CLAP's decrease is concentrated in acoustic-parameter specification (APS), partly offset by better role-play (RP) correlation. Ranking gains therefore do not transfer uniformly to human agreement. Still, Table 6 supports SLS's use as a lightweight evaluator, with significant gains over pretrained M2D-CLAP and PEAV S without training on human ratings.

Our contribution builds on CLSP's multi-granular supervision, SPAM's bidirectional SupCon, and counterfactual-caption learning, connecting structured caption relations with audio-negative selection for pretrained audio-text models.

Reviewer_2iVHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SLS to improve fine-grained alignment between speech and text. It uses accurate and inaccurate captions to teach models to distinguish speech attributes, such as speaking rate and emotion. The inaccurate captions also guide the selection of negative audio samples during training. The method is evaluated on retrieval, classification, ranking, and TTS instruction-adherence evaluation.

Critical strengths
  1. Clear motivation. The paper focuses on subtle differences in speech that existing audio–text models may overlook. The proposed supervision directly targets these differences.
  2. Evaluation across multiple backbones. Experiments across three backbone families strengthen the evidence that the training approach can benefit different pretrained models.
Critical weaknesses
  1. Limited methodological novelty. This is my main concern. Training with positive pairs and bidirectional supervised contrastive loss has already been explored for speech–text alignment, including in SPAM (https://arxiv.org/html/2601.05554v1). The main distinction appears to be using inaccurate captions to select negative audio samples. Although this is an interesting extension, the paper does not sufficiently establish its advantage over conventional hard-negative mining using accurate captions as queries.
  2. The benefit for TTS evaluation is inconsistent across models. In Table 6, SLS performs worse than SigLIP with structured captions on M2D-CLAP. Furthermore, the reported significance tests compare adapted models with their pretrained checkpoints, rather than directly testing SLS against structured-caption SigLIP. Therefore, the evidence for the full method’s advantage in this application is limited.
  3. The human evaluation covers a small number of underlying utterances. Although the study includes many listeners and ratings, it uses only 20 underlying utterances. This limits the diversity of the evaluation and makes it difficult to draw broad conclusions about TTS evaluation performance.
  4. The reliability of the negative samples needs further validation. The supervision depends on edited captions containing a genuine contradiction and mined audio being unsuitable for the corresponding accurate caption. Similarity to an inaccurate caption does not necessarily guarantee the latter. A human audit of these training examples would help establish the reliability of the supervision.
Questions for authors
  1. What is the main methodological advance over SPAM and existing hard-negative mining approaches? Can the authors compare inaccurate-caption-guided mining with mining based on accurate captions under the same training conditions?
  2. Are the additional gains over structured-caption SigLIP consistent across random seeds and statistically reliable, particularly for ranking?
  3. What explains the weaker TTS evaluation performance of SLS on M2D-CLAP? Are the improvements on the other backbones significant relative to structured-caption SigLIP?
  4. How were the 20 utterances selected, and how well do they cover different speaking styles and instructions? Can the evaluation be extended to more utterances?
  5. Were the generated caption contradictions and mined audio negatives manually checked? What proportion of selected negatives are actually compatible with the corresponding accurate caption?
3 comments
AuthorsOpenReview ↗

Response to reviewer 2iVH's initial review

We thank the reviewer for the constructive feedback and would like to address the questions raised.


Q1. Contribution and matched mining baseline.

SLS connects accurate caption variants, targeted inaccurate captions, and audio-negative mining to adapt pretrained audio-text models. SPAM already uses style-key positives and bidirectional SupCon. SLS's distinction is using inaccurate captions both as text negatives and as queries for real audio negatives, alongside paraphrases and valid partial descriptions.

With the full SLS objective fixed, we compare audio-negative selection for VoiceCLAP S (Tables 3 and 15). Scores below are percentages.

Audio-negative selectionRecall@5Macro F1Caption PairAccAudio PairAcc
Random42.160.797.576.8
Accurate-caption queries56.562.096.779.8
Inaccurate-caption queries (SLS)57.062.396.879.4

Accurate- and inaccurate-caption queries give close, mixed results. Similarity-based mining improves retrieval, classification, and audio ranking over random mining, with a small caption-ranking decrease.


Q2. Gains beyond structured-caption SigLIP.

Structured captions account for most of the mean PairAcc gain (Table 2). We compare full SLS with SigLIP using the same accurate and inaccurate captions (structured-caption SigLIP) across three matched training seeds to assess the added bidirectional SupCon and audio mining.

Values in the following table are SLS minus structured-caption SigLIP, as mean ± SD in percentage points. Bold marks Holm-adjusted p<0.05p<0.05. Mean PairAcc averages caption and audio ranking, while StrictSep requires every positive to outrank every negative.

BackboneMacro F1Mean PairAccCaption StrictSepAudio StrictSep
VoiceCLAP S+1.48 ± 0.18+0.16 ± 0.35+4.09 ± 0.46+0.84 ± 0.29
M2D-CLAP+4.08 ± 0.36+0.78 ± 0.14+3.32 ± 0.89+3.97 ± 1.29
PEAV S+1.75 ± 0.43+0.43 ± 0.26+0.50 ± 0.76+2.83 ± 1.58

Classification improves in every seed and is significant on all three backbones. The added ranking benefit is clearest in StrictSep: four of six comparisons are significant, with gains in every seed. Mean PairAcc is significant only for M2D-CLAP.


Q3. TTS transfer and M2D-CLAP.

We directly compare agreement with human TTS adherence ratings on the 20-source listening set. The table reports macro Spearman correlation (ρ\rho) and Holm-adjusted pp-values for SLS versus structured-caption SigLIP.

BackboneStructured-caption SigLIP ρ\rhoFull SLS ρ\rhoHolm pp
VoiceCLAP S0.3670.4030.402
M2D-CLAP0.3710.3470.556
PEAV S0.3420.4020.239

None of the direct differences are statistically significant. M2D-CLAP's decrease is concentrated in acoustic-parameter specification (APS), partly offset by better role-play (RP) correlation. Ranking gains therefore do not transfer uniformly to human agreement. Still, Table 6 supports SLS's use as a lightweight evaluator, with significant gains over pretrained M2D-CLAP and PEAV S without training on human ratings.


Q4. Listening-study coverage.

Appendix F.1 describes the selection: adult speakers with complete synthesis and judge coverage, balanced genders and accents, and varied emotion, personality, texture, and tone. Each source is evaluated across six audio conditions and three instruction types. Expanding the study would require substantial additional listening work because APS asks listeners to judge individual attributes separately.


Q5. Reliability of captions and mined negatives.

We asked two independent reviewers to audit 24 anchors and 72 mined audio pairs from SLS-PEAV S in random batches of 80 recordings.

Caption checkBoth reviewers agreed
Inaccurate caption introduces an audible contradiction24/24
Edit changes exactly one attribute23/24
Full caption accurately describes the anchor20/24
All three checks pass19/24

The edits consistently introduced audible contradictions in this sample, while full-caption accuracy was less reliable.

Next, a recording matches a description only if all stated attributes hold, while omitted attributes are ignored (partial description). Rates below summarize both reviewers' judgments.

Mined negative recording judgedFull captionPartial description
Matches0.0%2.9%
Does not match99.2%96.4%
Unclear0.8%0.8%

Most mined recordings were incompatible with both descriptions in this sample, while partial-caption matches were uncommon (2.9%).

Thanks for the response and additional experiments. The multi-seed results strengthen the evidence for the full objective, and the human audit partly addresses my concern about the reliability of negative samples. My main concern is still the novelty; inaccurate-caption mining gives small and mixed gains over accurate-caption mining. The direct tests also do not show a significant advantage over structured-caption SigLIP for TTS evaluation, and the human study remains limited to 20 source utterances.

Considering the additional validation, I will revise my score to 2, while retaining these concerns.

Thank you for taking time to review the additional experiments and reconsider your score.

We appreciate your candid feedback on the novelty of the mining approach, the comparison with structured-caption SigLIP, and the size of the human study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.