- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Structured Local Supervision (SLS), a fine-tuning framework for improving fine-grained speech-text alignment in pretrained dual-encoder audio-text models. SLS uses an LLM to construct multi-granularity accurate captions for each speech sample, including paraphrases and partial positives that omit some attributes, together with targeted inaccurate captions that flip exactly one speaking-style or acoustic attribute. Training combines a global multi-positive and multi-negative SigLIP loss with bidirectional supervised contrastive learning (SupCon). For text-to-audio SupCon, the inaccurate captions of the anchor recording are used as queries to mine the top most similar real recordings in the current batch as hard audio negatives. Across five models from three backbone families (VoiceCLAP, M2D-CLAP, and PEAV), SLS substantially improves speech retrieval, classification, and fine-grained ranking over the pretrained weights. A listening study with 150 participants further shows that SLS-adapted lightweight dual encoders (0.1B to 0.4B parameters) transfer zero-shot to the evaluation of instruction-conditioned TTS. Their correlation with human adherence ratings exceeds that of 7B to 30B open-source audio LLMs and matches Gemini 3.7 Flash.
Critical strengths
SLS is highly practical as a lightweight evaluator of expressive TTS. Judging whether instruction-conditioned speech follows a nuanced style prompt usually requires an expensive listening test or a multi-billion-parameter audio LLM. The paper shows that an SLS-adapted dual encoder with only 0.11B to 0.39B parameters reaches a macro Spearman correlation of 0.402 to 0.403 with human ratings on InstructTTSEval. This far exceeds a 30B open-source audio LLM such as Qwen3-Omni (0.159) and matches the closed model Gemini 3.7 Flash (0.402). Because a dual encoder supports precomputed embeddings and fast dot-product scoring, the result offers an efficient and differentiable tool for large-scale automatic TTS evaluation and, potentially, for reward modeling.
The experimental validation is broad, and the statistical checks are solid. The method is tested on five models from three backbone families and compared with more than 20 baselines on retrieval, four families of speech classification, and fine-grained ranking. The paper reports multi-seed variance, a CompA-style paired evaluation of same-speaker contrasts, and a perceptual study of 1,800 human ratings with utterance-clustered bootstrap confidence intervals and Holm-adjusted p-values. These results support the full training recipe.
Critical weaknesses
The marginal gain of bidirectional SupCon over standard SigLIP with structured captions is very small. Table 2 (row 4 versus row 7) and the TTS results in Table 6 leave open how much of the reported improvement comes from bidirectional SupCon and audio mining, and how much comes from the LLM-synthesized structured captions themselves. Multi-positive and multi-negative SigLIP, trained only on the synthetic accurate and inaccurate captions (row 4), already raises ranking PairAcc to 87.9%, 87.8%, and 88.7% on the three backbones. Adding the paper’s core algorithmic piece, bidirectional SupCon with in-batch audio mining (row 7), increases PairAcc by only 0.2, 0.7, and 0.7 points, while lowering retrieval on two backbones (VoiceCLAP S by 0.4 points and PEAV S by 1.9 points). In Table 6, M2D-CLAP trained with SigLIP and structured captions alone correlates better with humans (0.371) than the full SLS objective (0.347). About 90% of the gain therefore comes from the synthetic caption construction rather than from the proposed local contrastive loss.
Global retrieval drops sharply, and the ranking benchmark may fit the wording preferences of the same LLM family. Relative to standard SigLIP fine-tuning on the same training corpus (Table 2, row 2), structured captions plus SLS (row 7) lower ParaSpeechCaps speech-retrieval Recall@5 by 11.2 to 13.3 points on all three models. PEAV S, for example, falls from 85.0% to 73.8%. General audio retrieval on AudioCaps and Clotho in Table 9 also declines broadly. The paper mentions this only as a trade-off and does not explain why local supervision damages instance-level retrieval to this degree. Separately, the structured training captions and the ten-candidate ranking test are both generated by the same model family, Gemini 3.1 Flash-Lite or Gemini 3.5 Flash, under the same prompt rules. The adapted models may therefore fit Gemini’s lexical preferences and antonym substitutions. Gemini 3.7 Flash reaching 99.8% PairAcc on this ranking benchmark in Table 11 is consistent with that same-family match.
Questions for authors
-
What does bidirectional SupCon add beyond the constructed captions? Row 4 of Table 2, SigLIP with accurate and inaccurate captions, already accounts for most of the ranking gain of full SLS in row 7. Classification still rises by 1.3 to 4.5 points from row 4 to row 7, and on M2D-CLAP in Table 6 the structured-caption SigLIP run outperforms full SLS. In which settings do bidirectional SupCon and audio mining give a statistically significant advantage over multi-positive and multi-negative SigLIP trained on the structured captions alone?
-
Why does global retrieval degrade? In Table 2, why do accurate and inaccurate captions together with the local losses (rows 3, 4, and 7) reduce ParaSpeechCaps Recall@5 by 11 to 13 points relative to standard SigLIP fine-tuning (row 2), and why does retrieval also drop on AudioCaps and Clotho in Table 9? Please explain, theoretically or experimentally, why stronger local caption invariance and discrimination harms instance-level retrieval.
-
Can the authors measure same-family generation bias in the ranking benchmark? The positive and negative training captions and the ten-candidate test set are generated by Gemini 3.1 Flash-Lite or Gemini 3.5 Flash. Please regenerate the test candidates with a model from another family, such as GPT-4o or Qwen, either to show that this shared preference does not inflate the scores or to quantify the effect. If accuracy drops substantially under captions from another model family, the ranking benchmark should be rebuilt.
-
How acoustically pure are the mined audio negatives in a small mixed batch? Under the default global batch size (B=80), with 17 heterogeneous speech, environmental-sound, and music datasets, what fraction of the top () recordings selected by Equation 1 actually share the anchor’s speaker or acoustic background and differ only in the targeted attribute? How many are unrelated recordings?
-
If the authors address these concerns with a sufficient analysis or with targeted experiments in the rebuttal, I would consider raising my score.
4 comments
Response to reviewer Tkwh's initial review (part 2)
Q3. LLM generator-family bias.
We regenerated the ten-candidate ParaSpeechCaps caption-ranking test using GPT-6 Luna with the same generation template. Adapted-model scores average three training seeds. Cells show Gemini → GPT scores (%).
| Backbone | Method | PairAcc | StrictSep |
|---|---|---|---|
| VoiceCLAP S | Pretrained | 70.8 → 66.2 | 3.2 → 1.7 |
| VoiceCLAP S | Structured-caption SigLIP | 96.8 → 95.5 | 66.6 → 63.3 |
| VoiceCLAP S | Full SLS | 96.8 → 95.5 | 70.6 → 65.6 |
| M2D-CLAP | Pretrained | 56.4 → 57.0 | 0.2 → 0.5 |
| M2D-CLAP | Structured-caption SigLIP | 96.3 → 94.4 | 65.9 → 59.5 |
| M2D-CLAP | Full SLS | 96.5 → 94.9 | 69.2 → 63.6 |
| PEAV S | Pretrained | 65.1 → 66.0 | 1.8 → 2.0 |
| PEAV S | Structured-caption SigLIP | 96.9 → 95.7 | 72.5 → 68.5 |
| PEAV S | Full SLS | 97.3 → 95.9 | 73.0 → 68.5 |
SLS remains well above pretraining with GPT-generated captions, although performance drops. Its StrictSep advantage over structured-caption SigLIP persists for VoiceCLAP S and M2D-CLAP, while PEAV S is approximately tied. This supports transfer across generators while showing some sensitivity to the candidate captions. Table 13 provides complementary evidence: all five SLS models improve on manually reviewed same-speaker contrasts.
Q4. Composition of mined audio negatives.
We asked two independent reviewers to audit 24 anchors and 72 mined audio pairs from SLS-PEAV S in random batches of 80 recordings.
The reviewers checked speaker identity, audible background, the changed attribute in one selected inaccurate caption, and whether recordings were unrelated. Rates below summarize both reviewers' judgments.
| Property of the mined recording | Rate |
|---|---|
| Same speaker as the anchor | 0.0% |
| Similar audible background | 78.9% |
| Matches the changed attribute value | 35.9% |
| Unrelated / substantially different scene | 6.3% |
| Incompatible with the anchor's full caption | 99.2% |
The audited negatives usually share similar backgrounds, but none was confirmed as a same-speaker, single-attribute contrast. Also, 35.9% of the mined recordings actually match the changed attribute in the text queries.
Thanks for running the extra experiments and detailed replies.
The GPT-6 test and the instruction-caption ablation clear up my concerns on Q2 and Q3.
However, the Q1 and Q4 results reinforce my earlier concern. Full SLS helps on Macro F1 and StrictSep, but most of the gain still comes from the structured captions, and the human TTS correlation difference isn't statistically significant. Also, the audit shows audio mining doesn't really find same-speaker, single-attribute pairs (0% same-speaker, 35.9% target-attribute match)—it mostly grabs negatives with similar backgrounds.
Overall I still see this as a useful empirical paper, so I'll keep my rating at Weak Accept.
Thank you for taking the time to review the additional experiments. We're glad the GPT-6 test and the instruction-caption ablation addressed your concerns. We also appreciate your feedback on Q1 and Q4.
In the audit, we mine negatives within batches randomly sampled from our training mix (which combines several datasets). So, for a given anchor, the batch may contain few or no other recordings from the same speaker.
We also follow the ParaSpeechCaps audio preparation, which applies VoiceFixer noise removal to the VoxCeleb recordings. This may partly account for the similar audible backgrounds observed in the audit.
Reference: https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#221-voxceleb
Response to reviewer Tkwh's initial review (part 1)
We thank the reviewer for the constructive feedback and would like to address the questions raised.
Q1. Gains beyond structured-caption SigLIP.
Structured captions account for most of the mean PairAcc gain (Table 2). We compare full SLS with SigLIP using the same accurate and inaccurate captions (structured-caption SigLIP) across three matched training seeds to assess the added bidirectional SupCon and audio mining.
Values below are SLS minus structured-caption SigLIP, as mean ± SD in percentage points. Bold marks Holm-adjusted p<0.05. Mean PairAcc averages caption and audio ranking, while StrictSep requires every positive to outrank every negative.
Classification improves in every seed and is significant on all three backbones. The added ranking benefit is clearest in StrictSep: four of six comparisons are significant, with gains in every seed. Mean PairAcc is significant only for M2D-CLAP.
We directly compare agreement with human TTS adherence ratings on the 20-source listening set. The table reports macro Spearman correlation (ρ) and Holm-adjusted p-values for SLS versus structured-caption SigLIP.
None of the direct differences is statistically significant. M2D-CLAP's decrease is concentrated in acoustic-parameter specification (APS), partly offset by better role-play correlation. Ranking gains therefore do not transfer uniformly to human agreement. Still, Table 6 supports SLS's use as a lightweight evaluator, with significant gains over pretrained M2D-CLAP and PEAV S without training on human ratings.
Q2. Retrieval versus attribute discrimination.
Most speech-retrieval loss occurs when structured captions are introduced, before adding the local losses. Table 2 reports mean Recall@5 (%) over both retrieval directions. Columns add components cumulatively.
Across three seeds, the added local losses significantly improve M2D-CLAP retrieval and lower PEAV S retrieval, while the VoiceCLAP S change is not significant.
Moreover, instruction-style training captions differ from the descriptive captions used for ParaSpeechCaps retrieval. Removing these instruction captions recovers much of VoiceCLAP S's retrieval loss:
This suggests that the difference between training instructions and retrieval descriptions explains much of the loss. The caption supervision that supports broader style understanding can come at a cost to retrieving a specific recording.
General-audio retrieval also depends on the backbone and dataset. The component comparison below reports mean Recall@5 (%) over both directions.
Local losses reduce both M2D-CLAP scores, while VoiceCLAP S improves slightly, and PEAV S moves in the opposite direction across datasets. The speech-focused training mixture (Table 8) may contribute to this trade-off. General-audio retrieval is not consistently preserved and is beyond the scope of this paper.