- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a benchmark for speech style captioning. In this task, given a speech recording, a model should be able to describe the gender, age, pitch, accent, speed, clarity, fluency, etc. of the speech. The paper curates an evaluation dataset for this task containing 1,000 single-speaker utterances which were annotated along 13 different properties by 15 annotators.
The paper uses these 1,000 waveforms to create two versions of the benchmark: the first is the single-speaker version of the benchmark, which uses the utterances as-is. The second version of the benchmark takes a subset of these utterances and artificially mixes them into multi-speaker recordings; the utterances can be spaced apart or overlapped by a random amount to simulate conversational turn-taking. In the single-speaker evaluation, an LLM as a judge (producing a 1-5 score for each property for an utterance) is used as the evaluation. In the multi speaker case, a model must evaluate a caption for each different utterance/speaker in the mix and so additional metrics are introduced to measure this.
The paper evaluates 14 different off-the-shelf systems on the task, and compares their performance.
Critical strengths
Strengths:
- The paper is well-written and easy to follow
Critical weaknesses
Weaknesses:
-
The contribution of the paper is extremely limited. It boils down to annotating 1,000 speech audio clips with speech style captions. This is an extremely small amount of data even for an evaluation benchmark, and several orders of magnitude short of what is needed to be useful for training models. Considering the use of the data as just an evaluation benchmark, the paper also does not make a convincing case that the 1,000 speech clips it annotates are different or more useful in some compelling way compared to existing speech style captioning datasets such as SpeechCraft or ParaSpeechCaps which contain thousands of hours (amounting to millions of utterances + captions).
-
The multi-speaker evaluation setting is poorly motivated. The multi-speaker recordings are synthetic mixes of single-speaker utterances, and not real conversations, and so there will not be any meaningful context across dialog turns that could make this problem setting interesting. The paper fails to explain how this setting cannot be addressed by simply applying speaker diarization/separation followed by single-speaker captioning
-
The single-speaker setting uses as its evaluation LLM-as-a-judge, and the judge model is from the same Qwen family as the top performing models on the benchmark.
Questions for authors
What does the proposed dataset offer that existing speech style captioning datasets, which are orders of magnitude larger and much broader in their coverage, cannot?
7 comments
Response to Reviewer Xn56 (2/2)
W2. The multi-speaker setting is poorly motivated; why not diarization/separation followed by single-speaker captioning?
We agree this is the key question, and we now test it directly instead of arguing it (App. I.1–I.4 of the revised PDF):
- Cascade baselines. We now report two diarizers alone (pyannote 3.1, DiariZen) and two cascades, DiariZen → Qwen3.5-Omni-Plus and DiariZen → Gemini-3.7-Flash, all scored with the same protocol. On the speakers that both the cascade and the end-to-end model recover, caption quality is close: +0.15 CapQ for the Qwen cascade and −0.16 for the Gemini cascade. Both cascades still trail their end-to-end counterparts, by 17.9 Multi points (95% CI [16.4, 19.5]) and 12.9 points (CI [11.3, 14.5]), because the diarizer recovers only 54% of the speakers versus 83% and 75%. For Qwen the gap grows from +2.8 points on 2-speaker mixtures to +33 to +55 points on 4–6 speakers. The hard part of the task is therefore recovering every speaker and attaching the right description to each, which single-speaker captioning does not test. VocalCap scores the output, so cascades remain legitimate entries.
- The description attached to the correct speaker degrades in a mixture. Holding the source voice and its reference fixed across the ~4 mixtures in which it appears, judged caption quality drops by 0.45 points (1–10 rubric, SE 0.08) from no overlap to full overlap and rises by 0.26 points per +3 dB of speaker gain. The drop is at least as large for gender, age and accent (−0.56), whose references cannot change under mixing, as for the five mixing-sensitive fields (−0.42) (within-source regression over 16,666 matched speakers of the 8 strongest systems; App. I.2, Table S25).
- Re-entry. We agree that controlled mixtures lack conversational context. We added VocalCap-RE (465 mixtures, 6–26 s, built with the main-set pipeline), where each speaker returns for 2–3 non-adjacent turns, and half pair same-gender voices. A cascade must re-identify returning speakers there. With 2–3 speakers the two approaches are close: Coverage is tied, and the end-to-end lead of +3.3 Multi (95% CI [1.2, 5.4]) is no longer significant once within-turn pauses of up to 1 s are merged in the diarizer output. The errors differ in kind: the cascade merges 25.3% of returning speakers with another speaker (29.1% for same-gender voices), versus 9.7% for end-to-end, which more often splits one speaker into two (12.1% vs. 3.5%). On a more crowded variant with 4–5 speakers (200 mixtures), the lead of end-to-end Qwen3.5-Omni-Plus grows to +16.1 Multi (CI [12.2, 19.7]) and holds under the same merging, while end-to-end Gemini-3.7-Flash is no better than its cascade (−2.1, CI [−6.9, 2.7]), so the setting also separates end-to-end models from each other.
On synthetic mixtures versus real conversation: controlled mixing is deliberate. It gives exact speaker identities and intervals and lets every speaker keep a human reference, which we could not obtain at this scale from natural conversations. It also lets us vary overlap and gain systematically, which is what the within-source analysis above relies on. We agree that natural dialogue is the necessary next step and state this limit explicitly (Sec. 7). The revised PDF adds the re-entry sets and the cascade results to strengthen the motivation (Sec. 6.2 and 7; App. I.1–I.4).
W3. The judge belongs to the same family as the top models.
We checked this directly (App. E and App. I.10 of the revised PDF). Relative to an independent DeepSeek-V4-Pro judge with identical prompts, the Qwen judge is about 2.5–3 points more lenient on all systems on the single track. The Qwen-family offset is only +0.20 points beyond that after excluding the two floor-compressed systems (+0.45 with them; exact permutation p = 0.23). On Binding, the Qwen judge is slightly stricter for Qwen systems (−0.011 vs. −0.009). The order of the 11 multi-track systems is identical under both judges, and the single-track Spearman ρ is 0.952. The top two systems keep their positions under either judge. A third judge from another provider family, GPT-OSS-120B, confirms this on six systems × 1,000 mixtures: the order is unchanged, and the Qwen judge gives Qwen systems slightly less relative credit (−0.29 Multi points, 95% CI [−0.37, −0.21]; Table S35).
We hope these results make clearer what VocalCap adds as an evaluation benchmark and show that the multi-speaker setting measures abilities that current models, and simple decompositions, do not yet solve. We would be grateful if the reviewer would reconsider the assessment in light of the new evidence.
Follow-up to Reviewer Xn56: Listening Audit of the References (related to Q1 / W1)
Thank you again for your assessment. We do not expect this note to resolve your main concerns about the size of the benchmark and about synthetic mixtures, which we tried to address in our response. It adds evidence on one narrower point. In our answer to Q1 / W1 we argued that, for evaluation, VocalCap offers field-wise human references whose reliability is measured. For the four most subjective fields (texture, tone, emotion, personality), the only evidence we had was inter-annotator agreement, which is moderate (κ = 0.41–0.52). We have now tested these references directly in a blind listening audit, announced in our response to Reviewer biU5. We report it here in full, including the points that limit it.
Design. We drew 200 single-speaker clips at random from the 1,000. For each clip and each of the four fields, a listener heard the clip and rated two descriptions, shown in random order and without any indication of their source: the clip's own reference, and the reference of the same field from another clip (same speaker gender, similar length, selected by a rule fixed before any rating was collected). Each description was rated on its own as supported, partially supported, unsupported, or insufficient evidence. Each clip was rated by two of six listeners, which gives 400 listener–clip pairs and 1,600 ratings of each kind of description.
Result.
| Field | Own reference: supported / partially / unsupported | Mismatched reference: supported / partially / unsupported | Difference in "supported" (points, 95% CI) |
|---|---|---|---|
| Texture | 89.8% / 8.5% / 1.8% | 9.2% / 29.5% / 61.0% | +80.5 [74.8, 85.5] |
| Tone | 93.5% / 6.5% / 0.0% | 5.8% / 17.8% / 76.2% | +87.8 [83.2, 92.0] |
| Emotion | 92.8% / 7.2% / 0.0% | 6.2% / 20.5% / 73.0% | +86.5 [81.5, 91.0] |
| Personality | 92.5% / 6.2% / 1.2% | 6.8% / 24.8% / 68.0% | +85.8 [80.5, 90.3] |
| All four | 92.1% / 7.1% / 0.8% | 7.0% / 23.1% / 69.6% | +85.1 [82.4, 87.8] |
CIs are from a bootstrap over clips. "Insufficient evidence" was chosen 5 times, always for a mismatched description.
The clip's own reference was rated "supported" in 92.1% of ratings and "unsupported" in 0.8% (12 ratings: 7 for texture, 5 for personality, never by both listeners of a clip). A same-gender reference from another clip was rated "supported" in 7.0%. Both listeners of a clip rated the own reference "supported" in 84.4% of the 800 clip–field cases.
What we take from it, and what we do not. On the four fields where agreement is lowest, listeners accept the clip's own reference far more often than a plausible reference from another clip, and they almost never reject it. We read this as evidence that these references are specific to the clip and rarely wrong, which is what an evaluation target needs. It does not show that the reference is the only acceptable description, and it says nothing about the size of the benchmark. With 200 clips and six listeners, it is a check and not a re-annotation.
We will add the audit, with the full rating counts, to Appendix I. We appreciate the time you have given to this paper.
Response to Reviewer Xn56 (1/2)
We thank the reviewer for the careful assessment and for finding the paper well-written. We respond to each concern below and have added new experiments for the multi-speaker and judge questions. Each answer below is self-contained; all changes in the revised PDF are marked in blue, and the Common Response summarizes the points shared with the other reviewers.
Q1 / W1. What does VocalCap offer that SpeechCraft or ParaSpeechCaps cannot?
You assess VocalCap as an evaluation benchmark and ask what its 1,000 clips offer beyond these much larger datasets. That is the right comparison, and the paper did not make the case clearly enough. VocalCap does not compete with them on size. SpeechCraft and ParaSpeechCaps were built to train captioning and style-prompted TTS models, and VocalCap was built to measure audio-LLMs. Four differences matter for that purpose:
Is 1,000 enough? We understand the concern about size. One way to check whether 1,000 items are enough is whether they separate systems reliably. With paired bootstrap tests (10,000 resamples; App. I.9, Table S34), 73 of 91 single-track pairs and 63 of 66 multi-track pairs differ significantly. The 95% CI half-width of the leading system is about ±0.8 points (single) and ±1.1 points (multi). The limit is also clear: systems within about 1 point (single) or 2 points (multi) of each other are not separated, and the revised PDF reports them as tied tiers. This size is consistent with widely used evaluation sets: AudioCaps test ≈ 1k clips, Clotho evaluation 1,045 clips, MMAU test-mini 1,000 items.
More than an annotated set. The contribution also includes the multi-speaker protocol and its finding: single-speaker rankings do not predict multi-speaker rankings (15 of 66 system pairs reverse; Qwen3-Omni-Instruct moves from 4th to 10th). New in the rebuttal: (i) a decomposition showing that recovering speakers and describing them are distinct abilities (Qwen3-Omni-Instruct describes recovered speakers better than MiMo-V2.5 but recovers 41% vs. 70% of them; response to biU5, Q2); (ii) the evidence below that description quality itself degrades in mixtures.