acceptodds
Under review as a conference paper at ICLR 2027

VocalCap: A Speaker-Level Benchmark for Fine-Grained Speech Captioning

Abstract

Understanding speech beyond its words requires describing vocal attributes and associating them with the speakers who produced them. Existing audio-language benchmarks provide limited evaluation of this joint requirement in multi-speaker recordings. We introduce VocalCap, a benchmark for speaker-grounded speech captioning that links fine-grained vocal description with speaker coverage, tempo- ral localization, and attribute-speaker binding. VocalCap comprises 1,000 human- annotated single-speaker clips from film and television and 1,000 controlled mix- tures containing two to six speakers. Both tracks share a 13-field schema spanning perceived speaker characteristics, prosody, articulation, affect, and speaking style. Our evaluation combines field-wise comparison against human references with deterministic temporal matching, reporting coverage, timeline accuracy, and bind- ing for multi-speaker outputs. We evaluate 14 systems on the single-speaker track and 12 on the multi-speaker track. Qwen3.5-Omni-Plus leads both, achieving a normalized single-speaker score of 72.2 and a multi-speaker score of 69.7 on their respective 0–100 scales. Cross-track rank changes show that strong single-speaker captioning does not ensure reliable multi-speaker performance. Scores for affect and style are lower than those for gender presentation and accent, although these subjective fields also have lower annotation agreement. These findings highlight the need to assess caption quality together with speaker coverage and temporal grounding. We release the annotations, source pointers, construction metadata, and evaluation code.

Primary area:
datasets and benchmarks
Keywords:
Speech Captioning, Speaker-Level Understanding, Vocal Attribute Recognition, Multi-Speaker Speech, Attribute–Speaker Binding, Speaker Localization, Audio-Language Models, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xn56OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a benchmark for speech style captioning. In this task, given a speech recording, a model should be able to describe the gender, age, pitch, accent, speed, clarity, fluency, etc. of the speech. The paper curates an evaluation dataset for this task containing 1,000 single-speaker utterances which were annotated along 13 different properties by 15 annotators.

The paper uses these 1,000 waveforms to create two versions of the benchmark: the first is the single-speaker version of the benchmark, which uses the utterances as-is. The second version of the benchmark takes a subset of these utterances and artificially mixes them into multi-speaker recordings; the utterances can be spaced apart or overlapped by a random amount to simulate conversational turn-taking. In the single-speaker evaluation, an LLM as a judge (producing a 1-5 score for each property for an utterance) is used as the evaluation. In the multi speaker case, a model must evaluate a caption for each different utterance/speaker in the mix and so additional metrics are introduced to measure this.

The paper evaluates 14 different off-the-shelf systems on the task, and compares their performance.

Critical strengths

Strengths:

  • The paper is well-written and easy to follow
Critical weaknesses

Weaknesses:

  • The contribution of the paper is extremely limited. It boils down to annotating 1,000 speech audio clips with speech style captions. This is an extremely small amount of data even for an evaluation benchmark, and several orders of magnitude short of what is needed to be useful for training models. Considering the use of the data as just an evaluation benchmark, the paper also does not make a convincing case that the 1,000 speech clips it annotates are different or more useful in some compelling way compared to existing speech style captioning datasets such as SpeechCraft or ParaSpeechCaps which contain thousands of hours (amounting to millions of utterances + captions).

  • The multi-speaker evaluation setting is poorly motivated. The multi-speaker recordings are synthetic mixes of single-speaker utterances, and not real conversations, and so there will not be any meaningful context across dialog turns that could make this problem setting interesting. The paper fails to explain how this setting cannot be addressed by simply applying speaker diarization/separation followed by single-speaker captioning

  • The single-speaker setting uses as its evaluation LLM-as-a-judge, and the judge model is from the same Qwen family as the top performing models on the benchmark.

Questions for authors

What does the proposed dataset offer that existing speech style captioning datasets, which are orders of magnitude larger and much broader in their coverage, cannot?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Xn56 (1/2)

We thank the reviewer for the careful assessment and for finding the paper well-written. We respond to each concern below and have added new experiments for the multi-speaker and judge questions. Each answer below is self-contained; all changes in the revised PDF are marked in blue, and the Common Response summarizes the points shared with the other reviewers.

Q1 / W1. What does VocalCap offer that SpeechCraft or ParaSpeechCaps cannot?

You assess VocalCap as an evaluation benchmark and ask what its 1,000 clips offer beyond these much larger datasets. That is the right comparison, and the paper did not make the case clearly enough. VocalCap does not compete with them on size. SpeechCraft and ParaSpeechCaps were built to train captioning and style-prompted TTS models, and VocalCap was built to measure audio-LLMs. Four differences matter for that purpose:

  1. Field-wise references with measured reliability. SpeechCraft's descriptions come from an automatic annotation pipeline. ParaSpeechCaps does include human labels: 342 hours (PSC-Base) alongside 2,427 automatically annotated hours (PSC-Scaled). So human annotation alone is not what separates VocalCap. The difference is what the labels are for. ParaSpeechCaps provides style tags and prompts for training style-prompted TTS. VocalCap gives every clip a human-annotated reference on each of 13 fields, reports inter-annotator agreement per field (Table S3), reviews flagged items, and defines how a model output is scored against each field.
  2. Coverage of rare and expressive conditions by design. Our clips are expressive acted speech chosen by balanced sampling over 145 observed tags (Sec. 3.4), so whispering, shouting, child speech, rare accents and disfluent speech are deliberately represented (Table S20). ParaSpeechCaps also targets rich style tags, for training. VocalCap balances an evaluation sample across all 12 attributes, so that each of these conditions has tens to hundreds of scored clips.
  3. Speaker-grounded multi-speaker output. Neither SpeechCraft nor ParaSpeechCaps evaluates a complete per-speaker description with activity intervals in multi-speaker audio. The closest prior work we know of, SpeechCaps, tests speaker–attribute associations through question answering and does not require the complete output (Table S22).
  4. A validated scoring protocol. This includes field-wise judging calibrated against humans (ρ = 0.79 on the single track; ρ = 0.84 for the multi-track 1–10 rubric in a new two-annotator check), deterministic temporal matching, and a second judge that reproduces the ranking.

Is 1,000 enough? We understand the concern about size. One way to check whether 1,000 items are enough is whether they separate systems reliably. With paired bootstrap tests (10,000 resamples; App. I.9, Table S34), 73 of 91 single-track pairs and 63 of 66 multi-track pairs differ significantly. The 95% CI half-width of the leading system is about ±0.8 points (single) and ±1.1 points (multi). The limit is also clear: systems within about 1 point (single) or 2 points (multi) of each other are not separated, and the revised PDF reports them as tied tiers. This size is consistent with widely used evaluation sets: AudioCaps test ≈ 1k clips, Clotho evaluation 1,045 clips, MMAU test-mini 1,000 items.

More than an annotated set. The contribution also includes the multi-speaker protocol and its finding: single-speaker rankings do not predict multi-speaker rankings (15 of 66 system pairs reverse; Qwen3-Omni-Instruct moves from 4th to 10th). New in the rebuttal: (i) a decomposition showing that recovering speakers and describing them are distinct abilities (Qwen3-Omni-Instruct describes recovered speakers better than MiMo-V2.5 but recovers 41% vs. 70% of them; response to biU5, Q2); (ii) the evidence below that description quality itself degrades in mixtures.

AuthorsOpenReview ↗

Response to Reviewer Xn56 (2/2)

W2. The multi-speaker setting is poorly motivated; why not diarization/separation followed by single-speaker captioning?

We agree this is the key question, and we now test it directly instead of arguing it (App. I.1–I.4 of the revised PDF):

  • Cascade baselines. We now report two diarizers alone (pyannote 3.1, DiariZen) and two cascades, DiariZen → Qwen3.5-Omni-Plus and DiariZen → Gemini-3.7-Flash, all scored with the same protocol. On the speakers that both the cascade and the end-to-end model recover, caption quality is close: +0.15 CapQ for the Qwen cascade and −0.16 for the Gemini cascade. Both cascades still trail their end-to-end counterparts, by 17.9 Multi points (95% CI [16.4, 19.5]) and 12.9 points (CI [11.3, 14.5]), because the diarizer recovers only 54% of the speakers versus 83% and 75%. For Qwen the gap grows from +2.8 points on 2-speaker mixtures to +33 to +55 points on 4–6 speakers. The hard part of the task is therefore recovering every speaker and attaching the right description to each, which single-speaker captioning does not test. VocalCap scores the output, so cascades remain legitimate entries.
  • The description attached to the correct speaker degrades in a mixture. Holding the source voice and its reference fixed across the ~4 mixtures in which it appears, judged caption quality drops by 0.45 points (1–10 rubric, SE 0.08) from no overlap to full overlap and rises by 0.26 points per +3 dB of speaker gain. The drop is at least as large for gender, age and accent (−0.56), whose references cannot change under mixing, as for the five mixing-sensitive fields (−0.42) (within-source regression over 16,666 matched speakers of the 8 strongest systems; App. I.2, Table S25).
  • Re-entry. We agree that controlled mixtures lack conversational context. We added VocalCap-RE (465 mixtures, 6–26 s, built with the main-set pipeline), where each speaker returns for 2–3 non-adjacent turns, and half pair same-gender voices. A cascade must re-identify returning speakers there. With 2–3 speakers the two approaches are close: Coverage is tied, and the end-to-end lead of +3.3 Multi (95% CI [1.2, 5.4]) is no longer significant once within-turn pauses of up to 1 s are merged in the diarizer output. The errors differ in kind: the cascade merges 25.3% of returning speakers with another speaker (29.1% for same-gender voices), versus 9.7% for end-to-end, which more often splits one speaker into two (12.1% vs. 3.5%). On a more crowded variant with 4–5 speakers (200 mixtures), the lead of end-to-end Qwen3.5-Omni-Plus grows to +16.1 Multi (CI [12.2, 19.7]) and holds under the same merging, while end-to-end Gemini-3.7-Flash is no better than its cascade (−2.1, CI [−6.9, 2.7]), so the setting also separates end-to-end models from each other.

On synthetic mixtures versus real conversation: controlled mixing is deliberate. It gives exact speaker identities and intervals and lets every speaker keep a human reference, which we could not obtain at this scale from natural conversations. It also lets us vary overlap and gain systematically, which is what the within-source analysis above relies on. We agree that natural dialogue is the necessary next step and state this limit explicitly (Sec. 7). The revised PDF adds the re-entry sets and the cascade results to strengthen the motivation (Sec. 6.2 and 7; App. I.1–I.4).

W3. The judge belongs to the same family as the top models.

We checked this directly (App. E and App. I.10 of the revised PDF). Relative to an independent DeepSeek-V4-Pro judge with identical prompts, the Qwen judge is about 2.5–3 points more lenient on all systems on the single track. The Qwen-family offset is only +0.20 points beyond that after excluding the two floor-compressed systems (+0.45 with them; exact permutation p = 0.23). On Binding, the Qwen judge is slightly stricter for Qwen systems (−0.011 vs. −0.009). The order of the 11 multi-track systems is identical under both judges, and the single-track Spearman ρ is 0.952. The top two systems keep their positions under either judge. A third judge from another provider family, GPT-OSS-120B, confirms this on six systems × 1,000 mixtures: the order is unchanged, and the Qwen judge gives Qwen systems slightly less relative credit (−0.29 Multi points, 95% CI [−0.37, −0.21]; Table S35).

We hope these results make clearer what VocalCap adds as an evaluation benchmark and show that the multi-speaker setting measures abilities that current models, and simple decompositions, do not yet solve. We would be grateful if the reviewer would reconsider the assessment in light of the new evidence.

AuthorsOpenReview ↗

Follow-up to Reviewer Xn56: Listening Audit of the References (related to Q1 / W1)

Thank you again for your assessment. We do not expect this note to resolve your main concerns about the size of the benchmark and about synthetic mixtures, which we tried to address in our response. It adds evidence on one narrower point. In our answer to Q1 / W1 we argued that, for evaluation, VocalCap offers field-wise human references whose reliability is measured. For the four most subjective fields (texture, tone, emotion, personality), the only evidence we had was inter-annotator agreement, which is moderate (κ = 0.41–0.52). We have now tested these references directly in a blind listening audit, announced in our response to Reviewer biU5. We report it here in full, including the points that limit it.

Design. We drew 200 single-speaker clips at random from the 1,000. For each clip and each of the four fields, a listener heard the clip and rated two descriptions, shown in random order and without any indication of their source: the clip's own reference, and the reference of the same field from another clip (same speaker gender, similar length, selected by a rule fixed before any rating was collected). Each description was rated on its own as supported, partially supported, unsupported, or insufficient evidence. Each clip was rated by two of six listeners, which gives 400 listener–clip pairs and 1,600 ratings of each kind of description.

Result.

FieldOwn reference: supported / partially / unsupportedMismatched reference: supported / partially / unsupportedDifference in "supported" (points, 95% CI)
Texture89.8% / 8.5% / 1.8%9.2% / 29.5% / 61.0%+80.5 [74.8, 85.5]
Tone93.5% / 6.5% / 0.0%5.8% / 17.8% / 76.2%+87.8 [83.2, 92.0]
Emotion92.8% / 7.2% / 0.0%6.2% / 20.5% / 73.0%+86.5 [81.5, 91.0]
Personality92.5% / 6.2% / 1.2%6.8% / 24.8% / 68.0%+85.8 [80.5, 90.3]
All four92.1% / 7.1% / 0.8%7.0% / 23.1% / 69.6%+85.1 [82.4, 87.8]

CIs are from a bootstrap over clips. "Insufficient evidence" was chosen 5 times, always for a mismatched description.

The clip's own reference was rated "supported" in 92.1% of ratings and "unsupported" in 0.8% (12 ratings: 7 for texture, 5 for personality, never by both listeners of a clip). A same-gender reference from another clip was rated "supported" in 7.0%. Both listeners of a clip rated the own reference "supported" in 84.4% of the 800 clip–field cases.

What we take from it, and what we do not. On the four fields where agreement is lowest, listeners accept the clip's own reference far more often than a plausible reference from another clip, and they almost never reject it. We read this as evidence that these references are specific to the clip and rarely wrong, which is what an evaluation target needs. It does not show that the reference is the only acceptable description, and it says nothing about the size of the benchmark. With 200 clips and six listeners, it is a check and not a re-annotation.

We will add the audit, with the full rating counts, to Appendix I. We appreciate the time you have given to this paper.

Reviewer_T9bqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for speaker-grounded speech captioning, called VocalCap. To improve the existing benchmarks that score isolated attributes or cover single speakers, this paper evaluate fine-grained vocal description together with speaker coverage, timing, and attribute-to-speaker binding in multi-speaker audio. So VocalCap includes two tracks to compare: the single-speaker track and the multi-speaker track. The single-speaker track isolates how well a model describes one voice on a shared 13-field schema (gender, age, accent, pitch, volume, speed, texture, clarity, fluency, emotion, tone, personality, and a free-form summary). The multi-speaker track reuses those same clips and captions inside controlled mixtures. Validation covers the judge, the protocol, and the data. The text-only judge is calibrated against human ratings and cross-checked with a second judge that reproduces the ranking. Diagnostics such as caption shuffling, oracle speaker inventories, and overlap slices confirm the multi-speaker metrics respond as intended, and a 100-segment for checking if inherited captions still hold after mixing.

Critical strengths

The target output is well defined and the evaluation protocol is specified clearly enough that there is little ambiguity about what is being measured. A model must return one item per speaker with activity intervals and a 13-field caption on a shared schema so a complete answer has a concrete shape. Each metric comes with a formula and explicit boundary handling, including how empty inventories, unmatched speakers and parsing failures are treated and speaker correspondence is resolved by deterministic temporal matching before any caption is scored.

Beyond the protocol, the direction itself is worth pursuing. Attaching fine-grained vocal attributes to individual speakers, rather than to a recording as a whole, is intriguing and necessary for any downstream use where it matters who had the non-native accent or express their emotion. Single-speaker style captioning is well covered by prior datasets and some existing works cover multi-talker captions through question answering. But to my knowledge no prior benchmark asks for a complete per-speaker description with activity intervals in multi-speaker audio. Framing the task this way and committing to a shared schema across both tracks is a useful step even if as discussed below, the current data do not yet show that it has to be solved jointly.

Critical weaknesses

My main concern is whether speaker-grounded captioning is really a joint problem. The benchmark is built on the premise that producing a complete multi-speaker description is a single capability that should be evaluated end to end. From my point of view, though, the task decomposes naturally into diarization followed by captioning, and the paper never tests whether that decomposition is sufficient. A cascade baseline would settle this: a diarizer such as pyannote produces speakers and intervals, each speaker’s segments are concatenated or masked, and any of the 14 single-speaker captioners describes each one. On short, low-overlap mixtures like these, such a cascade would likely dominate every end-to-end system on Coverage and Timeline and reduce Binding to single-speaker caption quality.

The paper’s own diagnostics already point the same way. Caption quality on matched speakers is essentially flat across overlap bins, staying between 6.28 and 6.36 on the 1 to 10 rubric, and across speaker counts, between 6.16 and 6.36, while recall drops from 0.899 to 0.698 as overlap grows. Overlap hurts finding speakers, not describing them. When the reference speaker inventory and intervals are supplied, Coverage rises from 0.851 to 0.969 and Timeline from 0.803 to 0.967, but Binding only from 0.503 to 0.560, so nearly all of the 10.8-point gain comes from structural components a diarizer would provide. The schema contains only per-speaker absolute attributes, none of which depends on the other speakers, so nothing in the task requires the captioner to see the mixture as a whole.

Finally, while the judge validation is adequate for rankings, there is still some room for variation at finer granularity. Both judges are text-only and could share a preference for similar phrasing, the primary judge belongs to the same model family as the two leading systems, and human calibration covers only the single-speaker 1 to 5 rubric, so the 1 to 10 rubric used for Binding was never checked against humans.

Questions for authors
  • Would it be possible to add a cascade baseline, an off-the-shelf diarizer followed by per-speaker captioning with two or three of the evaluated models, and report its Coverage, Timeline, Binding and Multi alongside the end-to-end systems?
  • Could you also report the diarizer alone on Coverage and Timeline, so readers can see how far the end-to-end models are from a specialized component on this data?
  • Given that caption quality on matched speakers is flat across overlap and speaker count while recall falls, what evidence supports the claim that description itself, rather than speaker recovery, degrades in multi-speaker audio?
  • Is there any subset of the benchmark, for example the 149 mixtures with overlap above 0.2, on which an end-to-end model would be expected to beat a cascade?
  • Have you considered adding relational attributes or longer mixtures with speaker re-entry, which would create cases a cascade cannot solve trivially?
5 comments
AuthorsOpenReview ↗

Response to Reviewer T9bq (1/4)

We thank the reviewer for the thorough and constructive review, and for noting that the task definition and protocol leave "little ambiguity about what is being measured". Your central question, whether the task is genuinely joint or reduces to diarization plus captioning, is the right one. We ran every experiment you suggested and report the results here; each answer is self-contained, and all changes in the revised PDF are marked in blue.

Q1 / Q2. Cascade baseline, and the diarizer alone.

Thank you for the concrete design. We implemented it as specified. We ran two diarizers, pyannote 3.1 and the stronger DiariZen (WavLM-Large + Conformer, VBx clustering), and built the cascade on DiariZen. Each predicted speaker's segments are concatenated, with frames where another speaker is active removed, and captioned with the single-speaker prompt by two of the evaluated models, Qwen3.5-Omni-Plus and Gemini-3.7-Flash. The judge (Qwen3-235B-A22B-Instruct-2507), matcher and metrics are unchanged.

System (1,000 mixtures)CoverageTimelineRecallCapQ (1–10)BindingMulti
pyannote 3.1 alone0.5930.5100.496–––
DiariZen alone0.6730.5650.539–––
DiariZen → Qwen3.5-Omni-Plus0.6730.5650.5396.530.38451.9
End-to-end Qwen3.5-Omni-Plus0.8550.8140.8306.280.49769.7
DiariZen → Gemini-3.7-Flash0.6730.5650.5396.010.34750.0
End-to-end Gemini-3.7-Flash0.7820.7370.7536.110.44262.9
End-to-end Qwen3.8-Omni-Flash0.8290.7820.7796.370.48467.5

Your prediction holds for one half. Captioning isolated crops works about as well as end-to-end captioning. On the speakers matched by both systems, the cascade's CapQ is higher with Qwen (6.58 vs. 6.43 on 1,527 speakers, Δ +0.15, 95% CI [0.10, 0.19]) and lower with Gemini (6.06 vs. 6.22 on 1,406 speakers, Δ −0.16, CI [−0.23, −0.09]). It does not hold for the other half. The diarizer, not the end-to-end model, is the weak point on these mixtures. It recovers 54% of reference speakers versus 83% (Qwen) and 75% (Gemini). It predicts 2.16 speakers per mixture where 3.06 are present and misses 21% of speech. The cascades therefore trail by 17.9 Multi points with Qwen (95% CI [16.4, 19.5]) and 12.9 with Gemini (CI [11.3, 14.5]), and would rank 8th and 9th of 12 systems. Q4 shows where this gap comes from.

Both cascades and both diarizer-only rows are now in the full multi-speaker table of the revised PDF (Table S18), with details in App. I.1 (Tables S23–S24) and a summary in Sec. 6.2. VocalCap evaluates the required output, not a specific architecture, so cascades are first-class entries.

Q3. Evidence that the description attached to the correct speaker degrades, not just speaker recovery.

You are right that the flat conditional caption score (6.28–6.36 across overlap bins) does not show this, and we should not have implied it. That statistic is computed only over recovered speakers, and recovery selects against the hardest ones. We have now performed a direct test. Each source clip appears in about 4 mixtures under different conditions, so we compare the same voice against the same reference across acoustic contexts using a within-source fixed-effects regression (8 strongest systems, 16,666 matched speakers; SEs clustered by source clip, 733 clusters):

Fields scoredown-overlap fraction 0→1per +3 dB speaker gainadded noise
All 13−0.45 (0.08)+0.26 (0.02)−0.04 (0.03)
Low mixing-sensitivity 6 (gender, age, accent, pitch, speed, fluency)−0.48 (0.09)+0.26 (0.03)−0.04 (0.03)
Gender + age + accent−0.56 (0.11)+0.27 (0.03)−0.03 (0.03)

Per system, the gain effect is significant for all 8 (+0.12 to +0.37 per 3 dB). The overlap effect is negative for all 8 (−0.22 to −0.69) and significant for 5. Its size, −0.45 on the 1–10 rubric, is comparable to the entire caption-quality spread across these 8 systems (5.76–6.37). It is as large for stable traits (gender, age, accent) as for mixing-sensitive fields, so it is not an artifact of references changing after mixing. Holding the speaker fixed, the description attached to that speaker is worse when it overlaps others or is quieter. The revised PDF corrects the wording in Sec. 6.4 and reports this analysis in App. I.2 (Table S25), in place of the conditional caption scores as evidence about description quality.

AuthorsOpenReview ↗

Response to Reviewer T9bq (2/4)

Q4. A subset where end-to-end should beat a cascade.

Multi, end-to-end Qwen3.5-Omni-Plus vs. DiariZen → Qwen3.5-Omni-Plus. Δ = end-to-end − cascade. The brackets give 95% confidence intervals from a paired bootstrap over mixtures (2,000 resamples): an interval that excludes 0 means end-to-end is significantly better on that subset. Intervals are wider for smaller subsets (394 two-speaker vs. 136 five-to-six-speaker mixtures), so their width reflects sample size, not instability at higher speaker counts.

SubsetnEnd-to-endCascadeΔ [95% CI]
2 speakers39473.971.1+2.8 [0.9, 4.6]
3 speakers30668.851.0+17.8 [15.2, 20.3]
4 speakers16465.432.4+32.9 [29.2, 36.5]
5–6 speakers13665.221.4+43.7 [39.9, 47.4]
No overlap41774.153.5+20.6 [18.1, 22.9]
Overlap > 0.215157.552.8+4.7 [1.3, 8.1]
VocalCap-RE, 2 speakers25074.072.4+1.5 [−1.2, 4.3]
VocalCap-RE, 3 speakers21560.354.9+5.4 [2.0, 8.9]
  • Speaker count is the clearest case. With two speakers the cascade is nearly on par. From four speakers on, the diarizer recovers fewer than half of the speakers and the end-to-end model leads by 33–55 points. This is the regime the benchmark is built to probe. The Gemini pair shows the same pattern: +1.0 [−1.2, 3.1] with 2 speakers, +11.4 [8.6, 14.3] with 3, +25.9 [22.2, 29.5] with 4 and +34.9 [30.9, 39.1] with 5–6.
  • Re-entry becomes a second case once more speakers return. On VocalCap-RE, Coverage is tied (0.805 for both) and end-to-end leads by +3.3 [1.2, 5.4] overall, through Timeline. That lead is sensitive to pauses inside a turn, which the diarizer marks as non-speech and the references do not: after merging same-speaker gaps of up to 1 s in the diarizer output, the difference is no longer significant (Q5). What differs robustly is the kind of identity error. The cascade merges a returning speaker with another speaker 25.3% of the time (29.1% in same-gender mixtures), versus 9.7% (12.5%) for end-to-end. End-to-end splits returning speakers more often (12.1% vs. 3.5%), so the shares of exactly tracked speakers are close (65.2% vs. 62.0%). On the crowded re-entry set with 4–5 speakers (Q5), end-to-end Qwen3.5-Omni-Plus leads by +16.1 [12.2, 19.7], and this lead holds under the same merging.
  • Heavy overlap is not such a subset. We report this openly. At overlap > 0.2, the end-to-end advantage shrinks to +4.7 (still significant), because end-to-end Multi itself drops to 57.5 there. For the Gemini pair it disappears (+0.7, CI [−2.3, 3.8]). Overlapped speech hurts both approaches, consistent with the within-source degradation in Q3.
AuthorsOpenReview ↗

Response to Reviewer T9bq (3/4)

Q5. Relational attributes, longer mixtures and speaker re-entry.

Thank you; this pinpoints a real limitation. In the released mixtures each speaker has a single contiguous turn, so the requirement to merge all of a speaker's intervals into one item is never exercised. We have built VocalCap-RE: 465 new mixtures made with the same synthesis pipeline as the main set, 6–26 s long (mean 15.5 s, versus 9.9 s in the main set), in which each of the 2–3 speakers returns for 2–3 non-adjacent turns cut at word boundaries of the same source clip (A₁B₁A₂B₂…), and half pair same-gender voices. A cascade must re-identify returning speakers, and the output still requires one item per speaker. References reuse the same 13-field captions.

VocalCap-RE (465 mixtures)CoverageTimelineRecallBindingMulti
DiariZen → Qwen3.5-Omni-Plus0.8050.6850.7640.50064.3
End-to-end Qwen3.5-Omni-Plus0.8050.7860.8070.50767.7

The two approaches are close on this set. The end-to-end model leads by +3.3 Multi (95% CI [1.2, 5.4]), almost entirely through Timeline, and this lead is sensitive to pause handling: pauses inside a turn are non-speech for the diarizer but lie inside the reference intervals. After merging same-speaker gaps of up to 1 s in the diarizer output, the cascade's Timeline rises from 0.685 to 0.752 and the Multi difference is no longer significant (between −1.2 and +1.9 across scoring variants; App. I.3). What differs robustly is the identity error (Q4): the cascade merges a returning speaker with another speaker 25.3% of the time, versus 9.7% for end-to-end, while end-to-end more often splits one speaker into two (12.1% vs. 3.5%). The RE set has only 2–3 speakers, which favors diarizers (DiariZen Coverage 0.805 here vs. 0.673 on the main set).

New: a more crowded re-entry set. Following your suggestion, we also built VocalCap-RE-Crowded with the same synthesis pipeline: 200 mixtures with 4–5 speakers, each taking two non-adjacent turns, 17–41 s long (mean 27.9 s). Half pair same-gender voices, half use partial overlap between turns, and 90 carry real background noise.

VocalCap-RE-Crowded (200 mixtures)CoverageTimelineRecallBindingMulti
DiariZen → Qwen3.5-Omni-Plus0.5190.4680.4590.29340.4
End-to-end Qwen3.5-Omni-Plus0.6330.7090.6870.42356.5
DiariZen → Gemini-3.7-Flash0.5190.4680.4590.28039.8
End-to-end Gemini-3.7-Flash0.4280.5670.4500.27237.7

The result depends on the end-to-end model. With Qwen3.5-Omni-Plus the end-to-end lead grows from +3.3 to +16.1 Multi (95% CI [12.2, 19.7]): +7.8 [2.2, 13.1] with four speakers and +24.4 [19.8, 29.1] with five. This lead holds when the same pause merging is applied (at least +11.3, CI [7.3, 15.1]). DiariZen recovers 46% of the speakers and the end-to-end model 69%. Gemini-3.7-Flash gains nothing from end-to-end processing here. Its end-to-end run ties with its cascade (−2.1, CI [−6.9, 2.7]): it outputs 5.6 speakers per mixture where 4.5 are present and recovers only 45% of them. Crowded re-entry is therefore much harder for the diarizer, but an end-to-end model does not win automatically. The strongest one does, and a weaker one is no better than its own cascade.

On relational attributes, we ran a first analysis on the existing predictions. Pitch and speed have ordinal reference labels, so for two speakers in one mixture whose labels differ, the references say who is higher-pitched or faster. We test whether a system's two descriptions imply the same order. The order is read from the two phrases by the text judge (a keyword-rule reader agrees on 94% of pairs). Volume is excluded because mixing gains change it. When the two descriptions imply an order, the four strongest systems get it right for 75–79% of same-gender pitch pairs and 77–81% of speed pairs (chance is 50%). They give no order for 7–32% of pairs. On the pairs recovered by both systems, end-to-end Gemini-3.7-Flash orders pitch better than its cascade (score 0.75 vs. 0.63 on 136 same-gender pairs, Δ +0.13, 95% CI [0.03, 0.22]). For Qwen3.5-Omni-Plus and for speed, the end-to-end model is nominally ahead but the difference is not significant (Δ +0.01 to +0.05). Hearing the speakers together therefore gives current models at most a modest relational advantage, and relative order is far from solved. This analysis is now in App. I.5 (Table S29). Sec. 7 discusses its limits and names a subset with explicit comparative questions as the next step.

Reviewer_biU5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for speaker-aware speech captioning. It comprises of 1000 single-speaker clips, and 1000 mutli-speaker clips, and evaluates on 13 attributes, while introducing 3 metrics for multi-speaker evaluation - (i) coverage measuring speaker detection, (ii) timeline measuring localization accuracy, and (iii) binding measuring attribute and caption agreement for a speaker. 14 systems are then evaluated.

Critical strengths
  • Evaluating speech description and vocal attributes while localizing them to a speaker is a very interesting problem, even more so, when going beyond standard single-speaker benchmarks.
  • The ranking differences between single-speaker vs multi-speaker evaluations, and improvement of Qwen-3.8-Omni's score when assisted with reference, highlight the limitations of single speaker evaluation.
Critical weaknesses
  • Very few clips (~10%) were independently reviewed, and the subjective attributes like emotion and personality have low agreement (Table S3).
  • Multi-speaker captions are copied from isolated recordings. Mixing voices and adding noise can change how attributes such as clarity, loudness, and emotion are perceived.
  • A correctly described speaker receives zero Binding credit if the predicted timestamps fail the IoU threshold. Consequently, poor diarization can appear as poor captioning, making the scores difficult to interpret independently.
Questions for authors
  1. How reliable are the subjective annotations, particularly emotion and personality, given their relatively low inter-annotator agreement?
  2. Can coverage, timeline, and caption quality be reported separately, to better distinguish diarization errors from captioning errors?
  3. Are the reported model differences statistically significant, particularly on the single-speaker track? How would confidence intervals change when accounting for repeated source clips?
4 comments
AuthorsOpenReview ↗

Response to Reviewer biU5 (1/3)

We thank the reviewer for recognizing the importance of speaker-level description beyond single-speaker benchmarks and the value of the cross-track ranking changes. The new analyses below answer all three questions and are added to the revised PDF (App. I), where all changes are marked in blue. Each answer below is self-contained; the Common Response summarizes the points shared with the other reviewers.

Q1 / W1. Reliability of subjective annotations (emotion, personality) and the 10% review sample.

We agree that texture, tone, emotion and personality have only moderate agreement (κ = 0.41–0.52). This is typical of perceptual affect labels: categorical emotion annotation in IEMOCAP reports Fleiss' κ ≈ 0.27. We tested whether these fields distort conclusions:

(i) Rankings without the low-agreement fields. We recomputed every score after removing the four fields with κ < 0.6:

Field set used for the scoreKendall τ vs. all 13 (single, 14 systems)Kendall τ vs. all 13 (multi, 12 systems)Top-2
Drop texture / tone / emotion / personality0.821.00unchanged
Only the 8 fields with κ ≥ 0.60.761.00unchanged
Only the 4 low-κ fields0.821.00same two systems

The multi-speaker ranking is identical under every field subset. On the single-speaker track the top two and bottom two systems keep their positions (with only the four low-κ fields, the statistically tied top two swap), and every other reordering involves systems within 2 points of each other on the full score.

(ii) The low-κ fields carry consistent signal, not noise. Their between-system spread (SD 0.26–0.32 on the 1–5 scale) matches that of high-κ fields (0.18–0.39). A system's ranking on each low-κ field correlates strongly with its ranking on the remaining fields (Spearman ρ = 0.70 texture, 0.84 tone, 0.87 personality, 0.94 emotion). If these references were mostly noise, a field's ranking would not track the other fields.

(iii) Annotation process. The 10% sample (100 clips, 1,200 decisions) is the sample used to estimate agreement; it is not the only quality control. All 15 annotators flag uncertain judgments, flagged and disagreeing items are reviewed before references enter evaluation, and unresolved fields are excluded from scoring (Sec. 3.3, App. A.3). New human ratings also show that scores on these four fields follow human judgment. In a new check of the field-wise judge, two annotators independently scored each of 350 reference–prediction phrase pairs (50 per open field, stratified across the judge's score range) on the judge's 1–10 rubric. On texture, tone, emotion and personality (200 pairs), the judge agrees with the mean human rating at Spearman ρ = 0.85 (95% CI [0.80, 0.89]; 0.83–0.89 per field), on par with the agreement between the two annotators (ρ = 0.80). This does not re-measure the references themselves, but it shows that, given a reference, the score on these fields is as reliable as a second human rater (App. I.11, Table S36). To check the references themselves, we are also running a blind listening audit. For 200 single-speaker clips drawn at random from the 1,000, two listeners per clip (from a pool of six) judge, for each of the four fields, two descriptions shown anonymously and in random order: the clip's own reference and a reference of the same field taken from another clip. Each description is rated on its own as supported, partially supported, unsupported, or insufficient evidence, so the mismatched description gives the baseline acceptance rate. We will post the result in this thread when it is complete.

(iv) Core score. The revised PDF now reports a "core" score over the eight κ ≥ 0.6 fields next to the full score for every system (App. I.7, Table S32). The same table reports the four affect/style fields with κ < 0.6 as a separate group. The top two systems are the same on both tracks under the core score (single: 77.3 and 76.2), and the multi-speaker order is identical (τ = 1.00).

AuthorsOpenReview ↗

Response to Reviewer biU5 (2/3)

W2. Mixing may change perceived clarity, loudness and emotion.

We agree, and checked whether this affects conclusions. The App. C audit already reports 99% / 95% / 90% retention for low-, moderate- and high-sensitivity field groups. New analyses:

Multi computed on…Kendall τ vs. default ranking (12 systems)
Only the 6 low-sensitivity fields (gender, age, accent, pitch, speed, fluency)1.00
Only the 5 high-sensitivity fields (volume, texture, clarity, emotion, personality)1.00
Only the 550 clean mixtures1.00
Only the 450 mixtures with added noise1.00

Mixing-sensitive fields and noisy mixtures therefore leave the system ordering unchanged. In addition, we compared each source voice with itself across the ~4 mixtures that reuse it (within-source fixed-effects regression over the 8 strongest systems, 16,666 matched speakers; App. I.2, Table S25). Overlapped or quieter speakers receive lower scores on gender, age and accent at least as much as on the five mixing-sensitive fields (−0.56 vs. −0.42 from no overlap to full overlap). Mixing cannot change these three references, so the drop does not come from references that no longer fit the mixed audio; it reflects degraded descriptions.

Q2 / W3. Separating diarization errors from captioning errors.

Thank you for this suggestion. Binding is conjunctive by design: a caption attached to an unlocalized speaker cannot be used downstream. We agree, though, that the components should also be reported separately. The new table (full version in App. I.6, Table S30) reports caption quality on matched speakers (CapQ) next to speaker recovery:

SystemCoverageTimelineRecallCapQ (own matched, 1–10)BindingMulti
Qwen3.5-Omni-Plus0.8550.8140.8306.280.49769.7
Qwen3.8-Omni-Flash0.8290.7820.7796.370.48467.5
Gemini-3.5-Flash0.6760.6270.6306.180.38553.8
MiMo-V2.50.6840.6940.6985.760.38055.5
Qwen3-Omni-Instruct0.4410.4070.4055.980.23132.5
MOSS-Audio0.5320.4970.4755.550.26138.5

We should state plainly what this table shows: the Multi ranking is driven mainly by speaker recovery. Ranking the 12 systems by Coverage alone reproduces the Multi order except for one pair that is not significantly different (Gemini-3.1-Pro vs. Qwen3-Omni-Thinking; Kendall τ = 0.97), and own-matched CapQ spans only 5.55–6.37 over the top 10 systems. The τ = 1.00 results in Q1 and W2 therefore show that the description part does not change the ranking; they do not show that it is uninformative.

Description quality does separate systems once recovery is held fixed. Each system's matched speakers are a selected subset, so we compare CapQ on the speakers that both systems of a pair recover. 61 of the 66 pairs then differ significantly (95% paired bootstrap over mixtures), and the order is not the Multi order. In each pair below, the second system has the higher Multi, but the first describes the shared speakers better:

Pair (speakers both systems recover)Shared speakersCapQ, firstCapQ, secondFirst − second [95% CI]
Qwen3-Omni-Instruct vs. MiMo-V2.59086.115.87+0.24 [0.13, 0.35]
Qwen3-Omni-Instruct vs. MOSS-Audio7396.155.61+0.54 [0.35, 0.72]
Gemini-3.5-Flash vs. MiMo-V2.51,4636.255.87+0.38 [0.29, 0.46]
Qwen3.8-Omni-Flash vs. Qwen3.5-Omni-Plus2,2236.416.35+0.06 [0.02, 0.10]

The two error sources are therefore distinct. Qwen3-Omni-Instruct describes shared speakers better than MiMo-V2.5 but recovers only 41% of the speakers against 70%; MiMo-V2.5 has the reverse profile. Multi combines the two abilities, which do not rank systems the same way, so we report CapQ next to it (App. I.6).

We also relaxed the matching threshold so that poor timestamps do not zero out a correct caption:

Binding @ IoU 0.5 (paper)Binding @ IoU 0.1
Qwen3.5-Omni-Plus0.4970.539
Qwen3-Omni-Instruct0.2310.317
Kimi-Audio0.0280.208
Kendall τ of the 12-system Multi ranking vs. IoU 0.5–1.00

Relaxing the threshold raises Binding most for systems with poor timestamps, as expected, but leaves the ranking unchanged. Both are in App. I.6 of the revised PDF (Table S30), with the IoU-0.1 variant reported as a timing-tolerant view.

AuthorsOpenReview ↗

Response to Reviewer biU5 (3/3)

Q3. Statistical significance and dependence from repeated source clips.

We ran paired bootstrap tests with 10,000 resamples of clips or mixtures (App. I.9, Table S34):

  • Single track. The 1,000 clips come from 994 distinct episodes, so clustering by source leaves the CIs essentially unchanged (±0.8 points per system). 73 of 91 pairs are significant. The top two systems (72.2 vs. 71.9, 95% CI of the difference [−0.32, 0.85]) are statistically tied, and so is most of the 68.3–69.1 group.
  • Multi track. Each of the 735 sources appears in about 4.2 mixtures. A paired mixture bootstrap gives 63 of 66 significant pairs; the three exceptions are adjacent mid-table systems within 1.7 points of each other. To test whether this reuse inflates significance, we drew 20 random subsets in which no source clip is shared (≈205 mixtures each) and compared them with random subsets of the same size and speaker-count mix that do share sources. Both give the same picture (59.0 vs. 59.4 of 66 pairs significant; Kendall τ with the full ranking 0.94 vs. 0.95), so the reuse does not inflate significance. All adjacent pairs among the top five are significant: for example, Qwen3.5-Omni-Plus vs. Qwen3.8-Omni-Flash differ by +2.2 [1.2, 3.3].

The revised PDF replaces strict ranks with tied tiers where differences are not significant and reports all pairwise tests (App. I.9, Table S34).

AuthorsOpenReview ↗

Follow-up to Reviewer biU5: Result of the Listening Audit (Q1 / W1)

Thank you again for the question about the low-agreement fields. In our response to Q1 / W1 we wrote that we were running a blind listening audit of the references themselves and would post the result in this thread. It is now complete. We report it in full, including the points that limit it.

Design. We drew 200 single-speaker clips at random from the 1,000. For each clip and each of the four fields, a listener heard the clip and rated two descriptions, shown in random order and without any indication of their source: the clip's own reference, and the reference of the same field from another clip (same speaker gender, similar length, selected by a rule fixed before any rating was collected). Each description was rated on its own as supported, partially supported, unsupported, or insufficient evidence. Each clip was rated by two of six listeners, which gives 400 listener–clip pairs and 1,600 ratings of each kind of description.

Result.

FieldOwn reference: supported / partially / unsupportedMismatched reference: supported / partially / unsupportedDifference in "supported" (points, 95% CI)
Texture89.8% / 8.5% / 1.8%9.2% / 29.5% / 61.0%+80.5 [74.8, 85.5]
Tone93.5% / 6.5% / 0.0%5.8% / 17.8% / 76.2%+87.8 [83.2, 92.0]
Emotion92.8% / 7.2% / 0.0%6.2% / 20.5% / 73.0%+86.5 [81.5, 91.0]
Personality92.5% / 6.2% / 1.2%6.8% / 24.8% / 68.0%+85.8 [80.5, 90.3]
All four92.1% / 7.1% / 0.8%7.0% / 23.1% / 69.6%+85.1 [82.4, 87.8]

CIs are from a bootstrap over clips. "Insufficient evidence" was chosen 5 times, always for a mismatched description.

The clip's own reference was rated "supported" in 92.1% of ratings and "unsupported" in 0.8% (12 ratings: 7 for texture, 5 for personality, never by both listeners of a clip). A same-gender reference from another clip was rated "supported" in 7.0%. Both listeners of a clip rated the own reference "supported" in 84.4% of the 800 clip–field cases.

Three things we should state plainly.

  1. Counting rule. Before collecting ratings we had planned to count "supported or partially supported" as acceptance. Under that rule the rates are 99% (own reference) and 30% (mismatched). After seeing the ratings of the first four listeners (266 of 400) we judged "partially supported" too weak to count as acceptance. The table therefore shows all three levels, and the difference uses "supported" only.
  2. Revised ratings. Two of the six listeners changed their ratings on 45 of their 134 clips shortly after finishing. In 57 of the 60 changed fields, the own reference moved from "supported" to "partially supported" and the mismatched one to "supported". The table uses their final ratings, which is the least favourable of the three versions for the references. With their first submissions the overall difference is +92.1 [90.1, 93.9]; with the other four listeners alone it is +88.5 [85.9, 91.2].
  3. Limits of the protocol. There was no calibration round, so listeners may place the line between "supported" and "partially supported" differently. Playback was not enforced or timed for most ratings. Descriptions were shown in English with a translation into the listeners' first language made by an AI assistant. Listeners flagged 9 clips as containing more than one voice or no intelligible speech; without them the difference is +84.9 [82.1, 87.6].

What we think this shows, and what it does not. On texture, tone, emotion and personality, listeners accept the clip's own reference far more often than a plausible reference from another clip, and they almost never reject it. We read this as evidence that these references are specific to the clip and rarely wrong. It does not show that the reference is the only acceptable description. Another annotator may choose a different label that a listener would also accept. The moderate κ (0.41–0.52) is consistent with that, and the audit does not change it. With 200 clips and six listeners, this is a check and not a re-annotation. For readers who prefer to set these fields aside, the core score over the eight κ ≥ 0.6 fields remains available (App. I.7, Table S32).

We will add the audit, with the full rating counts, to Appendix I. Thank you for prompting this check.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response to All Reviewers (1/3)

We thank all reviewers for their careful reading. Reviewers agree that attaching vocal attributes to individual speakers is useful (biU5, T9bq) and that the protocol is precisely specified (T9bq). The main shared concerns are: (C1) whether the multi-speaker task reduces to diarization followed by captioning (T9bq, Xn56); (C2) whether the Qwen-family judge favors Qwen systems (T9bq, Xn56); and (C3) statistical reliability given the benchmark size (biU5, Xn56). We ran new experiments and analyses for each. All numbers below are computed on the submitted predictions and judge records unless marked as new runs. The analysis scripts and result files are in the updated supplement. All changes in the revised PDF are marked in blue, and the new material is collected in App. I.

C1. Is speaker-grounded captioning more than "diarize, then caption"?

We agree this is the central question. VocalCap scores the output (one item per speaker, with all intervals and a 13-field caption), not a particular architecture, so cascades are legitimate entries. We have now added them and test the decomposition directly.

(a) New cascade baselines. We evaluated two diarizers, pyannote 3.1 and DiariZen (WavLM-Large + Conformer with VBx clustering), and built the cascade on the stronger one. For each predicted speaker we concatenated that speaker's non-overlapped segments and captioned them with the single-speaker prompt (Listing 1). We used two captioners from different model families: Qwen3.5-Omni-Plus, the strongest end-to-end system, and Gemini-3.7-Flash, the strongest non-Qwen system. Each cascade is compared with the end-to-end run of the same model. Scoring used the same judge (Qwen3-235B-A22B-Instruct-2507), matcher and metrics as the main table.

System (1,000 mixtures)CoverageTimelineRecallBindingCapQ (matched, 1–10)Multi
pyannote 3.1 alone0.5930.5100.496–––
DiariZen alone0.6730.5650.539–––
Cascade: DiariZen → Qwen3.5-Omni-Plus0.6730.5650.5390.3846.5351.9
End-to-end Qwen3.5-Omni-Plus (paper)0.8550.8140.8300.4976.2869.7
Cascade: DiariZen → Gemini-3.7-Flash0.6730.5650.5390.3476.0150.0
End-to-end Gemini-3.7-Flash (paper)0.7820.7370.7530.4426.1162.9
  1. Both cascades lose on speaker recovery. The Qwen cascade scores 17.9 Multi points below end-to-end Qwen3.5-Omni-Plus (paired bootstrap 95% CI [16.4, 19.5]), and the Gemini cascade 12.9 points below end-to-end Gemini-3.7-Flash (CI [11.3, 14.5]). They would place 8th and 9th of the 12 systems in the main table. The diarizer recovers 54% of reference speakers versus 83% and 75% for the two end-to-end models. It predicts 2.16 speakers per mixture where 3.06 are present: it merges speakers and misses 21% of speech.
  2. The gap grows with the number of speakers. With Qwen, end-to-end leads by +2.8 Multi on 2-speaker mixtures (CI [0.9, 4.6]), +17.8 on 3, +32.9 on 4, +42.0 on 5 and +54.5 on 6 speakers. With Gemini, the two are tied on 2-speaker mixtures (+1.0, CI [−1.2, 3.1]), and end-to-end leads by +11.4 on 3, +25.9 on 4, +32.7 on 5 and +48.5 on 6 speakers.
  3. Isolated crops are not described consistently better. On the speakers matched by both the cascade and the end-to-end run, cascade CapQ is higher with Qwen (6.58 vs. 6.43 on 1,527 speakers, Δ +0.15, 95% CI [0.10, 0.19]) and lower with Gemini (6.06 vs. 6.22 on 1,406 speakers, Δ −0.16, CI [−0.23, −0.09]). The difference is small in both directions.

So "diarize, then caption" describes the voices it finds about as well as an end-to-end model. It fails at the part that is specific to this task: recovering every speaker with their intervals and attaching each description to the right one. Multi measures exactly that, and single-speaker benchmarks cannot.

AuthorsOpenReview ↗

Common Response to All Reviewers (2/3)

(b) New evidence that the description attached to the correct speaker degrades in mixtures. T9bq correctly notes that the conditional caption score in the paper is flat across overlap bins. That statistic, however, conditions on speakers that were recovered, and the hardest speakers are the ones that go unrecovered. Each source clip appears in about 4 different mixtures with different overlap, gain and noise. We therefore fit a within-source fixed-effects regression: the same voice, the same reference, different acoustic context, pooled over the 8 strongest systems (16,666 matched speakers; SEs clustered by source clip, 733 clusters):

Caption fields scoredown-overlap 0→1per +3 dB speaker gainadded noise
All 13 fields−0.45 (SE 0.08)+0.26 (0.02)−0.04 (0.03)
Gender + age + accent only−0.56 (0.11)+0.27 (0.03)−0.03 (0.03)

For the same speaker, the description attached to that speaker gets worse when the speaker overlaps others or is quieter. The full-overlap effect (−0.45 on the 1–10 rubric) is comparable to the entire spread of conditional caption quality across the 8 strongest systems (5.76–6.37). The drop is just as large for gender, age and accent, whose references cannot change under gain or overlap. It therefore reflects model degradation, not reference drift.

(c) New re-entry extension (VocalCap-RE). T9bq rightly points out that the released mixtures give each speaker a single contiguous turn. We built 465 additional mixtures with the same synthesis pipeline as the main set (speaker sampling, ASR-guided cuts at word boundaries, boundary-smoothed mixing, the same noise types and SNR levels). They are 6–26 s long (mean 15.5 s) with 2–3 speakers; every speaker returns for 2–3 non-adjacent turns from the same source clip, and half of the mixtures pair same-gender voices. A cascade must re-link a speaker's later turns to the earlier ones, and the output must still contain one item per speaker. References reuse the same 13-field captions.

VocalCap-RE (465)CoverageTimelineRecallBindingMulti
Cascade: DiariZen → Qwen3.5-Omni-Plus0.8050.6850.7640.50064.3
End-to-end Qwen3.5-Omni-Plus0.8050.7860.8070.50767.7

On VocalCap-RE the two approaches are close. Coverage is tied, and end-to-end leads by +3.3 Multi (95% CI [1.2, 5.4]) through Timeline (+0.10): +5.4 [2.0, 8.9] with three speakers and +1.5 [−1.2, 4.3] (not significant) with two. This lead is sensitive to pauses inside a turn, which the diarizer marks as non-speech and the references do not. After merging same-speaker gaps of up to 1 s in the diarizer output, the cascade's Timeline rises from 0.685 to 0.752 and the Multi difference is no longer significant (between −1.2 and +1.9, depending on how newly matched speakers are scored and on whether the end-to-end output is merged too). What differs robustly is the kind of identity error. The cascade merges 25.3% of the 1,145 returning speakers with another speaker (29.1% in same-gender mixtures), versus 9.7% for end-to-end. End-to-end more often splits a returning speaker into two identities (12.1% vs. 3.5%), so the shares of speakers tracked exactly are close (65.2% vs. 62.0%).

More speakers. The gap opens when more speakers return. We also built a more crowded re-entry set with the same pipeline: 200 mixtures with 4–5 speakers and two turns each (17–41 s, mean 27.9 s). With Qwen3.5-Omni-Plus the end-to-end lead grows to +16.1 Multi (95% CI [12.2, 19.7]; 56.5 vs. 40.4), because DiariZen recovers 46% of the speakers and the end-to-end model 69%. This lead holds under the same pause merging (at least +11.3, CI [7.3, 15.1]). With Gemini-3.7-Flash, end-to-end and cascade are tied (37.7 vs. 39.8, Δ −2.1, CI [−6.9, 2.7]), and the end-to-end model recovers only 45% of the speakers. Crowded re-entry is harder for the diarizer, but only the strongest end-to-end model handles it better (full table in the response to T9bq, Q5).

(a)–(c) are now in App. I.1–I.4 of the revised PDF (Tables S23–S28), and the within-source analysis corrects the wording of Sec. 6.4. Both cascades and both diarizer-only rows are added as reference systems to the full multi-speaker table (Table S18) and summarized in Sec. 6.2.

AuthorsOpenReview ↗

Common Response to All Reviewers (3/3)

C2. Judge self-preference

We compared the primary Qwen3-235B judge against the secondary DeepSeek-V4-Pro judge (identical prompts, identical predictions). If the Qwen judge favored Qwen systems, Qwen systems would gain more under it.

Qwen-family systemsOther systemsDifference
Single track: Qwen-judge minus DeepSeek-judge (0–100)+2.96+2.52+0.45 (exact permutation p = 0.23)
…excluding the two floor-compressed systems (Kimi-Audio, Voxtral)+2.96+2.76+0.20
Multi track: Binding, Qwen minus DeepSeek−0.011−0.009−0.002

The Qwen judge is uniformly about 2.5–3 points more lenient on the single track and slightly stricter on Binding, with no significant family-specific offset. Rankings are preserved: single-track Spearman ρ = 0.952, and the 11 multi-track systems keep an identical order under both judges (Table S12).

New: a third judge from another provider family. We re-scored the submitted multi-speaker predictions of six systems (three Qwen-family, three others; all 1,000 mixtures, about 175k field-level calls) with GPT-OSS-120B (OpenAI), using the same prompts, matcher and metrics:

Multi (6 systems × 1,000 mixtures)Qwen3-235B (paper)GPT-OSS-120BDeepSeek-V4-Pro
Qwen3.5-Omni-Plus69.768.770.3
Qwen3.8-Omni-Flash67.566.868.2
Gemini-3.7-Flash62.961.763.7
Gemini-3.8-Flash60.059.0–
MiMo-V2.555.553.856.0
Qwen3-Omni-Thinking51.450.051.9

GPT-OSS is uniformly stricter, scoring every system 0.7–1.7 Multi points lower. Relative to it, the Qwen judge gives Qwen-family systems slightly less credit than the others (family offset −0.29 Multi points, 95% CI [−0.37, −0.21]); against DeepSeek the offset is +0.05 [−0.01, +0.12]. The system order is identical under all three judges (Kendall τ = 1.00), with the same top two. Field-level scores agree with the Qwen judge at Spearman ρ = 0.86–0.88. The GPT-OSS run also covers Gemini-3.8-Flash, which the DeepSeek comparison had to exclude because its secondary run used a different prediction snapshot.

C3. Statistical reliability and benchmark size

We now report paired bootstrap tests (10,000 resamples of clips or mixtures):

  • Single track. The 1,000 clips come from 994 distinct source episodes, so episode clustering leaves the intervals unchanged. Per-system 95% CIs are about ±0.8 points (e.g., Qwen3.5-Omni-Plus 72.2 [71.4, 73.0]), and 73 of 91 system pairs differ significantly. The top two systems (72.2 vs 71.9) are not separable, and among the six systems between 68.3 and 69.1 only 1 of 15 pairs is. The revised PDF reports these as tied tiers instead of strict ranks (App. I.9, Table S34).
  • Multi track. With a paired mixture bootstrap, 63 of 66 pairs are significant. The three exceptions are adjacent mid-table systems within 1.7 points of each other (MiMo-V2.5 vs Gemini-3.5-Flash, Gemini-3.5-Flash vs Gemini-3.1-Pro, Gemini-3.1-Pro vs Qwen3-Omni-Thinking). All adjacent pairs among the top five are significant, e.g., Qwen3.5-Omni-Plus vs Qwen3.8-Omni-Flash +2.2 [1.2, 3.3]. The 1,000 mixtures reuse 735 source clips (about 4.2 times each). To test whether this dependence inflates significance, we drew 20 random subsets in which no source clip is shared (≈205 mixtures each) and compared them with random subsets of the same size and speaker-count mix that do share sources. Both give the same picture: 59.0 vs. 59.4 of 66 pairs significant, and Kendall τ = 0.94 vs. 0.95 with the full ranking. Source reuse therefore does not overstate the separability of systems.
  • Size. For the leading system, the 95% CI half-width of Multi is 2.2 points at n = 250, 1.6 at n = 500 and 1.1 at n = 1,000. The 1,000-item size is in line with established evaluation sets (AudioCaps test ≈ 1k clips, Clotho evaluation 1,045 clips, MMAU test-mini 1,000), and it separates systems whose scores differ by about 2 points or more.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.