- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a representation-level benchmark and resource suite for fine-grained voice understanding. It evaluates how well voice–text embedding models rank fine-grained emotion and talking-style descriptions against real, in-the-wild human speech.
Critical strengths
This paper provides substantial released resources, including VoiceNet-EMO provides 40-emotion, in-the-wild human speech annotated by three psychology experts; EMOLIA / EMOLIA-BALANCED and the MOSS-Audio annotation suite are large-scale; VoiceCLAP checkpoints, per-tag files, and evaluation scripts are promised for release.
Critical weaknesses
- The paper does not clearly state whether EMO clips were excluded from the training mixture. If not, VoiceCLAP’s advantage may be overestimated.
- EMOLIA emotion scores, EMOLIA-BALANCED balanced sampling, and VoiceNet-EMO candidate sampling all use the EmpathicInsight-Voice classifier. Although released labels are expert majority votes, the classifier still shapes both training distributions and benchmark sampling, creating potential circular bias.
- Insufficient comparison fairness: VoiceCLAP-LARGE is finetuned on EMOLIA-BALANCED, whereas the Omni bases are zero-shot and general CLAPs are not trained on EMOLIA.
- The model is scored against aggregate majority labels, while each expert is scored against other raters. The authors acknowledge this, but the abstract/contributions can still be misread as “surpassing humans.”
Questions for authors
- Were the 3,944 VoiceNet-EMO clips explicitly excluded from the VoiceCLAP training mixture? Please provide the deduplication protocol and, if possible, VoiceCLAP results on the deduplicated EMO subset.
- EMOLIA emotion scores, EMOLIA-BALANCED balancing, and VoiceNet-EMO candidate sampling all use the EmpathicInsight-Voice classifier. Can you report VoiceCLAP vs. baseline results on a subset sampled without classifier preselection, or on the “non-affirmative” task types only?
- Table 1 headlines oracle-calibrated bal@pp, with held-out calibration in the appendix. Would the main ranking change if held-out bal@pp or threshold-free ρ were the primary metric? Please report the held-out numbers alongside Table 1 for all models, not only VoiceCLAP and its base.
- VoiceCLAP-LARGE underperforms the zero-shot Omni base on EmoNet-Voice and RAVDESS. Is this due to taxonomy mismatch, acted-vs-naturalistic distribution shift, or overfitting to EMOLIA?
- Seed variance is reported only for VoiceCLAP-SMALL. Can you provide at least a limited variance estimate for VoiceCLAP-LARGE, e.g., different LoRA seeds/ranks or checkpoint selection runs? This affects confidence in the top-line ranking.
5 comments
Response to jpHB (2/2): fairness, metrics, external results, and seeds
We have added matched-recipe training controls, the full held-out calibration comparison, and three Large training seeds. These additions address the requested comparisons directly and separate the effects of data choice, calibration, and checkpoint selection.
W3: fairness and data attribution. We added controls using the same training recipe. Checkpoints below use held-out training retrieval, except that Nemotron uses the final checkpoint.
| Model / data | Runs | EMO ρ | EMO AUROC | EXT AUROC |
|---|---|---|---|---|
| LCO-Omni-7B, zero-shot | — | .305 | .650 | .554 |
| Large, full mixture | 3 | .310 | .654 | .576 |
| Large, public caption corpora only | 1 | .363 | .688 | .597 |
| Small, full mixture | 3 | .256 | .628 | .554 |
| Small, mixture without Emolia-Balanced | 3 | .296 | .663 | .547 |
| Nemotron-3B, zero-shot | — | .137 | .569 | .541 |
| Nemotron-3B, fine-tuned | 1 | .277 | .647 | .579 |
The matched-recipe Nemotron run improves EMO ρ from .137 to .277 and AUROC from .569 to .647, adding evidence that the recipe can benefit another backbone (one run). The data controls show that Emolia-Balanced is not necessary for the observed gains: Small trained without Emolia-Balanced scores higher on EMO (ρ +.039 [.014, .062]). The Large public-caption control is also strong, although its single run and substantially different mixture cannot isolate one corpus's causal effect. The revision reports these useful data-choice results without attributing the improvement uniquely to Emolia-Balanced (§4.5; App. A.18).
W4: human comparison. We have removed this comparison from the abstract and results (§4.3; App. A.22). The original evaluation targets differed, and even under matched scoring a construction-aware task-type rule scores above the model. The revision makes no claim of human-level or superior perception; it evaluates voice representations against the annotated labels.
Q3: calibration and ranking. For the submitted checkpoints, rankings are similar across oracle and held-out bal@pp (Kendall τ .96 on EMO, .89 on EXT). Frozen-prediction EMO held-out values for all eleven models are:
| Model | Held-out bal@pp |
|---|---|
| Cacophony | .507 |
| CLSP | .528 |
| LAION-CLAP | .529 |
| M2D-CLAP | .532 |
| MS-CLAP | .543 |
| GLAP | .554 |
| MGA-CLAP | .559 |
| LCO-Omni-3B | .624 |
| LCO-Omni-7B | .641 |
| VoiceCLAP-Small | .654 |
| VoiceCLAP-Large | .682 |
New embedding runs give .635/.684 for the base/submitted Large, despite similarity correlations of .9999; the revised Table 1 marks which rows come from which scoring run. Holding out threshold calibration does not undo prior checkpoint selection. The revised Table 1 shows AUROC, ρ and held-out bal@pp for every model, including the three requested omni baselines. e5-omni-7B's EMO AUROC is .667, versus .654 for selected Large and .650 for its base; close estimates are not proof of equivalence.
Q4: external degradation. The IEMOCAP/CREMA-D submission numbers were few-shot probes. In the new zero-shot evaluation, selected Large improves macro AUROC on all four external sets, but top-1 falls on RAVDESS (.317 → .238) and CREMA-D (.339 → .192), is similar on EmoNet-Voice (.166 → .165), and rises on IEMOCAP (.285 → .330). Prompt variation and post-hoc standardization suggest calibration contributes, but do not establish it as the sole cause. Longer full-mixture training reduces AUROC gains. We cannot isolate taxonomy, recording distribution, and training effects from these experiments; the full numbers are in our response to wab5.
Q5: three-seed results. Large EMO ρ trajectories are:
| Seed | Ep1 | Ep2 | Ep3 | Ep4 | Ep5 | Ep6 |
|---|---|---|---|---|---|---|
| 1 | .332 | .303 | .320 | .318 | .322 | .313 |
| 2 | .342 | .366 | .338 | .312 | .311 | .320 |
| 3 | .352 | .329 | .299 | .306 | .293 | .297 |
Epoch 1 has mean .342 ± .010; retrieval-selected epoch 6 has .310 ± .012, versus base .305. The clip-bootstrap differences are +.037 [.019, .056] and +.005 [−.013, +.026]. These intervals condition on the trained models; three seeds give limited precision about future runs. The submitted .372 is above the new full-mixture seed range at epoch 1. Its saved checkpoint reproduces the submitted predictions; fresh training does not recover that point estimate, and the reason remains unresolved.
A retrospective analysis selecting epochs on half the benchmark and evaluating on the other half gives mean ρ .348 versus .303. Because benchmark results had already informed development, this is a sensitivity analysis, not a fresh held-out test; the 500 split quantiles are not independent-replication confidence intervals. The revision makes the selection dependence explicit by reporting both early-budget and retrieval-selected results with the full trajectories (Table 1, Fig. 2, §4.2; App. A.17).
Thank you for the additional experiments and clarifications. I appreciate the authors' efforts, especially the strict speaker/video exclusion experiments, which help address my concerns about data overlap.
However, my main concern about annotation reliability remains. Both VoiceNet-EMO and VoiceNet-EXT show low inter-annotator agreement, and the additional analyses do not fully resolve this issue. The improvements over existing models are also somewhat mixed, particularly when compared with stronger baselines such as e5-omni-7B.
While I recognize the value of the dataset and the additional experiments, I believe reliable annotations are essential for this paper. I will therefore maintain my current rating.
Response to jpHB (1/2): training overlap and classifier preselection
Thank you for requesting these checks. We have completed an overlap audit, exclusion retrains, and the non-affirmative analysis. The revision reports their results explicitly so that the effect of sampling and training exposure can be assessed.
Q1: were EMO clips explicitly excluded? They were not explicitly excluded in the submitted training runs. We have corrected this omission in §4.4 and App. A.16 and added the following experiments.
We audited 8,383,075 training records and matched benchmark files to Emolia-Balanced using the first 80 transcript characters and duration within 20 ms. Candidate matches occur for 3,099 of 4,000 benchmark files, corresponding to 3,862 training records. This is a candidate-matching procedure, not acoustic identity verification; short transcripts can cause false positives and absence of a match is not proof of absence from training.
Partial exclusion reruns. Three seeds of each model were retrained after excluding 3,442 candidate records. A later matching pass added 420 records that remain in those runs. For Large, retrieval-selected epoch-6 EMO ρ is .310 with and without these exclusions; the paired clip-bootstrap difference is −.001 [−.006, +.004]. At epoch 1, exclusion raises mean ρ from .342 to .357. These results do not suggest a benefit from the removed records, but cannot rule out effects of residual overlap or shared speakers/videos.
Requested unmatched subset (per-prompt AUROC):
An improvement is not established on the unmatched subset: submitted minus base is +.002 [−.034, +.040]. This subset also differs in composition: 27% of its items are speaker-reference files versus 3% of matched items. The public-caption-only control, which used no Emolia clips, shows the same broad pattern. This suggests composition matters, but does not prove overlap harmless. The revision calls this an “unmatched subset,” not a verified clean test set (§4.4; App. A.16).
Stricter speaker/video exclusion. The benchmark is not speaker-disjoint: 3,394/3,718 identified speakers have other clips in Emolia-Balanced, and 92,035 training clips share a speaker or source video with a benchmark file. We completed three Small retrains excluding all 92,828 records that are a candidate match or share a speaker/video with a benchmark file. At the retrieval-selected epoch, EMO ρ is .248 ± .017 versus .256 ± .012 for the full mixture; at epoch 1 it is .298 ± .011 versus .314 ± .007. The selected-checkpoint difference is small relative to the observed seed variation; this is not an equivalence result. The early reduction is consistent with a modest overlap benefit. Update, 16 Nov: the strict exclusion is now complete for Large too (reply below). Loader replays cannot replace unavailable original batch logs (§4.4; App. A.16).
Q2: classifier dependence. Human labeling does not remove the dependence introduced by sampling. The same stored classifier scores helped construct Emolia-Balanced and EMO's affirmative and contrast items. A rule using only task type—present for affirmative, absent otherwise—scores balanced accuracy .716 and ρ .451; the stored classifier scores reach AUROC .693. These are construction-aware diagnostic references, not independent audio baselines.
Results within task type (AUROC):
The submitted-model differences from the base are +.004 [−.022, +.029] and +.001 [−.033, +.036]. Thus the pooled advantage does not establish better discrimination within either stratum. e5-omni-7B's non-affirmative result suggests some recoverable signal, although these are exploratory, unadjusted comparisons. Low within-stratum human agreement further limits interpretation.
EXT also has model-assisted sampling: an early VoiceCLAP prototype and sparse autoencoders grouped candidates before Gemini screening. The revision documents this dependence as well (§3.3; App. A.15).
We do not yet have a benchmark subset sampled independently of the classifier. The requested non-affirmative analysis is available above, but it too is classifier-selected. The revision limits EMO's interpretation to performance on an explicitly enriched sample and reports pooled and stratified scores together (§3.3, §4.3, Table 2). A new independently sampled evaluation would be needed to assess broader coverage; we do not claim the current analyses substitute for one.