acceptodds
Under review as a conference paper at ICLR 2027

VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale

Abstract

Expressive speech synthesis has outpaced expressive speech perception: systems now render fine-grained vocal performances that no public benchmark can score. Most benchmarks for this inverse problem stop at six to nine basic emotion categories, largely on acted speech. This paper introduces VoiceNet, a human-annotated representation-level benchmark for voice performance understanding on permissively-licensed in-the-wild speech. VoiceNet has two subsets: VoiceNet-Emo applies a 40-emotion taxonomy with three expert ratings per item, and VoiceNet-Ext, a preliminary subset, scores 57 talking-style attributes including speaking rate, vocal tension, breathiness, and register. The paper also releases Emolia, an emotion-annotated version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter VoiceCLAP-Small for fast large-scale data filtering and a 7B VoiceCLAP-Large for state-of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on VoiceNet-Emo. On VoiceNet-Emo, VoiceCLAP-Large aligns more closely with the aggregate expert consensus than individual experts agree with one another: a comparison against the majority label rather than evidence of surpassing human emotion perception. All systems evaluated here are voice-text embedding models: VoiceNet scores representation-level attribute recognition and retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering uncurated speech corpora into subsets that span diverse talking styles and emotions remains an open challenge; VoiceCLAP embeddings offer a promising tool for this task. VoiceNet, Emolia, and VoiceCLAP will be released for research use.

Primary area:
datasets and benchmarks
Keywords:
speech emotion recognition, paralinguistics, talking style, audio-text contrastive learning, voice-text embeddings, fine-grained emotion taxonomy, benchmark, expert annotation, inter-rater reliability, in-the-wild speech

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jpHBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a representation-level benchmark and resource suite for fine-grained voice understanding. It evaluates how well voice–text embedding models rank fine-grained emotion and talking-style descriptions against real, in-the-wild human speech.

Critical strengths

This paper provides substantial released resources, including VoiceNet-EMO provides 40-emotion, in-the-wild human speech annotated by three psychology experts; EMOLIA / EMOLIA-BALANCED and the MOSS-Audio annotation suite are large-scale; VoiceCLAP checkpoints, per-tag files, and evaluation scripts are promised for release.

Critical weaknesses
  1. The paper does not clearly state whether EMO clips were excluded from the training mixture. If not, VoiceCLAP’s advantage may be overestimated.
  2. EMOLIA emotion scores, EMOLIA-BALANCED balanced sampling, and VoiceNet-EMO candidate sampling all use the EmpathicInsight-Voice classifier. Although released labels are expert majority votes, the classifier still shapes both training distributions and benchmark sampling, creating potential circular bias.
  3. Insufficient comparison fairness: VoiceCLAP-LARGE is finetuned on EMOLIA-BALANCED, whereas the Omni bases are zero-shot and general CLAPs are not trained on EMOLIA.
  4. The model is scored against aggregate majority labels, while each expert is scored against other raters. The authors acknowledge this, but the abstract/contributions can still be misread as “surpassing humans.”
Questions for authors
  1. Were the 3,944 VoiceNet-EMO clips explicitly excluded from the VoiceCLAP training mixture? Please provide the deduplication protocol and, if possible, VoiceCLAP results on the deduplicated EMO subset.
  2. EMOLIA emotion scores, EMOLIA-BALANCED balancing, and VoiceNet-EMO candidate sampling all use the EmpathicInsight-Voice classifier. Can you report VoiceCLAP vs. baseline results on a subset sampled without classifier preselection, or on the “non-affirmative” task types only?
  3. Table 1 headlines oracle-calibrated bal@pp, with held-out calibration in the appendix. Would the main ranking change if held-out bal@pp or threshold-free ρ were the primary metric? Please report the held-out numbers alongside Table 1 for all models, not only VoiceCLAP and its base.
  4. VoiceCLAP-LARGE underperforms the zero-shot Omni base on EmoNet-Voice and RAVDESS. Is this due to taxonomy mismatch, acted-vs-naturalistic distribution shift, or overfitting to EMOLIA?
  5. Seed variance is reported only for VoiceCLAP-SMALL. Can you provide at least a limited variance estimate for VoiceCLAP-LARGE, e.g., different LoRA seeds/ranks or checkpoint selection runs? This affects confidence in the top-line ranking.
5 comments
AuthorsOpenReview ↗

Response to jpHB (1/2): training overlap and classifier preselection

Thank you for requesting these checks. We have completed an overlap audit, exclusion retrains, and the non-affirmative analysis. The revision reports their results explicitly so that the effect of sampling and training exposure can be assessed.

Q1: were EMO clips explicitly excluded? They were not explicitly excluded in the submitted training runs. We have corrected this omission in §4.4 and App. A.16 and added the following experiments.

We audited 8,383,075 training records and matched benchmark files to Emolia-Balanced using the first 80 transcript characters and duration within 20 ms. Candidate matches occur for 3,099 of 4,000 benchmark files, corresponding to 3,862 training records. This is a candidate-matching procedure, not acoustic identity verification; short transcripts can cause false positives and absence of a match is not proof of absence from training.

Partial exclusion reruns. Three seeds of each model were retrained after excluding 3,442 candidate records. A later matching pass added 420 records that remain in those runs. For Large, retrieval-selected epoch-6 EMO ρ is .310 with and without these exclusions; the paired clip-bootstrap difference is −.001 [−.006, +.004]. At epoch 1, exclusion raises mean ρ from .342 to .357. These results do not suggest a benefit from the removed records, but cannot rule out effects of residual overlap or shared speakers/videos.

Requested unmatched subset (per-prompt AUROC):

Matching statusItemsBaseSubmittedThree seeds, ep1Three seeds, ep6
Candidate match6,187.659.690.675.653
No match found1,799.622.625.614.600

An improvement is not established on the unmatched subset: submitted minus base is +.002 [−.034, +.040]. This subset also differs in composition: 27% of its items are speaker-reference files versus 3% of matched items. The public-caption-only control, which used no Emolia clips, shows the same broad pattern. This suggests composition matters, but does not prove overlap harmless. The revision calls this an “unmatched subset,” not a verified clean test set (§4.4; App. A.16).

Stricter speaker/video exclusion. The benchmark is not speaker-disjoint: 3,394/3,718 identified speakers have other clips in Emolia-Balanced, and 92,035 training clips share a speaker or source video with a benchmark file. We completed three Small retrains excluding all 92,828 records that are a candidate match or share a speaker/video with a benchmark file. At the retrieval-selected epoch, EMO ρ is .248 ± .017 versus .256 ± .012 for the full mixture; at epoch 1 it is .298 ± .011 versus .314 ± .007. The selected-checkpoint difference is small relative to the observed seed variation; this is not an equivalence result. The early reduction is consistent with a modest overlap benefit. Update, 16 Nov: the strict exclusion is now complete for Large too (reply below). Loader replays cannot replace unavailable original batch logs (§4.4; App. A.16).

Q2: classifier dependence. Human labeling does not remove the dependence introduced by sampling. The same stored classifier scores helped construct Emolia-Balanced and EMO's affirmative and contrast items. A rule using only task type—present for affirmative, absent otherwise—scores balanced accuracy .716 and ρ .451; the stored classifier scores reach AUROC .693. These are construction-aware diagnostic references, not independent audio baselines.

Results within task type (AUROC):

ItemsBaseSubmittedThree seeds, ep1Three seeds, ep6Public captions onlye5-omni-7B
Affirmative, n=3,999.585.589.578.565.614.593
Non-affirmative, n=3,987.504.505.514.501.527.563

The submitted-model differences from the base are +.004 [−.022, +.029] and +.001 [−.033, +.036]. Thus the pooled advantage does not establish better discrimination within either stratum. e5-omni-7B's non-affirmative result suggests some recoverable signal, although these are exploratory, unadjusted comparisons. Low within-stratum human agreement further limits interpretation.

EXT also has model-assisted sampling: an early VoiceCLAP prototype and sparse autoencoders grouped candidates before Gemini screening. The revision documents this dependence as well (§3.3; App. A.15).

We do not yet have a benchmark subset sampled independently of the classifier. The requested non-affirmative analysis is available above, but it too is classifier-selected. The revision limits EMO's interpretation to performance on an explicitly enriched sample and reports pooled and stratified scores together (§3.3, §4.3, Table 2). A new independently sampled evaluation would be needed to assess broader coverage; we do not claim the current analyses substitute for one.

AuthorsOpenReview ↗

Response to jpHB (2/2): fairness, metrics, external results, and seeds

We have added matched-recipe training controls, the full held-out calibration comparison, and three Large training seeds. These additions address the requested comparisons directly and separate the effects of data choice, calibration, and checkpoint selection.

W3: fairness and data attribution. We added controls using the same training recipe. Checkpoints below use held-out training retrieval, except that Nemotron uses the final checkpoint.

Model / dataRunsEMO ρEMO AUROCEXT AUROC
LCO-Omni-7B, zero-shot—.305.650.554
Large, full mixture3.310.654.576
Large, public caption corpora only1.363.688.597
Small, full mixture3.256.628.554
Small, mixture without Emolia-Balanced3.296.663.547
Nemotron-3B, zero-shot—.137.569.541
Nemotron-3B, fine-tuned1.277.647.579

The matched-recipe Nemotron run improves EMO ρ from .137 to .277 and AUROC from .569 to .647, adding evidence that the recipe can benefit another backbone (one run). The data controls show that Emolia-Balanced is not necessary for the observed gains: Small trained without Emolia-Balanced scores higher on EMO (ρ +.039 [.014, .062]). The Large public-caption control is also strong, although its single run and substantially different mixture cannot isolate one corpus's causal effect. The revision reports these useful data-choice results without attributing the improvement uniquely to Emolia-Balanced (§4.5; App. A.18).

W4: human comparison. We have removed this comparison from the abstract and results (§4.3; App. A.22). The original evaluation targets differed, and even under matched scoring a construction-aware task-type rule scores above the model. The revision makes no claim of human-level or superior perception; it evaluates voice representations against the annotated labels.

Q3: calibration and ranking. For the submitted checkpoints, rankings are similar across oracle and held-out bal@pp (Kendall τ .96 on EMO, .89 on EXT). Frozen-prediction EMO held-out values for all eleven models are:

ModelHeld-out bal@pp
Cacophony.507
CLSP.528
LAION-CLAP.529
M2D-CLAP.532
MS-CLAP.543
GLAP.554
MGA-CLAP.559
LCO-Omni-3B.624
LCO-Omni-7B.641
VoiceCLAP-Small.654
VoiceCLAP-Large.682

New embedding runs give .635/.684 for the base/submitted Large, despite similarity correlations of .9999; the revised Table 1 marks which rows come from which scoring run. Holding out threshold calibration does not undo prior checkpoint selection. The revised Table 1 shows AUROC, ρ and held-out bal@pp for every model, including the three requested omni baselines. e5-omni-7B's EMO AUROC is .667, versus .654 for selected Large and .650 for its base; close estimates are not proof of equivalence.

Q4: external degradation. The IEMOCAP/CREMA-D submission numbers were few-shot probes. In the new zero-shot evaluation, selected Large improves macro AUROC on all four external sets, but top-1 falls on RAVDESS (.317 → .238) and CREMA-D (.339 → .192), is similar on EmoNet-Voice (.166 → .165), and rises on IEMOCAP (.285 → .330). Prompt variation and post-hoc standardization suggest calibration contributes, but do not establish it as the sole cause. Longer full-mixture training reduces AUROC gains. We cannot isolate taxonomy, recording distribution, and training effects from these experiments; the full numbers are in our response to wab5.

Q5: three-seed results. Large EMO ρ trajectories are:

SeedEp1Ep2Ep3Ep4Ep5Ep6
1.332.303.320.318.322.313
2.342.366.338.312.311.320
3.352.329.299.306.293.297

Epoch 1 has mean .342 ± .010; retrieval-selected epoch 6 has .310 ± .012, versus base .305. The clip-bootstrap differences are +.037 [.019, .056] and +.005 [−.013, +.026]. These intervals condition on the trained models; three seeds give limited precision about future runs. The submitted .372 is above the new full-mixture seed range at epoch 1. Its saved checkpoint reproduces the submitted predictions; fresh training does not recover that point estimate, and the reason remains unresolved.

A retrospective analysis selecting epochs on half the benchmark and evaluating on the other half gives mean ρ .348 versus .303. Because benchmark results had already informed development, this is a sensitivity analysis, not a fresh held-out test; the 500 split quantiles are not independent-replication confidence intervals. The revision makes the selection dependence explicit by reporting both early-budget and retrieval-selected results with the full trajectories (Table 1, Fig. 2, §4.2; App. A.17).

Thank you for the additional experiments and clarifications. I appreciate the authors' efforts, especially the strict speaker/video exclusion experiments, which help address my concerns about data overlap.

However, my main concern about annotation reliability remains. Both VoiceNet-EMO and VoiceNet-EXT show low inter-annotator agreement, and the additional analyses do not fully resolve this issue. The improvements over existing models are also somewhat mixed, particularly when compared with stronger baselines such as e5-omni-7B.

While I recognize the value of the dataset and the additional experiments, I believe reliable annotations are essential for this paper. I will therefore maintain my current rating.

Reviewer_MrWzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces VOICENET, a benchmark for fine-grained voice understanding, consisting of VOICENET-EMO with 40 emotion categories and VOICENET-EXT with 57 vocal attributes. The authors also construct EMOLIA, a large-scale speech dataset with automatically generated emotion and vocal attribute annotations, and train two audio-text contrastive models, VOICECLAP-SMALL (110M) and VOICECLAP-LARGE (7B). Experiments on VOICENET and four public speech emotion benchmarks demonstrate improvements over general-purpose CLAP baselines. VOICECLAP-LARGE achieves the best performance on VOICENET and two external benchmarks. However, VOICENET-EXT shows very low inter-annotator agreement, and performance gains over the pretrained base model are inconsistent across external datasets, raising questions about annotation reliability and generalization.

Critical strengths

A More Comprehensive Voice Understanding Benchmark: The paper extends speech emotion evaluation beyond conventional emotion categories by introducing 40 fine-grained emotions and 57 vocal attributes. Using real-world speech and human annotations makes this a valuable step toward evaluating more diverse aspects of vocal expression. Large-Scale Data Construction and Empirical Improvements: The authors construct large-scale datasets with detailed voice annotations and train two contrastive models. The models outperform existing general-purpose CLAP baselines, demonstrating the potential of fine-grained voice-text supervision for improving speech understanding.

Critical weaknesses

Low Annotation Reliability: Both VOICENET subsets exhibit low inter-annotator agreement, particularly VOICENET-EXT (Fleiss' κ = -0.022), where none of the 57 attributes meets the authors' reliability criteria. This raises concerns about the validity of the benchmark and whether model performance reflects meaningful voice understanding.

Limited Generalization to Fine-Grained Voice Understanding: Although the proposed models outperform general-purpose CLAP baselines, improvements over the pretrained LCO-Omni-7B model are inconsistent across external datasets. In particular, VOICECLAP-LARGE performs worse on EmoNet-Voice, which shares the same 40-emotion taxonomy. Furthermore, no independent external evaluation is provided for the 57 vocal attributes, making it difficult to assess the generalizability of the claimed improvements.

Questions for authors

Annotation Reliability: What are the main causes of the low inter-annotator agreement, particularly in VOICENET-EXT? Can the authors provide further evidence that the current annotations are reliable enough to support meaningful model comparisons?

Fine-Grained Emotion Generalization: VOICECLAP-LARGE underperforms its pretrained base model on EmoNet-Voice, which uses the same 40-emotion taxonomy. Can the authors explain this performance degradation and clarify whether the improvements on VOICENET-EMO generalize to other datasets?

Vocal Attribute Understanding: Given the low annotation agreement in VOICENET-EXT, how can the authors establish that the reported improvements reflect genuine vocal attribute understanding? Can the authors provide additional validation using independently annotated data?

5 comments
AuthorsOpenReview ↗

Response to MrWz (1/2): annotation reliability

We have addressed this concern with annotator-level diagnostics, alternative label aggregation, acoustic-proxy checks, and an external attribute evaluation. These analyses distinguish which model comparisons persist across labeling choices from which claims require stronger labels. The revised paper treats EXT as exploratory and evaluates attribute transfer separately on ParaSpeechCaps; the latter results are in our next reply.

EXT: disagreement is concentrated in particular annotators. In the November 13 export (43,603 ratings; 18,632 items including the language category), three of eight annotators have maximum pairwise Cohen κ of only .02, .02 and .04. Together they supplied 49% of the ratings. The remaining annotators have κ of .22–.29 with the main annotator. These observations identify different response patterns, not their psychological cause. Three of the four psychology-degree annotators described in the paper are in the low-agreement group; qualifications should not be presented as evidence of label reliability.

We examined a post-hoc rule retaining annotators with κ ≥ .05 against the main annotator on at least 30 shared items. No model score enters this rule.

Annotation setItems with ≥2 ratingsKrippendorff α [95% CI]
All annotators17,174−.02 [−.03, −.01]
Five retained annotators3,536.19 [.15, .22]

Agreement remains low. Only 19% of items have two retained ratings, and the main annotator is involved in every retained pair. An exploratory 20/57 attributes reach α ≥ .20 with a positive bootstrap lower bound, but this changes the original reliable-core criterion and has no multiplicity correction. It does not establish a validated reliable core.

Two further sensitivity checks support investigating these annotator differences: a Dawid–Skene model also assigns low informativeness to the same three annotators; and votes from retained annotators align more with four acoustic proxies than votes from excluded annotators. Neither establishes ground truth. The proxies cover only four attributes, and acoustic onset rate is a weak proxy for perceived tempo.

What happens to model comparisons? The following are per-prompt AUROCs. Large is the three-seed mean at the retrieval-selected checkpoint.

EXT labelsBaseLargee5-omni-7B
Submitted labels; ties treated as absent.554.576.588
Main annotator only.583.613.632
Dawid–Skene.579.607.630
≥2 retained ratings, ties excluded.634.673.721

The direction of the Large–base difference persists across these label sets, but their absolute scores are not directly comparable: item populations differ. On the same items as the last row, using all annotators gives base AUROC .636. Thus the higher last-row score is not evidence that screening repaired the labels. The clip-bootstrap difference for submitted labels is +.022 [.010, .035], conditional on the trained models; seed SD is .011 and there are only three seeds. e5-omni-7B is competitive throughout.

EMO: ranking stability and rater sensitivity. The submitted models' ranking is relatively stable under individual-rater labels (Kendall τ .78–.96 versus the consensus ranking), and the gap between model families does not depend on the rater: under each expert's labels alone, both zero-shot LCO-Omni bases score above all seven general-audio CLAPs (weakest omni AUROC .716, .581, .545 versus best CLAP .573, .528, .521). Its three experts label 41%, 58% and 72% of pairs as emotion-present; pairwise κ is .23, .08 and .03, and agreement within affirmative/non-affirmative strata is weak (κ .08/−.09). Ranking stability therefore supports the robustness of the observed ordering to rater choice, but cannot alone establish validity. The revision retains the original labels and reports these sensitivity analyses (§3.3, §4.3).

The revision separates the exploratory EXT analysis (App. A.15) from the external evidence of attribute transfer. It now provides the requested diagnostics of disagreement, quantifies sensitivity to label aggregation, and includes evaluation against labels outside EXT. Agreement in EXT remains insufficient for confirmatory model ranking. Additional annotation is ongoing; the evidence in these replies is already complete.

AuthorsOpenReview ↗

Response to MrWz (2/2): external emotion and attribute validation

We have completed the requested external attribute validation and expanded the emotion evaluation to three training seeds. The results provide evidence of transfer beyond VoiceNet: Large improves per-emotion AUROC on all four external datasets and attribute AUROC on ParaSpeechCaps, including its human-annotated tags. We report top-1 accuracy alongside these ranking results because the two metrics reveal different behavior.

Q2: EmoNet-Voice degradation. The submitted model's zero-shot top-1 falls from .166 to .155 (difference −.011, 95% CI [−.017, −.005]). We retain this result. In three new runs, top-1 is .164 at epoch 1 and .165 at the retrieval-selected epoch 6, versus .166 for the base; the epoch-6 difference is −.001 [−.006, +.004]. Its macro one-vs-rest AUROC rises from .663 to .686 (+.023 [.020, .026]).

The two metrics ask different questions: top-1 compares emotion prompts for a clip; AUROC ranks clips for each emotion prompt. Better AUROC does not cancel the top-1 loss. Prompt sensitivity and post-hoc score standardization suggest cross-prompt calibration contributes, but do not isolate the cause. A nominal taxonomy mismatch is not needed to explain the result because both sets use 40 emotions; differences in audio and label distributions may still matter.

Across four external emotion sets, the retrieval-selected models improve AUROC over the base, but top-1 falls on RAVDESS and CREMA-D. The revision reports this mixed result and separates zero-shot evaluation from the IEMOCAP/CREMA-D few-shot probes previously described incorrectly (§4.6, Table 3; App. A.19). Longer training reduces the external AUROC gains. This is consistent with a mismatch between the training objective and evaluation tasks, without proving a specific causal explanation.

Q3: independent attribute validation. We evaluated ParaSpeechCaps on 14,026 holdout clips excluded from the training data. This supplies the requested test against labels outside VoiceNet-EXT. We report its 35 human-annotated tags separately from the full 57-tag set, which also includes 15 automatic basic tags and seven metadata-derived accent tags. We additionally map 34 tags to EXT rubric prompts; this tests partial attribute coverage and does not validate the original EXT labels.

EvaluationBase AUROCLarge, three seedsDifference [95% CI]e5-omni-7B
All 57 PSC tags.588.639 ± .020+.051 [.037, .062].647
35 human-annotated PSC tags.592.618 ± .019+.026 [.007, .040].647
34 mapped tags, EXT rubric prompts.570.633 ± .020+.063 [.042, .083].577
Mapped tags, unseen speakers.541.603 ± .014+.062 [.027, .097].574

Large uses the retrieval-selected epoch-6 checkpoints (revised §4.6; App. A.20). ± is seed SD; paired intervals resample speakers and condition on the trained models. The unseen-speaker analysis contains only 2,646 clips from ten speakers. Its 17 human-annotated tags give an inconclusive +.042 [−.021, +.109]. These limits prevent a strong claim of generalization across unseen speakers and all human-rated attributes.

The public-caption-only control reaches .711 on the 57-tag evaluation (one run). Its training captions come from other clips of the same corpora, so this evaluation is in-domain for that control; the full mixture also contains these caption corpora. e5-omni-7B is competitive with PSC prompts. These results support attribute transfer to held-out clips under the tested prompts, but not a unique benefit of Emolia-Balanced or uniformly better performance than other omni embeddings.

A complementary acoustic check. On 1,139 EXT clips covering four attributes, model similarities correlate more with the specified acoustic proxies after fine-tuning: mean signed Spearman ρ .163 versus .054 for the base, paired difference +.109 [.048, .169]. These are limited proxy measurements, not independent validation of the full taxonomy or its labels (App. A.21).

Presentation. The revised main text is organised around EMO and its stratified results (§4.1–4.3), separates data construction (§3.2–3.3, with a pipeline figure) from model training (§3.4), and places the EXT reliability analyses in App. A.15. The completed external evaluations add the missing evidence for emotion and attribute transfer, with its scope quantified above. We hope this evidence and the clearer presentation help you reassess the contribution.

Thank you to the authors for the detailed rebuttal and extensive additional experiments. I believe that fine-grained emotion and vocal attribute understanding is an important research direction, and I appreciate the substantial effort invested in large-scale data construction, human annotation, and model evaluation. The new experiments on external datasets also provide encouraging evidence of generalization.

However, I still have concerns about the reliability of the results. First, annotation agreement on VoiceNet-EXT remains low. While the additional analyses help explain some of the disagreement, the underlying label reliability issue remains unresolved. Second, potential overlap between training and evaluation data in the original experiments has not been fully ruled out, and strict speaker/video-level exclusion has not yet been completed for the Large model. These issues make it difficult to fully trust the reported improvements.

Overall, I appreciate the research direction, the potential value of the released resources, and the authors' efforts during the rebuttal. However, I consider annotation reliability and proper separation between training and evaluation data particularly important for a benchmark paper. Addressing these issues would make this a much more valuable and convincing contribution. For now, I will maintain my original rating of 1 (Clear Rejection).

Reviewer_wab5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a human-annotated representation-level benchmark for voice performance understanding, called VoiceNet. VoiceNet has two subsets, VoiceNet-Emo (40-emotion taxonomy) and VoiceNet-Ext (57 talking-style). The paper also releases Emolia, an emotion-annotated version of the Emilia corpus. Two voice-text contrastive baselines train on this data: VoiceClap-Small (110M) and VoiceClap-Large (7B). On VoiceNet-Emo, the VoiceClap achieve the SOTA performance. On 4 public speech-emotion benchmarks, VoiceClap wins two of four.

Critical strengths
  1. The paper makes a comprehensive contribution, including a benchmark, training data, and two CLAP models.
  2. The open-sourcing of VOICENET, EMOLIA, and VOICECLAP will be a valuable contribution to the community; beyond their utility as CLAP representation models, they may also prove beneficial for emotion-aware TTS.
Critical weaknesses
  1. The novelty is rather limited; it lacks any truly original ideas.
  2. On out-of-domain public benchmarks, it shows no significant advantage over zero-shot Omni-Embedding bases.
  3. A lack of sufficient comparisons involving Omni Embeddings, such as E5-Omni, EmbeddingGemma 2, and Omni-Embed-Nemotron.
Questions for authors
  1. For what other tasks might this data be potentially valuable?
  2. Have you tried other Omni Embeddings?
2 comments
AuthorsOpenReview ↗

Response to wab5: baselines, generalization, and contribution

Thank you for recognizing the resources' value. We have completed all three requested omni baselines and a four-dataset zero-shot evaluation across three Large training seeds. The additions strengthen the comparison and show consistent gains in external per-emotion ranking, over the base and over all three requested omni baselines, with mixed top-1 results as detailed below.

W3 / Q2: the requested baselines. We evaluated all three zero-shot with the same benchmark prompts. “Selected” below means caption–audio retrieval on held-out training clips; it does not use VoiceNet labels to select the checkpoint. Held-out bal@pp calibrates thresholds on other benchmark folds and is a separate operation.

ModelEMO held-out bal@ppEMO ρEMO AUROCEXT AUROC
e5-omni-7B.646.306.667.588
EmbeddingGemma 2.612.254.630.557
Omni-Embed-Nemotron-3B.550.137.569.541
LCO-Omni-7B base.635.305.650.554
Large, selected epoch 6, three seeds.649.310.654.576
Large, epoch 1, three seeds.661.342.675.566
Large, submitted / benchmark-selected.684.372.691.587

These additions make Table 1 a broader comparison of voice representations: e5-omni-7B is a strong zero-shot reference, while fine-tuning benefits depend on the evaluation and checkpoint. We have removed the blanket state-of-the-art claim. Table 1's frozen-prediction base/submitted held-out values are .641/.682, versus .635/.684 in the rescored rows above. Held-out bal@pp changes by up to .006 between embedding runs of the same base model, so small differences in this metric should not be overinterpreted.

W2: external generalization. The submission mixed protocols: IEMOCAP and CREMA-D were few-shot probes, not zero-shot. We have now evaluated all four sets zero-shot. Each cell below is base → selected epoch-6 Large, with Large averaged over three seeds.

DatasetMacro one-vs-rest AUROCZero-shot top-1Few-shot probe
EmoNet-Voice.663 → .686.166 → .165—
RAVDESS.726 → .780.317 → .238—
CREMA-D.631 → .746.339 → .192.209 → .416
IEMOCAP.700 → .745.285 → .330.260 → .283

The AUROC differences are +.023 [.020, .026], +.054 [.044, .069], +.115 [.110, .120] and +.045 [.034, .060], respectively. Intervals resample speakers/actors where available and examples for EmoNet-Voice; IEMOCAP has only ten speakers. They condition on the three trained models.

The selected Large also exceeds all three requested omni baselines in macro AUROC on all four datasets. Against e5-omni-7B, the strongest of them (.636, .709, .704, .712), the paired differences are +.050 [.046, .053], +.072 [.061, .084], +.042 [.037, .047] and +.033 [.027, .045]. In top-1 it is above e5-omni-7B on EmoNet-Voice and IEMOCAP, similar on RAVDESS and below on CREMA-D (.192 versus .280). App. A.19 lists all models.

These results support improved ranking of clips for an emotion prompt, not a general improvement in choosing the correct emotion among prompts. RAVDESS and CREMA-D top-1 deteriorate. Prompt wording and score calibration matter: the submitted model's RAVDESS top-1 varies from .277 to .501 across templates. Post-hoc standardization over unlabelled test clips reduces the losses, but uses a different, transductive protocol and does not replace the original results. We withdraw “wins two of four”; the revised §4.6 and Table 3 report both protocols, with prompt ranges and intervals in App. A.19.

W1: contribution. The contribution is the combination of fine-grained in-the-wild evaluation and annotation scale: expert judgments across 40 emotions, emotion annotations on 71.78M Emilia clips, and the MOSS-Audio suite on 8.64M clips. This enables comparison across general-audio CLAPs and omni embeddings and analysis of transfer, prompt sensitivity and classifier-assisted sampling. The new controls also yield a useful data-selection finding: the public-caption-only model reaches EMO ρ .363 and AUROC .688 at its retrieval-selected checkpoint (one run), so the gains cannot be uniquely attributed to Emolia-Balanced. The resource and empirical contributions stand independently of introducing a new learning objective.

Q1: other uses. Potential applications include emotion/style-conditioned TTS data selection, expressive-speech retrieval, corpus auditing, and instruction tuning on attribute descriptions. These are proposed uses, not experiments completed in this paper. The external transfer results above and the ParaSpeechCaps results in our response to MrWz are the uses directly evaluated here.

AuthorsOpenReview ↗

A concrete use for the revised attribute resource

Your question about what others can do with the resource helped us make one use explicit. A researcher studying whispering can start from 81 clip–rubric-level pairs with two or more screened ratings (α = .435, pointwise 95% interval [.21, .64]), inspect the individual vote counts, and test whether a new listener panel reproduces the judgment. The same information is available for nine other attributes. We narrowed exploratory EXT use to these ten after testing all 57, and we retain the rest of the taxonomy in the annotation archive (App. A.15).

For comparing representations today, the external emotion results address your stronger-baseline question directly. On RAVDESS, retrieval-selected Large improves macro AUROC over e5-omni-7B by .072 [.061, .084]. On CREMA-D its macro AUROC improves by .042 [.037, .047], while e5 has higher zero-shot top-1 accuracy (.280 vs .192). This distinction helps users choose an embedding for ranking clips by a given emotion, where we observe a gain, or for choosing a single label for a clip, where the result depends on the dataset.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: resources and new experimental evidence

Thank you for recognizing VoiceNet's resources and for the experimental suggestions. We have completed three-seed retraining, all three requested omni baselines, strict speaker/video exclusion for Small, and external attribute validation. The new results show that the fine-tuned baselines transfer: emotion ranking improves on all four external datasets and attribute ranking improves on human-annotated ParaSpeechCaps tags.

Resources and new transfer evidence. VoiceNet provides expert judgments across 40 emotions; Emolia provides emotion annotations on 71.78M Emilia clips; and the MOSS-Audio suite provides attribute annotations on 8.64M clips. These resources enable fine-grained evaluation and analysis of voice representations. The expanded comparisons reveal a substantial gap between general-audio CLAPs and omni embeddings.

Retrieval-selected Large checkpoints improve macro AUROC over their base and over all three requested omni baselines on all four external emotion sets, while zero-shot top-1 remains mixed. On 14,026 clip-disjoint ParaSpeechCaps examples, the three-seed mean improves AUROC by .051 [.037, .062]; on its 35 human-annotated tags, the gain is .026 [.007, .040]. This supplies evidence of attribute transfer using labels outside VoiceNet-EXT. Other clips from the caption corpora occur in training, and a public-caption-only control is strong; these results do not isolate a unique benefit from Emolia-Balanced.

Strict training-overlap control. We completed three Small retrains excluding all 92,828 training records identified as candidate EMO matches or sharing a benchmark speaker/video. This includes all 3,862 candidate matches, including the 420 found after the initial reruns. Selected-checkpoint EMO ρ is .248 ± .017 versus .256 ± .012 with the full mixture; at epoch 1, .298 ± .011 versus .314 ± .007. These results quantify overlap sensitivity for Small. The submitted runs did not explicitly exclude EMO; Update, 16 Nov: strict exclusion now also complete for Large. See our reply to jpHB and App. A.16.

Checkpoint selection and reproducibility. The submitted checkpoints were selected using VoiceNet-EMO; this was not disclosed. The reported “epoch” was a fixed sample budget, not a pass through the mixture: 76,000 samples for Large and 405,504 for Small. We now distinguish the submitted result, three new seeds at the historical training budget, and checkpoints selected by caption–audio retrieval on held-out training clips. This rule selects epoch 6 for all three full-mixture Large runs.

Large model / protocolEMO Spearman ρEMO AUROCEXT AUROC
LCO-Omni-7B base.305.650.554
Submitted, benchmark-selected.372.691.587
Three seeds, epoch 1.342 ± .010.675 ± .004.566 ± .008
Three seeds, retrieval-selected epoch 6.310 ± .012.654 ± .007.576 ± .011

Values are means of per-seed metrics; ± is seed standard deviation. At epoch 6, the EMO ρ difference from the base is +.005, with paired clip-bootstrap 95% CI [−.013, +.026]. Thus, an EMO advantage is not established under this selection rule. The early-budget gain remains a descriptive result, with its benchmark-informed choice disclosed. Clip-bootstrap intervals condition on the trained models and do not include seed variability.

Sampling and protocol corrections.

  • The submitted IEMOCAP and CREMA-D results were eight-examples-per-label linear probes, whereas RAVDESS and EmoNet-Voice were zero-shot. The revision separates the protocols and replaces “wins two of four” with the complete four-dataset results.
  • The audited training inventory is 8.38M clips, not 9.05M. The reported models used corpus-specific caption fields, not the concatenated MOSS attribute text described in the main text; 47,619 Emolia-Balanced records used transcript fallback. Emolia-Balanced combines emotion-threshold and speaker-cluster selection routes rather than joint balancing.
  • EXT candidate selection included early VoiceCLAP/SAE grouping before the Gemini prescreen. Its low agreement remains a limitation, and the revision treats EXT as exploratory. We removed blanket state-of-the-art and human-superiority claims.

Manuscript changes. App. A.1 maps the revisions: corrected data/training descriptions (§3.2, §3.4); AUROC, ρ and held-out calibration in Table 1; checkpoint trajectories (Fig. 2, App. A.17); separate external protocols (§4.6, Table 3); removal of the human comparison (§4.3); and exploratory EXT analyses in App. A.15. The limitation statement about retraining feasibility is also corrected (App. A.24).

The paper now combines these resources with a substantially broader evaluation of what voice embeddings capture, where fine-tuning transfers, and how sampling and checkpoint selection affect the conclusions. We ask reviewers to consider the completed experiments and revised manuscript when updating their assessment; the replies below address each concern directly.

AuthorsOpenReview ↗

What the revised VoiceNet benchmark supports

We have narrowed exploratory VoiceNet-EXT use to ten attributes and completed the strict speaker/video exclusion for both model sizes, which MrWz noted helps address the overlap concern. Three results bear on the remaining questions.

Which judgments recur? Of 57 attributes, ten show above-chance agreement among screened listeners after Holm correction (α = .32–.48), including cartoonish delivery, perceived voice age and whispering. We checked two earlier attribute selections on later votes from the same listeners, paired with the historical reference vote. The 20 attributes selected by the earlier exploratory rule, which include all ten, had higher agreement than the other 37 (α .31 vs .16, difference +.15 [.02, .28]). For the six selected by Holm on the earlier export alone, the difference was inconclusive (+.09 [−.11, +.28]). These are new judgments with a shared reference listener. The final ten still need replication by a different panel (App. A.15).

Which model comparisons survive disagreement? EMO experts often disagree on individual clips (Fleiss' κ = .086). Even so, the pooled ordering of 16 models is similar for each pair of experts (Kendall τ .77–.85), and both zero-shot LCO-Omni bases exceed all seven general-audio CLAPs under every expert. Within non-affirmative items the rankings diverge (τ −.10 to .13). The reproducible comparison is on the classifier-enriched pool (§4.3, App. A.22). On the ten exploratory EXT attributes, retrieval-selected Large improves over its base by +.076 [.017, .128] AUROC with screened majority labels and +.024 [.003, .046] using the reference listener on all items. The strict-exclusion retrains also improve under both label definitions (§4.4, App. A.15–A.16).

Where does fine-tuning transfer? With the full training mixture, retrieval-selected Large improves macro AUROC over both its base and e5-omni-7B on all four external emotion datasets. Top-1 accuracy is mixed. On 35 human-annotated ParaSpeechCaps tags, Large improves over its base (.618 vs .592), while e5 leads (.647). e5 also leads Large on pooled EMO (.667 vs .654) and the submitted EXT labels (.588 vs .576). On the ten exploratory attributes their ordering depends on the label set. The strict-exclusion Large retains the external gains over its base, including .623 on the 35 human tags. On EMO itself, a fine-tuning gain over the base remains uncertain.

These results identify a use for VoiceNet independent of which model wins: researchers can examine the same predictions against each listener, see which comparisons survive, and trace where sampling changes the conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.