- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a Chinese multimodal emotion recognition dataset containing 68,210 speech segments from 50 participants. Data are collected through three progressive stages: individual emotion elicitation, dyadic interaction, and daily-life recording. The dataset includes speech, timestamped photographs, heart-rate variability, and inertial signals, with nine original emotion categories. The authors propose a multi-annotator consensus protocol and evaluate 13 speech baselines, seven- and nine-class classification, curriculum learning, and multimodal fusion.
Critical strengths
- The paper addresses a relevant problem by combining controlled emotion elicitation, interpersonal interaction, and daily-life recordings within the same participant cohort.
- The dataset includes four information sources and provides modality-coverage statistics and speaker-independent evaluation.
- The authors employ independent human annotation and evaluate label reliability using Fleiss' κ, Krippendorff's α, and annotator-consensus experiments.
- The experimental evaluation includes 13 speech baselines, class-wise results, multimodal ablations, and confidence intervals for fusion improvements.
- The paper compares baseline performance with MELD and IEMOCAP and investigates seven- and nine-class emotion recognition.
Critical weaknesses
- The proposed dataset is not available to external researchers under the current participant authorization. Neither raw recordings nor participant-linkable derived features can be shared, substantially limiting independent evaluation and the practical value of the proposed benchmark.
- Despite presenting the three-stage design as a central contribution, the paper does not evaluate cross-stage generalization between individual elicitation, dyadic interaction, and daily-life recordings. The curriculum experiment improves UAR but reduces Accuracy and Macro-F1, providing limited evidence of the proposed progression's benefits.
- Emotion labels are assigned exclusively from speech, without access to visual or physiological information. This creates a potential modality-dependent annotation bias, as the multimodal models are evaluated against labels derived solely from acoustic evidence.
- The dataset contains only 50 participants, but their gender, age, and other demographic characteristics are not reported. Participant distributions across training, validation, and test sets are also insufficiently documented, limiting assessment of representativeness and demographic bias.
- The paper provides no actual examples of photographs, speech waveforms, physiological signals, or synchronized multimodal samples. Camera placement, facial and body visibility, image acquisition frequency, and sensor configurations are not sufficiently specified, making the collected visual and physiological information difficult to assess.
- Neutral accounts for 70.24% of all segments, while several emotion classes contain fewer than 1,500 examples. Although Macro-F1, UAR, and class-wise results are reported, dedicated class-balancing strategies are not systematically investigated, and minority-class recognition remains weak.
- Adding visual information improves Macro-F1 by only 0.10 percentage points over speech alone, while adding physiological and inertial signals reduces it by 0.12 points. The reported confidence intervals include zero, so the experiments do not establish a statistically reliable aggregate benefit from additional modalities.
- Although the consensus procedure is described, the total number and qualifications of annotators, the full distribution of ratings per segment, and the handling of unresolved cases are insufficiently documented. Agreement statistics are moderate, with Fleiss' κ = 0.4310 and Krippendorff's α = 0.4188.
- MELD, IEMOCAP, and the proposed dataset differ in annotation schemes, emotional categories, and recording conditions. Moreover, models are trained and tested separately on each corpus, so the comparisons do not establish cross-corpus generalization. MELD is particularly problematic for audio-only evaluation.
- The paper does not sufficiently characterize the number of participants, samples, emotion distributions, recording durations, and modality availability separately for the three collection stages. These differences may substantially affect the reported curriculum and multimodal results.
- The corpus is organized around speech activity detection and target-speaker segmentation. Consequently, emotional expressions occurring during silence or non-speech behavior are not independently evaluated, despite the claimed focus on real-world multimodal emotion recognition.
Questions for authors
- Given that the central contribution is a new dataset and benchmark, how can independent researchers reproduce the experiments when participant-level data cannot be shared? Could anonymized acoustic, visual, and physiological features be released, subject to renewed consent and privacy review, or could a controlled-access evaluation protocol be established?
- What experimental evidence demonstrates the advantage of collecting data across three progressively less controlled settings? Can the authors provide cross-stage evaluations, particularly training on individual elicitation or dyadic interaction and testing on daily-life recordings, together with stage-specific dataset statistics?
- Why are emotion labels determined exclusively from audio when the dataset is intended for multimodal emotion recognition? How would the labels change if annotators also observed visual information? How many annotators participated overall, what were their qualifications, and how many ratings were required per sample?
- What are the gender and age distributions of the 50 participants, including their allocation across data splits? Can the authors provide representative multimodal samples and specify the camera setup, visibility of faces and bodies, image sampling rate, sensor configurations, and synchronization accuracy?
- How is the severe class imbalance addressed during training? Can the authors evaluate class-balanced learning strategies and provide confusion matrices to clarify whether the reported Accuracy primarily reflects the dominant neutral category rather than reliable recognition of minority emotions?
- Why do visual, physiological, and inertial modalities provide almost no improvement in overall Macro-F1? To what extent could this result be caused by audio-derived ground-truth labels, incomplete modality coverage, or limitations of the fusion approach? Can the authors provide additional evidence that these modalities contain complementary emotional information?
- Given the differences in emotion taxonomies and annotation protocols, what conclusions can be drawn from the within-corpus comparisons with MELD and IEMOCAP? Can the authors provide cross-corpus experiments with compatible label mappings to assess whether the proposed dataset improves generalization beyond its own participants and recording conditions?
3 comments
Supplementary staged-training and class-balance experiments
The following supports Q2 and Q5. Scores are percentages; differences and confidence-interval endpoints are percentage points.
Q2. Cross-stage testing. Entries are seven-class Macro-F1; I/D denote individual/dyadic. The pretrained Chinese-HuBERT encoder remains frozen. Standardization and heads use source-stage training data only, with parameters selected on the corresponding validation set.
| Head source | Train/val | Test I | Test D | Test daily |
|---|---|---|---|---|
| Individual | 7,450/921 | 44.74 | 30.48 | 17.98 |
| Dyadic | 20,879/2,993 | 31.29 | 40.46 | 26.55 |
| Daily | 16,879/3,076 | 18.94 | 26.31 | 26.63 |
| Pooled | 45,208/6,990 | 41.00 | 42.23 | 31.07 |
Individual/dyadic/daily tests contain 2,235/5,630/8,147 clips from 10/9/2 held-out identifiers.
The curriculum trains on individual, individual+dyadic, and all data for 5 epochs each, starting from the preceding best checkpoint and resetting the optimizer. Complete denotes direct encoder classification on 10,993 strict-intersection test clips.
| Test set | Training | n | Accuracy | Macro-F1 | UAR |
|---|---|---|---|---|---|
| All | Mixed | 16012 | 78.42 | 52.59 | 51.63 |
| All | Curriculum | 16012 | 76.61 | 52.03 | 54.07 |
| Individual | Mixed | 2235 | 52.66 | 52.16 | 51.49 |
| Individual | Curriculum | 2235 | 55.17 | 53.14 | 55.00 |
| Dyadic | Mixed | 5630 | 79.41 | 52.23 | 49.19 |
| Dyadic | Curriculum | 5630 | 77.25 | 51.91 | 52.45 |
| Daily | Mixed | 8147 | 84.80 | 36.01 | 40.63 |
| Daily | Curriculum | 8147 | 82.04 | 34.81 | 40.87 |
| Complete | Mixed | 10993 | 78.38 | 51.42 | 49.55 |
| Complete | Curriculum | 10993 | 77.54 | 52.55 | 53.52 |
Overall Accuracy/Macro-F1/UAR changes are −1.81/−0.56/+2.44 points, characterizing learning-order effects on class recall.
Q5. Class-balanced control. Both runs use the same split, seed=42, effective batch size 40, and six-epoch budget, selecting by validation Macro-F1. Weights use training class frequencies only. Weighted minus ordinary CE differences and 95% participant-paired bootstrap intervals are: Accuracy −3.63 [−4.57,−1.40], Macro-F1 −0.76 [−1.74,+1.25], and UAR +4.82 [+3.51,+5.75].
| Class | Full-test n | CE Recall | Weighted Recall |
|---|---|---|---|
| Neutral | 11553 | 89.36 | 81.65 |
| Disgust | 319 | 34.17 | 42.63 |
| Joy | 1493 | 56.46 | 66.31 |
| Fear | 581 | 42.34 | 46.64 |
| Sadness | 566 | 41.70 | 52.12 |
| Surprise | 622 | 51.93 | 60.13 |
| Anger | 878 | 59.57 | 59.79 |
Neutral/Other acc are within-group accuracies; aggregate metrics use all 16,012 test clips.
| Training | Neutral acc | Other acc | Overall acc | Macro-F1 | UAR |
|---|---|---|---|---|---|
| CE | 89.36 | 51.13 | 78.72 | 53.67 | 53.65 |
| Weighted CE | 81.65 | 58.11 | 75.09 | 52.91 | 58.47 |
Weighting increases correct non-neutral predictions from 2,280 to 2,591 and changes correct neutral predictions from 10,324 to 9,433, trading improved minority-emotion recognition against neutral recognition.
Fusion results and class-specific confusion matrices
The following supports Q6 and provides count matrices for Q5. Scores are percentages; differences and interval endpoints are percentage points.
Q6. Fusion experiment. All systems use the same 10,993 seven-class test clips and 9 held-out participants. Heads are selected by validation Macro-F1 and class-coordinate blending weights by validation UAR, then fixed for testing. Count matrices are in our LEEC-Q2 response and Appendix Table 23.
| System | Accuracy | Macro-F1 | UAR | Delta MF1 [95% CI] |
|---|---|---|---|---|
| Audio | 73.77 | 50.84 | 55.12 | Reference |
| Audio+visual | 65.94 | 47.51 | 57.39 | -3.33 [-4.99,-1.65] |
| Three-modal | 66.60 | 47.79 | 57.15 | -3.05 [-5.10,-1.34] |
Class recalls from the same predictions:
| Class | n | Audio | Audio+visual | Three-modal |
|---|---|---|---|---|
| Neutral | 7840 | 81.49 | 68.51 | 69.52 |
| Disgust | 241 | 43.15 | 45.64 | 45.64 |
| Joy | 1039 | 67.76 | 74.11 | 74.40 |
| Fear | 409 | 43.28 | 46.94 | 40.10 |
| Sadness | 440 | 44.77 | 53.41 | 56.14 |
| Surprise | 376 | 52.93 | 59.57 | 60.11 |
| Anger | 648 | 52.47 | 53.55 | 54.17 |
Vision lowers neutral Recall by 12.98 points and raises all six non-neutral Recall point estimates; UAR changes by +2.27 [95% CI +0.25,+4.24]. Adding sensors to speech+vision raises sadness Recall by 2.73 points and lowers fear Recall by 6.84. Three-modal versus speech UAR changes by +2.03 [−0.10,+3.93]. None of 14 class-wise comparisons survives Holm correction, so class effects are reported as exploratory.
Q5. Class-balanced confusion matrices. Each cell gives ordinary/weighted CE counts, using the class order in LEEC-Q2 and identical supports.
| True | N | D | J | F | Sa | Su | A | n |
|---|---|---|---|---|---|---|---|---|
| N | 10324/9433 | 53/96 | 243/420 | 186/290 | 158/351 | 360/673 | 229/290 | 11553 |
| D | 66/37 | 109/136 | 8/15 | 29/31 | 21/23 | 21/20 | 65/57 | 319 |
| J | 490/314 | 13/14 | 843/990 | 16/18 | 22/33 | 66/80 | 43/44 | 1493 |
| F | 215/160 | 13/21 | 2/7 | 246/271 | 50/73 | 25/29 | 30/20 | 581 |
| Sa | 186/110 | 18/29 | 12/15 | 82/82 | 236/295 | 14/17 | 18/18 | 566 |
| Su | 190/126 | 10/13 | 31/44 | 26/23 | 5/4 | 323/374 | 37/38 | 622 |
| A | 154/84 | 33/57 | 36/44 | 43/45 | 9/18 | 80/105 | 523/525 | 878 |
Response to the seven questions from Reviewer rvVz
We first address the seven Questions for Authors, followed by the corresponding experimental tables.
Q1. Reproduction and access. The shared statement specifies the subset and code release on November 28, 2026. Its training/evaluation partitions will be documented and reported separately from the full-dataset benchmark.
Paper change: Access and reproducibility statements specify subset scope and evaluation protocols.
Q2. Evidence for three-stage collection. We added cross-stage tests with frozen Chinese-HuBERT features: heads trained on individual, dyadic, daily, or pooled data are tested in all three held-out settings. Individual-only training gives 44.74% Macro-F1 on individual tests versus 41.00% for pooled training; pooled training gives 42.23%/31.07% on dyadic/daily tests. Curriculum training diagnoses learning order: overall UAR changes by +2.44 points, Accuracy by −1.81, and Macro-F1 by −0.56 relative to mixed training. Both schedules cap at 15 epochs; stage sizes and cumulative training steps differ.
Paper change: Section 4 and Appendix B add cross-stage results and present curriculum training as a diagnostic analysis.
Q3. Labels and annotators. The target is speech-perceived emotion, judged from speech alone; photos and sensors are auxiliary observations. Evaluation units are target-speaker speech segments. The statistics cover 30 annotator identifiers from the collection pool. No training sessions were held; designated professionals independently reviewed no-majority cases. The 65,456 auditable segments have 3/4/5/6 judgments in 8,972/28,268/27,721/495 cases.
Paper change: The title, abstract, and methods consistently state the label source; Appendix A specifies annotators, judgment counts, and review rules.
Q4. Demographics and acquisition. Train/validation/test contain 35/5/10 identifiers, with male/female counts of 23/12, 3/2, and 9/1. Ages are 24.53±1.86, 23.80±2.17, and 24.20±1.40 years (mean±sample SD; valid n=34/5/10); overall age is 24.39±1.79, range 21–29. The camera is within 1 m, approximately 30° off frontal, covering the face and upper body, with a configured 0.5-second interval. Each inertial stream has a median of 260 records per segment; HRV offsets have median 22.94 seconds and tolerance 120 seconds. Paired modality examples will accompany the subset.
Paper change: Appendix A adds split-wise demographics and acquisition settings; Appendix E provides stage-wise records.
Q5. Class imbalance. We added ordinary versus inverse-frequency-weighted CE under the same split and six-epoch budget. UAR changes from 53.65% to 58.47%, Accuracy from 78.72% to 75.09%, and Macro-F1 from 53.67% to 52.91%. Within-group accuracy on 11,553 neutral/4,459 non-neutral test clips changes from 89.36%/51.13% to 81.65%/58.11%. Supporting tables give class recalls, intervals, and count matrices.
Paper change: Section 4 and Appendices B and E add class-balanced controls, class-wise results, and confusion matrices.
Q6. Auxiliary modalities. The corrected predictions give speech+vision/three-modal Macro-F1 changes of −3.33/−3.05 points and UAR changes of +2.27/+2.03 relative to speech. Vision raises six non-neutral Recall point estimates while lowering neutral Recall; none of 14 class-wise comparisons reaches 0.05 after Holm correction. We revise the conclusion to a class-recall trade-off under speech-derived supervision and the selected fusion operating point.
Paper change: Main and appendix interpretations are aligned, with selection objectives and statistical tests specified.
Q7. MELD/IEMOCAP comparisons. We accept this concern and remove the main MELD comparison and associated audio-model ranking conclusions. IEMOCAP remains only as a separate appendix reference under the standard four-class, five-fold protocol.
Paper change: The main MELD comparison and inferences are removed; the separate IEMOCAP reference table is retained.