acceptodds
Under review as a conference paper at ICLR 2027

A Progressive Multi-Stage Multimodal Dataset and Benchmark for Real-World Chinese Emotion Recognition

Abstract

Real-world emotion datasets must jointly balance acquisition control, ecological validity, cross-modal synchronization, label reliability, and reproducible evaluation. We construct a progressive multi-stage multimodal dataset for Chinese emotion research with 68,210 speech segments from 50 participants, nine perceived-emotion labels, synchronized timestamped photo sequences, heart-rate-variability (HRV) measurements, and inertial signals. A unified protocol links individual emotion elicitation, dyadic interaction, and daily-life recording while retaining the same participant cohort, label space, temporal reference, and quality controls. To address the subjectivity of perceived emotion, an auditable annotation pipeline combines independent ratings, absolute-majority aggregation, and adaptive annotator expansion. Across 65,456 segments with at least two alignable ratings, Fleiss' and Krippendorff's ; random resampling of 517 samples with ten complete ratings and fixed-model learnability tests further show that label agreement and learnability generally improve with more annotators. We establish unified, speaker-independent seven- and nine-class benchmarks; the complete seven-class benchmark covers 13 speech baselines, with matched-model comparisons on MELD and IEMOCAP. Chinese-HuBERT-Large achieves 78.42% Accuracy and 52.59% Macro-F1 on the seven-class task. A cumulative three-stage curriculum raises UAR from 51.63% to 54.07%, while yielding 76.61% Accuracy and 52.03% Macro-F1. The strict complete-modality intersection contains 37,043 segments (54.31%). On this subset, additional visual and sensor modalities mainly produce class-conditional performance changes: visual input improves accuracy for six non-neutral classes, while overall Macro-F1 remains close to the audio baseline. Together, the dataset, progressive acquisition framework, empirically validated consensus protocol, and systematic benchmark provide an auditable foundation for real-world Chinese emotion research.

Primary area:
datasets and benchmarks
Keywords:
Chinese emotion recognition, multimodal emotion dataset, real-world affective computing, consensus annotation, speaker-independent benchmark, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rvVzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a Chinese multimodal emotion recognition dataset containing 68,210 speech segments from 50 participants. Data are collected through three progressive stages: individual emotion elicitation, dyadic interaction, and daily-life recording. The dataset includes speech, timestamped photographs, heart-rate variability, and inertial signals, with nine original emotion categories. The authors propose a multi-annotator consensus protocol and evaluate 13 speech baselines, seven- and nine-class classification, curriculum learning, and multimodal fusion.

Critical strengths
  1. The paper addresses a relevant problem by combining controlled emotion elicitation, interpersonal interaction, and daily-life recordings within the same participant cohort.
  2. The dataset includes four information sources and provides modality-coverage statistics and speaker-independent evaluation.
  3. The authors employ independent human annotation and evaluate label reliability using Fleiss' κ, Krippendorff's α, and annotator-consensus experiments.
  4. The experimental evaluation includes 13 speech baselines, class-wise results, multimodal ablations, and confidence intervals for fusion improvements.
  5. The paper compares baseline performance with MELD and IEMOCAP and investigates seven- and nine-class emotion recognition.
Critical weaknesses
  1. The proposed dataset is not available to external researchers under the current participant authorization. Neither raw recordings nor participant-linkable derived features can be shared, substantially limiting independent evaluation and the practical value of the proposed benchmark.
  2. Despite presenting the three-stage design as a central contribution, the paper does not evaluate cross-stage generalization between individual elicitation, dyadic interaction, and daily-life recordings. The curriculum experiment improves UAR but reduces Accuracy and Macro-F1, providing limited evidence of the proposed progression's benefits.
  3. Emotion labels are assigned exclusively from speech, without access to visual or physiological information. This creates a potential modality-dependent annotation bias, as the multimodal models are evaluated against labels derived solely from acoustic evidence.
  4. The dataset contains only 50 participants, but their gender, age, and other demographic characteristics are not reported. Participant distributions across training, validation, and test sets are also insufficiently documented, limiting assessment of representativeness and demographic bias.
  5. The paper provides no actual examples of photographs, speech waveforms, physiological signals, or synchronized multimodal samples. Camera placement, facial and body visibility, image acquisition frequency, and sensor configurations are not sufficiently specified, making the collected visual and physiological information difficult to assess.
  6. Neutral accounts for 70.24% of all segments, while several emotion classes contain fewer than 1,500 examples. Although Macro-F1, UAR, and class-wise results are reported, dedicated class-balancing strategies are not systematically investigated, and minority-class recognition remains weak.
  7. Adding visual information improves Macro-F1 by only 0.10 percentage points over speech alone, while adding physiological and inertial signals reduces it by 0.12 points. The reported confidence intervals include zero, so the experiments do not establish a statistically reliable aggregate benefit from additional modalities.
  8. Although the consensus procedure is described, the total number and qualifications of annotators, the full distribution of ratings per segment, and the handling of unresolved cases are insufficiently documented. Agreement statistics are moderate, with Fleiss' κ = 0.4310 and Krippendorff's α = 0.4188.
  9. MELD, IEMOCAP, and the proposed dataset differ in annotation schemes, emotional categories, and recording conditions. Moreover, models are trained and tested separately on each corpus, so the comparisons do not establish cross-corpus generalization. MELD is particularly problematic for audio-only evaluation.
  10. The paper does not sufficiently characterize the number of participants, samples, emotion distributions, recording durations, and modality availability separately for the three collection stages. These differences may substantially affect the reported curriculum and multimodal results.
  11. The corpus is organized around speech activity detection and target-speaker segmentation. Consequently, emotional expressions occurring during silence or non-speech behavior are not independently evaluated, despite the claimed focus on real-world multimodal emotion recognition.
Questions for authors
  1. Given that the central contribution is a new dataset and benchmark, how can independent researchers reproduce the experiments when participant-level data cannot be shared? Could anonymized acoustic, visual, and physiological features be released, subject to renewed consent and privacy review, or could a controlled-access evaluation protocol be established?
  2. What experimental evidence demonstrates the advantage of collecting data across three progressively less controlled settings? Can the authors provide cross-stage evaluations, particularly training on individual elicitation or dyadic interaction and testing on daily-life recordings, together with stage-specific dataset statistics?
  3. Why are emotion labels determined exclusively from audio when the dataset is intended for multimodal emotion recognition? How would the labels change if annotators also observed visual information? How many annotators participated overall, what were their qualifications, and how many ratings were required per sample?
  4. What are the gender and age distributions of the 50 participants, including their allocation across data splits? Can the authors provide representative multimodal samples and specify the camera setup, visibility of faces and bodies, image sampling rate, sensor configurations, and synchronization accuracy?
  5. How is the severe class imbalance addressed during training? Can the authors evaluate class-balanced learning strategies and provide confusion matrices to clarify whether the reported Accuracy primarily reflects the dominant neutral category rather than reliable recognition of minority emotions?
  6. Why do visual, physiological, and inertial modalities provide almost no improvement in overall Macro-F1? To what extent could this result be caused by audio-derived ground-truth labels, incomplete modality coverage, or limitations of the fusion approach? Can the authors provide additional evidence that these modalities contain complementary emotional information?
  7. Given the differences in emotion taxonomies and annotation protocols, what conclusions can be drawn from the within-corpus comparisons with MELD and IEMOCAP? Can the authors provide cross-corpus experiments with compatible label mappings to assess whether the proposed dataset improves generalization beyond its own participants and recording conditions?
3 comments
AuthorsOpenReview ↗

Response to the seven questions from Reviewer rvVz

We first address the seven Questions for Authors, followed by the corresponding experimental tables.

Q1. Reproduction and access. The shared statement specifies the subset and code release on November 28, 2026. Its training/evaluation partitions will be documented and reported separately from the full-dataset benchmark.

Paper change: Access and reproducibility statements specify subset scope and evaluation protocols.

Q2. Evidence for three-stage collection. We added cross-stage tests with frozen Chinese-HuBERT features: heads trained on individual, dyadic, daily, or pooled data are tested in all three held-out settings. Individual-only training gives 44.74% Macro-F1 on individual tests versus 41.00% for pooled training; pooled training gives 42.23%/31.07% on dyadic/daily tests. Curriculum training diagnoses learning order: overall UAR changes by +2.44 points, Accuracy by −1.81, and Macro-F1 by −0.56 relative to mixed training. Both schedules cap at 15 epochs; stage sizes and cumulative training steps differ.

Paper change: Section 4 and Appendix B add cross-stage results and present curriculum training as a diagnostic analysis.

Q3. Labels and annotators. The target is speech-perceived emotion, judged from speech alone; photos and sensors are auxiliary observations. Evaluation units are target-speaker speech segments. The statistics cover 30 annotator identifiers from the collection pool. No training sessions were held; designated professionals independently reviewed no-majority cases. The 65,456 auditable segments have 3/4/5/6 judgments in 8,972/28,268/27,721/495 cases.

Paper change: The title, abstract, and methods consistently state the label source; Appendix A specifies annotators, judgment counts, and review rules.

Q4. Demographics and acquisition. Train/validation/test contain 35/5/10 identifiers, with male/female counts of 23/12, 3/2, and 9/1. Ages are 24.53±1.86, 23.80±2.17, and 24.20±1.40 years (mean±sample SD; valid n=34/5/10); overall age is 24.39±1.79, range 21–29. The camera is within 1 m, approximately 30° off frontal, covering the face and upper body, with a configured 0.5-second interval. Each inertial stream has a median of 260 records per segment; HRV offsets have median 22.94 seconds and tolerance 120 seconds. Paired modality examples will accompany the subset.

Paper change: Appendix A adds split-wise demographics and acquisition settings; Appendix E provides stage-wise records.

Q5. Class imbalance. We added ordinary versus inverse-frequency-weighted CE under the same split and six-epoch budget. UAR changes from 53.65% to 58.47%, Accuracy from 78.72% to 75.09%, and Macro-F1 from 53.67% to 52.91%. Within-group accuracy on 11,553 neutral/4,459 non-neutral test clips changes from 89.36%/51.13% to 81.65%/58.11%. Supporting tables give class recalls, intervals, and count matrices.

Paper change: Section 4 and Appendices B and E add class-balanced controls, class-wise results, and confusion matrices.

Q6. Auxiliary modalities. The corrected predictions give speech+vision/three-modal Macro-F1 changes of −3.33/−3.05 points and UAR changes of +2.27/+2.03 relative to speech. Vision raises six non-neutral Recall point estimates while lowering neutral Recall; none of 14 class-wise comparisons reaches 0.05 after Holm correction. We revise the conclusion to a class-recall trade-off under speech-derived supervision and the selected fusion operating point.

Paper change: Main and appendix interpretations are aligned, with selection objectives and statistical tests specified.

Q7. MELD/IEMOCAP comparisons. We accept this concern and remove the main MELD comparison and associated audio-model ranking conclusions. IEMOCAP remains only as a separate appendix reference under the standard four-class, five-fold protocol.

Paper change: The main MELD comparison and inferences are removed; the separate IEMOCAP reference table is retained.

AuthorsOpenReview ↗

Supplementary staged-training and class-balance experiments

The following supports Q2 and Q5. Scores are percentages; differences and confidence-interval endpoints are percentage points.

Q2. Cross-stage testing. Entries are seven-class Macro-F1; I/D denote individual/dyadic. The pretrained Chinese-HuBERT encoder remains frozen. Standardization and heads use source-stage training data only, with parameters selected on the corresponding validation set.

Head sourceTrain/valTest ITest DTest daily
Individual7,450/92144.7430.4817.98
Dyadic20,879/2,99331.2940.4626.55
Daily16,879/3,07618.9426.3126.63
Pooled45,208/6,99041.0042.2331.07

Individual/dyadic/daily tests contain 2,235/5,630/8,147 clips from 10/9/2 held-out identifiers.

The curriculum trains on individual, individual+dyadic, and all data for 5 epochs each, starting from the preceding best checkpoint and resetting the optimizer. Complete denotes direct encoder classification on 10,993 strict-intersection test clips.

Test setTrainingnAccuracyMacro-F1UAR
AllMixed1601278.4252.5951.63
AllCurriculum1601276.6152.0354.07
IndividualMixed223552.6652.1651.49
IndividualCurriculum223555.1753.1455.00
DyadicMixed563079.4152.2349.19
DyadicCurriculum563077.2551.9152.45
DailyMixed814784.8036.0140.63
DailyCurriculum814782.0434.8140.87
CompleteMixed1099378.3851.4249.55
CompleteCurriculum1099377.5452.5553.52

Overall Accuracy/Macro-F1/UAR changes are −1.81/−0.56/+2.44 points, characterizing learning-order effects on class recall.

Q5. Class-balanced control. Both runs use the same split, seed=42, effective batch size 40, and six-epoch budget, selecting by validation Macro-F1. Weights use training class frequencies only. Weighted minus ordinary CE differences and 95% participant-paired bootstrap intervals are: Accuracy −3.63 [−4.57,−1.40], Macro-F1 −0.76 [−1.74,+1.25], and UAR +4.82 [+3.51,+5.75].

ClassFull-test nCE RecallWeighted Recall
Neutral1155389.3681.65
Disgust31934.1742.63
Joy149356.4666.31
Fear58142.3446.64
Sadness56641.7052.12
Surprise62251.9360.13
Anger87859.5759.79

Neutral/Other acc are within-group accuracies; aggregate metrics use all 16,012 test clips.

TrainingNeutral accOther accOverall accMacro-F1UAR
CE89.3651.1378.7253.6753.65
Weighted CE81.6558.1175.0952.9158.47

Weighting increases correct non-neutral predictions from 2,280 to 2,591 and changes correct neutral predictions from 10,324 to 9,433, trading improved minority-emotion recognition against neutral recognition.

AuthorsOpenReview ↗

Fusion results and class-specific confusion matrices

The following supports Q6 and provides count matrices for Q5. Scores are percentages; differences and interval endpoints are percentage points.

Q6. Fusion experiment. All systems use the same 10,993 seven-class test clips and 9 held-out participants. Heads are selected by validation Macro-F1 and class-coordinate blending weights by validation UAR, then fixed for testing. Count matrices are in our LEEC-Q2 response and Appendix Table 23.

SystemAccuracyMacro-F1UARDelta MF1 [95% CI]
Audio73.7750.8455.12Reference
Audio+visual65.9447.5157.39-3.33 [-4.99,-1.65]
Three-modal66.6047.7957.15-3.05 [-5.10,-1.34]

Class recalls from the same predictions:

ClassnAudioAudio+visualThree-modal
Neutral784081.4968.5169.52
Disgust24143.1545.6445.64
Joy103967.7674.1174.40
Fear40943.2846.9440.10
Sadness44044.7753.4156.14
Surprise37652.9359.5760.11
Anger64852.4753.5554.17

Vision lowers neutral Recall by 12.98 points and raises all six non-neutral Recall point estimates; UAR changes by +2.27 [95% CI +0.25,+4.24]. Adding sensors to speech+vision raises sadness Recall by 2.73 points and lowers fear Recall by 6.84. Three-modal versus speech UAR changes by +2.03 [−0.10,+3.93]. None of 14 class-wise comparisons survives Holm correction, so class effects are reported as exploratory.

Q5. Class-balanced confusion matrices. Each cell gives ordinary/weighted CE counts, using the class order in LEEC-Q2 and identical supports.

TrueNDJFSaSuAn
N10324/943353/96243/420186/290158/351360/673229/29011553
D66/37109/1368/1529/3121/2321/2065/57319
J490/31413/14843/99016/1822/3366/8043/441493
F215/16013/212/7246/27150/7325/2930/20581
Sa186/11018/2912/1582/82236/29514/1718/18566
Su190/12610/1331/4426/235/4323/37437/38622
A154/8433/5736/4443/459/1880/105523/525878
Reviewer_LEECOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces 68k Chinese speech segments from 50 participants, spanning elicitation, dyadic interaction and daily life, with photos and wearable signals. It benchmarks speech models, annotation consensus, multimodal fusion and a staged training curriculum.

Critical strengths

Following the same participants across three settings is a neat design: it could help disentangle changes in recording context from differences between cohorts.

I appreciate the explicit reporting of annotation agreement, missing modalities and participant-level confidence intervals.

Critical weaknesses

The main contribution is a dataset, yet the consent explicitly prevents external access to participant-level data, including derived data (Appendix C.1). Releasing code and aggregate statistics leaves others unable to use or independently reproduce the benchmark.

The fusion results seem not to add up. Table 19 reports 75.08% speech+vision accuracy, but every corresponding class recall in Table 20 is at most 74.11%. Overall accuracy is a support-weighted average of class recalls, so these cannot describe the same predictions and test set. This undermines the central complementarity analysis.

Questions for authors

What concrete, consent-compatible route will let independent researchers access and evaluate this dataset?

Can you reconcile Tables 19–20 and provide class supports and confusion matrices from the same test predictions?

1 comment
AuthorsOpenReview ↗

Response to the two questions from Reviewer LEEC

Thank you for identifying this error. We address access and numerical consistency below.

Q1. Data access. The subset release on November 28, 2026 has been agreed with the data holder. The shared statement specifies its scope and accompanying materials.

Paper change: Appendix C.1 specifies the release schedule and access arrangements.

Q2. Cause of the Table 19–20 inconsistency. Your mathematical observation is correct: Accuracy is the support-weighted average of class recalls. The speech+vision recalls in original Table 20 range from 45.64% to 74.11% and cannot yield 75.08% Accuracy. We apologize for this error.

Original Table 19 retained aggregates from an earlier experiment and was not updated when the final experiment underlying original Table 20 was integrated. We have recomputed aggregate metrics, class-wise results, and confusion matrices from the same predictions underlying original Table 20.

SystemAccuracyMacro-F1UARDelta MF1 [95% CI]
Audio73.7750.8455.12Reference
Audio+visual65.9447.5157.39-3.33 [-4.99,-1.65]
Three-modal66.6047.7957.15-3.05 [-5.10,-1.34]

All three systems use the same 10,993 test clips and class supports, with 8,110, 7,249, and 7,321 correct predictions. The 95% intervals use 10,000 paired bootstrap resamples over 9 held-out participants. We now describe the result as a class-recall trade-off.

Paper change: Main Table 7 and Appendix Table 21 update original Table 19; Appendix Table 22 updates original Table 20; new Table 23 gives the corresponding count matrices. The abstract and interpretation are revised accordingly.

Rows are true labels and columns are predictions. N/D/J/F/Sa/Su/A denote neutral/disgust/joy/fear/sadness/surprise/anger; n is class support.

Audio only

TrueNDJFSaSuAn
N6389743821592164951257840
D351041014171942241
J19112704192564241039
F121136177622010409
Sa1072218661971317440
Su9783291019921376
A974548411562340648

Audio plus visual

TrueNDJFSaSuAn
N5371987562324737551557840
D101102115231943241
J9611770234369271039
F601420192862611409
Sa472232732351318440
Su65934111222421376
A494666462668347648

Audio plus visual plus sensors

TrueNDJFSaSuAn
N54501007461464667761567840
D101102115231943241
J9714773174269271039
F641723164983112409
Sa532431532471418440
Su7093251322621376
A494768372670351648
Reviewer_6RFtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multimodal dataset for Chinese emotion recognition across controlled tasks, conversations and daily-life settings, addressing a relevant need in affective computing. However, the data are currently unavailable to external researchers, and important collection and annotation details require clarification. Further validation of the audio-based labels for the other modalities is also needed. These limitations reduce the dataset’s current value as a reusable benchmark.

Critical strengths

The main strength is collecting speech, visual and wearable signals from the same participants across controlled tasks, conversations and daily-life settings. This design could support useful studies of how emotion recognition changes across contexts. The use of multiple annotators, participant-independent evaluation and a broad set of speech baselines provides a useful starting point.

Critical weaknesses
  1. My main concern is the availability of dataset. The current consent prohibits external sharing of participant-level data, while the proposed release includes only documentation, code and aggregate statistics. Will researchers be able to access the data and annotations upon publication? Please provide a concrete release or controlled-access plan, including the required consent, access conditions and timeline. Without a credible access plan, I would lean toward rejection because the paper’s main contribution cannot be independently evaluated or reused.
  2. The paper reports the total number of speech segments, but the number and duration of recordings per participant remain unclear. How many source recordings and visual sequences were collected from each participant? Please provide these statistics for each collection stage and distinguish original recordings from extracted segments.
  3. Annotators judge speech without access to visual or physiological information. It remains unclear how well these audio-based labels represent the emotional information available in the other modalities. Could the authors validate a representative subset using independent audiovisual ratings and available participant self-reports?
  4. The paper mentions category definitions and instructions, but does not provide the actual guidelines or describe annotator qualifications and training. Moderate overall agreement also leaves uncertainty about categories such as sadness and disgust. What guidance and examples did annotators receive? Please report agreement and unresolved cases for each emotion.
  5. The amount of visual and sensor information accompanying each speech segment needs clarification. How frequently were photos sampled, and how much sensor data was available for each segment? These details would help readers assess whether the additional modalities adequately capture the corresponding emotional expression.
  6. Only 54.31% of samples contain all recorded streams, and the main fusion experiments use this complete subset. Why are the other modalities missing? Please explain whether missingness varies across participants, emotions or collection stages, and how this affects the representativeness of the evaluated subset.
Questions for authors

Please refer to the weaknesses section

2 comments
AuthorsOpenReview ↗

Response to the six questions from Reviewer 6RFt

Thank you for your comments. We address the six questions below; the next comment provides supporting statistics.

Q1. Data access. The subset will be released on November 28, 2026, as agreed with the data holder. Its scope and accompanying materials are given in the shared data-access statement.

Paper change: Appendix C.1 adds the subset release and access arrangements.

Q2. Recordings and participant contributions. Acquisition comprises 1,449 sessions and 2,418 source audio files; the 68,210 retained segments reference 1,445 sessions and 2,406 files. Measured source recordings total 128.29 hours, and retained annotation intervals total 64.23 hours. Allocating half of dyadic durations to each contribution gives 105.92 and 46.98 hours, respectively. The next comment gives stage-wise recording, segment, and photo-sequence counts; Appendix E provides counts and durations for all 50 identifiers by stage.

Paper change: Appendix A distinguishes sessions, source recordings, and extracted segments; Appendix E adds the participant-by-stage inventory.

Q3. Label source. We define the target as speech-perceived emotion: an observer’s judgment based only on the target speaker’s speech. Photos and sensors serve as auxiliary observations for predicting this target. The title, abstract, and methods now state this consistently.

Paper change: The title, abstract, and annotation methods specify speech-derived supervision.

Q4. Annotation guide and agreement. Annotators used a written guide with eight emotion definitions, everyday examples, and four contrasts; neutral was an additional response. The guide distinguishes immediate threat from future worry (fear/anxiety), loss from blocked goals (sadness/frustration), confrontation from avoidance (anger/disgust), and sustained positive experience from a brief unexpected response (joy/surprise). Examples include a snake in view versus worry about exam failure, and arguing after being bullied versus avoiding something repellent. Intensity was subjectively rated weak/moderate/strong; no training sessions were held. Krippendorff alpha=0.4242 and Fleiss-type kappa=0.4294 were obtained from 65,456 segments and 281,563 judgments. Class-specific agreement and disagreement are given in the next comment.

Paper change: New Appendix G reproduces the original definitions and examples; Appendix Table 26 reports class-specific agreement and no-majority counts.

Q5. Auxiliary information per segment. The configured photo interval is 0.5 seconds. In the complete intersection, each inertial stream has a median of 260 records per annotated speech interval. HRV matching offsets have a median of 22.94 seconds, with a 120-second tolerance.

Paper change: Appendix A specifies sampling intervals, records per segment, and temporal matching rules.

Q6. Missingness and representativeness. Missing photos primarily relate to participants being unwilling to collect or retain them, particularly in daily life. Secondary reasons include device failures and data unavailability during storage and archiving, including stalls that lengthened photo intervals and unsuccessful camera startup. The complete intersection contains 37,043/68,210 clips, with individual/dyadic/daily inclusion rates of 67.96%/66.18%/36.69%. Class-specific inclusion ranges from 51.03% to 72.59%. Neutral falls from 70.24% of all clips to 66.00% of complete cases, and daily-life clips from 41.20% to 27.83%. We therefore report complete-case fusion separately from full-audio results and provide coverage by stage, class, and participant.

Paper change: Section 3 and Appendices B and E explain missingness and quantify distribution shifts in the strict intersection.

AuthorsOpenReview ↗

Supplementary statistics on recordings, annotation agreement, and modality coverage

The following tables support Q2 on recordings and photos, Q4 on annotation, and Q6 on modality coverage.

UnitCountScope
All sessions1,449full source inventory
All source files2,418one canonical audio per directory
Retained sessions1,445referenced by retained segments
Retained files2,406referenced by retained segments
Retained segments68,210final benchmark

Durations are hours. Measured denotes physical duration; allocated halves dyadic durations to assign individual contributions.

StageFilesAudio measuredAudio allocatedSegments measuredSegments allocated
Individual72218.3018.3011.9211.92
Dyadic55444.7422.3734.5017.25
Daily1,14265.2465.2417.8117.81
Total2,418128.29105.9264.2346.98

Entries are rounded independently; totals use unrounded durations. Allocation changes neither file lengths nor clip counts.

Identifiers are counted within stage and may overlap across stages. A photo sequence is an original continuous sequence saved for an acquisition session, distinct from photos matched to an individual speech clip.

StageIDsSessionsFilesClipsPhoto seq.Photos
Individual497217221060665354440
Dyadic4755455429502496129467
Daily1417411422810286607638
Total5014492418682101235791545

Appendix E provides stage-wise records for all 50 participant identifiers.

The following groups use final emotion labels and retain Ambiguous and Abnormal judgments. Unanimous is the fraction with all retrievable judgments identical; no-majority counts items where no response exceeds half the judgments. Supplementary review handles unresolved cases.

EmotionEligible nUnanimous %No-majority n
Neutral46,53341.472618
Disgust1,16926.60148
Joy7,40033.96506
Fear55819.3580
Sadness68919.8859
Surprise3,03420.24369
Anger2,71637.00292
Frustration1,82321.28266
Anxiety1,53419.62237

Across 65,456 auditable items, the absolute-majority rate is 93.01% overall and 84.55%–94.37% by class. These class-wise results show how agreement varies across emotions.

Q6. Class-specific modality coverage. Complete counts clips in the complete intersection; the denominator is all retained clips in the class.

EmotionAll nComplete nComplete %
Neutral47,91124,45051.03
Disgust1,35597972.25
Joy7,7144,48858.18
Fear69441059.08
Sadness75648564.15
Surprise3,1591,61851.22
Anger2,9622,05469.35
Frustration1,9701,33367.66
Anxiety1,6891,22672.59

Comments on the paper

2 comments
AuthorsOpenReview ↗

Dataset access and subset release

We thank all three reviewers for emphasizing data accessibility. We will release a verifiable subset on November 28, 2026. It comprises 12,696 speech clips from 19 participants, with measured clip durations totaling approximately 9.5 hours, paired photos, HRV, sensor records, and nine-class labels. Documentation, selection manifests, evaluation splits, and reproduction code will accompany the subset. The release arrangement has been agreed with the data holder; access conditions will be specified in the release documentation.

Numbering: Unless marked as original, references below use the revised manuscript. Appendices D–G are added in this revision.

Paper change: Appendix C.1 specifies the release date and access arrangements; Appendix F describes subset composition and selection.

AuthorsOpenReview ↗

Summary of Revisions and Responses to the Main Concerns

Dear Area Chair and Reviewers, Thank you for the constructive feedback. Our contribution combines a Chinese multimodal dataset, progressive three-stage collection, consensus annotation, and a speaker-independent benchmark. We summarize the principal revisions below.

  1. Data access. We will release a verifiable subset on November 28, 2026, as agreed with the data holder: 12,696 retained speech clips from 19 participants, totaling approximately 9.5 hours of measured clip duration, with paired photos, HRV, sensor records, and nine-class labels. Documentation, selection manifests, evaluation splits, and reproduction code will accompany the subset. (Appendix C.1; new Appendix F.)
  2. Numerical correction. We thank Reviewer LEEC for identifying the inconsistency between original Tables 19–20. Original Table 19 retained earlier experimental results. Aggregate metrics, class recalls, and confusion matrices now use the final predictions underlying original Table 20 on the same 10,993 test clips. Speech, speech+vision, and three-modality systems obtain Accuracy of 73.77%/65.94%/66.60% and Macro-F1 of 50.84%/47.51%/47.79%, respectively. We revised the interpretation to a class-recall trade-off; none of the 14 class-wise comparisons survives Holm correction. (Main Table 7; Appendix Tables 21–23.)
  3. Additional experiments and benchmark scope. We added frozen-encoder cross-stage tests and matched-budget ordinary versus class-weighted cross-entropy controls. Curriculum training is presented as a learning-order diagnostic. We removed the main MELD comparison and associated ranking claims; IEMOCAP remains a separate four-class appendix reference. (Sections 4.3–4.5; Appendix B.)
  4. Label definition and dataset documentation. The title, abstract, and methods consistently define the target as speech-perceived emotion, with photos and sensors as auxiliary observations. We document the original emotion definitions and examples, rating-count distributions, class-specific agreement, participant-by-stage recordings, sampling configurations, and modality missingness. (Appendices A and B; new Appendices E and G.) These updates are incorporated in the revised manuscript and detailed in our reviewer-specific replies. Numbering above follows the revised manuscript; Appendices D–G are added in this revision. We appreciate your consideration of the revised evidence. Authors of Paper #39153

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.