acceptodds
Under review as a conference paper at ICLR 2027

Pairwise Modality Evidence Transfer for Generalized Zero-Shot Emotion Recognition

Abstract

Generalized zero-shot emotion recognition requires identifying both seen and unseen emotions using task-specific labeled examples only from seen categories. Multimodal observations provide complementary evidence for this task, but audio, text and visual predictors may favor different candidates. Moreover, a correction learned from errors on familiar categories may not remain useful when new categories enter the decision. We propose Pairwise Modality Evidence Transfer (PMET), a shared nonlinear correction of comparative multimodal evidence, to learn a reusable correction from these comparative preferences. We utilize a nonlinear antisymmetric scorer to combine modality margins, the baseline margin and description similarity, generated from semantic recognizer; uniform aggregation converts its pair corrections into residual class scores. To expose the learner to unfamiliar-category predictions during source training, we construct teachers that exclude selected source categories from fitting. Ground-truth query supervision jointly trains uniform and auxiliary relational aggregation, with only the uniform path retained at inference. The formulation changes candidate-level ordering while preserving the underlying semantic recognizer and its calibration policy. Experiments show that our proposed PMET achieves best performance on multiple emotion recognition datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generalized zero-shot learning, Multimodal emotion recognition, Semantic transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_igg3OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal emotion recognition method that uses audio, visual, and text information. The proposed method uses a nonlinear antisymmetric scorer to learn pairwise correction between emotions. By comparing pairs of candidates, the model learns to refine semantic emotion prediction and improve recognition on unseen emotion categories.

Critical strengths
  1. The method of a class-agnostic, pairwise correction is an interesting contribution.

  2. The use of teachers to simulate unseen-category recognition is sound.

  3. The author conducted extensive experiments, and the results are promising.

Critical weaknesses
  1. The most significant concern is that the paper titled Generalized Zero-Shot Recognition, however, I have checked all math in the paper, it doesn't have any approach to encourage the generalizability. The only assumption is that if we can correct the disagreement on the training set on SS, it can be assumed to work in UU.

  2. Modality conflict can not be fully resolved. For example, if we have eck1=eck2e_{ck}^1 = e_{ck}^2, and the left hand side has noisy text, and the right hand side has noisy voice. The network will have the same correction.

  3. For (1), the evidence can be seen from the experiments, where on MELD, unseen-class WF1 improves by only 0.02 percentage points over ProEmoTrans-VAT.

Questions for authors
  1. The enforced antisymmetry only works for 2 attributes, if multiple attributes are coexist, is there anyway to enforce eab+ebc+eca=0e_{ab} + e_{bc} + e_{ca} = 0 to make sure a>b>ba>b>b

  2. How sensitive is the method to the number and semantic composition of unseen candidate classes, as the correction depends all other classes?

2 comments
AuthorsOpenReview ↗

Response to reviewer igg3 (Part 1/2, Questions)

Thank you for raising questions about consistency across candidate comparisons and transfer beyond the source vocabulary. We first clarify how the final ranking is obtained and how vocabulary sensitivity is evaluated. We then address the transfer-learning mechanism, the limitation from indistinguishable evidence, and the MELD result.

Q1 Antisymmetry with more than two candidates and cycle consistency

Antisymmetry applies to every candidate pair for any vocabulary size; it is not restricted to a two-candidate task. Equation (2) enforces that reversing a pair reverses its correction. However, antisymmetry alone does not require the raw edges to satisfy eab+ebc+eca=0e_{ab}+e_{bc}+e_{ca}=0.

Equations (3) and (6) instead project those edges onto scalar class potentials delta. For the projected differences, define e^ab=δa−δb\hat{e}_{ab}=\delta_a-\delta_b. Then

e^ab+e^bc+e^ca=(δa−δb)+(δb−δc)+(δc−δa)=0\hat{e}_{ab}+\hat{e}_{bc}+\hat{e}_{ca}=(\delta_a-\delta_b)+(\delta_b-\delta_c)+(\delta_c-\delta_a)=0

Final logit differences have the same telescoping property. Thus the deployed scalar scores define a coherent ranking, up to ties, even when raw pair preferences contain cycles.

Q2 Sensitivity to the number and semantic composition of unseen candidates

The correction depends on the available comparison set. Permutation equivariance means that reordering candidates consistently reorders their scores; it does not imply invariance to adding or removing competitors.

We evaluate with a frozen-roster experiment, retaining all seen candidates and enumerate nonempty subsets of the original unseen classes. They recompute roster-dependent evidence and corrections and compare against sliced full-roster scores on identical eligible examples. Single-label additions use the smaller roster’s fixed evaluation population.

DatasetUnseen rosterWF1Unseen WF1AUSUC
IEMOCAP-7frustration62.9172.120.5590
 frustration + surprise55.9665.470.3659
 frustration + fear59.6967.820.4579
 surprise57.5925.210.5810
 surprise + fear52.6121.130.3759
 fear64.0110.880.5975
 frustration + surprise + fear54.3062.620.3123
MELDdisgust63.4012.840.3821
 disgust + fear56.049.390.2035
 disgust + surprise55.3427.340.2273
 fear60.386.100.3621
 fear + surprise53.4129.190.2357
 surprise58.3232.550.3888
 frustration + surprise + fear52.0025.560.1560
MER2024worry75.0837.820.6985
 surprise79.608.160.6035
 worry + surprise71.3330.260.3440
AuthorsOpenReview ↗

Response to reviewer igg3 (Part 2/2, Weaknesses)

W1 Generalized zero-shot learning and mechanism clarification

Generalized zero-shot learning is defined to train a model for classifying data samples under the condition that some output classes are unknown during supervised learning[1]. We designed and stated in Section 3.3 with an explicit transfer-oriented learning: category-excluded teaching. Each teacher and its probes omit selected source categories and correction-query utterances from fitting. Excluded-category and query utterances are also removed from teacher context fragments. The correction learner then receives labeled source queries whose categories may be unfamiliar to the teacher producing the evidence.

Final-unseen categories and examples are absent from correction training. Table 2 includes a single all-seen teacher and a count-matched six-teacher all-seen bank, which provide controls for the teaching construction. These are distinct from simply assuming that correction of familiar-category predictions will transfer.

The generalization designs are more presented in pseudo-unseen classes: a category that is pseudo-unseen to a teacher is still a labeled source category for the correction head. The construction encourages learning from unfamiliar-category predictions but does not guarantee generalization to arbitrary unseen semantics.

[1] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, Q. M. Jonathan Wu. A Review of Generalized Zero-Shot Learning Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4051-4070, 2023.

W2 Indistinguishable evidence and different sources of noise

We agree that a deterministic model cannot distinguish different latent noise sources when its complete input evidence is completely identical. However, Equation (1) retains separate audio, text and visual margins, so noisy text and noisy audio need not produce identical input vectors. Also, equality of one pair’s correction does not imply equality of the final class residual, which aggregates the other incident comparisons and is added to the baseline score.

W3 The small unseen improvement on MELD

In generalized zero-shot learning (GZSL), both the ability of learning from seen classes and unseen classes are crucial, which leads to using overall F1 score and H score to evaluate performance. In MELD, our proposed transfer ensured high recognition ability preservation on seen categories, compared to ProEmoTrans models that drops seen F1 significantly, while achieving good performance on unseen transfer, which remains a pretty hard task due to the dataset native that it was obtained from TV shows, and put all other models except for ProEmoTrans and PMET struggling.

We present the precision, recall and F values for each classes in MELD here, to further highlight the ability of seen categories learning and overall GZSL emotion recognition.

ClassGroupVAT P/R/F1PMET P/R/F1ΔP/R/F1
joyseen67.66 / 28.11 / 39.7260.75 / 44.28 / 51.22-6.91 / +16.17 / +11.50
angerseen70.00 / 8.12 / 14.5556.40 / 28.12 / 37.52-13.60 / +20.00 / +22.97
neutralseen75.42 / 50.56 / 60.5372.86 / 69.27 / 71.02-2.56 / +18.71 / +10.49
sadnessseen50.00 / 4.81 / 8.7751.61 / 7.69 / 13.39+1.61 / +2.88 / +4.62
disgustunseen6.67 / 20.59 / 10.0810.39 / 11.76 / 11.03+3.72 / -8.83 / +0.95
fearunseen2.35 / 32.00 / 4.382.69 / 26.00 / 4.88+0.34 / -6.00 / +0.50
surpriseunseen23.66 / 54.80 / 33.0529.17 / 37.37 / 32.76+5.51 / -17.43 / -0.29
Reviewer_65mjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Pairwise Modality Evidence Transfer (PMET) for generalized zero-shot multimodal emotion recognition. The method learns antisymmetric pairwise corrections from audio, text, visual evidence, baseline scores, and semantic description similarity. Category-excluded teachers generate pseudo-unseen predictions for correction training. A shared nonlinear scorer is optimized through uniform aggregation and auxiliary relational supervision. Inference retains the uniform aggregation path. Experiments on IEMOCAP, MELD, and MER2024 demonstrate improvements over several adapted baselines.

Critical strengths
  1. PMET explicitly models how different modalities favor competing emotion categories rather than relying exclusively on fixed modality weights.
  2. The antisymmetric scorer guarantees reversal consistency, while uniform aggregation produces centered class residuals.
  3. The relationship to least-squares ranking is correctly established in Appendix A.
  4. The category-excluded teacher design addresses a relevant difficulty in generalized zero-shot learning. Table 2 includes both a single all-seen teacher and a matched bank of six all-seen teachers.
  5. Table 2 examines modality evidence, directional margins, direct MLP fusion, teacher construction, and both supervision paths.
  6. The paper reports seen and unseen metrics, calibration curves, per-class F1, and confusion matrices.
Critical weaknesses
  1. Pairwise ranking, adaptive multimodal fusion, semantic compatibility, and pseudo-unseen training have established precedents. PMET combines these ideas in a shared nonlinear correction framework. However, Table 2 does not include a linear pairwise scorer or a comparably trained candidate-set-aware score correction. Consequently, the experiments do not establish whether the specific nonlinear pairwise formulation provides a substantial contribution beyond established score fusion and ranking mechanisms.
  2. Equation 1 includes cosine similarity between emotion descriptions, which is intended to condition the pairwise correction on semantic relationships. However, no ablation removes this feature while retaining the remaining signed margins. Appendix A also establishes that a globally affine scorer eliminates the direct contribution of symmetric similarity during antisymmetrization. The experiments therefore do not determine whether the proposed nonlinear semantic interaction materially contributes to recognition accuracy.
  3. Equation 4 retains the baseline seen-class calibration penalty, but Appendix A shows that learned corrections can modify its effective operating point. Thus, retaining the numerical calibration parameter does not imply preserving the original decision boundary. The experiments do not compare PMET against simpler learned calibration or score-rescaling mechanisms trained on the same correction queries. The reported improvements cannot be attributed exclusively to transferable pairwise modality evidence.
  4. The substantial gains in overall weighted F1 do not consistently translate into stronger unseen-category recognition. On MELD, unseen weighted F1 increases by only 0.02 percentage points, while unseen macro recall decreases from 35.79% to 25.04%. The AUSUC improvement over the strongest competing method is only 0.0002. On MER2024, the harmonic mean decreases from 37.43% to 31.96% relative to ProEmoTrans-VAT, despite a 10.58-point gain in overall weighted F1. These results limit the evidence for consistently improved generalized zero-shot transfer.
  5. The main experiments use fixed category-transfer partitions with only four seen classes. IEMOCAP-4 includes three alternative holdouts, but the other datasets are evaluated under a single reported seen–unseen division. The paper does not establish how reliably the learned correction transfers when the semantic composition or number of unseen candidates changes. This is important because adaptability to changing candidate vocabularies is a central motivation of PMET.
  6. The weighted relational projection in Equations 5-8 is used only during training. Removing its loss reduces weighted F1 by 0.83 points on IEMOCAP-7 and 0.77 points on MER2024. However, the study does not compare learned relational weights against a fixed-weight auxiliary projection or an alternative auxiliary classification objective.
  7. Tables 1 and 2 report single numerical results without repeated-seed statistics, standard deviations, or confidence intervals. This is particularly important for the small improvements in AUSUC and the auxiliary supervision ablations. The experiments do not establish whether these differences are robust to initialization, teacher training variability, or correction optimization.
  8. Table 1 combines paper-based adaptations of methods with different encoder resources, observation modalities, parameter budgets, and training procedures. Appendix D.1 explicitly describes a custom reconstruction of VML rather than an evaluation of its original implementation. Although a common test population improves comparability, it does not establish that all methods received equally effective adaptations. The complete-system comparisons therefore provide weaker evidence of methodological superiority than the controlled comparisons against the shared ProEmoTrans-VAT foundation.
  9. The method requires multiple independently trained category-excluded teachers during correction learning, three modality probes, and quadratic pairwise scoring over candidate classes. The reported 315 trainable scorer parameters do not characterize the complete training or inference cost. No measurements of total training time, inference latency, peak memory consumption, or scaling with candidate vocabulary size are provided. The computational practicality of the method beyond the small evaluated label sets remains insufficiently established.
Questions for authors
  1. Can the authors compare PMET with a linear antisymmetric pairwise scorer and a parameter-matched candidate-set-aware correction network, using identical evidence generators, teacher presentations, and optimization budgets?
  2. What are the results when semantic description similarity is removed from Equation 1? Can the authors quantify whether nonlinear interactions involving this feature improve predictions on unseen categories?
  3. Can the authors compare PMET against learned temperature scaling, affine score correction, and class-independent adaptive calibration trained on the same source correction queries? Do the gains persist when calibration is optimized consistently across methods?
  4. How does PMET perform under additional seen–unseen class partitions and different candidate vocabulary sizes? In particular, does the advantage persist when emotion categories with different semantic relationships are held out?
  5. Can the authors provide an ablation in which the auxiliary relational projection uses fixed uniform weights rather than learned router weights? How much improvement is attributable specifically to the learned relational weighting?
  6. Can the authors provide repeated-seed results and confidence intervals for the principal comparisons and component ablations? Are the smaller improvements in AUSUC and relational supervision statistically distinguishable from training variability?
  7. What are the complete training and inference costs of PMET, including all category-excluded teachers and modality probes? How do latency and memory consumption scale with the number of candidate emotions?
3 comments
AuthorsOpenReview ↗

Response to reviewer 65mj (Part 1/3, Questions 1-3)

Thank you for distinguishing the method’s structural properties from the empirical evidence needed to attribute its gains. We agree that the controlled comparisons, calibration analysis, transfer coverage and uncertainty reporting need further clarification. We address your seven questions in order, followed by the additional concerns in W4 and W8 about unseen outcomes and adapted baselines.

Q1 Linear and candidate-set-aware correction controls

We present the correction controls to zero correction, four-coefficient signed fusion, a direct candidate-wise MLP, an approximately parameter-matched candidate-set-aware (CSA) network, and full PMET. The tables highlights our improvements compared to linear fusion and CSA network, proving the contribution of our proposed nonlinear correction.

Within the results, Linear fusion falls behind CSA network and direct MLP in the recognition performance. CSA network achieved near WF1 in IEMOCAP-7 while being outperformed in AUSUC, with contrast on MER2024 where AUSUC are close but WF1 having a gap between PMET. The closest results come from MELD, where it drops WF1 but performs nearly same on AUSUC, though this metric fails to differentiate MELD methods with all results in that 0.15-0.16 range. The results highlight our contribution to performance improvement more clearly and are merged into revised Table 2.

WF1/AUSUCIEMOCAP-7MELDMER2024
Zero correction48.01 / 0.237641.78 / 0.154360.75 / 0.2954
Linear fusion47.92 / 0.282851.07 / 0.152065.47 / 0.3290
Direct MLP50.22 / 0.286749.02 / 0.152969.26 / 0.3377
CSA network54.19 / 0.282351.24 / 0.155968.50 / 0.3420
PMET54.30 / 0.312352.00 / 0.156071.33 / 0.3440

Q2 Incremental value of explicit description similarity

The no-similarity intervention removes rho from both the pair scorer and the training router while retaining the signed baseline and modality margins. It retrains the correction under the same recipe. Descriptions still define the baseline and probe scores, so this tests explicit pair-similarity information rather than removing semantics altogether. Our proposed method keeps the performance advantage with the introduction of description feature similarities.

Overall WF1IEMOCAP-7MELDMER2024
No-rho, no-sim51.62 / 0.290351.26 / 0.154070.78 / 0.3418
PMET54.30 / 0.312352.00 / 0.156071.33 / 0.3440

Q3 Calibration, score rescaling and a common selection policy

Appendix A gives the example on retaining numerical calibration parameters: if a residual rescales the baseline by a positive factor 1 + alpha, the effective fitted-class penalty becomes gamma / (1 + alpha).

The matched alternatives include a positive global scale with a fitted-class offset (B1), positive groupwise affine correction (B2), and a small class-independent, sample-adaptive fitted-class penalty (B3). All learn from the same source correction queries. The masks refer to each teacher’s fitted categories during training and to the final seen categories at deployment. Those correction methods does not perform well, and introduced mixed results even compared to baseline VAT model on different datasets.

WF1/AUSUCIEMOCAP-7MELDMER2024
VAT48.01 / 0.237641.78 / 0.154360.75 / 0.2954
B145.42 / 0.254247.78 / 0.143562.07 / 0.2954
B246.09 / 0.254647.85 / 0.143658.08 / 0.2959
B347.55 / 0.226049.03 / 0.141960.76 / 0.3167
PMET54.30 / 0.312352.00 / 0.156071.33 / 0.3440
AuthorsOpenReview ↗

Response to reviewer 65mj (Part 2/3, Questions 4-6)

Q4 Additional semantic partitions and candidate vocabulary sizes

We separate transfer to a different fitting-label partition from sensitivity of a frozen model to its candidate roster.

For the first question, the added experiments hold out sadness or anger from the original seen set in each of MELD and MER2024. Each new task has three seen classes, with appropriately refitted final recognizers, three two-of-three teachers, probes and source calibration. Newly unseen categories are excluded from all affected task-fitting and source-context stages. Endpoint IDs stay fixed. This changes semantic composition and the seen/unseen balance, rather than merely relabeling an already trained model’s classes. The results show the strong robustness for our proposed PMET model, with sadness in both datasets and anger in MELD removed, the baseline VAT model suffered significant performance drops, while PMET retains the recognition performance better.

DatasetsSeen/unseen classesVAT (WF1/AUSUC)PMET (WF1/AUSUC)
MELD-sadness3/47.24 / 0.153145.54 / 0.1600
MELD-anger3/452.52 / 0.162355.70 / 0.1768
MER2024-sadness3/320.20 / 0.227156.03 / 0.2447
MER2024-anger3/316.38 / 0.154550.11 / 0.2229

For the second question, the original models retain all four seen candidates while evaluating every nonempty subset of the original unseen set (i.e. frustration, fear, surprise, frustration+surprise, frustration+fear, surprise+fear, for IEMOCAP and MELD. worry, surprise, worry+surprise for MER2024). We presented the detailed table in revised Appendix E.4, and a summarization here. All roster-dependent operations are recomputed in the test, compared to the baseline VAT model, our PMET presented performance improvements among all rosters, as shown with the ΔWF1 values in the table.

DatasetsChanged rostersΔWF1 range
IEMOCAP-76+4.62 ~ +6.20
MELD6+7.91 ~ +10.59
MER20242+8.18 ~ +9.71

Q5 Fixed uniform and fixed nonuniform relational weighting

Fixed uniform relational weights are equivalent to the uniform-only objective when the loss normalization is matched. With all weights fixed to one throughout training, Equation (6) yields Equation (3) for every edge tensor. Equation (8) then becomes one-half CE(uniform) plus one-half CE(uniform), which equals full-weight CE(uniform), including its scorer gradient.

The additional Fixed non-uniform auxiliary projection control keeps PMET’s dual objective but replaces the learned router with positive, symmetric weights based on the fixed ranks of description similarities. The rule is permutation-equivariant and has no outcome-tuned parameter. Both methods deploy the same uniform aggregation. Our comparison show the performance gain on unseen categories transfer by using the proposed relational weighting.

WF1/Unseen RecallIEMOCAP-7MELDMER2024
Fixed uniform54.13 / 43.6651.27 / 24.6071.00 / 19.52
Fixed nonuniform auxiliary projection53.79 / 42.0250.96 / 24.8370.75 / 19.52
PMET54.30 / 44.3952.00 / 25.0471.33 / 19.94

Q6 Repeated seeds and confidence intervals

We have conducted experiments on three separate seeds with same configuration and presented our improvements out of random fluctuations. updated table 1 in the revision to ensure more fair statistical comparison and highlight our statistical improvements from compared methods. We present some rows in this response to state our improvements to compared methods.

IEMOCAP-7FF_SF_UHAUSUC
MTEE38.72±0.2427.10±1.3945.11±0.4122.85±1.510.1070±0.0049
ProEmoTrans-VAT49.04±1.5733.39±3.6457.65±1.2135.51±1.200.2375±0.0169
SubT10.60±1.2314.54±1.028.43±1.5215.21±0.820.1169±0.0040
SubT-VAT32.03±1.7420.12±1.0438.57±2.1221.79±1.190.1615±0.0041
PMET54.08±0.5038.92±0.3062.42±0.8641.06±0.490.3094±0.0039
MELDFF_SF_UHAUSUC
MTEE36.92±2.4741.02±2.9914.23±0.6822.11±1.730.1038±0.0056
ProEmoTrans-VAT41.78±3.0344.70±3.6225.54±0.3026.97±1.960.1543±0.0027
SubT22.59±0.9724.35±1.1612.85±0.0612.86±0.490.0566±0.0035
SubT-VAT30.28±2.0033.13±2.2714.49±0.6420.29±1.270.1052±0.0013
PMET51.80±0.2156.51±0.2525.71±0.0730.01±0.080.1569±0.0008
AuthorsOpenReview ↗

Response to reviewer 65mj (Part 1/3, Question 7, Weaknesses 4&8)

Q7 Complete computational cost and candidate-count scaling

We present a table on theoretical computational cost (FLOPs, MACs), memory consumption, and measured time of our PMET compared to the baseline model in this table.

ModelGFLOPs/batchGMACs/batchPeak GPU Memory (training / inference, MiB)Training time (min)Inference time (s/batch)
Baseline125,224.9162,612.462,422.4 / 1,978.840.0727.905
PMET125,224.9662,612.482,422.4 / 1,980.042.03 + 0.30 (CPU head)28.945

Both rows include parallel fitting of the final recognizer and three teachers used for source calibration. PMET also fits 12 probes and a correction head. The CPU head time includes scoring and export. Our proposed method, as its presented small parameter form, does not introduce much FLOPs/MAC operations, as well as the GPU memory consumption, the CPU head operations introduces only ~5.6% and ~3.7% time increase in training and inference, respectively.

W4 Aggregate gains do not consistently improve unseen recognition

In generalized zero-shot learning, both the ability of learning from seen classes and unseen classes are crucial, which leads to using overall F1 score and H score to evaluate performance. In MELD, our proposed transfer ensured high recognition ability preservation on seen categories, compared to ProEmoTrans models that drops seen F1 significantly, while achieving good performance on unseen transfer, which remains a pretty hard task due to the dataset native that it was obtained from TV shows, and put all other models except for ProEmoTrans and PMET struggling.

We present the precision, recall and F values for each classes in MELD here, to further highlight the ability of seen categories learning and overall GZSL emotion recognition.

ClassGroupVAT P/R/F1PMET P/R/F1ΔP/R/F1
joyseen67.66 / 28.11 / 39.7260.75 / 44.28 / 51.22-6.91 / +16.17 / +11.50
angerseen70.00 / 8.12 / 14.5556.40 / 28.12 / 37.52-13.60 / +20.00 / +22.97
neutralseen75.42 / 50.56 / 60.5372.86 / 69.27 / 71.02-2.56 / +18.71 / +10.49
sadnessseen50.00 / 4.81 / 8.7751.61 / 7.69 / 13.39+1.61 / +2.88 / +4.62
disgustunseen6.67 / 20.59 / 10.0810.39 / 11.76 / 11.03+3.72 / -8.83 / +0.95
fearunseen2.35 / 32.00 / 4.382.69 / 26.00 / 4.88+0.34 / -6.00 / +0.50
surpriseunseen23.66 / 54.80 / 33.0529.17 / 37.37 / 32.76+5.51 / -17.43 / -0.29

W8 Fairness and validity of adapted baseline comparisons

We did not locate a verified official implementation or task checkpoint in the public sources for the VML paper, so we went for a paper-based adaptation.

Reviewer_T2WHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Pairwise Modality Evidence Transfer (PMET), a framework designed for generalized zero-shot emotion recognition using multimodal observations. It formulates a candidate-relative multimodal correction framework using a shared nonlinear antisymmetric scorer that maps signed modality preferences, baseline semantic margins, and description similarities into pair-level corrections. It proposes a category-excluded source teaching procedure paired with a dual uniform/relational supervision design, exposing the learner to pseudo-unseen evidence during training while maintaining a lightweight uniform projection path at deployment.

Critical strengths

The proposed approach significantly outperforms the compared approaches across datasets.

By structuring evidence transfer around pairwise comparative margins rather than absolute class slots, the model generalizes smoothly to unseen emotion descriptions introduced at deployment.

The training strategy cleverly constructs pseudo-unseen exposure from source data using category-omitting teachers, directly addressing the distribution shift inherent to generalized zero-shot learning.

The paper performs detailed ablation studies for the proposed approach.

Critical weaknesses

Writing can be improved. The paper does not define what "score" refers to at the beginning, which can be an issue for readers.

The paper reports single-run point estimates across all datasets without multi-seed standard deviations or confidence intervals, making it difficult to verify the statistical stability of the performance gains.

Unseen classes are represented by author-provided textual descriptions, with IEMOCAP surprise and fear borrowed from the baseline authors’ MELD descriptions. The results are not tested for sensitivity to the wording of the description.

Questions for authors

Could the "score" be described clearly before its use in the initial sections of the paper?

Can the authors provide multi-seed variance estimates (e.g., standard deviations over at least 3 runs) to confirm the statistical significance of the reported gains?

Could the authors evaluate the sensitivity of PMET to alternative descriptions or paraphrases of unseen classes to establish the robustness of semantic transfer to description wording?

1 comment
AuthorsOpenReview ↗

Response to reviewer T2WH

Thank you for the specific requests concerning score definitions, training variability and description wording. We clarify the score terminology below, explain the limits of the current uncertainty evidence, and specify the paired description-sensitivity evaluation.

Q1 Define score before its first use

A semantic score measures cosine compatibility between an utterance representation and a candidate-description representation. A modality probe produces the corresponding compatibility from audio, text or video alone. Their signed differences express which candidate the evidence favors.

These quantities are distinct from the corrected and calibrated logits used for the final prediction. In particular, raw cosine scores and signed margins are not automatically calibrated probabilities or confidence estimates. Equation (1) defines the comparative inputs, while Equation (4) defines the final logits.

We introduced this definition at the first method overview in revision Section 1, before referring to differences between scores.

Q2 Multi-seed variance and statistical stability

We have conducted experiments on three separate seeds with same configuration and presented our improvements out of random fluctuations. updated table 1 in the revision to ensure more fair statistical comparison and highlight our statistical improvements from compared methods. We present some rows in this response to state our improvements to compared methods.

IEMOCAP-7FF_SF_UHAUSUC
MTEE38.72±0.2427.10±1.3945.11±0.4122.85±1.510.1070±0.0049
ProEmoTrans-VAT49.04±1.5733.39±3.6457.65±1.2135.51±1.200.2375±0.0169
SubT10.60±1.2314.54±1.028.43±1.5215.21±0.820.1169±0.0040
SubT-VAT32.03±1.7420.12±1.0438.57±2.1221.79±1.190.1615±0.0041
PMET54.08±0.5038.92±0.3062.42±0.8641.06±0.490.3094±0.0039
MELDFF_SF_UHAUSUC
MTEE36.92±2.4741.02±2.9914.23±0.6822.11±1.730.1038±0.0056
ProEmoTrans-VAT41.78±3.0344.70±3.6225.54±0.3026.97±1.960.1543±0.0027
SubT22.59±0.9724.35±1.1612.85±0.0612.86±0.490.0566±0.0035
SubT-VAT30.28±2.0033.13±2.2714.49±0.6420.29±1.270.1052±0.0013
PMET51.80±0.2156.51±0.2525.71±0.0730.01±0.080.1569±0.0008

Q3 Sensitivity to alternative unseen-class descriptions

Appendix C.1 identifies the class descriptions as resources from the baseline authors. In IEMOCAP, the surprise and fear descriptions come from their MELD resources.

The added test uses two meaning-preserving unseen-description variants for each dataset. Seen descriptions, learned weights and numerical calibration remain fixed. We recompute every affected semantic input for both VAT and PMET, including description embeddings, baseline and probe compatibility scores, and pair similarity. Thus the test changes the semantic resource consistently rather than changing only the similarity coordinate.

On all three datasets, the model show a little performance degradation with custom descriptions, with IEMOCAP-7 showing a relative large gap, while MELD and MER2024 are not affected much. AUSUC in MELD paraphrase 1 even shows a little better AUSUC, with WF1 scores on both unseen and overall left behind.

DatasetDescription setWF1Unseen WF1AUSUC
IEMOCAP-7Original54.3062.620.3123
Paraphrase 152.9159.610.2866
Paraphrase 246.7253.600.2188
MELDOriginal52.0025.560.1560
Paraphrase 151.3024.000.1583
Paraphrase 250.2622.630.1417
MER2024Original71.3330.260.3440
Paraphrase 170.0628.930.3298
Paraphrase 271.3130.310.3339

We added Appendix E.3, “Sensitivity to unseen-description wording” specifying changes and reporting results, and Appendix F, “Exact unseen-description wording sets”, containing the exact original and alternative strings.

Reviewer_St8JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pairwise Modality Evidence Transfer (PMET) for generalized zero-shot emotion recognition. Starting from a frozen multimodal semantic recognizer and separately trained modality probes, PMET constructs candidate-pair features from semantic and modality score margins and description similarity. A shared nonlinear antisymmetric scorer predicts pairwise corrections, which are uniformly aggregated into residual class scores. Category-excluded source teachers expose the correction learner to pseudo-unseen predictions. Training combines supervision of the uniform deployment path with an auxiliary weighted relational projection. Experiments on IEMOCAP, MELD, and MER2024 report improvements in overall weighted F1 and AUSUC over the underlying ProEmoTrans-VAT recognizer.

Critical strengths
  • The paper addresses recognition over joint seen and unseen emotion vocabularies. Its introduction usefully distinguishes a correction function that can be evaluated on new labels from a learning procedure that makes that correction useful on unfamiliar categories.
  • The shared scorer avoids class-specific output parameters, antisymmetry enforces consistent reversal of candidate comparisons, and uniform aggregation has a clear least-squares interpretation. The auxiliary router is discarded at inference.
Critical weaknesses
  • The abstract describes modality disagreement and possible failure of corrections when new labels are introduced, but does not clearly articulate the candidate-pair-dependent discriminative value of modalities that motivates the design. The introduction provides this missing rationale, for example through sadness–neutral versus sadness–frustration comparisons, but these examples are not supported by a direct diagnostic analysis.
  • Section 3.1 mixes the interface required by PMET with implementation details of ProEmoTrans-VAT, including its BERT encoder, inherited CRF and prototype enrichment, and residual cross-attention. Defining the evidence generator is necessary, but the paper should distinguish essential assumptions from one implementation.
  • The authors should add citations where necessary. For example, Emotion recognition systems in introduction.

Typos:

  • brings should be bring in line 34.
Questions for authors

My primary concern is the motivation. In the abstract, the authors should provide a deeper analysis of the underlying problem rather than merely describing the observed phenomenon.

The authors should also check and correct other issues, such as formatting inconsistencies.

1 comment
AuthorsOpenReview ↗

Response to reviewer St8J

Thank you for identifying the gap between the motivation and the evidence supporting the design. Your comments help us separate the underlying hypothesis about modality usefulness from the observed disagreement between predictors. We address the motivation and presentation first, then clarify the required evidence interface and supporting citations.

Q1 Motivation and evidence for candidate-dependent modality usefulness

The underlying hypothesis is that a modality’s discriminative value depends on which emotions are being compared, which is especially important for generalized zero-shot learning models to learn and gain transfer ability. Evidence that separates sadness from neutral may be less useful for separating sadness from frustration. A single fixed weighting of modality scores cannot express every such change in relative usefulness. PMET therefore retains separate signed modality margins for each candidate pair and learns a shared nonlinear correction from those margins, the baseline margin and description similarity.

We chose some emotion pair subset teachers to present audio, text and visual margins, presenting the difference of modality contribution to different emotion pairs with those AUC values. For example, in MELD, joy–surprise and anger–surprise are both seen/unseen comparisons. Text exceeds audio for joy–surprise (AUC 0.7840 versus 0.6060), whereas audio exceeds text for anger–surprise (0.7319 versus 0.6450). And in IEMOCAP, seen-only examples also present similar modality disagreement, with text exceeds audio for happy–neutral (0.6079 versus 0.5217), while audio exceeds text for happy–sadness (0.9187 versus 0.8539).

Emotion PairsAudio AUCText AUCVideo AUC
IEMOCAP happy-neutral0.52170.60790.3751
IEMOCAP happy-sad0.91870.85390.5560
MELD joy-surprise0.60600.78400.6801
MELD anger-surprise0.73190.64500.6422
MER2024 anger-sad0.41370.51620.2829
MER2024 happy-sad0.78440.68190.9563

We revised Abstract and Section 1 to better present motivation. Appendix E.2, “Candidate-pair modality diagnostics” contains results for the emotion pairs, and identifies endpoint results as descriptive and preserves teacher/familiarity distinctions.

Q2 Formatting, terminology and other presentation issues

We agree that the presentation needs a consistent terminology and notation pass. We corrected “brings” to “bring,” replaced “confliction,” and standardize the notation for semantic scores, modality margins, residual corrections and calibrated logits. The revised introduction and Section 3.1 distinguish compatibility scores, signed margins and final logits.

W2 Required evidence interface versus one baseline implementation

The required interface consists of a semantic compatibility score for each candidate, separate modality compatibility scores, and candidate-description similarities. The correction uses this evidence; the particular BERT, CRF, prototype and cross-attention construction belongs to the ProEmoTrans-VAT implementation evaluated here.

Revised section 3.1 now states the required frozen-score/probe/description-similarity interface before the brief ProEmoTrans-VAT implementation and explicitly limits the empirical evidence to that evaluated foundation. Appendix C retains the detailed baseline implementation and fitting description.

W3 Supporting citations in the introduction

We agree that broad claims about emotion-recognition systems and multimodal evidence should be supported where they first appear. We added relevant affective-computing and conversational emotion-recognition citations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates

Dear Area Chair, Reviewers St8J, T2WH, 65mj and igg3,

We appreciate the time and effort you dedicated to throughout the review process. We are writing to provide a brief summary of our rebuttal progress to assist in your evaluation.

Below is the primary concerns raised by each reviewer and the outcomes of our rebuttal:

Reviewer St8J

Key Concern: The paper needed a clearer explanation of why modality usefulness depends on the emotion pair, along with direct evidence for that claim and a clearer definition of the inputs PMET requires.

Response & Outcome: We revised the abstract and introduction to explain the motivation and added pairwise diagnostics in Appendix E.2. We also revised Section 3.1 to define model’s required inputs, and added supporting citations. The reviewer has not yet responded further.

Reviewer T2WH

Key Concern: The reviewer asked us to define scores earlier, quantify variability across training runs, and test sensitivity to the wording of unseen emotion descriptions.

Response & Outcome: We now define compatibility scores, signed margins, and final logits at their first use. Revised Table 1 reports means and standard deviations across three seeds. We also tested two paraphrase sets per dataset with model weights and calibration fixed. Added Appendices E.3 and F report the results and provide the exact descriptions. The reviewer has not yet responded further.

Reviewer 65mj

Key Concern: The reviewer asked for stronger evidence that PMET’s design choices explain its gains, and for a clearer account of unseen-class performance, baseline comparability, and computational cost.

Response & Outcome: We added comparisons against linear fusion, candidate-set-aware correction, and learned calibration, as well as ablations for description similarity and learned relational weights. PMET achieved higher overall weighted F1 than the tested controls and improved over baseline model across the additional class splits and candidate sets. We also added three-seed statistics, reported end-to-end runtime and memory for the measured setup, and clarified how we adapted VML in revised paper. The reviewer has not yet responded further.

Reviewer igg3

Key Concern: The reviewer questioned how PMET learns to transfer to unseen classes, how it handles modality noise and potentially cyclic pairwise preferences, and how its predictions change with the candidate set.

Response & Outcome: We clarified how category-excluded teachers expose the correction learner to predictions for labels unfamiliar to the teacher, with evaluation classes held out of correction training. We explained how projecting pairwise corrections onto scalar class scores yields a coherent ranking and why identical input evidence produces identical corrections even when the underlying noise differs. We also tested changes in the number and composition of unseen candidates with the trained models fixed. The reviewer has not yet responded further.

Thank you again for all your efforts and management of the review process.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.