- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multimodal emotion recognition method that uses audio, visual, and text information. The proposed method uses a nonlinear antisymmetric scorer to learn pairwise correction between emotions. By comparing pairs of candidates, the model learns to refine semantic emotion prediction and improve recognition on unseen emotion categories.
Critical strengths
-
The method of a class-agnostic, pairwise correction is an interesting contribution.
-
The use of teachers to simulate unseen-category recognition is sound.
-
The author conducted extensive experiments, and the results are promising.
Critical weaknesses
-
The most significant concern is that the paper titled Generalized Zero-Shot Recognition, however, I have checked all math in the paper, it doesn't have any approach to encourage the generalizability. The only assumption is that if we can correct the disagreement on the training set on , it can be assumed to work in .
-
Modality conflict can not be fully resolved. For example, if we have , and the left hand side has noisy text, and the right hand side has noisy voice. The network will have the same correction.
-
For (1), the evidence can be seen from the experiments, where on MELD, unseen-class WF1 improves by only 0.02 percentage points over ProEmoTrans-VAT.
Questions for authors
-
The enforced antisymmetry only works for 2 attributes, if multiple attributes are coexist, is there anyway to enforce to make sure
-
How sensitive is the method to the number and semantic composition of unseen candidate classes, as the correction depends all other classes?
2 comments
Response to reviewer igg3 (Part 2/2, Weaknesses)
W1 Generalized zero-shot learning and mechanism clarification
Generalized zero-shot learning is defined to train a model for classifying data samples under the condition that some output classes are unknown during supervised learning[1]. We designed and stated in Section 3.3 with an explicit transfer-oriented learning: category-excluded teaching. Each teacher and its probes omit selected source categories and correction-query utterances from fitting. Excluded-category and query utterances are also removed from teacher context fragments. The correction learner then receives labeled source queries whose categories may be unfamiliar to the teacher producing the evidence.
Final-unseen categories and examples are absent from correction training. Table 2 includes a single all-seen teacher and a count-matched six-teacher all-seen bank, which provide controls for the teaching construction. These are distinct from simply assuming that correction of familiar-category predictions will transfer.
The generalization designs are more presented in pseudo-unseen classes: a category that is pseudo-unseen to a teacher is still a labeled source category for the correction head. The construction encourages learning from unfamiliar-category predictions but does not guarantee generalization to arbitrary unseen semantics.
[1] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, Q. M. Jonathan Wu. A Review of Generalized Zero-Shot Learning Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4051-4070, 2023.
W2 Indistinguishable evidence and different sources of noise
We agree that a deterministic model cannot distinguish different latent noise sources when its complete input evidence is completely identical. However, Equation (1) retains separate audio, text and visual margins, so noisy text and noisy audio need not produce identical input vectors. Also, equality of one pair’s correction does not imply equality of the final class residual, which aggregates the other incident comparisons and is added to the baseline score.
W3 The small unseen improvement on MELD
In generalized zero-shot learning (GZSL), both the ability of learning from seen classes and unseen classes are crucial, which leads to using overall F1 score and H score to evaluate performance. In MELD, our proposed transfer ensured high recognition ability preservation on seen categories, compared to ProEmoTrans models that drops seen F1 significantly, while achieving good performance on unseen transfer, which remains a pretty hard task due to the dataset native that it was obtained from TV shows, and put all other models except for ProEmoTrans and PMET struggling.
We present the precision, recall and F values for each classes in MELD here, to further highlight the ability of seen categories learning and overall GZSL emotion recognition.
| Class | Group | VAT P/R/F1 | PMET P/R/F1 | ΔP/R/F1 |
|---|---|---|---|---|
| joy | seen | 67.66 / 28.11 / 39.72 | 60.75 / 44.28 / 51.22 | -6.91 / +16.17 / +11.50 |
| anger | seen | 70.00 / 8.12 / 14.55 | 56.40 / 28.12 / 37.52 | -13.60 / +20.00 / +22.97 |
| neutral | seen | 75.42 / 50.56 / 60.53 | 72.86 / 69.27 / 71.02 | -2.56 / +18.71 / +10.49 |
| sadness | seen | 50.00 / 4.81 / 8.77 | 51.61 / 7.69 / 13.39 | +1.61 / +2.88 / +4.62 |
| disgust | unseen | 6.67 / 20.59 / 10.08 | 10.39 / 11.76 / 11.03 | +3.72 / -8.83 / +0.95 |
| fear | unseen | 2.35 / 32.00 / 4.38 | 2.69 / 26.00 / 4.88 | +0.34 / -6.00 / +0.50 |
| surprise | unseen | 23.66 / 54.80 / 33.05 | 29.17 / 37.37 / 32.76 | +5.51 / -17.43 / -0.29 |
Response to reviewer igg3 (Part 1/2, Questions)
Thank you for raising questions about consistency across candidate comparisons and transfer beyond the source vocabulary. We first clarify how the final ranking is obtained and how vocabulary sensitivity is evaluated. We then address the transfer-learning mechanism, the limitation from indistinguishable evidence, and the MELD result.
Q1 Antisymmetry with more than two candidates and cycle consistency
Antisymmetry applies to every candidate pair for any vocabulary size; it is not restricted to a two-candidate task. Equation (2) enforces that reversing a pair reverses its correction. However, antisymmetry alone does not require the raw edges to satisfy eab+ebc+eca=0.
Equations (3) and (6) instead project those edges onto scalar class potentials delta. For the projected differences, define e^ab=δa−δb. Then
e^ab+e^bc+e^ca=(δa−δb)+(δb−δc)+(δc−δa)=0
Final logit differences have the same telescoping property. Thus the deployed scalar scores define a coherent ranking, up to ties, even when raw pair preferences contain cycles.
Q2 Sensitivity to the number and semantic composition of unseen candidates
The correction depends on the available comparison set. Permutation equivariance means that reordering candidates consistently reorders their scores; it does not imply invariance to adding or removing competitors.
We evaluate with a frozen-roster experiment, retaining all seen candidates and enumerate nonempty subsets of the original unseen classes. They recompute roster-dependent evidence and corrections and compare against sliced full-roster scores on identical eligible examples. Single-label additions use the smaller roster’s fixed evaluation population.