- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VISTA, a value-informed appraisal framework for multimodal emotion recognition under conflicting affective evidence. It introduces a seven-field appraisal representation covering personal concerns, event relations, and expression conditions, which guides modality arbitration and emotion prediction. The method is motivated by a log-odds decomposition and a representation-refinement analysis.
Experiments using Qwen2.5-Omni-7B across five benchmarks demonstrate its effectiveness. VISTA achieves 64.5% conflict accuracy on CA-MER, outperforming Modality-Gate-SFT by 2.5 percentage points. Additional ablations and intervention experiments support the usefulness of appraisal-guided emotion recognition.
Critical strengths
-
Well-motivated framework. VISTA addresses an important problem in multimodal emotion recognition: emotional cues from different modalities may conflict, and their interpretation depends on the underlying event and personal concerns. The proposed appraisal interface provides a reasonable way to incorporate such contextual information into modality arbitration.
-
Comprehensive experimental evaluation. The paper evaluates VISTA across five benchmarks and includes informative ablation studies. The improvements are more pronounced on conflicting inputs, and experiments such as appraisal shuffling and removing the decision connection provide useful evidence for the contribution of the proposed framework.
Critical weaknesses
-
Limited modeling of cue–appraisal interactions. The paper emphasizes that appraisal changes the diagnostic meaning of emotional cues. However, the Semantic Arbitration module uses appraisal only to produce a scalar weight for each modality, while the unimodal classification logits remain unchanged. Thus, the method mainly performs appraisal-conditioned modality reweighting rather than explicitly reinterpreting individual emotional cues. The authors should clarify this distinction or moderate the corresponding claims.
-
High inference cost. VISTA requires separate unimodal forward passes and a joint multimodal pass, together with appraisal generation. While these operations serve the proposed design, they introduce considerable computational overhead. As reported in Table 30, VISTA has a median inference latency of 15.5 seconds, compared with 6.0 seconds for Modality-Gate-SFT.
Questions for authors
-
Implementation details. The architecture of the Semantic Arbitration module is not clearly specified. Is it a separate network or part of the backbone? How are the appraisal fields encoded and passed to this module? In addition, the construction of the supervision targets for the arbitration and conflict losses could be explained more clearly.
-
EmoMM evaluation protocol. Table 3 reports results on EmoMM, but the benchmark data and evaluation splits do not appear to be readily available from the official project repository. Could the authors clarify how the evaluation data were obtained and provide the exact evaluation protocol or instructions for reproducing these results?
4 comments
Inference efficiency and EmoMM evaluation (2/2)
3. Inference cost. Reusing media encodings and identical-prefix KV states reduces VISTA's end-to-end P50 and P95 by 20.00%. The submission reports median latencies of 15.5 seconds for VISTA and 6.0 seconds for Modality-Gate-SFT on one A100 at batch size 1 (Table 30). Gate's timing there includes explanation generation. The measurements below use native classification: Gate directly reads out the class in 3.2 seconds, while VISTA generates appraisal before the final class readout.
| Native classification configuration | P50 (s) | P95 (s) | Peak allocated memory (GiB) | Conflict accuracy (%) |
|---|---|---|---|---|
| Modality-Gate-SFT | 3.2 | 8.8 | 28.4 | 62.00 |
| VISTA | 15.5 | 31.0 | 31.8 | 64.50 |
| VISTA with media-encoding reuse and same-prefix KV caching | 12.4 | 24.8 | 34.0 | 64.50 |
Before optimization, appraisal generation takes a median of 10.8 seconds. With computation reuse, mean appraisal length remains 340.2 tokens, and reported conflict accuracy is unchanged. Peak memory rises from 31.8 to 34.0 GiB. Computation is reused within each input while preserving the fields and output length.
We timed inference on an A100-SXM4-80GB with batch size 1, BF16, and greedy decoding. We ran 20 warm-up inputs, followed by 200 fixed timed inputs with CUDA synchronization. Accuracy was evaluated on the full CA-MER set across the three trained seeds. Appraisal generation accounts for most of the measured inference cost.
4. EmoMM data and evaluation protocol. We followed the EmoMM v1 definitions using source media from the CH-SIMS v2.0 and CMU-MOSI releases. Our test set contains 800 original clips, 400 per language, grouped into 160 parent-source clusters. The submission's “800 sources” refers to these clips.
The 800 complete inputs comprise 300 aligned and 500 conflict inputs. Each clip also produces three single-modality-missing views, giving 2,400 missing-modality inputs, including 1,200 classified as conflict plus missing. These views share their original clips. The complete conflict subset contains 272 Chinese and 228 English clips, with 666 and 534 conflict-plus-missing views, respectively.
An input is conflicting when at least one available modality's annotated polarity differs from the multimodal reference. We score alignment, conflict, missingness, and conflict plus missing separately, then give these accuracies equal weight in the four-condition macro summary. Conflict plus missing is a subset of the missingness condition (Appendix E).
We directly evaluate the common-stage checkpoints selected on development data for seeds 42/43/44 on EmoMM. Video is sampled at 2 fps with at most 16 frames, audio at 16 kHz, and context is limited to five preceding utterances. Modality masks identify the available inputs. The five sentiment labels are N/WN/Neu/WP/P.
For the EmoMM evaluation (Appendix E and Table 16), removing parent sources shared with training and checkpoint-selection data leaves 454 clips in 48 clusters. On the strictly independent conflict subset, VISTA scores 49.50% versus 47.49% for Gate across 266 inputs. On the 624 conflict-plus-missing inputs, the scores are 41.35% and 38.89%. The approximate paired 95% intervals for these differences are [−2.60, 6.61] and [−1.56, 6.48] percentage points.
Please let us know if you have further questions. We would be glad to discuss these results with you.
If you find these explanations and results convincing, we would be grateful if you could reconsider your assessment of the paper. Such encouragement would mean a great deal to us as we pursue careful research at a time when AI is reshaping the field. Your evaluation and feedback help the community assess and build on the work, and we would be honored to see it make a useful contribution. Thank you for the time and care you have devoted to our paper.
Thank you for the detailed rebuttal. I am satisfied with the authors' responses and have no further questions.
Thank you for your positive assessment
Thank you for reading our responses so carefully and for thinking so highly of our work. Your recognition is a tremendous encouragement to our whole team. On behalf of my coauthors and myself, please accept our sincere thanks and respect. We are also glad that we could clarify most of the questions without adding too much to your reading burden.
You may also want to examine the work more closely. Perhaps the research question has sparked your curiosity and you would like to explore it further, or you would like more detail to support your judgment and positive assessment. Your questions have been especially valuable in helping us improve the work. We have updated both the main text and appendix in the PDF, and we hope the revision lives up to your high regard for our work.
We have also corrected several typos and formatting issues. We are proud to present this revised manuscript to you, and we do so with deep respect. Changes are marked in red.
You can find the revisions related to your questions here:
-
Concern 1. Modality weighting and cue–appraisal interaction. Main text: Sec. 3.2, p. 5, lines 221–227; Sec. 4.4, p. 8, lines 413–428 (Table 2). Appendix D.4, p. 26, lines 1371–1403; Appendix O.5, pp. 54–55, lines 2894–2949.
-
Concern 2. Inference cost. Main text: Sec. 4.5, p. 10, lines 514–520. Appendix O.6, p. 56, lines 2982–3012 (Tables 51–52).
-
Q1. Implementation and loss targets. Main text: Sec. 3.2, p. 4, lines 201–215; Sec. 3.3, p. 5, lines 239–244. Appendix A.2, pp. 16–17, lines 847–874; Appendix M.2, pp. 46–47, lines 2479–2519.
-
Q2. EmoMM evaluation protocol. Main text: Sec. 4.5, p. 9, lines 473–480. Appendix E.3, p. 32, lines 1688–1707; Appendix O.2, pp. 51–52, lines 2744–2800 (Table 42).
Once again, our sincere thanks.
Arbitration mechanism and supervision (1/2)
Thank you for asking us to clarify how appraisal changes a prediction and how the EmoMM evaluation was conducted. Your comments helped us improve both descriptions. We also appreciate your positive assessment of the motivation, five-benchmark evaluation, and ablations on conflicting inputs.
1. Cue–appraisal interaction. VISTA implements appraisal-conditioned modality reweighting. At fixed evidence, the unimodal logits um and joint residual stay fixed, and appraisal changes the scalar weights αm, giving
=\sum_m\Delta\alpha_m(u_{m,y}-u_{m,y'}).$$ In this mechanism (Section 3.2 and Appendix D.4), appraisal changes how much each modality contributes to a class preference. For example, suppose the visual branch favors happiness over sadness more strongly than the other branches. Reducing its weight under an appraisal of social courtesy reduces its contribution to that class contrast. We measured the interaction on 120 coherent quadruplets, each containing two cue conditions and two appraisal states. Within each cue condition, evidence representations and logits stayed fixed while appraisal changed. The mean class-pair log-odds were $L_{00}=-0.250$, $L_{10}=0.100$, $L_{01}=-0.080$, and $L_{11}=0.540$. The interaction from Eq. 3 was $$(L_{11}-L_{01})-(L_{10}-L_{00})=0.270,$$ with an approximate source-adjusted 95% CI of [0.132, 0.408]. The interaction had the prespecified direction in 86/120 quadruplets (71.67%). Annotation was blind to predictions, and the class pairs, directions, and seed-42 checkpoint were fixed before evaluation. These results support a cue–appraisal interaction in output log-odds through modality reweighting. **2. Arbitration architecture and supervision.** Semantic Arbitration is a separate small MLP attached to the Thinker representations. Each unimodal vector is projected from 3584 to 128 dimensions. The three projections, six entropy/maximum-probability features, and the 128-dimensional appraisal form a concatenated 518-dimensional input. Linear(518,128), GELU, and Linear(128,3) jointly produce the modality scores. A masked softmax normalizes these scores over available modalities. The weighted unimodal logits are added to a joint-evidence residual. The seven appraisal fields contain text, categorical, scalar, and binary values, serialized as compact JSON in the order `g,c,e,a,k,n,r`. Tokenizer offsets select sub-tokens overlapping valid field-value spans. We take a global mean of their final-layer hidden states, with equal token weights. The selected spans include nested keys and punctuation inside the compound `n` value. A trainable linear projection gives $z_0\in\mathbb R^{128}$. A field head predicts 21 logits from $z_0$, with slice widths 3/3/5/1/4/5 for `c,e,a,k,n,r`. Categorical slices use separate softmax operations; control and norm bits use sigmoid. Their concatenated probabilities $q^{\mathrm{field}}$ are projected through a fixed matrix and added to $z_0$, $$F_{jk}=\frac{\sin((j+1)(k+1))}{\sqrt{128}},\quad F\in\mathbb R^{21\times128},\qquad z=(z_0+q^{\mathrm{field}}F)\,\mathbf 1[\text{at least one field is valid}].$$ The goal field contributes through value-token pooling. Invalid top-level values are serialized as `null`, and the pooling mask selects valid value tokens. The field head supplies the predicted feedback vector. Confidence and evidence support field acceptance, supervision, and auditing. Field losses use validity masks. For CH-SIMS examples with complete training labels, the arbitration target is $q_m\propto\exp(-|y_m-y|/0.25)$. For classification examples with at least one correct available frozen-teacher branch, target probability is shared equally among the available branches that predict the training label. Validity-masked cross-entropy against $q$ has coefficient 0.5. The conflict head uses a continuous target $$\widetilde C=\frac{|y_V-y_A|+|y_V-y_T|+|y_A-y_T|}{4}\in[0,1],$$ with a validity-masked squared-error loss on CH-SIMS training examples that have all three modality labels, also with coefficient 0.5. Here $\widetilde C$ is the normalized form of the $[0,4]$ disagreement quantity in Appendix E.