acceptodds
Under review as a conference paper at ICLR 2027

VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

Abstract

Conflicting emotional cues can be individually valid: a subdued voice may reflect a blocked goal while a smile satisfies a social obligation. Their interpretation depends on what the event means to the person. We introduce VISTA (Value-Informed Semantic Trust Arbitration), a learned seven-field appraisal interface that conditions modality arbitration on concerns, event relations, and expression conditions while retaining a joint-evidence residual. A log-odds decomposition separates emotion expectation from cue diagnosticity, motivating an interface that lets appraisal change how evidence is interpreted. With a shared Qwen2.5-Omni-7B backbone and matched training examples and steps, VISTA reaches 64.5% conflict accuracy on CA-MER, improving on modality gating by 2.5 percentage points on conflict and 0.2 on consistency. Shuffling appraisal across scenes or removing its decision connection reduces this benefit. A common frozen-backbone probe reaches 0.600 macro CCC for appraisal readout, compared with 0.505 for emotion-only fine-tuning. Evaluations across five benchmarks connect recognition under increasing conflict with appraisal readout and downstream decision use. Together, the analyses and experiments support scene-specific appraisal as an intermediate representation that helps interpret conflicting emotional evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal emotion recognition, cross-modal conflict, value-informed appraisal, appraisal theory, modality arbitration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_mkj5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VISTA, a value-informed appraisal framework for multimodal emotion recognition under conflicting affective evidence. It introduces a seven-field appraisal representation covering personal concerns, event relations, and expression conditions, which guides modality arbitration and emotion prediction. The method is motivated by a log-odds decomposition and a representation-refinement analysis.

Experiments using Qwen2.5-Omni-7B across five benchmarks demonstrate its effectiveness. VISTA achieves 64.5% conflict accuracy on CA-MER, outperforming Modality-Gate-SFT by 2.5 percentage points. Additional ablations and intervention experiments support the usefulness of appraisal-guided emotion recognition.

Critical strengths
  1. Well-motivated framework. VISTA addresses an important problem in multimodal emotion recognition: emotional cues from different modalities may conflict, and their interpretation depends on the underlying event and personal concerns. The proposed appraisal interface provides a reasonable way to incorporate such contextual information into modality arbitration.

  2. Comprehensive experimental evaluation. The paper evaluates VISTA across five benchmarks and includes informative ablation studies. The improvements are more pronounced on conflicting inputs, and experiments such as appraisal shuffling and removing the decision connection provide useful evidence for the contribution of the proposed framework.

Critical weaknesses
  1. Limited modeling of cue–appraisal interactions. The paper emphasizes that appraisal changes the diagnostic meaning of emotional cues. However, the Semantic Arbitration module uses appraisal only to produce a scalar weight for each modality, while the unimodal classification logits remain unchanged. Thus, the method mainly performs appraisal-conditioned modality reweighting rather than explicitly reinterpreting individual emotional cues. The authors should clarify this distinction or moderate the corresponding claims.

  2. High inference cost. VISTA requires separate unimodal forward passes and a joint multimodal pass, together with appraisal generation. While these operations serve the proposed design, they introduce considerable computational overhead. As reported in Table 30, VISTA has a median inference latency of 15.5 seconds, compared with 6.0 seconds for Modality-Gate-SFT.

Questions for authors
  1. Implementation details. The architecture of the Semantic Arbitration module is not clearly specified. Is it a separate network or part of the backbone? How are the appraisal fields encoded and passed to this module? In addition, the construction of the supervision targets for the arbitration and conflict losses could be explained more clearly.

  2. EmoMM evaluation protocol. Table 3 reports results on EmoMM, but the benchmark data and evaluation splits do not appear to be readily available from the official project repository. Could the authors clarify how the evaluation data were obtained and provide the exact evaluation protocol or instructions for reproducing these results?

4 comments
AuthorsOpenReview ↗

Arbitration mechanism and supervision (1/2)

Thank you for asking us to clarify how appraisal changes a prediction and how the EmoMM evaluation was conducted. Your comments helped us improve both descriptions. We also appreciate your positive assessment of the motivation, five-benchmark evaluation, and ablations on conflicting inputs.

1. Cue–appraisal interaction. VISTA implements appraisal-conditioned modality reweighting. At fixed evidence, the unimodal logits umu_m and joint residual stay fixed, and appraisal changes the scalar weights αm\alpha_m, giving

=\sum_m\Delta\alpha_m(u_{m,y}-u_{m,y'}).$$ In this mechanism (Section 3.2 and Appendix D.4), appraisal changes how much each modality contributes to a class preference. For example, suppose the visual branch favors happiness over sadness more strongly than the other branches. Reducing its weight under an appraisal of social courtesy reduces its contribution to that class contrast. We measured the interaction on 120 coherent quadruplets, each containing two cue conditions and two appraisal states. Within each cue condition, evidence representations and logits stayed fixed while appraisal changed. The mean class-pair log-odds were $L_{00}=-0.250$, $L_{10}=0.100$, $L_{01}=-0.080$, and $L_{11}=0.540$. The interaction from Eq. 3 was $$(L_{11}-L_{01})-(L_{10}-L_{00})=0.270,$$ with an approximate source-adjusted 95% CI of [0.132, 0.408]. The interaction had the prespecified direction in 86/120 quadruplets (71.67%). Annotation was blind to predictions, and the class pairs, directions, and seed-42 checkpoint were fixed before evaluation. These results support a cue–appraisal interaction in output log-odds through modality reweighting. **2. Arbitration architecture and supervision.** Semantic Arbitration is a separate small MLP attached to the Thinker representations. Each unimodal vector is projected from 3584 to 128 dimensions. The three projections, six entropy/maximum-probability features, and the 128-dimensional appraisal form a concatenated 518-dimensional input. Linear(518,128), GELU, and Linear(128,3) jointly produce the modality scores. A masked softmax normalizes these scores over available modalities. The weighted unimodal logits are added to a joint-evidence residual. The seven appraisal fields contain text, categorical, scalar, and binary values, serialized as compact JSON in the order `g,c,e,a,k,n,r`. Tokenizer offsets select sub-tokens overlapping valid field-value spans. We take a global mean of their final-layer hidden states, with equal token weights. The selected spans include nested keys and punctuation inside the compound `n` value. A trainable linear projection gives $z_0\in\mathbb R^{128}$. A field head predicts 21 logits from $z_0$, with slice widths 3/3/5/1/4/5 for `c,e,a,k,n,r`. Categorical slices use separate softmax operations; control and norm bits use sigmoid. Their concatenated probabilities $q^{\mathrm{field}}$ are projected through a fixed matrix and added to $z_0$, $$F_{jk}=\frac{\sin((j+1)(k+1))}{\sqrt{128}},\quad F\in\mathbb R^{21\times128},\qquad z=(z_0+q^{\mathrm{field}}F)\,\mathbf 1[\text{at least one field is valid}].$$ The goal field contributes through value-token pooling. Invalid top-level values are serialized as `null`, and the pooling mask selects valid value tokens. The field head supplies the predicted feedback vector. Confidence and evidence support field acceptance, supervision, and auditing. Field losses use validity masks. For CH-SIMS examples with complete training labels, the arbitration target is $q_m\propto\exp(-|y_m-y|/0.25)$. For classification examples with at least one correct available frozen-teacher branch, target probability is shared equally among the available branches that predict the training label. Validity-masked cross-entropy against $q$ has coefficient 0.5. The conflict head uses a continuous target $$\widetilde C=\frac{|y_V-y_A|+|y_V-y_T|+|y_A-y_T|}{4}\in[0,1],$$ with a validity-masked squared-error loss on CH-SIMS training examples that have all three modality labels, also with coefficient 0.5. Here $\widetilde C$ is the normalized form of the $[0,4]$ disagreement quantity in Appendix E.
AuthorsOpenReview ↗

Inference efficiency and EmoMM evaluation (2/2)

3. Inference cost. Reusing media encodings and identical-prefix KV states reduces VISTA's end-to-end P50 and P95 by 20.00%. The submission reports median latencies of 15.5 seconds for VISTA and 6.0 seconds for Modality-Gate-SFT on one A100 at batch size 1 (Table 30). Gate's timing there includes explanation generation. The measurements below use native classification: Gate directly reads out the class in 3.2 seconds, while VISTA generates appraisal before the final class readout.

Native classification configurationP50 (s)P95 (s)Peak allocated memory (GiB)Conflict accuracy (%)
Modality-Gate-SFT3.28.828.462.00
VISTA15.531.031.864.50
VISTA with media-encoding reuse and same-prefix KV caching12.424.834.064.50

Before optimization, appraisal generation takes a median of 10.8 seconds. With computation reuse, mean appraisal length remains 340.2 tokens, and reported conflict accuracy is unchanged. Peak memory rises from 31.8 to 34.0 GiB. Computation is reused within each input while preserving the fields and output length.

We timed inference on an A100-SXM4-80GB with batch size 1, BF16, and greedy decoding. We ran 20 warm-up inputs, followed by 200 fixed timed inputs with CUDA synchronization. Accuracy was evaluated on the full CA-MER set across the three trained seeds. Appraisal generation accounts for most of the measured inference cost.

4. EmoMM data and evaluation protocol. We followed the EmoMM v1 definitions using source media from the CH-SIMS v2.0 and CMU-MOSI releases. Our test set contains 800 original clips, 400 per language, grouped into 160 parent-source clusters. The submission's “800 sources” refers to these clips.

The 800 complete inputs comprise 300 aligned and 500 conflict inputs. Each clip also produces three single-modality-missing views, giving 2,400 missing-modality inputs, including 1,200 classified as conflict plus missing. These views share their original clips. The complete conflict subset contains 272 Chinese and 228 English clips, with 666 and 534 conflict-plus-missing views, respectively.

An input is conflicting when at least one available modality's annotated polarity differs from the multimodal reference. We score alignment, conflict, missingness, and conflict plus missing separately, then give these accuracies equal weight in the four-condition macro summary. Conflict plus missing is a subset of the missingness condition (Appendix E).

We directly evaluate the common-stage checkpoints selected on development data for seeds 42/43/44 on EmoMM. Video is sampled at 2 fps with at most 16 frames, audio at 16 kHz, and context is limited to five preceding utterances. Modality masks identify the available inputs. The five sentiment labels are N/WN/Neu/WP/P.

For the EmoMM evaluation (Appendix E and Table 16), removing parent sources shared with training and checkpoint-selection data leaves 454 clips in 48 clusters. On the strictly independent conflict subset, VISTA scores 49.50% versus 47.49% for Gate across 266 inputs. On the 624 conflict-plus-missing inputs, the scores are 41.35% and 38.89%. The approximate paired 95% intervals for these differences are [−2.60, 6.61] and [−1.56, 6.48] percentage points.

Please let us know if you have further questions. We would be glad to discuss these results with you.

If you find these explanations and results convincing, we would be grateful if you could reconsider your assessment of the paper. Such encouragement would mean a great deal to us as we pursue careful research at a time when AI is reshaping the field. Your evaluation and feedback help the community assess and build on the work, and we would be honored to see it make a useful contribution. Thank you for the time and care you have devoted to our paper.

Thank you for the detailed rebuttal. I am satisfied with the authors' responses and have no further questions.

AuthorsOpenReview ↗

Thank you for your positive assessment

Thank you for reading our responses so carefully and for thinking so highly of our work. Your recognition is a tremendous encouragement to our whole team. On behalf of my coauthors and myself, please accept our sincere thanks and respect. We are also glad that we could clarify most of the questions without adding too much to your reading burden.

You may also want to examine the work more closely. Perhaps the research question has sparked your curiosity and you would like to explore it further, or you would like more detail to support your judgment and positive assessment. Your questions have been especially valuable in helping us improve the work. We have updated both the main text and appendix in the PDF, and we hope the revision lives up to your high regard for our work.

We have also corrected several typos and formatting issues. We are proud to present this revised manuscript to you, and we do so with deep respect. Changes are marked in red.

You can find the revisions related to your questions here:

  • Concern 1. Modality weighting and cue–appraisal interaction. Main text: Sec. 3.2, p. 5, lines 221–227; Sec. 4.4, p. 8, lines 413–428 (Table 2). Appendix D.4, p. 26, lines 1371–1403; Appendix O.5, pp. 54–55, lines 2894–2949.

  • Concern 2. Inference cost. Main text: Sec. 4.5, p. 10, lines 514–520. Appendix O.6, p. 56, lines 2982–3012 (Tables 51–52).

  • Q1. Implementation and loss targets. Main text: Sec. 3.2, p. 4, lines 201–215; Sec. 3.3, p. 5, lines 239–244. Appendix A.2, pp. 16–17, lines 847–874; Appendix M.2, pp. 46–47, lines 2479–2519.

  • Q2. EmoMM evaluation protocol. Main text: Sec. 4.5, p. 9, lines 473–480. Appendix E.3, p. 32, lines 1688–1707; Appendix O.2, pp. 51–52, lines 2744–2800 (Table 42).

Once again, our sincere thanks.

Reviewer_aaMqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

VISTA introduces a seven-field event-appraisal representation to guide modality weighting while retaining a joint-evidence pathway. Evaluations across five benchmarks show gains concentrated on conflicting samples. The idea is appealing and the mechanism controls are thoughtful, although uncertainty estimates, source-level data independence, and some implementation details need clarification.

Critical strengths

Even reliable emotional cues can mean different things in different contexts, so reliability-based weighting alone may be insufficient.

The controls are thoughtful. Shared backbones, examples, and training steps, together with length-matched CoT, gating, appraisal shuffling, and connection ablations, help identify where the gains come from.

Beyond recognition accuracy, the paper examines appraisal readout, downstream utility, and pseudo-label quality, while explicitly reporting the additional computational cost.

Critical weaknesses

I see promise in this paper, particularly in its controlled comparisons. My hesitation concerns the strength and verifiability of the evidence rather than the value of the idea. I would consider raising my score if the overlap audit, uncertainty analysis, and implementation details are satisfactorily addressed.

Questions for authors

1.How stable are the mechanism gains? Three-seed means are reported without variability. Could the authors provide per-seed results and paired uncertainty estimates, particularly for the 0.8-point within-emotion shuffle effect and the 1.3-point gain over the generic semantic bottleneck?

2.EmoMM and the training pool share CH-SIMS v2 source material, but overlap counts are not reported. Please quantify any overlap and evaluate after removing it. Absence of additional adaptation alone does not establish independent transfer.

3.How are the seven fields encoded for the arbitrator, how are targets for LarbL_{\mathrm{arb}} constructed, and how are CA-MER’s open-ended labels mapped to the normalized classification interface? These details matter for reproducibility and fair comparison.

4.Current experiments establish dependence on appraisal, but do not directly measure the theoretical interaction. Coherent appraisal replacements at fixed evidence, with changes in modality weights and class log-odds, would help, alongside details of how intervention directions were annotated.

5 comments
AuthorsOpenReview ↗

Uncertainty estimates and source-independent evaluation (1/3)

Thank you for pointing out the source overlap between EmoMM and our training pool. Your comment brought an easily overlooked issue to our attention and helped us improve the evaluation. We also appreciate your positive assessment of the motivation, controlled comparisons, and appraisal evaluation beyond recognition accuracy.

Q1. Stability of the mechanism gains. We evaluated the original checkpoints for seeds 42, 43, and 44 using the original implementation and native nine-class argmax predictions. The submission reports evaluation through a normalized output interface.

MethodSeed 42Seed 43Seed 44Mean ± seed SD (%)
VISTA62.8065.0065.7064.50 ± 1.51
Within-emotion shuffle62.1063.9065.1063.70 ± 1.51
Generic semantic bottleneck61.5063.6064.5063.20 ± 1.54
Modality-Gate-SFT60.7062.0063.3062.00 ± 1.30

Each shuffle value is the mean of 50 permutations within that training seed. The paired differences below are in percentage points.

ComparisonDifferences for seeds 42 / 43 / 44Mean ± seed SDApproximate paired 95% CI
VISTA − within-emotion shuffle0.70 / 1.10 / 0.600.80 ± 0.26[−0.40, 2.00]
VISTA − generic semantic bottleneck1.30 / 1.40 / 1.201.30 ± 0.10[−0.60, 3.20]
VISTA − Modality-Gate-SFT2.10 / 3.00 / 2.402.50 ± 0.46[0.55, 4.45]

Both mechanism gains are positive across all three seeds; their test-set intervals include zero. Across-seed SD measures training variability, while the paired intervals describe test-set uncertainty conditional on these checkpoints. We estimated the intervals using a normal approximation and a CA-MER source design-effect adjustment of 1.25.

For the accuracy intervals, we first average paired correctness differences within each original clip over the fixed seeds and relevant views. The 50 shuffle assignments are averaged within each clip and seed before this step. For a view-weighted mean, let xix_i be the clip-level difference and wiw_i its number of evaluated views. With W=∑iwiW=\sum_iw_i, we define d=∑iwixi/Wd=\sum_iw_ix_i/W, r=∑iwixi2/Wr=\sum_iw_ix_i^2/W, and Neff=W2/∑iwi2N_{\mathrm{eff}}=W^2/\sum_iw_i^2. The source adjustment uses the CR1 cluster variance

V_{\mathrm{cluster}}=\frac{G}{G-1}\frac{\sum_gU_g^2}{W^2},\qquad DE=\frac{V_{\mathrm{cluster}}}{(r-d^2)/N_{\mathrm{eff}}}.$$ Here $G$ is the number of mutually exclusive parent-source clusters, formed by merging shared source relations into connected components. The interval in percentage points is $100d\pm1.95996398454\times100\sqrt{V_{\mathrm{cluster}}}$. CA-MER conflict accuracy gives equal weight to the video-aligned and audio-aligned subsets. Their equal sizes in the table above make this the pooled conflict mean. **Q2. EmoMM source overlap.** Of the 800 original test clips, 202 overlap training clips and 62 overlap clips used for checkpoint selection, based on clip identity or temporal overlap. These sets are disjoint. Of the 160 parent-source clusters, 112 overlap training or checkpoint-selection sources, counting their intersection once. Excluding these parent sources and all derived views leaves 454 clips from 48 source clusters. All twelve clips with unresolved identities belong to the excluded parent sources. The conflict portion of this strictly source-independent set contains 266 clips from 34 source clusters and 624 corresponding conflict-plus-missing inputs. We evaluated the frozen original VISTA and Modality-Gate-SFT checkpoints on exactly the same retained inputs. | Strictly source-independent subset | VISTA (%) | Modality-Gate-SFT (%) | Difference (pp) | Approximate paired 95% CI (pp) | | --- | ---: | ---: | ---: | --- | | Conflict, 266 inputs | 49.50 | 47.49 | 2.01 | [−2.60, 6.61] | | Conflict plus missing, 624 inputs | 41.35 | 38.89 | 2.46 | [−1.56, 6.48] | VISTA retains higher point estimates in both source-independent conditions. The paired intervals account for dependence by aggregating missing-modality views within each clip and adjusting for parent sources.
AuthorsOpenReview ↗

Field encoding, arbitration targets, and CA-MER scoring (2/3)

Q3. Field encoding, arbitration targets, and CA-MER scoring. In the original implementation, the goal field is text, congruence/expectation/agency/expression regulation have 3/3/5/5 categories, control is a scalar, and norms have four binary indicators. We serialize the fields as compact JSON in the order g,c,e,a,k,n,r, with invalid top-level values written as null. Tokenizer offsets select non-padding sub-tokens whose character spans overlap a non-null value. We average their final-layer hidden states globally with equal token weights. The selected value spans include keys and punctuation inside the compound n value.

Let mtm_t denote the value-token mask and Ht∈R3584H_t\in\mathbb R^{3584} a hidden state. The pooled representation is hz=∑tmtHt/max⁡(1,∑tmt)h_z=\sum_tm_tH_t/\max(1,\sum_tm_t). A trainable linear layer maps hzh_z to z0∈R128z_0\in\mathbb R^{128}. A typed-field head predicts 21 logits from z0z_0, with slice widths 3/3/5/1/4/5 for c,e,a,k,n,r. We apply a separate softmax to each categorical field and sigmoid to control and the norm bits, then concatenate the probabilities into qfieldq^{\mathrm{field}}. The goal field enters through value-token pooling. Feedback is added through a fixed matrix:

F\in\mathbb R^{21\times128},\qquad z=(z_0+q^{\mathrm{field}}F)\,\mathbf 1[\text{at least one field is valid}].$$ The field head supplies predicted-probability feedback for all typed fields, while validity masks select the tokens used for pooling. The `no_r` ablation removes the regulation value tokens and sets its feedback slice to zero. When all fields are invalid, the appraisal state is zero. Confidence and evidence govern field acceptance, supervision, and auditing. Supervision is applied to valid targets, with separate masks for the norm bits. The arbitrator in Section 3.2 is a separate MLP. Each unimodal representation is projected to 128 dimensions. The three projections, six entropy/maximum-probability features, and the 128-dimensional appraisal form its 518-dimensional input. Linear(518,128), GELU, and Linear(128,3) jointly produce the modality scores. A masked softmax normalizes these scores over the available modalities. For CH-SIMS training examples with joint and unimodal labels, the arbitration target is $q_m\propto\exp(-|y_m-y|/0.25)$. For classification examples with at least one correct available frozen-teacher branch, target mass is shared equally among the correct branches. The arbitration loss is validity-masked cross-entropy against this distribution, with coefficient 0.5. All targets come from training labels. For CA-MER, we retain all 1,500 inputs and use the original multimodal decision label as the reference. Of these inputs, 1,418 map uniquely to the nine classes, 68 have multiple candidates, and 14 require disambiguation. The latter two groups were independently reviewed against the original multimodal reference and media, blind to model outputs. Generated predictions are parsed from a single JSON `final_label` using NFKC normalization and a fixed synonym map (Appendix E), such as `happiness` to `happy`. Unknown labels or multiple answers count as errors. Native classifiers return the canonical argmax class directly. Conflict accuracy gives equal weight to the 500 video-aligned and 500 audio-aligned inputs. We report single-label accuracy in the normalized interface; the benchmark's original metric uses open-vocabulary set matching.
AuthorsOpenReview ↗

Appraisal interventions at fixed evidence (3/3)

Q4. Fixed-evidence replacements and the theoretical interaction. In the original 150-pair input-edit test (Appendix H.1), we fixed the target emotion and expected direction before scoring either prediction. A pair succeeded if the target-category probability moved in that direction. Zero changes and invalid outputs counted as failures. The new experiments replace appraisal while holding evidence fixed.

We built a held-out set from unused MER2023 candidates, excluding parent sources shared with CA-MER or the training/development pools. The coherence review retained 256 of 300 proposed replacements. We replaced appraisal while fixing the checkpoint, evidence representations, modality availability, unimodal logits, and joint residual. Class-pair log-odds moved in the prespecified direction for 195/256 pairs (76.17%), and the label changed for 70/256 (27.34%).

For each scene, we also paired two cue conditions with two appraisal states. All four combinations had to pass the coherence review, leaving 120 complete quadruplets from 150 candidates. Within each cue condition, evidence stayed fixed as appraisal changed. Every mean below uses the same 120 quadruplets.

ConditionClass-pair log-odds LLαT\alpha_TαA\alpha_AαV\alpha_V
u0,z0u_0,z_0−0.2500.3100.4600.230
u1,z0u_1,z_00.1000.3000.4480.252
u0,z1u_0,z_1−0.0800.3150.4050.280
u1,z1u_1,z_10.5400.2850.3650.350

The interaction in Eq. 3 is therefore

I=(L11−L01)−(L10−L00)=0.270,\mathcal I=(L_{11}-L_{01})-(L_{10}-L_{00})=0.270,

with an approximate source-adjusted 95% CI of [0.132, 0.408]. The interaction follows the prespecified direction in 86/120 quadruplets (71.67%). It measures the interaction between cue and appraisal in output log-odds through modality reweighting. Unimodal logits stay fixed during appraisal replacement (Section 3.2 and Appendix D.4).

The original input-edit test in Appendix H.1 and the new interventions used the same annotation protocol. Two trained annotators independently judged coherence and direction, blind to model identity and predictions. A third adjudicator resolved disagreements. For the new experiments, annotators specified the intervention-supported emotion y1y_1, a fixed comparison emotion y0y_0, and the expected direction of L=log⁡(py1/py0)L=\log(p_{y_1}/p_{y_0}) before model evaluation. Before adjudication, direction agreement was 87.50% for accepted replacements and 86.67% for accepted quadruplets, with Cohen's κ\kappa of 0.750 and 0.733. We selected the seed 42 checkpoint in advance for these new interventions.

Thank you especially for your willingness to consider raising your score if we satisfactorily address your concerns. Please let us know if any explanation remains unclear or any question unanswered. We would be glad to answer follow-up questions and discuss whichever aspects of the work interest you. Your careful review has been a great encouragement to us, and we are deeply grateful for your time and attention.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Matched structured control (1/2)

Thank you for pointing out how supervision quality and donor matching can affect these comparisons. Both issues are easy to overlook, and your suggestions helped us improve the controls. We also appreciate your recognition of the motivation and the value of evaluating conflicting and consistent inputs separately.

Q1. Matched structured non-appraisal control. The new control records observable scene facts in seven slots: an event summary, sentence form, acoustic-energy trend, number of visible actors, proportion of audible speech, four cue-presence indicators, and mouth movement. These slots match the appraisal fields' text/categorical/scalar/binary types.

We trained the control with the original implementation. It uses the same 8,000 examples, seeds 42/43/44, and 375 training steps per seed as VISTA. Both use the same trainable architecture and parameter count, including a 128-dimensional intermediate representation and the same connection to the arbitrator. Both conditions use three label-blind samples from the same Qwen2.5-Omni-7B teacher and share the candidate pool and review budget. Their per-example validity masks are identical, including masks for individual norm/presence bits.

CheckVISTA appraisalStructured scene facts
Valid fields36,760/56,000 (65.64%)36,760/56,000 (65.64%)
Grounded fields in a blind audit of the same 400 examples1,574/1,838 (85.64%)1,588/1,838 (86.40%)
Mean supervised text tokens per valid example24.624.4
Total supervised text tokens398,520395,280

Both conditions cap the text target at 48 tokens, and their total supervised-token counts differ by 0.81%. These counts cover the tokens used in the language-model loss; the other fields use typed losses.

MethodConflict accuracy (%)Consistent accuracy (%)
VISTA64.5074.20
Matched structured non-appraisal control63.6074.13
Generic semantic bottleneck63.2074.07
Modality-Gate-SFT62.0074.00

VISTA's mean conflict accuracy is 0.90 percentage points higher than the structured control's, with an approximate paired 95% CI of [−0.72, 2.52]. This comparison matches structured supervision and field types while varying the intermediate content, extending the controls in Table 19.

AuthorsOpenReview ↗

Matched appraisal donors and evidence of interaction (2/2)

Q2. Matching donor validity and confidence. We compared four conditions on the same recipients while keeping input evidence and the original checkpoints fixed. Donor selection progressively matches emotion class, the exact validity-mask pattern, and confidence. Confidence distance is the mean absolute difference over commonly valid fields, with threshold 0.05. Each repetition pairs recipients one-to-one with eligible donors from other examples. Gold emotion defines the classes for this offline diagnostic.

The common set of recipients matchable across all three seeds contains 1,296/1,500 examples (86.40%): 420 video-aligned, 432 audio-aligned, and 444 consistent inputs. All four conditions use this set. Each replacement result averages 50 donor assignments per trained seed. Conflict accuracy gives equal weight to the video-aligned and audio-aligned rates.

Appraisal conditionConflict accuracy (%)Original − replacement (pp)Approximate paired 95% CI (pp)
Original65.17ReferenceReference
Same emotion64.350.82[−0.50, 2.14]
Same emotion and validity masks64.590.58[−0.58, 1.74]
Same emotion, masks, and confidence64.860.31[−0.64, 1.26]

In a blind review of 400 recipients, coherence increased from 66.75% with emotion-only donors to 81.75% with full matching. Mean confidence distance fell from 0.146 to 0.024. The smaller accuracy change under tighter matching supports your concern that atypical pairings contribute to the shuffle effect. We treat shuffling as a sensitivity test and examine the decision mechanism directly through coherent interventions with fixed evidence.

Direct evidence of the interaction. We evaluated 256 coherent appraisal replacements with evidence, unimodal logits, and the joint residual fixed. Class-pair log-odds moved in the prespecified direction in 76.17% of pairs. A separate set of 120 complete cue–appraisal quadruplets gave a mean interaction of 0.270, with an approximate source-adjusted 95% CI of [0.132, 0.408]. We fixed the class pairs, directions, and seed-42 checkpoint before evaluation. These tests examine how appraisal changes the contribution of observed evidence through modality weights.

The accuracy intervals use normal approximations to paired differences, adjusted for source dependence and conditional on the fixed training seeds. The source design effect is the ratio of the CR1 parent-source cluster variance to the independent-sample approximation. For shuffling, repeated assignments are averaged within each recipient and training seed before aggregation by source. CA-MER conflict comparisons give equal weight to the video-aligned and audio-aligned subsets. We would be happy to discuss either comparison further.

If you find these explanations and results convincing, we would be grateful if you could reconsider your assessment of the paper. Such encouragement would mean a great deal to us as we pursue careful research at a time when AI is reshaping the field. Your evaluation and feedback help the community assess and build on the work, and we would be honored to see it make a useful contribution. Thank you for the time and care you have devoted to our paper.

AuthorsOpenReview ↗

Revised manuscript and responses to your questions

We understand that reviewing is voluntary work for the community, and we deeply respect the generosity and character behind that commitment. It also adds to the demands of your own work. Because long replies can be difficult to follow in a comment thread, we do not want to add to your reading burden by giving every detail the same space and emphasis. We are confident that we have done our best in these comments to clarify the points you raised and answer your questions concisely and accurately.

You may also want a more detailed account of the work to support your judgment or reconsider your assessment. We recognize the value of your questions and how much they can help us improve the paper. We have updated the PDF of the main text and appendix, with fuller explanations that we hope will address your concerns more concretely.

We have also corrected several typos and formatting issues. We are proud to present this revised manuscript to you, and we do so with deep respect. Changes are marked in red.

You can find the revisions related to your questions here:

  • Q1. Matched structured non-appraisal control. Main text: Sec. 4.3, p. 7, lines 374–377, and p. 8, lines 388–390. Appendix O.3, pp. 52–53, lines 2802–2852 (Tables 43–44).

  • Q2. Donors matched on emotion, field validity, and confidence. Main text: Sec. 4.3, p. 8, lines 394–400. Appendix O.4, pp. 53–54, lines 2854–2891 (Tables 45–46).

We look forward to hearing your thoughts on the revised paper and would be happy to discuss any remaining questions. If the new experiments and clarifications have addressed your concerns, we would be very grateful if you would consider raising your score.

Once again, our sincere thanks.

Thanks for the authors' response. I will maintain my score as weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.