acceptodds
Under review as a conference paper at ICLR 2027

Disagreement Is Not Degradation: Evaluating Reliability in Multimodal Sentiment Analysis

Abstract

Multimodal reliability evaluations can conflate annotation disagreement, input degradation, task utility, and output-format compliance. We audit these distinctions with three omni-modal models: a 21,714-request, seven-view evaluation on 1,034 CH-SIMS v2 clips, followed by paired raw-media transformations and fixed MELD and CMU-MOSEI cohorts. Adding a transcript to audio-visual input increases Qwen2.5-Omni's error on the annotation-disagreeing CH-SIMS subset, while the corresponding MiniCPM-o estimate remains uncertain. A training-median control reveals different target distributions: none of the three model-minus-prior intervals on the disagreeing subset excludes zero. Qwen3-Omni's format failures further separate schema compliance from task performance. Narrow syntax rules developed post hoc on the original run recover its malformed numerical responses; fixed before the follow-up, complete-fence normalization also restores high coverage on the external tasks. Additional modalities do not yield uniform gains: on the locked MELD cohort, MiniCPM-o's text accuracy exceeds its text-audio-visual accuracy by 9.96 percentage points despite complete strict coverage. Paired physical transformations have nonuniform predictive effects, and an earlier matched-supervision feature study does not support a factorized reliability gate. The evidence supports joint reporting of absolute error, paired input effects, train-only prior controls, output coverage, and distinct failure causes. It does not establish a human-validated perceptual-quality effect or a superior fusion architecture.

Primary area:
datasets and benchmarks
Keywords:
multimodal sentiment analysis, omni-modal evaluation, modality disagreement, output-format compliance, robustness evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1JKpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits reliability evaluation in multimodal sentiment analysis, distinguishing annotation disagreement, input degradation, predictive utility, and output-format compliance. The authors evaluate three omni-modal models across seven input combinations on CH-SIMS v2, supplemented by paired noise and blur experiments and experiments on fixed MELD/MOSEI evaluation subsets. The results show that adding modalities does not necessarily improve prediction, format failures can distort reliability assessments, target-distribution differences affect the interpretation of groupwise errors, and the tested factorized gate shows no clear advantage.

Critical strengths
  1. Under full three-modality input, Qwen3-Omni’s strict-format scoring risk is 0.8132, whereas its mean absolute error after format normalization is 0.3539; the apparent improvement under stronger blur largely disappears after normalization. This demonstrates that format compliance can substantially affect assessments of model reliability.
  2. The study uses paired comparisons on the same samples, confidence intervals clustered by source, and a prior control fitted using training labels only, while explicitly retaining negative results and exploratory analyses. These designs strengthen the credibility of the paper’s conclusions within its limited scope.
Critical weaknesses
  1. Robust-MSA [1] and EmoMM [2] have already studied modality conflict, missingness, and noise. The main addition of this paper is jointly reporting different evaluation metrics, but it lacks a demonstration of how this practice changes substantive conclusions in existing research.
  2. Each task uses only one prompt and a fixed media input budget. Is a corresponding sensitivity analysis missing? Can the resulting format failures and performance degradation caused by adding modalities persist consistently?
  3. In Table 33, the AUROC of the text and audio Gaussian-noise detectors is approximately 0.49–0.50, close to chance. Therefore, the overall absence of improvement does not distinguish insufficient detector quality from a lack of utility in the factorized gate itself.

[1] Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis, AAAI 2023. [2] EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness, ACL 2026.

Questions for authors
  1. After jointly reporting absolute error, prior controls, and output coverage, which substantive conclusion in existing conflict or noise evaluations would change? Can you provide corresponding quantitative evidence?
  2. Do the main modality effects and format failures persist under semantically equivalent prompts or a reasonable alternative media input budget?
  3. For visual perturbations with relatively accurate detection in Table 33, does the factorized gate still offer no improvement over the control using only utility gating? Please provide paired comparisons broken down by perturbation condition.
3 comments

Thank you for highlighting prior-work overlap, configuration sensitivity, and the interpretation of our gate result.We have kept the revised paper's contribution to an evaluation audit, not a new conflict benchmark or fusion architecture.

Prior work and substantive interpretation changes. Joint reporting changes two interpretations within our study: a lower strict operational risk under blur does not establish better numerical sentiment prediction, and degradation relative to clean input does not determine whether deleting the degraded modality helps. Robust-MSA studies simulated modality noise and defenses; EmoMM provides conflict/missingness evaluations and a steering method. We have not re-run either study or claimed to change its published conclusions. Our contribution is the bounded joint audit of annotation-protocol disagreement, physical degradation, predictive input utility and output-format compliance (revised Sections 1 and 5, Table 1).

In the original Qwen3-Omni physical campaign, strong blur minus clean appears beneficial under strict all-intended risk: −0.1040, source-cluster 95% CI [−0.1638,−0.0450], with strict-valid counts of 750/818 out of 1,034 per arm. Under the fixed narrow syntax normalization, the all-intended difference is +0.0034 [−0.0092,+0.0152]; recovered valid-pair MAE differs by +0.0018 [−0.0099,+0.0123] on 1,031 pairs. The four strict-format cells are 654/96/164/120 (both valid/clean only/blur only/neither), a descriptive decomposition rather than a causal mechanism (revised Sections 1 and 6; Appendix G, Table 30). In a separate post-review same-campaign comparison, MiniCPM-o's stronger blur worsens MAE versus clean TAV by +0.02456 [+.01231,+.03715] (marginal interval), yet retaining blurred video beats removing it by −0.03511 [−.06251,−.00771] (exploratory approximate 12-comparison simultaneous interval). This is a contrast of different reference inputs, not a claim that blur improves prediction (revised Section 7, Table 6; Appendix O, Table 48).

Prompt and media-budget sensitivity. We ran separate post-review fixed-cohort checks, each with a same-campaign P0/B16 reference. P1 paraphrases the task description while preserving the output contract; Qwen3-only P2 paraphrases the format instruction; Bhalf changes the native media budget. On the original CH-SIMS v2 D group (149 clips), Qwen2.5-Omni's paired AV−TAV MAE is −.06644 [−.10932,−.02945] at P0/B16, −.08792 [−.13740,−.04362] at P1/B16, and −.07181 [−.11889,−.03182] at P0/Bhalf. On 512 fixed MELD utterances, MiniCPM-o's paired T−TAV accuracy advantage is +9.96 [6.56,13.69], +8.40 [4.56,12.53], and +9.18 [5.42,13.04] percentage points, respectively. Both arms are strict-valid for every one of the 149 or 512 paired items in these checks. The directions persist across these tested settings; all four configuration-interaction intervals cross zero, which does not prove equal effect sizes. Qwen3's strict-valid clean/strong-blur counts change from 750/819 at new-campaign P0/B16 to 594/741 at P1/B16 and 554/744 at Qwen3-only P2/B16; Bhalf gives 680/723, each out of 1,034 per arm. Fixed syntax normalization makes all produced outputs scorable in these new checks, and every corresponding numerical blur-minus-clean interval spans zero. The original campaign's 750/818 counts are distinct from the new P0 campaign's 750/819. We did not isolate a change to the media input template; Bhalf also changes native frame-rate metadata and MiniCPM-o audio chunk allocation. Revised Section 7, Table 5, and Appendix O give the settings, intervals, coverage and anonymous verbatim-prompt configuration archive.

Detector quality versus gate utility. The matched-supervision study does not show a Factorized advantage over Utility. In visual temporal dropout at severity .3, the Factorized detector's three AUROCs are .9534/.9527/.9527, yet Factorized−Utility absolute MAE is +.01007 [+.00486,+.01514] and its paired error-increment difference is +.00844 [+.00496,+.01241]. In audio temporal dropout .3 the increment difference instead favors Factorized (−.00126 [−.00212,−.00038]) while its absolute-MAE difference is uncertain (+.00038 [−.00367,+.00450]). These 12 condition-level comparisons are exploratory, source-cluster intervals without multiplicity adjustment; revised Appendix N, Tables 43–44, provides every F/U absolute MAE, increment, difference, interval and corresponding detector AUROC. High visual detection discriminability rules out a detection-is-always-near-chance explanation for that one visual condition, but does not separate detector calibration, intervention task relevance or the gate-combination rule. The joint variant changes two gradient boundaries together. To address readability, revised Table 1 defines the four objects, the introduction leads with the two interpretation changes, and the gate result is presented as supporting negative evidence (revised Sections 1, 4.4 and 8).

Thank you to the authors for supplementing the experiment; the issues of cue word sensitivity and factorization gating have been addressed. I maintain the original score because the two cases illustrate the value of their respective controls, but do not yet fully demonstrate what additional insights joint auditing can provide.

I suggest demonstrating item by item how each control alters the assessment conclusions on the same batch of samples, and examining whether labeling discrepancies are related to modal utility in samples with comparable sentiment polarity and intensity. The latter could more directly support the motivation to include labeling discrepancies in joint auditing.

Thank you for clarifying the remaining issue. We completed a CPU-only reanalysis of the same-campaign Task 4 predictions, with rules fixed before calculation and no new outputs or labels. These analyses are reported here, not in the uploaded PDF.

1. Following the controls on the same batch. All three models have 1,034 clip intentions per arm. We retain (a) all-intended operational risk and format coverage and (b) a fixed seven-arm numerical intersection of 1,032 clips/141 sources per model, excluding the same two clips restricted in the 10-dB arm. Every numerical control below uses that fixed intersection; it is not compared to a different complete-case population.

For MiniCPM-o under strong blur, the itemwise controls give this assessment (MAE):

Control/referenceResultPermitted assessment
Clean TAV / blurred TAV / deleted-video TA.26773 / .29234 / .32752Compare absolute task error
Training-only median prior.50543All three arm means are below this constant reference
Blur minus clean+.02461 [.00543,.04380]Blur increases error
Blur minus deleted-video TA-.03517 [-.06559,-.00476]Keeping blurred video still reduces error versus deletion

Both intervals are exploratory approximate simultaneous 95% source-cluster intervals over all 24 degradation/retention contrasts (three models, four transformations, two references), using 20,000 shared draws. Thus the same clips can show degradation and retained input utility together. The decomposition is an accounting identity, not a new mechanism; subtracting the same per-clip prior loss cancels in the utility contrast.

Format coverage is a separate control on the same intended cohort. In Task 4, Qwen3's clean/strong-blur strict-valid counts are 750/819 of 1,034. Strict operational risks are .81315/.70745, whereas fixed-normalized numerical errors are .35392/.35580. On the fixed 1,032-clip intersection, normalized blur-minus-clean is +.00189 [-.01553,.01931] under the 24-comparison interval. Lower strict operational risk therefore does not establish improved numerical prediction. These are new same-campaign calculations, not the original physical campaign's 750/818 counts.

2. Is disagreement associated with utility at comparable polarity/intensity? We defined video utility as MAE(TAV)-MAE(TA), and audio utility as MAE(TAV)-MAE(TV), so negative values favor adding the modality. We prespecified exact-zero separation, positive/negative polarity, and absolute-target bins (0,.2), [.2,.6), [.6,1]. Requiring at least ten clips and ten sources in every A/D/N group retained four signed moderate/strong strata: A=458, D=133, N=349 (940 clips). Identical weights from the intended cohort were applied to every group/model. Effective sample sizes are A=340.0, D=112.5, N=285.6. We report full-cohort raw, common-support raw, and standardized contrasts separately: Qwen2.5 audio D-A is +.00484, +.00194, and -.01928, respectively; its simultaneous interval still includes zero.

The standardized D-minus-A utility differences are:

ModelVideo, estimate [simultaneous 95% CI]Audio, estimate [simultaneous 95% CI]
Qwen2.5-.05370 [-.11891,.01152]-.01928 [-.06119,.02264]
MiniCPM-o-.05807 [-.17237,.05623]-.04264 [-.10645,.02117]
Qwen3, fixed normalization+.00718 [-.10516,.11953]-.00139 [-.06412,.06134]

These six comparisons form a separate exploratory simultaneous-interval family, using 20,000 joint source-cluster draws; all draws were valid. All six intervals include zero. We therefore have not established a stable disagreement-utility association; this also does not establish its absence. Coarse-stratum standardization leaves residual intensity imbalance: weighted mean absolute targets are .5903/.5128/.5141 for A/D/N, despite identical polarity shares. This is not exact matching or independent confirmation, and the intervals condition on fixed support/weights. Task 4 has no AV arm, so this does not reassess the original transcript-addition effect.

We retain the bounded contribution: the audit makes different reference inputs, target composition, and format outcomes explicit when interpreting reliability. The evidence supports those interpretive safeguards, but does not validate annotation disagreement as a predictor of modality utility or establish a causal conflict effect under full model inputs.

Reviewer_o286OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reliability evaluation in multimodal sentiment analysis, distinguishing human unimodal annotation disagreement, physical input degradation, task utility, and output-format compliance. The authors evaluate three omni-modal models across seven input views on CH-SIMS v2. They also conduct paired raw-media transformations, a training-median control analysis, and external-task experiments on MELD and CMU-MOSEI. The paper shows that additional modalities do not necessarily improve prediction performance, that target-distribution differences across annotation groups affect the interpretation of model errors, and that output-format noncompliance does not necessarily imply an incorrect numerical prediction. The authors consequently recommend jointly reporting absolute risk, paired input effects, train-only prior controls, and output coverage.

Critical strengths

S1. The paper clearly distinguishes its evaluation targets and interpretive limits. The paper does not equate annotation disagreement with input degradation or directly attribute performance changes across input combinations to a model’s internal fusion mechanism. These distinctions help prevent overinterpretation of model reliability. S2. The experimental records are complete, and the paper provides concrete paired statistical evidence. The authors retain records of all 21,714 requests in the original seven-view CH-SIMS v2 evaluation, including invalid outputs, and perform paired comparisons on the same clips. On the annotation-disagreeing subset, Qwen2.5-Omni’s AV−TAV error contrast is −0.0664, with a conditional 95% interval of [−0.1090, −0.0317]. This supports the conclusion that adding a transcript increases error under the fixed protocol. MiniCPM-o’s corresponding contrast is +0.0215, with an interval of [−0.0133, 0.0609]. Its direction remains uncertain, so these results do not establish opposite-direction effects for the two models. On the fixed 512-clip MELD evaluation cohort, MiniCPM-o’s text-only accuracy exceeds its text–audio–visual accuracy by 9.96 percentage points, further demonstrating that additional inputs do not necessarily improve prediction. S3. The prior-control and syntax-recovery analyses help prevent evaluation misinterpretations. The exploratory training-median control analysis reveals target-distribution differences across annotation groups. It shows why higher raw error on the annotation-disagreeing group alone is insufficient to conclude that disagreement itself makes prediction more difficult. This analysis neither removes all confounding nor identifies an independent disagreement effect. The syntax-recovery analysis shows that responses that are invalid under strict parsing may still contain recoverable numerical predictions. The authors explicitly distinguish evaluation of recovered numerical predictions from original output-format compliance, avoiding the conflation of parsing failure with prediction error.

Critical weaknesses

W1. The joint analysis of physical transformations and task utility could be developed further. The paper compares prediction performance across seven input views and examines the effects of specified physical transformations through audio noise and frame blur. However, these results are primarily presented separately. The paper does not directly report paired omni-modal model performance for retaining a transformed modality versus removing that modality. For example, an increase in overall error after adding audio noise does not imply that removing audio would produce better results. A paired comparison would more directly establish whether retaining the modality remains beneficial under the specified transformation. This does not invalidate the existing results, but represents an opportunity to deepen the analysis. W2. Further interpretation of the prior-control analysis is limited by its exploratory design and sparse strata. The training-median control usefully shows that raw error alone cannot establish that the annotation-disagreeing group is more difficult. However, this analysis was introduced after observing the test results, and some target strata contain very few samples. The existing results demonstrate the relevance of target distributions to error interpretation, but do not support a stable separation of the respective effects of disagreement and target distribution. The authors explicitly disclose these limitations and make no causal claim. This concern therefore does not invalidate the descriptive results; it limits further interpretation of the sources of error in the annotation-disagreeing group. The external tasks lack corresponding unimodal annotation-disagreement labels and therefore do not directly confirm this analysis.

Questions for authors

Q1. Could the authors compare average prediction performance when retaining noise-corrupted audio versus removing audio entirely, using prespecified samples or annotation groups, consistent scoring rules, and comparable execution conditions? An analogous comparison for blurred visual input would also be useful. Together with the corresponding clean-input results, these comparisons could establish whether retaining the modality remains beneficial under the specified transformation. Q2. If independent samples with corresponding unimodal annotations are already available, could the authors use fixed analysis rules to reassess target distributions across annotation groups and model performance relative to the training-median control?

2 comments

Thank you for the direct retain-versus-remove proposal and for recognizing the exploratory limits of the train-median control. We completed a new same-campaign comparison for the first question and have narrowed, rather than overstated, the answer to the second.

Q1: Is it better to retain a physically transformed modality than remove it? On the same 1,034 intended CH-SIMS v2 test clips per arm, we evaluated three models with clean TAV, noisy-audio TAV at 20 and 10 dB, blurred-video TAV at σ=1 and 2, clean TV (audio removed), and clean TA (video removed). The paired estimand is MAE(degraded TAV)−MAE(modality removed) on the valid intersection; negative favors retention. Clean TAV was rerun in the same campaign. Of 21,714 intended requests, 21,708 have responses and six are prespecified 10-dB media restrictions (two per model), with no unattempted records. Those restrictions stay in all-intended operational accounting, while the 10-dB paired-MAE intersection is n=1,032 rather than 1,034. Qwen3 comparisons below use the fixed narrow syntax normalization; native strict coverage is separately reported in revised Appendix O, Table 46.

ModelDegraded TAV minus deletionPaired MAE differenceApproximate 12-comparison simultaneous 95% CI
Qwen2.5Audio 20 dB − TV+.01035[−.00222,+.02291]
Qwen2.5Audio 10 dB − TV+.00950[−.00284,+.02183]
Qwen2.5Blur σ1 − TA−.01103[−.02589,+.00384]
Qwen2.5Blur σ2 − TA−.00387[−.01765,+.00991]
MiniCPM-oAudio 20 dB − TV−.00542[−.02435,+.01352]
MiniCPM-oAudio 10 dB − TV−.00484[−.02299,+.01330]
MiniCPM-oBlur σ1 − TA−.05493[−.08353,−.02633]
MiniCPM-oBlur σ2 − TA−.03511[−.06251,−.00771]
Qwen3, normalizedAudio 20 dB − TV+.00614[−.01195,+.02424]
Qwen3, normalizedAudio 10 dB − TV+.00673[−.01028,+.02375]
Qwen3, normalizedBlur σ1 − TA+.00150[−.02613,+.02913]
Qwen3, normalizedBlur σ2 − TA+.00464[−.02275,+.03204]

Revised Section 7, Table 6 gives every original marginal 95% source-cluster interval; Appendix O, Table 48 gives all marginal and simultaneous intervals, per-row n, the common-source 20,000-draw max-|Z| procedure, and its approximate scope. The simultaneous family covers the 12 retain-minus-delete contrasts only, not blur-minus-clean references. Only MiniCPM-o's two blurred-video retention intervals exclude zero simultaneously. Importantly, stronger blur makes MiniCPM-o worse than its same-campaign clean TAV by +.02456 [+.01231,+.03715] (marginal, unadjusted 95% CI), while retaining the blurred video beats clean TA by −.03511 [−.06251,−.00771] (exploratory approximate 12-comparison simultaneous 95% CI; marginal CI [−.05469,−.01630]). Its same-campaign clean TAV and clean TA absolute MAEs are .26741 and .32708, respectively (revised Appendix O, Table 47); these are arm means, not a new paired interval. Thus clean-to-degraded harm does not by itself establish that deleting video helps, and the retention contrast does not isolate a blur-resistance mechanism. The clean-arm absolute MAEs and native Qwen3 coverages are in Appendix O, Tables 46–47. This is conditional on the specified transformation, input routes, model revisions and cohort; it establishes neither a universal policy to retain video nor an internal fusion mechanism. Strict all-intended penalties and normalized paired MAE describe different outcomes, especially for Qwen3.

Q2: Can the target-distribution/prior result be independently checked? The constant prior was fitted using training labels only; in the original test cohort it shows that raw groupwise MAE and target distribution should be read together. It does not remove all confounding or identify a separate effect of D. The additional read-only, post-review five-bin check in revised Appendix O is descriptive. It uses target intervals [−1,−.6), [−.6,−.2), [−.2,.2), [.2,.6), [.6,1], different from Table 36's strong/weak/exact-zero partition. Requiring each A/D/N group to have at least ten clips and ten distinct sources in a bin excludes only the first interval (D: six clips/six sources), discarding 144/459 A clips; Appendix O reports every group/bin count. That changed target population and has no confirmatory interval. We have no untouched, protocol-compatible cohort with the same human unimodal labels: the evaluated MELD and CMU-MOSEI subsets lack these labels, and the available CH-SIMS validation material informed earlier checkpoint selection. We therefore cannot present an independent confirmation of a disagreement-specific or prior-adjusted effect. Revised Sections 2 and 8 retain the original prior analysis as exploratory and explicitly state this limit.

I had read the response, and appreciate the authors' efforts in addressing the comments. Nevertheless, my evaluation of the paper remains unchanged

Reviewer_oiJpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the distinctions among annotation disagreement, input degradation, task utility, and output-format compliance in the evaluation of multimodal sentiment analysis. It conducts a seven-view evaluation of Qwen2.5-Omni, MiniCPM-o-4.5, and Qwen3-Omni on 1,034 CH-SIMS v2 test samples, and analyzes predictive performance and output coverage using a train-median prior, restricted format recovery, paired raw-media transformations, and fixed MELD and CMU-MOSEI cohorts. A separate matched-supervision feature study examines a factorized reliability gate. The results show that the effects of adding modalities depend on the model and task, and that target distributions and format failures can change the interpretation of results. In the feature study's overall seen-intervention MAE and paired error increment comparisons, Factorized does not demonstrate an advantage over Utility. The paper consequently recommends jointly reporting absolute error, paired input effects, train-only prior controls, output coverage, and distinct failure causes.

Critical strengths

1.By distinguishing annotation disagreement, input degradation, task utility, and output-format compliance, the paper provides a clear evaluation perspective for understanding reliability in multimodal sentiment analysis.

2.The experiments use multiple input combinations, physical transformations, and external datasets to examine the relationship between input information and model performance from different perspectives.

3.The paper treats format failures, statistical uncertainty, and negative results carefully, and clearly states the scope of its conclusions.

Critical weaknesses

1.The annotation strata have limited explanatory scope for cross-modal disagreement under the models' actual input conditions. Section 3 explicitly states that acoustic labels were obtained with words obscured, and visual labels without audio or subtitles, whereas the actual model inputs retain spoken content and may include subtitles. Thus, D denotes annotation disagreement under the original annotation conditions. The reported group-level results leave unclear the extent to which these samples retain cross-modal disagreement under the full model input conditions. This limits the strata's ability to explain model behavior across input conditions.

2.The stability of the key findings across evaluation configurations remains unclear. The current Omni evaluations use one prompt and one media budget per task, and most of Qwen3-Omni's strict-invalid outputs in the original evaluation were complete Markdown-fenced objects. The reported results leave unclear whether the frequency of format failures, Qwen2.5-Omni's AV/TAV MAE difference on the CH-SIMS D subset, and MiniCPM-o's T/TAV accuracy difference on MELD would remain consistent after changes to the output-format instructions, model input templates, or frame budget. The paper frames these as frozen pipeline comparisons; stability under closely related configurations still affects the usefulness of these findings.

3.The negative results for Factorized relative to Utility still lack a clear explanation. In Table 29, Factorized has a higher overall seen-intervention MAE than Utility, and Section 4.4 also reports a larger paired error increment. In Table 33, detector AUROC is close to 0.5 for text/audio Gaussian interventions, while some visual interventions are more detectable. The connection between the existing detector diagnostics and the condition-specific prediction-error differences between Factorized and Utility remains unclear. It is therefore difficult to distinguish the respective roles of detector capability, the interventions' effects on the task, and the gate combination scheme. The joint variant in Appendix I changes two gradient boundaries simultaneously, which also limits the assessment of their individual effects. This limits the guidance that this supporting study provides for reliability-aware fusion design.

Questions for authors
  1. Could you select a subset of D using a predefined sampling rule, repeat modality-specific annotation under conditions matching the models' actual inputs, and report its consistency with the original disagreement grouping? This would help assess how spoken content and subtitles affect the interpretation of the groups.

  2. When the prompt or frame budget is varied separately across a small set of predefined settings, do Qwen2.5-Omni's AV/TAV MAE difference on the CH-SIMS D subset and MiniCPM-o's T/TAV accuracy difference on MELD retain their respective directions? Could you also report the paired intervals and strict coverage for each setting?

  3. Drawing on Table 33, could you explain how the differences in absolute MAE and paired error increment between Factorized and Utility vary across interventions? How do these differences relate to detector AUROC under the corresponding conditions?

1 comment

Thank you for separating the meaning of CH-SIMS v2's unimodal annotations from what the omni-modal models actually see. We have made the annotation-condition boundary explicit throughout the revised manuscript.

Q1: Does the original D group persist under model-visible inputs? CH-SIMS v2 already supplies independently elicited T/A/V human sentiment labels. Acoustic annotators heard speech with words obscured, and visual annotators saw silent material without subtitles or audio. Our model input instead retains intelligible speech, and sampled frames can contain embedded subtitles. D therefore denotes disagreement under the dataset's original restricted annotation conditions, not a verified conflict under every model-visible T/A/V combination (revised Sections 1–3, especially Section 2). We did not complete independent reannotation under the models' actual presentation conditions; no D-persistence percentage can be estimated from the existing labels. The original Qwen2.5-Omni AV−TAV result on 149 D clips from 70 source clusters remains a descriptive contrast under that fixed group definition. It does not identify internal fusion or prove that each of those clips is still cross-modally conflicting under full input. This is an unresolved construct-validity limit, now stated in revised Section 8.

Q2: Do the main input effects and format behavior survive nearby configurations? We varied a task-description paraphrase (P1) at fixed input budget and, separately, an input-media budget (Bhalf) at the reference instruction; Qwen3 alone also received a format-instruction paraphrase (P2). Each comparison has a same-campaign baseline. The paired results are:

SettingQwen2.5 CH-SIMS D, AV−TAV MAE (149 pairs)MiniCPM-o MELD, T−TAV accuracy (512 pairs)
P0/B16−.06644 [−.10932,−.02945]+9.96 [6.56,13.69] pp
P1/B16−.08792 [−.13740,−.04362]+8.40 [4.56,12.53] pp
P0/Bhalf−.07181 [−.11889,−.03182]+9.18 [5.42,13.04] pp

Both arms have strict coverage of 149/149 or 512/512 in every listed setting. The directions remain in these finite checks. All four configuration-interaction intervals (two models × prompt/budget changes) include zero; this does not establish effect-size equivalence or arbitrary-template robustness (revised Section 7, Table 5; Appendix O). Qwen3-Omni shows the opposite kind of sensitivity: native strict-valid clean/strong-blur counts are 750/819 at new-campaign P0/B16, 594/741 at P1/B16 and 554/744 at Qwen3-only P2/B16, each out of 1,034 per arm. At P0/Bhalf they are 680/723. Strict all-intended blur-minus-clean risk differences [95% source-cluster CI] are −.10571 [−.16622,−.04164], −.22611 [−.29451,−.15418], −.29971 [−.36754,−.22841] and −.06083 [−.12554,+.00393], respectively. Under the fixed narrow fence/numeric-quote normalization, the corresponding numerical differences are +.00189 [−.00994,+.01236], +.00982 [−.00348,+.02220], +.00280 [−.00898,+.01431] and +.00319 [−.01168,+.01557], all with intervals crossing zero. The original physical follow-up has strict-valid 750/818, not the new campaign's 750/819. Hence strict format failure is configuration-sensitive, while no tested normalized contrast establishes improved numerical prediction from blur (revised Section 7; Appendix G, Table 30; Appendix O, Table 45). The paper and anonymous configuration archive now specify the verbatim prompts and Bhalf sampling rule; the media input template was not varied alone, and Bhalf changes native frame-rate metadata and MiniCPM-o audio chunk allocation.

Q3: What do the Factorized–Utility differences show across interventions? We can localize the negative result by intervention, but cannot identify its mechanism from these comparisons. Revised Appendix N, Tables 43–44, lists all 12 seen interventions, with each method's absolute MAE and paired error increment, their F−U differences and source-cluster intervals, plus detector AUROC. For visual temporal dropout .3, AUROC is .9534/.9527/.9527 across replicas, yet F−U absolute MAE is +.01007 [+.00486,+.01514] and paired-increment difference is +.00844 [+.00496,+.01241]. High intervention discrimination is therefore insufficient for a gate gain in this condition. The pattern is not uniform: audio temporal dropout .3 has an F−U increment of −.00126 [−.00212,−.00038], while the absolute-MAE difference is +.00038 [−.00367,+.00450]. These post-review intervals are unadjusted exploratory comparisons conditional on the fitted models. Text/audio Gaussian AUROCs are near .5, while some visual conditions have much higher AUROC; these diagnostics do not isolate detector calibration, intervention task effects or the gate-combination rule. The joint variant changes two gradient boundaries simultaneously, so its result cannot be assigned to either boundary (revised Sections 4.4 and 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.