- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper audits reliability evaluation in multimodal sentiment analysis, distinguishing annotation disagreement, input degradation, predictive utility, and output-format compliance. The authors evaluate three omni-modal models across seven input combinations on CH-SIMS v2, supplemented by paired noise and blur experiments and experiments on fixed MELD/MOSEI evaluation subsets. The results show that adding modalities does not necessarily improve prediction, format failures can distort reliability assessments, target-distribution differences affect the interpretation of groupwise errors, and the tested factorized gate shows no clear advantage.
Critical strengths
- Under full three-modality input, Qwen3-Omni’s strict-format scoring risk is 0.8132, whereas its mean absolute error after format normalization is 0.3539; the apparent improvement under stronger blur largely disappears after normalization. This demonstrates that format compliance can substantially affect assessments of model reliability.
- The study uses paired comparisons on the same samples, confidence intervals clustered by source, and a prior control fitted using training labels only, while explicitly retaining negative results and exploratory analyses. These designs strengthen the credibility of the paper’s conclusions within its limited scope.
Critical weaknesses
- Robust-MSA [1] and EmoMM [2] have already studied modality conflict, missingness, and noise. The main addition of this paper is jointly reporting different evaluation metrics, but it lacks a demonstration of how this practice changes substantive conclusions in existing research.
- Each task uses only one prompt and a fixed media input budget. Is a corresponding sensitivity analysis missing? Can the resulting format failures and performance degradation caused by adding modalities persist consistently?
- In Table 33, the AUROC of the text and audio Gaussian-noise detectors is approximately 0.49–0.50, close to chance. Therefore, the overall absence of improvement does not distinguish insufficient detector quality from a lack of utility in the factorized gate itself.
[1] Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis, AAAI 2023. [2] EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness, ACL 2026.
Questions for authors
- After jointly reporting absolute error, prior controls, and output coverage, which substantive conclusion in existing conflict or noise evaluations would change? Can you provide corresponding quantitative evidence?
- Do the main modality effects and format failures persist under semantically equivalent prompts or a reasonable alternative media input budget?
- For visual perturbations with relatively accurate detection in Table 33, does the factorized gate still offer no improvement over the control using only utility gating? Please provide paired comparisons broken down by perturbation condition.
3 comments
Thank you to the authors for supplementing the experiment; the issues of cue word sensitivity and factorization gating have been addressed. I maintain the original score because the two cases illustrate the value of their respective controls, but do not yet fully demonstrate what additional insights joint auditing can provide.
I suggest demonstrating item by item how each control alters the assessment conclusions on the same batch of samples, and examining whether labeling discrepancies are related to modal utility in samples with comparable sentiment polarity and intensity. The latter could more directly support the motivation to include labeling discrepancies in joint auditing.
Thank you for clarifying the remaining issue. We completed a CPU-only reanalysis of the same-campaign Task 4 predictions, with rules fixed before calculation and no new outputs or labels. These analyses are reported here, not in the uploaded PDF.
1. Following the controls on the same batch. All three models have 1,034 clip intentions per arm. We retain (a) all-intended operational risk and format coverage and (b) a fixed seven-arm numerical intersection of 1,032 clips/141 sources per model, excluding the same two clips restricted in the 10-dB arm. Every numerical control below uses that fixed intersection; it is not compared to a different complete-case population.
For MiniCPM-o under strong blur, the itemwise controls give this assessment (MAE):
| Control/reference | Result | Permitted assessment |
|---|---|---|
| Clean TAV / blurred TAV / deleted-video TA | .26773 / .29234 / .32752 | Compare absolute task error |
| Training-only median prior | .50543 | All three arm means are below this constant reference |
| Blur minus clean | +.02461 [.00543,.04380] | Blur increases error |
| Blur minus deleted-video TA | -.03517 [-.06559,-.00476] | Keeping blurred video still reduces error versus deletion |
Both intervals are exploratory approximate simultaneous 95% source-cluster intervals over all 24 degradation/retention contrasts (three models, four transformations, two references), using 20,000 shared draws. Thus the same clips can show degradation and retained input utility together. The decomposition is an accounting identity, not a new mechanism; subtracting the same per-clip prior loss cancels in the utility contrast.
Format coverage is a separate control on the same intended cohort. In Task 4, Qwen3's clean/strong-blur strict-valid counts are 750/819 of 1,034. Strict operational risks are .81315/.70745, whereas fixed-normalized numerical errors are .35392/.35580. On the fixed 1,032-clip intersection, normalized blur-minus-clean is +.00189 [-.01553,.01931] under the 24-comparison interval. Lower strict operational risk therefore does not establish improved numerical prediction. These are new same-campaign calculations, not the original physical campaign's 750/818 counts.
2. Is disagreement associated with utility at comparable polarity/intensity? We defined video utility as MAE(TAV)-MAE(TA), and audio utility as MAE(TAV)-MAE(TV), so negative values favor adding the modality. We prespecified exact-zero separation, positive/negative polarity, and absolute-target bins (0,.2), [.2,.6), [.6,1]. Requiring at least ten clips and ten sources in every A/D/N group retained four signed moderate/strong strata: A=458, D=133, N=349 (940 clips). Identical weights from the intended cohort were applied to every group/model. Effective sample sizes are A=340.0, D=112.5, N=285.6. We report full-cohort raw, common-support raw, and standardized contrasts separately: Qwen2.5 audio D-A is +.00484, +.00194, and -.01928, respectively; its simultaneous interval still includes zero.
The standardized D-minus-A utility differences are:
| Model | Video, estimate [simultaneous 95% CI] | Audio, estimate [simultaneous 95% CI] |
|---|---|---|
| Qwen2.5 | -.05370 [-.11891,.01152] | -.01928 [-.06119,.02264] |
| MiniCPM-o | -.05807 [-.17237,.05623] | -.04264 [-.10645,.02117] |
| Qwen3, fixed normalization | +.00718 [-.10516,.11953] | -.00139 [-.06412,.06134] |
These six comparisons form a separate exploratory simultaneous-interval family, using 20,000 joint source-cluster draws; all draws were valid. All six intervals include zero. We therefore have not established a stable disagreement-utility association; this also does not establish its absence. Coarse-stratum standardization leaves residual intensity imbalance: weighted mean absolute targets are .5903/.5128/.5141 for A/D/N, despite identical polarity shares. This is not exact matching or independent confirmation, and the intervals condition on fixed support/weights. Task 4 has no AV arm, so this does not reassess the original transcript-addition effect.
We retain the bounded contribution: the audit makes different reference inputs, target composition, and format outcomes explicit when interpreting reliability. The evidence supports those interpretive safeguards, but does not validate annotation disagreement as a predictor of modality utility or establish a causal conflict effect under full model inputs.
Thank you for highlighting prior-work overlap, configuration sensitivity, and the interpretation of our gate result.We have kept the revised paper's contribution to an evaluation audit, not a new conflict benchmark or fusion architecture.
Prior work and substantive interpretation changes. Joint reporting changes two interpretations within our study: a lower strict operational risk under blur does not establish better numerical sentiment prediction, and degradation relative to clean input does not determine whether deleting the degraded modality helps. Robust-MSA studies simulated modality noise and defenses; EmoMM provides conflict/missingness evaluations and a steering method. We have not re-run either study or claimed to change its published conclusions. Our contribution is the bounded joint audit of annotation-protocol disagreement, physical degradation, predictive input utility and output-format compliance (revised Sections 1 and 5, Table 1).
In the original Qwen3-Omni physical campaign, strong blur minus clean appears beneficial under strict all-intended risk: −0.1040, source-cluster 95% CI [−0.1638,−0.0450], with strict-valid counts of 750/818 out of 1,034 per arm. Under the fixed narrow syntax normalization, the all-intended difference is +0.0034 [−0.0092,+0.0152]; recovered valid-pair MAE differs by +0.0018 [−0.0099,+0.0123] on 1,031 pairs. The four strict-format cells are 654/96/164/120 (both valid/clean only/blur only/neither), a descriptive decomposition rather than a causal mechanism (revised Sections 1 and 6; Appendix G, Table 30). In a separate post-review same-campaign comparison, MiniCPM-o's stronger blur worsens MAE versus clean TAV by +0.02456 [+.01231,+.03715] (marginal interval), yet retaining blurred video beats removing it by −0.03511 [−.06251,−.00771] (exploratory approximate 12-comparison simultaneous interval). This is a contrast of different reference inputs, not a claim that blur improves prediction (revised Section 7, Table 6; Appendix O, Table 48).
Prompt and media-budget sensitivity. We ran separate post-review fixed-cohort checks, each with a same-campaign P0/B16 reference. P1 paraphrases the task description while preserving the output contract; Qwen3-only P2 paraphrases the format instruction; Bhalf changes the native media budget. On the original CH-SIMS v2 D group (149 clips), Qwen2.5-Omni's paired AV−TAV MAE is −.06644 [−.10932,−.02945] at P0/B16, −.08792 [−.13740,−.04362] at P1/B16, and −.07181 [−.11889,−.03182] at P0/Bhalf. On 512 fixed MELD utterances, MiniCPM-o's paired T−TAV accuracy advantage is +9.96 [6.56,13.69], +8.40 [4.56,12.53], and +9.18 [5.42,13.04] percentage points, respectively. Both arms are strict-valid for every one of the 149 or 512 paired items in these checks. The directions persist across these tested settings; all four configuration-interaction intervals cross zero, which does not prove equal effect sizes. Qwen3's strict-valid clean/strong-blur counts change from 750/819 at new-campaign P0/B16 to 594/741 at P1/B16 and 554/744 at Qwen3-only P2/B16; Bhalf gives 680/723, each out of 1,034 per arm. Fixed syntax normalization makes all produced outputs scorable in these new checks, and every corresponding numerical blur-minus-clean interval spans zero. The original campaign's 750/818 counts are distinct from the new P0 campaign's 750/819. We did not isolate a change to the media input template; Bhalf also changes native frame-rate metadata and MiniCPM-o audio chunk allocation. Revised Section 7, Table 5, and Appendix O give the settings, intervals, coverage and anonymous verbatim-prompt configuration archive.
Detector quality versus gate utility. The matched-supervision study does not show a Factorized advantage over Utility. In visual temporal dropout at severity .3, the Factorized detector's three AUROCs are .9534/.9527/.9527, yet Factorized−Utility absolute MAE is +.01007 [+.00486,+.01514] and its paired error-increment difference is +.00844 [+.00496,+.01241]. In audio temporal dropout .3 the increment difference instead favors Factorized (−.00126 [−.00212,−.00038]) while its absolute-MAE difference is uncertain (+.00038 [−.00367,+.00450]). These 12 condition-level comparisons are exploratory, source-cluster intervals without multiplicity adjustment; revised Appendix N, Tables 43–44, provides every F/U absolute MAE, increment, difference, interval and corresponding detector AUROC. High visual detection discriminability rules out a detection-is-always-near-chance explanation for that one visual condition, but does not separate detector calibration, intervention task relevance or the gate-combination rule. The joint variant changes two gradient boundaries together. To address readability, revised Table 1 defines the four objects, the introduction leads with the two interpretation changes, and the gate result is presented as supporting negative evidence (revised Sections 1, 4.4 and 8).