- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces Selective Evidence Loss (SEL) to study task-relevant partial missingness in multimodal sentiment analysis, and proposes Selective Relational Distillation and Preservation (SRDP) to improve robustness under such corruption.
Critical strengths
- Clearly identifies task relevance as an dim in missing-modality learning fields.
- Structure-matched controls are carefully designed.
- Broad evaluation across 10 models, two datasets, and multiple severity levels.
Critical weaknesses
- The real-world prevalence of SEL-style missingness is not sufficiently justified and it currently appears more like a human-made missingness. Could the authors provide more concrete evidence, such as industry or consulting reports, demonstrating that this type of selective missingness occurs in practice? Otherwise, the proposed setting may be viewed as a carefully constructed missing-modality task with limited practical relevance, which is important given the strongly application-driven nature of missing-modality learning.
- The main observation is somewhat intuitive: removing sentiment-relevant words should naturally be more harmful than random deletion.
- The reported SRDP performance is partially confounded by checkpoint averaging: the final deployed model is the midpoint between the native and SRDP checkpoints, and this midpoint often outperforms the SRDP student itself. Comparable averaging controls for alternative training strategies are missing.
- The proposed mitigation is validated much less broadly than the SEL phenomenon itself. The most comprehensive SRDP comparison focuses on MOSEI at AER60 in Table 1, leaving its robustness across datasets (e.g., IEMOCAP) and severity levels less clearly established.
- (a) MOSI and MOSEI do not appear to be properly cited. In addition, IEMOCAP is also commonly used in multimodal sentiment analysis and would provide a useful complementary benchmark, especially since MOSI and MOSEI are closely related datasets. Furthermore, it is very strange that the baselines were not correctly cited in the main paper but only in the appendix. (b) For those terms: clean–corrupted joint training and pointwise distillation, they are not explained well in the main paper. Furthermore, the metrics used are not well explained. The current experimental presentation of the paper is in a state of disarray.The authors should correct all these errors.
Questions for authors
See above
6 comments
Response to Reviewer HSuv (2/4)
W2 Implications of the Matched Deletion Comparison
Reviewer comment
The main observation is somewhat intuitive: removing sentiment-relevant words should naturally be more harmful than random deletion.
Response
We share the reviewer’s intuition that removing sentiment-relevant words is likely to cause a larger performance drop than deleting words at random. When part of the text is missing, the model can still use the remaining text, audio, and video. We measure how large the gap between the two deletion strategies is with those inputs still available and test whether it also appears in models designed for incomplete inputs. To do so, we compare SEL and Structure-Matched Random on the same utterances, matching the number of deleted words, the number of spans, and their lengths while keeping the audio, video, and model checkpoint fixed.
Across all 60 combinations of ten models, two datasets, and three deletion severities, the three-seed mean Non0-Acc2 is lower under SEL than under Structure-Matched Random. The gap also appears in all four models designed for incomplete inputs. At AER60, the mean gap across models is 14.2 percentage points on MOSI and 6.5 on MOSEI. Thus, even when deletion amount and span structure are matched, random deletion can overestimate a model’s robustness to the selective loss of sentiment evidence. See Section 5.3, Figure 4, and Appendix Table A5 for the full comparison.
Response to Reviewer HSuv (3/4)
W3 Parameter Averaging Controls
Reviewer comment
The reported SRDP performance is partially confounded by checkpoint averaging: the final deployed model is the midpoint between the native and SRDP checkpoints, and this midpoint often outperforms the SRDP student itself. Comparable averaging controls for alternative training strategies are missing.
Response
We thank the reviewer for pointing out the need for comparable parameter-averaging controls. We added results before and after parameter averaging for joint training and pointwise distillation, using the same equal-weight rule as for SRDP. For Self-MM on MOSEI at AER60, the final SRDP model reduces mean Affected MAE by 0.0174 relative to Native. Parameter averaging accounts for 0.0125 of that reduction, or about 72%. When trained with the same Native reference, pointwise distillation and SRDP benefit by similar amounts from averaging, with mean reductions of 0.0088 and 0.0111, respectively. The same rule increases Affected MAE for joint training in all three seeds.
We evaluate all checkpoints on the same 1,505 SEL-affected test samples. Within each seed, we average each trained checkpoint with the same Native checkpoint using equal weights. The last column reports the mean and sample standard deviation of the reductions computed within each of the three paired seeds.
| Training strategy | Affected MAE before averaging | Affected MAE at the fixed midpoint | Reduction from averaging |
|---|---|---|---|
| Joint training on clean and corrupted inputs | 0.7569 | 0.7716 | −0.0146 ± 0.0061 |
| Pointwise distillation with the Native reference | 0.7713 | 0.7625 | 0.0088 ± 0.0190 |
| SRDP with the Native reference | 0.7699 | 0.7588 | 0.0111 ± 0.0049 |
| Final SRDP with an independent reference | 0.7706 | 0.7581 | 0.0125 ± 0.0117 |
With the same Native reference and averaging rule, SRDP achieves a slightly lower mean Affected MAE than pointwise distillation (0.7588 vs. 0.7625). Unaveraged joint training achieves a mean Affected MAE of 0.7569, slightly below 0.7581 for the final SRDP midpoint. The improvement over Native therefore reflects the full SRDP deployment procedure, including averaging. The additional benefit of SRDP’s training objective over these simpler alternatives remains uncertain in this setting. See Section 5.6, Appendix C.2, and Tables A7 and A10 of the revised manuscript.
Response to Reviewer HSuv (1/4)
W1 Real-World Motivation for SEL
Reviewer comment
Response
We thank the reviewer for asking for concrete examples of content-dependent missingness. Captioning and transcription systems already mask or remove words selectively based on their content. By default, YouTube masks potentially inappropriate words in automatic captions while leaving the original audio unchanged [1]. Amazon Transcribe supports removing words on a custom vocabulary list and identifies profanity filtering as a common use case [2]. McDonnell et al. analyzed 300 TikTok videos with on-screen captions and found deliberate departures from verbatim captioning in 55 of them, most often involving profanity or other vulgar expressions [3]. Cross-linguistic word ratings show that taboo words tend to have high emotional arousal [4].
Such edits can obscure sentiment cues in the text even when the audio remains available. How often these edits lead to sentiment evidence loss in downstream applications remains to be measured.
We use SEL to test the effects of selective evidence loss under controlled conditions. To assess how the choice of removed content affects model performance, we compare SEL with Structure-Matched Random on the same utterances from MOSI and MOSEI, matching deletion amount and span structure while keeping audio and video fixed (Sections 3.3 and 5.3).
References