acceptodds
Under review as a conference paper at ICLR 2027

Selective Evidence Loss: Task-Relevant Missingness in Multimodal Sentiment Analysis

Abstract

Real-world multimodal systems can lose information selectively through content filtering and privacy protection. Existing incomplete multimodal learning mainly characterizes missingness by its amount and structure, leaving the task relevance of missing content underexplored. We introduce Selective Evidence Loss (SEL) to study task-relevant missingness under controlled deletion, and operationalize task relevance in multimodal sentiment analysis using contextual affective intensity. SEL causes greater performance degradation than random deletion under matched deletion amounts, and the gap persists after matching affected utterances and within-utterance deletion structure. This vulnerability is not consistently resolved by mechanism-matched training. We propose Selective Relational Distillation and Preservation (SRDP), which provides relational supervision for affected samples while preserving model behavior on unaffected samples. SRDP produces a single deployment checkpoint and requires only the corrupted multimodal input at inference. In the main evaluation, SRDP reduces affected-sample MAE for all ten models, with a model-average reduction of 0.012. These results show that robustness depends on what evidence is lost, highlighting task relevance as a key dimension of missing-information robustness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Selective Evidence Loss, Task-Relevant Missingness, Multimodal Sentiment Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HSuvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Selective Evidence Loss (SEL) to study task-relevant partial missingness in multimodal sentiment analysis, and proposes Selective Relational Distillation and Preservation (SRDP) to improve robustness under such corruption.

Critical strengths
  1. Clearly identifies task relevance as an dim in missing-modality learning fields.
  2. Structure-matched controls are carefully designed.
  3. Broad evaluation across 10 models, two datasets, and multiple severity levels.
Critical weaknesses
  1. The real-world prevalence of SEL-style missingness is not sufficiently justified and it currently appears more like a human-made missingness. Could the authors provide more concrete evidence, such as industry or consulting reports, demonstrating that this type of selective missingness occurs in practice? Otherwise, the proposed setting may be viewed as a carefully constructed missing-modality task with limited practical relevance, which is important given the strongly application-driven nature of missing-modality learning.
  2. The main observation is somewhat intuitive: removing sentiment-relevant words should naturally be more harmful than random deletion.
  3. The reported SRDP performance is partially confounded by checkpoint averaging: the final deployed model is the midpoint between the native and SRDP checkpoints, and this midpoint often outperforms the SRDP student itself. Comparable averaging controls for alternative training strategies are missing.
  4. The proposed mitigation is validated much less broadly than the SEL phenomenon itself. The most comprehensive SRDP comparison focuses on MOSEI at AER60 in Table 1, leaving its robustness across datasets (e.g., IEMOCAP) and severity levels less clearly established.
  5. (a) MOSI and MOSEI do not appear to be properly cited. In addition, IEMOCAP is also commonly used in multimodal sentiment analysis and would provide a useful complementary benchmark, especially since MOSI and MOSEI are closely related datasets. Furthermore, it is very strange that the baselines were not correctly cited in the main paper but only in the appendix. (b) For those terms: clean–corrupted joint training and pointwise distillation, they are not explained well in the main paper. Furthermore, the metrics used are not well explained. The current experimental presentation of the paper is in a state of disarray.The authors should correct all these errors.
Questions for authors

See above

6 comments
AuthorsOpenReview ↗

Response to Reviewer HSuv (1/4)

W1 Real-World Motivation for SEL

Reviewer comment

The real-world prevalence of SEL-style missingness is not sufficiently justified and it currently appears more like a human-made missingness. Could the authors provide more concrete evidence, such as industry or consulting reports, demonstrating that this type of selective missingness occurs in practice? Otherwise, the proposed setting may be viewed as a carefully constructed missing-modality task with limited practical relevance, which is important given the strongly application-driven nature of missing-modality learning.

Response

We thank the reviewer for asking for concrete examples of content-dependent missingness. Captioning and transcription systems already mask or remove words selectively based on their content. By default, YouTube masks potentially inappropriate words in automatic captions while leaving the original audio unchanged [1]. Amazon Transcribe supports removing words on a custom vocabulary list and identifies profanity filtering as a common use case [2]. McDonnell et al. analyzed 300 TikTok videos with on-screen captions and found deliberate departures from verbatim captioning in 55 of them, most often involving profanity or other vulgar expressions [3]. Cross-linguistic word ratings show that taboo words tend to have high emotional arousal [4].

Such edits can obscure sentiment cues in the text even when the audio remains available. How often these edits lead to sentiment evidence loss in downstream applications remains to be measured.

We use SEL to test the effects of selective evidence loss under controlled conditions. To assess how the choice of removed content affects model performance, we compare SEL with Structure-Matched Random on the same utterances from MOSI and MOSEI, matching deletion amount and span structure while keeping audio and video fixed (Sections 3.3 and 5.3).

References

AuthorsOpenReview ↗

Response to Reviewer HSuv (2/4)

W2 Implications of the Matched Deletion Comparison

Reviewer comment

The main observation is somewhat intuitive: removing sentiment-relevant words should naturally be more harmful than random deletion.

Response

We share the reviewer’s intuition that removing sentiment-relevant words is likely to cause a larger performance drop than deleting words at random. When part of the text is missing, the model can still use the remaining text, audio, and video. We measure how large the gap between the two deletion strategies is with those inputs still available and test whether it also appears in models designed for incomplete inputs. To do so, we compare SEL and Structure-Matched Random on the same utterances, matching the number of deleted words, the number of spans, and their lengths while keeping the audio, video, and model checkpoint fixed.

Across all 60 combinations of ten models, two datasets, and three deletion severities, the three-seed mean Non0-Acc2 is lower under SEL than under Structure-Matched Random. The gap also appears in all four models designed for incomplete inputs. At AER60, the mean gap across models is 14.2 percentage points on MOSI and 6.5 on MOSEI. Thus, even when deletion amount and span structure are matched, random deletion can overestimate a model’s robustness to the selective loss of sentiment evidence. See Section 5.3, Figure 4, and Appendix Table A5 for the full comparison.

AuthorsOpenReview ↗

Response to Reviewer HSuv (3/4)

W3 Parameter Averaging Controls

Reviewer comment

The reported SRDP performance is partially confounded by checkpoint averaging: the final deployed model is the midpoint between the native and SRDP checkpoints, and this midpoint often outperforms the SRDP student itself. Comparable averaging controls for alternative training strategies are missing.

Response

We thank the reviewer for pointing out the need for comparable parameter-averaging controls. We added results before and after parameter averaging for joint training and pointwise distillation, using the same equal-weight rule as for SRDP. For Self-MM on MOSEI at AER60, the final SRDP model reduces mean Affected MAE by 0.0174 relative to Native. Parameter averaging accounts for 0.0125 of that reduction, or about 72%. When trained with the same Native reference, pointwise distillation and SRDP benefit by similar amounts from averaging, with mean reductions of 0.0088 and 0.0111, respectively. The same rule increases Affected MAE for joint training in all three seeds.

We evaluate all checkpoints on the same 1,505 SEL-affected test samples. Within each seed, we average each trained checkpoint with the same Native checkpoint using equal weights. The last column reports the mean and sample standard deviation of the reductions computed within each of the three paired seeds.

Training strategyAffected MAE before averagingAffected MAE at the fixed midpointReduction from averaging
Joint training on clean and corrupted inputs0.75690.7716−0.0146 ± 0.0061
Pointwise distillation with the Native reference0.77130.76250.0088 ± 0.0190
SRDP with the Native reference0.76990.75880.0111 ± 0.0049
Final SRDP with an independent reference0.77060.75810.0125 ± 0.0117

With the same Native reference and averaging rule, SRDP achieves a slightly lower mean Affected MAE than pointwise distillation (0.7588 vs. 0.7625). Unaveraged joint training achieves a mean Affected MAE of 0.7569, slightly below 0.7581 for the final SRDP midpoint. The improvement over Native therefore reflects the full SRDP deployment procedure, including averaging. The additional benefit of SRDP’s training objective over these simpler alternatives remains uncertain in this setting. See Section 5.6, Appendix C.2, and Tables A7 and A10 of the revised manuscript.

Reviewer_HdF5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies task-relevant missingness in multimodal sentiment analysis and introduces Selective Evidence Loss (SEL) to evaluate its impact. The authors further propose SRDP, combining relational distillation and behavior preservation to improve robustness. Experiments on MOSI and MOSEI across ten models show that SEL causes greater performance degradation than random deletion, while SRDP improves prediction performance under selective evidence loss.

Critical strengths

1. Well-motivated research problem. The motivation is well justified, as the paper identifies a practical issue that has indeed been overlooked in previous incomplete multimodal learning research.

2. Well-designed controlled experiments. The paper introduces two control settings, Budget-Matched Random and Structure-Matched Random, to control for deletion amount and deletion structure, respectively, enabling a clearer evaluation of the impact of task relevance. The human annotation study further supports the validity of the proposed SEL benchmark.

3. Comprehensive experimental evaluation. The experimental evaluation is relatively comprehensive, covering multiple datasets and a diverse set of baseline models.

Critical weaknesses

1. Limited generalizability beyond synthetic textual corruption. The experiments are limited to artificially constructed text deletion scenarios on MOSI and MOSEI, without evaluating naturally occurring missingness or selective information loss in audio and video modalities. Therefore, it remains unclear whether the findings can generalize to broader multimodal tasks, real-world missingness mechanisms, or other modalities.

2. Limited methodological novelty. The proposed method mainly combines several existing techniques, and the improvements are marginal for some models. More importantly, the combination of these components is not clearly motivated by or directly connected to the core research problem identified in the paper.

3. Unclear paper organization and presentation. The overall organization of the paper lacks clarity, and the writing shows noticeable signs of AI assistance. In addition, several figures and tables are poorly positioned relative to their corresponding sections, making the paper difficult to follow.

Questions for authors

1. Connection between SRDP and task-relevant missingness. While Table 2 provides component-level ablations demonstrating the effectiveness of relational distillation and behavior preservation, it remains unclear whether these components specifically address task-relevant missingness or simply improve robustness to missing information in general. Could the authors provide further evidence or clarification regarding the specific advantages of SRDP under SEL compared with matched random deletion settings?

2. Attribution of performance gains. Figure 5 shows that checkpoint averaging contributes to the improvements. Could the authors clarify how much of the gain comes from the proposed training objectives versus parameter averaging, and whether similar gains can be achieved by averaging other baselines?

5 comments
AuthorsOpenReview ↗

Response to Reviewer HdF5 (1/3)

W1 Limited Generalizability Beyond Synthetic Textual Corruption

Reviewer comment

The experiments are limited to artificially constructed text deletion scenarios on MOSI and MOSEI, without evaluating naturally occurring missingness or selective information loss in audio and video modalities. Therefore, it remains unclear whether the findings can generalize to broader multimodal tasks, real-world missingness mechanisms, or other modalities.

Response

We appreciate the reviewer’s point about generalizability. Our current evaluation is limited to controlled text deletion on MOSI and MOSEI. Further evidence is needed to determine whether the findings extend to naturally occurring missingness and to selective evidence loss in audio and vision.

We started with text because it provides rich sentiment cues, as highlighted by prior work on incomplete multimodal sentiment analysis such as LNLN and TF-Mamba. Word and span boundaries let us inspect what was removed and match the number of deleted words, the number of spans, and their lengths within each utterance. By keeping audio and visual inputs fixed, we can examine how the choice of removed text affects model performance. See Sections 3.1 and 3.3.

Work to extend this controlled comparison to audio and visual evidence is underway. We have clarified the scope of our findings in Section 6 of the revised manuscript.

W2 Limited Methodological Novelty

Reviewer comment

The proposed method mainly combines several existing techniques, and the improvements are marginal for some models. More importantly, the combination of these components is not clearly motivated by or directly connected to the core research problem identified in the paper.

Response

We thank the reviewer for asking us to clarify the methodological contribution. The paper’s primary contribution is SEL, a controlled setting for evaluating task-relevant evidence loss. The matched-deletion experiments show that model performance depends on which text is removed even when deletion amount and span structure are matched.

SRDP is a mitigation approach motivated by this setting. Its design assigns supervision according to whether an example loses textual evidence under SEL. For affected examples, the clean view provides relational information that is harder to recover from the corrupted input. For examples whose text remains intact, Native predictions and representations provide targets for preserving the behavior the model has already learned. Section 4 explains how these two forms of supervision are applied.

On MOSEI at AER60, deployed SRDP reduced mean Affected MAE relative to Native for all ten backbones, with an average reduction of about 0.012 across backbones. The size of the gain varied by backbone. Q1 compares the gains under SEL and Structure-Matched Random deletion, and Q2 examines how much parameter averaging contributes. See Sections 5.3–5.6 and Table 1 of the revised paper.

W3 Paper Organization and Presentation

Reviewer comment

The overall organization of the paper lacks clarity, and the writing shows noticeable signs of AI assistance. In addition, several figures and tables are poorly positioned relative to their corresponding sections, making the paper difficult to follow.

Response

We thank the reviewer for the suggestions on the paper’s organization, writing, and placement of figures and tables. The revised Section 5 makes the purpose and setup of each comparison explicit. Section 5.1 introduces the datasets, backbones, metrics, and evaluation subsets. Sections 5.2–5.4 then validate the task-relevance proxy, present the two matched deletion comparisons, and examine mechanism-matched training. Section 5.5 presents the deployed SRDP results, and Section 5.6 covers the training controls, component ablations, and parameter-averaging analysis. Section 4 also introduces the two forms of supervision before defining their objectives.

Figure 3 accompanies the amount-matched comparison, and Figure 4 appears on the page immediately following the structure-matched discussion. Tables 1 and 2 appear together, and the parameter-averaging discussion in Section 5.6 points directly to Figure 5. The AI Use Statement describes the assistance used for language editing, figure prototyping, and code drafting, as well as how the authors checked the assisted material against the methods and experimental records.

AuthorsOpenReview ↗

Response to Reviewer HdF5 (2/3)

Q1 Connection Between SRDP and Task-Relevant Missingness

Reviewer comment

While Table 2 provides component-level ablations demonstrating the effectiveness of relational distillation and behavior preservation, it remains unclear whether these components specifically address task-relevant missingness or simply improve robustness to missing information in general. Could the authors provide further evidence or clarification regarding the specific advantages of SRDP under SEL compared with matched random deletion settings?

Response

We thank the reviewer for asking whether SRDP yields larger gains under SEL. The full SRDP method reduced the three-seed mean Affected MAE for all ten backbones under both SEL and Structure-Matched Random deletion. Only two backbones showed a larger mean reduction under SEL, and each had at least one seed with an equal or larger reduction under random deletion.

SEL motivates the two training objectives. For training utterances that lose sentiment cues under SEL, the paired clean views retain those cues. We use these views to supervise pairwise representation similarities and differences between predicted sentiment scores, while the original task loss anchors individual predictions. For utterances whose text remains intact, we preserve Native predictions and representations to limit drift as updates to shared parameters adapt the model to affected utterances. See Sections 4.2–4.4. Random deletion can also remove useful sentiment cues, which may help explain why the same SEL-trained model improves performance under both conditions.

On MOSEI at AER60, we hold the paired SEL-trained Native and deployed SRDP checkpoints fixed for each backbone and seed. Of the 1,505 SEL-affected utterances, 1,498 admit a valid alternative deletion that matches the number of deleted words, the number of spans, and their lengths. We evaluate both deletion settings on this shared subset. We average MAE across three random draws within each seed and compare the MAE reduction from Native to SRDP across the two settings. Table 2 reports component ablations under SEL. The new comparison evaluates the full deployed method across both deletion settings.

Table A11. Affected MAE and SRDP gains under SEL and Structure-Matched Random deletion on MOSEI at AER60

BackboneSEL MAE, Native → SRDPRandom MAE, Native → SRDPSEL gainRandom gainSEL gain − Random gain
Self-MM0.776851 → 0.7595910.644689 → 0.6237870.017260 ± 0.0071740.020902 ± 0.003980−0.003642 ± 0.006746
MMIM0.810143 → 0.7746900.718476 → 0.6802640.035454 ± 0.0076360.038211 ± 0.006272−0.002758 ± 0.005838
MISA0.792985 → 0.7780650.671970 → 0.6569710.014921 ± 0.0093250.014999 ± 0.003757−0.000078 ± 0.005655
ALMT0.771853 → 0.7621420.651278 → 0.6377460.009711 ± 0.0106890.013532 ± 0.014581−0.003821 ± 0.005177
CENET0.769607 → 0.7664810.633072 → 0.6264020.003126 ± 0.0044290.006670 ± 0.008038−0.003544 ± 0.004467
MulT0.801751 → 0.7948500.695498 → 0.6814690.006901 ± 0.0096820.014029 ± 0.014960−0.007128 ± 0.007164
LNLN0.775881 → 0.7738530.643880 → 0.6412000.002027 ± 0.0013980.002680 ± 0.002413−0.000653 ± 0.002013
TF-Mamba0.763766 → 0.7587140.708117 → 0.6984380.005052 ± 0.0120310.009679 ± 0.005617−0.004627 ± 0.013712
RECAP-MSA0.871799 → 0.8661780.789236 → 0.7836780.005621 ± 0.0046510.005558 ± 0.006236+0.000063 ± 0.002922
EBMC0.872289 → 0.8590070.777382 → 0.7649960.013282 ± 0.0045540.012387 ± 0.006223+0.000895 ± 0.001672

The arrows show three-seed mean Affected MAEs. Gain is Native MAE minus SRDP MAE. Gains and gain differences are computed within each paired seed and reported as mean ± sample standard deviation across three seeds.

See Section 5.5, Appendix C.4, and Table A11 of the revised manuscript for the experimental setup and results.

AuthorsOpenReview ↗

Response to Reviewer HdF5 (3/3)

Q2 Attribution of Performance Gains

Reviewer comment

Figure 5 shows that checkpoint averaging contributes to the improvements. Could the authors clarify how much of the gain comes from the proposed training objectives versus parameter averaging, and whether similar gains can be achieved by averaging other baselines?

Response

We thank the reviewer for asking us to clarify how much of the improvement comes from parameter averaging. For Self-MM on MOSEI at AER60, the final SRDP student reduced mean Affected MAE by 0.0049 relative to Native before parameter averaging. The fixed midpoint reduced mean Affected MAE by a further 0.0125, accounting for about 72% of the total reduction of 0.0174. The same averaging rule also reduced mean Affected MAE for pointwise distillation by 0.0088, bringing its total reduction relative to Native to 0.0130.

We use the training controls in Appendix Table A7 and evaluate all checkpoints on the same 1,505 SEL-affected test samples. For each seed, we average each trained checkpoint with the same Native checkpoint, giving each a weight of 0.5. Native has a mean Affected MAE of 0.7755. We compute the reduction from averaging within each seed and report the mean ± sample standard deviation across three seeds. Positive values indicate lower error after averaging.

Training strategyAffected MAE before averagingAffected MAE at the fixed midpointReduction from averaging
Joint training on clean and corrupted inputs0.75690.7716−0.0146 ± 0.0061
Pointwise distillation with the Native reference0.77130.76250.0088 ± 0.0190
SRDP with the Native reference0.76990.75880.0111 ± 0.0049
Final SRDP with an independent reference0.77060.75810.0125 ± 0.0117

When both methods use Native as the training reference, the same averaging rule reduces mean Affected MAE by 0.0111 for SRDP and 0.0088 for pointwise distillation. The resulting midpoint errors are 0.7588 and 0.7625, respectively. The final SRDP variant with an independent reference reaches 0.7581. Its mean error is lower than that of pointwise distillation, and the ordering varies across seeds.

Unaveraged joint training achieved a mean Affected MAE of 0.7569, slightly lower than 0.7581 for the final SRDP midpoint. Averaging increased the error of joint training in all three seeds. The final SRDP midpoint had lower error than the joint-training midpoint in every paired seed. See Section 5.6, Appendix C.2, and Table A10 for the full paired comparison.

Reviewer_8zjsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies task-relevant missingness in multimodal sentiment analysis. It introduces Selective Evidence Loss (SEL), a controlled setting that preferentially deletes textual expressions with high contextual affective intensity to simulate task-relevant evidence loss.

The authors propose Selective Relational Distillation and Preservation (SRDP), which trains a native model and a student with sample-conditional supervision, then averages their checkpoints for inference.

Experiments across two datasets and ten backbones demonstrate that SEL is more challenging than matched random deletion, while SRDP reduces affected-sample MAE across all ten backbones on MOSEI AER60.

Critical strengths

Strength 1

The paper highlights task relevance as an underexplored dimension of missing-information robustness. The controlled comparisons across different models, datasets, and missingness severities, together with the human audit, provide convincing evidence that the relevance of missing content significantly affects model performance.

Strength 2

The paper propose SRDP, which can be applied to diverse backbones and deployed as a single checkpoint without additional inference requirements. It improves mean Affected MAE across all ten evaluated models on MOSEI AER60. The authors also provide careful experimental controls, including paired training seeds, validation-based checkpoint selection, and analyses of mechanism-matched training and reference-view choices, which strengthen the empirical evaluation.

Critical weaknesses

Weakness 1

Generality of the SRDP component analysis Section 5.4 motivates SRDP by showing that SEL-matched training yields inconsistent improvements across different model–dataset–severity configurations. However, the key component ablations in Table 2 are conducted only on Self-MM with MOSEI AER60. Given that the authors motivate sample-conditional supervision through these cross-condition differences, it remains unclear whether Relational Transfer and Native Preservation consistently contribute to improvements across different backbones, datasets, and missingness severities. More representative component ablations would strengthen the evidence for the proposed design.

Weakness 2

Lack of Comparison with alternative training strategies. Section 5.4 compares Random-trained and SEL-trained models, showing that mechanism-matched training does not consistently improve performance. However, the main SRDP evaluation in Table 1 only compares SRDP against the SEL-trained Native baseline. Since the latter does not consistently outperform Random-trained models, improvements over Native alone may not establish SRDP's advantage over this existing alternative. A direct comparison including Random-trained models on the same affected samples and primary Affected MAE metric would provide stronger evidence for SRDP's effectiveness.

Questions for authors

Question 1

Could the authors provide additional component ablations across representative models, datasets, and missingness severities to verify whether Relational Transfer and Native Preservation consistently contribute to the improvements? Section 5.4 motivates SRDP through the inconsistent gains of SEL-matched training across different model–dataset–severity configurations, while the component ablations in Table 2 are limited to Self-MM on MOSEI AER60.

Question 2

Could the authors include the Budget-Matched Random-trained baseline in Table 1, using the same evaluation setting and metrics, particularly Affected MAE? The current main comparison only includes Native, which is trained directly on SEL-corrupted data. As shown in Figure A1, SEL-matched training sometimes underperforms random-corruption training. Therefore, it remains unclear whether SRDP consistently outperforms this alternative baseline.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8zjs (1/2)

Q1 Component Ablations

Reviewer comment

Could the authors provide additional component ablations across representative models, datasets, and missingness severities to verify whether Relational Transfer and Native Preservation consistently contribute to the improvements? Section 5.4 motivates SRDP through the inconsistent gains of SEL-matched training across different model–dataset–severity configurations, while the component ablations in Table 2 are limited to Self-MM on MOSEI AER60.

Response

We thank the reviewer for suggesting a broader component analysis. Native Preservation lowered mean Unaffected MAE at deployment in all four configurations. The effects of Relational Transfer varied across sample subsets and checkpoints.

Starting from the Self-MM ablation on MOSEI at AER60, we added three configurations by varying the dataset, deletion severity, and backbone one at a time. We removed one auxiliary loss at a time and evaluated all four configurations on held-out test data. Within each configuration and seed, all variants used the same test examples, and each student checkpoint was averaged with the same Native checkpoint using equal weights of 0.5.

The table reports the deployed MAE difference between each ablated variant and full SRDP as the mean ± sample standard deviation across three paired seeds. Positive values indicate higher error after removing the component.

Configurationw/o Relational Transfer, Affectedw/o Relational Transfer, Unaffectedw/o Native Preservation, Affectedw/o Native Preservation, Unaffected
Self-MM / MOSEI / AER60−0.0025 ± 0.0049−0.0004 ± 0.0038+0.0223 ± 0.0097+0.0016 ± 0.0023
Self-MM / MOSI / AER60−0.0026 ± 0.0250−0.0072 ± 0.0147+0.0337 ± 0.0241+0.0345 ± 0.0296
Self-MM / MOSEI / AER40−0.0060 ± 0.0113−0.0039 ± 0.0045+0.0378 ± 0.0062+0.0103 ± 0.0055
TF-Mamba / MOSEI / AER60−0.0033 ± 0.0129+0.0028 ± 0.0019−0.0005 ± 0.0093+0.0037 ± 0.0044

Native Preservation lowered deployed Unaffected MAE in 10 of the 12 paired-seed comparisons. This supports retaining Native supervision on utterances whose text remains intact while adapting to affected utterances. For Self-MM, it also lowered deployed Affected MAE in all nine paired-seed comparisons across the three configurations.

Relational Transfer lowered the unaveraged Self-MM student’s Affected MAE by 0.0067 on MOSEI at AER40. For TF-Mamba, it lowered deployed Unaffected MAE in all three seeds. Removing Relational Transfer lowered mean deployed Affected MAE by 0.0025–0.0060 across the four configurations, with the direction of the difference varying across seeds.

See Section 5.6 and Table 2 for the deployed results, Appendix C.1 for the paired-seed results, and Appendix C.5 and Table A12 for the student results before averaging.

AuthorsOpenReview ↗

Response to Reviewer 8zjs (2/2)

Q2 Budget-Matched Random Training

Reviewer comment

Could the authors include the Budget-Matched Random-trained baseline in Table 1, using the same evaluation setting and metrics, particularly Affected MAE? The current main comparison only includes Native, which is trained directly on SEL-corrupted data. As shown in Figure A1, SEL-matched training sometimes underperforms random-corruption training. Therefore, it remains unclear whether SRDP consistently outperforms this alternative baseline.

Response

We thank the reviewer for requesting this important baseline. We added Budget-Matched Random-trained results for all ten backbones and evaluated these models alongside SEL-trained Native and SRDP on the same SEL-corrupted MOSEI test split at AER60. Across the 1,505 SEL-affected test samples, SRDP achieved a lower three-seed mean Affected MAE than Random-trained for nine of the ten backbones. RECAP-MSA was the exception. Appendix Table A3 reports the size of the affected subset.

BackboneVenue / YearBudget-Matched Random-trainedSEL-trained NativeSRDP
Self-MMAAAI 20210.79560.77550.7581
MMIMEMNLP 20210.83580.81000.7739
MISAACM MM 20200.78610.79350.7788
ALMTEMNLP 20230.79130.77270.7622
CENETIEEE TMM 20230.78150.77000.7654
MulTACL 20190.81410.80340.7969
LNLNNeurIPS 20240.78330.77570.7737
TF-MambaFindings of EMNLP 20250.77440.76400.7585
RECAP-MSAAAAI 20260.79110.87350.8680
EBMCCVPR 20260.87950.87260.8597

For RECAP-MSA, the gap between SRDP and Random-trained was driven mainly by seed 1111 (Appendix C.2).

Section 5.5 discusses the results, and Table 1 reports the full comparison across all six metrics.

Thank you for your response. I have no further concerns and will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.