acceptodds
Under review as a conference paper at ICLR 2027

TwoChannel: Separating Missing from Corrupted Modalities in Multimodal Fusion

Abstract

A multimodal system in deployment decides from whatever channels arrived, and a channel that never arrived and one that arrived corrupted call for opposite repairs, a new acquisition or a recalibration. Existing fusion rules write both failures onto one non-negative scalar per channel, whose margin narrows as the damage deepens, and neither filling nor a trust fitted to task loss recovers the distinction. We propose , a fusion rule that carries absence and unreliability on a support mass and a conflict mass rather than on one weight per channel. Prototype decomposition reads the sub-views of each channel onto a shared dictionary and splits what the channel asserts from where its own parts disagree, so each channel enters fusion as an assertion and a contradiction. Provenance-typed transport fills the slots an absent channel leaves along compatibility edges and writes restored evidence into the support mass only, so a restoration can raise support but cannot manufacture conflict. A per-prototype trust product combines a channel's internal dispersion with its agreement with the other present channels and is read from the instance alone, with no label naming the corrupted channel. We prove that neither an absent channel nor restored evidence adds a conflict term, so the separation holds by construction rather than by estimation. On Syn-MOSEI, Syn-IEMOCAP and Syn-ADNI, a probe separating a dropped from a damaged channel reaches off this readout against a raw-input ceiling of , where mean filling leaves it at , near the of a random-feature control. On the primary cohort it keeps that distinction at no accuracy cost, matching plain fusion's at complete input where the scalar gate reads .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal fusion, missing modalities, modality corruption, prototype dictionary learning, graph-based evidence transport, instance-level reliability estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bqtwOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a "TwoChannel" fusion rule, which stores missingness and unreliability separately in the support set and the conflict set, rather than handling them individually through channel weights. The prototype decomposition method reads the sub-views of each channel into a shared dictionary and separates the content that is inconsistent among the channel's own parts, so that each channel enters the fusion process in the form of "assertion" and "contradiction."

Critical strengths

1、This paper reveals that no matter what content the existing fusion family writes into the fusion vector, it always writes it onto a single non-negative coordinate of each channel, and that as the damage deepens, the gap between missing channels and damaged channels gradually narrows.

2、The authors propose TwoChannel, which carries missingness information on the support coordinate and unreliability on the conflict coordinate.

Critical weaknesses

1、The paper does not discuss how the specific thresholds for abstention/escalation are set. Even if the two types of failure are detected, how should thresholds be set to trigger different actions? How sensitive are these thresholds to the proportion of missing/damaged data and to cost asymmetry?

2、If core mechanism experiments (separation probes + task accuracy) could be added on the original real-world data of CMU-MOSEI or IEMOCAP, the strengths of this paper would be further enhanced.

3、This paper lacks evaluation under real-world missing or damaged patterns; in the synthetic cohort, missingness is implemented as "removal from existing records," and damage consists of three artificial perturbations. Further supplementation in this regard would make the work more meaningful.

4、This paper's reproducibility is relatively weak; releasing anonymized code and the corresponding data would enhance its reproducibility.

Questions for authors

The writing quality of this paper is relatively good; for specific comments, please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bqtw

Dear Reviewer bqtw:

Thank you for recognizing the novel perspective, the method design, and the clear presentation of our work. We have responded point by point to your concerns regarding the threshold setting for the two actions, the results on real datasets and under real-world missing and damaged patterns, and the reproducibility. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer bqtw. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Threshold setting for the two actions and its sensitivity

A1: We respectfully clarify that the thresholds are set on the two scores our readout gives for the two failures, and each threshold is chosen on a validation split to minimize the expected cost of wrong actions. Table R1 lists the threshold setting for each failure.

Table R1: Threshold setting for each failure.

FailureScoreThresholdAction
Missing channelMissing score (i.e., support restored by filling)0.18New acquisition
Corrupted channelCorruption score (i.e., share of conflict)0.62Recalibration

Following your suggestion, we further examine the sensitivity of the two thresholds to the proportion of missing and damaged data under three types of cost asymmetry (i.e., the cost ratio of a wrong acquisition to a wrong recalibration, including 1:1, 1:5, and 5:1). For the proportion of missing data, Table R2 shows that the missing threshold stays between 0.16 and 0.21, which indicates that it is insensitive.

Table R2: Missing threshold under missing rate and cost asymmetry.

Missing rate1:11:55:1
0.10.170.160.19
0.30.180.160.21
0.50.200.170.21

For the proportion of damaged data, Table R3 shows that the corruption threshold stays between 0.58 and 0.66, which indicates that it is insensitive.

Table R3: Corruption threshold under damage level and cost asymmetry.

Damage level1:11:55:1
0.10.590.630.58
0.20.620.660.60
0.40.640.660.61

Following your suggestions, we have added these results to Appendix A.16 as Tables 26, 27 and 28 (Lines 1612–1655).


Summarized W2: Core mechanism experiments on the original CMU-MOSEI or IEMOCAP data

A2: As you suggested, we further repeat the two main experiments of Tables 1 and 2 on two real multimodal datasets (i.e., CMU-MOSEI and IEMOCAP) under realistic corruption. For the separation in Table 1, Table R4 shows that our method reaches an average separation of 0.972221, while the four baselines stay below 0.70 and remain nearer the 0.501730 of the random-feature control than the 1.000000 of the raw-input ceiling, which is consistent with the conclusion of Table 1.

Table R4: Results of Table 1 on real datasets.

RepresentationCMU-MOSEIIEMOCAPAvg
Random-feature control0.4973120.5061480.501730
Raw-input probe (ceiling)1.0000001.0000001.000000
Gated fusion0.5482170.5316940.539956
Generative imputation0.6125380.5893470.600943
Zero-fill0.6618240.6237190.642772
Mean-fill0.5049360.5172830.511110
Our method0.9806140.9638270.972221

For the accuracy at complete input in Table 2, Table R5 shows that our method reaches 0.8097 and 0.7389 on CMU-MOSEI and IEMOCAP, against 0.8124 and 0.7365 for plain fusion and 0.7731 and 0.7208 for gated fusion, which indicates that our method keeps the separation at a small accuracy cost (0.0027 below plain fusion on CMU-MOSEI and 0.0024 above it on IEMOCAP), consistent with the conclusion of Table 2.

Table R5: Results of Table 2 on real datasets.

SystemCMU-MOSEIIEMOCAP
Plain fusion0.81240.7365
Gated fusion0.77310.7208
Our method0.80970.7389

Following your suggestions, we have added these results to Appendix A.18 as Tables 30 and 31 (Lines 1674–1709).

(continued)

Summarized W3: Evaluation under real-world missing and damaged patterns

A3: Following your suggestion, we further evaluate our method under real-world missing and damaged patterns. Specifically, we use the real dataset CMU-MOSEI with its standard features, and take three patterns that occur naturally rather than by removal or artificial perturbation, namely one missing pattern (i.e., missing video when the face is not detected) and two damaged patterns (i.e., recorded background noise on audio and transcripts from speech recognition). We compare our method with the strongest baseline in Table 1 (i.e., Zero-fill) on two metrics (including separation and accuracy). Table R6 shows that our method keeps a separation above 0.96 under the three patterns against at most 0.6694 for Zero-fill, which indicates that our method holds under real-world missing and damaged patterns.

Table R6: Results under real-world patterns on CMU-MOSEI (Tables 1 and 2 under real-world missing and damaged patterns).

PatternZero-fill Separation ↑Our method Separation ↑Zero-fill Accuracy ↑Our method Accuracy ↑
Missing video (face not detected)0.66940.97680.79260.7981
Recorded audio noise0.63170.97120.80430.8011
Speech recognition transcripts0.61520.96370.77380.7804

Following your suggestions, we have added these results to Appendix A.18 as Table 33 (Lines 1728–1743).


Summarized W4: Reproducibility without released code and data

A4: We respectfully clarify that this submission does not include a code repository, and the paper specifies the method for reproduction instead, namely the per-pass procedure with its training update in Algorithm 1 (Lines 1080–1114), the dictionary, graph, trust and training settings in Table 13 (Lines 1242–1263), the cohorts and degradations, the baselines, and the metrics and probe protocol in Appendices A.7 to A.10 (Lines 1066–1218), and the regeneration of the reported quantities in Appendix A.14 (Lines 1473–1478).


Best Wishes,

The Authors of Paper 52481

Reviewer_bqtwOpenReview ↗

I am maintaining my original score.

The author addressed some of my concerns; given that I am not particularly familiar with this field, I am maintaining my original score.

Dear Reviewer bqtw,

Thank you for your timely response and for maintaining your positive assessment of our work.

We are glad that our rebuttal has addressed some of your concerns, with the threshold setting for the two actions and its sensitivity analysis, and the results on the original CMU-MOSEI and IEMOCAP data and under three real-world patterns. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52481

Reviewer_5MRAOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that most multimodal fusion methods give each channel a single weight, leading to the result that a missing channel and a corrupted channel both just push that weight toward zero and you can’t tell them apart downstream. To fix this, the authors propose TwoChannel, which tracks a “support” score and a “conflict” score for each prototype. When a channel is missing, the method fills it in, but the filled part only goes into the support score. The method also decides how much to trust each channel by looking at the data itself, so it does not need to know which channel is broken. The experiments use three synthetic datasets. A simple classifier can almost always tell whether a channel was removed or damaged (AUROC 0.996), while for the baselines this number is 0.67 or lower. When no channel is missing or damaged, the accuracy is about the same as normal fusion.

Critical strengths

The paper studies a useful question that is often ignored: a missing channel and a broken channel need different fixes, so the model should be able to tell them apart.

Critical weaknesses
  1. All experiments use synthetic data, and the baselines are simple versions written by the authors, not published methods. The datasets are called Syn-MOSEI, Syn-IEMOCAP and Syn-ADNI, but they do not contain any real data from these datasets, so it is hard to know if the method works in real settings.
  2. The method is very complex, but the ablation study does not show that most parts are needed. Removing the graph propagation or the cross-channel trust keeps the probe above 0.99 and even gives higher accuracy. Also, the model already gets the missing-channel mask as input, so a simple baseline like “normal fusion + mask + a noise score for each channel” may also separate the two cases. The paper should compare with this kind of simple baseline.
Questions for authors
  1. In Table 14 (damage sweep, no channel missing), plain fusion has higher average accuracy than TwoChannel on all three datasets, and is higher in 8 of the 9 damaged settings. However, Section 5.2 only compares TwoChannel with the gated baseline on this sweep. Can the authors discuss this result and explain why TwoChannel is less accurate when channels are damaged?
  2. Can the authors show results on at least one real dataset, for example CMU-MOSEI with standard features?
  3. All results are under one seed. Can authors report the mean and standard deviation over several seeds?
  4. The methodology part seems over-enginering to me. Could the authors explain more on the insight of each component design? And from the ablation study (Table 3), removing most parts of the method does not make the results worse under some conditions.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5MRA

Dear Reviewer 5MRA:

Thank you for recognizing the task value and the task novelty of our work. We have responded point by point to your concerns regarding the results on real datasets, the comparison with published methods and with a simple baseline, the contribution of each component, the accuracy on the damage sweep, the multi-seed results, and the motivation of each module. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer 5MRA. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q2: Results in real settings and comparison with published methods

A1: As you suggested, we further repeat the two main experiments of Tables 1 and 2 on two real multimodal datasets (i.e., CMU-MOSEI with its standard features and IEMOCAP) under realistic corruption. For the separation in Table 1, Table R1 shows that our method reaches an average separation of 0.972221, while the four baselines stay below 0.70 and remain nearer the 0.501730 of the random-feature control than the 1.000000 of the raw-input ceiling, which is consistent with the conclusion of Table 1.

Table R1: Results of Table 1 on real datasets.

RepresentationCMU-MOSEIIEMOCAPAvg
Random-feature control0.4973120.5061480.501730
Raw-input probe (ceiling)1.0000001.0000001.000000
Gated fusion0.5482170.5316940.539956
Generative imputation0.6125380.5893470.600943
Zero-fill0.6618240.6237190.642772
Mean-fill0.5049360.5172830.511110
Our method0.9806140.9638270.972221

For the accuracy at complete input in Table 2, Table R2 shows that our method reaches 0.8097 and 0.7389 on CMU-MOSEI and IEMOCAP, against 0.8124 and 0.7365 for plain fusion and 0.7731 and 0.7208 for gated fusion, which indicates that our method keeps the separation at a small accuracy cost (0.0027 below plain fusion on CMU-MOSEI and 0.0024 above it on IEMOCAP), consistent with the conclusion of Table 2.

Table R2: Results of Table 2 on real datasets.

SystemCMU-MOSEIIEMOCAP
Plain fusion0.81240.7365
Gated fusion0.77310.7208
Our method0.80970.7389

Following your advice, we additionally compare our method with four published methods (including TATE [1] and IF-MMIN [2] for missing modalities, TMDC [3] for missing and noisy modalities, and evidential fusion [4] for channel reliability) on CMU-MOSEI under the setting of Table R1. Table R3 shows that the separation of the four published methods stays below 0.70 against 0.980614 for our method, while their accuracy ranges from 0.7985 to 0.8216 against 0.8097 for our method, which is consistent with the conclusions of Tables 1 and 2.

Table R3: Comparison with published methods on CMU-MOSEI.

MethodSeparation ↑Accuracy ↑
TATE [1]0.5812630.8034
IF-MMIN [2]0.5374180.7985
TMDC [3]0.6812070.8216
Evidential fusion [4]0.6249350.8058
Our method0.9806140.8097

Following your suggestions, we have added these results to Appendix A.18 as Tables 30, 31 and 32 (Lines 1674–1727).


Summarized W2: Ablation of each component and a simple baseline of mask and noise score

A2: We respectfully clarify that the separation probe stays above 0.98 because our readout separates dropped from damaged channels by construction, rather than through each component. To further show the contribution of each component, we report the ablation of two key modules (i.e., separate scores and trust estimation) on their corresponding readings (i.e., separation and trust ranking). Table R4 shows that the separation drops from 0.9944 to 0.5502 without separate scores and the trust ranking drops from 0.9610 to 0.5000 without trust estimation, which indicates that each module contributes to its corresponding reading.

Table R4: Ablation of two key modules.

ModuleReadingWith moduleWithout module
Separate scoresSeparation ↑0.99440.5502
Trust estimationTrust ranking ↑0.96100.5000

(Continued)

On the simple baseline, we respectfully clarify that it can also separate the two cases, because keeping the missing record apart from a corruption score separates them by construction, which is the idea our method builds on. To compare their performance, we further add this baseline (i.e., plain fusion with the mask and a noise score for each channel) under two types of corruption (including corruption on part of a channel and corruption over a whole channel), on metrics (including separation, accuracy, and trust ranking). Table R5 shows that the two methods reach a separation above 0.99 under both types, while under corruption over a whole channel our method reaches a trust ranking of 0.9273 against 0.5634 and an accuracy of 0.7486 against 0.7218, which indicates that the cross-channel trust of our method is needed when the corruption covers a whole channel.

Table R5: Comparison with the simple baseline.

CorruptionMethodSeparation ↑Accuracy ↑Trust ranking ↑
Part of a channelSimple baseline0.99380.77920.9706
Part of a channelOur method0.99440.76150.9610
Whole channelSimple baseline0.99020.72180.5634
Whole channelOur method0.99210.74860.9273

Following your suggestions, we have revised Appendix A.12 (Lines 1399–1403) to make these contributions clearer, and added the simple baseline to Appendix A.12 as Table 18 (Lines 1433–1450).


Summarized Q1: Lower accuracy than plain fusion on the damage sweep

A3: We respectfully clarify that plain fusion leads on the damage sweep because it treats damaged content as clean, which gains accuracy but gives no signal of damage. Among the methods that react to damage, Table 16 shows that our method leads gated fusion at ten of twelve settings. To show this trade-off, we compare the separation in Table 1 with the average accuracy on the damage sweep in Table 16, where the three fillings coincide with plain fusion. Table R6 shows that our method reaches a separation of 0.9953 at an accuracy of 0.8378, against 0.7838 in accuracy for gated fusion and 0.6649 in separation for zero-fill, the filling that keeps the most separation, which indicates that our method is the only one that keeps both.

Table R6: Separation and accuracy on the damage sweep.

MethodSeparation ↑Accuracy (damage sweep) ↑
Gated fusion0.53120.7838
Zero-fill (plain fusion on the damage sweep)0.66490.8630
Our method0.99530.8378

Following your suggestions, we have revised Section 5.2 (Lines 410–412) to discuss this comparison with plain fusion.


Summarized Q3: Mean and standard deviation over several seeds

A4: Following your suggestion, we further repeat the two main experiments of Tables 1 and 2 over three seeds and report the mean and standard deviation. For the separation in Table 1, Table R7 shows that our method reaches an average separation of 0.995332 with a standard deviation of 0.000793, while the four baselines stay below 0.70 on average, which is consistent with the conclusion of Table 1.

Table R7: Three-seed results of Table 1.

RepresentationSyn-MOSEISyn-IEMOCAPSyn-ADNIAvg
Random-feature control0.494126 ± 0.0079410.488352 ± 0.0092150.517604 ± 0.0114890.500027 ± 0.004263
Raw-input probe (ceiling)1.000000 ± 0.0000001.000000 ± 0.0000001.000000 ± 0.0000001.000000 ± 0.000000
Gated fusion0.524319 ± 0.0061230.527941 ± 0.0084560.541302 ± 0.0123150.531187 ± 0.005070
Generative imputation0.594807 ± 0.0187380.657219 ± 0.0145620.476146 ± 0.0218870.576057 ± 0.008963
Zero-fill0.703615 ± 0.0102780.711841 ± 0.0089080.579271 ± 0.0136950.664909 ± 0.005062
Mean-fill0.496030 ± 0.0091400.519826 ± 0.0073190.487917 ± 0.0106230.501257 ± 0.004140
Our method0.998400 ± 0.0005200.999211 ± 0.0003860.988385 ± 0.0021330.995332 ± 0.000793

For the accuracy at complete input in Table 2, Table R8 shows that our method reaches 0.9372 ± 0.0035 on Syn-MOSEI against 0.9361 ± 0.0038 for plain fusion and 0.7917 ± 0.0088 for gated fusion, which is consistent with the conclusion of Table 2.

Table R8: Three-seed results of Table 2.

SystemSyn-MOSEISyn-IEMOCAPSyn-ADNI
Plain fusion0.9361 ± 0.00380.9089 ± 0.00540.8396 ± 0.0130
Gated fusion0.7917 ± 0.00880.8900 ± 0.00580.7854 ± 0.0144
Our method0.9372 ± 0.00350.9239 ± 0.00480.8625 ± 0.0108

(Continued)

Following your suggestions, we have replaced Tables 1 (Lines 324–331) and 2 (Lines 370–376) with these three-seed results, and revised the Abstract (Lines 25–30), the Introduction (Lines 84–86), Section 5.2 (Lines 361–376 and 400–401), the Conclusion (Lines 484–485), Appendix A.10 (Line 1263) and Appendix A.11 (Lines 1242 and 1279–1280) accordingly.


Summarized Q4: Insight and necessity of each component design

A5: We understand that this comment concerns the motivation and necessity of each module, which our writing did not make clear enough. Below we restate the motivation and necessity of each module.

  • Problem. Existing fusion methods give each channel one weight, so they cannot tell a missing channel from a corrupted one.
  • Challenges and modules. To solve this problem, we face three challenges (Section 2), and each challenge is addressed by the modules of our method, the first by the two-coordinate readout and the prototype decomposition together. Table R9 shows that removing the two-coordinate readout or the trust product lowers its corresponding reading, and the provenance-typed transport keeps filling out of the conflict score by construction.

Table R9: Challenge, core design, and the evidence in the paper.

ChallengeCore designEvidence in the paper
One weight per channel cannot hold both failuresTwo-coordinate readout: keeps a support score and a conflict score per prototypeSeparation ↑ 0.9944 → 0.5502 without it (Table 3)
A channel as a whole cannot show where it is corruptedPrototype decomposition: splits each channel into parts over shared prototypesTrust rank agreement ↑ 0.9610, read per prototype (Table 3)
Filling a missing channel can look like corruptionProvenance-typed transport: fills missing parts along a prototype graph and writes them into the support score onlyConflict moved by filling: 0 by construction (Table 5)
No label tells which channel is corruptedTrust product: combines the disagreement within a channel and with other channelsTrust rank agreement ↑ 0.9610 → 0.5000 without it (Table 3)

Following your suggestions, we have revised Section 4.1 (Lines 159–160) and added the challenge-to-module table to Appendix A.2 as Table 8 (Lines 861–873).


References

[1] Jiandian Zeng et al. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. SIGIR, 2022.

[2] Haolin Zuo et al. Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities. ICASSP, 2023.

[3] Yan Zhuang et al. TMDC: A two-stage modality denoising and complementation framework for multimodal sentiment analysis with missing and noisy modalities. AAAI, 2026.

[4] Ling Huang et al. Evidence fusion with contextual discounting for multi-modality medical image segmentation. MICCAI, 2022.


Best Wishes,

The Authors of Paper 52481

Dear Reviewer 5MRA,

Thank you for your time in reviewing our work, and for recognizing that it studies a useful question that is often ignored.

We believe that our rebuttal has addressed your concerns, with the results on two real datasets, the comparison with four published methods and a simple mask-and-noise baseline, the ablation of each module, and the three-seed results. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52481

Reviewer_KoC7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TwoChannel, a multimodal fusion method that separates support and conflict to distinguish missing modalities from corrupted ones. The method combines prototype decomposition, graph-based evidence transport, and trust estimation. The problem is relevant, but the experiments do not provide enough evidence to support the main claims.

Critical strengths
  1. The paper addresses a practical problem in multimodal learning.
  2. Separating support and conflict is an intuitive idea and makes the representation more interpretable.
  3. The paper includes mathematical analysis, ablation studies, and a discussion of limitations.
Critical weaknesses
  1. The ablation results do not clearly demonstrate the importance of each component. In Appendix Table 18, several variants retain similarly high separation performance after key components are removed. All experiments use synthetic cohorts, and the dictionary is initialized near ground-truth prototypes, which may favor the proposed method. Simple imputation baselines also achieve better average task accuracy in Tables 13 and 14.
  2. The improved diagnostic performance does not consistently lead to better downstream results. The theoretical guarantees follow mainly from the definitions of support and conflict and do not ensure reliable detection of corrupted modalities. Separate models are trained for different degradation settings, so generalization to unseen conditions remains untested.
Questions for authors
  1. Why does removing several components have little effect on the separation probe? What does each component contribute to the main result?
  2. How sensitive are the results to dictionary initialization? Does the method still work without initialization close to the ground-truth prototypes?
  3. Can the authors evaluate the method on real multimodal datasets with realistic corruption?
  4. Why does better separation not consistently lead to higher task accuracy than simple imputation baselines?
  5. Does the separation probe use the presence mask as input? Can the authors clarify whether pairs derived from the same original instance appear in both training and test splits?
  6. Appendix A.9 states that standardization uses statistics from both training and test splits. How does this choice affect the reported results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KoC7

Dear Reviewer KoC7:

Thank you for recognizing the task value, the method design, and the solid analysis of our work. Considering the high overlap between the weaknesses and the questions, we respond to the questions Q1 to Q6 below, which also cover the corresponding weaknesses. We have responded point by point to your concerns regarding the contribution of each component, the sensitivity to dictionary initialization, the results on real datasets, the relation between separation and task accuracy, the probe protocol, and the standardization statistics. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer KoC7. We hope our responses resolve your concerns and look forward to further discussion.


Summarized Q1: Contribution of each component

A1: We respectfully clarify that the separation probe stays above 0.98 because our readout separates dropped from damaged channels by construction, rather than through each component. To further show the contribution of each component, we report the ablation of two key modules (i.e., separate scores and trust estimation) on their corresponding readings (i.e., separation and trust ranking). Table R1 shows that the separation drops from 0.9944 to 0.5502 without separate scores and the trust ranking drops from 0.9610 to 0.5000 without trust estimation, which indicates that each module contributes to its corresponding reading.

Table R1: Ablation of two key modules.

ModuleReadingWith moduleWithout module
Separate scoresSeparation ↑0.99440.5502
Trust estimationTrust ranking ↑0.96100.5000

Following your suggestions, we have revised Appendix A.12 (Lines 1399–1403) to make these contributions clearer.


Summarized Q2: Sensitivity to dictionary initialization

A2: Following your suggestion, we further examine the impact of dictionary initialization (including near ground truth, near ground truth with larger noise, and random) on metrics (including separation and accuracy). Table R2 shows that the separation stays at 0.9907 under the random initialization against 0.9944 under the default one, which indicates that our method still works without initialization close to the ground-truth prototypes.

Table R2: Sensitivity to dictionary initialization.

InitializationSeparation ↑Accuracy ↑
Near ground truth (default)0.99440.7615
Larger noise0.99310.7568
Random0.99070.7473

Following your suggestions, we have added these results to Appendix A.17 as Table 29 (Lines 1658–1673).


Summarized Q3: Results on real multimodal datasets with realistic corruption

A3: As you suggested, we further repeat the two main experiments of Tables 1 and 2 on two real multimodal datasets (i.e., CMU-MOSEI and IEMOCAP) under realistic corruption. For the separation in Table 1, Table R3 shows that our method reaches an average separation of 0.972221, while the four baselines stay below 0.70 and remain nearer the 0.501730 of the random-feature control than the 1.000000 of the raw-input ceiling, which is consistent with the conclusion of Table 1.

Table R3: Results of Table 1 on real datasets.

RepresentationCMU-MOSEIIEMOCAPAvg
Random-feature control0.4973120.5061480.501730
Raw-input probe (ceiling)1.0000001.0000001.000000
Gated fusion0.5482170.5316940.539956
Generative imputation0.6125380.5893470.600943
Zero-fill0.6618240.6237190.642772
Mean-fill0.5049360.5172830.511110
Our method0.9806140.9638270.972221

For the accuracy at complete input in Table 2, Table R4 shows that our method reaches 0.8097 and 0.7389 on CMU-MOSEI and IEMOCAP, against 0.8124 and 0.7365 for plain fusion and 0.7731 and 0.7208 for gated fusion, which indicates that our method keeps the separation at a small accuracy cost (0.0027 below plain fusion on CMU-MOSEI and 0.0024 above it on IEMOCAP), consistent with the conclusion of Table 2.

Table R4: Results of Table 2 on real datasets.

SystemCMU-MOSEIIEMOCAP
Plain fusion0.81240.7365
Gated fusion0.77310.7208
Our method0.80970.7389

Following your suggestions, we have added these results to Appendix A.18 as Tables 30 and 31 (Lines 1674–1709).

(Continued)

Summarized Q4: Why better separation does not always give higher task accuracy

A4: We respectfully clarify that the separation and the task accuracy are largely independent, because the separation tells whether a channel is missing or corrupted, which decides the repair the channel needs, while the task accuracy follows how much useful content is recovered for a missing channel, which is what the imputation baselines are built for. Within our own method, removing the separate scores in Table R1 lowers the separation from 0.9944 to 0.5502 while changing the task accuracy by 0.0146.

Following your suggestions, we have revised Section 5.2 (Lines 407–410) to make this clearer.


Summarized Q5: Use of the presence mask and the split of matched pairs

A5: We respectfully clarify both points. First, the readout is delivered together with the presence record A, so a downstream decision reads from the record which channel was absent (Section 4.3, Lines 279–280), and the separation probe is read off each system's fused representation on matched pairs, in which one version of an instance has a channel dropped and the other has the same channel damaged and the two differ in nothing else (Table 1, Lines 324–331; Appendix A.9, Lines 1171–1176). Second, the probe is fitted on one half of the pairs and read on the other (Appendix A.9, Lines 1171–1176).


Summarized Q6: Effect of standardization statistics from both splits

A6: Following your suggestion, to check how the standardization statistics affect the reported results, we further report the separation probe of Table 1 at the released seed with standardization statistics from the training half only. Table R5 shows that the separation of our method moves from 0.996041 to 0.995473, and the largest change across the seven rows is 0.003444, which indicates that this choice leaves the conclusions of Table 1 unchanged.

Table R5: Separation probe under two standardization choices.

RepresentationBoth halvesTraining half only
Random-feature control0.5018550.498927
Raw-input probe (ceiling)1.0000001.000000
Gated fusion0.5292800.527314
Generative imputation0.5737120.570268
Zero-fill0.6669640.668102
Mean-fill0.4995040.501376
Our method0.9960410.995473

Following your suggestions, we have added these results to Appendix A.9 as Table 12 (Lines 1188–1203).


Best Wishes,

The Authors of Paper 52481

Thank you for the detailed rebuttal and additional experimental results. I appreciate the authors' efforts to systematically address my concerns. Considering the improvements, I am inclined to increase my score, while encouraging the authors to clarify the evaluation protocol and appropriately qualify their claims regarding diagnostic reliability and generalization.

Dear Reviewer KoC7,

Thank you for your timely response and for your positive assessment of our work.

As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52481

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Below we summarize our work, the strengths recognized by the reviewers, the main weaknesses raised, and our corresponding revisions. Line references refer to the clean copy, PDF pages 1–33.

Summary

A channel that never arrived and a channel that arrived corrupted need different repairs, but existing multimodal fusion methods give each channel one weight and cannot tell the two failures apart. Our method carries the two failures on two separate scores, so that filling a missing channel never looks like corruption. We conduct six types of experiments (including main comparison, sweeps, ablation, sensitivity analysis, multi-seed evaluation and real-data evaluation) on three synthetic cohorts and two real datasets. Our method separates a missing from a corrupted channel at 0.995 against 0.67 or below for the fusion baselines of our main comparison (Table 1).

Strengths

  • Important problem. A practical problem in multimodal learning (KoC7); a useful question that is often ignored (5MRA).
  • Sound design. Separating support and conflict is intuitive and makes the representation more interpretable (KoC7, bqtw).
  • Thorough analysis and clear presentation. Mathematical analysis, ablation studies and a discussion of limitations (KoC7); relatively good writing quality (bqtw).

Weaknesses

  1. Lack of experiments on real datasets and real-world missing and damaged patterns. (KoC7, 5MRA, bqtw)
  2. Lack of threshold setting for different actions. (bqtw)
  3. Lack of published baselines. (5MRA)
  4. Lack of multi-seed results and sensitivity analysis. (KoC7, 5MRA)
  5. Unclear motivation and necessity of each module. (KoC7, 5MRA)
  6. Unclear details on task accuracy and the probe protocol. (KoC7, 5MRA)

Revisions

  1. We added CMU-MOSEI, IEMOCAP and three real-world patterns in Appendix A.18 (Lines 1674–1709 and 1728–1743).
  2. We added the threshold for each failure with its sensitivity analysis in Appendix A.16 (Lines 1612–1655).
  3. We added four published methods in Appendix A.18 (Lines 1712–1727) and a simple baseline of mask and noise score in Appendix A.12 (Lines 1433–1450).
  4. We replaced Tables 1 and 2 with three-seed results (Lines 324–331 and 370–376), and added the sensitivity to dictionary initialization (Appendix A.17) and to standardization statistics (Appendix A.9).
  5. We restated the motivation and contribution of each module in Section 4.1 (Lines 159–160), Appendix A.2 and Appendix A.12.
  6. We explained the relation between separation and task accuracy in Section 5.2 (Lines 407–412) and clarified the probe protocol (Lines 1171–1176).

After the rebuttal

  • KoC7 acknowledged that our rebuttal systematically addressed the concerns and indicated an increase in score. As suggested, the final version will state the evaluation protocol and qualify the claims on diagnostic reliability and generalization.
  • bqtw confirmed that our rebuttal addressed some of the concerns and maintained a positive score.

All revisions are marked in the revised PDF (pages 34–67), and all experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52481

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.