acceptodds
Under review as a conference paper at ICLR 2027

Architectural Missing-Modality Robustness in Semantic Segmentation

Abstract

Multimodal semantic segmentation is typically evaluated with every modality present, but real-world deployment conditions often differ. A model trained on several modalities should therefore stay accurate on any non-empty subset of them at test-time. We propose Adapted Mixture of Modalities (A-MoM), which—instead of relying on auxiliary signals, training strategies, or model-specific losses—attains robustness architecturally through: (1) token-level, context-dependent mixing, since a sensor's advantage is usually local; (2) an initialization that keeps mixing low by default, so the model does not learn to depend on all modalities; and (3) low-rank adapters that translate features between modality pairs. Across 4 datasets spanning 8 modalities, A-MoM improves the average robustness score by 1.5 and retains on average 92.7% of the mIoU of instances of A-MoM specialized to each modality-subset on MCubeS, while also improving the average all-modalities score by 0.7. Thus, A-MoM narrows the gap from research to deployment under real-world conditions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
robustness, sensor failure, missing modalities, segmentation, robotics, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pRFTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Adapted Mixture of Modalities (A-MoM), an architectural approach to multimodal semantic segmentation under missing modalities. It combines token-level modality weighting, an initialization favoring within-modality information retention, and lightweight pairwise modality adapters. Experiments cover four datasets, multiple backbone scales, and both SegFormer and SwinV2.

Critical strengths
  1. Comprehensive eval.

  2. Competitive full-modality performance + relatively small parameter and inference-time overhead.

Critical weaknesses
  1. Average robustness does not fully characterize failure resilience. The proposed method achieves the best average scores, but it does not consistently perform best in difficult individual subsets. For example, its depth-only DSEC score is 19.6 versus 25.0 for MAGIC, and its event-only MUSES score is 24.2 versus 35.9 for CHARM. The subset results are provided, but the discussion would benefit from distinguishing average performance from worst-case robustness and from performance under realistic sensor-failure distributions.

  2. The scope of the baseline comparison needs clarification. Using common hyperparameters helps isolate architectural differences, but it does not necessarily represent the strongest configuration of each baseline. It is unclear whether all method-specific losses, modality-dropout procedures, and missing-input handling mechanisms are retained. This distinction matters for interpreting the results as either a controlled architectural comparison or a comparison against complete competing methods.

  3. Several improvements over the strongest baselines are modest.

Questions for authors
  1. Responses to the weaknesses.

  2. And also:

  • Which method-specific training components are retained for each baseline?
  • How does A-MoM perform when a modality is corrupted or spatially misaligned but remains present?
  • Can the authors report worst-subset performance and variability across multiple training seeds?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for their time and effort in reviewing our work. We appreciate the encouraging assessment of our contributions. We address the points below and will incorporate the results into the camera-ready.

W1/Q2.c: worst-case performance

Thank you for raising this point and helping strengthening the paper. We agree that the worst-case subset is relevant for understanding missing-modality robustness. We discuss this here and report worst-case robustness across multiple seeds. The discussion and results will be incorporated into the camera-ready.

Summary of findings: Worst-case performance is a difficult problem. A-MoM raises the average worst-case score by 0.80.8, but none of the models (A-MoM included) are consistently the best for more than a single dataset. Modality dropout greatly increases worst-case performance and shows A-MoM being either matched or highest-scoring on 3 out of 4 datasets.

We have obtained multiple seeds from the additional experiments in response to W3.

We report the results on the worst-case subsets below:

FusionDSECFMBMCubeSMUSESAvg
CHARM11.6 ± 2.340.8 ± 7.914.9 ± 7.631.4 ± 9.124.7 ± 3.6
CMNeXt1.4 ± 0.033.82.0 ± 0.90.19.3
GeminiFusion1.6 ± 0.440.1 ± 1.83.8 ± 3.04.0 ± 0.312.4 ± 0.9
MAGIC22.9 ± 1.942.0 ± 0.618.2 ± 2.829.6 ± 0.428.2 ± 0.9
StitchFusion17.7 ± 2.644.1 ± 0.823.0 ± 0.927.7 ± 1.628.1 ± 0.8
TokenFusion5.0 ± 2.045.2 ± 1.812.9 ± 1.44.2 ± 2.416.8 ± 1.0
A-MoM (ours)16.6 ± 3.045.0 ± 0.228.9 ± 0.925.5 ± 1.229.0 ± 0.9

On average, A-MoM raises the mean worst-case performance by 0.80.8, but it varies a lot for each dataset with MCubeS being the only dataset where A-MoM clearly stands out, and with a close second-best mean score on FMB. The high variance across datasets is consistent for all baselines–not just A-MoM–as the best-performing model in the worst case is different for each dataset. This shows that worst-case robustness is a difficult problem and that none of the models have solved it natively.

We also evaluate the worst-case performance with modality dropout (new experiment in response to reviewer CoHn/W1):

FusionDSECFMBMCubeSMUSESAvg
CHARM45.953.733.943.644.3
CMNeXt45.453.522.927.037.2
GeminiFusion46.555.929.838.642.7
MAGIC46.656.336.443.145.6
StitchFusion45.856.532.644.744.9
TokenFusion45.855.132.340.043.3
A-MoM (ours)44.157.736.544.845.8

We observe greatly improved worst-case robustness with modality dropout. A-MoM lags behind by −2.5-2.5 on DSEC, but otherwise maintains the highest score on 3 remaining datasets. We acknowledge that the margins are small on MCubeS and MUSES, but also highlight that A-MoM is more consistently high-scoring than the baselines. Specifically, StitchFusion is close to A-MoM by only −0.1-0.1 on MUSES, but trails by −1.2-1.2 on FMB and −3.9-3.9 on MCubeS. Likewise, MAGIC has a −0.1-0.1 difference to A-MoM on MCubeS, but −1.4-1.4 on FMB and −1.7-1.7 on MUSES.

W2/Q2.b: baseline scope

Thank you for the question. We appreciate that it deserves more clarification.

A-MoM is competing against complete methods and we clarify why below.

Here is what we mean by shared hyperparameters: all models are trained with the same learning rate, batch size. learning rate schedule, number of epochs, optimizer, weight decay, backbone (MiT and SwinV2), and data augmentations. We note that MiT is also used as the backbone in all of the baseline papers.

What we do not mean by shared hyperparameters:

  • Model-specific losses. We consider these to be a part of the fusion method and include them in our implementations.
  • Modality dropout. None of the baselines apply it. However, we are happy to include the modality dropout results in the camera-ready.
  • Missing-input handling mechanisms. Surprisingly, none of the baseline papers describe how this is handled. Neither in the paper nor in the official code. We use the setup described in Appendix F as the mechanism to handle missing inputs, but would have considered this model-specific if it had been described in the baseline papers.

We are happy to provide these clarifications in the camera-ready.

AuthorsOpenReview ↗

Rebuttal (2/3)

W3: modest improvements

We extend the results with multiple seeds to show that they, though modest, are consistent across trials. In summary, the findings are robust across multiple seeds and we will include the new results in the camera-ready.

We report results across 3 seeds below. Note that CMNeXt is unstable, resulting in scores close to 0 for the additional seeds on FMB and MUSES. We therefore report the single-seed case without standard deviation for CMNeXt on those datasets.

Robustness (Table 1):

FusionDSECFMBMCubeSMUSESAvg
CHARM47.5 ± 2.152.3 ± 1.632.6 ± 0.347.8 ± 2.345.0 ± 0.9
CMNeXt37.4 ± 1.348.524.2 ± 0.429.634.9
GeminiFusion43.0 ± 1.651.6 ± 0.130.0 ± 1.042.2 ± 0.841.7 ± 0.5
MAGIC49.0 ± 2.254.0 ± 0.436.9 ± 2.250.5 ± 0.947.6 ± 0.8
StitchFusion48.1 ± 1.054.5 ± 0.541.2 ± 0.551.3 ± 0.648.8 ± 0.3
TokenFusion46.5 ± 0.754.5 ± 0.733.0 ± 0.731.7 ± 0.641.4 ± 0.3
A-MoM49.2 ± 1.255.0 ± 0.242.9 ± 0.654.9 ± 0.550.5 ± 0.4

We show the difference in means between A-MoM and the best baseline for both the single-seed and 3-seed cases in the table below:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.4+0.5+1.7+2.4+1.5
3 seeds+0.2+0.5+1.7+3.6+1.7

We find that the 3-seed case is mostly comparable to the single-seed case and so does not change the conclusions from the single-seed case. A notable difference is that the A-MoM increase in robustness is higher by +1.2 on MUSES in the 3-seed case.

Overall (Table 4):

FusionSizeParams (M)DSECFMBMCubeSMUSESAvg
CHARMB243.666.0 ± 0.661.5 ± 2.643.6 ± 2.265.5 ± 2.359.2 ± 1.0
CMNeXtB258.166.3 ± 0.359.347.6 ± 0.959.658.2
GeminiFusionB240.766.2 ± 0.762.3 ± 2.450.6 ± 0.571.8 ± 0.462.7 ± 0.6
GeminiFusion$\dagger$B122.564.0 ± 0.460.2 ± 2.345.9 ± 1.865.1 ± 1.658.8 ± 0.8
MAGICB228.465.8 ± 0.564.4 ± 1.148.3 ± 3.067.7 ± 1.861.6 ± 0.9
StitchFusionB226.667.0 ± 0.565.0 ± 0.548.6 ± 0.670.8 ± 1.162.8 ± 0.4
TokenFusionB235.562.1 ± 0.763.0 ± 1.042.4 ± 1.754.9 ± 1.455.6 ± 0.6
A-MoMB226.667.5 ± 0.364.7 ± 0.349.9 ± 1.072.6 ± 0.263.7 ± 0.3

Differences in means for single-seed and 3-seed cases, with and without parameter matching:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.2-0.1-1.2+1.1+0.7
3 seeds+0.5-0.3-0.7+0.8+0.9
1 seed param. matched+0.2-0.1+1.5+1.3+0.7
3 seeds param. matched+0.5-0.3+1.3+1.8+0.9

Similar as for robustness, we find the 3-seed case to be comparable to the single-seed case and to not change the existing conclusions.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q2.b: corrupted or spatially misaligned modalities

That is a very good question. We discuss it here and present new results on spatial misalignment, which we will include in the camera-ready.

Corrupted. We have explored this in the adverse weather and lighting conditions study in Table 2 where some sensors experience degraded signals due to e.g. fog, snow and night-time lighting. We find A-MoM to be highly robust in those scenarios. Particularly in fog and snow with respective increases of +7.6+7.6 and +1.2+1.2 over the strongest baselines, showing robustness to partial outage.

Spatial misalignment. Considering the token-level design of A-MoM, we think this is an interesting experiment. We evaluate at test-time how the models perform under spatial misalignment by translating each modality separately by a random offset from the [−16,16][-16, 16] interval. We seed the random number generator based on sample index and modality. This ensures a reproducible offset sampling such that all models are evaluated using the same offsets.

Overall scores under spatial misalignment:

FusionDSECFMBMCubeSMUSESAvg
CHARM52.7 ± 0.350.8 ± 2.134.8 ± 1.058.0 ± 2.349.1 ± 0.8
CMNeXt51.6 ± 0.845.335.946.844.9
GeminiFusion52.5 ± 0.449.4 ± 2.636.6 ± 0.260.5 ± 0.449.8 ± 0.7
MAGIC52.7 ± 0.248.8 ± 1.736.8 ± 1.157.8 ± 1.849.0 ± 0.7
StitchFusion52.3 ± 0.651.9 ± 0.536.6 ± 0.358.8 ± 1.549.9 ± 0.4
TokenFusion49.7 ± 0.750.9 ± 0.831.3 ± 2.044.5 ± 0.444.1 ± 0.6
A-MoM (ours)52.8 ± 0.351.6 ± 0.337.9 ± 1.161.6 ± 1.051.0 ± 0.4

All models are affected by the misalignment, but we notice that A-MoM still outperform the baselines. A-MoM's average score is +1.1+1.1 higher than the best baseline, StitchFusion, and A-MoM is highest-scoring on 3 out of 4 datasets with a small difference of −0.3-0.3 on FMB. Interestingly, we find that the token-level design of A-MoM does not cause it to degrade more than the baselines under spatial misalignment.

Reviewer_CoHnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes A-MoM, a simple architecture for handling missing modalities in multimodal semantic segmentation, without additional losses or training strategies. It uses shared weights and lightweight modality-specific adapters to mix information across available modalities, with self-modality dominance to limit unnecessary mixing. Experiments on DSEC, FMB, MCubeS, and MUSES show consistent gains, improving average robustness by 1.5 mIoU over StitchFusion. The method also approaches specialist performance on MCubeS and is evaluated across model scales, backbones, and inference costs.

Critical strengths
  1. A useful evaluation protocol. Testing every non-empty modality subset and comparing against 98 specialist models exposes how poorly existing fusion methods handle missing modalities. The results are striking, with CMNeXt achieving only 0.1 mIoU on RGB-only MUSES. The robustness-to-specialization ratio is a useful metric for future work.

  2. A simple method with broad, fair evaluation. All methods use the same training setup, and the paper includes parameter-matched comparisons, inference costs, four datasets, multiple backbone scales, and SwinV2 transfer. The ablation also shows that initialization is critical: Kaiming initialization reduces the robustness score by 12.9 points.

Critical weaknesses
  1. No modality-dropout baseline. The paper does not compare A-MoM against methods trained with random modality dropout. It remains unclear whether A-MoM retains its advantage when other methods use this common strategy.

  2. Missing specialized methods. Methods designed for missing-modality robustness, such as Reza et al. (2025), Fourier Prompt Tuning (Liu et al., 2024), and SimMLM, are cited but not evaluated.

  3. Potentially unfair baseline inputs. Fixed-modality baselines receive zero-filled inputs for missing modalities, despite never seeing such inputs during training. This may disadvantage the baselines and affect the fairness of the comparison.

  4. Small performance margins. A-MoM improves average robustness by 1.5 points and all-modality performance by 0.7 points. Several dataset-level gains are marginal, and A-MoM does not consistently outperform the strongest baseline.

  5. No statistical variance. The paper reports no results across random seeds or standard deviations, making it difficult to determine whether the small performance gains are statistically significant.

  6. Robustness mainly comes from initialization. Removing the p=0.9p=0.9 initialization reduces the robustness score from 51.9 to 39.0, while the Eq. 2-only variant still achieves 48.0. This suggests that self-modality dominance may contribute more to robustness than the proposed token-level mixing mechanism.

  7. The token-level fusion claim is not validated. The paper does not analyze learned weights or compare token-level fusion with image-level or channel-level gating. It therefore remains unclear whether token-level mixing is necessary or provides a meaningful advantage.

Questions for authors
  1. How does A-MoM compare with StitchFusion, GeminiFusion, and MAGIC when all methods use modality dropout? Are the gains complementary?
  2. How are missing modalities handled in each baseline? Would alternatives to zero-filling, such as learned missing-modality tokens, reduce the performance gap?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

We thank the reviewer for their time and effort in reviewing our work. We are grateful for the praise of the evaluation and simplicity of the method. We address each point below, and we will incorporate all of the results into the camera-ready.

W1/Q1: modality dropout

Thank you for asking this question. It is good feedback and makes the paper stronger. We include the modality dropout results below and we will include the experiment/results in the camera-ready.

We find that A-MoM keeps its advantage over the baselines for both robustness and overall scores.

Robustness:

FusionDSECFMBMCubeSMUSESAvg
CHARM58.758.541.355.253.4
CMNeXt57.859.137.938.548.3
GeminiFusion59.459.637.357.053.3
MAGIC59.860.443.756.355.1
StitchFusion59.560.842.558.255.2
TokenFusion56.457.539.451.351.1
A-MoM59.661.445.758.656.3

We observe a small drop on DSEC by −0.2-0.2 to MAGIC, but otherwise A-MoM maintains consistently high robustness across the remaining 3 datasets with +0.6+0.6 on FMB, +2.0+2.0 on MCubeS, and +0.4+0.4 on MUSES. Together, A-MoM raises the average robustness with modality dropout by +1.1+1.1.

Overall:

FusionSizeParamsDSECFMBMCubeSMUSESAvg
CHARMB243.6M64.264.643.966.059.7
CMNeXtB258.1M64.664.445.949.256.0
GeminiFusionB240.7M64.465.642.469.860.5
GeminiFusion$\dagger$B122.5M61.662.741.166.157.9
MAGICB228.4M64.864.347.466.960.8
StitchFusionB226.6M65.164.446.069.361.2
TokenFusionB235.5M60.161.841.760.356.0
A-MoMB226.6M65.465.248.670.762.5

The overall scores with modality dropout supports A-MoM. A-MoM increases the MCubeS score by +1.2 and +0.9 on MUSES. Consistent with Table 4, A-MoM only lags behind GeminiFusion on FMB, but surpasses all baselines by at least +0.3 when matching parameter budgets.

W2: additional baselines

We report results on Fourier Prompt Tuning (Liu et al., 2024) below and will include the additional results in the camera-ready. Thank you for the suggestion.

We don't think that Reza et al. and SimMLM are comparable baselines for this work. We seek to train a model once (L037–040) and show that it is robust to missing modalities. Given mm modalities, Reza et al. train a total of 2m−12^m -1 models (one for every modality subset) with parameter-efficient fine-tuning. SimMLM trains mm models (one for every modality) and mixes the predictions of each model in a late-fusion manner. As such, we consider them incomparable as baselines, because they do not adhere to the constraint of a single trainable and deployable model.

Robustness:

MethodDSECFMBMCubeSMUSESAvg
FPT53.555.037.438.346.1
A-MoM50.554.942.654.350.6

FPT improves robustness on DSEC (+3.0) and slightly on FMB (+0.1), but it is inconsistent with big drops in robustness on MCubeS (-5.2) and MUSES (-16.0). This is despite FPT being trained with a specialized form of modality dropout to improve its robustness. (The results here are for A-MoM without modality dropout.) We find A-MoM to be more robust to missing modalities.

Overall:

MethodParams (M)DSECFMBMCubeSMUSESAvg
FPT103.060.260.042.850.253.3
A-MoM26.667.564.449.672.963.6

A-MoM performs consistently better than FPT across all datasets with increases ranging from +4.4 to +22.7, yielding an average increase of +10.3.

AuthorsOpenReview ↗

Rebuttal (2/4)

W3/Q2: baseline inputs

The modality dropout results (see reply to W1/Q1) ensure that baselines have seen zero-filled inputs during training, and A-MoM still maintains its advantage.

Regarding learned missing-modality tokens: all models in this paper use the same embedding function (see code/src/amom/nn/embedding.py in the supplementary material). Namely, each modality is embedded using a modality-specific convolution with a bias term. This means that the bias term has a very similar role to a learned missing-modality token for zero-filled inputs, as w∗x+b=bw \ast x + b = b for x=0x = 0, and so bb acts similarly to a learned token.

AuthorsOpenReview ↗

Rebuttal (3/4)

W4/W5: small performance margins and multiple seeds

Thank you for the feedback. Multiple seeds are an excellent point and we report performance across 3 seeds further below. In summary, the findings are robust across multiple seeds and we will include the new results in the camera-ready.

Regarding the marginal improvements, we would like to clarify the -1.2% change on MCubeS in Table 4 (overall case) only applies in the setting where baselines are not parameter-matched. GeminiFusion adds 14.1M (53% increase) parameters over A-MoM. A direct comparison is unfair as we know that larger models tend to lead to higher performance. We therefore control for parameter budgets and show that A-MoM is the best-performing model by +1.5 on MCubeS in this setting (L364-372), resulting in A-MoM outperforming the strongest baseline on 3 out of 4 datasets in the overall case under controlled parameter budgets.

We report results across 3 seeds below. Note that CMNeXt is unstable, resulting in scores close to 0 for the additional seeds on FMB and MUSES. We therefore report the single-seed case without standard deviation for CMNeXt on those datasets.

Robustness (Table 1):

FusionDSECFMBMCubeSMUSESAvg
CHARM47.5 ± 2.152.3 ± 1.632.6 ± 0.347.8 ± 2.345.0 ± 0.9
CMNeXt37.4 ± 1.348.524.2 ± 0.429.634.9
GeminiFusion43.0 ± 1.651.6 ± 0.130.0 ± 1.042.2 ± 0.841.7 ± 0.5
MAGIC49.0 ± 2.254.0 ± 0.436.9 ± 2.250.5 ± 0.947.6 ± 0.8
StitchFusion48.1 ± 1.054.5 ± 0.541.2 ± 0.551.3 ± 0.648.8 ± 0.3
TokenFusion46.5 ± 0.754.5 ± 0.733.0 ± 0.731.7 ± 0.641.4 ± 0.3
A-MoM49.2 ± 1.255.0 ± 0.242.9 ± 0.654.9 ± 0.550.5 ± 0.4

We show the difference in means between A-MoM and the best baseline for both the single-seed and 3-seed cases in the table below:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.4+0.5+1.7+2.4+1.5
3 seeds+0.2+0.5+1.7+3.6+1.7

We find that the 3-seed case is mostly comparable to the single-seed case and so does not change the conclusions from the single-seed case. A notable difference is that the A-MoM increase in robustness is higher by +1.2 on MUSES in the 3-seed case.

Overall (Table 4):

FusionSizeParams (M)DSECFMBMCubeSMUSESAvg
CHARMB243.666.0 ± 0.661.5 ± 2.643.6 ± 2.265.5 ± 2.359.2 ± 1.0
CMNeXtB258.166.3 ± 0.359.347.6 ± 0.959.658.2
GeminiFusionB240.766.2 ± 0.762.3 ± 2.450.6 ± 0.571.8 ± 0.462.7 ± 0.6
GeminiFusion$\dagger$B122.564.0 ± 0.460.2 ± 2.345.9 ± 1.865.1 ± 1.658.8 ± 0.8
MAGICB228.465.8 ± 0.564.4 ± 1.148.3 ± 3.067.7 ± 1.861.6 ± 0.9
StitchFusionB226.667.0 ± 0.565.0 ± 0.548.6 ± 0.670.8 ± 1.162.8 ± 0.4
TokenFusionB235.562.1 ± 0.763.0 ± 1.042.4 ± 1.754.9 ± 1.455.6 ± 0.6
A-MoMB226.667.5 ± 0.364.7 ± 0.349.9 ± 1.072.6 ± 0.263.7 ± 0.3

Differences in means for single-seed and 3-seed cases, with and without parameter matching:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.2-0.1-1.2+1.1+0.7
3 seeds+0.5-0.3-0.7+0.8+0.9
1 seed param. matched+0.2-0.1+1.5+1.3+0.7
3 seeds param. matched+0.5-0.3+1.3+1.8+0.9

Similar as for robustness, we find the 3-seed case to be comparable to the single-seed case and to not change the existing conclusions.

Reviewer_Ke61OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposed a token- and modality-wise multimodality fusion for semantic segmentation. By allowing information exchange after each shared transformer block with dedicated scorer, the model shows competitive performance with improved efficiency and generalizability for inference with unavailable modalities.

Critical strengths

1.Clear ablation study, mathematical proof and qualitative results showing reasonable argument for the core designs of the paper.

2.Novel view of measuring the generalizability for inference scenario with incomplete modalities in Table 3, showing model’s high robustness. Also, comprehensive experiments on datasets with various modalities, showing the versatility of the model.

Critical weaknesses

1.The percentage measurement in Table 3 only shows the relative performance compared with trained specialists, but not the absolute performance. This remains great concern, since high relative performance in percentage can occur while the actual performance is bad, which is exactly observed for MCubeS dataset combining Table 3 with Table 11 for settings with only A or D single modality. This weakens the robustness statement.

2.The performance improvement on 4 datasets remains marginal (+0.2%, -0.1%, -1.2%, +1.1%, respectfully) compared with other SOTA methods, that might not have statistical significancy.

Questions for authors

1.As W1, could the authors clarify how can the relative robustness translate into reliable performance under the scenario with limited available modalities? What’s the results (relative and absolute) under this inference setting on other datasets other than MCubeS?

2.As W2, could the author provide results across multiple seeds to establish the performance improvements?

3.Regarding Q2 and respecting the improved inference time, could the author give more evidence and results in terms of the robustness-accuracy-efficiency tradeoff?

5 comments
AuthorsOpenReview ↗

Request for clarification regarding Q3

Thank you for your time and effort in reviewing our work. We are conducting additional experiments and will give a detailed reply as soon as they are completed.

In the meantime, we are unsure how to interpret Q3. We kindly ask if the reviewer can make the question more concrete or elaborate on the intention?

AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for their time and effort in reviewing our work, and we thank for the praise of the thorough core design evaluations and novelty of robustness vs specialization study (Table 3). We believe our responses below will alleviate many of their concerns.

Q1/W1: relative scores in Table 3

To clarify, we report absolute robustness in Table 1 and in the dataset-specific tables in Appendix E. Please note that Table 3 is not a standalone robustness result, but a complementary view of robustness as we briefly expand upon below.

The purpose of Table 3 is to answer how close a model is to its own specialist performance (L340-343). We think relative scoring is appropriate to answer that question. The purpose is not to answer how robust a model is in absolute terms. That is a different question and we answer it in Table 1, showing absolute robustness increases from 0.4 to 2.4 (average improvement of 1.5).

AuthorsOpenReview ↗

Rebuttal (2/3)

Q2/W2: marginal improvements and seeds

Thank you for the feedback. Multiple seeds are an excellent point and we report performance across 3 seeds further below. In summary, the findings are robust across multiple seeds and we will include the new results in the camera-ready.

Regarding the marginal improvements, we would like to clarify two points:

  1. Missing-modality robustness is our primary objective and the mentioned scores in W2 are the full-modality scores in Table 4. We report those to support that A-MoM remains a strong model in the full-modality case. As such, while the Table 4 results are relevant, it is the results in Table 1 that are the most important to the primary objective.

  2. Fairness of comparison. The -1.2% change on MCubeS only applies in the setting where baselines are not parameter-matched. GeminiFusion adds 14.1M (53% increase) parameters over A-MoM. A direct comparison is unfair as we know that larger models tend to lead to higher performance. We therefore control for parameter budgets and show that A-MoM is the best-performing model by +1.5 on MCubeS in this setting (L364-372).

We report results across 3 seeds below. Note that CMNeXt is unstable, resulting in scores close to 0 for the additional seeds on FMB and MUSES. We therefore report the single-seed case without standard deviation for CMNeXt on those datasets.

Robustness (Table 1):

FusionDSECFMBMCubeSMUSESAvg
CHARM47.5 ± 2.152.3 ± 1.632.6 ± 0.347.8 ± 2.345.0 ± 0.9
CMNeXt37.4 ± 1.348.524.2 ± 0.429.634.9
GeminiFusion43.0 ± 1.651.6 ± 0.130.0 ± 1.042.2 ± 0.841.7 ± 0.5
MAGIC49.0 ± 2.254.0 ± 0.436.9 ± 2.250.5 ± 0.947.6 ± 0.8
StitchFusion48.1 ± 1.054.5 ± 0.541.2 ± 0.551.3 ± 0.648.8 ± 0.3
TokenFusion46.5 ± 0.754.5 ± 0.733.0 ± 0.731.7 ± 0.641.4 ± 0.3
A-MoM49.2 ± 1.255.0 ± 0.242.9 ± 0.654.9 ± 0.550.5 ± 0.4

We show the difference in means between A-MoM and the best baseline for both the single-seed and 3-seed cases in the table below:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.4+0.5+1.7+2.4+1.5
3 seeds+0.2+0.5+1.7+3.6+1.7

We find that the 3-seed case is mostly comparable to the single-seed case and so does not change the conclusions from the single-seed case. A notable difference is that the A-MoM increase in robustness is higher by +1.2 on MUSES in the 3-seed case.

Overall (Table 4):

FusionSizeParams (M)DSECFMBMCubeSMUSESAvg
CHARMB243.666.0 ± 0.661.5 ± 2.643.6 ± 2.265.5 ± 2.359.2 ± 1.0
CMNeXtB258.166.3 ± 0.359.347.6 ± 0.959.658.2
GeminiFusionB240.766.2 ± 0.762.3 ± 2.450.6 ± 0.571.8 ± 0.462.7 ± 0.6
GeminiFusion$\dagger$B122.564.0 ± 0.460.2 ± 2.345.9 ± 1.865.1 ± 1.658.8 ± 0.8
MAGICB228.465.8 ± 0.564.4 ± 1.148.3 ± 3.067.7 ± 1.861.6 ± 0.9
StitchFusionB226.667.0 ± 0.565.0 ± 0.548.6 ± 0.670.8 ± 1.162.8 ± 0.4
TokenFusionB235.562.1 ± 0.763.0 ± 1.042.4 ± 1.754.9 ± 1.455.6 ± 0.6
A-MoMB226.667.5 ± 0.364.7 ± 0.349.9 ± 1.072.6 ± 0.263.7 ± 0.3

Differences in means for single-seed and 3-seed cases, with and without parameter matching:

A-MoM - best baselineDSECFMBMCubeSMUSESAvg
1 seed+0.2-0.1-1.2+1.1+0.7
3 seeds+0.5-0.3-0.7+0.8+0.9
1 seed param. matched+0.2-0.1+1.5+1.3+0.7
3 seeds param. matched+0.5-0.3+1.3+1.8+0.9

Similar as for robustness, we find the 3-seed case to be comparable to the single-seed case and to not change the existing conclusions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of author-reviewer discussion

We thank the AC and reviewers for their time and effort. We believe the paper is stronger as a result, and we will be sure to incorporate all the additional results and clarifications into the camera-ready.

We present a summary of the discussions here for convenience. Please see the responses to each reviewer for full details.

Reviewer Ke61

Relative scores. The reviewer expressed concerns about using relative scores instead of absolute scores to measure robustness. We clarified the purpose of the relative scores in Table 3 and that they are complementary to the primary/absolute robustness scores reported in Table 1.

Marginal improvements and seeds. Some of the improvements are small. We extended the main results (Table 1 and 4) with comparisons across 3 seeds for A-MoM and all 6 baselines, finding that the improvements are consistent with multiple seeds. We also clarified that the robustness results are presented in Table 1, whereas the reviewer commented on the relevant, but secondary results in Table 4.

Robustness-accuracy-efficiency tradeoff. It is unclear to us how to interpret the question. We asked the reviewer to clarify, but have not received a reply. We would otherwise have been happy to provide a response to a clarified question.

Reviewer CoHn

Modality dropout. The reviewer asked how A-MoM performs when using modality dropout during training. We retrained A-MoM and the baselines under this setting and found that A-MoM maintains its advantage: +1.1+1.1 average increase in robustness and +1.3+1.3 average increase in overall score.

Additional baselines. The reviewer asked for additional baselines: FPT, SimMLM, and Reza et al. We presented additional results for FPT, finding that A-MoM increases the average robustness and overall scores by +4.5+4.5 and +10.3+10.3, respectively. We also clarified how SimMLM and Reza et al. are incomparable to this work, because they do not adhere to the constraint of a single trainable model (L037–040).

Baseline inputs. The reviewer questioned the fairness of using zero-filled inputs for the fixed-modality baselines, since such inputs have not been seen during training. The modality dropout models include those inputs during training, and A-MoM maintains its advantage.

Small performance margins and multiple seeds. Similarly to reviewer Ke61, some of the improvements are small. We extended the main results (Table 1 and 4) with comparisons across 3 seeds for A-MoM and all 6 baselines, finding that the improvements are consistent with multiple seeds.

Source of robustness. The reviewer found that the ablation study shows the initialization to contribute more to robustness than the token-level scoring. It is unclear to us why that is considered a weakness as both parts contribute positively and meaningfully, even if initialization contributes more. We asked the reviewer to clarify the position, but we have not received a reply.

Validation of token-level claim. The reviewer asked for more evidence to validate the token-level motivation. We provided 4 new ablations with image-level fusion instead of token-level fusion. A-MoM (with token-level fusion) improves robustness and overall scores over all 4 new ablations.

Reviewer pRFT

Worst-case performance. The reviewer asked for worst-case subset results. We provided those results and showed that this (without modality dropout) is an unsolved problem as not a single model is consistently high-performing across datasets in this setting. We also showed that this changes when applying modality dropout, with A-MoM matching or exceeding all baselines on 3 out of 4 dataset. The reviewer expressed that the question was not fully answered.

Baseline scope. The reviewer asked for clarifications about which parts of the baseline methods are retained. We provided these clarifications and our reasoning for why we think the comparison is fair. The reviewer expressed that the question was not fully answered, but did not specify why.

Modest improvements. Similarly to reviewer Ke61 and CoHn, some of the improvements are small. We extended the main results (Table 1 and 4) with comparisons across 3 seeds for A-MoM and all 6 baselines, finding that the improvements are consistent with multiple seeds. The reviewer expressed the answer as satisfactory.

Corrupted or spatially misaligned modalities. The reviewer asked how A-MoM performs under corruption or spatial misalignment. For corruption, we referred to the adverse weather and lightning conditions study in Table 2, with particularly strong robustness in fog and snow. For spatial misalignment, we evaluated this with shifted inputs at inference time, showing that A-MoM increases the average overall score by +1.1+1.1 over the baselines in this setting. The reviewer expressed the answer as satisfactory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.