- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Adapted Mixture of Modalities (A-MoM), an architectural approach to multimodal semantic segmentation under missing modalities. It combines token-level modality weighting, an initialization favoring within-modality information retention, and lightweight pairwise modality adapters. Experiments cover four datasets, multiple backbone scales, and both SegFormer and SwinV2.
Critical strengths
-
Comprehensive eval.
-
Competitive full-modality performance + relatively small parameter and inference-time overhead.
Critical weaknesses
-
Average robustness does not fully characterize failure resilience. The proposed method achieves the best average scores, but it does not consistently perform best in difficult individual subsets. For example, its depth-only DSEC score is 19.6 versus 25.0 for MAGIC, and its event-only MUSES score is 24.2 versus 35.9 for CHARM. The subset results are provided, but the discussion would benefit from distinguishing average performance from worst-case robustness and from performance under realistic sensor-failure distributions.
-
The scope of the baseline comparison needs clarification. Using common hyperparameters helps isolate architectural differences, but it does not necessarily represent the strongest configuration of each baseline. It is unclear whether all method-specific losses, modality-dropout procedures, and missing-input handling mechanisms are retained. This distinction matters for interpreting the results as either a controlled architectural comparison or a comparison against complete competing methods.
-
Several improvements over the strongest baselines are modest.
Questions for authors
-
Responses to the weaknesses.
-
And also:
- Which method-specific training components are retained for each baseline?
- How does A-MoM perform when a modality is corrupted or spatially misaligned but remains present?
- Can the authors report worst-subset performance and variability across multiple training seeds?
5 comments
Rebuttal (2/3)
W3: modest improvements
We extend the results with multiple seeds to show that they, though modest, are consistent across trials. In summary, the findings are robust across multiple seeds and we will include the new results in the camera-ready.
We report results across 3 seeds below. Note that CMNeXt is unstable, resulting in scores close to 0 for the additional seeds on FMB and MUSES. We therefore report the single-seed case without standard deviation for CMNeXt on those datasets.
Robustness (Table 1):
| Fusion | DSEC | FMB | MCubeS | MUSES | Avg |
|---|---|---|---|---|---|
| CHARM | 47.5 ± 2.1 | 52.3 ± 1.6 | 32.6 ± 0.3 | 47.8 ± 2.3 | 45.0 ± 0.9 |
| CMNeXt | 37.4 ± 1.3 | 48.5 | 24.2 ± 0.4 | 29.6 | 34.9 |
| GeminiFusion | 43.0 ± 1.6 | 51.6 ± 0.1 | 30.0 ± 1.0 | 42.2 ± 0.8 | 41.7 ± 0.5 |
| MAGIC | 49.0 ± 2.2 | 54.0 ± 0.4 | 36.9 ± 2.2 | 50.5 ± 0.9 | 47.6 ± 0.8 |
| StitchFusion | 48.1 ± 1.0 | 54.5 ± 0.5 | 41.2 ± 0.5 | 51.3 ± 0.6 | 48.8 ± 0.3 |
| TokenFusion | 46.5 ± 0.7 | 54.5 ± 0.7 | 33.0 ± 0.7 | 31.7 ± 0.6 | 41.4 ± 0.3 |
| A-MoM | 49.2 ± 1.2 | 55.0 ± 0.2 | 42.9 ± 0.6 | 54.9 ± 0.5 | 50.5 ± 0.4 |
We show the difference in means between A-MoM and the best baseline for both the single-seed and 3-seed cases in the table below:
| A-MoM - best baseline | DSEC | FMB | MCubeS | MUSES | Avg |
|---|---|---|---|---|---|
| 1 seed | +0.4 | +0.5 | +1.7 | +2.4 | +1.5 |
| 3 seeds | +0.2 | +0.5 | +1.7 | +3.6 | +1.7 |
We find that the 3-seed case is mostly comparable to the single-seed case and so does not change the conclusions from the single-seed case. A notable difference is that the A-MoM increase in robustness is higher by +1.2 on MUSES in the 3-seed case.
Overall (Table 4):
| Fusion | Size | Params (M) | DSEC | FMB | MCubeS | MUSES | Avg |
|---|---|---|---|---|---|---|---|
| CHARM | B2 | 43.6 | 66.0 ± 0.6 | 61.5 ± 2.6 | 43.6 ± 2.2 | 65.5 ± 2.3 | 59.2 ± 1.0 |
| CMNeXt | B2 | 58.1 | 66.3 ± 0.3 | 59.3 | 47.6 ± 0.9 | 59.6 | 58.2 |
| GeminiFusion | B2 | 40.7 | 66.2 ± 0.7 | 62.3 ± 2.4 | 50.6 ± 0.5 | 71.8 ± 0.4 | 62.7 ± 0.6 |
| GeminiFusion$\dagger$ | B1 | 22.5 | 64.0 ± 0.4 | 60.2 ± 2.3 | 45.9 ± 1.8 | 65.1 ± 1.6 | 58.8 ± 0.8 |
| MAGIC | B2 | 28.4 | 65.8 ± 0.5 | 64.4 ± 1.1 | 48.3 ± 3.0 | 67.7 ± 1.8 | 61.6 ± 0.9 |
| StitchFusion | B2 | 26.6 | 67.0 ± 0.5 | 65.0 ± 0.5 | 48.6 ± 0.6 | 70.8 ± 1.1 | 62.8 ± 0.4 |
| TokenFusion | B2 | 35.5 | 62.1 ± 0.7 | 63.0 ± 1.0 | 42.4 ± 1.7 | 54.9 ± 1.4 | 55.6 ± 0.6 |
| A-MoM | B2 | 26.6 | 67.5 ± 0.3 | 64.7 ± 0.3 | 49.9 ± 1.0 | 72.6 ± 0.2 | 63.7 ± 0.3 |
Differences in means for single-seed and 3-seed cases, with and without parameter matching:
| A-MoM - best baseline | DSEC | FMB | MCubeS | MUSES | Avg |
|---|---|---|---|---|---|
| 1 seed | +0.2 | -0.1 | -1.2 | +1.1 | +0.7 |
| 3 seeds | +0.5 | -0.3 | -0.7 | +0.8 | +0.9 |
| 1 seed param. matched | +0.2 | -0.1 | +1.5 | +1.3 | +0.7 |
| 3 seeds param. matched | +0.5 | -0.3 | +1.3 | +1.8 | +0.9 |
Similar as for robustness, we find the 3-seed case to be comparable to the single-seed case and to not change the existing conclusions.
Rebuttal (3/3)
Q2.b: corrupted or spatially misaligned modalities
That is a very good question. We discuss it here and present new results on spatial misalignment, which we will include in the camera-ready.
Corrupted. We have explored this in the adverse weather and lighting conditions study in Table 2 where some sensors experience degraded signals due to e.g. fog, snow and night-time lighting. We find A-MoM to be highly robust in those scenarios. Particularly in fog and snow with respective increases of and over the strongest baselines, showing robustness to partial outage.
Spatial misalignment. Considering the token-level design of A-MoM, we think this is an interesting experiment. We evaluate at test-time how the models perform under spatial misalignment by translating each modality separately by a random offset from the interval. We seed the random number generator based on sample index and modality. This ensures a reproducible offset sampling such that all models are evaluated using the same offsets.
Overall scores under spatial misalignment:
| Fusion | DSEC | FMB | MCubeS | MUSES | Avg |
|---|---|---|---|---|---|
| CHARM | 52.7 ± 0.3 | 50.8 ± 2.1 | 34.8 ± 1.0 | 58.0 ± 2.3 | 49.1 ± 0.8 |
| CMNeXt | 51.6 ± 0.8 | 45.3 | 35.9 | 46.8 | 44.9 |
| GeminiFusion | 52.5 ± 0.4 | 49.4 ± 2.6 | 36.6 ± 0.2 | 60.5 ± 0.4 | 49.8 ± 0.7 |
| MAGIC | 52.7 ± 0.2 | 48.8 ± 1.7 | 36.8 ± 1.1 | 57.8 ± 1.8 | 49.0 ± 0.7 |
| StitchFusion | 52.3 ± 0.6 | 51.9 ± 0.5 | 36.6 ± 0.3 | 58.8 ± 1.5 | 49.9 ± 0.4 |
| TokenFusion | 49.7 ± 0.7 | 50.9 ± 0.8 | 31.3 ± 2.0 | 44.5 ± 0.4 | 44.1 ± 0.6 |
| A-MoM (ours) | 52.8 ± 0.3 | 51.6 ± 0.3 | 37.9 ± 1.1 | 61.6 ± 1.0 | 51.0 ± 0.4 |
All models are affected by the misalignment, but we notice that A-MoM still outperform the baselines. A-MoM's average score is higher than the best baseline, StitchFusion, and A-MoM is highest-scoring on 3 out of 4 datasets with a small difference of on FMB. Interestingly, we find that the token-level design of A-MoM does not cause it to degrade more than the baselines under spatial misalignment.
Rebuttal (1/3)
We thank the reviewer for their time and effort in reviewing our work. We appreciate the encouraging assessment of our contributions. We address the points below and will incorporate the results into the camera-ready.
W1/Q2.c: worst-case performance
Thank you for raising this point and helping strengthening the paper. We agree that the worst-case subset is relevant for understanding missing-modality robustness. We discuss this here and report worst-case robustness across multiple seeds. The discussion and results will be incorporated into the camera-ready.
Summary of findings: Worst-case performance is a difficult problem. A-MoM raises the average worst-case score by 0.8, but none of the models (A-MoM included) are consistently the best for more than a single dataset. Modality dropout greatly increases worst-case performance and shows A-MoM being either matched or highest-scoring on 3 out of 4 datasets.
We have obtained multiple seeds from the additional experiments in response to W3.
We report the results on the worst-case subsets below:
On average, A-MoM raises the mean worst-case performance by 0.8, but it varies a lot for each dataset with MCubeS being the only dataset where A-MoM clearly stands out, and with a close second-best mean score on FMB. The high variance across datasets is consistent for all baselines–not just A-MoM–as the best-performing model in the worst case is different for each dataset. This shows that worst-case robustness is a difficult problem and that none of the models have solved it natively.
We also evaluate the worst-case performance with modality dropout (new experiment in response to reviewer CoHn/W1):
We observe greatly improved worst-case robustness with modality dropout. A-MoM lags behind by −2.5 on DSEC, but otherwise maintains the highest score on 3 remaining datasets. We acknowledge that the margins are small on MCubeS and MUSES, but also highlight that A-MoM is more consistently high-scoring than the baselines. Specifically, StitchFusion is close to A-MoM by only −0.1 on MUSES, but trails by −1.2 on FMB and −3.9 on MCubeS. Likewise, MAGIC has a −0.1 difference to A-MoM on MCubeS, but −1.4 on FMB and −1.7 on MUSES.
W2/Q2.b: baseline scope
Thank you for the question. We appreciate that it deserves more clarification.
A-MoM is competing against complete methods and we clarify why below.
Here is what we mean by shared hyperparameters: all models are trained with the same learning rate, batch size. learning rate schedule, number of epochs, optimizer, weight decay, backbone (MiT and SwinV2), and data augmentations. We note that MiT is also used as the backbone in all of the baseline papers.
What we do not mean by shared hyperparameters:
We are happy to provide these clarifications in the camera-ready.