- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method for training multi-modal fusion networks that is robust to missing inputs from some of the sensing modes (incomplete multi-modal training IMT). The method is couched in the language of genetic programming, but essentially involves constructing a missing data proxy via the method of - when data is missing in one modality - finding similar samples in the modalities that are present, and then combining the corresponding data of those samples in the missing-modalities into an interpolated “proxy” sample used to fill in for the missing data during training. The proposed method is then compared experimentally to other IMT approaches and found to have favorable performance.
Critical strengths
- The general approach of proxy data generation via genetic-style combination is an interesting approach to the problem.
- Internal hyperparameters for attentive aggregation seem to be learned from data instead of relying on fixed approaches.
Critical weaknesses
- It appears that each evaluation randomly masked missing data at training time for that evaluation, instead of masking once randomly and using the same training set for each method. Maybe this makes no difference, but seems to leave room for unequal evaluation of the methods.
- The method does not appear to be tested on actual missing modality data, all missing pairs are constructed. In theory there is no difference between theory and practice, in practice there often is.
Questions for authors
- I am curious how the method would compare a more simplistic data augmentation of the missing-modality data for example by selecting the nearest neighbor in the present-modality and adding random data augmentation.
- I am also curious how the method would compare to only using fully-present data for training the fusion network and then falling back to single-modality networks when a modality is missing.
2 comments
Follow-up to Reviewer oRx9
Dear Reviewer oRx9,
We are following up on our detailed rebuttal above. We would appreciate hearing whether it addresses your concerns about shared missing masks and realistic missingness, as well as your questions about the two alternatives you suggested. Below, we briefly recap our earlier response and note what has changed in the revised PDF.
1. What we addressed in our initial rebuttal
All methods use the same data split and the same missing-modality assignment for each sample within an experiment. For example, if HSI is masked for an Augsburg test sample, every method evaluates that sample without HSI.
The two Augsburg evaluations address different questions. The spatially disjoint evaluation tests performance in a held-out part of the scene, with no shared pixels between training and test patches. The simulated coverage-gap evaluation changes which modalities are available at test time: without retraining either model, we masked HSI in one contiguous area and SAR in another within that held-out region. Under the shared gap masks, GenFuse achieved 83.17% OA versus 78.71% for REDEEM, with higher OA in both missing directions (Table 3; Appendices B.2–B.3, Table 14).
We also tested both alternatives you suggested: nearest-neighbor filling with training-time augmentation and fusion trained on complete pairs with a single-modality fallback at test time. At 70% probabilistic missingness on Augsburg, their overall OA was 60.21% and 73.40%, respectively, compared with 78.00% for GenFuse. GenFuse also achieved higher overall OA than both alternatives on Trento (Appendix Table 12).
2. What is new in the revised PDF
We expanded the Stage 2 discussion to explain why we recommend keeping the encoders trainable and the feature banks frozen by default (Section 4.3; Appendix B.4). Refreshing the banks after each epoch improved OA on the spatially disjoint split and lowered it on the random split. We therefore keep the banks frozen by default and discuss refreshing as an option for the spatially held-out setting we tested.
We also expanded the per-class analysis across three Augsburg splits (Appendix B.5). Residential Area accounts for 38.8% of test pixels, and GenFuse achieves 85.37% mean accuracy on this class versus 64.36% for REDEEM. This improvement carries substantial weight in overall pixel accuracy, whereas average class accuracy gives every class equal weight.
Finally, we streamlined Sections 3.1–3.3 to show how GenFuse retrieves features of the missing modality, combines them with the available feature, and uses the resulting substitute for classification. These revisions are marked in blue.
If any of your original concerns remain, or if there are additional questions about the comparisons or the revised PDF, we would be glad to discuss them further.
Response to Reviewer oRx9
Thank you for the concrete suggestions. We tested both alternatives you proposed: nearest-neighbor filling with random augmentation and fusion trained on complete pairs with a single-modality fallback at test time. We also clarified that all methods share the same per-sample missing masks and added two evaluations on Augsburg: a spatially disjoint split and a test with contiguous modality gaps.
W1. Did all methods use the same missing masks?
Yes. GenFuse and all baselines use the same data split and the same missing-modality assignment for each sample within a given setting. Under the specific protocol, all methods use the same training masks and the same designated missing modality at test time. Under the probabilistic protocol, both the training and test masks are shared across methods, so each test sample is evaluated with the same available modalities. The masks are not resampled independently for each method. We clarified this in the experimental setup and Appendix A.
W2. Do the results hold under missingness that resembles real sensor coverage gaps?
The Augsburg data available to us do not provide verified sensor-availability masks. To approximate a real coverage gap, we made HSI unavailable across one contiguous region and SAR unavailable across another, rather than masking test samples independently. This simulates a sensor coverage gap at test time.
We separated this test from the question of where the models are evaluated. First, we used a spatially disjoint split, so the 9 × 9 training and test patches share no image pixels. With random masking on this held-out region, GenFuse improves over REDEEM by 3.17 pp in OA and 8.64 pp in AA (Table 3; Appendix B.2). Then, without retraining either model, we replaced random test masking with the contiguous sensor gaps on the same held-out region. On the 22,139 samples evaluated under the shared gap map, GenFuse achieves 83.17% OA and 70.58% AA, compared with 78.71% and 62.55% for REDEEM. The OA advantage holds with either sensor missing: +1.81 pp for HSI and +11.85 pp for SAR (Appendix B.3, Table 14).
Q1. How does GenFuse compare with nearest-neighbor filling and random augmentation?
We implemented the suggested NN + Aug. baseline. It uses the available modality to find the nearest complete training pair and uses that neighbor’s other modality feature as the proxy. During training, we randomly perturb the retrieved feature; at test time, we use it without perturbation. Under 70% probabilistic missingness in both training and testing, GenFuse improves overall OA by 17.79 pp on Augsburg and 2.50 pp on Trento over NN + Aug. (Appendix Table 12).
Q2. Is it enough to train fusion on complete pairs and fall back to a single-modality classifier?
We evaluated Fusion + Fallback. Its fusion classifier is trained only on complete pairs and used when both modalities are available at test time. When one modality is missing, it uses the classifier for the available modality without generating a proxy. Under the same protocol, GenFuse improves overall OA by 4.60 pp on Augsburg and 1.17 pp on Trento.
Fusion + Fallback achieves higher OA on Augsburg when HSI is missing (63.88% vs. 53.15%). Across the full test sets, which mix complete inputs and both missing directions, GenFuse achieves higher OA and AA on both datasets. Appendix Table 12 reports the results for each test group.
We welcome any further questions during the discussion.