acceptodds
Under review as a conference paper at ICLR 2027

GenFuse: Cross-Sample Genetic Fusion for Incomplete Multimodal Training in Remote Sensing

Abstract

Multimodal remote sensing classifiers typically assume complete training pairs and handle missing modalities only at inference, limiting their applicability to incomplete multimodal training (IMT), where modality absence also occurs during training. We propose GenFuse, a cross-sample genetic fusion framework that generates a missing-modality proxy by recombining two complementary parents rather than relying solely on intra-sample mapping. The current sample's available feature serves as the semantic parent, while features retrieved from the training bank corresponding to the missing modality are aggregated into the modality parent. Feature-wise crossover and range-scaled mutation with a learnable offset produce the proxy, which is fused with the available feature for classification. The complete subset supervises proxy alignment, while genetic consistency anchors proxies for incomplete samples to their available features without ground-truth features from the missing modality. Experiments on three datasets under specific and probabilistic modality-missing protocols demonstrate superior classification performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
incomplete multimodal learning, missing modality, multimodal representation learning, cross-sample retrieval, proxy feature generation, remote sensing image classification

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_oRx9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for training multi-modal fusion networks that is robust to missing inputs from some of the sensing modes (incomplete multi-modal training IMT). The method is couched in the language of genetic programming, but essentially involves constructing a missing data proxy via the method of - when data is missing in one modality - finding similar samples in the modalities that are present, and then combining the corresponding data of those samples in the missing-modalities into an interpolated “proxy” sample used to fill in for the missing data during training. The proposed method is then compared experimentally to other IMT approaches and found to have favorable performance.

Critical strengths
  • The general approach of proxy data generation via genetic-style combination is an interesting approach to the problem.
  • Internal hyperparameters for attentive aggregation seem to be learned from data instead of relying on fixed approaches.
Critical weaknesses
  • It appears that each evaluation randomly masked missing data at training time for that evaluation, instead of masking once randomly and using the same training set for each method. Maybe this makes no difference, but seems to leave room for unequal evaluation of the methods.
  • The method does not appear to be tested on actual missing modality data, all missing pairs are constructed. In theory there is no difference between theory and practice, in practice there often is.
Questions for authors
  • I am curious how the method would compare a more simplistic data augmentation of the missing-modality data for example by selecting the nearest neighbor in the present-modality and adding random data augmentation.
  • I am also curious how the method would compare to only using fully-present data for training the fusion network and then falling back to single-modality networks when a modality is missing.
2 comments
AuthorsOpenReview ↗

Response to Reviewer oRx9

Thank you for the concrete suggestions. We tested both alternatives you proposed: nearest-neighbor filling with random augmentation and fusion trained on complete pairs with a single-modality fallback at test time. We also clarified that all methods share the same per-sample missing masks and added two evaluations on Augsburg: a spatially disjoint split and a test with contiguous modality gaps.

W1. Did all methods use the same missing masks?

Yes. GenFuse and all baselines use the same data split and the same missing-modality assignment for each sample within a given setting. Under the specific protocol, all methods use the same training masks and the same designated missing modality at test time. Under the probabilistic protocol, both the training and test masks are shared across methods, so each test sample is evaluated with the same available modalities. The masks are not resampled independently for each method. We clarified this in the experimental setup and Appendix A.

W2. Do the results hold under missingness that resembles real sensor coverage gaps?

The Augsburg data available to us do not provide verified sensor-availability masks. To approximate a real coverage gap, we made HSI unavailable across one contiguous region and SAR unavailable across another, rather than masking test samples independently. This simulates a sensor coverage gap at test time.

We separated this test from the question of where the models are evaluated. First, we used a spatially disjoint split, so the 9 × 9 training and test patches share no image pixels. With random masking on this held-out region, GenFuse improves over REDEEM by 3.17 pp in OA and 8.64 pp in AA (Table 3; Appendix B.2). Then, without retraining either model, we replaced random test masking with the contiguous sensor gaps on the same held-out region. On the 22,139 samples evaluated under the shared gap map, GenFuse achieves 83.17% OA and 70.58% AA, compared with 78.71% and 62.55% for REDEEM. The OA advantage holds with either sensor missing: +1.81 pp for HSI and +11.85 pp for SAR (Appendix B.3, Table 14).

Q1. How does GenFuse compare with nearest-neighbor filling and random augmentation?

We implemented the suggested NN + Aug. baseline. It uses the available modality to find the nearest complete training pair and uses that neighbor’s other modality feature as the proxy. During training, we randomly perturb the retrieved feature; at test time, we use it without perturbation. Under 70% probabilistic missingness in both training and testing, GenFuse improves overall OA by 17.79 pp on Augsburg and 2.50 pp on Trento over NN + Aug. (Appendix Table 12).

Q2. Is it enough to train fusion on complete pairs and fall back to a single-modality classifier?

We evaluated Fusion + Fallback. Its fusion classifier is trained only on complete pairs and used when both modalities are available at test time. When one modality is missing, it uses the classifier for the available modality without generating a proxy. Under the same protocol, GenFuse improves overall OA by 4.60 pp on Augsburg and 1.17 pp on Trento.

MethodAugsburg OA (%)Augsburg AA (%)Trento OA (%)Trento AA (%)
NN + Aug.60.2161.9294.8492.42
Fusion + Fallback73.4056.5296.1794.49
GenFuse78.0069.7497.3496.16

Fusion + Fallback achieves higher OA on Augsburg when HSI is missing (63.88% vs. 53.15%). Across the full test sets, which mix complete inputs and both missing directions, GenFuse achieves higher OA and AA on both datasets. Appendix Table 12 reports the results for each test group.

We welcome any further questions during the discussion.

AuthorsOpenReview ↗

Follow-up to Reviewer oRx9

Dear Reviewer oRx9,

We are following up on our detailed rebuttal above. We would appreciate hearing whether it addresses your concerns about shared missing masks and realistic missingness, as well as your questions about the two alternatives you suggested. Below, we briefly recap our earlier response and note what has changed in the revised PDF.

1. What we addressed in our initial rebuttal

All methods use the same data split and the same missing-modality assignment for each sample within an experiment. For example, if HSI is masked for an Augsburg test sample, every method evaluates that sample without HSI.

The two Augsburg evaluations address different questions. The spatially disjoint evaluation tests performance in a held-out part of the scene, with no shared pixels between training and test patches. The simulated coverage-gap evaluation changes which modalities are available at test time: without retraining either model, we masked HSI in one contiguous area and SAR in another within that held-out region. Under the shared gap masks, GenFuse achieved 83.17% OA versus 78.71% for REDEEM, with higher OA in both missing directions (Table 3; Appendices B.2–B.3, Table 14).

We also tested both alternatives you suggested: nearest-neighbor filling with training-time augmentation and fusion trained on complete pairs with a single-modality fallback at test time. At 70% probabilistic missingness on Augsburg, their overall OA was 60.21% and 73.40%, respectively, compared with 78.00% for GenFuse. GenFuse also achieved higher overall OA than both alternatives on Trento (Appendix Table 12).

2. What is new in the revised PDF

We expanded the Stage 2 discussion to explain why we recommend keeping the encoders trainable and the feature banks frozen by default (Section 4.3; Appendix B.4). Refreshing the banks after each epoch improved OA on the spatially disjoint split and lowered it on the random split. We therefore keep the banks frozen by default and discuss refreshing as an option for the spatially held-out setting we tested.

We also expanded the per-class analysis across three Augsburg splits (Appendix B.5). Residential Area accounts for 38.8% of test pixels, and GenFuse achieves 85.37% mean accuracy on this class versus 64.36% for REDEEM. This improvement carries substantial weight in overall pixel accuracy, whereas average class accuracy gives every class equal weight.

Finally, we streamlined Sections 3.1–3.3 to show how GenFuse retrieves features of the missing modality, combines them with the available feature, and uses the resulting substitute for classification. These revisions are marked in blue.

If any of your original concerns remain, or if there are additional questions about the comparisons or the revised PDF, we would be glad to discuss them further.

Reviewer_6HPCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies land-cover classification when some remote-sensing inputs are missing during training or testing. The key ideas is to use features retrieved from other training samples to construct a replacement for the missing input’s features. They evaluate the method on three datasets under different amounts and patterns of missing data, and report improvements over the compared methods in most settings, with larger gains on Houston and Augsburg and smaller gains on Trento land cover rasters.

Critical strengths
  1. The problem motivation makes sense and is interesting. Handling missing sensors during both training and testing makes better use of incomplete remote-sensing data.
  2. The papers' comparisons with simple alternatives when sensor data is missing is interesting. The simple comparisons helped assess whether the method strictly helps with missing modalities.
  3. The reported performance improvements are impressive.
Critical weaknesses

I structure the weaknesses by two major points: limited novelty, and insufficient evidence that the proposed method (GenFuse) shows a strong improvement over prior work.

  1. Novelty: From quick searches, I see that RAGPT and REDEEM also retrieve information from other samples in the data to compensate for missing modalities. The paper does discuss these methods, but these are not included as baselines.
  2. The benchmark dataset choices are very unconventional. The "Augsburg," "Houston,", and "Trento" datasets are not standard multimodal benchmarks in machine learning for remote sensing. Even when considered as standard datasets, the breadth of evaluation is too little to convincingly establish the method.
  3. I struggle to understand the "genetic" terminology used in the paper. From what I understand, "Crossover" is learned feature mixing, "mutation" is a learned additive correction. I don't understand why the biological analogies are warranted here (I encourage the authors to correct me here if this is common in this specific problem setting).

Minor comments

  1. Please use \cite{} and \citep{} appropriately. The paper contains citations that do not follow ICLR 2027 style guidelines (should be \citep{}).
  2. RS is never defined before used in the introduction.

Writing: I find the writing of the paper to be incredibly dense and ineligible. For example in the introduction:

Rather than using this parent directly as the proxy, GGM recombines it with the semantic parent through feature-wise crossover and then applies range-scaled mutation using a learnable offset scaled by the feature range of the corresponding bank.

I appreciate the authors for attempting to condense the entire method's description and nomenclature into one sentence, but I believe this can be significantly simplified. I did not understand any of this during a sequential read of the manuscript.

We develop CSPM to construct the modality parent and GGM to generate the proxy through feature-wise crossover and range-scaled mutation, together with training objectives tailored to complete and incomplete samples.

This is incredibly hard to parse and understand. What is the proxy? What is feature-wise "crossover"? What is a range-scaled "mutation"? While multiple reads of the paper can help some of this, clearer writing would significantly improve the quality of this paper.

Overall, I do not believe this paper meets the standard for publication at ICLR. I suggest the authors to incorporate better baselines, more benchmark datasets, and submit to the relevant domain venue.

Questions for authors

I'm curious if table 1 could include the single-modality baselines at every missingness rate. The large gains over the multimodal methods are good, but compared to the smaller improvements in Table 6, it would be interesting to see whether generating missing features is actually necessary.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6HPC

Thank you for raising these questions. We added REDEEM and ConfSMoE to both main comparison tables, included Unimodal at every training missing rate, and rewrote the two passages highlighted in the introduction.

W1. What is new compared with existing retrieval-based compensation methods, and how does GenFuse perform against them?

RAGPT and REDEEM use the available modality to match complete reference samples and then use the corresponding features of the missing modality. GenFuse searches directly in the bank for the missing modality. This lets incomplete training samples contribute the features they observe, even when they cannot provide complete pairs. Compared with ConfSMoE, which averages sampled features to initialize imputation and then refines the estimate, GenFuse uses the current sample’s available feature to select and weight candidates from the start. That feature also contributes directly to the generated substitute. See Appendix A.4 for details.

GenFuse achieves higher OA than REDEEM in 8 of 9 specific settings and 5 of 6 probabilistic settings, and higher OA than ConfSMoE in 8 of 9 specific settings and all 6 probabilistic settings (Tables 1–2). The specific results average OA over the two missing directions. To isolate the effect of allowing incomplete samples to contribute to the bank, we compared GenFuse with a version that uses only complete pairs as references. On Augsburg at 70% probabilistic missingness, including the observed features of incomplete samples raises OA from 69.45% to 78.00% (Appendix B.1).

W2. Why use Houston, Augsburg, and Trento, and what test conditions do they cover?

All three datasets have been used in prior multimodal remote sensing classification studies. DPMamba (IJCAI 2025) evaluates missing-modality classification on the same three datasets. DGDNet (IEEE TGRS 2023) uses Houston and Augsburg, while LDS2AE (AAAI 2024) and MCFNet (IEEE TGRS 2025) both evaluate on Houston and Trento. These datasets let us study two sensor pairings: HSI–LiDAR on Houston and Trento, and HSI–SAR on Augsburg.

We also added two tests on Augsburg. In the spatially disjoint split, the training and test patches share no image pixels. On that held-out region, GenFuse exceeds REDEEM by 3.17 pp in OA and 8.64 pp in AA (Table 3). Using the same trained models, we then imposed contiguous modality gaps at test time to simulate gaps in sensor coverage. GenFuse achieves 83.17% OA and 70.58% AA, compared with 78.71% and 62.55% for REDEEM (Appendix B.3).

W3 + M3. What do crossover, mutation, and proxy mean in GenFuse?

The two “parents” are the current sample’s available feature and an aggregate of features retrieved for the missing modality. Crossover learns how much each parent contributes to each feature dimension. Mutation then applies a learned additive correction scaled by the feature range of the bank. The resulting feature stands in for the missing modality and is fused with the available feature for classification. We call this substitute feature the proxy.

We revised the introduction to explain these steps directly and removed the string of module and operator names from the second contribution. We also tested the role of range scaling: on Augsburg at 30% training missingness under the specific protocol, it improves MeanOA from 65.13% with a plain learnable bias to 73.07%, while keeping the mixing step unchanged (Table 5; Appendix Table 11).

M1–M2. Citation style and the definition of RS.

We revised the parenthetical citations and defined remote sensing (RS) at its first use in the introduction.

Q1. Does generating a substitute feature help compared with using the available modality alone?

Table 1 now includes Unimodal at every training missing rate: 0%, 30%, and 70%. Unimodal trains a separate classifier for each modality using samples where that modality is observed, then uses only the available modality at test time. GenFuse achieves higher direction-averaged OA in all 9 dataset–missing-rate settings and higher OA in 15 of 18 individual missing-direction settings (Appendix G).

On Augsburg at 70% training missingness, GenFuse reaches 70.9% OA vs. 33.7% for Unimodal when HSI is missing. When SAR is missing, the two are close (86.2% vs. 87.1%). The direction-level comparison shows where generating a substitute feature makes the largest difference.

We welcome any further questions during the discussion.

Reviewer_6HPCOpenReview ↗

Reviewer 6HPC Response to Authors

Thank you for performing additional experiments, and fixing the specific writing issues I pointed out in my review. My issues with writing quality remain. However, I appreciate the authors' efforts in incorporating additional baselines to their work and increase my score from a clear reject to a weak reject.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 6HPC

Thank you for your follow-up and for acknowledging the additional comparisons. We intended the genetic terminology to make the combination of two complementary feature sources intuitive. Some of these terms and repeated explanations made parts of the method harder to follow. We apologize for the lack of clarity in our presentation.

We have made a focused revision to Section 3, marked in blue in the updated PDF, to present the method in computational order.

In the quoted text below, mm denotes the available modality and m′m' the missing modality.

Section 3.1 (Problem Setting and Overview) now introduces the feature banks, the two parent features, and the generated substitute step by step:

For an incomplete sample with available modality mm and missing modality m′m', where m≠m′m\neq m', GenFuse generates a substitute feature fm′proxyf_{m'}^{\mathrm{proxy}} and fuses it with fmf_m for classification.

As illustrated in Figure 2, GenFuse maintains a first-in, first-out (FIFO) bank Bm\mathcal B_m of observed training features for each modality. For an incomplete sample, CSPM uses the available feature fmf_m to retrieve and aggregate features from Bm′\mathcal B_{m'}. The available feature serves as the semantic parent, and the aggregate f~m′\tilde f_{m'} serves as the modality parent. GGM combines them to generate fm′proxyf_{m'}^{\mathrm{proxy}}, which is fused with fmf_m for classification. The same procedure applies in both modality directions. For incomplete training samples, genetic consistency supervision encourages the proxy to preserve the scene semantics carried by the available feature. Complete samples provide observed features for aligning the proxies generated in both modality directions.

Section 3.2 (Cross-Sample Parental Matching, CSPM) now focuses on candidate retrieval and aggregation. Its opening replaces the repeated discussion of parental complementarity:

CSPM constructs the modality parent by retrieving and aggregating features from the bank corresponding to the missing modality. It combines multiple candidates to reduce dependence on the content of a single retrieved sample.

We also removed the repeated explanation of single-candidate limitations from Attentive Aggregation, which now leads directly into the computation:

Since the relevance of the top-kk candidates varies, CSPM computes their attention weights using the semantic parent fmf_m. It projects fmf_m and each selected candidate into a query–key space:

Section 3.3 (Genetic Generation Module, GGM) now starts from its two inputs and explains the operations before presenting the equations:

Given the semantic parent fmf_m and modality parent f~m′\tilde f_{m'}, GGM generates a substitute feature for the missing modality. Feature-wise crossover learns how to mix the two parents, and range-scaled mutation applies a learned additive correction scaled by the feature range of the bank.

These changes retain the motivation in the introduction and let the method subsections concentrate on retrieval, aggregation, feature mixing, correction, and classification. All equations and implementation details are preserved.

We sincerely appreciate your time and constructive feedback. If there are any particular passages you would like us to clarify further—or any additional explanations or analyses that would help address remaining concerns—please let us know, and we will be glad to work on them during the discussion period.

Reviewer_u9xvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GenFuse, a cross-sample genetic fusion framework for incomplete multimodal training (IMT) in remote sensing image classification, where modalities may be missing during both training and inference. The framework introduces Cross-Sample Parental Matching (CSPM) to retrieve and aggregate relevant features from a missing-modality feature bank, and a Genetic Generation Module (GGM) to combine these features with the available modality through feature-wise crossover and range-scaled mutation, generating proxy representations for missing modalities. Additionally, the authors design a genetic consistency loss for incomplete samples and a proxy similarity loss for complete samples to guide feature generation.

Critical strengths
  1. The paper addresses incomplete multimodal training in remote sensing, where modalities may be missing during both training and inference. This setting is practically meaningful and extends beyond the complete-training assumptions of many existing missing-modality methods.
  2. GenFuse combines current-sample semantic information with missing-modality features retrieved from other training samples, enabling incomplete samples to contribute to representation learning without requiring their missing-modality ground truth. This provides a reasonable and effective strategy for handling incomplete multimodal data.
  3. Experiments on three remote sensing datasets under specific and probabilistic modality-missing settings demonstrate substantial improvements over several baselines, particularly on Houston and Augsburg. Ablation studies and additional analyses further support the effectiveness of the proposed components.
Critical weaknesses
  1. The proposed CSPM primarily relies on conventional Top-k retrieval and attention-based aggregation, while the genetic crossover and mutation operations essentially correspond to feature-wise gated interpolation and learnable offsets. The technical novelty beyond existing retrieval-based feature compensation methods remains insufficiently established.
  2. Several compared methods were originally developed under complete-modality training assumptions, and missing inputs are handled through zero-filled tensors. It remains unclear whether these baselines are adequately adapted to the IMT setting, weakening the conclusions drawn from the substantial performance improvements.
  3. The feature banks remain frozen during Stage 2 while the encoders continue updating, potentially introducing inconsistencies between stored and current representations. Moreover, the substantial performance fluctuations across bank sizes on Augsburg raise concerns about the stability and robustness of the retrieval mechanism.
Questions for authors

Please responds to my concerns in the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer u9xv

Thank you for the feedback. We added ConfSMoE and REDEEM to both main comparison tables. We also tested the contribution of observed features from incomplete samples, compared range scaling with a plain learnable bias, and examined how Stage 2 encoder updates and bank capacity affect retrieval.

W1. What is technically new compared with existing retrieval-based compensation methods?

ConfSMoE starts by averaging sampled features of the missing modality and then refines that estimate using the available modality. GenFuse uses the current sample’s available feature from the start: it guides which candidates are retrieved and how they are weighted, and it contributes directly to the proxy. REDEEM uses the available modality to match complete reference pairs and then takes their missing-modality counterparts. GenFuse searches directly in the bank for the missing modality, so incomplete training samples can also supply features they observe. See Appendix A.4 for details on these design differences.

GenFuse has higher OA than ConfSMoE in 8 of 9 specific and all 6 probabilistic settings, and higher OA than REDEEM in 8 of 9 specific and 5 of 6 probabilistic settings (Tables 1–2; Appendix A). The specific results average OA over the two missing directions.

Two comparisons test the value of these design choices. On Augsburg at 70% probabilistic missingness, including observed features from incomplete samples raises OA from 69.45% to 78.00% compared with using a bank built only from complete pairs (Appendix B.1). On Augsburg at 30% training missingness under the specific protocol, range scaling raises MeanOA from 65.13% with a plain learnable bias to 73.07%, with the mixing step held fixed (Table 5; Appendix Table 11).

W2. How were the baselines adapted to incomplete training?

GenFuse and the baselines use the same data splits and fixed per-sample missing masks. Under the specific protocol, methods adapted to a particular missing direction use the same designated test-time direction as GenFuse. DPMamba is fine-tuned for that direction. MCF-Net selects samples with the required modality available before masking the other modality. LDS2AE computes reconstruction loss only on observed modalities. Zero filling supplies the input tensor where required by the implementation.

ConfSMoE and REDEEM use the same encoder architecture as GenFuse with independently initialized weights. Both run for 200 epochs in Stage 1 and 400 in Stage 2 and are evaluated at the final epoch. Appendix A.3 describes the baseline adaptations and training schedules.

W3. Do encoder updates degrade retrieval from a frozen bank, and how sensitive is retrieval to bank capacity?

Encoder updates. On Augsburg with 30% training missingness and HSI missing at test time, we compared SAR queries from the start and end of Stage 2 on the same 78,014 test samples, using the same frozen 16,384-slot HSI bank. The median top-32 slot overlap is 0%, but the same-class retrieval ratio increases from 39.83% to 60.94%. The encoder update changes which slots are retrieved, and the retrieved candidates more often match the query’s class at the end of Stage 2 (Appendix Table 16).

Freezing the encoders or refreshing the banks. With the FIFO banks fixed at 30% training missingness under the specific protocol, freezing the encoders raises MeanOA from 73.07% to 75.90% but lowers MeanAA from 68.07% to 64.25% (Table 5; Appendix Table 11). In a separate experiment at 70% probabilistic missingness, both runs start Stage 2 from the same re-encoded banks on each split, and both continue updating the encoders. Frozen keeps the initial bank features unchanged throughout Stage 2. Refreshed recomputes the bank features with the updated encoders after each epoch. Compared with Frozen, Refreshed raises OA from 83.77% to 84.44% on the spatially disjoint split and lowers it from 69.12% to 67.53% on the random split (Table 5; Appendix Table 15).

Bank capacity. We repeated the Augsburg HSI-missing capacity sweep using the same preprocessed inputs, split, and missing masks across all four capacities, with k = 32 fixed in both stages. The top-32 same-class retrieval ratio exceeds the random baseline from the same bank at every capacity, by at least 19.87 percentage points. At 8,192 and 16,384 slots, the same-class retrieval ratios are close (59.22% and 60.94%). Thus, although classification OA varies across capacities, retrieval retains an advantage over random selection at every capacity tested (Appendix Tables 25–26).

We welcome any further questions during the discussion.

AuthorsOpenReview ↗

Follow-up to Reviewer u9xv

Dear Reviewer u9xv,

We are following up on our detailed rebuttal above. We would appreciate hearing whether it addresses your concerns about how GenFuse differs from existing methods, how the baselines were trained, and how encoder updates and bank capacity affect retrieval. Below, we briefly recap our earlier response and note what has changed in the revised PDF.

1. What we addressed in our initial rebuttal

GenFuse searches directly in the bank for the missing modality, so incomplete training samples can contribute the features they observe. On Augsburg at 70% probabilistic missingness, including those features raises overall accuracy (OA) from 69.45% to 78.00% compared with building the bank only from complete pairs. We added ConfSMoE and REDEEM to both main comparison tables and clarified how the baselines were trained using the same data splits and per-sample missing-modality assignments (Tables 1–2; Appendices A.3–A.4 and B.1).

To test whether updating the encoder degrades retrieval from the stored features, we encoded the same SAR test inputs with the encoder at the start and at the end of Stage 2. We used each set of SAR features to search the same frozen HSI bank. The fraction of top-32 candidates that shared the test sample’s class rose from 39.83% to 60.94% (Appendix Table 16). In the four-capacity comparison, same-class retrieval exceeded random selection at every capacity tested. Classification OA varied across capacities and reached its highest value, 73.82%, at 16,384 slots (Appendix Tables 25–26).

2. What is new in the revised PDF

We expanded the Stage 2 discussion to explain why we recommend keeping the encoders trainable and the feature banks frozen by default, and when other settings may be useful (Section 4.3; Appendix B.4). A trainable encoder can adapt how it queries a fixed bank, as the retrieval comparison above illustrates. Freezing the encoders is an option when overall pixel accuracy is the priority: in our Augsburg comparison, it raised mean OA from 73.07% to 75.90%, while mean class accuracy fell from 68.07% to 64.25% (Appendix Table 11).

Refreshing the banks changes the features used to rank and aggregate candidates, as well as the per-dimension feature ranges used to scale the additive correction. Refreshing improved OA on the spatially disjoint split and lowered it on the random split (Appendix Table 15). Keeping the banks frozen by default holds these reference quantities fixed and avoids re-encoding training features after every epoch. Refreshing remains an option in the spatially held-out setting we tested.

We also streamlined Sections 3.1–3.3 so readers can follow retrieval, aggregation, and feature mixing in the order they are performed. These revisions are marked in blue.

If any of your original concerns remain, or if the revised discussion raises a new question, we would be glad to discuss it further.

Reviewer_gifBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GenFuse for incomplete multimodal training in remote sensing. It retrieves features of the missing modality from a training feature bank and combines them with the available feature to generate a proxy for classification. The method is evaluated on three datasets under different missing-modality protocols.

Critical strengths
  1. The paper addresses a relevant setting where modalities can be missing during both training and inference.
  2. The framework is intuitive and clearly presented: it combines current-sample semantics with cross-sample features of the missing modality.
  3. The experiments include component ablations, alternative proxy strategies, and a same-ID retrieval exclusion control.
Critical weaknesses
  1. Training and test pixels are randomly split without spatial separation, while each input is a spatial patch. Overlapping patches and nearby samples may make retrieval easier. Spatially disjoint evaluation would help determine whether the gains extend to unseen areas.

  2. Some gains are unusually large, but the paper gives limited detail on how baselines were adapted to incomplete training. It is unclear whether they received training budgets comparable to GenFuse and, in the specific protocol, the same knowledge of the test-time missing direction.

  3. The proxy improves classification, but it is unclear whether it captures missing-modality-specific information beyond transforming the available feature or retrieving semantically similar samples.

  4. The crossover and mutation operations resemble gated fusion and a scaled learnable offset. The ablations establish their usefulness within GenFuse, but comparisons with conventional alternatives would better justify the specific design.

  5. The main results are single-run, and the three-seed check covers only one setting with a fixed data split. Results across multiple splits and per-class performance would clarify how consistent the gains are.

Questions for authors

The comparison with other banking-base approach would be useful. Check papers:

https://arxiv.org/abs/2410.08245

https://arxiv.org/abs/2505.19525

Does range-scaled mutation outperform a standard learnable bias or residual?

How sensitive is performance to freezing the bank while continuing to update the encoders in Stage 2?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gifB (Part 1/2)

Thank you for the constructive suggestions. We added ConfSMoE and REDEEM to both main comparison tables. We also added a spatially disjoint evaluation, comparisons with simpler proxy designs, and results across multiple data splits.

W1. Do the gains hold when training and test patches do not overlap?

We added a spatially disjoint Augsburg split in which the 9 × 9 training and test patches share no image pixels. Under 70% probabilistic missingness, GenFuse exceeds REDEEM by 3.17 pp in OA and 8.64 pp in AA on the held-out region. When HSI is missing, GenFuse achieves 75.53% OA, compared with 64.93% for REDEEM (Table 3; Appendix B.2).

W2 + Q1. How were the baselines trained, and how does GenFuse compare with other missing-modality methods?

GenFuse and the baselines use the same data splits and the same missing mask for each sample. Under the specific protocol, direction-specific adaptations use the same designated test-time missing modality as GenFuse. DPMamba is fine-tuned for that direction. MCF-Net selects samples with the required modality available before masking the other modality. LDS2AE computes reconstruction loss only on observed modalities. Zero filling supplies the input tensor where required by the implementation.

We added ConfSMoE and REDEEM under both protocols. Both use the same encoder architecture as GenFuse with independently initialized weights and run for 200 epochs in Stage 1 and 400 in Stage 2, with evaluation at the final epoch. GenFuse has higher OA than ConfSMoE in 8 of 9 specific and all 6 probabilistic settings, and higher OA than REDEEM in 8 of 9 specific and 5 of 6 probabilistic settings (Tables 1–2; Appendix A). The specific results average OA over the two missing directions.

ConfSMoE uses sampled features to initialize imputation, while REDEEM retrieves from complete reference pairs. GenFuse uses the current sample’s available feature to select and weight candidates directly from the bank for the missing modality. This lets incomplete training samples contribute their observed features. See Appendix A.4 for details on these design differences. Appendix A.4 also discusses Flex-MoE, whose learned bank embeddings are indexed by the observed modality combination and the identity of the missing modality. On Augsburg at 70% probabilistic missingness, including these features raises OA from 69.45% to 78.00% compared with using a bank built only from complete pairs.

W3. What does the proxy capture beyond transforming the available feature or using retrieved features directly?

On Augsburg with 30% training missingness and HSI missing at test time, GenFuse achieves 59.13% OA, compared with 52.92% for intra-sample compensation (ISC) and 36.40% for Retrieval-only. ISC maps the available feature to a proxy using alignment supervision; Retrieval-only uses the retrieved aggregation as the proxy.

We also fit ridge regressions to predict the same observed 180-band HSI spectra from each proxy. Standardized test RMSE is 0.9101 for GenFuse, 0.9280 for ISC, and 1.0393 for Retrieval-only. In this setting, GenFuse supports higher classification accuracy and more accurate HSI spectral prediction than either alternative proxy (Figure 4; Table 4; Appendix C.3).

W4 + Q2. Why combine the two features, and does range scaling help over a plain bias?

The mixing step combines the current sample’s available feature with retrieved features of the missing modality. On Augsburg at 30% specific missingness, GenFuse achieves 73.07% MeanOA across the two test directions, compared with 70.91% for ISC (an intra-sample compensation method that generates the proxy from the current sample’s available feature) and 62.90% for using the retrieved aggregation directly. To test the additive correction separately, we held the mixing step fixed. MeanOA is 73.07% with range scaling, 65.13% with a plain learnable bias, and 61.15% without the correction. Range scaling improves MeanOA by 7.94 pp over the plain bias in this setting (Figure 4; Table 5; Appendix Table 11).

AuthorsOpenReview ↗

Response to Reviewer gifB (Part 2/2)

W5. Are the gains consistent across runs, data splits, and classes?

GenFuse achieves higher OA than REDEEM and DPMamba in all three matched Houston runs and all three Augsburg splits. Values are mean ± sample standard deviation.

EvaluationGenFuse OA (%)REDEEM OA (%)DPMamba OA (%)
Houston: three runs on a fixed split, 70% training missingness, LiDAR missing at test time92.13 ± 0.7087.70 ± 2.5638.91 ± 18.38
Augsburg: three splits, 70% probabilistic missingness80.21 ± 2.6974.24 ± 0.9959.99 ± 7.60

On Augsburg, GenFuse outperforms REDEEM on Forest, Residential Area, and Industrial Area in every split. The largest mean per-class gain is on Residential Area (85.37% vs. 64.36%). Mean AA across all classes is closer (68.47% vs. 67.95%). Table 4 and Appendix B.5 provide the aggregate and per-class results.

Q3. What happens if we freeze the encoders or refresh the banks in Stage 2?

With the FIFO banks fixed on Augsburg at 30% specific missingness, freezing the encoders raises MeanOA from 73.07% to 75.90% and lowers MeanAA from 68.07% to 64.25%. In a separate comparison at 70% probabilistic missingness, we re-encoded the training features before Stage 2 so that both runs on each split started with identical banks. Both runs continued updating the encoders. Frozen keeps the initial bank features unchanged throughout Stage 2. Refreshed recomputes the bank features with the updated encoders after each epoch. Compared with Frozen, Refreshed raises OA on the spatially disjoint split (83.77% → 84.44%) and lowers it on the random split (69.12% → 67.53%). The results show a metric trade-off for encoder freezing and a split-dependent effect for bank refreshing (Table 5; Appendix Tables 11 and 15).

We welcome any further questions during the discussion.

Thanks for the effort and time for the response.

After reviewing the response, I found the rebuttal is somehow answer the questions, there is space for further discussion once more experiment results are obtained. But the author seems not able to provide the discussion.

For example, response for Q3 obtain additional results, but the author simply mentioned there is a a metric trade-off for encoder freezing and a split-dependent effect for bank refreshing. What may be the best setting instead? what is your recommendation?

I decide to keep my original judgment for this paper.

Kind regards

AuthorsOpenReview ↗

Follow-up Response to Reviewer gifB

Thank you very much for your prompt follow-up and valuable feedback. Our recommended default is to keep the encoders trainable while freezing the feature banks in Stage 2. We have added this recommendation to Section 4.3 and expanded the discussion of its basis and alternatives in Appendix B.4.

1. Why trainable encoders and frozen banks are our default

A frozen bank provides a fixed set of reference features, while the trainable encoder changes how the current sample queries those references. In CSPM, the encoder output determines the initial top-k ranking; the learned query–key projections then reweight the selected candidates. On Augsburg with 30% training missingness and HSI missing at test time, we evaluated the same 78,014 samples against the same frozen 16,384-slot HSI bank before and after Stage 2. The same-class retrieval ratio increases from 39.83% to 60.94% (Table 16). This shows that the SAR encoder learns to query the unchanged HSI bank more effectively in this experiment and supports retaining query adaptation against a fixed reference bank.

2. When we recommend refreshing the banks

Refreshing changes the vectors used for candidate ranking and aggregation, as well as the per-dimension range used to scale mutation. Keeping the banks frozen holds these reference quantities fixed while the remaining network adapts. At 70% probabilistic missingness, we compared Frozen and Refreshed starting from identical re-encoded banks (one entry per training sample with the corresponding modality observed) on each split, with trainable encoders in both runs. Frozen retains those initial bank features; Refreshed recomputes them after each epoch.

Refreshing raises spatially disjoint OA from 83.77% to 84.44% and lowers random-split OA from 69.12% to 67.53% (Table 15). We retain frozen banks as the default for their simpler update procedure and lower feature-extraction cost. For the spatially held-out setting evaluated here, refreshing is an option when its additional computation is acceptable.

3. When we recommend freezing the encoders

OA weights each class by its number of test pixels, whereas AA gives every class equal weight. Augsburg has an uneven test distribution, so the two metrics can favor different encoder policies. At 30% specific missingness, freezing the encoders raises MeanOA from 73.07% to 75.90% and lowers MeanAA from 68.07% to 64.25% (Table 11). The higher OA and lower AA indicate that its gains are weighted more strongly by pixel frequency. We therefore recommend trainable encoders when class-averaged performance matters, and regard encoder freezing as an alternative when overall pixel accuracy is the primary objective.

4. Why the multi-split OA and AA gains differ

The per-class results explain the larger OA gain over REDEEM. Residential Area accounts for 38.8% of the test pixels and improves from 64.36% to 85.37% accuracy. This large gain has substantial weight in OA. AA gives the same weight to Residential Area and each smaller class, producing a closer comparison between GenFuse (68.47%) and REDEEM (67.95%). The class frequencies and per-class improvements therefore explain how GenFuse maintains a larger overall-pixel advantage across all three splits (Appendix B.5).

5. What the proxy comparisons tell us

The two sources serve distinct roles: retrieved features of the missing modality supply observed representations of that modality, and the available feature carries the current sample’s scene information. The complete-pair-bank comparison tests the value of admitting observed features from incomplete samples: on Augsburg at 70% probabilistic missingness, OA increases from 69.45% to 78.00%. This lets a sample contribute useful references even when it cannot supply a complete pair (Appendix A.4).

Range scaling gives each dimension’s learned correction a scale derived from the bank’s observed feature range. Keeping crossover unchanged, we replace the range-scaled mutation offset with a plain learnable bias. Range scaling improves MeanOA by 7.94 pp on Augsburg (Appendix B.1). The spectral prediction experiment provides a common observed target for comparing the proxies: GenFuse yields lower HSI spectral RMSE than ISC and Retrieval-only, alongside higher classification OA in the tested missing direction (Appendix C.3).

We welcome any further questions during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Revisions

We thank all reviewers for their constructive feedback.

Summary of revisions. We added comparisons with REDEEM, ConfSMoE, and unimodal classifiers, expanded the evaluation to spatially disjoint testing and contiguous modality gaps, and clarified how GenFuse differs from existing compensation methods. The main changes are listed below.

  • Baseline comparisons and evaluation protocol. Tables 1–2 now include REDEEM and ConfSMoE under both missingness protocols. Table 1 also includes a unimodal baseline at every training missing rate. Appendix A documents the shared data splits and per-sample missing masks, the adaptations for incomplete training, and the differences between GenFuse and prior compensation methods.

  • Spatially disjoint testing and contiguous modality gaps. On Augsburg, we added a split in which training and test patches share no image pixels (Table 3; Appendix B.2). We then used the same trained models to evaluate simulated contiguous HSI and SAR coverage gaps at test time. Under these gaps, GenFuse achieves 83.17% OA and 70.58% AA, compared with 78.71% OA and 62.55% AA for REDEEM, with higher OA in both missing directions (Appendix B.3, Table 14).

  • Proxy generation and feature-bank design. We compared GenFuse with intra-sample compensation, retrieval-only proxies, nearest-neighbor filling with augmentation, and fusion with a unimodal fallback. We also evaluated spectral prediction from the proxies against the same observed 180-band HSI spectra. Additional experiments examine which samples contribute to the bank, the effect of range scaling versus a plain learnable bias, and how Stage 2 encoder updates and bank capacity affect retrieval (Figure 4; Tables 4–5; Appendices B–D).

  • Consistency across runs, splits, and classes. We report three matched runs on a fixed Houston split, results across three Augsburg data splits, and per-class results on Augsburg. GenFuse achieves higher OA than REDEEM and DPMamba in each of these runs and splits (Table 4; Appendix B.5).

We also rewrote the introduction to explain in plain language which features GenFuse combines, how it combines them, and how the resulting feature substitutes for the missing modality during classification. The individual replies provide the results and manuscript references for each reviewer’s questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.