- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies adapting pretrained remote sensing models to changing input modalities without annotating the new modality. It considers three scenarios, namely transfer, using only the new modality; addition, using both modalities; and peeking, retaining the original modality at inference after observing the new modality during adaptation. The proposed DeluluNet combines modality-specific encoders, a masking transformer for missing-modality feature hallucination, and a fusion transformer. Training integrates prediction distillation, pre-fusion reconstruction, latent feature matching, and replay of existing labeled old-modality data.
Critical strengths
1.The paper proposes a coherent framework for a practically relevant adaptation problem.
2.Feature hallucination serves a dual purpose: accommodating missing modalities at inference and enabling old-modality labeled samples to train the multimodal model.
3.Alternating between real multimodal inputs with pseudo-labels and hallucinated multimodal inputs with ground-truth labels coherently links architecture and training strategy.
4.The experiments are extensive.
Critical weaknesses
1.Some experimental settings remain inconsistent and unclear. (1) Several results cannot be reconciled across the manuscript. EuroSAT RGB→VRE transfer accuracy is 90.6 ± 0.4 in Tab. 1 but 96.4 ± 0.2 in Tab. 7. Tab. 3 and 8 also reverse the DeluluNet–MixMatch ranking for RGB peeking at VRE. (2) Appendix C describes agreement with the frozen teacher but also refers to pseudo-labels from the peeking branch, while Fig. 7 additionally multiplies agreement by peeking performance. The exact reference predictor, score formulation, and task-specific definitions remain unclear. Differences in tuning or model selection should be explicitly documented.
2.DeluluNet reuses old-modality ground-truth labels during adaptation, whereas KD/TTM rely on teacher predictions. This is legitimate under the proposed setting, but confounds the contribution of hallucination with supervised replay. Tab. 4 provides useful partial controls, yet covers only one dataset–modality pair under a separately tuned protocol. Comparisons with simpler fusion or modality-dropout alternatives, matched for label access, capacity, and optimization budget, are needed to establish the incremental value of the proposed design.
3.Section 3 broadly characterizes cross-modal distillation as requiring labeled multimodal data. However, previous methods (e.g., Gupta et al., 2016) already used paired unlabeled modalities for representation supervision transfer. These precedents do not subsume the complete framework, but they narrow its novelty to unified scenario coverage and the incremental benefits of its mechanisms.
4.Masking and batch-mixing ratios are tuned separately for transfer, addition, and peeking. Consequently, it is unclear whether the reported results demonstrate one trained model supporting changing inputs or several scenario-specific models sharing an architecture. This distinction is consequential because flexible inference under changing modality availability is a central motivation.
5.The reported improvements are not consistent across settings. On DFC2020 with S2 as the original modality and S1 as the new modality, addition and peeking reduce performance from 67.0 to 65.1 and 64.1 mIoU, respectively. These failures warrant analysis of when adaptation preserves existing capability and when teacher-based supervision restricts the exploitation of complementary information. The current evidence does not resolve this distinction.
Questions for authors
1.What explains the discrepancies between Tab.s 1/7, 3/8, and 2/4?
2.What is the exact validation-score formula for each scenario? Does agreement reference frozen or the adapted peeking branch, and is it multiplied by peeking performance? How is agreement defined for multilabel classification and segmentation, and which validation labels are used?
3.How much improvement remains over simple fusion or modality dropout with identical old-label replay and matched capacity, training, and search budgets? For peeking, does the gain persist against an old-modality-only control matched in architecture and optimization, thereby isolating the contribution of observing the new modality?
4.Do Tab.s 1–3 evaluate a shared checkpoint or separately optimized models?
5.What explains the performance degradation in the DFC2020 S2→S1 addition and peeking settings? Does adaptation correct teacher errors or primarily preserve teacher predictions, and how does this behavior depend on teacher quality and modality informativeness?
Please refer to weaknesses.
5 comments
Response to Reviewer Bars (2/2)
4. One shared vs. separate weights for the three scenarios. Our main tables report separately trained checkpoints for each of transfer, peeking, and addition; masking and batch-mixing ratios are set per scenario (tuned once on DFC2020 S1→S2¬rgb). However, the point of having a shared set of weights is interesting, practical, and possible with DeluluNets. Here we report the performance of how a set of shared weights can perform well for all three scenarios if we select by average validation score across the three scenarios (new Appendix G). Although the shared weights generally trail the scenario-specific DeluluNet checkpoints from Tables 1–3, the shared weights still win against baselines and on 7/9, 7/9, and 8/9 of the tested dataset-modality pairs, even though each baseline is tuned and trained for its own scenario, showing that DeluluNet can achieve good performance for all three scenarios using one set of weights. We thank the reviewer for the insightful question, which led to this additional experiment that meaningfully strengthens the case for deploying a single DeluluNet.
5. For DFC2020 S2->S1, Sentinel 1 (S1) does not provide sizable value in addition to Sentinel 2 (S2). This is evidenced by observing that even the label-oracle supervised multimodal S1+S2 checkpoint only achieves +1.4 mIoU compared to the S2 . As a result, there's not much for DeluluNet to learn from the unlabeled S1. This issue is worsened by our hyperparameter selection strategy, where we directly use the hyperparameters selected with DFC2020 S1->S2_noRGB to save computational cost across the many dataset and modality pairs we experimented with. We have added an analysis on this specific dataset-modality pair in Appendix F, where we tune hyperparameters for this dataset and modality pairing, and show that although DeluluNet can't improve with S1, it largely preserves performance (within 1 mIoU), and still has a smaller drop than any baseline (MixMatch −6.1, MKE −1.0), while tying KD on transfer.
On whether adaptation corrects teacher errors or primarily preserves teacher predictions: DeluluNet is partly trained through distillation, which preserves much of the teacher's predictions. However, DeluluNet exceeds in 8 of 9 settings for both peeking and addition (Tables 2–3), so it is correcting errors made by . These gains over come from the unlabeled multimodal data.
We thank the reviewer for the careful reading and the questions, which led to additional experiments on shared DeluluNet weights for all three scenarios, and a robustness analysis when the new modality is not useful. We believe these changes should address the reviewer's concern, and would like to hear the reviewer's feedback on the revision.
As the response phase is coming to a close, we wanted to check whether there is anything further we can address or discuss. We believe our rebuttal and revisions have addressed your concerns, but we would be glad to answer any remaining questions. Thank you for your time and feedback.
Response to Reviewer Bars (1/2)
We thank the reviewer for the careful review and the questions. We address the reviewer's concerns through additional experiments, protocol clarification, and manuscript adjustments.
1. On cross-table performance difference:
a. The submission's EuroSAT tables in the Appendix report results obtained with an outdated experiment protocol. We've updated the tables with the main text protocol, so now the RGB->VRE runs on EuroSAT are internally consistent across main text and the appendix. We thank the reviewer for catching this.
b. Table 4 (Ablation) reports slightly different numbers from Tables 1–3 because of its independent tuning of hyperparameters (which ensures fairness for the ablation configurations), while Tables 1–3 use structural hyperparameters tuned once on DFC2020 S1→S2¬rgb. We regenerated Table 4 from the logged sweeps with clarified protocol, with each column reporting the run with the highest validation score, and have updated the table caption and Section 5.1 for clarification.
On validation score: We select DeluluNet transfer and addition based on the simple agreement between DeluluNet and the frozen teacher f0. We have updated Figure 7 and Appendix C with additional results and discussion, alongside correcting the figure caption, which was outdated: agreement is not multiplied by peeking performance, and no pseudo-labels from the adapted peeking branch are used. Specifically: (i) peeking predicts from MA alone, so it is selected by standard validation performance on the labeled uni-modal MA validation split; (ii) transfer (MB input) and addition (MA+MB input) are selected on the unlabeled multimodal validation split by agreement with the frozen f0's predictions on MA; (iii) agreement is top-class agreement for single- and multi-label classification, and per-pixel top-class agreement for segmentation. We thank the reviewer for catching this inconsistency.
2. On isolating modality hallucination and supervised replay: We'd like to emphasize that DeluluNet's modality hallucination mechanism is precisely what makes the (pseudo-) supervised replay possible. The modality-dropout alternative is the "+Masking Token" row of Table 4: it replaces masking transformers with per-modality learned mask tokens (i.e., modality dropout) while keeping the same old-label replay via batch mixing, with an independently tuned set of hyperparameters. It trails DeluluNet in all three scenarios.
3. On parameters and architecture-controlled supervised baseline for DeluluNet's peeking mode. We thank the reviewer for this valuable suggestion. We report the performance of an architecture-matched supervised control in a new Appendix section H, where we show that DeluluNet's improvement from modality peeking can be cleanly separated out from the added parameters and additional supervised training: the control (same architecture, same initialization, same training steps as f0, at a tuned learning rate) does not improve on f0 while DeluluNets do, and every DeluluNet seed scores higher than every control seed.
On related work in cross-modal distillation. We agree, and have corrected the related-work characterization (L125–129): some cross-modal distillation methods such as UniDistill (Zhou et al., 2023) assume labeled multimodal data, while Gupta et al. (2016) and Zhao et al. (2018) use paired unlabeled modalities to train a model on the new modality, but do not learn a fused multimodal model or address modality addition and peeking.