acceptodds
Under review as a conference paper at ICLR 2027

Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors

Abstract

Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination—predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep Learning, Remote Sensing, Transfer Learning, Domain Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BarsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adapting pretrained remote sensing models to changing input modalities without annotating the new modality. It considers three scenarios, namely transfer, using only the new modality; addition, using both modalities; and peeking, retaining the original modality at inference after observing the new modality during adaptation. The proposed DeluluNet combines modality-specific encoders, a masking transformer for missing-modality feature hallucination, and a fusion transformer. Training integrates prediction distillation, pre-fusion reconstruction, latent feature matching, and replay of existing labeled old-modality data.

Critical strengths

1.The paper proposes a coherent framework for a practically relevant adaptation problem.

2.Feature hallucination serves a dual purpose: accommodating missing modalities at inference and enabling old-modality labeled samples to train the multimodal model.

3.Alternating between real multimodal inputs with pseudo-labels and hallucinated multimodal inputs with ground-truth labels coherently links architecture and training strategy.

4.The experiments are extensive.

Critical weaknesses

1.Some experimental settings remain inconsistent and unclear. (1) Several results cannot be reconciled across the manuscript. EuroSAT RGB→VRE transfer accuracy is 90.6 ± 0.4 in Tab. 1 but 96.4 ± 0.2 in Tab. 7. Tab. 3 and 8 also reverse the DeluluNet–MixMatch ranking for RGB peeking at VRE. (2) Appendix C describes agreement with the frozen teacher but also refers to pseudo-labels from the peeking branch, while Fig. 7 additionally multiplies agreement by peeking performance. The exact reference predictor, score formulation, and task-specific definitions remain unclear. Differences in tuning or model selection should be explicitly documented.

2.DeluluNet reuses old-modality ground-truth labels during adaptation, whereas KD/TTM rely on teacher predictions. This is legitimate under the proposed setting, but confounds the contribution of hallucination with supervised replay. Tab. 4 provides useful partial controls, yet covers only one dataset–modality pair under a separately tuned protocol. Comparisons with simpler fusion or modality-dropout alternatives, matched for label access, capacity, and optimization budget, are needed to establish the incremental value of the proposed design.

3.Section 3 broadly characterizes cross-modal distillation as requiring labeled multimodal data. However, previous methods (e.g., Gupta et al., 2016) already used paired unlabeled modalities for representation supervision transfer. These precedents do not subsume the complete framework, but they narrow its novelty to unified scenario coverage and the incremental benefits of its mechanisms.

4.Masking and batch-mixing ratios are tuned separately for transfer, addition, and peeking. Consequently, it is unclear whether the reported results demonstrate one trained model supporting changing inputs or several scenario-specific models sharing an architecture. This distinction is consequential because flexible inference under changing modality availability is a central motivation.

5.The reported improvements are not consistent across settings. On DFC2020 with S2 as the original modality and S1 as the new modality, addition and peeking reduce performance from 67.0 to 65.1 and 64.1 mIoU, respectively. These failures warrant analysis of when adaptation preserves existing capability and when teacher-based supervision restricts the exploitation of complementary information. The current evidence does not resolve this distinction.

Questions for authors

1.What explains the discrepancies between Tab.s 1/7, 3/8, and 2/4?

2.What is the exact validation-score formula for each scenario? Does agreement reference frozen f0f_0 or the adapted peeking branch, and is it multiplied by peeking performance? How is agreement defined for multilabel classification and segmentation, and which validation labels are used?

3.How much improvement remains over simple fusion or modality dropout with identical old-label replay and matched capacity, training, and search budgets? For peeking, does the gain persist against an old-modality-only control matched in architecture and optimization, thereby isolating the contribution of observing the new modality?

4.Do Tab.s 1–3 evaluate a shared checkpoint or separately optimized models?

5.What explains the performance degradation in the DFC2020 S2→S1 addition and peeking settings? Does adaptation correct teacher errors or primarily preserve teacher predictions, and how does this behavior depend on teacher quality and modality informativeness?

Please refer to weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bars (1/2)

We thank the reviewer for the careful review and the questions. We address the reviewer's concerns through additional experiments, protocol clarification, and manuscript adjustments.

1. On cross-table performance difference:

a. The submission's EuroSAT tables in the Appendix report results obtained with an outdated experiment protocol. We've updated the tables with the main text protocol, so now the RGB->VRE runs on EuroSAT are internally consistent across main text and the appendix. We thank the reviewer for catching this.

b. Table 4 (Ablation) reports slightly different numbers from Tables 1–3 because of its independent tuning of hyperparameters (which ensures fairness for the ablation configurations), while Tables 1–3 use structural hyperparameters tuned once on DFC2020 S1→S2¬rgb. We regenerated Table 4 from the logged sweeps with clarified protocol, with each column reporting the run with the highest validation score, and have updated the table caption and Section 5.1 for clarification.

On validation score: We select DeluluNet transfer and addition based on the simple agreement between DeluluNet and the frozen teacher f0f_0. We have updated Figure 7 and Appendix C with additional results and discussion, alongside correcting the figure caption, which was outdated: agreement is not multiplied by peeking performance, and no pseudo-labels from the adapted peeking branch are used. Specifically: (i) peeking predicts from MAM_A alone, so it is selected by standard validation performance on the labeled uni-modal MAM_A validation split; (ii) transfer (MBM_B input) and addition (MAM_A+MBM_B input) are selected on the unlabeled multimodal validation split by agreement with the frozen f0f_0's predictions on MAM_A; (iii) agreement is top-class agreement for single- and multi-label classification, and per-pixel top-class agreement for segmentation. We thank the reviewer for catching this inconsistency.

2. On isolating modality hallucination and supervised replay: We'd like to emphasize that DeluluNet's modality hallucination mechanism is precisely what makes the (pseudo-) supervised replay possible. The modality-dropout alternative is the "+Masking Token" row of Table 4: it replaces masking transformers with per-modality learned mask tokens (i.e., modality dropout) while keeping the same old-label replay via batch mixing, with an independently tuned set of hyperparameters. It trails DeluluNet in all three scenarios.

3. On parameters and architecture-controlled supervised baseline for DeluluNet's peeking mode. We thank the reviewer for this valuable suggestion. We report the performance of an architecture-matched supervised control in a new Appendix section H, where we show that DeluluNet's improvement from modality peeking can be cleanly separated out from the added parameters and additional supervised training: the control (same architecture, same initialization, same training steps as f0f_0, at a tuned learning rate) does not improve on f0f_0 while DeluluNets do, and every DeluluNet seed scores higher than every control seed.

On related work in cross-modal distillation. We agree, and have corrected the related-work characterization (L125–129): some cross-modal distillation methods such as UniDistill (Zhou et al., 2023) assume labeled multimodal data, while Gupta et al. (2016) and Zhao et al. (2018) use paired unlabeled modalities to train a model on the new modality, but do not learn a fused multimodal model or address modality addition and peeking.

AuthorsOpenReview ↗

Response to Reviewer Bars (2/2)

4. One shared vs. separate weights for the three scenarios. Our main tables report separately trained checkpoints for each of transfer, peeking, and addition; masking and batch-mixing ratios are set per scenario (tuned once on DFC2020 S1→S2¬rgb). However, the point of having a shared set of weights is interesting, practical, and possible with DeluluNets. Here we report the performance of how a set of shared weights can perform well for all three scenarios if we select by average validation score across the three scenarios (new Appendix G). Although the shared weights generally trail the scenario-specific DeluluNet checkpoints from Tables 1–3, the shared weights still win against baselines and f0f_0 on 7/9, 7/9, and 8/9 of the tested dataset-modality pairs, even though each baseline is tuned and trained for its own scenario, showing that DeluluNet can achieve good performance for all three scenarios using one set of weights. We thank the reviewer for the insightful question, which led to this additional experiment that meaningfully strengthens the case for deploying a single DeluluNet.

5. For DFC2020 S2->S1, Sentinel 1 (S1) does not provide sizable value in addition to Sentinel 2 (S2). This is evidenced by observing that even the label-oracle supervised multimodal S1+S2 checkpoint only achieves +1.4 mIoU compared to the S2 f0f_0. As a result, there's not much for DeluluNet to learn from the unlabeled S1. This issue is worsened by our hyperparameter selection strategy, where we directly use the hyperparameters selected with DFC2020 S1->S2_noRGB to save computational cost across the many dataset and modality pairs we experimented with. We have added an analysis on this specific dataset-modality pair in Appendix F, where we tune hyperparameters for this dataset and modality pairing, and show that although DeluluNet can't improve f0f_0 with S1, it largely preserves performance (within 1 mIoU), and still has a smaller drop than any baseline (MixMatch −6.1, MKE −1.0), while tying KD on transfer.

On whether adaptation corrects teacher errors or primarily preserves teacher predictions: DeluluNet is partly trained through distillation, which preserves much of the teacher's predictions. However, DeluluNet exceeds f0f_0 in 8 of 9 settings for both peeking and addition (Tables 2–3), so it is correcting errors made by f0f_0. These gains over f0f_0 come from the unlabeled multimodal data.


We thank the reviewer for the careful reading and the questions, which led to additional experiments on shared DeluluNet weights for all three scenarios, and a robustness analysis when the new modality is not useful. We believe these changes should address the reviewer's concern, and would like to hear the reviewer's feedback on the revision.

As the response phase is coming to a close, we wanted to check whether there is anything further we can address or discuss. We believe our rebuttal and revisions have addressed your concerns, but we would be glad to answer any remaining questions. Thank you for your time and feedback.

Reviewer_pVnyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author analyzed three scenarios where remote sensing machine learning models need to adapt to new modalities, including Modality Transfer (substitution), Addition (superset), and Peeking (subset). The proposed DeluluNet that can handle these three types of changing modality scenarios within an end-to-end framework.

Critical strengths
  1. The paper focuses on the adaptation problem of remote sensing models when the input modalities change, which has practical application value.
  2. DeluluNet uses masking transformers to predict the missing modality representations. The design of the method is reasonably related to the three changing modality scenarios.
Critical weaknesses
  1. The comparative experiments only cover few baselines and are insufficient in comparing the recently proposed methods. Besides, some experimental results are inconsistent, for example, the corresponding results in Tables 1-3 and Table 4 of the reBEN dataset are different.
  2. Although incorporating the three changing modality scenarios into a unified framework has certain value, the novelty of the proposed method is limited. It is more an application of existing technologies under the proposed changing modality settings.
Questions for authors
  1. The author needs to clarify the inconsistency between the complete model results in the main result and the ablation table.
  2. Under certain experimental settings, the model performance has deteriorated compared to the initial model. The reasons for this should be analyzed and explained.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pVny (1/2)

We address the reviewer's questions through tuning protocol clarification and manuscript adjustments.

1. On cross-table performance difference: The difference between the main results in Tables 1–3 and the ablations in Table 4 is from hyperparameter tuning settings. As we describe in Section 5 (Optimization and hyper-parameter tuning), Tables 1–3 report DeluluNets trained with structural hyperparameters (masking ratios and batch-mixing ratio) tuned on one dataset and one modality pairing (DFC2020 with Sentinel-1 as MAM_A and S2¬rgb as MBM_B), with only learning rate and loss weights tuned per dataset and modality pair. Table 4 (Ablations) aims to ablate DeluluNet components, which is why reusing the hyperparameters tuned jointly for the full DeluluNet recipe would be unfair to the four ablation configurations. All ablation runs in Table 4, including full and ablation configs of DeluluNets, report performance on independently tuned hyper-parameters over a shared search space, and each column reports the run with the highest validation score. We have regenerated Table 4 from the logged sweeps with this explicit selection rule, and adjusted the writing in the Table 4 caption and the loss ablation paragraph in subsection 5.1 to clearly reflect the hyper-parameter tuning protocol for the ablation study. We hope our response has clarified the experiment and the manuscript revision has addressed the concern.

2. On cases where Delulu Peeking and Addition lose to f0f_0: Two dataset-modality pairings previously showed DeluluNet achieving lower performance than f0f_0: reBEN S1->S2 and DFC2020 S2->S1.

a. The reBEN S1->S2 degradation is due to a slight bug in our experiments: the DeluluNets and the baselines needing teacher models (KD, TTM, MKE) reported in the main text Tables 1–3 were silently trained with a weaker f0f_0 since the best f0f_0 checkpoint file was corrupted. We have retrained DeluluNet and these baselines for this pairing, and updated the manuscript. DeluluNets outperform f0f_0 and all baselines with the teacher checkpoint fixed. We thank the reviewer for helping us catch this bug.

b. DFC2020 S2->S1 is a more interesting case. For this particular dataset, Sentinel 1 (S1) does not provide any sizable value in addition to Sentinel 2 (S2). This is evidenced by observing that even the label-oracle supervised multimodal S1+S2 checkpoint only achieves +1.4 mIoU compared to the S2 f0f_0 (67.0 → 68.4). As a result, there's not much for DeluluNet to learn from the unlabeled S1. This issue is worsened by our hyperparameter selection strategy, where we directly use the hyperparameters selected with DFC2020 S1->S2_noRGB to save computational cost across the many dataset and modality pairs we experimented with. We have added an analysis on this specific dataset-modality pair in Appendix F, where we tune hyperparameters for this dataset and modality pairing. This is how machine learning engineers would tune hyperparameters in practice, and these experiments show that DeluluNet would select hyperparameters that minimize the effect of the uninformative modality, and as a result largely preserve the already-strong f0f_0: within 1 mIoU on both peeking (−0.8) and addition (−0.6), a smaller drop than any baseline (MixMatch −6.1, MKE −1.0), while tying KD on transfer.

AuthorsOpenReview ↗

Response to Reviewer pVny (2/2)

3. On more baselines. Because the changing modality setting has not been studied before this paper (to the best of our knowledge), there aren't many baselines that can directly be applied in our setup. We compare against five label-free baselines covering each scenario's closest sub-field (KD and TTM for transfer, MixMatch and FreeMatch for peeking, MKE for addition), plus three supervised label-oracles (DINOv3, Panopticon, OlmoEarth). We have also extended the related work in Appendix A with recent methods (kindly provided by reviewer AMze) developed specifically for remote sensing such as multimodal (geographical) domain adaptation methods, and discuss how they differ from our setting: they require labeled data over a fixed set of input modalities, adapting to geographic or domain shift rather than to a change in sensing modality. As a result, they cannot be trained when the new modality comes without labels, which makes them orthogonal to our setting rather than applicable baselines.

4. On Novelty. Although all independent layers use existing techniques (e.g. masking, cross-attention), we'd like to clarify that DeluluNet itself is a new architecture with novel adaptive and conditioned computation paths designed for multi-modal remote sensing models where modalities can be missing at test time, and prediction is possible with completely unlabeled modalities alone. No existing single architecture addresses the transfer, addition, and peeking scenarios.


We thank the reviewer for the careful reading and the questions, which led to a correction in our experiment (which strengthened DeluluNet results), and a robustness analysis when the new modality is not useful. We believe these changes should address the reviewer's concern, and would like to hear the reviewer's feedback on the revision.

As the response phase is coming to a close, we wanted to check whether there is anything further we can address or discuss.

We believe our rebuttal and revisions have addressed your concerns, but we would be glad to answer any remaining questions. Thank you for your time and feedback.

Thank you for the detailed responses. However, some experimental inconsistencies still remain in the revised manuscript (e.g., EuroSAT RGB→VRE Transfer is reported as 90.6 in Tables 1 and 7 but 96.0 in Table 11), which continues to affect my confidence in the results. Besides, while I think the motivation to unify multiple scenarios is meaningful, the method's components are mostly adaptations of established techniques. After considering the rebuttal and other reviewers' comments, I will maintain my original score.

Reviewer_AMzeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

First, the paper proposes to study the different scenarios of updating a model to new modalities in remote sensing. The paper identifies three scenarios (modality transfer, sustitution and peeking) that all belongs to transfer learning paradigm. He explains how these real-world Earth observation scenarios relate to the various problems studied in machine learning The paper also propsoe a new architecture DeluluNet handling the three scenarios. This architecture can be trained end-to-end in a semi-supervised setting where labels are only available for one of the two modalities. The approach is based on the prediction of missing modalities representation from the present ones thanks to a cross attention module. The proposed architecture is evaluated on three datasets (EuroSAT, DFC2020 and reBEN) for the 3 scenarios. Important design choices and core technical contributions are correctly ablated, showing their actual contribution to overall performance.

Critical strengths
  • The initial idea is good and sensible. We are seeing an increasing number of new sensors in Earth Observation.
  • The paper is clear and well written.
  • The paper provides experiments on three different datasets for 3 scenarios. The performances of some recent models such as DinoV3 or Panopticon are reported in the table for comparisons.
  • The architecture is based on recent modules from computer Vision (ie attention module and transformer). In particular, the cross-attention module is used for predicting missing modality.
  • The Appendix A about the adjacent fields is very important and useful to understand the difference with well know topics such as continual learning, domain adaptation and learning from previleged information.
Critical weaknesses
  • Several experiments are conducted using subsets of Sentinel-2 bands. Therefore, this is not really a different sensor. There may be a very strong correlation between the different Sentinel-2 bands from a spatial, spectral and temporal point of view.
  • It should be noted, however, that retraining a new model using annotated data is still several orders of magnitude less expensive than developing and launching a new satellite.
  • The term “hallucinations” can be confusing even it comes from (Hoffman et al. 2016) beacause it can refer to other issues in computer vision. "Missing modality prediction" could be a replacement name.
  • It would have been interesting to see more comparisons with methods from semi-supervised learning [6,7] or domain adaptation learning (such as [3,4,5]).
  • On EuroSAT the performance reachs 99.0 Acc, one might wonder if the dataset is not simply saturated.
  • Comparison with SOTA fondation models are missing to really showing the performance that could be reach on those datasets. For instance [1], (Fuller et al., 2023), [2] reach better performances on EuroSAT dataset.

Readability :

  • Page 2 : D_l, D_a and D_e should be introduced and also used for Figure 1.

[1] Wang, Y, et al. "Towards a unified copernicus foundation model for earth vision." 2025 ICCV. [2] Wand, Y. et al. "Multi-label guided soft contrastive learning for efficient Earth observation pretraining". 2024 IEEE Transactions on Geoscience and Remote Sensing [3] Y. Li, et al. (2022) DKDFN: domain knowledge-guided deep collaborative fusion network for multimodal unitemporal remote sensing land cover classification. ISPRS Journal of Photogrammetry and Remote Sensing [4] Hong, D., et al. "Cross-city matters: A multimodal remote sensing benchmark dataset for cross-city semantic segmentation using high-resolution domain adaptation networks." Remote Sensing of Environment (2023) [5] Li, G., et al. "A multimodal cross-domain segmentation network of remote sensing imagery with multilevel deep cross-fusion and adversarial domain adaptation." IEEE Transactions on Geoscience and Remote Sensing 63 (2025) [6] Luo, Han, et al. "Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing." arXiv preprint arXiv:2609.13654 (2026). [7] Zhou, Yi, et al. "Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion." IEEE Transactions on Geoscience and Remote Sensing 64 (2026)

Questions for authors

Clarification requests:

  • Could you please explain in more detail how the term “hallucination” is used in the context of predicting a missing modality based on nearby data?
  • What is your explanation / intuition for the model's ability to predict the missing modality?
  • Is it necessary for the data to be synchronized in time or space at any point during the training ?
  • On EuroSAT is it the Overall Accuracy (OA) or only accuracy that is reported ?
  • Another limitation that has been mentioned is the difference in spatial resolution between sensors. What would you suggest in that case?

Suggestions for future work:

  • It could be interesting to try the proposed methods with other sensors for instance LandSat or SPOT (with PASTIS-HD for instance (Astruc et al. 2024)) or Planet images or aerial data [A].
  • Quantitative and qualitative evaluation coule be maded in classical dataset such as "Vaihingen, Potsdam datasets" [B,C] or datasets from [3] or [4].
  • Experiments with hyperspectral image for the peeking scenario as mentionned in the text, should be added.

[A] Liu, K., Wu, A., Wan, X., and Li, S.: MRSSC: A BENCHMARK DATASET FOR MULTIMODAL REMOTE SENSING SCENE CLASSIFICATION, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLIII-B2-2021 [B] https://www.isprs.org/resources/datasets/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx [C] https://www.isprs.org/resources/datasets/benchmarks/UrbanSemLab/2d-sem-label-vaihingen.aspx

4 comments
AuthorsOpenReview ↗

Response to Reviewer AMze (1/2)

We thank the reviewer for the positive feedback and the questions. We address the reviewer's questions and concerns through clarification and manuscript revision.

On modality hallucination's naming, data, and mechanistic analysis. DeluluNet's masking transformers predict the latents of unavailable modalities using available modalities. There is shared information to be inferred between such pairs, because these are geo-referenced data pairs aligned in space and close in time. Specifically, the sub-band groups of Sentinel-2 from EuroSAT are from the exact same acquisition, so the time aligns perfectly; the paired Sentinel-1 and Sentinel-2 acquisitions for reBEN can be up to 5 days apart; and the DFC2020 S1 S2 data were only enforced to be collected from the same season. This shows that DeluluNets do not require perfect temporal alignment to work, but only spatial co-registration of the training pairs. Critically, instead of predicting the missing modality in the input space, we jointly train the network to enable cross-modality prediction in the latent space. This is important because this avoids enforcing prediction targets that might be physically impossible because of the sensors' differences in physical property, but encourages a latent prediction (see Figure 4, and Table 6) that is useful for downstream prediction.

We acknowledge the reviewer's concern that "hallucination" in modern deep learning usually refers to the undesirable artifact in language (and sometimes computer vision) modeling. However, as the reviewer also pointed out, we inherit this term from (Hoffman et al., 2016), and it is a fitting name for DeluluNet's masking transformer mechanism.

On EuroSAT dataset and metric. We report Overall Accuracy (OA) on EuroSAT. Since EuroSAT is nearly class-balanced, OA and macro accuracy (mean of per-class accuracy) differ very little.

We agree that EuroSAT is a mostly saturated dataset under standard evaluation (full-S2 band), which is why we've chosen to focus on the harder datasets in our main text. This also applies to the numbers reported by [1], [2], etc., which are obtained with labels on the evaluated bands (typically full S2 or RGB, which is a known strong subgroup for EuroSAT), whereas our setting has no labels for the new modality. The comparable SOTA references are the supervised RSFM oracles in Tables 1–3 (Panopticon, OlmoEarth), which receive target-modality labels. However, we note that sub-group transfer has not been shown before, and EuroSAT, as one of the most popular remote sensing benchmarks, is still suitable and informative there (Appendix E, Tables 7–9).

On Sentinel-2 band subsets as new sensors. We agree that the Sentinel-2 band-group settings are a milder shift than a new sensor. For this reason, our main text has focused on reBEN and DFC2020 settings, pairing Sentinel-1 SAR with Sentinel-2 optical imagery in various directions. Figure 4 (right) quantifies how different they are: the patch-level Pearson correlation between real S1 and real S2 tokens is only 0.138. Even so, masking transformers produce hallucinated tokens that correlate with the real ones at 0.951 (S2) and 0.795 (S1). DeluluNet's gains hold in these cross-sensor settings.

On Different Resolution Inputs. We agree that future work on adapting between modalities of different resolutions is an interesting and practical research direction. DeluluNet can be applied to such cases with very little change for token alignment in the prefusion and latent distillation objectives. Examples of such changes include doing feature upsampling on the lower resolution modality with e.g. AnyUp (Wimmer et al., 2026), downsampling high-res modality with bilinear interpolation, or directly encoding different resolution modalities at different patch sizes (Beyer et al. 2023; Herzog et al. 2026).

AuthorsOpenReview ↗

Response to Reviewer AMze (2/2)

On labeling cost vs. satellite cost, and DeluluNet scope. While we agree that the cost of launching a satellite outweighs labeling efforts, we note that new remote sensing data products are made available at a rate that would make relabeling and training from scratch too expensive for the average practitioner. We would like to clarify that the scope of DeluluNet is to show that we can transfer/peek/add modalities without new labels, and DeluluNets are not limited to the new-label-free regime. When some labels for the new modalities are available, DeluluNets can easily incorporate and learn from those labels through supervised loss, and even batch mixing (by showing only the new modality and training with supervised loss). Labels for new modality or paired modalities can also help in model selection. This is an important point on the scope and future direction of the paper, and we have added a paragraph to Section 6 (Discussion) titled "Beyond the label-free regime".

On recent remote sensing domain adaptation methods. We thank the reviewer for the specific pointers. These methods represent the standard static multimodal fusion methods, and highlight the novel and practical setup of our changing modalities scenarios. DKDFN [3], HighDAN [4], MC-Seg [5], and DROS [6] all require labeled multimodal supervision on the full modality set, with [4], [5], and [6] putting specific focus on shifts in geographical location, but not modalities; CoS2 [7] proposes to use a CLIP-DINO student pair for pseudo labeling in semi-supervised learning, but only works on RGB modality and does not consider train-test modality shift. We have added further discussions of [4], [5], and [6] to the Domain Adaptation paragraph of Appendix A to highlight the related works specifically developed for remote sensing, and discuss the differences between these works and the proposed setting.

On readability. DlD_l, DaD_a, and DeD_e are introduced in Section 2 (L78–80), before the three scenarios are defined. We have updated figure 1 to include the data split annotations there.

On Future extensions to DeluluNet and the changing modalities setting. We thank the reviewer for the good pointers for future steps; we agree these are exciting future directions, and have extended the "Adapting foundation models to new sensor modalities" paragraph in Section 6 accordingly.


References

Wimmer et al. 2026 "Anyup: Universal feature upsampling" ICLR

Beyer et al. 2023 "Flexivit: One model for all patch sizes." CVPR

Herzog et al. 2026 "OlmoEarth: Stable latent image modeling for multimodal earth observation" CVPR

Thank you for your answer. I have a few remarks.

On modality hallucination's naming, data, and mechanistic analysis. You should You should clearly specify this distinction regarding the term “hallucination” to avoid confusion between the different uses of the term in the LLM and computer vision communities.

On Sentinel-2 band subsets as new sensors. I remain convinced that using different Sentinel-2 bands (which are very strongly correlated) as a new sensor severely limits the conclusions that can be drawn and the transferability of the results to situation with actual different sensors. The idea and proof of concept you present are very good, but unfortunately they are supported only by experiments on Sentinel 1 and Sentinel 2.

On Different Resolution Inputs. Recent architectures such as Scale-MAE or UniverSat are a good solution for natively handling different resolutions without upsampling or downsampling (either traditional or machine-learning-based).

Reed, C. J., Gupta, R., Li, S., Brockman, S., Funk, C., Clipp, B., ... & Darrell, T. (2023, October). Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 4065-4076). IEEE. Perron, Y., Astruc, G., Gonthier, N., Mallet, C., & Landrieu, L. (2026). UniverSat: Resolution-and Modality-Agnostic Transformers for Earth Observation. arXiv preprint arXiv:2606.23503.

AuthorsOpenReview ↗

Follow-up: non-Sentinel sensor with a large resolution gap

Thank you for the suggestion on including more sensors and resolutions. We agree these are settings that can strengthen our submission.

We have performed additional experiments on TreeSatAI, a multi-label benchmark pairing 0.2 m aerial RGB with 10 m Sentinel-2: i.e. different sensors with a ~50× resolution gap. Both modalities are resampled to a shared 96×96 grid for simplicity, by upsampling Sentinel-2 16× (nearest-neighbour, so each token corresponds to one 10 m pixel) and downsampling aerial RGB ~3.2× (anti-aliased bilinear).

One DeluluNet per direction (aerial RGB ↔ Sentinel-2), selected by mean validation score over the three scenarios, is compared against the baselines tuned for each scenario. Here we report mean ± std over 3 seeds of weighted F1 (wF1, as standard for TreeSatAI); a dash denotes non-applicable.


MAM_A = aerial RGB (0.2 m), MBM_B = Sentinel-2 (10 m)

f0f_0 (MAM_A only): 52.9

MethodTransferPeekingAddition
KD49.6 ± 0.9--
TTM50.4 ± 0.9--
MixMatch-45.4 ± 0.9-
FreeMatch-47.8 ± 1.0-
MKE--53.6 ± 0.4
DeluluNet (ours)50.9 ± 0.453.1 ± 0.555.4 ± 0.8

MAM_A = Sentinel-2 (10 m), MBM_B = aerial RGB (0.2 m)

f0f_0 (MAM_A only): 53.8

MethodTransferPeekingAddition
KD40.5 ± 0.3--
TTM39.9 ± 0.5--
MixMatch-46.8 ± 0.9-
FreeMatch-52.3 ± 0.9-
MKE--54.2 ± 0.1
DeluluNet (ours)41.9 ± 0.554.9 ± 0.154.8 ± 0.9

Bold = best method per scenario.

The single DeluluNet model is the best method in every scenario, in both directions.

This adds to our existing experiments on cross-sensor changing-modality scenarios between Sentinel-1 and Sentinel-2, and shows the effectiveness of DeluluNet beyond Sentinel sensors.

We thank the reviewer for sharing our interest in evaluating DeluluNet on more sensors at different resolutions, which led to this additional experiment that shows the generalizability of DeluluNet. We will revise the manuscript to include this new benchmark.

Comments on the paper

3 comments
Area_Chair_qEkVOpenReview ↗

[Open Discussion] Thank you for your review and please check authors' rebuttal

Dear All Reviewers,

We have received mixed opinions on this paper. Thank you all for your thoughtful reviews and constructive feedback. Currently, it is still in the open discussion phase, so reviewers are welcome to have further discussions as well as the authors.

Could you please review the authors' rebuttal and share your updated opinions on whether your concerns have been adequately addressed? Please also let us know if you have any remaining questions or concerns that require further clarification.

Dear Reviewer AMze,

You provided a positive assessment of this paper. Could you please confirm whether the authors' rebuttal has adequately addressed your remaining concerns, such as regarding cross-sensor evaluation and baseline comparisons?

Dear Reviewers pVny and Bars,

You both expressed reservations about this paper. Could you please check whether the authors have provided sufficient evidence and clarification to address your concerns, such as experimental inconsistencies and performance degradation in certain settings?

Thank you all again for your time and valuable contributions to the review process.

Best regards,
AC.

I thank the authors for their detailed rebuttal, which addresses my concerns. However, the evaluation is still somewhat limited (using different groups in the Sentinel-2 bands) given the paper's claims to truly make the proposed method a ready-made solution for the proposed cases of changes in modalities. I don't think I'll raise my rating for that reason.

AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers and the AC for their constructive feedback. The main revisions are summarized below, and are highlighted in blue in the revised manuscript for ease of reading.

  • Updated results and protocol (pVny, Bars) for reBEN S1→S2 in Tables 1–3 show that DeluluNet now outperforms f0f_0 and all baselines on this pairing. We regenerated Table 4 and the EuroSAT appendix tables, and clarified the tuning and validation protocols (Section 5.1, Appendix C).
  • DFC2020 S2→S1 (pVny, Bars), new Appendix F analyzes DeluluNet behavior when the new modality is uninformative. DeluluNet tuned for the pairing largely preserves the already-strong f0f_0 (within 1 mIoU), outperforming every baseline.
  • Architecture-matched control (Bars), new Appendix H compares DeluluNet against a supervised control with the same architecture, initialization, and training budget. The control does not improve on f0f_0, while every DeluluNet seed outperforms every control seed. This shows that DeluluNet's peeking gain comes from observing the new modality, not from added parameters or extra training.
  • Single shared model (Bars), new Appendix G shows that training one DeluluNet once, with a single set of parameters shared among the three scenarios, still outperforms f0f_0 and the scenario-tuned baselines on 7/9, 7/9, and 8/9 pairs for transfer, peeking, and addition. A single DeluluNet from one shared training run can therefore handle all three changing-modality settings.
  • Scope and related work (AMze) are expanded with a new Section 6 paragraph, "Beyond the label-free regime", an extended Appendix A covering remote sensing domain adaptation methods, and data splits added to Figure 1. For reference, our evaluation spans three datasets, two sensor types (Sentinel-1 SAR and Sentinel-2 optical), and Sentinel-2 band groups.

More details on each revision, including the corresponding manuscript changes, can be found in our response to each reviewer. We would appreciate any further feedback before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.