acceptodds
Under review as a conference paper at ICLR 2027

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

Abstract

Multimodal remote sensing image generation emerges as a promising paradigm for recovering complete and complementary observations from heterogeneous and partially observed remote sensing data. However, existing methods predominantly depend on sensor-specific fine-tuning of natural image generation models, limiting unified multimodal generation capacity due to the absence of shared representations and robust conditioning mechanisms. To address these challenges, we propose OmniRSI, a unified multimodal remote sensing image generation model. Specifically, we develop a remote sensing-specific variational autoencoder that captures multimodal priors and promotes a locally smooth latent space for unified representation and robust reconstruction across diverse observed data. Building upon that, we further fully finetune Stable Diffusion 3.5 with a ControlNet branch to support unified multimodal generation and translation conditioned on textual and visual cues. Furthermore, we devise a training-free cyclic latent alignment mechanism that iteratively enriches incomplete visual conditions with generated multimodal information, progressively improving guidance reliability while preserving observed measurements. In addition, we construct RSI-MM, a large-scale dataset comprising 11.5 million multi-resolution global images with text annotations and partially paired observations, to enable training and evaluation. Extensive experiments verify that OmniRSI achieves superior performance in multimodal reconstruction, generation and translation across diverse scenarios.

Primary area:
generative models
Keywords:
Remote Sensing, Image Generation, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_rgAMOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OmniRSI, a unified latent diffusion framework for multimodal remote sensing image generation under heterogeneous and partially observed inputs. The method first trains an RS-VAE to map different remote-sensing modalities into a shared latent space, and then adapts Stable Diffusion 3.5 with ControlNet to support both text-to-image generation and cross-modal translation. It further introduces Cyclic Latent Alignment (CLA), a training-free sampling strategy that progressively completes missing modalities using intermediate predictions and feeds the enriched visual condition back into the denoising process. The authors also construct RSI-MM, a large-scale multimodal dataset containing about 11.5M images from 16 public datasets. Experiments on reconstruction, text-to-image generation, and cross-modal translation show substantial improvements over existing methods, including zero-shot transfer to unseen modality pairs.

Critical strengths
  1. Well-motivated problem formulation. The paper targets a practically important problem in remote sensing: unified generation under heterogeneous and partially observed multimodal inputs. The motivation is clear and well justified.
  2. Comprehensive and convincing experiments. The evaluation covers reconstruction, text-to-image generation, and cross-modal translation, with extensive comparisons and ablations. The consistent improvements across different tasks and modalities provide strong empirical support for the proposed framework.
  3. A coherent unified framework. Rather than designing separate models for individual modality pairs, OmniRSI integrates multimodal representation, generation, and translation within a shared latent space, providing a clean and scalable formulation for heterogeneous remote-sensing modalities.
Critical weaknesses

The main weakness is that the performance gains are not fully disentangled from model and data scale.

Questions for authors
  1. OmniRSI uses a substantially larger backbone and training corpus than several baselines. Could the authors provide additional analysis or controlled experiments to clarify how much of the performance gain comes from the proposed unified formulation (RS-VAE/CLA) rather than increased model and data scale?
  2. CLA progressively fills missing modalities using intermediate predictions and feeds them back as visual conditions. How sensitive is this process to inaccurate predictions at early denoising steps? Could prediction errors in the missing slots accumulate or negatively affect the subsequent denoising trajectory?
  3. The zero-shot PAN–SAR experiments suggest that OmniRSI can generalize to unseen modality pairs. Could the authors further clarify what enables this compositional generalization, and whether similar behavior is observed for other unseen modality combinations?
  4. A key motivation of RS-VAE is to establish a shared latent space across heterogeneous modalities. Beyond reconstruction quality and latent-space visualization, could the authors provide more direct quantitative evidence (e.g., cross-modal retrieval performance, latent similarity, or alignment metrics) that semantically corresponding observations from different modalities are well aligned in the learned latent space?
6 comments

Response to Reviewer#4 (rgAM)

We sincerely thank the reviewer for the thoughtful and constructive comments, and for recognizing the practical relevance, comprehensive evaluation, and coherent unified formulation of OmniRSI. Your suggestions have helped us clarify the sources of improvement and strengthen the evidence supporting our design. All additional experiments, analyses, and clarifications will be incorporated into the main text or appendix.

Q1&Q4: Component-wise Ablation

We thank the reviewer for these important questions. We add the controlled RS-VAE ablations, fine-tuned models for generation comparison, and high-capacity translation baselines.

RS-VAE. RS-VAE consistently outperforms the standard VAE and modality-specific autoencoders in reconstruction quality (Tab.d1), while supporting heterogeneous modalities within a single encoder–decoder. For the standard VAE, single-band PAN, SAR, and NIR inputs are replicated across three channels to match its native RGB input format. It combines improved reconstruction fidelity with joint multimodal encoding and decoding, replacing separate modality-specific representations with a shared latent interface for generation and translation. Tab.d2 and Tab.d3 reveal that LmaskL_{mask} improves PSNR and SSIM across all groups and slightly improves latent consistency, whereas adding LsmoothL_{smooth} substantially improves latent consistency. Hence, RS-VAE benefits from slot-based organization with comprehensive remote sensing multimodal data and two compatible losses.

ModelPANRGBSARNIR
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
VAE36.990.83038.380.94025.680.79429.410.946
VAE(Any2Any)37.680.83839.060.95126.850.84629.990.951
RS-VAE42.790.97239.710.95827.150.87832.830.953

Table d1: Reconstruction of VAE variants.

Config.PANOPTSARNIRMSIAverage
VAE baselineLmaskL_{mask}LsmoothL_{smooth}PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
√××41.700.96339.540.95126.910.87731.990.94224.130.82834.300.927
√√×42.840.97739.740.96127.240.87932.950.95527.100.87235.030.937
√×√41.660.96039.440.94726.820.87631.960.94124.110.82334.240.924
√√√42.790.97239.710.95827.150.87832.830.95326.910.86934.950.935

Table d2: RS-VAE reconstruction performance under different loss configurations

VAE baselineLmaskL_{mask}LsmoothL_{smooth}N=0N=1N=2N=3N=4N=5N=6N=7N=8N=9N=10Weighted Avg.
√××1.00000.80110.74140.72990.71740.64780.62730.60890.55490.39620.16850.6428
√√×1.00000.82170.75270.74620.73390.66510.63690.61770.57640.42200.25470.6686
√×√1.00000.94070.93820.90800.82410.75690.73100.69610.59200.54170.36890.7934
√√√1.00000.96320.94130.93820.89830.82120.79800.75060.68590.58430.40260.8316

Table d3: Latent Semantic Alignment Evaluation of excluded bands N

Text-to-image Generation. We select the original SD3.5 Medium, MMDiff and Txt2Img-MHN for additional RGB fine-tuning, as they provide publicly available training implementations compatible with our RGB setting. As reported in Tab.d4, OmniRSI outperforms these additionally fine-tuned baselines. In particular, our configuration achieves better quality than the standard-VAE configuration with the same SD3.5 Medium backbone. Hence, generation benefits from a domain-adapted backbone with a shared multimodal representation by RS-VAE.

Config.Text-to-imageParam.
--FID↓NIQE↓CLIPIQA↑CLIPScore↑(M)
VAESD3.5Medium93.0896.3970.5970.7582139.25
VAESD3.5Medium*45.1335.9070.6000.7662139.25
RS-VAESD3.5Medium*(Ours)28.6934.3990.6950.8302139.25
VQVAETxt2Img-MHN150.0635.3980.3770.67229.33
VQVAETxt2Img-MHN*126.2218.6880.2900.71729.33
VAEMMDiff80.9225.8240.3770.7801066.24
VAEMMDiff*61.7245.6390.3860.7851066.24

Table d4: RGB text-to-image generation performance. * denotes fine-tuned models.

Cross-modal Translation. Leveraging the spatial compression by RS-VAE, we adapt RDBM to the latent space and substantially enlarge its network for unified translation training. OmniRSI outperforms this enlarged baseline across all translation directions without text (Tab.d5). OmniRSI supports these translation directions within one shared model, whereas the original baselines are trained separately for each task. Furthermore, we train the ControlNet branches attached to DiffusionSat and standard-VAE SD3.5 using the available data and associated metadata for RGB-target translation (Tab.d6). SD3.5-based configuration outperforms DiffusionSat, while incorporating RS-VAE further improves metrics across all tasks. Besides, CLA further improves translation performance, as detailed in our response to Q2. Together, these comparisons demonstrate the complementary benefits of backbone adaptation, shared multimodal representation learning and cyclic latent refinement.

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverageParam
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
RDBM21.020.65631.110.88112.160.36718.010.24522.590.61639.17 (Total: 156.68)
RS-VAE+ScaledRDBM21.430.65832.280.88914.370.36919.890.27423.830.6242274.98
Ours (× text)22.070.66235.170.91417.360.37320.320.29125.630.6392789.06
Ours (RGB-derived text)25.160.78636.100.92518.390.38220.500.30227.100.6832789.06

Table d5: Scaling baseline RDBM with RS-VAE.

Config.PAN->RGBNIR->RGBSAR->RGBParam.
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
DiffusionSAT+ControlNet*24.200.73915.890.48216.670.3891682.888
VAE+SD3.5Medium+ControlNet*26.110.85720.330.65618.090.4312789.06
VAE+SD3.5Medium*+ControlNet*27.280.86621.260.68319.020.4632789.06
Ours(RS-VAE+SD3.5Medium*+ControlNet*)29.090.88923.010.73420.240.5222789.06

Table d6: Text-free translation comparisons with models of comparable parameter counts. For configurations with a fine-tuned SD3.5 backbone, generation fine-tuning precedes ControlNet training for translation. * denotes fine-tuned models. CLA is disabled.

Q2: Robustness of CLA

We thank the reviewer for raising this question. We evaluate the same trained model with and without CLA across five random seeds. Tab.d7 reports μ±σ2\mu \pm \sigma^{2}, with the coefficient of variation CV=σ/μ×100%\mathrm{CV}= \sigma /\mu\times100\% in parentheses, where μ\mu and σ\sigma denote the mean and standard deviation across seeds.

CLA improves mean PSNR and SSIM while reducing both variance and CV across all four translation directions, demonstrating improved fidelity and reduced sensitivity to sampling randomness. This stability is consistent with its annealed condition-update mechanism, which limits reliance on generated completions early in sampling and progressively incorporates their information while preserving observed measurements. These results support the robustness of CLA’s improvements across the evaluated seeds.

Config.PAN→MSIMSI→PANSAR→MSIMSI→SAR
CLAPSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
×21.772±1.3965(5.4278%)0.661±0.0008(4.2790%)34.589±1.2959(3.2911%)0.913±0.0018(4.6471%)17.352±1.8143(7.7626%)0.369±0.0024(13.2793%)20.256±0.5701(3.7275%)0.280±0.0006(8.7471%)
√22.032±0.2992(2.4830%)0.662±0.0001(1.5107%)35.227±0.2488(1.4158%)0.914±0.0001(1.0941%)17.577±0.4913(3.9881%)0.373±0.0001(2.6809%)20.43±0.3233(2.7831%)0.291±0.0002(4.8596%)

Table d7: Translation performance with and without CLA across five random seeds.

Furthermore, we compare CLA with two controlled variants: (i) direct latent feedback, which directly uses the estimated clean latent while retaining CLA’s annealed interpolation with the initial condition; and (ii) unannealed condition replacement, which preserves observation-space completion and re-encoding but immediately adopts the resulting latent without annealed interpolation. As shown in Tab.d9, (i) improves over fixed conditioning but remains modestly below CLA, whereas (ii) substantially degrades performance. This degradation may arise because hard replacement allows unreliable early predictions to bias the jointly encoded condition, diverting the sampling trajectory and impairing reconstruction. Preserving measured values in observation space alone does not prevent errors in predicted modalities from distorting latent guidance. CLA’s annealed condition updates mitigate this risk by anchoring early conditioning to the original observations and progressively incorporating generated complementary information. Together, these results support the complementary roles of observation-preserving completion and gradual feedback incorporation

MethodCoreupdateRefinementObject
(i)Direct latent feedbackZ^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>ψt−=λtψ1+(1−λt)Z^0t\boldsymbol{\psi_{t^-}=\lambda_t\psi_{1}+(1-\lambda_t)\hat Z_0^t}<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})No <u>slot-replacement<u>Condition
(ii)Condition replacementZ^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>X~t=M⊙X+(1−M)⊙D(Z^0t)‾\underline{\tilde X_t=M\odot X+(1-M)\odot D(\hat Z_0^t)}<br>ψt−=E(X~t)\psi_{t^-}=E(\tilde X_t)<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})No annealingCondition
(iii)CLA(Ours)Z^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>X~t=M⊙X+(1−M)⊙D(Z^0t)‾\underline{\tilde X_t=M\odot X+(1-M)\odot D(\hat Z_0^t)}<br>ψt−=λtψ1+(1−λt)E(X~t)\boldsymbol{\psi_{t^-}=\lambda_t\psi_1+(1-\lambda_t)E(\tilde X_t)}<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})Multimodal completion+<br>Annealed condition updateCondition

Table d8: Implementations of CLA variants

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverage
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
× CLA21.730.66034.770.91017.090.36519.540.27825.210.632
(i)22.030.66135.020.91317.170.36920.110.28725.500.636
(ii)11.750.38015.980.4748.7680.1329.5010.14912.360.331
(iii) CLA22.070.66235.170.91417.360.37320.320.29125.630.639

Table d9: Performance of CLA variants without text prompts

Reviewer_HxiuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents OmniRSI, a single latent diffusion model for reconstruction, text-to-image generation, and translation across PAN, MSI, and SAR imagery. Observations from different sensors are packed into 18 fixed band "slots" with an availability mask, and a new VAE (RS-VAE) is trained on these with masked reconstruction plus a smoothness term. SD3.5 (Stable Diffusion 3.5) medium is then fully fine-tuned on the resulting latents for text-to-image, and a ControlNet is trained on top for translation from available modalities. At inference, the authors add Cyclic Latent Alignment (CLA): at every step it decodes the predicted clean image, fills in the missing slots, and re-encodes the result to update the condition. They also put together RSI-MM, about 11.5M images from 16 public datasets, with GPT-generated captions for sources that had no text.

Critical strengths

(S1): The slot-and-mask idea is simple, and I think it’s the most useful part of the paper. One model can take in any combination of sensors happens to be available. OmniRSI can generate MSI, PAN, and SAR. CLA is easy to add without training, and the authors compare it against a plain sampler at the same number of function evaluations.

(S2): The dataset is also a valuable contribution. The datasheet and split protocol are more thorough than usual, and the authors commit to open-sourcing everything. I also appreciated that the evaluation goes beyond PSNR/SSIM to spectral indices and band-availability experiments.

Critical weaknesses

(W1): My main concern is that I can’t tell the degree of the improvement from the method vs the experimental setup.

In translation, the text prompts seem to carry information about the target. Captions were generated from RGB images, and for PAN->MSI and SAR->MSI the RGB is what the model is supposed to produce. The pattern of gains seems to support this: text adds 3.09 dB on PAN->MSI but only 0.18 dB on MSI->SAR, and NDXI RMSE drops from 0.321 to 0.196. None of the baselines get text.

There are also issues with baselines. 1) The translation baselines are also small (9–60M parameters) and trained from scratch, against a 2.79B model initialized from SD3.5, and some look like they didn't train properly (DDBM’s SSIM is 0.02). 2) The experiments should include SD3.5 + ControlNet with the original VAE, DiffusionSat’s ControlNet for optical targets, and a simple linear regression for MSI->PAN.

For text-to-image, OmniRSI is fine-tuned on data from the same sources as the test set, while the baselines are used off the shelf. Without SD3.5 Medium fine-tuned on the same data with its original VAE, I can’t attribute the improvement in FID to RS-VAE. DiffusionSat also seems to have been run without its metadata inputs and with 256-word captions, which is far from what it was trained on.

Moreover, it seems omega was tuned on the test set, and there was no deduplication across sources. Git-10M is itself built from public datasets, so some test images may have appeared in training.

(W2): My second concern is that the paper claims more than it shows and undersells prior work. DiffusionSat is the closest predecessor but only appears as a row in Table 3. It already fine-tunes SD2.1 on aggregated satellite data and uses a ControlNet with multispectral inputs, and the introduction or related work does not mention this at all. This is a glaring omission - proper contextualization with prior work is important, especially given the degree of similarity in the method's motivation. The real differences should be stated plainly: non-RGB outputs, conditioning on arbitrary modality subsets, CLA, and the newer backbone. I'd also like to know why DiffusionSat's GSD and location conditioning was dropped. RSI-MM spans 0.15–128 m resolution, and the authors already have coordinates.

Also, the "unified generation" claim is only measured on RGB. About 90% of the text-to-image training images are RGB-only, and because the VAE losses are masked, slots that were never observed get no supervision. The sharp sub-meter TIR and SAR in figures A2–A3 don't look physically plausible to me. The authors claim the VAE’s alignment based on a single t-SNE plot, and do not ablate its losses. CLA’s gain at matched compute is small (0.15–0.27 dB, single seed), and the method looks a lot like self-conditioning combined with RePaint-style replacement, neither of which is discussed.

Questions for authors

(Q1): Can the authors report the translation results with captions generated only from the source modality, or with no text, and give the baselines the same text?

(Q2): How does SD3.5 Medium fine-tuned on RSI-MM with its original VAE compare, both for text-to-image and, with a ControlNet, for translation?

(Q3): Why was metadata conditioning (GSD, location) omitted? What inputs did DiffusionSat receive in the Table 3 evaluation?

(Q4): Was any deduplication done across sources, particularly between Git-10M and the test sources? Were neighboring tiles from the same scenes as test samples excluded from training?

(Q5): What is CLA’s variance across seeds, and how does it compare with plain self-conditioning and with replacement-only variants?

10 comments

Response to Reviewer#3 (Hxiu)

We thank the reviewer for the insightful and constructive comments. Following the order of your concerns, we organize our responses into two sections: W1 (Q1–Q4) addresses experimental settings, baseline comparisons, and evaluation protocols; W2 (Q5) discusses connections to prior work, RS-VAE ablations, and CLA’s mechanism and robustness. All additional experiments, analyses, and clarifications will be incorporated into the main text or appendix.

W1&Q1&Q3&Q4: Experimental Setups

Q1: Text Prompts for Translation. We apologize for the lack of clarity and misunderstanding. Tab.5 already reports text-free translation results (in bold), which provide our primary comparison with image-only baselines. The additional text-conditioned results examine the benefit of supplementary semantic information. As described in Appendix A, the default captions are generated from RGB composites and are therefore target-derived for PAN→MSI and SAR→MSI, but source-derived for MSI→PAN and MSI→SAR. We further generate captions using GPT with the same base prompt in Appendix A, augmented with an explicit input-modality description (The image is a panchromatic (PAN) / synthetic aperture radar (SAR) remote sensing image rendered in grayscale) for grayscale PAN and SAR images; MSI captions use RGB composites. Therefore, we evaluate the entire translation test set under three settings: no text, source-derived captions, and target-derived captions. Tab.c1 shows that textual guidance can improve translation performance. Neither source-derived nor target-derived captions consistently dominate. Instead, RGB-derived captions achieve the best performance. This pattern suggests that caption utility depends on the informativeness and reliability of the extracted semantics, reflecting both the information available in each modality and the captioning model’s ability to interpret and express it. Access to target observations therefore does not necessarily produce more effective textual guidance.

Thanks for your suggestion, we promise to provide modality-specific captioning prompts for all PAN and SAR images across the RSI-MM datasets. These additions will strengthen the impact and reuse of RSI-MM, supporting further research on multimodal remote sensing generation.

Source→TargetPAN→MSIMSI→PANSAR→MSIMSI→SARAverage
TextPSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
w/o text22.070.66235.170.91417.360.37320.320.29125.630.639
Source text22.690.75936.100.92517.450.37820.500.30226.200.674
Target text25.160.78635.630.91818.390.38220.360.29326.910.679
Default(RGB-derived)25.160.78636.100.92518.390.38220.500.30227.100.683

Table c1: Translation performance with different captions.

Q2: Translation Baseline Comparisons. Leveraging the spatial compression by RS-VAE, we adapt RDBM to the latent space and substantially enlarge its network for unified translation training. OmniRSI outperforms this enlarged baseline across all translation directions without text (Tab.c2). OmniRSI supports these translation directions within one shared model, whereas the original baselines are trained separately for each task. Furthermore, we train the ControlNet branches attached to DiffusionSat and standard-VAE SD3.5 using the available data and associated metadata for RGB-target translation (Tab.c3). The SD3.5-based configuration outperforms DiffusionSat, while incorporating RS-VAE further improves metrics. Additionally, we evaluate a linear regression baseline for MSI→PAN on PSBench. We fit PAN^=∑iαiMSi+b\widehat{\mathrm{PAN}}=\sum_i\alpha_i\mathrm{MS}_i+b by minimizing the squared error on the training split. This baseline achieves a PSNR of 21.78 dB and an SSIM of 0.815 on the held-out test set, whereas a simple mean MSI yields only 18.03 dB and 0.797.

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverageParam
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
RDBM21.020.65631.110.88112.160.36718.010.24522.590.61639.17 (Total: 156.68)
RS-VAE+ScaledRDBM21.430.65832.280.88914.370.36919.890.27423.830.6242274.98
Ours (× text)22.070.66235.170.91417.360.37320.320.29125.630.6392789.06
Ours (RGB-derived text)25.160.78636.100.92518.390.38220.500.30227.100.6832789.06

Table c2: Scaling baseline RDBM with RS-VAE.

Config.PAN->RGBNIR->RGBSAR->RGBParam.
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
DiffusionSAT+ControlNet*24.200.73915.890.48216.670.3891682.888
VAE+SD3.5Medium+ControlNet*26.110.85720.330.65618.090.4312789.06
VAE+SD3.5Medium*+ControlNet*27.280.86621.260.68319.020.4632789.06
Ours(RS-VAE+SD3.5Medium*+ControlNet*)29.090.88923.010.73420.240.5222789.06

Table c3: Text-free translation comparisons with models of comparable parameter counts. For configurations with a fine-tuned SD3.5 backbone, generation fine-tuning precedes ControlNet training for translation. * denotes fine-tuned models. CLA is disabled.

Q3: DiffusionSat Setting For fairness, all evaluations are conducted under identical text conditions and other necessary inputs, which will be detailed in the text. In addition, we do not apply method-specific template-based prompt transformations to the annotated text. Instead, we uniformly use the same natural-language captions to verify each model’s end-to-end generation performance under a shared prompting protocol. For Tab. 3, DiffusionSat is run with all available metadata (longitude, latitude, GSD, cloud cover, year, month, and day), since omitting metadata causes runtime errors and prevents generation. Source information is used where available, and default values fill missing fields (e.g., unreported cloud cover). For the training in Tab.c3, we preserve the original generation backbone and train the ControlNet by supplying the required metadata. RSI-MM retains auxiliary information, including metadata, segmentation labels, and change-detection annotations. Since our method does not involve the use of these auxiliary annotations, they are not separately reported in the initial manuscript. We will revise the dataset description and evaluation protocol to explicitly report their availability and usage. Besides, our current formulation focuses on shared multimodal generation conditioned on text and available observations. It does not yet address explicit metadata control and temporal sequence modeling, which we identify as a highly promising avenue for future research. We sincerely appreciate the reviewer's valuable insight, which provides significant inspiration for our subsequent work.

DDBM Setting Thank you for raising this concern. Based on the results, the lower SSIM of DDBM is primarily due to the presence of significant noise in the generated images. Initially, we used the default official implementation directly to avoid any suspicion of deliberately tuning the baseline's performance. However, we found that the official default implementation sets the noise-scale parameter to σmax=80\sigma_{max}=80, making the intermediate bridge states strongly noise-dominated relative to the normalized image intensities. The experiment results in Tab.5 indicate that this noise scale was poorly suited to our remote sensing translation setting. We therefore retrain DDBM with σmax=20\sigma_{max}=20, which substantially reduces residual noise and improves reconstruction quality. The corrected results are reported in Tab.c4. This correction does not affect our overall conclusion. Our method is tailored for remote sensing data adaptation and achieves the best performance. We will report this modification in the Appendix.

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverage
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
DDBM*(σmax=20\sigma_{max}=20)18.120.46026.490.73012.410.17918.450.26120.190.474
Ours(× text)22.070.66235.170.91417.360.37320.320.29125.630.639

Table c4: Translation performance of modified DDBM.

Text-to-image Generation Improvement. We select original SD3.5 Medium, MMDiff and Txt2Img-MHN for additional RGB fine-tuning, as they provide publicly available training implementations compatible with our RGB setting. Other baselines adopt distinct conditioning or output configurations: DiffusionSat incorporates metadata, some of which are unavailable across all our subsets; Text2Earth's official training code remains unreleased; CC-Diff relies on object annotations; and HSIGene is tailored for hyperspectral generation. As reported in Tab.c5, OmniRSI outperforms these additionally fine-tuned baselines. In particular, our configuration achieves better quality than the standard-VAE configuration with the same SD3.5 Medium backbone. Hence, generation benefits from a domain-adapted backbone with a shared multimodal representation by RS-VAE.

Config.Text-to-imageParam.
--FID↓NIQE↓CLIPIQA↑CLIPScore↑(M)
VAESD3.5Medium93.0896.3970.5970.7582139.25
VAESD3.5Medium*45.1335.9070.6000.7662139.25
RS-VAESD3.5Medium*(Ours)28.6934.3990.6950.8302139.25
VQVAETxt2Img-MHN150.0635.3980.3770.67229.33
VQVAETxt2Img-MHN*126.2218.6880.2900.71729.33
VAEMMDiff80.9225.8240.3770.7801066.24
VAEMMDiff*61.7245.6390.3860.7851066.24

Table c5: RGB text-to-image generation performance. * denotes fine-tuned models.

Omega Selection We thank the reviewer for raising this concern. We would like to clarify that we empirically set ω=5.0\omega=5.0 for generation before inspecting the test-set results. Tab.4 is not intended for tuning parameters on the test set. It examines the sensitivity of task performance to classifier-free guidance and we will revise its caption to <Generation performance on varied ω\omega.>. Importantly, all six evaluated settings outperform every baseline in Tab.3 across all four metrics, demonstrating that OmniRSI’s comparative advantage is maintained across the tested guidance range rather than depending on a single best-performing setting. We additionally evaluate OmniRSI on the external HRRSD dataset[1], using captions provided by MMM-RS[2]. As shown in Tab.c6, ω=5.0\omega=5.0 again achieves the best performance across all four metrics among the evaluated settings, consistent with the original evaluation.

Guidance ω\omegaFID↓NIQE↓CLIPIQA↑CLIPScore↑
1.053.94795.73550.46690.6924
2.040.69085.63110.49280.7135
3.033.31735.61840.51010.7193
4.029.39095.57760.52040.7601
5.027.60835.23720.52830.8188
6.030.93135.49130.51870.8005

Table c6: Our OmniRSI on the external dataset TGRSHRRSD

[1] Hierarchical and Robust Convolutional Neural Network for Very High-Resolution Remote Sensing Object Detection. TGRS2019.

[2] MMM-RS: A multi-modal, multi-GSD, multi-scene remote sensing dataset and benchmark for text-to-image generation. NeurIPS2024.

Reviewer_Kx1VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniRSI, a unified framework for multimodal remote sensing image generation. Specifically, the authors first develop a remote sensing-specific VAE (RS-VAE) to learn a shared latent representation for heterogeneous and incomplete observations. Based on this representation, SD3.5 is fine-tuned with a ControlNet branch to achieve text-to-image generation and cross-modal image translation. Moreover, a training-free cyclic latent alignment (CLA) strategy is proposed to progressively refine incomplete visual conditions during inference. The authors also construct a large-scale multimodal remote sensing dataset, termed RSI-MM, for training and evaluation. Experimental results on different tasks demonstrate the effectiveness of the proposed framework.

Critical strengths

The paper studies an interesting and meaningful problem. Learning a unified generative model for heterogeneous remote sensing modalities is challenging due to different imaging mechanisms and incomplete observations. The proposed framework provides a relatively comprehensive solution for multimodal reconstruction, generation, and translation.

The overall framework is reasonably designed. In particular, the modality-aware slots and the proposed RS-VAE enable different remote sensing modalities to be represented in a shared latent space. Moreover, the CLA strategy provides a way to progressively update incomplete visual conditions during sampling, which is a useful design for multimodal image translation.

The experimental results are promising. The proposed method achieves better performance than the compared methods on different tasks. The authors also conduct experiments on spectral indices and different sampling solvers, which further demonstrate the effectiveness of the proposed framework.

Critical weaknesses

The technical novelty is not sufficiently clear. Most components of the proposed framework are developed based on existing techniques, including VAE, masked reconstruction, latent regularization, SD3.5, and ControlNet. The main algorithmic contribution appears to be the proposed CLA strategy. However, its relationship with existing iterative self-conditioning and condition refinement methods is not sufficiently discussed. It is unclear whether CLA introduces a fundamentally different mechanism or mainly provides an alternative implementation of iterative condition updating.

The experimental comparisons are not entirely convincing. The proposed method employs a much larger model than most competing methods, especially for image translation.

The effectiveness of several key designs is not sufficiently investigated. The authors introduce masked reconstruction and latent smoothness regularization to learn a unified latent representation. However, the contributions of these two regularizations are not evaluated. In addition, although the proposed CLA improves translation performance, its improvement over the baseline with the same number of function evaluations is relatively limited, as shown in Table 8.

The physical consistency of generated multimodal observations requires further investigation. Different remote sensing sensors may have different spectral responses and radiometric characteristics, even when their observations are assigned to the same modality-aware slot. Although the authors evaluate several spectral indices, these results are still insufficient to fully demonstrate the physical reliability of generated observations.

Questions for authors

What is the main difference between the proposed CLA and existing iterative self-conditioning or condition refinement strategies?

Can the authors provide ablation studies on the masked reconstruction loss and latent smoothness regularization?

Considering the substantial differences in model size and computational complexity, can the authors provide more controlled comparisons to demonstrate that the improvements mainly come from the proposed framework rather than the larger backbone?

How are the textual conditions obtained during the evaluation of cross-modal image translation? Are they generated from the source observations or the target images?

How are the average PSNR and SSIM values in Table 5 calculated? The reported values do not appear to be the arithmetic averages of the four translation tasks. It would be helpful to clarify the evaluation protocol.

9 comments

Response to Reviewer#2 (Kx1V)

We sincerely thank the reviewer for the valuable and constructive comments. Your feedback has helped us clarify our methodological contributions, strengthen the experimental evidence, and improve the transparency of our evaluation. All additional experiments, analyses, and clarifications will be incorporated into the main text or appendix.

W1&Q1: Method Contribution and CLA Novelty

We thank the reviewer for encouraging us to clarify the contribution. We acknowledge that the VAE and diffusion formulation are widely adopted techniques in generative modeling. Our contribution lies not in claiming these building blocks as individually novel, but in systematically adapting the full Stable Diffusion pipeline to address the incompleteness of heterogeneous multimodal remote sensing observations. Representative approaches specialize in particular output modalities and lack a unified framework that integrates shared representation learning, multimodal generation, and cross-modal translation, as shown in Tab.b1. Any2Any (Chen et al., ICML 2026) performs pairwise cross-modal translation using modality-specific VAEs and a shared DiT backbone with target-specific adapters. Txt2Img-MHN(Xu et al. TIP2023), MMDiff(Tang et al. ISPRS2026), Text2earth(Liu et al. GRSM2025), DiffusionSat(Khanna et al. ICLR2024) mainly focus on RGB generation, HSIGene (Pang et al. TPAMI2026) on hyperspectral synthesis, and HuiYanEarth-SAR (Liu et al.) on SAR generation. MMDiff and Text2earth use LoRA adaptation to generate other modalities with extra parameters, whereas DiffusionSat uses ControlNet to incorporate auxiliary information for RGB generation.

MethodRepresentationGenerationTranslationDescription
Any2AnyModality-specific VAEs—one-to-one mappingDiT backbone with target-specific adapters
Txt2Img-MHNVQ-VAE/VQGANRGB—VQ-VAE/VQGAN representations and Hopfield network.
MMDiff—OPT/SAR/IR—Pretrained VAE and LoRA-based modality adaptation
Text2Earth—RGBone-to-one translationExtends SAR, NIR, PAN generation and translation via task-specific lora and controlnet
DiffusionSat—RGBMSI→RGBPretrained VAE, metadata-conditioned generation, with ControlNet-based extensions
HSIGeneHyperspectral VAEHSIHED,Sketch→HSIHyperspectral generation and RGB-guided controlnet
HuiYanEarth-SAR—SAR—Pretrained VAE and metadata conditioned SAR generation using LoRA adaptation
OmniRSI(Ours)Unified RS-VAEMulti-modalUnified translation + CLAMultimodal representation, generation and translation with CLA refinement

Table b1: Comparisons with representative methods.

To the best of our knowledge, OmniRSI is the first remote sensing framework to couple a shared latent representation learned from partially observed multimodal data with unified generation, cross-modal translation, and an iterative condition refinement strategy. Specifically, RS-VAE establishes a shared representation for heterogeneous and incomplete observations, where LmaskL_{mask} improves the reconstruction fidelity (Tab.b9) and LsmoothL_{smooth} promotes the semantic alignment (Tab.b10). Crucially, the resulting representation serves both as the generative target space and as the visual-conditioning interface, enabling the adapted SD3.5 Medium backbone and ControlNet branch to support generation and translation from different modality subsets. CLA updates the explicit conditioning representation by multimodal completion and annealed latent re-encoding, which shares the general principle of iterative prediction feedback with existing methods. Its distinction lies in how the feedback is constructed and where it acts. As summarized in Tab.b2, Analog Bits [1] feeds back clean-sample estimates, whereas ILVR [2], RePaint [3], and DPS [4] directly refine sampling states using reference or measurement constraints. Instead, the novelty of CLA lies in this coupled observation–latent feedback path, rather than iterative updating or any individual operation alone. The difference is not simply that CLA updates a condition, but that it constructs that condition from a combination of actual observations and predicted missing modalities through the shared encoder–decoder. Hence, OmniRSI treats incomplete multimodal observations as a central design consideration throughout representation, generation, and translation, providing a reusable foundation for further research on multimodal remote sensing generation. We sincerely appreciate the reviewer's comment, and we promise to clarify these connections and distinctions in the Related Work and Method sections.

MethodCore updateRefinementObject
Analog Bitsx^0t=fθ(xt,t,x^0prev)\hat x_0^t=f_\theta(x_t,t,\hat x_0^{prev})Improve prediction using previous estimatesCondition
ILVRxt−1=xt−1+ϕN(yt−1)−ϕN(xt−1)x_{t-1}=x_{t-1}+\phi_N(y_{t-1})-\phi_N(x_{t-1})Low-pass filter \phi_N(·) for states correctionState
RePaintxt−1=M⊙yt−1+(1−M)⊙xt−1x_{t-1}=M\odot y_{t-1}+(1-M)\odot x_{t-1}Known-region M replacement in noisy stateState
DPSxt−1=xt−1−ηt∇xt∥y−A(x^0(xt))∥22x_{t-1}=x_{t-1}-\eta_t\nabla_{x_t}\lVert y-\mathcal A(\hat x_0(x_t))\rVert_2^2Measurement-gradient state correctionState
CLA (Ours)X~t=M⊙X+(1−M)⊙Dθ(Z^0t)\tilde X_t=M\odot X+(1-M)\odot D_\theta(\hat Z_0^t)Known-slot replacementCondition
↳ψt−=λtψ1+(1−λt)Eθ(X~t)\psi_{t^-}=\lambda_t\psi_1+(1-\lambda_t)E_\theta(\tilde X_t)and annealed latent condition update
↳Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})

Table b2: Comparisons with representative self-conditioning and iterative refinement methods.

[1] Chen T, et al. Analog bits: Generating discrete data using diffusion models with self-conditioning, ICLR 2023.

[2] Choi J, et al. ILVR: Conditioning method for denoising diffusion probabilistic models. ICCV 2021.

[3] Lugmayr A, et al. RePaint: Inpainting using denoising diffusion probabilistic models. CVPR 2022.

[4] Chung H, et al. Diffusion posterior sampling for general noisy inverse problems. ICLR 2023

Inspired by Tab.b2, we evaluate two controlled variants (Tab.b3): (i) direct latent feedback, which directly uses the estimated clean latent while retaining CLA’s annealed interpolation with the initial condition; and (ii) unannealed condition replacement, which preserves observation-space completion and re-encoding but immediately adopts the resulting latent without annealed interpolation. As shown in Tab.b4, (i) improves over fixed conditioning but remains modestly below CLA, whereas (ii) substantially degrades performance. This degradation may arise because hard replacement allows unreliable early predictions to bias the jointly encoded condition, diverting the sampling trajectory and impairing reconstruction. Preserving measured values in observation space alone does not prevent errors in predicted modalities from distorting latent guidance. CLA’s annealed conditioning mitigates this risk by anchoring early conditioning to the original observations and progressively incorporating generated complementary information. Together, these results support the complementary roles of observation-preserving completion and gradual feedback incorporation.

MethodCoreupdateRefinementObject
(i)Direct latent feedbackZ^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>ψt−=λtψ1+(1−λt)Z^0t\boldsymbol{\psi_{t^-}=\lambda_t\psi_{1}+(1-\lambda_t)\hat Z_0^t}<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})No slot-replacementCondition
(ii)Condition replacementZ^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>X~t=M⊙X+(1−M)⊙D(Z^0t)‾\underline{\tilde X_t=M\odot X+(1-M)\odot D(\hat Z_0^t)}<br>ψt−=E(X~t)\psi_{t^-}=E(\tilde X_t)<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})No annealingCondition
(iii)CLA(Ours)Z^0t=fθ(Zt,t,c,ψt)\hat Z_0^t=f_\theta(Z_t,t,c,\psi_t)<br>X~t=M⊙X+(1−M)⊙D(Z^0t)‾\underline{\tilde X_t=M\odot X+(1-M)\odot D(\hat Z_0^t)}<br>ψt−=λtψ1+(1−λt)E(X~t)\boldsymbol{\psi_{t^-}=\lambda_t\psi_1+(1-\lambda_t)E(\tilde X_t)}<br>Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})Multimodal completion+<br>Annealed condition updateCondition

Table b3: Implementations of CLA variants

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverage
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
× CLA21.730.66034.770.91017.090.36519.540.27825.210.632
(i)22.030.66135.020.91317.170.36920.110.28725.500.636
(ii)11.750.38015.980.4748.7680.1329.5010.14912.360.331
(iii) CLA22.070.66235.170.91417.360.37320.320.29125.630.639

Table b4: Performance of CLA variants without text prompts

W2&Q3: Baseline Comparisons

Thanks for your constructive advice. OmniRSI builds on the SD3.5 Medium backbone and ControlNet architecture to provide a reusable foundation for unified multimodal remote sensing generation. We supplement component-wise ablation analysis as below.

VAE. RS-VAE consistently outperforms the standard VAE and modality-specific autoencoders in reconstruction quality (Tab.b5), while supporting heterogeneous modalities within a single encoder–decoder. For the standard VAE, single-band PAN, SAR, and NIR inputs are replicated across three channels to match its native RGB input format. It combines improved reconstruction fidelity with joint multimodal encoding and decoding, replacing separate modality-specific representations with a shared latent interface for generation and translation. Tab.b9 and Tab.b10 reveal that LmaskL_{mask} improves PSNR and SSIM across all groups and slightly improves latent consistency, whereas adding LsmoothL_{smooth} substantially improves latent consistency. Hence, RS-VAE benefits from slot-based organization with comprehensive remote sensing multimodal data and two compatible losses.

ModelPANRGBSARNIR
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
VAE36.990.83038.380.94025.680.79429.410.946
VAE(Any2Any)37.680.83839.060.95126.850.84629.990.951
RS-VAE42.790.97239.710.95827.150.87832.830.953

Table b5: Reconstruction of VAE variants.

Reviewer_8LUHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OmniRSI, a unified framework for multimodal remote sensing image generation, covering image reconstruction, text-to-image generation, and cross-modal translation. The framework consists of a remote sensing-specific VAE (RS-VAE) for shared latent representation learning, a fine-tuned Stable Diffusion 3.5 backbone with ControlNet for conditional generation, and a training-free cyclic latent alignment (CLA) mechanism that iteratively refines incomplete visual conditions. The authors also construct RSI-MM, a large-scale dataset containing approximately 11.5 million images from 16 public datasets. Experiments across multiple remote sensing modalities demonstrate improvements over the evaluated baselines in reconstruction, generation, and translation tasks.

Critical strengths

The paper addresses the challenging problem of unified multimodal remote sensing image generation under incomplete observations. Integrating reconstruction, text-to-image generation, and cross-modal translation into a single framework is practically meaningful. Furthermore, the authors construct RSI-MM, a large-scale dataset containing approximately 11.5 million images, and conduct extensive experiments across multiple modalities and tasks. The reported results demonstrate competitive performance and highlight the practical potential of the proposed framework.

Critical weaknesses

Limited Methodological Novelty. OmniRSI primarily combines established techniques: (1) RS-VAE integrates masked reconstruction and latent smoothness regularization into a standard VAE; (2) the generation framework builds on SD3.5 fine-tuning and ControlNet; and (3) CLA shares similarities with existing self-conditioning and iterative refinement strategies. Overall, the framework appears to be largely an integration of existing techniques, making its methodological novelty unclear. The authors should more explicitly clarify the key mechanistic differences between OmniRSI and prior approaches.

Fairness of Baseline Comparisons. OmniRSI has substantially more parameters (2.79B) than the baselines in Table 5, while the text-to-image baselines in Table 3 use official pretrained weights without comparable fine-tuning. It is therefore unclear whether the reported improvements arise from the proposed methods or differences in model capacity and training data.

Questions for authors

Potential Error Propagation in CLA. CLA uses generated multimodal information to refine subsequent denoising steps. However, inaccurate predictions may propagate errors and degrade generation quality. I suggest comparing three conditioning strategies: (1) fixed incomplete conditioning, (2) CLA with generated information, and (3) CLA with ground-truth information (oracle guidance), to evaluate the impact of conditioning reliability.

Limited novelty of RS-VAE. RS-VAE primarily integrates existing techniques, including masked reconstruction and latent-space smoothness regularization, into a standard VAE framework. The paper does not clearly establish the methodological novelty of these components.

Insufficient justification for cross-modal semantic alignment. The authors argue that modality-specific VAEs lead to semantically misaligned latent spaces. However, using a shared encoder and decoder does not necessarily ensure cross-modal alignment. As shown in Eq. (1), the reconstruction objectives do not explicitly enforce alignment between modalities from the same scene. Moreover, the t-SNE visualization in Figure 4 and the reconstruction results in Table 2 provide insufficient evidence of semantic alignment, as good reconstruction performance does not necessarily imply aligned latent representations. Additional quantitative evaluations are needed to substantiate this claim.

Insufficient Component-wise Ablations. The paper does not provide ablation studies on the masked reconstruction and latent smoothness regularization in RS-VAE. Without isolating these components, it is difficult to determine their individual contributions to the reported improvements.

Potential Target Information Leakage. Some captions in RSI-MM are generated from RGB images using GPT. If the captions used in PAN→MSI or SAR→MSI evaluation are derived from target MSI images, they may inadvertently reveal target information, leading to unfair comparisons. The authors should clarify the source of evaluation captions and compare three settings: no text, source-derived captions, and target-derived captions.

Overall, the paper is well-written, clearly structured, and logically organized. However, I still have several concerns regarding the issues raised above. I hope the authors can adequately address these concerns in their rebuttal, as their responses may influence my final assessment.

6 comments

Response to Reviewer#1 (8LUH)

We sincerely thank the reviewer for the thoughtful and constructive comments, which have helped us clarify our methodological contributions, strengthen the experimental evidence, and improve the discussion. We address each concern in the point-by-point responses below, and all additional experiments, analyses, and clarifications will be incorporated into the camera-ready version.

W1&Q2: Method Novelty

We thank the reviewer for encouraging us to clarify the contribution. We acknowledge that the VAE and diffusion formulation are widely adopted techniques in generative modeling. Our contribution lies not in claiming these building blocks as individually novel, but in systematically adapting the full Stable Diffusion pipeline to address the incompleteness of heterogeneous multimodal remote sensing observations. Representative approaches specialize in particular output modalities and lack a unified framework that integrates shared representation learning, multimodal generation, and cross-modal translation, as shown in Tab.a1. Any2Any (Chen et al., ICML 2026) performs pairwise cross-modal translation using modality-specific VAEs and a shared DiT backbone with target-specific adapters. Txt2Img-MHN(Xu et al. TIP2023), MMDiff(Tang et al. ISPRS2026), Text2earth(Liu et al. GRSM2025), DiffusionSat(Khanna et al. ICLR2024) mainly focus on RGB generation, HSIGene (Pang et al. TPAMI2026) on hyperspectral synthesis, and HuiYanEarth-SAR (Liu et al.) on SAR generation. MMDiff and Text2earth use LoRA adaptation to generate other modalities with additional parameters, whereas DiffusionSat uses ControlNet to incorporate auxiliary information for RGB generation.

MethodRepresentationGenerationTranslationDescription
Any2AnyModality-specific VAEs—one-to-one mappingDiT backbone with target-specific adapters for translation
Txt2Img-MHNVQ-VAE/VQGANRGB—VQ-VAE/VQGAN representations and Hopfield network.
MMDiff-OPT/SAR/IR—Pretrained VAE and LoRA-based modality adaptation
Text2Earth-RGBone-to-one translationExtends SAR, NIR, PAN generation and translation via task-specific lora and controlnet
DiffusionSat-RGBMSI→RGBPretrained VAE, metadata-conditioned generation, with ControlNet-based extensions
HSIGeneHyperspectral VAEHSIHED,Sketch→HSIHyperspectral generation and RGB-guided controlnet
HuiYanEarth-SAR-SAR—Pretrained VAE and metadata conditioned SAR generation using LoRA adaptation
OmniRSI(Ours)Unified RS-VAEMulti-modalUnified translation + CLAMultimodal representation, generation and translation with CLA refinement

Table a1: Comparisons with representative methods.

To the best of our knowledge, OmniRSI is the first remote sensing framework to couple a shared latent representation learned from partially observed multimodal data with unified generation, cross-modal translation, and an iterative condition refinement strategy. Specifically, RS-VAE establishes a shared representation for heterogeneous and incomplete observations, where LmaskL_{mask} improves the reconstruction fidelity (Tab.a9) and LsmoothL_{smooth} promotes the semantic alignment (Tab.a10). Crucially, the resulting representation serves both as the generative target space and as the visual-conditioning interface, enabling the adapted SD3.5 Medium backbone and ControlNet branch to support generation and translation from different modality subsets. CLA updates the explicit conditioning representation by multimodal completion and annealed latent re-encoding.

As summarized in Tab.a2, Analog Bits [1] feeds back clean-sample estimates, whereas ILVR [2], RePaint [3], and DPS [4] directly refine sampling states using reference or measurement constraints. Instead, the novelty of CLA lies in this coupled observation–latent feedback path, rather than iterative updating or any individual operation alone. Hence, OmniRSI treats incomplete multimodal observations as a central design consideration throughout representation, generation, and translation, providing a reusable foundation for further research on multimodal remote sensing generation. We sincerely appreciate the reviewer's comment, and we promise to clarify these connections and distinctions in the Related Work and Method sections.

MethodCore updateRefinementObject
Analog Bitsx^0t=fθ(xt,t,x^0prev)\hat x_0^t=f_\theta(x_t,t,\hat x_0^{prev})Improve prediction using previous estimatesCondition
ILVRxt−1=xt−1+ϕN(yt−1)−ϕN(xt−1)x_{t-1}=x_{t-1}+\phi_N(y_{t-1})-\phi_N(x_{t-1})Low-pass filter \phi_N(·) for states correctionState
RePaintxt−1=M⊙yt−1+(1−M)⊙xt−1′x_{t-1}=M\odot y_{t-1}+(1-M)\odot x'_{t-1}Known-region M replacement in noisy stateState
DPSxt−1=xt−1−ηt∇xt∥y−A(x^0(xt))∥22x_{t-1}=x_{t-1}-\eta_t\nabla_{x_t}\lVert y-\mathcal A(\hat x_0(x_t))\rVert_2^2Measurement-gradient state correctionState
CLA (Ours)X~t=M⊙X+(1−M)⊙Dθ(Z^0t)\tilde X_t=M\odot X+(1-M)\odot D_\theta(\hat Z_0^t)Known-slot replacement andCondition
↳ψt−=λtψ1+(1−λt)Eθ(X~t)\psi_{t^-}=\lambda_t\psi_1+(1-\lambda_t)E_\theta(\tilde X_t)annealed latent condition update
↳Zt−=Zt−Δσt vθ(Zt,t,c,ψt−)Z_{t^-}=Z_t-\Delta\sigma_t\,v_\theta(Z_t,t,c,\psi_{t^-})

Table a2: Comparisons with representative self-conditioning and iterative refinement methods.

[1] Chen T, et al. Analog bits: Generating discrete data using diffusion models with self-conditioning, ICLR 2023.

[2] Choi J, et al. ILVR: Conditioning method for denoising diffusion probabilistic models. ICCV 2021.

[3] Lugmayr A, et al. RePaint: Inpainting using denoising diffusion probabilistic models. CVPR 2022.

[4] Chung H, et al. Diffusion posterior sampling for general noisy inverse problems. ICLR 2023.

W2: Baseline Comparisons

Thanks for your constructive advice. OmniRSI builds on the SD3.5 Medium backbone and ControlNet architecture to provide a reusable foundation for unified multimodal remote sensing generation. We supplement component-wise ablation analysis as below.

VAE. RS-VAE consistently outperforms the standard VAE and modality-specific autoencoders in reconstruction quality (Tab.a3), while supporting heterogeneous modalities within a single encoder–decoder. For the standard VAE, single-band PAN, SAR, and NIR inputs are replicated across three channels to match its native RGB input format. It combines improved reconstruction fidelity with joint multimodal encoding and decoding, replacing separate modality-specific representations with a shared latent interface for generation and translation. Tab.a9 and Tab.a10 reveal that LmaskL_{mask} improves PSNR and SSIM across all groups and slightly improves latent consistency, whereas adding LsmoothL_{smooth} substantially improves latent consistency. Hence, RS-VAE benefits from slot-based organization with comprehensive remote sensing multimodal data and two compatible losses.

ModelPANRGBSARNIR
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
VAE36.990.83038.380.94025.680.79429.410.946
VAE(Any2Any)37.680.83839.060.95126.850.84629.990.951
RS-VAE42.790.97239.710.95827.150.87832.830.953

Table a3: Reconstruction of VAE variants.

Text-to-image Generation. We select the original SD3.5 Medium, MMDiff and Txt2Img-MHN for additional RGB fine-tuning, as they provide publicly available training implementations compatible with our RGB setting. Other baselines adopt distinct conditioning or output configurations: Text2Earth and DiffusionSat incorporate metadata unavailable across all our subsets; Text2Earth's official training code remains unreleased; CC-Diff uses object annotations; and HSIGene is tailored for hyperspectral generation. As reported in Tab.a4, OmniRSI outperforms these additionally fine-tuned baselines. In particular, our configuration achieves better quality than the standard-VAE configuration with the same SD3.5 Medium backbone. Hence, generation benefits from a domain-adapted backbone with a shared multimodal representation by RS-VAE.

Config.Text-to-imageParam.
--FID↓NIQE↓CLIPIQA↑CLIPScore↑(M)
VAESD3.5Medium93.0896.3970.5970.7582139.25
VAESD3.5Medium*45.1335.9070.6000.7662139.25
RS-VAESD3.5Medium*(Ours)28.6934.3990.6950.8302139.25
VQVAETxt2Img-MHN150.0635.3980.3770.67229.33
VQVAETxt2Img-MHN*126.2218.6880.2900.71729.33
VAEMMDiff80.9225.8240.3770.7801066.24
VAEMMDiff*61.7245.6390.3860.7851066.24

Table a4: RGB text-to-image generation performance. * denotes fine-tuned models.

Cross-modal Translation. Leveraging the spatial compression by RS-VAE, we adapt RDBM to the latent space and substantially enlarge its network for unified translation training. OmniRSI outperforms this enlarged baseline across all translation directions without text (Tab.a5), with textual guidance providing a further improvement reported separately (Tab.a11). OmniRSI supports these translation directions within one shared model, whereas the original baselines are trained separately for each task. Besides, train the ControlNet branches attached to DiffusionSat and standard-VAE SD3.5 using the available data and associated metadata for RGB-target translation (Tab.a6). The SD3.5-based configuration outperforms DiffusionSat, while incorporating RS-VAE further improves metrics across all tasks. Besides, CLA further improves translation performance, as detailed in our response to Q1. Together, these comparisons demonstrate the complementary benefits of backbone adaptation, shared multimodal representation learning and cyclic latent refinement.

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverageParam
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
RDBM21.020.65631.110.88112.160.36718.010.24522.590.61639.17 (Total: 156.68)
RS-VAE+ScaledRDBM21.430.65832.280.88914.370.36919.890.27423.830.6242274.98
Ours (× text)22.070.66235.170.91417.360.37320.320.29125.630.6392789.06
Ours (RGB-derived text)25.160.78636.100.92518.390.38220.500.30227.100.6832789.06

Table a5: Scaling baseline RDBM with RS-VAE.

Config.PAN->RGBNIR->RGBSAR->RGBParam.
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑(M)
DiffusionSAT+ControlNet*24.200.73915.890.48216.670.3891682.888
VAE+SD3.5Medium+ControlNet*26.110.85720.330.65618.090.4312789.06
VAE+SD3.5Medium*+ControlNet*27.280.86621.260.68319.020.4632789.06
Ours(RS-VAE+SD3.5Medium*+ControlNet*)29.090.88923.010.73420.240.5222789.06

Table a6: Text-free translation comparisons with models of comparable parameter counts. For configurations with a fine-tuned SD3.5 backbone, generation fine-tuning precedes ControlNet training. * denotes fine-tuned models. CLA is disabled.

Q1: CLA Variants

We thank the reviewer for suggesting this analysis. In fact, the comparison between incomplete and CLA condition was already reported in Tab.8. We compare 6text-free variants using 40 steps: (i) Incomplete conditioning, (ii) standard CLA, (iii) two-stage conditioning, with the 40-step output from (i) fixed as the condition for subsequent translation without CLA, (iv) 80-step output from (i), (v) two-stage conditioning, with an extra 40-step generation providing a fixed completion for subsequent translation with CLA, and (vi) Oracle CLA, replacing generated values in slots with available ground truth while retaining the same blending schedule.

Tab.a7 shows that standard CLA improves both PSNR and SSIM across all four translation directions and outperforms both fixed two-stage conditioning and extended sampling at matched NFE (ii–iv). Completed initialization followed by CLA provides further gains, while oracle CLA achieves the best performance, demonstrating the benefit of richer and more accurate conditioning. The oracle gap identifies remaining opportunities to improve completion accuracy, while the gains over fixed incomplete conditioning show that generated completions already provide effective guidance. Tab.a8 further shows that CLA delivers larger gains under limited-step sampling, while its incremental benefit narrows at higher step counts, as the underlying SD3.5 model achieves better translation performance and leaves less room for further refinement.

Config.PAN->MSIMSI->PANSAR->MSIMSI->SARAverage
-PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
i21.730.66034.770.91017.090.36519.540.27825.210.632
ii22.070.66235.170.91417.360.37320.320.29125.630.639
iii21.910.65635.110.90817.220.36820.280.28625.540.633
iv21.780.65335.090.90717.190.36620.270.28525.480.631
v22.430.66535.210.91517.570.37720.420.29325.810.641
vi25.200.80535.380.92117.790.41220.640.30326.780.693

Table a7: Performance of compared Variants

StepsPAN->MSIMSI->PANSAR->MSIMSI->SARAverage
-CLAPSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑PSNR↑SSIM↑
10×20.070.52333.880.89115.750.31918.240.21723.940.566
10√20.930.57434.470.90616.150.34319.010.26424.610.600
20×20.660.55834.150.89716.020.33718.580.23624.320.585
20√21.890.66134.800.91217.250.36820.030.28325.380.635
40×21.730.66034.770.91017.090.36519.540.27825.210.632
40√22.070.66235.170.91417.360.37320.320.29125.630.639

Table a8: Performance of CLA without text prompts

Comments on the paper

1 comment

General Response to Reviewers:

We thank all reviewers for their constructive feedback. We summarize the additional experiments and clarifications below, with details provided in our point-by-point responses. These additions will be incorporated into the revised manuscript.

Reviewer 8LUH

  • Clarify OmniRSI's and CLA’s novelty (Tab.a1–a2).
  • Add fine-tuned baselines and RS-VAE ablations (Tab.a3–a6, a9–a10).
  • Compare CLA variants and translation with different captions (Tab.a7–a8, a11).

Reviewer Kx1V

  • Clarify OmniRSI’s and CLA’s novelty, and add CLA comparisons (Tab.b1–b4).
  • Add baselines, RS-VAE and CLA ablations, slot identification and cross-modal round-trip reconstruction (Tab.b5–b14).
  • Clarify caption sources and metric aggregation (Tab.b15).

Reviewer Hxiu

  • Add caption-source comparisons, baselines, DDBM(σ=20\sigma=20), and external-dataset evaluation (Tab.c1–c6).
  • Expand the discussion of DiffusionSat, choice of ω\omega, and source deduplication.
  • Add slot identification, RS-VAE ablations, CLA variants, five-seed robustness tests, and sampling-step analysis (Tab.c7–c14).

Reviewer rgAM

  • Add RS-VAE ablations, and more generation/translation comparisons (Tab.d1–d6).
  • Evaluate CLA across five random seeds and compare alternative feedback mechanisms (Tab.d7–d9).
  • Discuss compositional generalization, and add cross-modal round-trip reconstruction and slot identification (Tab.d10–d11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.