- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes OmniRSI, a unified latent diffusion framework for multimodal remote sensing image generation under heterogeneous and partially observed inputs. The method first trains an RS-VAE to map different remote-sensing modalities into a shared latent space, and then adapts Stable Diffusion 3.5 with ControlNet to support both text-to-image generation and cross-modal translation. It further introduces Cyclic Latent Alignment (CLA), a training-free sampling strategy that progressively completes missing modalities using intermediate predictions and feeds the enriched visual condition back into the denoising process. The authors also construct RSI-MM, a large-scale multimodal dataset containing about 11.5M images from 16 public datasets. Experiments on reconstruction, text-to-image generation, and cross-modal translation show substantial improvements over existing methods, including zero-shot transfer to unseen modality pairs.
Critical strengths
- Well-motivated problem formulation. The paper targets a practically important problem in remote sensing: unified generation under heterogeneous and partially observed multimodal inputs. The motivation is clear and well justified.
- Comprehensive and convincing experiments. The evaluation covers reconstruction, text-to-image generation, and cross-modal translation, with extensive comparisons and ablations. The consistent improvements across different tasks and modalities provide strong empirical support for the proposed framework.
- A coherent unified framework. Rather than designing separate models for individual modality pairs, OmniRSI integrates multimodal representation, generation, and translation within a shared latent space, providing a clean and scalable formulation for heterogeneous remote-sensing modalities.
Critical weaknesses
The main weakness is that the performance gains are not fully disentangled from model and data scale.
Questions for authors
- OmniRSI uses a substantially larger backbone and training corpus than several baselines. Could the authors provide additional analysis or controlled experiments to clarify how much of the performance gain comes from the proposed unified formulation (RS-VAE/CLA) rather than increased model and data scale?
- CLA progressively fills missing modalities using intermediate predictions and feeds them back as visual conditions. How sensitive is this process to inaccurate predictions at early denoising steps? Could prediction errors in the missing slots accumulate or negatively affect the subsequent denoising trajectory?
- The zero-shot PAN–SAR experiments suggest that OmniRSI can generalize to unseen modality pairs. Could the authors further clarify what enables this compositional generalization, and whether similar behavior is observed for other unseen modality combinations?
- A key motivation of RS-VAE is to establish a shared latent space across heterogeneous modalities. Beyond reconstruction quality and latent-space visualization, could the authors provide more direct quantitative evidence (e.g., cross-modal retrieval performance, latent similarity, or alignment metrics) that semantically corresponding observations from different modalities are well aligned in the learned latent space?
6 comments
Text-to-image Generation. We select the original SD3.5 Medium, MMDiff and Txt2Img-MHN for additional RGB fine-tuning, as they provide publicly available training implementations compatible with our RGB setting. As reported in Tab.d4, OmniRSI outperforms these additionally fine-tuned baselines. In particular, our configuration achieves better quality than the standard-VAE configuration with the same SD3.5 Medium backbone. Hence, generation benefits from a domain-adapted backbone with a shared multimodal representation by RS-VAE.
| Config. | Text-to-image | Param. | ||||
|---|---|---|---|---|---|---|
| - | - | FID↓ | NIQE↓ | CLIPIQA↑ | CLIPScore↑ | (M) |
| VAE | SD3.5Medium | 93.089 | 6.397 | 0.597 | 0.758 | 2139.25 |
| VAE | SD3.5Medium* | 45.133 | 5.907 | 0.600 | 0.766 | 2139.25 |
| RS-VAE | SD3.5Medium*(Ours) | 28.693 | 4.399 | 0.695 | 0.830 | 2139.25 |
| VQVAE | Txt2Img-MHN | 150.063 | 5.398 | 0.377 | 0.672 | 29.33 |
| VQVAE | Txt2Img-MHN* | 126.221 | 8.688 | 0.290 | 0.717 | 29.33 |
| VAE | MMDiff | 80.922 | 5.824 | 0.377 | 0.780 | 1066.24 |
| VAE | MMDiff* | 61.724 | 5.639 | 0.386 | 0.785 | 1066.24 |
Table d4: RGB text-to-image generation performance. * denotes fine-tuned models.
Cross-modal Translation. Leveraging the spatial compression by RS-VAE, we adapt RDBM to the latent space and substantially enlarge its network for unified translation training. OmniRSI outperforms this enlarged baseline across all translation directions without text (Tab.d5). OmniRSI supports these translation directions within one shared model, whereas the original baselines are trained separately for each task. Furthermore, we train the ControlNet branches attached to DiffusionSat and standard-VAE SD3.5 using the available data and associated metadata for RGB-target translation (Tab.d6). SD3.5-based configuration outperforms DiffusionSat, while incorporating RS-VAE further improves metrics across all tasks. Besides, CLA further improves translation performance, as detailed in our response to Q2. Together, these comparisons demonstrate the complementary benefits of backbone adaptation, shared multimodal representation learning and cyclic latent refinement.
| Config. | PAN->MSI | MSI->PAN | SAR->MSI | MSI->SAR | Average | Param | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| - | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | (M) |
| RDBM | 21.02 | 0.656 | 31.11 | 0.881 | 12.16 | 0.367 | 18.01 | 0.245 | 22.59 | 0.616 | 39.17 (Total: 156.68) |
| RS-VAE+ScaledRDBM | 21.43 | 0.658 | 32.28 | 0.889 | 14.37 | 0.369 | 19.89 | 0.274 | 23.83 | 0.624 | 2274.98 |
| Ours (× text) | 22.07 | 0.662 | 35.17 | 0.914 | 17.36 | 0.373 | 20.32 | 0.291 | 25.63 | 0.639 | 2789.06 |
| Ours (RGB-derived text) | 25.16 | 0.786 | 36.10 | 0.925 | 18.39 | 0.382 | 20.50 | 0.302 | 27.10 | 0.683 | 2789.06 |
Table d5: Scaling baseline RDBM with RS-VAE.
| Config. | PAN->RGB | NIR->RGB | SAR->RGB | Param. | |||
|---|---|---|---|---|---|---|---|
| - | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | (M) |
| DiffusionSAT+ControlNet* | 24.20 | 0.739 | 15.89 | 0.482 | 16.67 | 0.389 | 1682.888 |
| VAE+SD3.5Medium+ControlNet* | 26.11 | 0.857 | 20.33 | 0.656 | 18.09 | 0.431 | 2789.06 |
| VAE+SD3.5Medium*+ControlNet* | 27.28 | 0.866 | 21.26 | 0.683 | 19.02 | 0.463 | 2789.06 |
| Ours(RS-VAE+SD3.5Medium*+ControlNet*) | 29.09 | 0.889 | 23.01 | 0.734 | 20.24 | 0.522 | 2789.06 |
Table d6: Text-free translation comparisons with models of comparable parameter counts. For configurations with a fine-tuned SD3.5 backbone, generation fine-tuning precedes ControlNet training for translation. * denotes fine-tuned models. CLA is disabled.
Q2: Robustness of CLA
We thank the reviewer for raising this question. We evaluate the same trained model with and without CLA across five random seeds. Tab.d7 reports , with the coefficient of variation in parentheses, where and denote the mean and standard deviation across seeds.
CLA improves mean PSNR and SSIM while reducing both variance and CV across all four translation directions, demonstrating improved fidelity and reduced sensitivity to sampling randomness. This stability is consistent with its annealed condition-update mechanism, which limits reliance on generated completions early in sampling and progressively incorporates their information while preserving observed measurements. These results support the robustness of CLA’s improvements across the evaluated seeds.
| Config. | PAN→MSI | MSI→PAN | SAR→MSI | MSI→SAR | ||||
|---|---|---|---|---|---|---|---|---|
| CLA | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ |
| × | 21.772±1.3965(5.4278%) | 0.661±0.0008(4.2790%) | 34.589±1.2959(3.2911%) | 0.913±0.0018(4.6471%) | 17.352±1.8143(7.7626%) | 0.369±0.0024(13.2793%) | 20.256±0.5701(3.7275%) | 0.280±0.0006(8.7471%) |
| √ | 22.032±0.2992(2.4830%) | 0.662±0.0001(1.5107%) | 35.227±0.2488(1.4158%) | 0.914±0.0001(1.0941%) | 17.577±0.4913(3.9881%) | 0.373±0.0001(2.6809%) | 20.43±0.3233(2.7831%) | 0.291±0.0002(4.8596%) |
Table d7: Translation performance with and without CLA across five random seeds.
Furthermore, we compare CLA with two controlled variants: (i) direct latent feedback, which directly uses the estimated clean latent while retaining CLA’s annealed interpolation with the initial condition; and (ii) unannealed condition replacement, which preserves observation-space completion and re-encoding but immediately adopts the resulting latent without annealed interpolation. As shown in Tab.d9, (i) improves over fixed conditioning but remains modestly below CLA, whereas (ii) substantially degrades performance. This degradation may arise because hard replacement allows unreliable early predictions to bias the jointly encoded condition, diverting the sampling trajectory and impairing reconstruction. Preserving measured values in observation space alone does not prevent errors in predicted modalities from distorting latent guidance. CLA’s annealed condition updates mitigate this risk by anchoring early conditioning to the original observations and progressively incorporating generated complementary information. Together, these results support the complementary roles of observation-preserving completion and gradual feedback incorporation
| Method | Coreupdate | Refinement | Object |
|---|---|---|---|
| (i)Direct latent feedback | <br><br> | No <u>slot-replacement<u> | Condition |
| (ii)Condition replacement | <br><br><br> | No annealing | Condition |
| (iii)CLA(Ours) | <br><br><br> | Multimodal completion+<br>Annealed condition update | Condition |
Table d8: Implementations of CLA variants
| Config. | PAN->MSI | MSI->PAN | SAR->MSI | MSI->SAR | Average | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| - | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ | PSNR↑ | SSIM↑ |
| × CLA | 21.73 | 0.660 | 34.77 | 0.910 | 17.09 | 0.365 | 19.54 | 0.278 | 25.21 | 0.632 |
| (i) | 22.03 | 0.661 | 35.02 | 0.913 | 17.17 | 0.369 | 20.11 | 0.287 | 25.50 | 0.636 |
| (ii) | 11.75 | 0.380 | 15.98 | 0.474 | 8.768 | 0.132 | 9.501 | 0.149 | 12.36 | 0.331 |
| (iii) CLA | 22.07 | 0.662 | 35.17 | 0.914 | 17.36 | 0.373 | 20.32 | 0.291 | 25.63 | 0.639 |
Table d9: Performance of CLA variants without text prompts
Response to Reviewer#4 (rgAM)
We sincerely thank the reviewer for the thoughtful and constructive comments, and for recognizing the practical relevance, comprehensive evaluation, and coherent unified formulation of OmniRSI. Your suggestions have helped us clarify the sources of improvement and strengthen the evidence supporting our design. All additional experiments, analyses, and clarifications will be incorporated into the main text or appendix.
Q1&Q4: Component-wise Ablation
We thank the reviewer for these important questions. We add the controlled RS-VAE ablations, fine-tuned models for generation comparison, and high-capacity translation baselines.
RS-VAE. RS-VAE consistently outperforms the standard VAE and modality-specific autoencoders in reconstruction quality (Tab.d1), while supporting heterogeneous modalities within a single encoder–decoder. For the standard VAE, single-band PAN, SAR, and NIR inputs are replicated across three channels to match its native RGB input format. It combines improved reconstruction fidelity with joint multimodal encoding and decoding, replacing separate modality-specific representations with a shared latent interface for generation and translation. Tab.d2 and Tab.d3 reveal that Lmask improves PSNR and SSIM across all groups and slightly improves latent consistency, whereas adding Lsmooth substantially improves latent consistency. Hence, RS-VAE benefits from slot-based organization with comprehensive remote sensing multimodal data and two compatible losses.
Table d1: Reconstruction of VAE variants.
Table d2: RS-VAE reconstruction performance under different loss configurations
Table d3: Latent Semantic Alignment Evaluation of excluded bands N