- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. It introduces a Scene-Guided Two-Stage Training strategy to connect an MLLM with a diffusion transformer, Detailed Visual Conditioning to support image-conditioned generation, and UniGen for multi-task training. Experiments demonstrate competitive performance across various understanding and generation benchmarks.
Critical strengths
- Omni-RS effectively integrates four understanding tasks and five generation tasks within a unified framework, significantly extending the capabilities of existing remote sensing models. The UniGen dataset further facilitates multi-task learning across heterogeneous generation tasks.
- Extensive evaluations demonstrate competitive performance across diverse benchmarks, with particularly notable improvements in image inpainting, outpainting, and cross-modal translation, highlighting the practical effectiveness of the framework.
Critical weaknesses
- Although the proposed framework achieves promising results, additional ablation studies on the two-stage training strategy and Detailed Visual Conditioning would help clarify their respective contributions to the overall performance.
- While Omni-RS supports a broad range of tasks, the paper lacks comparisons between unified multi-task training and task-specific training, leaving the benefits and potential trade-offs of joint learning insufficiently explored.
Questions for authors
Does joint training across multiple generation tasks provide measurable benefits over independently trained task-specific models? A comparison under similar training budgets would help substantiate the advantages of the unified framework.
3 comments
Response to Reviewer kzuG (2/3)
(Q1, continued.)
(3) DVC vs. conditioning through the MLLM
To examine the role of DVC, we built a baseline that replaces the reference-image conditioning pathway. Since editing tasks require access to the reference image, we did not simply remove all reference-image inputs. Instead, we fed the reference image into the MLLM and conditioned the generator on its hidden states, while removing the direct VAE reference-image pathway. This experiment compares the full Omni-RS with this alternative, which passes reference-image information through the MLLM.
Table R4. Replacing the reference-image conditioning pathway.
\hline \text{Conditioning setting} & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{w/o DVC} & 50.02 & 57.94 & 24.04 & 11.04 & 8.54 \\\\ \text{Full Omni-RS} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ NIR and SAR denote RGB→NIR and RGB→SAR, respectively. Compared with the alternative baseline, the full model reduces inpainting and outpainting FID from 50.02 and 57.94 to 31.23 and 35.33, raises layout CAS from 24.04 to 78.77, and raises NIR and SAR PSNR from 11.04 and 8.54 to 18.68 and 12.89, respectively. The full model achieves better results on all five reference-conditioned evaluation. This indicates that, under the compared training settings, passing reference-image information only through the MLLM cannot replace the direct VAE visual conditioning pathway used in the full model, supporting the practical value of DVC for reference-conditioned generation tasks. *(Continued in the next comment.)*Response to Reviewer kzuG (3/3)
Q2/W2: Benefits and trade-offs of multi-task joint training
We thank the reviewer for raising this question. We added a controlled comparison between single-task and multi-task training to examine whether the unified model brings performance benefits beyond task coverage.
In the single-task setting (S), a separate model is trained for each generation task; in the multi-task setting (E), one model learns all generation tasks jointly. All models are initialized from the same generator weights and the same Stage I MLLM, which is kept frozen during generation training. Both settings use the same training samples for each task and the same generator optimization configuration.
Table R5. Single-task training vs. multi-task joint training.
\hline \text{Training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Single-task (S)} & 42.97 & 23.06 & 44.43 & 37.65 & 56.11 & 18.44 & \mathbf{12.99} \\\\ \text{Multi-task (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & 12.89 \\\\ \hline \end{array}$$ NIR and SAR denote RGB→NIR and RGB→SAR, respectively. The multi-task model achieves better metrics in five of the six evaluation settings. T2I FID decreases from 42.97 to 21.90 and CLIP increases from 23.06 to 24.85; inpainting FID decreases from 44.43 to 31.23; layout CAS increases from 56.11 to 78.77. Outpainting FID also decreases from 37.65 to 35.33, and NIR PSNR increases from 18.44 to 18.68. T2I shows a substantial improvement. In UniGen, T2I and outpainting share the same Git-10M source images (Appendix Table 8), allowing the multi-task model to additionally learn scene reconstruction under partial-image conditioning. This complementary training signal may contribute to the improvement, although this comparison does not isolate its contribution from that of other tasks. These results support that, under the compared training settings, one shared generation model outperforms separately trained models on most tasks, so unified training has measurable performance value, along with task-dependent trade-offs. Some tasks in UniGen share source scenes and spatial conditions, which provides correlated information for cross-task learning. In the revision, we will include these results together with the detailed training budgets and per-task sample exposure, so that the benefits of joint training and the comparison conditions can be directly verified. We thank the reviewer again for the constructive comments. The additional experiments above examine the individual contributions of Stage I understanding adaptation, freezing the MLLM during generation training, and DVC, as well as the benefits and trade-offs of multi-task joint training relative to single-task training. These results will be added to the revision. We hope these additions address your concerns, and we are happy to provide further clarification if needed.
Response to Reviewer kzuG (1/3)
We thank the reviewer for the positive assessment of Omni-RS, in particular for recognizing the unified framework that integrates four understanding tasks and five generation tasks, the role of UniGen in supporting multi-task learning, and the improvements in inpainting, outpainting, and cross-modal translation. To address the two suggestions, we conducted the following controlled experiments.
Q1/W1: Individual contributions of two-stage training and DVC
We separately examine Stage I RS understanding adaptation, MLLM freezing during Stage II, and the DVC reference-image conditioning pathway.
(1) Contribution of Stage I RS understanding adaptation
We compare a model using the original frozen MLLM (F) with Omni-RS, which uses the frozen MLLM after Stage I adaptation (E). Both generators are initialized from the same weights, use the same UniGen data, task sampling ratios, training budget, and generator optimization settings, and retain the same reference-image conditioning pathway. The only key difference is whether the MLLM has undergone Stage I RS understanding adaptation.
Table R1. Contribution of Stage I RS understanding adaptation.
\hline \text{MLLM setting} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Original (F)} & 62.97 & 21.98 & 39.23 & 40.52 & 47.90 & 12.17 & 12.19 \\\\ \text{Stage I-adapted (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Both MLLMs remain frozen during generation training. NIR and SAR denote RGB→NIR and RGB→SAR, respectively. After Stage I adaptation, all listed metrics improve across all six evaluation settings. T2I FID drops from 62.97 to 21.90, layout CAS rises from 47.90 to 78.77, and NIR PSNR rises from 12.17 to 18.68. This shows that, under the same settings, RS understanding adaptation provides additional benefits for generation. **(2) Role of freezing the MLLM in Stage II** We further compare updating the Stage I MLLM during generation training (D) with keeping it frozen (E). Both settings use the same initialization, generation training data, task sampling, and training budget, with identical generator optimization settings. *Table R2. Updating vs. freezing the MLLM during generation training.* $$\begin{array}{l|ccccccc} \hline \text{MLLM training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Updated (D)} & 39.31 & 23.24 & 36.67 & 36.49 & 72.42 & 15.70 & 11.81 \\\\ \text{Frozen (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Both settings start from the same Stage I-adapted MLLM. Freezing the MLLM yields better results across all six generation evaluation settings. In particular, T2I FID decreases from 39.31 to 21.90, layout CAS increases from 72.42 to 78.77, and NIR PSNR increases from 15.70 to 18.68. These results support retaining the fixed, Stage I-adapted conditioning representation rather than updating the MLLM with the generation objective under the evaluated training configuration. **(3) Further evaluation of understanding** To examine whether updating the MLLM during generation training affects its understanding ability, we evaluate the final MLLM from D on the same understanding benchmarks and compare it with the Stage I-adapted MLLM retained in E. *Table R3. Understanding performance before and after generation training.* $$\begin{array}{l|ccccc} \hline \text{MLLM state} & \text{RSVQA-LR Avg. Acc.}\uparrow & \text{VRSBench Acc.}\uparrow & \text{Region R-1}\uparrow & \text{Region R-L}\uparrow & \text{Region METEOR}\uparrow \\\\ \hline \text{Updated (D)} & 92.09 & 49.78 & 45.94 & 45.94 & 40.13 \\\\ \text{Frozen (E)} & \mathbf{92.52} & \mathbf{49.89} & \mathbf{85.30} & \mathbf{85.30} & \mathbf{80.60} \\\\ \hline \end{array}$$ When the MLLM is updated with the generation objective, RSVQA-LR and VRSBench accuracy drop by 0.43 and 0.11 percentage points, respectively, while region understanding ROUGE-1, ROUGE-L, and METEOR drop by 39.36, 39.36, and 40.47 points. Together with the generation results above, updating the MLLM not only fails to improve generation but is also accompanied by a substantial degradation in region understanding. Freezing the MLLM preserves the understanding ability acquired in Stage I and yields better generation results, supporting the freezing choice under the current training configuration. *(Continued in the next comment.)*