acceptodds
Under review as a conference paper at ICLR 2027

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Abstract

Practical remote sensing applications often require both understanding and generation capabilities for the same geographic scene. However, most existing methods address either understanding or generation in isolation, while existing unified remote sensing models' generative ability is limited to text to image. To bridge this gap, we propose Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. Specifically, we introduce the "Scene-Guided Two-Stage Training" strategy, which first adapts a multimodal large language model to remote sensing understanding and then uses its hidden states to guide a diffusion transformer for image generation. To enable joint learning across diverse generation tasks, we construct UniGen, a unified multi-task generation dataset, by organizing heterogeneous remote sensing data into a unified format. Furthermore, we introduce "Detailed Visual Conditioning", which uses visual conditions encoded by a VAE to guide the generation process, thereby supporting additional generation tasks that require reference images. Extensive experiments demonstrate that Omni-RS performs exceptionally well in understanding tasks and has achieved outstanding results in a variety of generative tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Remote sensing, Understanding, Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kzuGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. It introduces a Scene-Guided Two-Stage Training strategy to connect an MLLM with a diffusion transformer, Detailed Visual Conditioning to support image-conditioned generation, and UniGen for multi-task training. Experiments demonstrate competitive performance across various understanding and generation benchmarks.

Critical strengths
  1. Omni-RS effectively integrates four understanding tasks and five generation tasks within a unified framework, significantly extending the capabilities of existing remote sensing models. The UniGen dataset further facilitates multi-task learning across heterogeneous generation tasks.
  2. Extensive evaluations demonstrate competitive performance across diverse benchmarks, with particularly notable improvements in image inpainting, outpainting, and cross-modal translation, highlighting the practical effectiveness of the framework.
Critical weaknesses
  1. Although the proposed framework achieves promising results, additional ablation studies on the two-stage training strategy and Detailed Visual Conditioning would help clarify their respective contributions to the overall performance.
  2. While Omni-RS supports a broad range of tasks, the paper lacks comparisons between unified multi-task training and task-specific training, leaving the benefits and potential trade-offs of joint learning insufficiently explored.
Questions for authors

Does joint training across multiple generation tasks provide measurable benefits over independently trained task-specific models? A comparison under similar training budgets would help substantiate the advantages of the unified framework.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kzuG (1/3)

We thank the reviewer for the positive assessment of Omni-RS, in particular for recognizing the unified framework that integrates four understanding tasks and five generation tasks, the role of UniGen in supporting multi-task learning, and the improvements in inpainting, outpainting, and cross-modal translation. To address the two suggestions, we conducted the following controlled experiments.

Q1/W1: Individual contributions of two-stage training and DVC

We separately examine Stage I RS understanding adaptation, MLLM freezing during Stage II, and the DVC reference-image conditioning pathway.

(1) Contribution of Stage I RS understanding adaptation

We compare a model using the original frozen MLLM (F) with Omni-RS, which uses the frozen MLLM after Stage I adaptation (E). Both generators are initialized from the same weights, use the same UniGen data, task sampling ratios, training budget, and generator optimization settings, and retain the same reference-image conditioning pathway. The only key difference is whether the MLLM has undergone Stage I RS understanding adaptation.

Table R1. Contribution of Stage I RS understanding adaptation.

\hline \text{MLLM setting} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Original (F)} & 62.97 & 21.98 & 39.23 & 40.52 & 47.90 & 12.17 & 12.19 \\\\ \text{Stage I-adapted (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Both MLLMs remain frozen during generation training. NIR and SAR denote RGB→NIR and RGB→SAR, respectively. After Stage I adaptation, all listed metrics improve across all six evaluation settings. T2I FID drops from 62.97 to 21.90, layout CAS rises from 47.90 to 78.77, and NIR PSNR rises from 12.17 to 18.68. This shows that, under the same settings, RS understanding adaptation provides additional benefits for generation. **(2) Role of freezing the MLLM in Stage II** We further compare updating the Stage I MLLM during generation training (D) with keeping it frozen (E). Both settings use the same initialization, generation training data, task sampling, and training budget, with identical generator optimization settings. *Table R2. Updating vs. freezing the MLLM during generation training.* $$\begin{array}{l|ccccccc} \hline \text{MLLM training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Updated (D)} & 39.31 & 23.24 & 36.67 & 36.49 & 72.42 & 15.70 & 11.81 \\\\ \text{Frozen (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Both settings start from the same Stage I-adapted MLLM. Freezing the MLLM yields better results across all six generation evaluation settings. In particular, T2I FID decreases from 39.31 to 21.90, layout CAS increases from 72.42 to 78.77, and NIR PSNR increases from 15.70 to 18.68. These results support retaining the fixed, Stage I-adapted conditioning representation rather than updating the MLLM with the generation objective under the evaluated training configuration. **(3) Further evaluation of understanding** To examine whether updating the MLLM during generation training affects its understanding ability, we evaluate the final MLLM from D on the same understanding benchmarks and compare it with the Stage I-adapted MLLM retained in E. *Table R3. Understanding performance before and after generation training.* $$\begin{array}{l|ccccc} \hline \text{MLLM state} & \text{RSVQA-LR Avg. Acc.}\uparrow & \text{VRSBench Acc.}\uparrow & \text{Region R-1}\uparrow & \text{Region R-L}\uparrow & \text{Region METEOR}\uparrow \\\\ \hline \text{Updated (D)} & 92.09 & 49.78 & 45.94 & 45.94 & 40.13 \\\\ \text{Frozen (E)} & \mathbf{92.52} & \mathbf{49.89} & \mathbf{85.30} & \mathbf{85.30} & \mathbf{80.60} \\\\ \hline \end{array}$$ When the MLLM is updated with the generation objective, RSVQA-LR and VRSBench accuracy drop by 0.43 and 0.11 percentage points, respectively, while region understanding ROUGE-1, ROUGE-L, and METEOR drop by 39.36, 39.36, and 40.47 points. Together with the generation results above, updating the MLLM not only fails to improve generation but is also accompanied by a substantial degradation in region understanding. Freezing the MLLM preserves the understanding ability acquired in Stage I and yields better generation results, supporting the freezing choice under the current training configuration. *(Continued in the next comment.)*
AuthorsOpenReview ↗

Response to Reviewer kzuG (2/3)

(Q1, continued.)

(3) DVC vs. conditioning through the MLLM

To examine the role of DVC, we built a baseline that replaces the reference-image conditioning pathway. Since editing tasks require access to the reference image, we did not simply remove all reference-image inputs. Instead, we fed the reference image into the MLLM and conditioned the generator on its hidden states, while removing the direct VAE reference-image pathway. This experiment compares the full Omni-RS with this alternative, which passes reference-image information through the MLLM.

Table R4. Replacing the reference-image conditioning pathway.

\hline \text{Conditioning setting} & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{w/o DVC} & 50.02 & 57.94 & 24.04 & 11.04 & 8.54 \\\\ \text{Full Omni-RS} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ NIR and SAR denote RGB→NIR and RGB→SAR, respectively. Compared with the alternative baseline, the full model reduces inpainting and outpainting FID from 50.02 and 57.94 to 31.23 and 35.33, raises layout CAS from 24.04 to 78.77, and raises NIR and SAR PSNR from 11.04 and 8.54 to 18.68 and 12.89, respectively. The full model achieves better results on all five reference-conditioned evaluation. This indicates that, under the compared training settings, passing reference-image information only through the MLLM cannot replace the direct VAE visual conditioning pathway used in the full model, supporting the practical value of DVC for reference-conditioned generation tasks. *(Continued in the next comment.)*
AuthorsOpenReview ↗

Response to Reviewer kzuG (3/3)

Q2/W2: Benefits and trade-offs of multi-task joint training

We thank the reviewer for raising this question. We added a controlled comparison between single-task and multi-task training to examine whether the unified model brings performance benefits beyond task coverage.

In the single-task setting (S), a separate model is trained for each generation task; in the multi-task setting (E), one model learns all generation tasks jointly. All models are initialized from the same generator weights and the same Stage I MLLM, which is kept frozen during generation training. Both settings use the same training samples for each task and the same generator optimization configuration.

Table R5. Single-task training vs. multi-task joint training.

\hline \text{Training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Single-task (S)} & 42.97 & 23.06 & 44.43 & 37.65 & 56.11 & 18.44 & \mathbf{12.99} \\\\ \text{Multi-task (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & 12.89 \\\\ \hline \end{array}$$ NIR and SAR denote RGB→NIR and RGB→SAR, respectively. The multi-task model achieves better metrics in five of the six evaluation settings. T2I FID decreases from 42.97 to 21.90 and CLIP increases from 23.06 to 24.85; inpainting FID decreases from 44.43 to 31.23; layout CAS increases from 56.11 to 78.77. Outpainting FID also decreases from 37.65 to 35.33, and NIR PSNR increases from 18.44 to 18.68. T2I shows a substantial improvement. In UniGen, T2I and outpainting share the same Git-10M source images (Appendix Table 8), allowing the multi-task model to additionally learn scene reconstruction under partial-image conditioning. This complementary training signal may contribute to the improvement, although this comparison does not isolate its contribution from that of other tasks. These results support that, under the compared training settings, one shared generation model outperforms separately trained models on most tasks, so unified training has measurable performance value, along with task-dependent trade-offs. Some tasks in UniGen share source scenes and spatial conditions, which provides correlated information for cross-task learning. In the revision, we will include these results together with the detailed training budgets and per-task sample exposure, so that the benefits of joint training and the comparison conditions can be directly verified. We thank the reviewer again for the constructive comments. The additional experiments above examine the individual contributions of Stage I understanding adaptation, freezing the MLLM during generation training, and DVC, as well as the benefits and trade-offs of multi-task joint training relative to single-task training. These results will be added to the revision. We hope these additions address your concerns, and we are happy to provide further clarification if needed.
Reviewer_yKHyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Omni-RS, a unified framework for remote sensing image understanding and multi-task generation built on Qwen2.5-VL and OmniGen2. It first adapts the understanding model to remote sensing, then trains the generator using the model’s hidden states and VAE features from reference images. The authors also introduce UniGen, a dataset covering a wide range of remote sensing image generation tasks. Omni-RS achieves leading or competitive results across several understanding and generation benchmarks.

Critical strengths
  1. Omni-RS covers many important and meaningful remote sensing understanding and generation tasks, showing that a single framework can scale to a broader range of tasks.

  2. The authors put together UniGen by organizing data from multiple sources into a consistent format and achieve strong results across several tasks using this dataset.

Critical weaknesses
  1. The paper covers many tasks, but some of the baselines are relatively weak, and it does not compare against OmniGen2, the most directly relevant baseline. This makes it hard to tell whether the gains come from the proposed method or simply from fine-tuning on remote sensing data. I don’t find the current results convincing enough.

  2. The experiments mostly report final results without much deeper analysis. For example, does better understanding actually help generation? Does training these tasks together improve performance compared with training them separately? These questions matter more to me than the number of tasks the framework supports.

  3. The methodological novelty is also unclear. The approach uses an understanding model to extract semantic features and guide a separate diffusion model, which is already a common way to connect understanding and generation.

Overall, from a remote sensing domain perspective, I’m not sure what scaling up the number of tasks adds beyond broader coverage. Do these tasks actually benefit each other? To me, that is the key motivation for bringing understanding and generation together. From the perspective of unified understanding and generation models, the method feels old and offers limited insight into how to adapt to a specialized domain.

Questions for authors

My main concerns are covered in the weaknesses above. I might reconsider my assessment if the paper provided more insights specific to remote sensing or ablation studies on the key design choices.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yKHy (1/3)

We thank the reviewer for the careful review. We have added a direct comparison with OmniGen2, as well as controlled experiments on RS understanding adaptation and multi-task training. Below we respond to the three Critical Weaknesses in turn.

Q1/W1: Baseline comparison and the contribution of RS fine-tuning

Concern (summarized): Some baselines are relatively weak, and the lack of a direct comparison with OmniGen2 makes it difficult to determine whether the gains come from the proposed approach or simply from fine-tuning on RS data.

We thank the reviewer for raising this key issue. We added a zero-shot evaluation of the original OmniGen2, as well as a controlled comparison with OmniGen2 fine-tuned on the same RS generation data.

Specifically, the OmniGen2 RS fine-tuned baseline (F) retains its original Qwen2.5-VL-3B MLLM, keeps it frozen, and performs multi-task generation training on UniGen. Omni-RS (E) uses the same MLLM after Stage I RS understanding adaptation, likewise frozen during generation training. Both generators are initialized from the same OmniGen2 weights and use identical generation training data, task sampling ratios, training steps, and generator optimization settings.

Table R1. Direct comparison with OmniGen2.

\hline \text{Method} & \text{Stage I} & \text{UniGen} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{OmniGen2 zero-shot (O)} & \text{No} & \text{No} & 159.06 & 24.23 & 97.18 & 99.05 & 11.25 & 14.13 & 10.81 \\\\ \text{OmniGen2 RS fine-tuned (F)} & \text{No} & \text{Yes} & 62.97 & 21.98 & 39.23 & 40.52 & 47.90 & 12.17 & 12.19 \\\\ \text{Omni-RS (E)} & \text{Yes} & \text{Yes} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ NIR and SAR denote RGB→NIR and RGB→SAR, respectively. The most direct evidence comes from comparing F and E. With Stage I understanding adaptation, T2I FID drops from 62.97 to 21.90 and CLIP rises from 21.98 to 24.85; inpainting and outpainting FID drop from 39.23 and 40.52 to 31.23 and 35.33; layout CAS rises from 47.90 to 78.77; and NIR and SAR PSNR increase by 6.51 dB and 0.70 dB, respectively. We would also like to clarify how Omni-RS differs from OmniGen2. Omni-RS initializes its generator from OmniGen2, as stated in the paper, but OmniGen2 conditions generation on a frozen general-domain MLLM and does not target RS understanding. In Omni-RS, the Scene-Guided Two-Stage Training first turns the MLLM into an RS understanding model (RSVQA-LR 67.15→92.52, GeoChat ROUGE-1 17.53→85.30; Table R2 in Q2) and then uses its adapted hidden states as generation conditions, so that a single model serves both RS understanding and multi-task RS generation.
AuthorsOpenReview ↗

Response to Reviewer yKHy (2/3)

Q2/W2: Does understanding help generation, and does joint training help?

Concern (summarized): Does better understanding actually help generation? Does training these tasks together improve performance compared with training them separately?

(1) Does better RS understanding help generation?

Stage I substantially improves the RS understanding of the MLLM:

Table R2. RS understanding before and after Stage I.

\hline \text{MLLM} & \text{RSVQA-LR Avg. Acc.}\uparrow & \text{VRSBench Acc.}\uparrow & \text{GeoChat R-1}\uparrow & \text{GeoChat R-L}\uparrow & \text{GeoChat METEOR}\uparrow \\\\ \hline \text{Original Qwen2.5-VL-3B} & 67.15 & 47.13 & 17.53 & 16.81 & 15.75 \\\\ \text{Stage I-adapted} & \mathbf{92.52} & \mathbf{49.89} & \mathbf{85.30} & \mathbf{85.30} & \mathbf{80.60} \\\\ \hline \end{array}$$ This improvement transfers to generation. To further rule out the effect of generator training, we fix the native OmniGen2 generator (without RS generation training) and only replace the MLLM, keeping the VAE, condition projection layers, test samples, prompts, reference images, and sampling settings identical. *Table R3. Replacing only the MLLM, with the generator fixed.* $$\begin{array}{l|ccccccc} \hline \text{MLLM} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR MAE}\downarrow & \text{SAR MAE}\downarrow \\\\ \hline \text{Original (A)} & 159.06 & \mathbf{24.23} & 97.18 & 99.05 & 11.25 & 0.40 & 0.61 \\\\ \text{Stage I-adapted (B)} & \mathbf{134.39} & 23.92 & \mathbf{86.78} & \mathbf{94.38} & \mathbf{16.00} & \mathbf{0.23} & \mathbf{0.39} \\\\ \hline \end{array}$$ Even with the generator unchanged, the adapted MLLM improves T2I, inpainting, and outpainting FID, raises layout CAS, and reduces cross-modal translation error; T2I CLIP decreases slightly (24.23→23.92), so we do not claim gains on every metric. Together, the two comparisons show that Stage I adaptation, which improves RS understanding, also benefits generation. **(2) Does joint multi-task training outperform separate training?** We compare separately trained single-task models (S) with one multi-task model (E). All models start from the same generator weights and the same frozen Stage I MLLM, and use the same per-task training samples, training steps, and optimization settings; each single-task model sees its own task's samples as many times as the multi-task model does. *Table R4. Single-task vs. multi-task training.* $$\begin{array}{l|ccccccc} \hline \text{Training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Single-task (S)} & 42.97 & 23.06 & 44.43 & 37.65 & 56.11 & 18.44 & \mathbf{12.99} \\\\ \text{Multi-task (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & 12.89 \\\\ \hline \end{array}$$ The multi-task model is better in five of the six settings, while SAR PSNR drops slightly (12.99→12.89). Joint training therefore has measurable value beyond task coverage, with task-dependent trade-offs. T2I shows a substantial improvement (FID 42.97→21.90). In UniGen, T2I and outpainting share the same Git-10M source images (Appendix Table 8), with source-image splits aligned across tasks to prevent training–test overlap. The multi-task model additionally learns scene reconstruction under partial-image conditioning, which may provide a complementary training signal. However, this comparison does not isolate the contribution of outpainting or shared-scene construction. We additionally evaluated understanding retention after generation training. Updating the Stage I-adapted MLLM reduced region-understanding ROUGE-1 from 85.30 to 45.94 and METEOR from 80.60 to 40.13. Generation performance also declined relative to freezing the MLLM, with T2I FID increasing from 21.90 to 39.31 and NIR PSNR decreasing from 18.68 to 15.70. These results further support freezing the adapted MLLM under the evaluated training configuration.
AuthorsOpenReview ↗

Response to Reviewer yKHy (3/3)

Q3/W3: Methodological contribution and remote sensing-specific insights

Concern (summarized): Using an understanding model to provide semantic features for a diffusion generator is an established paradigm, and supporting more tasks alone does not establish a clear contribution.

We thank the reviewer for asking us to clarify what the work contributes beyond this established architecture. Our central question is how to build a unified remote sensing model in which domain understanding provides useful conditions for multiple generation tasks, and whether training these tasks together offers benefits beyond broader task coverage.

Our contribution has three connected aspects.

First, a remote sensing understanding-guided training strategy with controlled evidence of its value. Omni-RS first adapts the MLLM to remote sensing understanding and then uses its representations to condition multi-task generation. Rather than assuming that better domain understanding helps generation, we test this directly. Under matched generation training conditions, the adapted MLLM improves generation over the original MLLM; replacing the MLLM also improves several generation metrics when the generator is held fixed. These experiments establish the value of RS understanding adaptation within this framework, beyond direct fine-tuning on RS generation data.

Second, UniGen and a unified model for heterogeneous remote sensing generation tasks. UniGen organizes different data sources into a common instruction–reference–target format and constructs multiple tasks from some of the same source scenes. This supports joint training across T2I, inpainting, outpainting, layout control, and cross-modal translation. The single-task comparison shows that the resulting model improves five of six evaluation settings, demonstrating performance benefits beyond simply combining task interfaces.

Third, extending the capabilities of unified RS understanding–generation models. Our comparison context for DVC is existing remote sensing unified models such as Uni-RS, whose generation capability is primarily limited to T2I. Reference-conditioned tasks require a pathway for transmitting information from the source image. DVC provides this pathway in Omni-RS, enabling these tasks within the unified framework. It adopts a VAE reference-feature injection mechanism similar to OmniGen2; we will make this inheritance explicit rather than present the injection mechanism itself as a new general-purpose architectural component.

We acknowledge that the underlying MLLM–diffusion architecture and reference-feature injection are established techniques. The contribution lies in the remote sensing understanding-guided multi-task framework, the data construction supporting it, and the controlled evidence establishing its benefits and trade-offs. We will revise the abstract, contribution list, and method description to distinguish these contributions from the inherited components.

Reviewer_bemAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Omni-RS, a unified framework for remote sensing understanding and multi-task image generation. It adopts a Scene-Guided Two-Stage Training strategy, first adapting a MLLM to remote sensing understanding and then using its hidden states to guide a DiT for image generation. The authors also introduce Detailed Visual Conditioning to incorporate reference-image features and construct UniGen, a unified dataset covering five generation tasks. Experiments show competitive understanding performance and improvements on several generation benchmarks.

Critical strengths
  1. The paper addresses a relevant problem in remote sensing. Combining visual understanding and image generation within a unified framework is useful for applications requiring both scene interpretation and image manipulation. Extending generation beyond text-to-image synthesis to inpainting, outpainting, controllable generation, and cross-modal translation broadens the framework's practical applicability.
  2. The proposed framework is clearly presented and reasonably well organized. The two-stage training strategy separates remote sensing understanding from generation training, while the visual conditioning pathway provides additional information for reference-based generation. UniGen also organizes heterogeneous remote sensing datasets into a consistent format for multi-task training.
  3. The experimental evaluation covers a broad range of understanding and generation tasks, with comparisons against several understanding models and task-specific generation methods. The results are encouraging, particularly for inpainting, outpainting, and RGB-to-NIR/SAR translation, where Omni-RS achieves notable improvements over the evaluated baselines.
Critical weaknesses
  1. The methodological novelty relative to existing unified understanding and generation models is not sufficiently clear. OmniGen2 already uses a Qwen2.5-VL-based model to provide hidden-state conditioning for a diffusion transformer and incorporates VAE features from reference images. Omni-RS follows a similar architecture and explicitly initializes its generative backbone from OmniGen2. The proposed Detailed Visual Conditioning also appears closely related to the visual conditioning mechanism in OmniGen2, while freezing the understanding model during generation training is an established design choice. Although extending these techniques to remote sensing tasks is useful, the authors should clarify what is methodologically new beyond domain adaptation and multi-task integration.
  2. The paper lacks sufficient ablation experiments to support its main design choices. The Scene-Guided Two-Stage Training strategy is intended to reduce interference between understanding and generation, the paper does not compare this strategy with joint training or directly fine-tuning the generation backbone without the Stage I understanding adaptation. Similarly, the contribution of Detailed Visual Conditioning is not isolated by removing or replacing the reference-image conditioning pathway. It is also unclear whether multi-task generation training outperforms training separate task-specific models. Given the strong pretrained backbones and large training dataset, these comparisons are important for attributing the reported gains. A controlled comparison with OmniGen2 fine-tuned on the same data, together with component-wise ablations, would strengthen the evidence.
  3. Some experimental comparisons and evaluation settings need further clarification. In Table 3, Omni-RS achieves only a marginal FID improvement over Uni-RS (21.90 vs. 22.11), while its CLIP score is lower (24.85 vs. 25.68). Further clarification of the evaluation splits, particularly for UniGen and GeoChat, would strengthen the reliability of the results.
Questions for authors

The paper addresses a meaningful problem and reports promising results across a broad range of remote sensing tasks. The construction of UniGen and the integration of understanding and generation are useful contributions. However, the methodological differences from OmniGen2 remain unclear, and the lack of controlled ablations makes it difficult to attribute the reported improvements to the proposed designs. Questions about baseline comparability and evaluation protocols also remain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bemA (1/4)

We thank the reviewer for the careful review and for recognizing the practical value of unifying RS understanding and multi-task generation. We respond below to the questions on methodological contribution, controlled experiments, and evaluation protocols.

Q1/W1: Methodological novelty relative to OmniGen2

Concern (summarized): The authors should clarify what is methodologically new beyond domain adaptation and multi-task integration.

We thank the reviewer for raising the question of how our contributions are positioned. We agree that conditioning a DiT on MLLM hidden states, injecting VAE features of reference images, and freezing the MLLM during generation training all have precedents, and we do not claim these underlying mechanisms themselves as independent innovations.

Regarding DVC, the relevant point of comparison is RS unified understanding and generation models represented by Uni-RS. DVC adopts a reference-image VAE feature injection similar to OmniGen2; we introduce it to fill the capability gap of RS unified models in reference-conditioned generation. Uni-RS connects understanding and generation only through semantic conditions and lacks a pathway that directly passes the visual features of a reference image, and its generation is largely limited to T2I. Inpainting, outpainting, layout control, and cross-modal translation, in contrast, require the spatial structure and visual details of the reference image.

To verify the practical role of this pathway, Q2/W2 (4) adds a replacement experiment: the reference image is fed into the MLLM to condition the generator through its hidden states, the direct VAE reference pathway is removed, and the generator is retrained. The full model outperforms this alternative on all five reference-conditioned settings, supporting the value of DVC in a unified RS framework.

Since the original manuscript did not clearly separate the inherited techniques from our RS-specific contributions, we will make explicit, in the abstract, contribution list, and method section, the technical lineage from OmniGen2 and the capability extension over Uni-RS.

AuthorsOpenReview ↗

Response to Reviewer bemA (2/4)

Q2/W2: Controlled experiments and attribution of gains

Concern (summarized): A controlled comparison with OmniGen2 fine-tuned on the same data, together with component-wise ablations, would strengthen the evidence.

We thank the reviewer for pointing out this weakness. We added a direct comparison with OmniGen2 fine-tuned on the same RS generation data, and separately examined Stage I understanding adaptation, the freezing strategy, multi-task joint training, and the reference-image conditioning pathway.

(1) Contribution of Stage I (vs. OmniGen2 fine-tuned on the same data)

We compare the OmniGen2 RS fine-tuned baseline with the original frozen MLLM (F) and Omni-RS with the Stage I-adapted frozen MLLM (E). Both generators are initialized from the same weights and use the same UniGen data, task sampling ratios, training budget, generator optimization settings, and reference-image conditioning pathway, which isolates the effect of RS understanding adaptation.

Table R1. Contribution of Stage I under identical generation training.

\hline \text{Method} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{w/o Stage I (F)} & 62.97 & 21.98 & 39.23 & 40.52 & 47.90 & 12.17 & 12.19 \\\\ \text{w/ Stage I (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Before T2I evaluation, both are further fine-tuned for two epochs on the same RSICD training set. NIR and SAR denote RGB→NIR and RGB→SAR. With Stage I, all listed metrics improve across all six settings; for example, T2I FID drops from 62.97 to 21.90 and layout CAS rises from 47.90 to 78.77. Under the same RS generation data and training budget, RS understanding adaptation thus provides additional value. **(2) Freezing the MLLM and two-stage training** We compare updating the Stage I MLLM during generation training (D) with keeping it frozen (E), using the same initialization, generation training data, task sampling, training budget, and generator optimization settings. *Table R2. Updating vs. freezing the MLLM during generation training.* $$\begin{array}{l|ccccccc} \hline \text{MLLM training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Updated (D)} & 39.31 & 23.24 & 36.67 & 36.49 & 72.42 & 15.70 & 11.81 \\\\ \text{Frozen (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Freezing improves all reported generation metrics across the six evaluation settings. We further evaluate whether updating the MLLM affects its retained understanding ability. *Table R3. Understanding performance before and after generation training.* $$\begin{array}{l|ccccc} \hline \text{MLLM state} & \text{RSVQA-LR Avg. Acc.}\uparrow & \text{VRSBench Acc.}\uparrow & \text{Region R-1}\uparrow & \text{Region R-L}\uparrow & \text{Region METEOR}\uparrow \\\\ \hline \text{Updated (D)} & 92.09 & 49.78 & 45.94 & 45.94 & 40.13 \\\\ \text{Frozen (E)} & \mathbf{92.52} & \mathbf{49.89} & \mathbf{85.30} & \mathbf{85.30} & \mathbf{80.60} \\\\ \hline \end{array}$$ When the MLLM is updated with the generation objective, RSVQA-LR and VRSBench accuracy drop by 0.43 and 0.11 percentage points, while region understanding ROUGE-1, ROUGE-L, and METEOR drop by 39.36, 39.36, and 40.47 points. Updating the MLLM thus not only fails to improve generation but is also accompanied by a substantial degradation in region understanding, whereas freezing preserves the understanding ability acquired in Stage I and yields better generation results, supporting the Stage II freezing strategy under the current configuration. This comparison evaluates freezing during generation training, rather than joint optimization of understanding and generation losses.
AuthorsOpenReview ↗

Response to Reviewer bemA (3/4)

(Q2/W2, continued.)

(3) Benefits of multi-task joint training

We compare separately trained single-task models (S) with one multi-task model (E). All models are initialized from the same Stage I MLLM and generator weights, keep the MLLM frozen, and use the same per-task training samples. Each single-task model sees its own task's samples at least as many times as the multi-task model does, ensuring that the single-task baselines are not disadvantaged in per-task sample exposure.

Table R4. Single-task vs. multi-task training.

\hline \text{Training} & \text{T2I FID}\downarrow & \text{T2I CLIP}\uparrow & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{Single-task (S)} & 42.97 & 23.06 & 44.43 & 37.65 & 56.11 & 18.44 & \mathbf{12.99} \\\\ \text{Multi-task (E)} & \mathbf{21.90} & \mathbf{24.85} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & 12.89 \\\\ \hline \end{array}$$ All T2I models are further fine-tuned for two epochs on the same RSICD training set. The single-task and multi-task settings use the same number of training steps and the same generator optimization configuration. The multi-task model achieves better metrics in five of the six settings: T2I FID drops from 42.97 to 21.90, inpainting FID from 44.43 to 31.23, and layout CAS rises from 56.11 to 78.77, while SAR PSNR decreases slightly from 12.99 to 12.89. Unified training therefore brings performance gains beyond task coverage. **(4) DVC vs. conditioning through the MLLM** To examine the role of DVC, we built a baseline that replaces the reference-image conditioning pathway. Since editing tasks require the reference image, we did not simply remove all reference inputs; instead, we fed the reference image into the MLLM and conditioned the generator on its hidden states, while removing the direct VAE pathway. The generator is retrained under this alternative setting, rather than merely disabling DVC at inference. *Table R5. Replacing the reference-image conditioning pathway.* $$\begin{array}{l|ccccc} \hline \text{Conditioning setting} & \text{Inpaint FID}\downarrow & \text{Outpaint FID}\downarrow & \text{Layout CAS}\uparrow & \text{NIR PSNR}\uparrow & \text{SAR PSNR}\uparrow \\\\ \hline \text{w/o DVC} & 50.02 & 57.94 & 24.04 & 11.04 & 8.54 \\\\ \text{Full Omni-RS} & \mathbf{31.23} & \mathbf{35.33} & \mathbf{78.77} & \mathbf{18.68} & \mathbf{12.89} \\\\ \hline \end{array}$$ Both settings retain reference-image information; the baseline conditions through the MLLM, while the full model uses the direct DVC pathway. Compared with the baseline, the full model reduces inpainting and outpainting FID from 50.02 and 57.94 to 31.23 and 35.33, raises layout CAS from 24.04 to 78.77, and raises NIR and SAR PSNR from 11.04 and 8.54 to 18.68 and 12.89, performing better on all five reference-conditioned settings. This supports the practical contribution of the direct VAE reference pathway.
AuthorsOpenReview ↗

Response to Reviewer bemA (4/4)

Q3/W3: T2I comparison and data split protocols

Concern (summarized): In Table 3, Omni-RS achieves only a marginal FID improvement over Uni-RS, while its CLIP score is lower, Further clarification of the evaluation splits, particularly for UniGen and GeoChat.

(1) T2I comparison with Uni-RS and the number of fine-tuning epochs

We thank the reviewer for highlighting this comparison. The Uni-RS results in Table 3 were taken directly from its paper, which reports five epochs of RSICD fine-tuning before evaluation. In contrast, the Omni-RS results in our original manuscript were obtained after two epochs of RSICD fine-tuning. We acknowledge that this difference was not clearly stated. We have therefore additionally evaluated Omni-RS with five epochs of RSICD fine-tuning and will clarify the result sources and fine-tuning settings in the revision.

Table R6. T2I results on RSICD and the number of fine-tuning epochs.

\hline \text{Method} & \text{RSICD epochs} & \text{FID}\downarrow & \text{CLIP}\uparrow \\\\ \hline \text{Uni-RS} & 5 & 22.11 & 25.68 \\\\ \text{Omni-RS (original)} & 2 & 21.90 & 24.85 \\\\ \text{Omni-RS (new)} & 5 & \mathbf{13.07} & \mathbf{26.63} \\\\ \hline \end{array}$$ With five epochs instead of two, Omni-RS improves FID from 21.90 to 13.07 and CLIP from 24.85 to 26.63. Under the matched five-epoch setting, Omni-RS reduces FID by 9.04 and improves CLIP by 0.95 over Uni-RS, i.e., it is better on both T2I metrics. We will clearly mark the number of fine-tuning epochs in the revision to avoid comparing results obtained under different training settings without explanation. **(2) Training/test splits of UniGen** The data sources and train/test sizes of each UniGen task are listed in Appendix Table 8. For source data with official splits, we follow the official train/test split and propagate each source image's split assignment to all task samples built from it. For example, T2I and outpainting samples built from the same source image always belong to the same split, so no image is used for training in one task and for testing in another. We will specify the split protocol for each data source in the revision. **(3) Train/test isolation for GeoChat** Stage I training and region-understanding evaluation on GeoChat follow the official train/test split: Stage I uses the training portion, and region understanding is evaluated on the corresponding test portion. We will state the subset and evaluation protocol in the revision to clarify that training and evaluation do not use the same data. The original Qwen2.5-VL and the Stage I model use the same evaluation set, prompt template, decoding, and answer post-processing. These additions will allow readers to distinguish dataset composition, actual evaluation sets, and cross-set isolation, and to interpret the T2I comparison accurately. We thank the reviewer again; these comments helped us clarify the scope of our contributions and strengthen the evidence for our core conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.