- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SynUrbanSAT, a controllable satellite imagery synthesis framework for urban semantic 3D reconstruction. The framework combines OpenStreetMap-based 3D scene construction with diffusion-based image generation, introducing a heterogeneous dual-condition adapter (HDCA) to incorporate semantic and height information. The authors construct SynUrbanSAT-20, containing over 20,000 samples across 20 cities, and demonstrate improvements in image fidelity and downstream semantic segmentation, height estimation, and 3D reconstruction.
Critical strengths
- Generating geographically diverse training data is an important and practical problem.
- This paper integrates real-world city layout and controllable semantics into the diffusion-based image generation process.
- The authors conduct extensive experiments to evaluate image fidelity, downstream segmentation, height estimation, and 3D reconstruction
Critical weaknesses
-
The technical novelty is limited. The overall framework primarily integrates existing techniques, including OSM-based procedural modeling, pretrained diffusion models, ControlNet, and LoRA. Additionally, please include an ablation of each submodule.
-
Although the pipeline produces aligned semantic and height annotations, diffusion-generated RGB images may not faithfully reflect the prescribed geometry. The paper lacks quantitative evaluation of semantic boundary alignment and height consistency between generated images and their conditioning maps.
-
More existing ControlNet or diffusion-based studies should be considered as baselines.
-
ControlNet provides conditional guidance but does not guarantee that generated RGB images strictly follow the input semantic layouts and metric-height maps. Some theoretical guarantees or quantitative results are expected to be provided.
Questions for authors
-
Spatial consistency of generated imagery. ControlNet provides conditional guidance but does not guarantee that generated RGB images strictly follow the input semantic layouts and metric-height maps. Given the importance of aligned supervision for 3D reconstruction, the authors should quantitatively evaluate semantic boundary alignment and geometric consistency, rather than relying primarily on qualitative visualizations and downstream performance.
-
Fair comparison with existing ControlNet methods. Although several ControlNet-based baselines are included, the experiments do not clearly isolate the contribution of HDCA from domain adaptation and LoRA fine-tuning. A comparison with standard semantic- and height-conditioned ControlNets under the same backbone, training budget, and adaptation settings would better establish the necessity of HDCA.
-
Comparison with unified multimodal generation models. Recent unified understanding-and-generation models, such as OmniGen2 and BAGEL, support instruction-based and image-conditioned generation without requiring task-specific conditioning architectures. How does SynUrbanSAT compare with these approaches when directly generating satellite imagery from semantic layouts?
-
Zero-shot multimodal generation baselines. It would also be informative to evaluate whether commercial multimodal image-generation models, such as Gemini Image or GPT Image, can directly synthesize realistic satellite imagery from semantic maps. Such comparisons could help demonstrate the practical advantages of the proposed specialized pipeline, particularly in terms of spatial consistency, geometric fidelity, and downstream utility.
4 comments
Response to Reviewer ggdB (2/3)
W2 / W4 / Q1. Spatial boundary alignment, geometric height consistency, and quantitative evaluation
Diffusion conditioning provides soft guidance. We therefore evaluate label adherence using an independent reader (DINOv2-L/14 + DPT) trained exclusively on real US3D images, with real held-out RGB as the reference. The results support semantic and height adherence on the evaluated layouts.
1. Semantic boundary alignment. On held-out real layouts, our generated RGB closely matches real-image references in 6-class mIoU ( vs. ), building BF@2px ( vs. ), and boundary displacement ( vs. ). Across shared OSM layouts, HDCA improves pooled mIoU by points and BF@2px by points over unadapted S&H ControlNets:
| Images ( OSM layouts) | mIoU (%) | Bldg IoU (%) | BF@2px (%) | Boundary disp. (m) |
|---|---|---|---|---|
| Real US3D test52 (reference) | 70.02 | 75.28 | 74.39 | 3.0 |
| Blender PBR render | 36.53 | 35.49 | 45.18 | 7.2 |
| FLUX.2 + PBR reference | 32.84 | 36.79 | 55.66 | 5.4 |
| FLUX.2 + depth ControlNet | 27.58 | 27.45 | 35.39 | 7.9 |
| FLUX.2 + S & H ControlNets | 26.66 | 21.23 | 32.66 | 8.5 |
| Ours (HDCA) | 60.43 | 68.11 | 66.71 | 3.1 |
2. Geometric height consistency and controlled edits. We evaluate static height readout ( held-out tiles), height-appearance slope ratio ( val tiles), and controlled height edits across buildings (fixing layout, camera, prompt, and noise):
| Height Consistency & Control Metric | Real Ref. | Semantic-only | Ours (HDCA) |
|---|---|---|---|
| Building-height readout RMSE on held-out tiles (m) | 2.93 | 3.34 | 2.86 |
| Generated-to-real height slope ratio on val tiles (ref ) | 1.00 | 0.51 | 0.92 |
| Correct direction under height edits (%) | — | — | 85.8% |
HDCA closely matches real-RGB height fidelity and accurately tracks directional height edits, providing more reliable 2D/3D alignment for downstream learning.
W3. Additional ControlNet and diffusion baselines
We expand our evaluation with six additional baselines:
- Equally trained ControlNet variants (
semantic-only,concatenation): Under identical backbones and training budgets, HDCA achieves the best height rank preservation and downstream height estimation (Q2). - Open-source multimodal (OmniGen2, BAGEL) & remote-sensing (GeoSynth) models: Our pipeline substantially outperforms all three in semantic and building boundary alignment (Q3).
- Commercial image generation (GPT-Image-2): Our pipeline achieves higher overall mIoU and lower height error, and responds far more reliably to height edits (Q4).
Q2. Fair comparison with ControlNet conditioners under the same backbone and budget
To isolate HDCA from domain adaptation and LoRA fine-tuning, we train
semantic-only and concatenation baselines under the exact same configuration. While simpler conditioners perform better on 2D semantics, HDCA improves pooled height rank correlation and downstream height RMSE.
1. Condition adherence ( OSM layouts).
Evaluated by the real-trained independent reader, semantic-only and
concatenation score higher on 2D semantics, whereas HDCA achieves the highest building-height rank correlation (), best preserving input height ordering:
| Conditioning | 6-class mIoU (%) | Bldg BF@2px (%) | Height rank |
|---|---|---|---|
| Semantic-only | 63.44 | 75.97 | 0.102 |
| Concatenation | 64.56 | 72.01 | 0.186 |
| Ours (HDCA) | 60.43 | 66.71 | 0.240 |
2. Downstream height estimation (Building-height RMSE in meters ). Training identical height predictors on each configuration yields:
| Synthesis Cohort | Test Set | Semantic-only | Concatenation | Ours (HDCA) |
|---|---|---|---|---|
| SynRS3D layouts | US3D / GAMUS | 25.37 / 26.61 | 17.89 / 20.46 | 8.05 / 14.43 |
| OSM layouts | US3D / GAMUS | 8.27 / 10.38 | 8.23 / 10.23 | 7.69 / 10.12 |
HDCA substantially reduces height RMSE on SynRS3D ( vs.
concatenation, vs. semantic-only) and Fixed OSM.
3. Why we select HDCA for our target downstream tasks. We choose HDCA because this work specifically targets urban semantic 3D reconstruction and height-sensitive urban simulations (e.g., city-scale wind-field modeling and seismic assessment of regional buildings, Sec. 4.3). We therefore prioritize height fidelity via HDCA. Meanwhile, because SynUrbanSAT is a modular pipeline, users whose downstream tasks only require 2D land-cover segmentation can readily swap in the semantic-only or concatenation module.
Response to Reviewer ggdB (3/3)
Q3. Comparison with unified multimodal and remote-sensing generation models
We benchmark our pipeline against unified multimodal models (OmniGen2, BAGEL) and a remote-sensing OSM-conditioned diffusion model (GeoSynth) across the held-out layouts using our real-trained independent reader. Our pipeline gives better building alignment and lower height readout error under the tested workflows.
| Generator | 6-class mIoU (%) | Bldg IoU (%) | Bldg BF@2px (%) | Height RMSE (m) |
|---|---|---|---|---|
| Real RGB (reference) | 71.12 | 79.18 | 78.87 | 2.93 |
| OmniGen2 | 16.12 | 12.44 | 12.02 | 6.42 |
| BAGEL | 21.99 | 27.62 | 34.23 | 6.85 |
| GeoSynth | 22.97 | 55.67 | 44.58 | 4.46 |
| Ours (HDCA) | 70.07 | 75.75 | 74.39 | 2.86 |
General multimodal models struggle with overhead urban layouts, and even against the strongest external baseline (GeoSynth), our pipeline improves building IoU by points, BF@2px by points, and reduces height readout RMSE from to . When OmniGen2 and BAGEL additionally receive the height map through their native multi-image interfaces ( OSM layouts), only and of buildings respond to a height edit in the requested direction, against for ours. These comparisons form Table 1 of the revised paper.
Q4. Comparison with zero-shot commercial image-generation models
We benchmark GPT-Image-2 against our pipeline with identical semantic and height inputs. The commercial model reproduces building outlines well, while our pipeline achieves higher overall semantic agreement, lower height error, and a far stronger response to height edits.
1. Label adherence ( held-out layouts, independent reader).
| Generator | 6-class mIoU (%) | Bldg IoU (%) | Bldg BF@2px (%) | Height RMSE (m) |
|---|---|---|---|---|
| Real RGB (reference) | 71.12 | 79.18 | 78.87 | 2.93 |
| GPT-Image-2 | 59.82 | 79.83 | 75.04 | 4.44 |
| Ours | 70.07 | 75.75 | 74.39 | 2.86 |
2. Height controllability & downstream utility ( matched layouts). We evaluate height edits on buildings, and train identical roof-height probes on each generator's images, tested on real GAMUS roofs from unseen cities:
| Evaluation Aspect | Metric | GPT-Image-2 | Ours (HDCA) |
|---|---|---|---|
| Height control | Correct edit direction (%) | 59.7 | 81.6 |
| Height control | Read height change per requested m (m/m) | 0.021 | 0.065 |
| Downstream | Real-roof height RMSE of the probe (m) | 6.94 | 4.63 |
The commercial model does not translate metric height maps into consistent visual cues: its height readout error is higher ( vs. ), and only of edited buildings respond in the requested direction, against for ours ( points). Its overall mIoU is also points lower. These comparisons are reported in Table 1 and Appendix A.9 of the revised paper.
Follow-up to Reviewer ggdB: New Comparisons with OmniGen2, BAGEL, GPT-Image-2 & HDCA Ablations
Dear Reviewer ggdB,
Thank you very much for your valuable feedback, which prompted us to add six new baselines and comprehensive quantitative evaluations. As the discussion period nears its end, we wanted to briefly summarize how our rebuttal and revised paper address all of your concerns:
- Novelty & HDCA Submodule Ablations (W1): We clarified our framework- and module-level contributions and ran submodule ablations (Table 5), confirming that replacing learned gated fusion with fixed fusion drops mIoU by points and height rank by , while disabling boundary features lowers BF@2px by points.
- Quantitative Spatial & Height Consistency (W2/W4/Q1): Using an independent predictor trained on real US3D images, HDCA closely matches real held-out RGB in 6-class mIoU ( vs. ), boundary displacement ( vs. ), and height RMSE ( vs. ), while driving of buildings in the target direction under height edits.
- Equal-Budget ControlNet Baselines (W3/Q2): Under the exact same backbone, real triplets, and LoRA budget, HDCA achieves the best height rank preservation () and lowest downstream height RMSE over
semantic-onlyandconcatenationvariants. - Unified Multimodal & Commercial Baselines (Q3/Q4): In Table 1 and Appendix A.9, our pipeline substantially outperforms OmniGen2, BAGEL, and GeoSynth (e.g., building IoU and height RMSE over GeoSynth). Against GPT-Image-2, our method achieves higher overall mIoU, lower height error ( vs. ), and far stronger height-edit adherence ( vs. ).
We hope these new experiments and comparisons satisfactorily resolve your concerns. Please let us know if any further clarification is needed before the window closes!
Best regards,
The Authors
Response to Reviewer ggdB (1/3)
We thank the reviewer for the careful reading and for recognizing the importance of generating geographically diverse training data. Below we present the corresponding experiments and comparisons, with all revisions highlighted in blue in the updated manuscript.
W1. Technical novelty, and ablation of each submodule
We clarify our novelty at the framework and module levels, followed by HDCA ablations.
1. Core novelty and comprehensive advantages. Existing remote-sensing synthesis methods face a dilemma: rendering-based pipelines offer 3D labels but lack visual realism and real-world layouts, while diffusion-based methods generate realistic RGB but focus on 2D semantics without aligned metric height supervision. The primary novelty of SynUrbanSAT lies in bridging this gap with a scalable, controllable synthesis framework that unifies real-world OSM layouts, configurable 3D scene modeling, and joint semantic-height conditioned diffusion. Beyond cost-effectively constructing SynUrbanSAT-20 dataset, our pipeline shows clear advantages across three dimensions:
At the module level, HDCA is a task-oriented conditioner designed for height-critical applications (e.g., city-scale wind CFD and seismic simulations of regional buildings). While our modular pipeline allows swapping in
semantic-onlyconditioning for 2D tasks (Q2), HDCA achieves the strongest height adherence and downstream building-height RMSE.2. HDCA submodule evaluation. (The full ablation of HDCA is in Q2, Section 4.4, Table 5)
We test HDCA via (i) a retrained ablation using fixed 0.5/0.5 fusion under the same budget, and (ii) inference interventions disabling Fourier or boundary features.
Fixed fusion hurts mIoU (−3.06) and height ρ (0.240→0.171), and disabling the boundary features lowers BF@2px by 15.02 points. The trained model also relies on its Fourier features (height RMSE 2.86→3.30 m when disabled), although a variant retrained with a scalar height input reaches similar adherence (mIoU 59.83%, ρ=0.261). The separate branches and the learned fusion therefore matter more than the height encoding itself.