acceptodds
Under review as a conference paper at ICLR 2027

SynUrbanSAT: Controllable Satellite Data Synthesis for Urban Semantic 3D Reconstruction

Abstract

Urban semantic 3D reconstruction from monocular satellite imagery requires geographically diverse data with aligned semantic and height supervision, yet acquiring precise annotations at scale remains costly. Existing synthesis methods struggle to jointly provide real-world urban layouts, realistic appearance, and spatially aligned supervision. To address this gap, we propose SynUrbanSAT, a scalable and controllable pipeline that combines OpenStreetMap (OSM)-based 3D scene construction with diffusion-based satellite image synthesis. A heterogeneous dual-condition adapter (HDCA) integrates categorical semantics and continuous metric heights to guide the generation of realistic satellite imagery. The pipeline supports controllable data augmentation by varying building-height distributions, vegetation configurations, viewpoints, and appearance. It enables scalable synthesis of geographically diverse data across cities with a single trained generator, while offering the option to train a dedicated generator on limited target-city imagery for local data augmentation. To support a broad range of downstream urban perception and reconstruction tasks, we construct SynUrbanSAT-20, comprising 20,097 samples across 20 cities with synthetic RGB images and aligned semantic labels, height maps, polygon maps, point clouds, and 3D meshes. Experiments demonstrate that our image generation method improves visual fidelity and the utility of synthetic data for downstream learning. The complete pipeline further outperforms the evaluated synthetic training sources in urban semantic 3D reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, Satellite Image Synthesis, Controllable Diffusion Models, Urban Semantic 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ggdBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SynUrbanSAT, a controllable satellite imagery synthesis framework for urban semantic 3D reconstruction. The framework combines OpenStreetMap-based 3D scene construction with diffusion-based image generation, introducing a heterogeneous dual-condition adapter (HDCA) to incorporate semantic and height information. The authors construct SynUrbanSAT-20, containing over 20,000 samples across 20 cities, and demonstrate improvements in image fidelity and downstream semantic segmentation, height estimation, and 3D reconstruction.

Critical strengths
  • Generating geographically diverse training data is an important and practical problem.
  • This paper integrates real-world city layout and controllable semantics into the diffusion-based image generation process.
  • The authors conduct extensive experiments to evaluate image fidelity, downstream segmentation, height estimation, and 3D reconstruction
Critical weaknesses
  • The technical novelty is limited. The overall framework primarily integrates existing techniques, including OSM-based procedural modeling, pretrained diffusion models, ControlNet, and LoRA. Additionally, please include an ablation of each submodule.

  • Although the pipeline produces aligned semantic and height annotations, diffusion-generated RGB images may not faithfully reflect the prescribed geometry. The paper lacks quantitative evaluation of semantic boundary alignment and height consistency between generated images and their conditioning maps.

  • More existing ControlNet or diffusion-based studies should be considered as baselines.

  • ControlNet provides conditional guidance but does not guarantee that generated RGB images strictly follow the input semantic layouts and metric-height maps. Some theoretical guarantees or quantitative results are expected to be provided.

Questions for authors
  1. Spatial consistency of generated imagery. ControlNet provides conditional guidance but does not guarantee that generated RGB images strictly follow the input semantic layouts and metric-height maps. Given the importance of aligned supervision for 3D reconstruction, the authors should quantitatively evaluate semantic boundary alignment and geometric consistency, rather than relying primarily on qualitative visualizations and downstream performance.

  2. Fair comparison with existing ControlNet methods. Although several ControlNet-based baselines are included, the experiments do not clearly isolate the contribution of HDCA from domain adaptation and LoRA fine-tuning. A comparison with standard semantic- and height-conditioned ControlNets under the same backbone, training budget, and adaptation settings would better establish the necessity of HDCA.

  3. Comparison with unified multimodal generation models. Recent unified understanding-and-generation models, such as OmniGen2 and BAGEL, support instruction-based and image-conditioned generation without requiring task-specific conditioning architectures. How does SynUrbanSAT compare with these approaches when directly generating satellite imagery from semantic layouts?

  4. Zero-shot multimodal generation baselines. It would also be informative to evaluate whether commercial multimodal image-generation models, such as Gemini Image or GPT Image, can directly synthesize realistic satellite imagery from semantic maps. Such comparisons could help demonstrate the practical advantages of the proposed specialized pipeline, particularly in terms of spatial consistency, geometric fidelity, and downstream utility.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ggdB (1/3)

We thank the reviewer for the careful reading and for recognizing the importance of generating geographically diverse training data. Below we present the corresponding experiments and comparisons, with all revisions highlighted in blue in the updated manuscript.

W1. Technical novelty, and ablation of each submodule

We clarify our novelty at the framework and module levels, followed by HDCA ablations.

1. Core novelty and comprehensive advantages. Existing remote-sensing synthesis methods face a dilemma: rendering-based pipelines offer 3D labels but lack visual realism and real-world layouts, while diffusion-based methods generate realistic RGB but focus on 2D semantics without aligned metric height supervision. The primary novelty of SynUrbanSAT lies in bridging this gap with a scalable, controllable synthesis framework that unifies real-world OSM layouts, configurable 3D scene modeling, and joint semantic-height conditioned diffusion. Beyond cost-effectively constructing SynUrbanSAT-20 dataset, our pipeline shows clear advantages across three dimensions:

  • Vs. synthetic datasets: Best visual fidelity (DINO-FID/KID) and downstream 3D reconstruction among the evaluated synthetic datasets (Sec. 4).
  • Vs. open-source models: Improves building IoU over OmniGen2, BAGEL, and GeoSynth (W3/Q3).
  • Vs. commercial models: Higher overall mIoU, lower height error, and a stronger response to height edits than GPT-Image-2, which retains higher building IoU and boundary F-score (Q4).

At the module level, HDCA is a task-oriented conditioner designed for height-critical applications (e.g., city-scale wind CFD and seismic simulations of regional buildings). While our modular pipeline allows swapping in semantic-only conditioning for 2D tasks (Q2), HDCA achieves the strongest height adherence and downstream building-height RMSE.

2. HDCA submodule evaluation. (The full ablation of HDCA is in Q2, Section 4.4, Table 5)

We test HDCA via (i) a retrained ablation using fixed 0.5/0.50.5/0.5 fusion under the same budget, and (ii) inference interventions disabling Fourier or boundary features.

Submodule TestTypeEval SetMetricFull HDCAModified
Gate →\rightarrow Fixed 0.5/0.50.5/0.5Retrained400400 OSMmIoU (%) ↑\uparrow60.4357.37
Gate →\rightarrow Fixed 0.5/0.50.5/0.5Retrained400400 OSMHeight ρ\rho ↑\uparrow0.2400.171
No Fourier featuresIntervention1313 held-outHeight RMSE (m) ↓\downarrow2.863.30
No boundary featuresIntervention1313 held-outBF@2px (%) ↑\uparrow74.3959.37

Fixed fusion hurts mIoU (−3.06-3.06) and height ρ\rho (0.240→0.1710.240 \rightarrow 0.171), and disabling the boundary features lowers BF@2px by 15.0215.02 points. The trained model also relies on its Fourier features (height RMSE 2.86→3.30 m2.86 \rightarrow 3.30\text{ m} when disabled), although a variant retrained with a scalar height input reaches similar adherence (mIoU 59.83%59.83\%, ρ=0.261\rho = 0.261). The separate branches and the learned fusion therefore matter more than the height encoding itself.

AuthorsOpenReview ↗

Response to Reviewer ggdB (2/3)

W2 / W4 / Q1. Spatial boundary alignment, geometric height consistency, and quantitative evaluation

Diffusion conditioning provides soft guidance. We therefore evaluate label adherence using an independent reader (DINOv2-L/14 + DPT) trained exclusively on 400400 real US3D images, with real held-out RGB as the reference. The results support semantic and height adherence on the evaluated layouts.

1. Semantic boundary alignment. On 1313 held-out real layouts, our generated RGB closely matches real-image references in 6-class mIoU (70.07%70.07\% vs. 71.12%71.12\%), building BF@2px (74.39%74.39\% vs. 78.87%78.87\%), and boundary displacement (2.692.69 vs. 2.37 px2.37\text{ px}). Across 400400 shared OSM layouts, HDCA improves pooled mIoU by 33.7733.77 points and BF@2px by 34.0534.05 points over unadapted S&H ControlNets:

Images (400400 OSM layouts)mIoU (%) ↑\uparrowBldg IoU (%) ↑\uparrowBF@2px (%) ↑\uparrowBoundary disp. (m) ↓\downarrow
Real US3D test52 (reference)70.0275.2874.393.0
Blender PBR render36.5335.4945.187.2
FLUX.2 + PBR reference32.8436.7955.665.4
FLUX.2 + depth ControlNet27.5827.4535.397.9
FLUX.2 + S & H ControlNets26.6621.2332.668.5
Ours (HDCA)60.4368.1166.713.1

2. Geometric height consistency and controlled edits. We evaluate static height readout (1313 held-out tiles), height-appearance slope ratio (4848 val tiles), and controlled 0.5×→1.5×0.5\times \rightarrow 1.5\times height edits across 493493 buildings (fixing layout, camera, prompt, and noise):

Height Consistency & Control MetricReal Ref.Semantic-onlyOurs (HDCA)
Building-height readout RMSE on 1313 held-out tiles (m) ↓\downarrow2.933.342.86
Generated-to-real height slope ratio on 4848 val tiles (ref =1.0=1.0) ↑\uparrow1.000.510.92
Correct direction under 0.5×→1.5×0.5\times \rightarrow 1.5\times height edits (%) ↑\uparrow——85.8%

HDCA closely matches real-RGB height fidelity and accurately tracks directional height edits, providing more reliable 2D/3D alignment for downstream learning.

W3. Additional ControlNet and diffusion baselines

We expand our evaluation with six additional baselines:

  • Equally trained ControlNet variants (semantic-only, concatenation): Under identical backbones and training budgets, HDCA achieves the best height rank preservation and downstream height estimation (Q2).
  • Open-source multimodal (OmniGen2, BAGEL) & remote-sensing (GeoSynth) models: Our pipeline substantially outperforms all three in semantic and building boundary alignment (Q3).
  • Commercial image generation (GPT-Image-2): Our pipeline achieves higher overall mIoU and lower height error, and responds far more reliably to height edits (Q4).

Q2. Fair comparison with ControlNet conditioners under the same backbone and budget

To isolate HDCA from domain adaptation and LoRA fine-tuning, we train semantic-only and concatenation baselines under the exact same configuration. While simpler conditioners perform better on 2D semantics, HDCA improves pooled height rank correlation and downstream height RMSE.

1. Condition adherence (400400 OSM layouts). Evaluated by the real-trained independent reader, semantic-only and concatenation score higher on 2D semantics, whereas HDCA achieves the highest building-height rank correlation (ρ=0.240\rho = 0.240), best preserving input height ordering:

Conditioning6-class mIoU (%) ↑\uparrowBldg BF@2px (%) ↑\uparrowHeight rank ρ\rho ↑\uparrow
Semantic-only63.4475.970.102
Concatenation64.5672.010.186
Ours (HDCA)60.4366.710.240

2. Downstream height estimation (Building-height RMSE in meters ↓\downarrow). Training identical height predictors on each configuration yields:

Synthesis CohortTest SetSemantic-onlyConcatenationOurs (HDCA)
SynRS3D layoutsUS3D / GAMUS25.37 / 26.6117.89 / 20.468.05 / 14.43
OSM layoutsUS3D / GAMUS8.27 / 10.388.23 / 10.237.69 / 10.12

HDCA substantially reduces height RMSE on SynRS3D (−55.0%-55.0\% vs. concatenation, −68.3%-68.3\% vs. semantic-only) and Fixed OSM.

3. Why we select HDCA for our target downstream tasks. We choose HDCA because this work specifically targets urban semantic 3D reconstruction and height-sensitive urban simulations (e.g., city-scale wind-field modeling and seismic assessment of regional buildings, Sec. 4.3). We therefore prioritize height fidelity via HDCA. Meanwhile, because SynUrbanSAT is a modular pipeline, users whose downstream tasks only require 2D land-cover segmentation can readily swap in the semantic-only or concatenation module.

AuthorsOpenReview ↗

Response to Reviewer ggdB (3/3)

Q3. Comparison with unified multimodal and remote-sensing generation models

We benchmark our pipeline against unified multimodal models (OmniGen2, BAGEL) and a remote-sensing OSM-conditioned diffusion model (GeoSynth) across the 1313 held-out layouts using our real-trained independent reader. Our pipeline gives better building alignment and lower height readout error under the tested workflows.

Generator6-class mIoU (%) ↑\uparrowBldg IoU (%) ↑\uparrowBldg BF@2px (%) ↑\uparrowHeight RMSE (m) ↓\downarrow
Real RGB (reference)71.1279.1878.872.93
OmniGen216.1212.4412.026.42
BAGEL21.9927.6234.236.85
GeoSynth22.9755.6744.584.46
Ours (HDCA)70.0775.7574.392.86

General multimodal models struggle with overhead urban layouts, and even against the strongest external baseline (GeoSynth), our pipeline improves building IoU by +20.08+20.08 points, BF@2px by +29.81+29.81 points, and reduces height readout RMSE from 4.46 m4.46\text{ m} to 2.86 m2.86\text{ m}. When OmniGen2 and BAGEL additionally receive the height map through their native multi-image interfaces (1818 OSM layouts), only 45.4%45.4\% and 40.4%40.4\% of buildings respond to a 0.5×→1.5×0.5\times \rightarrow 1.5\times height edit in the requested direction, against 81.6%81.6\% for ours. These comparisons form Table 1 of the revised paper.

Q4. Comparison with zero-shot commercial image-generation models

We benchmark GPT-Image-2 against our pipeline with identical semantic and height inputs. The commercial model reproduces building outlines well, while our pipeline achieves higher overall semantic agreement, lower height error, and a far stronger response to height edits.

1. Label adherence (1313 held-out layouts, independent reader).

Generator6-class mIoU (%) ↑\uparrowBldg IoU (%) ↑\uparrowBldg BF@2px (%) ↑\uparrowHeight RMSE (m) ↓\downarrow
Real RGB (reference)71.1279.1878.872.93
GPT-Image-259.8279.8375.044.44
Ours70.0775.7574.392.86

2. Height controllability & downstream utility (1818 matched layouts). We evaluate 0.5×→1.5×0.5\times \rightarrow 1.5\times height edits on 1,1511,151 buildings, and train identical roof-height probes on each generator's images, tested on 12,34012,340 real GAMUS roofs from unseen cities:

Evaluation AspectMetricGPT-Image-2Ours (HDCA)
Height controlCorrect edit direction (%) ↑\uparrow59.781.6
Height controlRead height change per requested m (m/m) ↑\uparrow0.0210.065
DownstreamReal-roof height RMSE of the probe (m) ↓\downarrow6.944.63

The commercial model does not translate metric height maps into consistent visual cues: its height readout error is 55%55\% higher (4.444.44 vs. 2.86 m2.86\text{ m}), and only 59.7%59.7\% of edited buildings respond in the requested direction, against 81.6%81.6\% for ours (+21.9+21.9 points). Its overall mIoU is also 10.2510.25 points lower. These comparisons are reported in Table 1 and Appendix A.9 of the revised paper.

AuthorsOpenReview ↗

Follow-up to Reviewer ggdB: New Comparisons with OmniGen2, BAGEL, GPT-Image-2 & HDCA Ablations

Dear Reviewer ggdB,

Thank you very much for your valuable feedback, which prompted us to add six new baselines and comprehensive quantitative evaluations. As the discussion period nears its end, we wanted to briefly summarize how our rebuttal and revised paper address all of your concerns:

  • Novelty & HDCA Submodule Ablations (W1): We clarified our framework- and module-level contributions and ran submodule ablations (Table 5), confirming that replacing learned gated fusion with fixed 0.5/0.50.5/0.5 fusion drops mIoU by 3.063.06 points and height rank ρ\rho by 0.0690.069, while disabling boundary features lowers BF@2px by 15.0215.02 points.
  • Quantitative Spatial & Height Consistency (W2/W4/Q1): Using an independent predictor trained on real US3D images, HDCA closely matches real held-out RGB in 6-class mIoU (70.07%70.07\% vs. 71.12%71.12\%), boundary displacement (2.692.69 vs. 2.37 px2.37\text{ px}), and height RMSE (2.862.86 vs. 2.93 m2.93\text{ m}), while driving 85.8%85.8\% of buildings in the target direction under 0.5×→1.5×0.5\times \to 1.5\times height edits.
  • Equal-Budget ControlNet Baselines (W3/Q2): Under the exact same backbone, 400400 real triplets, and LoRA budget, HDCA achieves the best height rank preservation (ρ=0.240\rho = 0.240) and lowest downstream height RMSE over semantic-only and concatenation variants.
  • Unified Multimodal & Commercial Baselines (Q3/Q4): In Table 1 and Appendix A.9, our pipeline substantially outperforms OmniGen2, BAGEL, and GeoSynth (e.g., +20.08+20.08 building IoU and −1.60 m-1.60\text{ m} height RMSE over GeoSynth). Against GPT-Image-2, our method achieves +10.25+10.25 higher overall mIoU, lower height error (2.862.86 vs. 4.44 m4.44\text{ m}), and far stronger height-edit adherence (81.6%81.6\% vs. 59.7%59.7\%).

We hope these new experiments and comparisons satisfactorily resolve your concerns. Please let us know if any further clarification is needed before the window closes!

Best regards,

The Authors

Reviewer_m22aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SynUrbanSAT constructs urban scenes from OpenStreetMap and uses rendered semantic and height maps to condition satellite RGB generation. HDCA encodes both conditions and adapts a pretrained generator through ControlNet LoRA using 400 real annotated samples. The paper contributes a 20-city synthetic dataset and reports downstream improvements. Its central claim is realistic RGB with semantic and metric-height labels that remain aligned after generation.

Critical strengths
  1. The proposed pipeline has practical value. Combining configurable urban geometry with generated appearance could produce diverse training data. Separate encoders for categorical semantics and continuous heights are a reasonable design.

  2. The downstream gains are meaningful. Replacing SynRS3D RGB while retaining its layouts and labels improves height RMSE from 7.15 to 5.27 m on US3D and from 11.26 to 9.09 m on GAMUS (Table 1). The complete pipeline also improves building-volume IoU over the tested synthetic sources (Table 2).

Critical weaknesses
  1. The paper does not establish that HDCA effectively controls metric height. Encoding height and injecting its features into ControlNet does not show that the generator uses them accurately. Semantic layouts already provide strong structural cues, and there is no equally trained semantic-only comparison to establish what the height branch adds. Figure 3(b) shows changed input height maps, but not corresponding RGB outputs whose geometric cues follow those changes.

    Moreover, Appendix A.3.1 clips height inputs at 30 m while retaining the original metric labels. With the same semantic map and prompt, height maps that become identical after clipping give identical generation conditions. The height channel therefore cannot distinguish these requests. The default procedural range may avoid this limitation, but the paper does not quantify its impact on retained OSM heights and real training heights. This directly limits the claimed metric-height control.

  2. The generated RGB is not shown to preserve its semantic and geometric labels reliably. Rendered RGB and labels originate from the same 3D scene. Once diffusion replaces the RGB, that correspondence must be verified. HDCA's boundary features and conditional training are plausible mechanisms, but their adequacy is not directly evaluated. Appendix A.8 already reports grass-to-soil appearance changes when only the random seed changes, showing that semantic conditions are not always preserved.

    DINO-FID/KID assess appearance distributions; downstream scores assess models trained on the generated data. Neither directly measures agreement between a generated image and its conditioning labels. Useful training data can still contain label errors. Volume IoU also mixes footprint and height accuracy: in Table 3, it improves substantially although US3D height RMSE is 6.95 m versus 6.91 m for the PBR-reference baseline. These results cannot substitute for a direct test of label adherence.

  3. HDCA's architectural novelty appears incremental, and its benefit over simpler conditioning is not established. HDCA combines class embeddings and boundary features, Fourier height encoding, CNNs, and MLP-based gated fusion with an existing ControlNet and LoRA. These are familiar components; the task-specific design lies in how they handle categorical semantics and continuous heights together. Such a combination can be valuable, but the paper needs to show that it improves joint condition adherence. Table 3 compares adapted HDCA with unadapted alternatives, so it does not isolate the benefit of this architecture from real-domain adaptation. The current evidence therefore does not establish why HDCA is needed over a simpler, equally trained semantic-plus-height conditioner.

Questions for authors
  1. Does height conditioning change the output correctly? Please compare full HDCA with an equally trained semantic-only model and evaluate paired height changes with layout and camera fixed. Show the generated RGB response, not only the modified input maps, and report the supported height range.

  2. How accurate are the labels after generation? Please quantify class agreement, building-boundary displacement, and height adherence on a representative generated subset across random seeds. If an independent predictor is used for evaluation, report its own error on real held-out images. What fraction of supplied heights exceeds 30 m?

  3. What does HDCA add beyond ordinary dual conditioning? Please compare it with a simpler semantic-plus-height conditioner using the same backbone, 400 real training triplets, LoRA configuration, and training budget. Does HDCA improve semantic and height adherence? Ablating its boundary features and gated fusion would also help identify which design choices contribute to those gains.

6 comments
AuthorsOpenReview ↗

Response to Reviewer m22a (1/3)

We thank the reviewer for the careful reading and for recognizing the practical value of generating paired RGB and supervision for downstream training. Below we present the corresponding experiments within the proposed data-synthesis framework. The main comparisons appear in the revised paper (changes in blue): Table 1, Table 5, Fig. 11, and Appendices A.6.3, A.8, and A.9.

W1 / Q1. Metric-height control over semantic-only, paired height edits, and supported [0,30] m[0, 30]\text{ m} range

We compare HDCA against an equally trained semantic-only baseline (identical backbone, real training data, LoRA settings, and budget) and quantify the impact of the 30 m30\text{ m} clipping threshold.

1. Height cues beyond semantics & paired height edits. On 4848 validation layouts, we use a real-trained height predictor to measure the regression slope between assigned heights and heights read from generated RGB (normalized by the real-RGB slope, where 1.01.0 is ideal). On 1313 held-out layouts (493493 eligible buildings), we also test paired 0.5×→1.5×0.5\times \rightarrow 1.5\times height edits with layout, camera, prompt, and noise fixed:

MetricSemantic-onlyOurs (HDCA)
Generated-to-real height slope ratio (ref =1.0=1.0) ↑\uparrow0.510.92
Correct height increase under 0.5×→1.5×0.5\times \rightarrow 1.5\times ↑\uparrow0.0% (invariant)85.8%

HDCA closely matches the real-image height-appearance slope (0.920.92 vs. 0.510.51) and drives 85.8%85.8\% of edited buildings in the prescribed direction (whereas semantic-only receives unchanged inputs under height-only edits). The paired RGB outputs and their response maps are shown in Fig. 11 of the revised paper.

2. Supported height range ([0,30] m[0, 30]\text{ m}) & limitations. We set the 30 m30\text{ m} encoder clipping threshold based on the empirical height distributions of our training data and SynUrbanSAT-20:

Dataset / CohortFraction exceeding 30 m30\text{ m}
Real generator-training data2.60% of building pixels
SynUrbanSAT-20 (3.09M3.09\text{M} buildings)0.33% of buildings; 1.82% of building pixels
Procedural heights0.00%

Clipping makes heights above 30 m30\text{ m} indistinguishable to the height encoder while retaining their original metric labels. This affects 1.82%1.82\% of building pixels in SynUrbanSAT-20. Future work will expand high-rise training coverage and extend this range.

AuthorsOpenReview ↗

Response to Reviewer m22a (2/3)

W2 / Q2. Label adherence after generation, cross-seed stability, and >30 m fraction

We directly evaluate whether generated RGB preserves the supplied semantic and height labels using an independent predictor trained exclusively on real US3D images, using its performance on real held-out RGB as the reference.

1. Direct label adherence (1313 held-out layouts). Evaluated against the real-RGB reference, HDCA closely matches real imagery across semantic agreement, boundary alignment, and height readout:

Images6-class mIoU (%) ↑\uparrowBldg BF@2px (%) ↑\uparrowBoundary disp. (px) ↓\downarrowHeight RMSE (m) ↓\downarrow
Real RGB (reference)71.1278.872.372.93
Ours (HDCA)70.0774.392.692.86

2. Stability across random seeds (100100 OSM layouts ×3\times 3 seeds). Across 300300 generated images, aggregate mIoU varies by only 0.780.78 points. However, local semantics still vary: cross-seed class agreement is 78.92%78.92\%, and 20.27%20.27\% of condition-grass pixel comparisons switch between grass and ground. This is consistent with the local drift in Appendix A.8.

Metric (100100 OSM layouts)HDCA (mean ±\pm SD)
Six-class mIoU (%) ↑\uparrow61.04±0.4261.04 \pm 0.42
Building BF@2px (%) ↑\uparrow66.27±0.4266.27 \pm 0.42
Pooled building-height rank ρ\rho ↑\uparrow0.228±0.0130.228 \pm 0.013

3. Conditioning trade-offs. We specifically designed HDCA and evaluate 3D Volume IoU to serve our target downstream applications—such as urban wind-field CFD simulation and regional seismic assessment (Sec. 4.3)—where joint 3D building geometry and height accuracy are critical. Although semantic-only conditioning yields slightly higher 2D mIoU on the OSM layouts (63.44%63.44\% vs. 60.43%60.43\%), HDCA substantially reduces downstream building-height RMSE. Meanwhile, due to the modular nature of the SynUrbanSAT pipeline, it therefore supports selecting the conditioner according to the downstream task.

4. Fraction exceeding 30 m30\text{ m}. As detailed in W1/Q1, only 0.33%0.33\% of building records (1.82%1.82\% of building pixels) in SynUrbanSAT-20 exceed 30 m30\text{ m}.

5. Volume IoU versus height RMSE (original Table 3; revised Table 4). Volume IoU is a joint measure of footprint and height. In revised Table 4, the reported US3D building volume IoU is 37.09%37.09\% for our pipeline versus 6.87%6.87\% for the unadapted FLUX.2 (PBR reference) baseline, a difference of 30.2230.22 percentage points. Their all-pixel height RMSEs are similar (6.956.95 vs. 6.91 m6.91\text{ m}), so height RMSE alone does not explain the difference in the joint reconstruction metric. We agree that this cannot substitute for a direct test of label adherence, which is why the reader-based evaluation above is now reported in the paper (Table 1 and Appendix A.9; the original Table 3 is Table 4 in the revised paper).

AuthorsOpenReview ↗

Response to Reviewer m22a (3/3)

W3 / Q3. Comparison with equally trained conditioners and HDCA submodule ablations

We compare semantic-only, concatenation, and HDCA under the exact same backbone, 400400 real training triplets, LoRA settings, and training budget, followed by component ablations.

1. Equally trained conditioner comparison. Evaluated by our real-trained predictor across 400400 OSM layouts, 4848 validation layouts (height-appearance slope ratio), and 1313 held-out layouts (0.5×→1.5×0.5\times \rightarrow 1.5\times paired height edits across 493493 buildings):

MetricSemantic-onlyConcatenationOurs (HDCA)
6-class mIoU on 400400 OSM layouts (%) ↑\uparrow63.4464.5660.43
Building BF@2px on 400400 OSM layouts (%) ↑\uparrow75.9772.0166.71
Building-height rank ρ\rho on 400400 OSM layouts ↑\uparrow0.1020.1860.240
Generated-to-real height slope ratio (ref =1.0=1.0) ↑\uparrow0.510.820.92
Correct direction for 0.5×→1.5×0.5\times \rightarrow 1.5\times edits (%) ↑\uparrow0.0 (invariant)86.285.8

While simpler conditioners score higher on 2D semantics, HDCA best preserves relative height ordering (ρ=0.240\rho = 0.240) and most closely reproduces the real-image height-appearance slope (0.920.92).

2. HDCA submodule evaluation. We evaluate submodules via (i) a retrained ablation replacing the learned gate with fixed 0.5/0.50.5/0.5 fusion under the same training budget, and (ii) inference interventions disabling Fourier height or boundary features:

Submodule TestRetrained?Eval SetMetricFull HDCAModified
Learned gate →\rightarrow Fixed 0.5/0.50.5/0.5Yes400400 OSM6-class mIoU (%) ↑\uparrow60.4357.37
Learned gate →\rightarrow Fixed 0.5/0.50.5/0.5Yes400400 OSMHeight rank ρ\rho ↑\uparrow0.2400.171
Disable Fourier height featuresNo1313 held-outHeight RMSE (m) ↓\downarrow2.863.30
Disable boundary enhancementNo1313 held-outBldg BF@2px (%) ↑\uparrow74.3959.37

Replacing the learned gate degrades both mIoU (−3.06-3.06) and height ρ\rho (−0.069-0.069), and disabling the boundary features lowers BF@2px by 15.0215.02 points. The trained model also relies on its Fourier features (height RMSE +0.44 m+0.44\text{ m} when disabled), although a variant retrained with a scalar height input reaches similar adherence (mIoU 59.83%59.83\%, ρ=0.261\rho = 0.261). The separate branches and the learned fusion therefore matter more than the height encoding itself.

3. Role of HDCA. As detailed in W2/Q2 (Point 3), HDCA is specifically designed for height-and volume-critical 3D urban simulations, while the modular nature of SynUrbanSAT allows swapping in other conditioning.

Reviewer_1hMaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SynUrbanSAT, a pipeline for synthesizing satellite imagery with aligned semantic and height annotations. The pipeline first uses OpenStreetMap (OSM) data to procedurally construct 3D urban scenes and derive the corresponding annotations, then trains a diffusion-based generator to synthesize satellite images conditioned on semantic maps and height maps. The authors propose a Heterogeneous Dual-Condition Adapter (HDCA) to integrate these two types of input. They also introduce SynUrbanSAT-20, a dataset comprising 20,097 samples from 20 cities, with accompanying geometric annotations. Experiments evaluate the visual fidelity of the generated images and their utility for downstream tasks, including semantic segmentation, height estimation, and building-volume reconstruction.

Critical strengths

The paper addresses an important data bottleneck. SynUrbanSAT combines real-world OSM layouts with synthetic satellite imagery and corresponding semantic and height annotations, offering a scalable approach to creating training data for urban tasks that require costly 3D supervision.

Critical weaknesses

1.The paper’s main contribution is the synthesis of satellite imagery and corresponding annotations; 3D reconstruction is primarily used to evaluate the downstream utility of the synthetic data. The paper should clarify its positioning as a data-synthesis method and dataset for urban perception and reconstruction tasks. 2.The paper’s current notion of “3D reconstruction” is mainly limited to height estimation and building-volume recovery. It does not reconstruct or evaluate roof shapes, structural details, or richer urban geometry. 3.For each city in SynUrbanSAT-20, what proportions of building-height labels come from valid OSM attributes, conversion from floor counts, and procedural sampling? How do results differ when training or evaluating with labels from each source separately? 4.The realism of the synthetic height labels, and its effect on the reconstruction results, are not sufficiently validated. 5.The ablation study does not clearly establish the contribution of the core HDCA module.

Questions for authors

The questions are focused on the issues raised in the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1hMa (1/3)

We thank the reviewer for the careful reading and for recognizing the data bottleneck that SynUrbanSAT addresses. Below we answer each weakness/question in detail. The corresponding changes in the revised paper are highlighted in blue.

W1. Positioning as a data-synthesis method and dataset

We agree. The proposed SynUrbanSAT is a controllable data-synthesis pipeline and SynUrbanSAT-20 is a dataset for urban perception and reconstruction tasks, while urban semantic 3D reconstruction serves as a downstream evaluation task. We have revised the abstract, introduction, and experiments (highlighted in blue) to state this explicitly.

W2. Scope of "3D reconstruction"

Thank you for pointing this out; we have clarified the task definition and LoD1 geometric scope in Sec. 4.3. Following the single-view semantic 3D track of the 2019 IEEE GRSS Data Fusion Contest, "urban semantic 3D reconstruction" in our work refers to predicting per-pixel semantic labels and above-ground heights from a monocular satellite image to assemble a scene-level semantic 3D model, where buildings are extruded as LoD1 blocks without reconstructing or evaluating roof shapes and facade details (LoD2+). While our evaluation covers the entire urban scene across all classes (mIoU) and pixels including vegetation (height RMSE), 3D volumetric extrusion (Building Volume IoU) is intentionally restricted to buildings because extruding non-building classes such as tree canopies or elevated roads would erroneously fill the open spaces beneath them with solid volumes, compromising downstream physical applications like urban wind and seismic simulations. Since our pipeline constructs explicit 3D scenes in Blender and SynUrbanSAT-20 already provides aligned 3D meshes and point clouds, extending this framework to finer LoD2+ roof geometries is a natural direction for future work.

W3. Per-city height sources and results by source

Thank you for raising this insightful question, which inspired a dedicated analysis now included in Appendix A.6.3 (Tables 9 and 10). We audited all 3,089,0043,089,004 buildings across the 20,09720,097 layouts in SynUrbanSAT-20 and evaluated downstream transfer by height source.

1. Height assignment priority & per-city statistics. Because OSM height records are often incomplete, Stage 1 assigns building heights via a three-step priority: (1) explicit OSM height tag (46.25%46.25\% overall), (2) floor count ×3 m\times 3\text{ m} (10.37%10.37\%), or (3) procedural sampling (43.38%43.38\%). The per-city breakdown (% of buildings) is summarized below (Table 9):

City# BldgsHeightFloorSampledCity# BldgsHeightFloorSampled
Austin131,04477.20.722.1Omaha137,0350.00.899.2
Chicago459,9330.148.951.0Philadelphia540,67888.61.010.4
Columbia, SC26,4600.95.094.0Potsdam31,0582.339.258.5
Des Moines16,1940.723.076.3Tallahassee20,9340.128.072.0
Houston356,2650.31.098.7Tampa158,85387.80.411.8
Jacksonville95,6410.10.299.7Topeka10,4970.10.599.4
Lincoln66,7660.00.799.3Vaihingen12,2260.14.195.8
Little Rock9,0850.02.397.7Vienna74,3522.537.560.0
New York City823,49685.40.314.2Washington, D.C.77,5990.936.862.3
Oklahoma City26,9800.00.0100.0Wichita13,9080.00.599.5

2. Training with each source separately. We trained separate height regressors exclusively on each synthetic height source (with matched height-bin counts) and evaluated them on 12,34012,340 real GAMUS buildings across Washington, D.C., New York City, and Philadelphia (Table 10):

Training heightsReal-image height RMSE (m) ↓\downarrowMAE (m) ↓\downarrow
OSM height tag5.544.87
Floor count ×3 m\times 3\text{ m}5.464.86
Procedural sampling6.455.90

Across unseen test cities, OSM-derived heights outperform procedural sampling by ∼1 m\sim 1\text{ m} RMSE (5.46–5.54 m5.46\text{--}5.54\text{ m} vs. 6.45 m6.45\text{ m}); in separate comparisons strictly matched for both training city and height bin, explicit height tags and floor counts still reduce RMSE by 0.36 m0.36\text{ m} and 1.60 m1.60\text{ m}, respectively. We sincerely thank the reviewer for this insightful comment, which directly inspired a promising direction for our future procedural rule design: aligning procedural height priors more closely with real-world urban height distributions.

AuthorsOpenReview ↗

Response to Reviewer 1hMa (2/3)

W4. Realism of the height labels and their effect on reconstruction

Thank you for this constructive comment. We added two validation experiments in Appendix A.6.3 comparing constructed heights against airborne LiDAR and evaluating downstream reconstruction under a real-data-matched height prior. Results confirm that OSM floor records substantially improve instance-level label realism, and aligning the synthetic height distribution with real data further boosts downstream reconstruction.

1. Label realism against airborne LiDAR. Evaluated against 2022 Cook County LiDAR on 1,4711,471 floor-tagged buildings across 12 Chicago tiles (using identical reference footprints), floor-derived heights (floor ×3 m\times 3\text{ m}) achieve 3.97 m3.97\text{ m} MAE and 58.50%58.50\% Building Volume IoU, compared with 8.33 m8.33\text{ m} MAE and 53.41%53.41\% Volume IoU when masking the tags and applying the expected procedural heights of the same tiles:

Height assignmentBldgsHeight MAE (m) ↓\downarrowVolume IoU (%) ↑\uparrow
Recorded floor counts1,4713.9758.50
Procedural sampling1,4718.3353.41

2. Effect of height prior on downstream reconstruction. We regenerated the synthetic RGB imagery and retrained downstream models under identical settings on the same 400400 Jacksonville and Omaha layouts, comparing our default procedural height prior against a height prior matched to the 400400 real US3D training images. Matching the height distribution reduces downstream building-height RMSE by 2.16 m2.16\text{ m} on US3D and 0.44 m0.44\text{ m} on GAMUS, while raising GAMUS Building Volume IoU from 43.39%43.39\% to 45.95%45.95\%:

Test setHeight priorBuilding-height RMSE (m) ↓\downarrow
US3DDefault procedural heights7.80
US3DMatched to real training heights5.64
GAMUSDefault procedural heights10.14
GAMUSMatched to real training heights9.70

Discussion & future work. We acknowledge that a domain gap remains: our conditional generator is currently adapted on 400400 US3D training pairs from two cities with heights primarily within 30 m30\text{ m}, which makes generalizing to out-of-distribution building heights and unseen urban morphologies challenging. However, as the primary goal of this work is to establish the end-to-end controllable SynUrbanSAT synthesis pipeline, its modular design leaves ample room for plug-and-play improvements. Your questions in W3 and W4 provide highly valuable inspiration for our next steps—specifically, incorporating region-specific real-world height priors into Stage 1 procedural sampling and expanding generator training across broader geographic and height distributions.

AuthorsOpenReview ↗

Response to Reviewer 1hMa (3/3)

W5. Contribution of HDCA

Thank you for this comment. We have expanded the conditioning ablation in Sec. 4.4 (Table 5 and Figure 11), comparing Semantic-only, Concatenation, and HDCA within SynUrbanSAT under identical backbones, training data, and optimization settings. Results demonstrate that HDCA achieves the strongest height consistency, accurately localizes building-height edits, and substantially reduces downstream building-height RMSE.

1. Semantic & height consistency. Evaluated on 400400 OSM layouts using the real-trained evaluator, simpler adapters retain slightly higher 2D semantic agreement, whereas HDCA delivers the highest building-height rank correlation (ρ\rho) and a height response slope ratio closest to real imagery (Table 5):

Conditioning6-class mIoU (%) ↑\uparrowBF@2px (%) ↑\uparrowHeight rank ρ\rho ↑\uparrowSlope ratio ↑\uparrow
Semantic-only63.4475.970.1020.51
Concatenation64.5672.010.1860.82
HDCA (ours)60.4366.710.2400.92

Furthermore, in paired building-height editing (0.5×→1.5×0.5\times \rightarrow 1.5\times with all other conditions fixed; Figure 11), HDCA concentrates RGB appearance changes much more precisely onto the edited buildings.

2. Downstream building-height estimation. Under fixed layouts and identical downstream training settings, HDCA consistently achieves the lowest building-height RMSE (m,↓\text{m}, \downarrow) across both layout cohorts and real-world test sets (Table 5):

Layout cohortTest setSemantic-onlyConcatenationHDCA (ours)
SynRS3D layoutsUS3D25.3717.898.05
SynRS3D layoutsGAMUS26.6120.4614.43
OSM-Fixed-400US3D8.278.237.69
OSM-Fixed-400GAMUS10.3810.2310.12

3. Why we select HDCA for our target downstream tasks. This ablation reveals an inherent trade-off between 2D boundary sharpness and continuous 3D metric-height control when injecting heterogeneous conditions into ControlNet. We adopt HDCA as our default adapter because this work specifically targets urban semantic 3D reconstruction and height-sensitive downstream applications (e.g., city-scale wind-field simulation, solar access analysis, and seismic risk assessment, Sec. 4.3), where accurate metric height recovery is critical. At the same time, because the conditioning adapter functions as a replaceable module within our pipeline, users targeting purely 2D land-cover segmentation can seamlessly substitute the Semantic-only or Concatenation variant.

AuthorsOpenReview ↗

Follow-up to Reviewer 1hMa: New LiDAR Validation, Per-City Height Audits & HDCA Ablations

Dear Reviewer 1hMa,

Thank you again for your insightful review, which significantly strengthened our paper. As the discussion period nears its end, we wanted to briefly highlight how our rebuttal and revised manuscript address all five of your concerns:

  • Positioning & Scope (W1–W2): We explicitly reframed the paper around controllable data synthesis and clarified the LoD1 urban semantic 3D reconstruction scope in Sec. 4.3.
  • Per-City Height Sources & LiDAR Validation (W3–W4): We audited all 3.09M buildings across SynUrbanSAT-20 (Tables 9–10), validated floor-derived heights against airborne Cook County LiDAR (reducing MAE from 8.33 m to 3.97 m), and showed that aligning procedural priors with real height distributions reduces downstream US3D RMSE by 2.16 m.
  • HDCA Ablations (W5): In Sec. 4.4 (Table 5 & Fig. 11), we compared HDCA against Semantic-only and Concatenation baselines, showing HDCA achieves the best height consistency (ρ=0.240\rho = 0.240, slope ratio 0.920.92) and lowest downstream height RMSE across all test cohorts.

We hope these new experiments and revisions satisfactorily resolve your concerns. Please let us know if any further clarification is needed before the window closes!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General responses

We sincerely thank all three reviewers for their time and constructive feedback. Your comments really helped us sharpen the positioning of SynUrbanSAT and solidify our empirical validation. We have updated the manuscript (changes highlighted in blue), with our main additions summarized below:

  1. Clearer positioning and task scope. We revised the abstract, introduction, and experiments to clearly distinguish SynUrbanSAT as a controllable data-synthesis pipeline and SynUrbanSAT-20 as its resulting dataset. We also clarified the definition and LoD1 scope of urban semantic 3D reconstruction, which is the target task motivating our design.

  2. Direct label-adherence evaluation and broader baselines. We added evaluations using an off-the-shelf predictor trained strictly on real images to directly test semantic agreement, building boundaries, and height consistency, alongside controlled height edits. We also expanded our comparisons to include recent multimodal and closed-source models, specifically OmniGen2, BAGEL, GeoSynth, and GPT-Image-2.

  3. Matched conditioning comparisons and HDCA ablations. Under the exact same backbone, training data, and budget, we added semantic-only and concatenation baselines to evaluate both label adherence and downstream height estimation. Our new component analyses further examine learned fusion, height encoding, and boundary features, giving a much clearer picture of HDCA's height-related benefits and its trade-offs with 2D semantic accuracy.

  4. Height-label provenance and realism. We included per-city height-source statistics, validation against airborne LiDAR, and transfer experiments using each height source separately. To see how label realism actually affects reconstruction, we also regenerated synthetic images and retrained downstream models under a height prior matched to the real training data.

We provide detailed results and point-by-point responses below, and we truly appreciate the reviewers' suggestions in helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.