acceptodds
Under review as a conference paper at ICLR 2027

LayGraph: Graph Guided Diffusion with Consistent Inference for Remote Sensing Layout-to-Image Generation

Abstract

Layout-to-image generation can synthesize remote sensing images from oriented layouts and expand training data for detectors. Existing diffusion models improve box-level placement, but encode objects largely independently and leave terrain to a global caption or unconstrained synthesis. Consequently, prescribed boxes need not yield compatible object–background pairings or plausible local arrangements. To address these failures, LayGraph first adopts a novel graph-structured condition that constrains the complex relations in layout-to-image generation, and then constructs a geometry features bank both to bind instance descriptors to this graph and to enable cluster retrieval at inference. Specifically, the graph represents each oriented layout with object nodes linked to spatial neighbors and to a background super-node, so relative geometry, category co-occurrence, and object–terrain couplings condition the denoiser rather than independent box tokens. To bind these nodes to instance descriptors and to keep the same slots when only a layout is given, the bank stores training-crop features on the graph; source-matched rows are read during training, and geometry-aware cluster retrieval fills those slots at layout-only inference. We evaluate this design on DIOR-R and DOTA against recent layout-conditioned generators. Relative to the best competing score on each metric,LayGraph reduces FID by 6.0% and raises YOLOScore by 27.3%; the same synthetic images further improve Oriented R-CNN mAP50 by 8.9% when the detector is trained without real photographs. Controlled ablations further show that the relation graph and the geometry features bank contribute complementary gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, Diffusion Model, Layout Condition, Image Generation, Object Detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eRTMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This work introduces LayGraph, a latent diffusion model for remote sensing synthesis that conditions on class-labeled bounding box layouts. Prior approaches have typically encoded instances independently, while LayGraph explicitly models object-object and object-background structure through a relational graph-based conditioning method. It also adds a retrieval-augmented component with a geometry feature bank used with cluster retrieval at inference. The proposed approach shows a reduction in problems like instances placed on incompatible backgrounds or unlikely combinations of neighboring objects, and its generated images can be used to improve downstream detectors via synthetic augmentation.

Critical strengths

The paper shows good improvements over baselines. The proposed approach consistently gets best or second best performance across many metrics on the two evaluated datasets. I especially find the synthetic augmentation results in Table 7 and 8 in Appendix D impressive, with LayGraph's generated images improving downstream detectors more than both classical augmentations and other generative models.

Critical weaknesses
  1. Incomplete baselines:

    (a) Recent layout-control remote sensing models are not compared against. TerraDiT-Ω\Omega conditions on polygons or bounding boxes, and bounding boxes inputs may be rasterized for map based conditioning methods (i.e., VectorSynth, CRS-Diff, TODSynth, SatSynth). Comparing against some of these works would be more convincing.

    (b) GLIGEN and LayoutDiff are strong baselines, but it is unclear if they are finetuned on DIOR or DOTA datasets. Finetuning one of these models can help better demonstrate the importance of the conditioning method.

  2. Generalization to current architectures: This work uses Stable Diffusion 1.5, which uses a UNet architecture. Recently, advancements in image generation have been dominated by Diffusion Transformers (DiTs). It's unclear if the conditioning method would work similarly well on these stronger backbones. Seeing if it can generalize to more modern architectures (i.e., SD3) would strengthen the paper.

  3. Limited evaluation datasets: LayGraph is evaluated on DIOR-R and DOTA-v1.0, and their training splits are smaller (~6k images with 20 classes and ~1.5k images with 15 classes, respectively) compared to what is available. The geometry features bank and co-occurence prior are built from these training sets, and it's unclear if the reported gains reflect a more generally useful conditioning mechanism or a model that has memorized the appearance and co-occurrence statistics of a small corpus. Larger and more diverse benchmarks are available, such as LAE-1M (80 classes), TerraDiT data (~1k classes), FAIR1M (37 classes), and xView (60 classes). Evaluating on at least one of these would show if the proposed approach can generalize to more categories and more varied scenes.

Questions for authors

Several parts of the method are unclear.

  1. How are the the spectral feature fif_i and text feature tit_i encoded in Eq. 6? (i.e., how is the color spectrum computed, do you use CLIP text encoder, etc.)
  2. How are the visual and spectral features computed in the scene descriptor ss in Eq. 4? Are the crops masked out when passed through a CLIP encoder?
  3. How is co-occurence defined? Two classes within the same image, or the two classes are in each others' k-nearest-neighbor?
  4. Can you give the equation for Le\mathcal{L}_e in Eq. 5?
  5. What is the MrM_r in Eq. 11? The text says "a validity mask scaled by a relation warmup". What is the validity mask? What is a relation warmup?
  6. What values of kk and τ\tau are used during training? (Graph relation encoder paragraph in Sec 3.3). How are they chosen?
  7. Is βp,i\beta_{p, i} in Eq. 12 the same βi\beta_{i} in Eq. 10? If so, since βp,i\beta_{p, i} varies by instance and the softmax is per instance, βi\beta_{i} in Eq. 10 would have no effect since it would effectively be a scalar shift that the softmax operation is invariant to.

I also have some questions about the experimental setup and evaluation, as well as a few experimental asks.

  1. Do you train separate models on DIOR-R and DOTA?
  2. Where do the layouts used for generation come from? For example, AeroGen generates its own layouts, while OF-Diff uses the training annotations.
  3. Lines 292-293 mention that all images are resized to 512x512. Is this interpolation or random/tiling cropping? I ask this because of the smaller dataset sizes, and because DOTA scenes of up to 4,000 px would lose small objects such as vehicles under plain resizing.
  4. In the ablation study (Table 5), what does not including the relation graph (RG) mean? Is that taking away OrO_r while keeping OhO_h?
  5. Are baselines retrained on DIOR-R and DOTA? If so, what is their training protocol (i.e., LoRA, learning rate, etc.)? If not, can you retrain them on those datasets?
  6. Can you finetune and compare against a few of the recent models in Weakness 1a? If baselines are not trained on each dataset, can you finetune them and compare?
  7. Can your conditioning method be applied to a DiT-based model (i.e. SD3)? A comparison between independent instance conditioning and the graph-based conditioning on a DiT backbone would help show that gains come from the conditioning design, rather than the limitations specific to SD1.5 UNet.
  8. Can you evaluate LayGraph against larger datasets with more classes? Zero-shot evaluation, or finetuning on those larger datasets (or both) on one of the datasets in Weaknesses would convince me that this approach is useful in application.
7 comments
AuthorsOpenReview ↗

Response to the Reviewer's questions on the Graph-Relational Conditioning and the Geometry Features Bank(Q1–Q5)

We sincerely thank the reviewer for these careful and clarifying questions. We address each point below and indicate where the manuscript has been revised. All equation and section numbers refer to the revised manuscript.

Q1. The instance descriptor is ai=[vi;fi;ti]a_i = [v_i; f_i; t_i] (Eq. 6), with blocks of dimension 384, 384, and 256 (1024 in total). viv_i is the CLIP image embedding of the oriented crop, computed with a frozen CLIP ViT-L/14. fif_i is a compact spectral summary of the same crop: per-channel color statistics concatenated with a low-frequency Fourier spectrum of the crop, projected into the 384-dimensional spectral block. tit_i is a 256-dimensional embedding of a short phrase that names the category and records the size and heading bins; the phrase is encoded with a frozen CLIP text encoder, and because the category, size bin, and heading bin are discrete, this embedding is precomputed and stored in the text block of the bank. Appendix G, paragraph "What is stored", now states this 384/384/256 decomposition and what each block contains.

Q2. The scene descriptor ss is computed only on the background: all annotated boxes are zeroed out before the visual and spectral encoders are applied, so no instance content leaks into ss. The encoded background is then summarized by a visual block and a spectral block into a 480-dimensional vector. This is now stated explicitly in Section 3.2 ("computed on the region outside all labeled boxes") and in Appendix G: "All boxes are masked out before encoding." Instance crops are, in contrast, encoded without masking, since their content is exactly what the instance descriptor should carry.

Q3. Co-occurrence is used at two distinct levels, and we now define them side by side in the manuscript:

1. Image-level co-occurrence is the event that two categories appear together in one image.

2. The co-occurrence prior MM aggregates this event over the training set: for every training image we record the unordered category pairs present (the diagonal included), then row-normalize the counts, giving Mij=P(cj∣ci)M_{ij} = P(c_j \mid c_i).

It is therefore neither a per-instance count nor a kk-nearest-neighbor count. The kkNN graph and MM play separate roles: kkNN defines the spatial candidate edges, and MM decides which cross-category edges survive the threshold. We make this distinction explicit in the definition sentence after Eq. 9 in Section 3.2 and in the paragraph "Co-occurrence prior" in Appendix H.

Q4. The appendix paragraph "Relation-graph details" in Appendix H now states the objective explicitly:

Le=1∣P∣∑(i,j)∈P[ℓBCE(y^ij, yij)+12 ℓBCE(y^ij, M~ij)]L_e = \frac{1}{|P|} \sum_{(i,j)\in P} \Big[ \ell_{BCE}\big(\hat y_{ij},\, y_{ij}\big) + \frac{1}{2}\, \ell_{BCE}\big(\hat y_{ij},\, \widetilde M_{ij}\big) \Big]

where PP is the set of all ordered, valid, non-self instance pairs; yijy_{ij} is the binary adjacency produced by the kkNN and prior-threshold rule; M~ij=clip(Mcicj, 0,1)\widetilde M_{ij} = \mathrm{clip}(M_{c_i c_j},\, 0, 1) is the soft target from the training prior; y^ij\hat y_{ij} is the edge logit; and ℓBCE(a,b)=−blog⁡σ(a)−(1−b)log⁡(1−σ(a))\ell_{BCE}(a, b) = -b\log\sigma(a) - (1-b)\log(1-\sigma(a)). This is the term weighted by λe=0.05\lambda_e = 0.05 in Eq. 5.

Q5. MrM_r is a validity indicator for the padded relation-token buffer: retained edge embeddings, including edges to the background super-node, are packed into a fixed-length buffer of size R=48R = 48 and padded when fewer edges exist; Mr=0M_r = 0 removes the padded slots before the softmax, and if the buffer contains no valid token the relation branch returns zero. The "relation warmup" is srel∈[0,1]s_{rel} \in [0, 1], a scalar multiplying the whole relation branch that rises linearly from 0 to 1 over the first 2000 training steps, so neighbor conditioning is introduced gradually and in parallel with the layout warmup ss in Eq. 12. Eq. 11 now writes srels_{rel} explicitly, and Appendix H, paragraph "Relation tokens and warmup", defines both the buffer and the schedule.

AuthorsOpenReview ↗

Response to the Reviewer's Follow-up Questions (Q6–Q9)

We thank the reviewer for these follow-up questions. We answer each point below and state where the manuscript records the corresponding detail.

Q6. We use k=4k=4 and τ=0.02\tau=0.02 for every dataset and every reported run, and this is now stated in Section 3.2. Neither value was searched per dataset; they are fixed once and shared by the DIOR-R and DOTA runs. The two choices rest on the statistics of the prior and on the local degree used elsewhere in the pipeline.

1. τ=0.02\tau=0.02 follows from the shape of the prior. Over the 5,862 training images the 20-class prior has 224 nonzero ordered off-diagonal entries, with a median conditional probability of 0.0080.008, so most cross-category pairs are nearly absent. At τ=0.02\tau=0.02 the threshold removes 148 of these 224 entries, but those entries carry only 11% of the off-diagonal mass: it discards pairs whose conditional probability is essentially zero while keeping every coupling that carries weight. The threshold also lands at the same scale as the neighbor budget, since a class then keeps 3.83.8 partners on average, close to k=4k=4. A stricter value such as 0.050.05 would keep only 2.22.2 partners and discard a quarter of the off-diagonal mass, leaving the graph too sparse in exactly the dense scenes the method targets.

2. k=4k=4 matches the degree used elsewhere. The same budget of four neighbors appears in two other parts of the pipeline: retrieval queries four spatial neighbors, and RelF1 draws its reference edges from the four nearest instances, so the graph used in training and the graph used in evaluation have the same local degree. On DIOR-R an image holds 8.28.2 instances on average, so k=8k=8 would connect every object to every other one and leave the entire decision to τ\tau; with k=4k=4 at least half of the scene stays outside each object's neighborhood, and the prior only has to prune a local candidate set.

3. Same-category pairs bypass τ\tau, and the soft target keeps the rest. Same-category pairs are always retained regardless of the threshold, and the soft term in LeL_e still provides a graded signal for pairs that fall below it, so a rare but genuine pairing is not discarded outright. The value 0.050.05 that appears in the RelF1 reference-edge rule and in Algorithm 2 is a threshold on the evaluation metric, not the training threshold, so the two are never interchanged.

Q7. Yes, they are the same quantity, and we thank the reviewer for catching a notation error on our side: the βi\beta_i that appeared in Eq. 10 of the submitted version should have been βp,i\beta_{p,i}, and the missing subscript was a typo rather than a different variable. The bias has always been computed per spatial position and per instance in our implementation, so no reported result is affected; only the notation was imprecise. Eq. 10 now writes βp,i\beta_{p,i} throughout. The bias depends on both the spatial position pp and the instance index ii, and the softmax in Eq. 10 is taken over the instance index at each fixed pp. Because βp,i\beta_{p,i} varies across ii, it is not a constant shift of the logits and does change the attention distribution. We have also added the corresponding sentence in Section 3.3: the normalization is over the instance index, and if no instance covers pp, all αp,i\alpha_{p,i} are set to zero.

Q8. Yes. We train one model per dataset with the same architecture and the same recipe; only the data-dependent parts are rebuilt, namely the geometry features bank, the co-occurrence matrix MM, and the subtype and scene clusters, each constructed from that dataset's training split alone. Both models are initialized from Stable Diffusion 1.5 and trained for 100 epochs with the settings in Section 4.1. We do not share a model across the two benchmarks because they differ in category vocabulary (20 vs. 15 classes) and in scene statistics, and the bank index is tied to both. The released configuration is the DIOR-R one; the DOTA run uses the identical pipeline with its own data root, bank, and prior, and we will state this in the experimental settings.

Q9. Layouts always come from the dataset annotations; they are never generated by a layout sampler. During training, the layout of each sample is the annotation of a real training image, and the same image supplies the matched bank row. During generation and evaluation, we take the ground-truth layout of each held-out test image (oriented boxes plus category labels) and synthesize one image conditioned on it. All generators receive exactly the same layouts on a given dataset (Section 4.1), so differences in the tables come from the conditioning mechanism, not from different layout sources. This is also why retrieval only queries the training bank and never reads validation or test images. Unlike AeroGen, which samples its own layouts, LayGraph needs no layout generator: the layout is the input condition, and the method is evaluated on the layouts the user actually provides.

AuthorsOpenReview ↗

Response to the Reviewer's Questions on Baselines, Backbones, and Data Scale (Q10–Q12)

We thank the reviewer for pressing on exactly the questions that decide whether the gains are methodological. We answer each point below.

Q10. The operation is plain bilinear interpolation; we never use random cropping, and no large scene is ever squeezed into the canvas. DIOR-R images are all 800×800800\times800 and are mapped to the 512×512512\times512 canvas with a single resize, a scale of 0.640.64, with the oriented boxes scaled by the same factors sx=512/Ws_x=512/W and sy=512/Hs_y=512/H so that the layout stays aligned with the image. DOTA images span 800800 to 40004000 pixels, so each scene is first cut into 800×800800\times800 patches with a sliding window, following the standard DOTA protocol, and each patch is then resized to 512×512512\times512 by the same 0.640.64 factor during training. A 40004000-pixel scene is therefore never downscaled by 7.8×7.8\times: the scale factor the model sees is the same on both datasets, which is also what keeps the two benchmarks comparable and preserves small objects such as vehicles at a usable size. We keep a single 512512 canvas for two reasons: the comparison with layout-conditioned baselines and every reported metric are then defined on the same frame, and the pretrained latent diffusion prior operates at this resolution. We will state the patch size and the scale factor explicitly, and correct the corresponding sentence in Section 4.1.

Q11. Yes, that is exactly the intended reading. Removing the relation graph keeps the instance-masked layout attention OhO_h and removes three things: the relation tokens OrO_r (Eq. 11) together with the edges they carry, the bias term βp,i\beta_{p,i} inside the softmax of Eq. 10, and the edge objective LeL_e with its adaptive edge weights. With RG off, each instance still attends only inside its own rasterized oriented box, but there is no neighbor-to-neighbor relational signal. The background super-node is a separate ablation factor (BG): in the "RG off, BG on" row, the scene summary still reaches the denoiser through the background path, which is why that row raises OTAcc while RelCooc and RelF1 stay low. Appendix J records this split explicitly: relation tokens and the layout-attention bias belong to RG, the scene summary belongs to BG, and the edge loss and adaptive weights belong to the default graph. Instance message passing is off by default in all rows, so RG off does not mean removing message passing.

Q12. Yes. Every layout-conditioned baseline was trained on the target dataset; no model is evaluated with weights fitted only to its original corpus, and none of them sees the validation or test splits. All methods share the training split, the layout set, the 512×512512\times512 resolution, and the metric definitions on each dataset (Section 4.1). Where the authors provide a checkpoint trained on the target dataset we use it; otherwise the model is fine-tuned from its public weights with its official training code, keeping its native backbone so that the comparison isolates the conditioning interface rather than the backbone. The optimization settings follow each baseline's own published recipe, since forcing our LoRA schedule onto a model with a different denoiser would disadvantage it. For completeness we will add a per-baseline table in the appendix with initialization, optimizer, learning rate, batch size, and number of epochs, so the protocol is auditable line by line. We can also add, if the reviewer prefers, an explicitly matched-setting rerun of GLIGEN and LayoutDiffusion with our LoRA schedule, but we expect it to move those rows by a small margin rather than change their rank.

Reviewer_TBT1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LayGraph for oriented layout-to-image generation in remote sensing. The method mainly consists of two components: explicit relation-graph construction and conditioning, and an offline geometry features bank. The authors evaluate the method on DIOR-R and DOTA in terms of generation quality, layout consistency, relation consistency, and downstream detection performance.

Critical strengths

1The paper explicitly models relational consistency in remote-sensing layout generation. Beyond conditioning on individual oriented boxes, the method explicitly considers both object-object and object-background relations. 2 The experimental evaluation is relatively comprehensive and demonstrates downstream utility. The paper reports generation quality, layout consistency, relation-based metrics, and synthetic-only detection performance.

Critical weaknesses

1 The necessity of explicit graph modeling is not sufficiently validated. The main gains are attributed to the relation graph, but there is no comparison with a token-based Transformer baseline. Relative position, category co-occurrence, and background information could potentially be incorporated directly as attention biases or pairwise tokens without explicitly constructing a graph. The current ablation mainly compares variants with and without the graph, which is insufficient to establish that the gains specifically come from the graph structure itself. 2 The geometry features bank may restrict generation diversity, and some details required to assess experimental fairness are missing. At inference time, instance and scene descriptors are retrieved from an offline bank constructed from the training set. This introduces a strong data prior and may bias the generated appearance space toward training samples, potentially reducing diversity. In addition, although the paper states that all methods use the same data splits, layouts, and evaluation metrics, it is unclear whether the baselines in Tables 1–3 were reimplemented and retrained under a unified setup or taken from the original papers. The amount of synthetic training data used in the synthetic-only detection experiment in Table 3 is also not clearly specified.

Questions for authors

1 Could the authors add a token-based relation baseline with a parameter count comparable to LayGraph? For example, relative relations could be modeled directly through Transformer attention without explicitly constructing a graph. 2 Does the geometry features bank cause generated samples to become overly similar to local appearances in the training set? It would be helpful to provide a quantitative analysis of generation diversity. 3 In the synthetic-only detection experiment in Table 3, how many synthetic images are generated by each method? Do all methods use exactly the same number of layouts, class distribution, prompts, and detector training settings to ensure a fair comparison? 4The main experiments focus on oriented object synthesis and detection augmentation. How do the authors position the contribution of this work to general layout-to-image generation, as opposed to a task-specific method for remote-sensing oriented-object data synthesis?

4 comments
AuthorsOpenReview ↗

Response to the Reviewer's Comment1 on Token-based Relation Baseline

Response to the Reviewer's Comment1 on Token-based Relation Baseline

We sincerely thank the reviewer for this insightful suggestion. We completely agree that modeling relative relations directly through Transformer attention (without explicit graph construction) is a highly relevant baseline.

1. Token-based nature of existing baselines. In fact, the majority of existing layout-to-image methods we compared in the main paper are inherently token-based. For instance, GLIGEN uses grounding tokens, LayoutDiffusion uses layout tokens with Object-aware Cross Attention, and AeroGen uses layout control tokens via dual cross-attention. All of these rely purely on Transformer attention to implicitly learn spatial and categorical relations, which is exactly the token-based paradigm you suggested.

2. Our dedicated Transformer (DiT) baseline. To directly address your concern, we have constructed a Diffusion Transformer (DiT) variant, denoted as LayGraph (Transformer), and added it to our supplementary material. Specifically, instead of appending raw coordinates to tokens, we migrated our core condition injection mechanism into the DiT architecture. We integrated our main innovations, such as the Graph Relational Attention, into the attention calculations at every layer of the DiT to inject the structural and relational conditions. The parameter count of this DiT variant is strictly aligned with our original UNet-based framework.

3. Comparison with recent methods (including VectorSynth). Furthermore, as suggested in previous discussions, we also added the recent VectorSynth method to our comparison. For clarity, we report the quantitative results of VectorSynth, LayGraph (Transformer), and our full model LayGraph (UNet) on the DIOR-R dataset in Table 1.

Table 1: Quantitative comparison on DIOR-R. Bold and underlined values denote the best and second-best results, respectively.

MethodFID↓KID↓CMMD↓CAS↑IoU↑Dice↑CD↓HD↓SSIM↑
VectorSynth53.460.0270.39537.190.0910.163‾\underline{0.163}5.5520.950.100
LayGraph (Transformer)23.91‾\underline{23.91}0.009‾\underline{0.009}0.30374.26‾\underline{74.26}0.097‾\underline{0.097}0.1696.56‾\underline{6.56}22.08‾\underline{22.08}0.119‾\underline{0.119}
LayGraph (UNet)22.740.0080.418‾\underline{0.418}76.830.0990.16113.9524.650.195

4. Key findings. As shown in Table 1, even with a straightforward migration of our condition injection to a pure Transformer backbone, the DiT variant significantly outperforms the token-based VectorSynth on distributional and layout metrics (FID, CAS, IoU), while achieving highly competitive shape metrics. More importantly, our full UNet-based LayGraph still achieves the best overall distributional fidelity (FID, KID) and layout alignment (CAS, IoU). This demonstrates that our proposed graph-guided condition is highly generalizable across architectures, and the explicit graph topology provides a strong structural inductive bias that cannot be fully replaced by simply relying on the global attention of a Transformer backbone.

We have updated the supplementary material with these detailed results. We hope this effectively addresses your concern.

AuthorsOpenReview ↗

Response to the Reviewer's Comment2 on Generation Diversity

Response to the Reviewer's Comment2 on Generation Diversity

We sincerely thank the reviewer for raising this critical and insightful concern. It is indeed a potential risk that a geometry features bank with cluster retrieval could collapse the generation diversity toward local training crops. We actually shared this exact concern at the very beginning of our design process.

To address this, we have already provided t-SNE visualizations of feature distributions in the main paper. These visualizations intuitively demonstrate that our generated samples span diverse clusters and do not simply collapse into the local training appearances.

To further provide rigorous quantitative evidence, we have added a dedicated diversity analysis to the latest supplementary material (as shown in Table 1 below). We compare LayGraph with existing layout-conditioned generators on DIOR-R under independent layout-conditioned inference. We extract ResNet-50 embeddings from every generated image and the paired real images, and evaluate four complementary metrics:

  • PairDist (mean pairwise cosine distance): measures the diversity of generated embeddings (higher is more diverse).
  • FeatStd (mean per-dimension standard deviation): measures the spread of the embeddings (higher is more diverse).
  • NN-Cos (mean maximum cosine similarity to a real image other than the paired ground truth): measures the degree of near-duplicate copying of other real images (lower is better).
  • PairCos (mean cosine similarity to the paired real image): measures fidelity to the target layout.

Table 1: Generation diversity and real-image proximity on DIOR-R after independent layout-conditioned inference. Bold and underlined values denote the best and second-best results, respectively.

MethodPairDist↑FeatStd↑NN-Cos↓PairCos↑
LayoutDiff0.753‾\underline{0.753}0.225‾\underline{0.225}0.615‾\underline{0.615}0.386
GLIGEN0.6860.2070.7150.457
AeroGen0.5840.1690.7270.492
CC-Diff0.5490.1910.6730.440
OF-Diff0.6050.1860.7370.470
LayGraph0.7680.2320.5980.512

(Note: PairDist and FeatStd correspond to the standard average pairwise cosine distance and feature distribution variance used for diversity evaluation, while NN-Cos and PairCos measure nearest-neighbor similarity (memorization) and pairwise fidelity to the reference image, respectively.)

Key Findings: As shown in Table 1, LayGraph leads all four scores. It achieves the highest diversity (PairDist, FeatStd) and the highest fidelity (PairCos), while having the lowest NN-Cos (indicating it does not simply copy near-duplicates from other real images).

In summary, the geometry features bank improves fidelity to the target layout's real counterpart while preserving (and slightly expanding) appearance coverage relative to box-only baselines, rather than collapsing onto near-duplicates of other training images. We have updated the supplementary material with these detailed results and analysis.

AuthorsOpenReview ↗

Response to the Reviewer's Comment3 on the Synthetic-Only Detection Setup

Response to the Reviewer's Comment3 on the Synthetic-Only Detection Setup

We sincerely thank the reviewer for raising this crucial point regarding experimental fairness. We confirm that all methods were evaluated under strictly controlled conditions regarding data volume, layouts, and detector training.

1. Number of Synthetic Images: For the DIOR-R dataset (Table 3), each method generated 5,862 synthetic images, exactly matching the official DIOR-R training split size. For the DOTA dataset, since the original images vary significantly in size (ranging from roughly 800 to 4,000 pixels), we applied a consistent sliding window cropping strategy to the training set before generation. All comparison methods generated synthetic images based on these identical cropped layouts. To ensure absolute fairness, every baseline generated the exact same number of synthetic cropped patches as LayGraph.

2. Layouts and Class Distribution: We strictly used the ground-truth oriented bounding boxes and category labels from the official training splits (and the cropped patches for DOTA) as the layout conditions. Therefore, all methods used identical layouts. Consequently, the class distribution of the generated synthetic datasets is identical across all methods and matches the real training distribution.

3. Prompts and Injection Mechanisms: The text prompts may have slight differences, but each baseline strictly adopted the original prompt settings and implementations from its respective paper to ensure they operate under their optimal conditions. For our LayGraph, we utilized a two-step injection mechanism: (1) Global text injection to guide the overall scene context, and (2) Instance text injection to bind text features to specific object nodes. We believe this evaluation reflects the true capability of each method under their native configurations.

4. Detector Training Settings: For the downstream synthetic-only detection, all generated images (at 512×512512 \times 512 resolution) were used to train the Oriented R-CNN detector from scratch. The detector training settings were kept strictly identical across all baselines and LayGraph, including the optimizer, learning rate, batch size, training epochs, and data augmentation strategies. No real photographs were used during this training phase.

By keeping the generation volume, layout conditions, and detector training pipeline identical, we ensured that the performance differences in Table 3 solely reflect the generative quality and downstream utility of the synthetic images produced by each method.

AuthorsOpenReview ↗

Response to the Reviewer's Comment4 on the General Contribution of LayGraph

Response to the Reviewer's Comment4 on the General Contribution of LayGraph

We appreciate this question, as it asks us to state precisely where the contribution of this work lies. Our position is that the two framings in the comment are not mutually exclusive: the contribution is methodological and general, while the evaluation is domain-specific. We chose remote sensing as the evaluation setting not because the method is tailored to it, but because it constitutes a harder test of the very property the method is built around — explicit and consistent layout reasoning.

The work addresses three problems that are intrinsic to layout-to-image (L2I) generation itself, none of which relies on any remote-sensing-specific prior:

  1. From independent tokens to an explicit layout graph. Most existing L2I methods (e.g., GLIGEN, LayoutDiffusion) represent a layout as a set of independent box tokens and leave the model to recover inter-object relations implicitly through attention. LayGraph instead constructs an explicit relation graph directly from the layout, in which relative geometry, category co-occurrence, and object–background coupling are modeled as first-class structure. The only assumption this requires — that a layout can be decomposed into instances together with their spatial and semantic relations — is precisely the assumption natural-image L2I makes as well: modeling a "person–chair" or "boat–water" interaction follows the same graph logic as modeling an "aircraft–runway" interaction in remote sensing.

  2. Closing the train–test gap of layout-only inference. A well-known weakness of many L2I methods is that training can draw on ground-truth image features, whereas inference receives only a layout; the missing appearance signal then degrades generation quality. We address this with a geometry feature bank and cluster-retrieval mechanism that fills each instance's appearance slot from the training distribution using layout signatures alone. Because this mechanism is conditioned only on geometric and categorical layout cues, it transfers to any L2I setting without modification.

  3. A conditioning framework independent of box format and backbone. The oriented bounding box (OBB) is only one way to instantiate the graph nodes: the same construction degrades to standard horizontal boxes (HBB) or extends to polygon layouts. Likewise, the graph-condition injection is not bound to a U-Net; in the supplementary material we integrate the same mechanism into a Diffusion Transformer (DiT) backbone and obtain comparable or better results. This indicates that the contribution lies in how a layout is represented and conditioned, rather than in any particular box type or network architecture.

To state the positioning directly: what is domain-specific about LayGraph is the data it is evaluated on; what is general is the graph-based representation of layouts and the retrieval-based handling of layout-only inference. We hope this makes the intended scope of the contribution clear.

Reviewer_cH45OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LayGraph, a graph-guided diffusion framework for remote sensing layout-to-image generation. It introduces a relation graph to model object–object and object–background interactions, along with a geometry features bank that retrieves instance descriptors during layout-only inference. Experiments on DIOR-R and DOTA demonstrate improvements in generation fidelity, layout consistency, and downstream object detection. Ablation studies further validate the contributions of the proposed components.

Critical strengths
  1. The proposed graph-based conditioning explicitly captures spatial relationships and object–background interactions, addressing important limitations of independent box-based layout conditioning in remote sensing generation.

  2. The paper evaluates generation quality, spatial consistency, and downstream detection performance on two benchmarks. The consistent improvements and component-level ablations provide convincing evidence of the framework's effectiveness.

Critical weaknesses
  1. LayGraph is developed and evaluated exclusively on a U-Net-based diffusion architecture. Its applicability to modern Diffusion Transformer (DiT) architectures remains unexplored, and comparisons with DiT-based layout-to-image generation methods are missing. This limits the generalizability and relevance of the proposed framework to recent generative architectures.

  2. The relation graph relies on dataset-derived category co-occurrence statistics and local k-nearest-neighbor connections, which may restrict its ability to model rare object combinations and complex spatial relationships. Its robustness to unseen layouts remains unclear.

Questions for authors
  1. The proposed method is only evaluated on a U-Net-based diffusion architecture. Can the authors clarify whether the graph-guided conditioning mechanism can be extended to DiT-based architectures? Comparisons with recent DiT-based generation methods would help establish its generalizability.

  2. During training, edge representations are constructed using source-matched node features, whereas inference relies on retrieved features. How does this discrepancy affect the consistency and reliability of the resulting graph representations?

2 comments
AuthorsOpenReview ↗

Response to the Reviewer's Comment1 on DiT Architectures and Comparisons

Response to the Reviewer's Comment1 on Graph Consistency

We sincerely thank the reviewer for this constructive feedback. We agree that DiT-based diffusion models are highly effective. Our initial decision to use a U-Net architecture was primarily driven by two considerations. First, the vast majority of existing layout-to-image generation methods for remote sensing object detection adopt U-Net. Second, we were concerned that employing a significantly stronger backbone like DiT might introduce a baseline so strong that it could overshadow the specific performance gains contributed by our proposed innovations (e.g., Graph Relational Attention). Therefore, we initially opted for U-Net for a fair evaluation.

However, we fully understand your concern regarding generalizability to modern architectures. We have conducted substantial additional experiments and updated our paper:

1. DiT-based Retraining: In our latest supplementary material, we retrained our framework using a DiT backbone. We integrated our main innovations, such as the Graph Relational Attention, into each diffusion layer of the DiT. Even with such a straightforward adaptation, the DiT-based variant outperforms the U-Net-based counterpart on multiple metrics (Table 1), demonstrating the effectiveness and generalizability of our method.

2. Additional Comparisons: We added comparisons with two recent works, VectorSynth and TerraDiT, in the supplementary material. Notably, to the best of our knowledge, there is currently no existing layout-to-image generation work specifically designed for object detection that utilizes a DiT backbone.

  • VectorSynth: It is the most relevant work, employing polygon-based, pixel-level dense control. Despite its direct pixel-level injection, LayGraph significantly outperforms it in distributional fidelity (FID, KID, CMMD) and layout/category fidelity (CAS, IoU, Dice). While VectorSynth achieves commendable shape metrics (CD, HD) due to dense pixel-alignment, it sacrifices overall image realism and semantic consistency.
  • TerraDiT: We conducted qualitative comparisons on DOTA under fine-tuning and retraining settings. Its quantitative metrics were excessively poor, so we only provided visual comparisons. This degradation stems from TerraDiT being not specifically designed for layout-to-image object detection. It inherently requires geospatial source information (e.g., point conditions); when forced to rely solely on the layout and global prompt, its conditional inputs degrade significantly.

Table 1: Additional DIOR-R comparison including VectorSynth and LayGraph with DiT / UNet backbones. Bold and underlined values denote the best and second-best results, respectively.

MethodFID↓KID↓CMMD↓CAS↑IoU↑Dice↑CD↓HD↓SSIM↑
LayoutDiff46.930.0240.90252.690.0570.08447.0355.820.216
GLIGEN28.290.0100.306‾\underline{0.306}47.040.0770.11337.0446.710.220‾\underline{0.220}
AeroGen39.090.0250.57364.210.0930.14226.9637.340.210
CC-Diff50.910.0290.70348.210.0810.11638.9447.940.241
OF-Diff24.180.0130.56673.530.0960.15318.2729.060.197
VectorSynth53.460.0270.39537.190.0910.163‾\underline{0.163}5.5520.950.100
LayGraph (DiT)23.91‾\underline{23.91}0.009‾\underline{0.009}0.30374.26‾\underline{74.26}0.097‾\underline{0.097}0.1696.56‾\underline{6.56}22.08‾\underline{22.08}0.119
LayGraph (UNet)22.740.0080.41876.830.0990.16113.9524.650.195

We believe these additions significantly strengthen the paper. We are truly grateful for the reviewer's insightful suggestions.

References

[1] D. Cher, B. Wei, S. Sastry, and N. Jacobs. "VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026. arXiv:2511.07744.

[2] S. Sastry, D. Cher, B. Wei, A. Dhakal, S. Khanal, D. Gupta, and N. Jacobs. "TerraDiT: Point-Conditioned Diffusion Transformer for Satellite Image Synthesis." arXiv preprint arXiv:2603.02172, 2026. (ECCV 2026).

AuthorsOpenReview ↗

Response to the Reviewer's Comment2 on Graph Consistency

Response to the Reviewer's Comment2 on Graph Consistency

We sincerely thank the reviewer for raising this insightful question. We agree that there is a discrepancy between training and inference regarding the source of the node features. However, we would like to clarify that this discrepancy does not degrade the graph structural representation; rather, it only affects the feature fidelity used for image reconstruction.

Our clarification is based on the following two aspects:

  1. The Graph Structure Remains Consistent and Intact: As defined in the paper (Eq. (7) and Eq. (9)), the graph topology (edges) and node geometric embeddings are constructed based on the input layout (i.e., oriented box coordinates and category labels) and the empirical co-occurrence matrix MM. Since the layout is explicitly provided at inference, the spatial relations dij\mathbf{d}_{ij} and category co-occurrences are identical to those during training. Therefore, the structural logic of the relation graph does not degrade during inference.

  2. The Discrepancy Only Affects Feature Fidelity: The only element that changes is the instance descriptor ai\mathbf{a}_i (Eq. (6)). During training, it is sourced from the ground-truth image, while during inference, it is filled via our geometry-aware cluster retrieval (Eq. (14)). This retrieval mechanism carefully selects features by matching the category prototype, area, aspect ratio, and heading of the box, as well as the mean co-occurrence with spatial neighbors. This ensures that the retrieved features are highly compatible with the established graph structure.

In summary, the graph's structural representation remains reliable and consistent throughout inference. The only degradation occurs in the fine-grained appearance features, which is an inherent challenge in layout-only generation and is effectively mitigated by our retrieval strategy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.