- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This work introduces LayGraph, a latent diffusion model for remote sensing synthesis that conditions on class-labeled bounding box layouts. Prior approaches have typically encoded instances independently, while LayGraph explicitly models object-object and object-background structure through a relational graph-based conditioning method. It also adds a retrieval-augmented component with a geometry feature bank used with cluster retrieval at inference. The proposed approach shows a reduction in problems like instances placed on incompatible backgrounds or unlikely combinations of neighboring objects, and its generated images can be used to improve downstream detectors via synthetic augmentation.
Critical strengths
The paper shows good improvements over baselines. The proposed approach consistently gets best or second best performance across many metrics on the two evaluated datasets. I especially find the synthetic augmentation results in Table 7 and 8 in Appendix D impressive, with LayGraph's generated images improving downstream detectors more than both classical augmentations and other generative models.
Critical weaknesses
-
Incomplete baselines:
(a) Recent layout-control remote sensing models are not compared against. TerraDiT- conditions on polygons or bounding boxes, and bounding boxes inputs may be rasterized for map based conditioning methods (i.e., VectorSynth, CRS-Diff, TODSynth, SatSynth). Comparing against some of these works would be more convincing.
(b) GLIGEN and LayoutDiff are strong baselines, but it is unclear if they are finetuned on DIOR or DOTA datasets. Finetuning one of these models can help better demonstrate the importance of the conditioning method.
-
Generalization to current architectures: This work uses Stable Diffusion 1.5, which uses a UNet architecture. Recently, advancements in image generation have been dominated by Diffusion Transformers (DiTs). It's unclear if the conditioning method would work similarly well on these stronger backbones. Seeing if it can generalize to more modern architectures (i.e., SD3) would strengthen the paper.
-
Limited evaluation datasets: LayGraph is evaluated on DIOR-R and DOTA-v1.0, and their training splits are smaller (~6k images with 20 classes and ~1.5k images with 15 classes, respectively) compared to what is available. The geometry features bank and co-occurence prior are built from these training sets, and it's unclear if the reported gains reflect a more generally useful conditioning mechanism or a model that has memorized the appearance and co-occurrence statistics of a small corpus. Larger and more diverse benchmarks are available, such as LAE-1M (80 classes), TerraDiT data (~1k classes), FAIR1M (37 classes), and xView (60 classes). Evaluating on at least one of these would show if the proposed approach can generalize to more categories and more varied scenes.
Questions for authors
Several parts of the method are unclear.
- How are the the spectral feature and text feature encoded in Eq. 6? (i.e., how is the color spectrum computed, do you use CLIP text encoder, etc.)
- How are the visual and spectral features computed in the scene descriptor in Eq. 4? Are the crops masked out when passed through a CLIP encoder?
- How is co-occurence defined? Two classes within the same image, or the two classes are in each others' k-nearest-neighbor?
- Can you give the equation for in Eq. 5?
- What is the in Eq. 11? The text says "a validity mask scaled by a relation warmup". What is the validity mask? What is a relation warmup?
- What values of and are used during training? (Graph relation encoder paragraph in Sec 3.3). How are they chosen?
- Is in Eq. 12 the same in Eq. 10? If so, since varies by instance and the softmax is per instance, in Eq. 10 would have no effect since it would effectively be a scalar shift that the softmax operation is invariant to.
I also have some questions about the experimental setup and evaluation, as well as a few experimental asks.
- Do you train separate models on DIOR-R and DOTA?
- Where do the layouts used for generation come from? For example, AeroGen generates its own layouts, while OF-Diff uses the training annotations.
- Lines 292-293 mention that all images are resized to 512x512. Is this interpolation or random/tiling cropping? I ask this because of the smaller dataset sizes, and because DOTA scenes of up to 4,000 px would lose small objects such as vehicles under plain resizing.
- In the ablation study (Table 5), what does not including the relation graph (RG) mean? Is that taking away while keeping ?
- Are baselines retrained on DIOR-R and DOTA? If so, what is their training protocol (i.e., LoRA, learning rate, etc.)? If not, can you retrain them on those datasets?
- Can you finetune and compare against a few of the recent models in Weakness 1a? If baselines are not trained on each dataset, can you finetune them and compare?
- Can your conditioning method be applied to a DiT-based model (i.e. SD3)? A comparison between independent instance conditioning and the graph-based conditioning on a DiT backbone would help show that gains come from the conditioning design, rather than the limitations specific to SD1.5 UNet.
- Can you evaluate LayGraph against larger datasets with more classes? Zero-shot evaluation, or finetuning on those larger datasets (or both) on one of the datasets in Weaknesses would convince me that this approach is useful in application.
7 comments
Response to the Reviewer's Follow-up Questions (Q6–Q9)
We thank the reviewer for these follow-up questions. We answer each point below and state where the manuscript records the corresponding detail.
Q6. We use and for every dataset and every reported run, and this is now stated in Section 3.2. Neither value was searched per dataset; they are fixed once and shared by the DIOR-R and DOTA runs. The two choices rest on the statistics of the prior and on the local degree used elsewhere in the pipeline.
1. follows from the shape of the prior. Over the 5,862 training images the 20-class prior has 224 nonzero ordered off-diagonal entries, with a median conditional probability of , so most cross-category pairs are nearly absent. At the threshold removes 148 of these 224 entries, but those entries carry only 11% of the off-diagonal mass: it discards pairs whose conditional probability is essentially zero while keeping every coupling that carries weight. The threshold also lands at the same scale as the neighbor budget, since a class then keeps partners on average, close to . A stricter value such as would keep only partners and discard a quarter of the off-diagonal mass, leaving the graph too sparse in exactly the dense scenes the method targets.
2. matches the degree used elsewhere. The same budget of four neighbors appears in two other parts of the pipeline: retrieval queries four spatial neighbors, and RelF1 draws its reference edges from the four nearest instances, so the graph used in training and the graph used in evaluation have the same local degree. On DIOR-R an image holds instances on average, so would connect every object to every other one and leave the entire decision to ; with at least half of the scene stays outside each object's neighborhood, and the prior only has to prune a local candidate set.
3. Same-category pairs bypass , and the soft target keeps the rest. Same-category pairs are always retained regardless of the threshold, and the soft term in still provides a graded signal for pairs that fall below it, so a rare but genuine pairing is not discarded outright. The value that appears in the RelF1 reference-edge rule and in Algorithm 2 is a threshold on the evaluation metric, not the training threshold, so the two are never interchanged.
Q7. Yes, they are the same quantity, and we thank the reviewer for catching a notation error on our side: the that appeared in Eq. 10 of the submitted version should have been , and the missing subscript was a typo rather than a different variable. The bias has always been computed per spatial position and per instance in our implementation, so no reported result is affected; only the notation was imprecise. Eq. 10 now writes throughout. The bias depends on both the spatial position and the instance index , and the softmax in Eq. 10 is taken over the instance index at each fixed . Because varies across , it is not a constant shift of the logits and does change the attention distribution. We have also added the corresponding sentence in Section 3.3: the normalization is over the instance index, and if no instance covers , all are set to zero.
Q8. Yes. We train one model per dataset with the same architecture and the same recipe; only the data-dependent parts are rebuilt, namely the geometry features bank, the co-occurrence matrix , and the subtype and scene clusters, each constructed from that dataset's training split alone. Both models are initialized from Stable Diffusion 1.5 and trained for 100 epochs with the settings in Section 4.1. We do not share a model across the two benchmarks because they differ in category vocabulary (20 vs. 15 classes) and in scene statistics, and the bank index is tied to both. The released configuration is the DIOR-R one; the DOTA run uses the identical pipeline with its own data root, bank, and prior, and we will state this in the experimental settings.
Q9. Layouts always come from the dataset annotations; they are never generated by a layout sampler. During training, the layout of each sample is the annotation of a real training image, and the same image supplies the matched bank row. During generation and evaluation, we take the ground-truth layout of each held-out test image (oriented boxes plus category labels) and synthesize one image conditioned on it. All generators receive exactly the same layouts on a given dataset (Section 4.1), so differences in the tables come from the conditioning mechanism, not from different layout sources. This is also why retrieval only queries the training bank and never reads validation or test images. Unlike AeroGen, which samples its own layouts, LayGraph needs no layout generator: the layout is the input condition, and the method is evaluated on the layouts the user actually provides.
Response to the Reviewer's Questions on Baselines, Backbones, and Data Scale (Q10–Q12)
We thank the reviewer for pressing on exactly the questions that decide whether the gains are methodological. We answer each point below.
Q10. The operation is plain bilinear interpolation; we never use random cropping, and no large scene is ever squeezed into the canvas. DIOR-R images are all and are mapped to the canvas with a single resize, a scale of , with the oriented boxes scaled by the same factors and so that the layout stays aligned with the image. DOTA images span to pixels, so each scene is first cut into patches with a sliding window, following the standard DOTA protocol, and each patch is then resized to by the same factor during training. A -pixel scene is therefore never downscaled by : the scale factor the model sees is the same on both datasets, which is also what keeps the two benchmarks comparable and preserves small objects such as vehicles at a usable size. We keep a single canvas for two reasons: the comparison with layout-conditioned baselines and every reported metric are then defined on the same frame, and the pretrained latent diffusion prior operates at this resolution. We will state the patch size and the scale factor explicitly, and correct the corresponding sentence in Section 4.1.
Q11. Yes, that is exactly the intended reading. Removing the relation graph keeps the instance-masked layout attention and removes three things: the relation tokens (Eq. 11) together with the edges they carry, the bias term inside the softmax of Eq. 10, and the edge objective with its adaptive edge weights. With RG off, each instance still attends only inside its own rasterized oriented box, but there is no neighbor-to-neighbor relational signal. The background super-node is a separate ablation factor (BG): in the "RG off, BG on" row, the scene summary still reaches the denoiser through the background path, which is why that row raises OTAcc while RelCooc and RelF1 stay low. Appendix J records this split explicitly: relation tokens and the layout-attention bias belong to RG, the scene summary belongs to BG, and the edge loss and adaptive weights belong to the default graph. Instance message passing is off by default in all rows, so RG off does not mean removing message passing.
Q12. Yes. Every layout-conditioned baseline was trained on the target dataset; no model is evaluated with weights fitted only to its original corpus, and none of them sees the validation or test splits. All methods share the training split, the layout set, the resolution, and the metric definitions on each dataset (Section 4.1). Where the authors provide a checkpoint trained on the target dataset we use it; otherwise the model is fine-tuned from its public weights with its official training code, keeping its native backbone so that the comparison isolates the conditioning interface rather than the backbone. The optimization settings follow each baseline's own published recipe, since forcing our LoRA schedule onto a model with a different denoiser would disadvantage it. For completeness we will add a per-baseline table in the appendix with initialization, optimizer, learning rate, batch size, and number of epochs, so the protocol is auditable line by line. We can also add, if the reviewer prefers, an explicitly matched-setting rerun of GLIGEN and LayoutDiffusion with our LoRA schedule, but we expect it to move those rows by a small margin rather than change their rank.
Response to the Reviewer's questions on the Graph-Relational Conditioning and the Geometry Features Bank(Q1–Q5)
We sincerely thank the reviewer for these careful and clarifying questions. We address each point below and indicate where the manuscript has been revised. All equation and section numbers refer to the revised manuscript.
Q1. The instance descriptor is ai=[vi;fi;ti] (Eq. 6), with blocks of dimension 384, 384, and 256 (1024 in total). vi is the CLIP image embedding of the oriented crop, computed with a frozen CLIP ViT-L/14. fi is a compact spectral summary of the same crop: per-channel color statistics concatenated with a low-frequency Fourier spectrum of the crop, projected into the 384-dimensional spectral block. ti is a 256-dimensional embedding of a short phrase that names the category and records the size and heading bins; the phrase is encoded with a frozen CLIP text encoder, and because the category, size bin, and heading bin are discrete, this embedding is precomputed and stored in the text block of the bank. Appendix G, paragraph "What is stored", now states this 384/384/256 decomposition and what each block contains.
Q2. The scene descriptor s is computed only on the background: all annotated boxes are zeroed out before the visual and spectral encoders are applied, so no instance content leaks into s. The encoded background is then summarized by a visual block and a spectral block into a 480-dimensional vector. This is now stated explicitly in Section 3.2 ("computed on the region outside all labeled boxes") and in Appendix G: "All boxes are masked out before encoding." Instance crops are, in contrast, encoded without masking, since their content is exactly what the instance descriptor should carry.
Q3. Co-occurrence is used at two distinct levels, and we now define them side by side in the manuscript:
1. Image-level co-occurrence is the event that two categories appear together in one image.
2. The co-occurrence prior M aggregates this event over the training set: for every training image we record the unordered category pairs present (the diagonal included), then row-normalize the counts, giving Mij=P(cj∣ci).
It is therefore neither a per-instance count nor a k-nearest-neighbor count. The kNN graph and M play separate roles: kNN defines the spatial candidate edges, and M decides which cross-category edges survive the threshold. We make this distinction explicit in the definition sentence after Eq. 9 in Section 3.2 and in the paragraph "Co-occurrence prior" in Appendix H.
Q4. The appendix paragraph "Relation-graph details" in Appendix H now states the objective explicitly:
Le=∣P∣1∑(i,j)∈P[ℓBCE(y^ij,yij)+21ℓBCE(y^ij,Mij)]
where P is the set of all ordered, valid, non-self instance pairs; yij is the binary adjacency produced by the kNN and prior-threshold rule; Mij=clip(Mcicj,0,1) is the soft target from the training prior; y^ij is the edge logit; and ℓBCE(a,b)=−blogσ(a)−(1−b)log(1−σ(a)). This is the term weighted by λe=0.05 in Eq. 5.
Q5. Mr is a validity indicator for the padded relation-token buffer: retained edge embeddings, including edges to the background super-node, are packed into a fixed-length buffer of size R=48 and padded when fewer edges exist; Mr=0 removes the padded slots before the softmax, and if the buffer contains no valid token the relation branch returns zero. The "relation warmup" is srel∈[0,1], a scalar multiplying the whole relation branch that rises linearly from 0 to 1 over the first 2000 training steps, so neighbor conditioning is introduced gradually and in parallel with the layout warmup s in Eq. 12. Eq. 11 now writes srel explicitly, and Appendix H, paragraph "Relation tokens and warmup", defines both the buffer and the schedule.