acceptodds
Under review as a conference paper at ICLR 2027

From Tokens to Canvases: Rethinking Dense Layout-to-Image Generation in Remote Sensing

Abstract

Layout-to-image generation controls image synthesis through object categories and bounding boxes, yet existing instance-centric conditioning becomes difficult to scale in dense scenes, particularly in remote-sensing imagery where numerous objects often coexist. Practical capacity limits force excess object conditions to be truncated, creating a condition–image mismatch when omitted objects remain visible in the training images, which later manifests as missing or spurious objects during inference. We revisit this representation by rasterizing all bounding boxes onto a fixed-resolution global canvas, avoiding object-wise growth in the spatial conditioning input. Surprisingly, this simple representation achieves comparable or even better layout consistency than richer edge- and shape-based conditions. Building on this finding, we propose DenseGen, a category-centric layered conditioning framework that decouples category-level semantic conditioning from instance-level spatial control. Category-Level Layout Mask Attention injects shared category semantics into corresponding spatial regions, while Instance-Aware Spatial Adaptive Layering separates overlapping boxes to reduce geometric ambiguity. Experiments on DIOR-R, DOTA-v1.0, and SODA-A demonstrate improved layout consistency and semantic accuracy in dense scenes, as well as greater utility of generated images for downstream object detection, without requiring additional structural annotations.

Primary area:
generative models
Keywords:
Layout-to-Image Generation, Controllable Image Generation, Diffusion Models, Remote Sensing, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W1P5OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DenseGen studies dense layout-to-image generation using fixed-resolution box canvases rather than one semantic token sequence per instance. Category-level masked attention supplies regional semantics, while spatial layering reduces ambiguity from overlapping boxes. Experiments on DIOR-R, DOTA-v1.0, and SODA-A, plus VisDrone, report better layout consistency and examine synthetic-data utility.

Critical strengths
  1. The representation is simple and practical for dense layouts. Its advantage is supported by a meaningful fixed-canvas baseline: DOTA mAP improves from 45.33 to 56.05 over ControlNet (box), and SODA-A from 41.09 to 49.37 (Table 1). Thus, the evidence is not limited to comparisons against truncated instance-token methods.

  2. Table 2 directly studies training-time truncation while holding the untruncated evaluation layouts fixed. Table 4 separately evaluates semantic conditioning and spatial layering. These controls make the main design argument more informative than a collection of benchmark wins.

Critical weaknesses
  1. The downstream protocol compromises the interpretation of some results. Appendix A.4 uses DOTA validation layouts to generate detector-training images and evaluates on the same real validation split. Removing 10% of objects and flipping layouts does not eliminate evaluation-label information; the reported DOTA augmentation gain therefore is not a clean held-out result. The synthetic-only protocol also evaluates detectors on real images used to train the generator, so it cannot establish generalization of the full pipeline to unseen real images. This does not by itself invalidate the independent image-generation comparisons.

  2. The main density comparison mixes representation quality with input capacity: instance methods are capped at 20 objects during training and 60 at inference, including evaluation on layouts above 80 objects. Figure 5 increases these limits for timing, but does not supply corresponding generation-quality results with matched complete conditioning. The practical resource advantage is convincing, while the stronger attribution to the representation itself needs this additional separation.

Questions for authors
  1. Can DOTA augmentation be rerun using layouts from a partition disjoint from the final detector evaluation split?

  2. Can you report generation quality for a representative instance baseline receiving all layout objects, alongside the resource-matched capped comparison?

  3. For synthetic-only trainability, can evaluation use real images unseen by both the generator and detector?

1 comment

We thank the reviewer for the careful scrutiny and positive feedback on our work, as well as for these constructive suggestions that substantially improve our evaluation protocol. Our responses to your questions are provided below.

W1P5 (Q1). We appreciate the reviewer's careful examination of our downstream evaluation protocol and identification of potential label leakage, which is crucial for a rigorous assessment of synthetic-data utility. Following this suggestion, we reran the DOTA-v1.0 downstream detection experiments using layouts disjoint from the final detector evaluation split.

Disjoint-layout construction. To avoid reusing DOTA validation annotations, we constructed a proxy layout pool exclusively from DIOR-R test annotations. We mapped corresponding categories to the DOTA taxonomy and separated vehicles into small and large categories based on oriented-box areas. After removing layouts without mapped objects, the pool contains 8,236 layouts and 113,899 oriented boxes across 11 DOTA categories. All compared generators use the same pool.

Protocol. The generators are unchanged (trained on the official DOTA training split), and each generates one image per proxy layout. ControlNet (shape), ControlNet (edge) and OF-Diff require structural conditions that boxes do not provide; these are extracted from the real DIOR-R images corresponding to the proxy layouts. DenseGen and the instance-centric methods use only boxes and category labels. All detectors are evaluated on real DOTA validation images over the 11 categories; these images and their annotations are never used for synthesis or detector training. Detector settings follow Appendix A.5.

  • Synthetic-only: detectors are trained on the 8,236 synthetic images and their conditioning layouts. The real-image reference is trained on the 8,236 real DIOR-R images from which the layouts were taken, with the same 11-category annotations.
  • Real + Synthetic: detectors are trained on the real DOTA training images (11-category annotations) plus all 8,236 synthetic images. The baseline uses the real DOTA training images only.

The revised detection results (mAP50mAP_{50}) are:

MethodSynthetic-onlyReal + Synthetic
Real-image baseline68.7383.41
AeroGen32.4583.37
CC-Diff36.7984.21
FICGen45.9383.85
OF-Diff60.8684.25
ControlNet (shape)52.6884.05
ControlNet (edge)53.7284.03
ControlNet (box)53.4183.89
DenseGen64.2384.74

DenseGen achieves the best performance among the compared generators in both settings. Under Synthetic-only, it outperforms OF-Diff by 3.37 mAP50mAP_{50} points. Under Real + Synthetic, it improves the real-only baseline by 1.33 points and exceeds OF-Diff by 0.49 points. Compared with ControlNet (box), which uses the same base box-canvas representation, DenseGen gains 10.82 and 0.85 points in the two settings, respectively.

These results confirm that DenseGen's augmentation gains persist without reusing annotations from the final evaluation split. We thank the reviewer for prompting this more rigorous evaluation. We have provided 11-class mapping statistics, vehicle assignment criteria, and the proxy-pool construction procedure in Appendix A.5 and Table 5.

Reviewer_E4YqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed DenseGen, a ControlNet-based layout-to-image model for remote sensing generation. Instead of encoding instance tokens for generation, DenseGen rasterizes oriented boxes onto a canvas. A categorical mask attention and spatial adaptive layering target densely overlapping remote sensing scenes. Improvements are shown on multiple downstream tasks as a synthetic data generation model.

Critical strengths

The paper shows good improvements over baselines. I find the augmenting real training data downstream prediction (Table 3) to be impressive.

The approach comes across as a well engineered pipeline for realistic remote sensing conditions. Dense, overlapping layouts are common in the remote sensing space, and this approach tackles that problem.

Critical weaknesses
  1. Baselines: LayoutDiffusion and GLIGEN are horizontal-box methods scored with rotated IoU, and their YOLOScores under 10 mAP on DIOR-R suggest they were run with axis-aligned boxes. Recent box and polygon conditioned RS models (TerraGen, RSGen, TerraDiT-Ω\Omega) are also not discussed.
  2. Truncation seems overstated: Figure 4 shows baselines with a miss rate of 60-70% with 10 or fewer boxes (truncation limit is 20). Table 2 moves missing rates 2-5%. DenseGen misses significantly less. Unsure what is driving that gap.
  3. Limited novelty: The approach is a well-constructed conditioning pipeline for remote sensing, but its parts are adaptations of known approaches. C-LMA is region-masked cross-attention (FreestyleNet) with box-union masks. The newer component (I-SL) shows less impact in ablations.
Questions for authors
  1. Eq. (9): where boxes of two categories overlap (a ship inside a harbor), the two attention outputs are summed with no ordering or normalization. Is this intended, and does it hurt nested classes?
  2. Which pretrained ControlNet checkpoint do you start from?
  3. How were LayoutDiffusion and GLIGEN given oriented boxes?
  4. Are the synthetic-only detectors (A.4) evaluated on the real training images the generator was trained on?
  5. When a layout exceeds the cap, which boxes are dropped? Are YOLOScore and the missing rate scored against the full annotation or the boxes the model was given?
2 comments

Dear Reviewer E4Yq,

Thank you for your time and effort in reviewing our paper. Your constructive feedback has been valuable in improving our work.

As the author–reviewer discussion period is approaching its end, we would like to kindly invite you to review our rebuttal and let us know if any of your concerns remain unaddressed.

We sincerely appreciate your time and consideration.

Best regards, The Authors

Reviewer_E4YqOpenReview ↗

Response to Authors

I thank the authors for their detailed responses. My questions have been predominantly answered, and the revisions address my main concerns. Therefore, I have raised my score to 3.

Reviewer_Z7wtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a straightforward, potentially useful solution: replace variable-length object-level conditioning with a fixed-resolution bounding-box canvas, then add category-aware semantic attention and spatial layering. The experiments are promising, particularly for dense layouts. However, the technical contributions of this paper are not significant. Encoding layouts from multiple objects as rasterized canvas doesn't sounds novel enough. Besides, the I-SL module is also not new. Therefore, I vote for weak reject, although some competing results are provided.

Critical strengths
  • Overall, the paper is clearly written and well organized.

  • The motivation is clear. The paper addresses dense remote-sensing layout-to-image generation, where scenes contain too many objects and conventional variable-length object-token representations may become inefficient or difficult to scale.

  • DenseGen uses a fixed-resolution rasterized layout canvas, combined with Instance-Aware Spatial Adaptive Layering (I-SL) and Category-Level Layout Mask Attention (C-LMA). The design is relatively straightforward and directly targets dense spatial conditioning.

  • Solid experiments and ablations. The paper evaluates on several remote-sensing datasets, including DIOR-R, DOTA-v1.0, and SODA-A, and reports image-quality, category-consistency, layout-consistency, and downstream detection metrics. Furthermore, ablations of its components and spatial-layer configurations are provided.

Critical weaknesses
  • Limited and insufficiently established novelty. The central methodological contribution appears to be a combination of rasterized layout conditioning, greedy spatial layering, and category-conditioned spatial attention. These ideas are intuitive and conceptually related to existing spatial-conditioning and layered-representation techniques. The paper does not sufficiently establish what fundamental technical advance each component introduces over prior work.
  • Rasterized canvas: Replacing variable-length object tokens with a fixed-resolution spatial representation is a plausible design choice, but the paper needs to distinguish it more clearly from existing box-map, semantic-map, and spatial-conditioning approaches.
  • I-SL: The proposed greedy assignment of overlapping boxes to a small number of layers appears relatively straightforward. The paper should explain how its algorithm differs from established overlap-resolution, layered-layout, and occlusion-aware approaches, and why the proposed assignment strategy is specifically advantageous.
  • C-LMA: Combining category-level semantic embeddings with spatial masks resembles existing forms of spatially constrained attention and layout-guided conditioning. The paper should identify the specific technical innovation beyond applying category information within spatial regions.
  • Novelty of the combination: Even if the combination is useful, the paper needs to show why integrating these components constitutes a substantial methodological advance rather than an engineering adaptation of established ideas. Direct comparisons against the closest prior methods are needed to support this claim.
  • Insufficient isolation of individual contributions. The ablations do not fully establish whether the gains come from the rasterized representation, I-SL, C-LMA, or their interaction. Additional controlled comparisons, stronger baselines, and results across multiple random seeds would make the experimental evidence more convincing.
Questions for authors

How are object boundaries, category labels, and instance identities encoded in the canvas? When multiple objects occupy the same pixels, how does the representation prevent loss of instance-level information?

Resolution sensitivity: How does the canvas resolution affect small-object preservation and spatial alignment? Have the authors evaluated whether rasterization introduces quantization errors for small or densely overlapping objects?

1 comment

Dear Reviewer Z7wt,

Thank you for your time and effort in reviewing our paper. Your constructive feedback has been valuable in improving our work.

As the author–reviewer discussion period is approaching its end, we would like to kindly invite you to review our rebuttal and let us know if any of your concerns remain unaddressed.

We sincerely appreciate your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment

W1P5 (Q3). We thank the reviewer for emphasizing the need to evaluate synthetic-only trainability on unseen real images. We therefore reevaluated Synthetic-only detectors on the official real validation splits, which were held out from both generator and detector training.

Revised protocol. For DIOR-R and SODA-A, the generators were trained on the official training splits, while synthetic detector-training images were generated from official test-set layouts; we directly evaluated the existing detectors on real validation images. For DOTA-v1.0, we used the independent DIOR-R-derived proxy layouts described in Q1, regenerated synthetic training images, and retrained all detectors. Evaluation was conducted on the official validation split, which was not used for either generator or detector training.

The revised Synthetic-only results (mAP50mAP_{50}) on unseen real validation images are:

MethodDIOR-RDOTA-v1.0SODA-A
Real-image baseline88.3468.7382.51
AeroGen73.7532.4555.49
CC-Diff54.4936.7931.52
FICGen55.5345.9352.45
OF-Diff54.9360.8663.91
ControlNet (shape)71.2852.6857.84
ControlNet (edge)70.2153.7261.66
ControlNet (box)64.0853.4164.32
DenseGen77.1364.2378.20

DenseGen achieves the highest mAP50mAP_{50} among generators on all three datasets, exceeding the strongest competing generator by 3.38 points on DIOR-R, 3.37 on DOTA-v1.0, and 13.88 on SODA-A. Despite using only synthetic images for detector training, its results approach the real-image baselines, with gaps of 11.21, 4.50, and 4.31 points, respectively.

This confirms that DenseGen's synthetic images support detection on real images unseen by both the generator and detector, rather than benefiting from overlap with the generator's training data. We appreciate the reviewer's suggestion, which strengthened our generalization evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.