acceptodds
Withdrawn from ICLR 2027

Build the Scaffold, Then Generate: Explicit Spatial Planning for Spatially Faithful Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image models produce high-fidelity images but remain unreliable in realizing complex spatial compositions. This limitation is more pronounced in open-source models, with a notable gap to proprietary models. We propose ScaffoldGen, an agent-driven framework that separates spatial planning from visual synthesis. Given a prompt, the planning agent parses constraints from the prompt and constructs a global layout plan in HTML, which is rendered into a visual scaffold. The generator is guided by the visual scaffold to synthesize detailed visual appearance. For systematic evaluation, we introduce SLAB, a Spatial LAyout Benchmark covering ten task categories involving complex spatial composition and layout organization. The benchmark assesses semantic and spatial constraint satisfaction through dependency-aware question-answer accuracy (QAA). On SLAB, ScaffoldGen outperforms all evaluated open-source methods and achieves competitive performance with certain proprietary models. Further evaluation on SpatialGenEval also shows consistent improvements over the baseline, confirming that the enhanced spatial awareness generalizes beyond SLAB. Together, the results demonstrate the effectiveness of ScaffoldGen and point to a broader research direction: leveraging the spatial reasoning capabilities of LLM agents to compensate for the limited spatial perception of generation models.

Primary area:
generative models
Keywords:
Image Generation, Spatial Layout Generation, Controllable Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5QZ9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper uses LLM to create an HTML layout, renders it as a visual scaffold, and conditions image generation on that scaffold. The paper also introduces SLAB, a benchmark for complex spatial compositions. Results show substantial gains on SLAB and SpatialGenEval.

Critical strengths

None

Critical weaknesses
  1. It seems that the paper was written by AI without adequate human checking. The figures, particularly Figure 3, also appear to have been generated by AI. Overall, it reads like the output of an automated research pipeline.

  2. The evaluation has only been validated on 291 questions from 20 images produced by one model. Under dependency scoring, a mistaken prerequisite judgment can invalidate several other answers. The reported 90.03% agreement on individual questions therefore does not establish the reliability of the final scores or model rankings. This matters because SLAB provides the main evidence for the paper.

  3. The comparison with existing layout methods is difficult to interpret. PlanGen, Ranni, and RPG all score below 7%, with a dependency penalty of exactly zero. The paper provides no examples or diagnostics explaining these results.

Questions for authors
  1. In the human validation subset, how closely do the final scores agree after applying prerequisite gating? How were the dependency graphs checked?

  2. Why do PlanGen, Ranni, and RPG all have zero dependency penalties?

Reviewer_ubVMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ScaffoldGen, a two-stage framework for spatially faithful text-to-image generation. An LLM first produces an HTML-based spatial plan, which is rendered into a visual scaffold containing simple geometric shapes and semantic labels. A fine-tuned image generator then uses the scaffold and original prompt to synthesize the final image. The authors construct ScaffoldGen-10K for training and introduce SLAB, a benchmark covering ten spatial task categories with dependency-aware evaluation. Experiments demonstrate substantial improvements on SLAB and smaller gains on SpatialGenEval, with ablations supporting the effectiveness of visual scaffold conditioning and generator adaptation.

Critical strengths
  1. The framework is intuitive, with an interpretable visual scaffold connecting spatial planning and image generation.
  2. The method shows substantial improvements on SLAB, and the ablations support the benefits of both visual guidance and generator adaptation.
  3. SLAB covers diverse spatial tasks and uses dependency-aware evaluation to assess structural consistency.
Critical weaknesses
  1. The distinction from closely related work needs clarification. CoCo[1] and GenClaw[2] also explore code-generated visual intermediates for image synthesis. Discussing these methods would help clarify the specific contributions of the proposed HTML scaffolds and generator adaptation.

  2. Scaffold granularity and its effects are insufficiently explored. The paper does not clearly define the scaffold’s level of detail or evaluate sensitivity to it. It also remains unclear how reliably the generator preserves intended spatial constraints while discarding incidental primitive geometry and appearance.

  3. Inference overhead is not evaluated. The framework introduces LLM planning and HTML rendering before image generation, but provides no end-to-end latency or inference-cost analysis. This makes it difficult to assess the practical tradeoff between spatial accuracy and computational overhead.

[1] CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation, ECCV 2026

[2] GenClaw: Code-Driven Agentic Image Generation, arxiv 05/2026.

Questions for authors
  1. How does your approach differ technically from CoCo and GenClaw, beyond the choice of intermediate representation?

  2. How is scaffold granularity determined? Have you tested whether changing its detail or visual style affects performance or causes unintended copying of primitive shapes?

  3. What are the end-to-end inference latency and cost, including planning, rendering, and image generation, compared with the base generator?

  4. Section 2.1 discusses benchmarks for compositional and spatial image generation. How does SLAB relate to LayoutSAM[1] and OverLayBench[2], particularly in its coverage of global composition and spatial structure?

[1] CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, ICCV 2025.

[2] OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps, NeurIPS 2025.

Reviewer_MtvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SCAFFOLDGEN, an agent-driven framework that decouples spatial planning from image synthesis. An LLM agent parses prompts into HTML layouts, which are rendered into visual scaffolds to guide a diffusion generator via token concatenation. The authors curate a 10K training dataset and introduce SLAB, a 10-category spatial benchmark utilizing a novel dependency-aware question-answer accuracy (QAA) metric. Validated on SLAB and SpatialGenEval, SCAFFOLDGEN significantly outperforms open-source baselines (boosting QAA from 60.9% to 80.8% on SLAB) and rivals leading proprietary models.

Critical strengths

Elegant HTML-to-Scaffold Bridging. Using rendered HTML as a visual condition is a clever, scalable design. LLMs excel at generating structured HTML code, while Diffusion Transformers naturally process pixel-aligned visual latents. This avoids the ambiguity of text-only layout tokens and provides far richer structural guidance than simple bounding boxes. This strongly supports acceptance due to its practical and generalizable architecture.

Critical weaknesses

1、The 10K training dataset is synthesized using the exact same 10 task categories as the SLAB benchmark. Furthermore, Table 4 shows prompt-only SFT already yields nearly half the total gain (+9.32%). Meanwhile, on the out-of-distribution SpatialGenEval benchmark, the improvement over the baseline is marginal (+1.9%, from 61.9% to 63.8%). This suggests the massive gains on SLAB heavily stem from in-domain distribution matching rather than true compositional generalization. 2、The Visual Scaffold contains stark geometric shapes and rendered text labels, fed directly as clean latent tokens. While the authors claim the model only inherits structure, they provide no quantitative metrics (e.g., FID, CLIP-IQA) to prove that rigid box boundaries or text artifacts do not bleed into the final image and degrade visual naturalness.

Questions for authors

Since the rendered visual scaffold includes explicit text labels and sharp geometric borders, how does the generator prevent text-bleeding artifacts or rigid, unnatural boundaries in free-form natural scenes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.