acceptodds
Under review as a conference paper at ICLR 2027

PlanYourPixel: Structured Visual Planning for Fine-Grained T2I generation Control and Failure Localization

Abstract

Recent advancements in text-to-image (T2I) generation have improved visual quality, yet fine-grained control remains a fundamental challenge. Current layout-conditioned models struggle with numerical reasoning, attribute binding, and spatial grounding. These failures stem from the inherent ambiguity of natural language, where abstract prompts result in underspecified layouts. Even with LLM-based planning, the absence of an underlying scene logic causes models to assign coordinates before resolving the semantic relationships between objects, leading to floating items, incorrect spatial positioning, and attribute leakage. To bridge this gap, we introduce PlanYourPixel (PYP), an agent-based framework that utilizes specialized roles to decompose natural text into a structured scene representation. By resolving semantic constraints into a verifiable plan, PYP provides a unified foundation for two applications: layout-guided generation for Image synthesis and automated atomic question generation for fine-grained failure localization. We evaluated our paradigm on PYP-Bench, a diagnostic suite of 1,800 prompts and corresponding layouts. Our results demonstrated that PYP-guided image generation consistently outperforms standard layout baselines, while PYP generated 12,503 questions and formed a diagnostic framework that uncovers critical failure patterns that aggregate metrics typically conceal.

Primary area:
generative models
Keywords:
Visual Planning, Text to Image Generation, Layout Guided Generation, Compositional Reasoning, Failure Localization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r7TwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose PlanYourPixel (PYP), a training-free pipeline that uses sequential LLM agents to decompose a prompt into a structured scene representation S(T) = {N,NP,R}. They first use a prompt parse that extracts object nodes and enrich them with LLM's prior knowledge, then PYP considers the relation of the nodes, it converts each relation into a directed edge with explicit geometric constraints like dominant axis, ordering , size etc and finally it uses its layout generator to generate then check the bounding boxes on a 512 canvas grid to place the objects.

PYP proposes two applications:1. Layout-guided generation, and 2. failure localization. The layout guided generation just applied to the generated layout using PYP's scene representation to the layout2image models like GLIGEN, MIGC etc. The failure localization uses the scene representation to generate atomic visual questions and use VLM judge to answer to identify the fine-grained errors in the results.

Based on the scene representation and the failure localization application, the authors propose PYP-Bench to perform systematic evaluation for image generation with fine-grained failure localization which are ignored by the current evaluation benchmarks.

Quantitatively, PYP can generate more accurate layouts and boost the performance of the Layout2image generators.

Critical strengths
  1. The authors motivation is well defined and useful, that the current evaluation benchmarks are not aware of the fine-grained failures such as inter-object geometry (contact, relative size).

  2. The design of the scene representation is universal, it fits in current generation pipelines as well as the evaluation pipeline.

  3. Improvement in downstream generation on Layout2Image models is significant.

Critical weaknesses
  1. I believe the authors should also include the layout generation pipeline in previous agentic image generation like GenArtist, CompAgent, LayerCraft or even the CoT reasoning pipeline from close sourced models like GPT image 2 and Nanobanana.

  2. I think the overall framework is still only using the LLM's text reasoning ability, there must be a gap between the real image level reasoning, defining a more structured representation is still somehow incremental.

Questions for authors

I believe the comparison between PYP layout and LaySyn layout for downstream task is not fair. in the Figure 4. PYP Layout has label "a red cow", but the others is just "a cow" without any other descriptions. I want to know how this differences changes the downstream performance. Could the authors elaborate more on this?

What if the LLM itself is not able to generate correct relations or it cannot refine the output from its own hallucinations? I hope authors can provide some LLM failure cases in the revised version.

I will raise the score if the authors solve my questions.

Reviewer_2HzcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PlanYourPixel (PYP), a multi-agent framework that parses text prompts into structured scene representations with explicit geometric constraints. This unified representation powers two applications: layout-guided T2I generation and PYP-VQA for fine-grained failure localization. The authors also introduce PYP-Bench, comprising 1,800 prompts and 12,503 diagnostic questions across seven error categories. Experiments on NSR-1K and T2I-CompBench validate that PYP achieves superior layout accuracy, enhances prompt adherence across multiple L2I backbones, and aligns closely with human judgment in diagnosing generation failures.

Critical strengths

(1) Unified Generation and Diagnosis via Explicit Constraints. PYP resolves spatial ambiguity before coordinate assignment by inferring pairwise geometric constraints (e.g., contact mode, relative scale). Sharing this structured representation for both layout synthesis and VQA-based error localization is highly novel and practical. It transforms abstract prompt adherence into verifiable sub-tasks, which strongly supports acceptance. (2) Current benchmarks only provide coarse, skill-level scores. PYP-VQA goes further by diagnosing how a model fails (e.g., separating attribute swap from attribute leakage). Its strong correlation with human judgment across all categories makes it a consequential diagnostic tool for the T2I community.

Critical weaknesses

In Table 2, PYP uses a powerful 31B model (Gemma 4), while training-free baselines use much smaller models (e.g., Llama-3.1-8B, GPT-4o-mini). It is unclear if the gains stem from the multi-agent design or the stronger LLM backbone. Furthermore, there are no ablation studies isolating the impact of the Prompt Parser versus the Relation Extractor. This compromises the validity of the experimental conclusions.

Questions for authors

How does PYP perform on NSR-1K when using the exact same LLM backbones (e.g., Llama-3.1-8B or GPT-4o-mini) as the LayoutGPT baselines?

Reviewer_c1rPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PlanYourPixel (PYP), a training-free LLM-agent framework that converts text prompts into structured scene representations containing objects, attributes, and pairwise geometric constraints. This representation supports both bounding-box layout generation and seven categories of binary VQA questions for failure diagnosis. Using 1,800 prompts from T2I-CompBench, the authors construct PYP-Bench with corresponding layouts and 12,503 diagnostic questions, and report improved layout accuracy, downstream prompt adherence, and correlation with human judgments over selected baselines.

Critical strengths
  • PYP introduces an interpretable scene representation that makes semantic and geometric constraints explicit before layout generation.
  • The method achieves substantial spatial improvements over LaYouSyn across multiple L2I backbones, while its diagnostic taxonomy provides more actionable feedback than aggregate alignment scores.
Critical weaknesses
  • The paper does not adequately compare PYP with relevant prior work. Although the authors claim that PYP achieves the highest numerical and spatial accuracy on NSR-1K, SKE-Layout (CVPR 2025) reports 97.24% and 95.05%, compared with PYP's 95.96% and 93.29% on the same benchmark. Without including SKE-Layout, the reported results do not support this claim. PYP-VQA also appears similar to existing evaluation methods such as TIFA, DSG, L-VQAScore, and FineGRAIN, but these methods are not discussed, leaving the novelty of PYP-VQA unclear.

  • The accuracy of PYP-VQA's failure localization is not sufficiently validated. Some evaluation targets rely on LLM-inferred constraints rather than explicit prompt requirements. For example, Figure 8 treats physical support of the plate by the fish as an expected CC relation, although the prompt does not explicitly require it. This may lead to incorrect failure labels based on inferred constraints, a concern that Table 4 does not address. The table only reports the correlation between overall PYP-VQA scores and human ratings, without validating the individual OP, AA, AL, AS, RA, RS, and CC labels. Per-category human annotations are needed to support the paper's failure-localization claim.

  • The experiments do not adequately isolate the contribution of the proposed planning mechanism. PYP uses Gemma 4, while the training-free baselines use different LLMs, with no same-model comparison against direct layout generation or ablations of geometric enrichment and constraint validation. This makes it unclear whether the gains stem from the proposed mechanism or differences in the LLM and prompting setup. The downstream comparison is also limited to LaYouSyn, raising concerns about generalizability. In addition, inconsistent 31B/27B model descriptions and missing full in-context prompts make the experimental setup difficult to reproduce.

Questions for authors
  • How do the authors reconcile the claimed highest accuracy on NSR-1K with SKE-Layout, which reports higher numerical and spatial accuracy on the same benchmark?
  • How do the authors distinguish requirements in the original prompt from assumptions introduced during planning?
  • Can the authors provide a same-model comparison with direct layout generation and ablations of geometric enrichment and constraint validation?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Concerns and Weaknesses

[W1]

See SKE-Layout discussion in Q1. For layout generation, we have expanded our comparisons to include GenArtist and LayerCraft. We also investigated CompAgent, but its official GitHub repository contains only a placeholder README without publicly available implementation code. See updated results in the revised paper Table 2.

For evaluation methods, our PYP is different in the following aspects:

  1. Fine-grained failure modes. PYP-VQA reports more precise failure modes (e.g., object presence, attribute swap) and more fine-grained categories related to attributes compared to existing work such as TIFA, DSG, L-VQAScore, and FineGRAIN. For example, in Figure 5, PYP can report three different types of attribute errors, while other tools just say “attribute failures”. As a result, model designers can understand detailed model weaknesses.

  2. Shared scene representation. PYP-VQA uses the same scene representation for planning and evaluation stages. We therefore can locate precise failures during planning and/or in existing layouts, which is not done in prior work like TIFA, DSG, L-VQAScore, and FineGRAIN.

[W2]

Part 1: Yes, good point. When we calculate the overall score to judge whether a generation fails, we focus on relational constraints directly from the prompt. We limit the influence of the relative size (RS) and physical contact (CC) which sometimes can come from enriched contents and sometimes directly from prompt. As shown in Table 4, our PYP outperformed the baselines regarding evaluation.

Part 2: In addition to overall human ratings provided by the ground truth given in Table 4, we have done an additional manual evaluation on 300 images from six T2I-CompBench categories, covering 2,197 questions across seven diagnostic types. Two independent reviewers have done the annotations and then a third reviewer evaluated cases where there is a disagreement. The final manual label goes with the majority voting among the three reviewers.

Category#of Questions NAccuracy(%) ↑
Object Presence (OP)65891.49
Attribute Adherence (AA)44487.61
Attribute Swap (AS)36095.28
Attribute Leakage(AL)23195.24
Relation Adherence(RA)19483.51
Relative Size(RS)19679.59
Contact Consistency11478.07
Summary219789.26

[W3]

We added experiments on the same GPT-4o-mini backbone, including two new baselines GenArtist (NeurIPS 2024) and LayerCraft (NeurIPS 2025). We also updated our results in the revised paper Table 2.

BackboneMethodPrecision ↑Recall ↑Accuracy ↑Accuracy ↑
GPT-4o-miniLLMGroundedDiffusion84.3695.9489.9472.46
LLm Blueprint87.2167.2938.3673.52
LayoutGPT73.8286.8477.5192.01
GenArtist87.8697.2891.2275.30
LayerCraft76.7386.5780.4076.78
PYP82.0998.8396.1993.57

Downstream baseline selection. We selected LaYouSyn because its layout accuracy is closest to PYP among the compared baselines (Table 2). We tested both methods across four image-generation backbones to examine whether similar layout accuracy leads to similar prompt adherence. PYP generally performs better, particularly on spatial relationships (Table 3).

We address the planning-component ablation in our response to Q3 (Appendix A1.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.