acceptodds
Under review as a conference paper at ICLR 2027

Structural Illumination under Control–Realization Mismatch for Compositional Text-to-Image Generation

Abstract

Compositional text-to-image generation is typically optimized to produce a faithful image, yet the same compositional constraint can admit many valid realizations with different spatial configurations. We study the complementary problem of : given a fixed generation budget, how can we uncover diverse realizations that satisfy the same compositional constraint? We formalize this objective as , which seeks to cover distinct structural niches using constraint-satisfying generated images. A key challenge is : inference-time search selects among candidate control configurations that specify intended structure, but stochastic generation may realize a different structure in the resulting image. To address this mismatch, we introduce Archive-Guided Structural Illumination (AGSI), which uses cheap structural proxies derived from these controls to guide search. Only constraint-satisfying generated images can expand structural coverage, and their realized structure determines the niche they occupy. On GenEval spatial relations with GLIGEN, AGSI achieves higher realized structural coverage than strong search baselines under matched generation budgets. The same principle transfers to exact counting and T2I-CompBench++ prompts, and remains effective when replacing GLIGEN with InstanceDiffusion. A matched ablation further shows that tracking discovered niches from realized outputs rather than proxy predictions improves coverage on both tasks. Notably, AGSI achieves broader coverage than the strongest baseline with a similar or lower fraction of feasible generations, indicating that its gains come from using feasible generations more effectively rather than simply producing more of them.

Primary area:
generative models
Keywords:
Compositional Text-to-Image Generation, Structural Repertoire Discovery, Structural Illumination, Control–Realization Mismatch, Quality-Diversity Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_irTzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies structural repertoire discovery for compositional text-to-image generation. It proposes Archive-Guided Structural Illumination (AGSI), a training-free search method that uses control layouts to prioritize unexplored structural niches while updating its archive only from feasible generated outputs and their measured structure. Experiments on spatial relations and exact counting show improved structural coverage across two benchmarks (GenEval, T2I-CompBench++) and two controllable generators(GLIGEN, InstanceDiffusion). Ablation studies support the benefit of using realized rather than intended structures to guide search.

Critical strengths
  1. The paper proposes a clear and useful objective on how to uncover diverse realizations that satisfy the same compositional constraint, offering users more structural choices rather than a single best image.
  2. The proposed training-free method AGSI is simple, which uses inexpensive layout proxies to select candidates and actual generated outputs to update the archive. It can be combined with existing layout-controlled generators.
  3. The paper is well-written and easy-to-follow.
Critical weaknesses
  1. The evaluation seems to be limited to a coarse, handcrafted definition of structural diversity: Diversity is measured using two handcrafted geometric descriptors and nine bins. Greater coverage does not necessarily imply more meaningful visual diversity.
  2. Modest gains over baselines, and baselines (stable diffusion v1.5) are outdated. On the main spatial-relation task (Table 1), AGSI discovers only about 0.39 additional niches out of nine compared with Proxy-Balanced after 64 generations.The performances seems to be incremental compared to the outdated baselines.
  3. Lack of efficiency analysis: Although Table 10 reports UNet evaluation counts, end-to-end runtime comparisons with the main search baselines are missing. The analysis of efficiency helps to assess practical efficiency.
Questions for authors
  1. Do the gains still hold with different bin boundaries, finer grids, or alternative structural descriptors? This would help establish whether the improvement extends beyond the chosen 3×3 archive.
  2. What is the total runtime of AGSI compared with Proxy-Balanced and BOP-Elites, including candidate selection and image evaluation? Does its advantage remain under a matched efficiency budget?
Reviewer_sA6LOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses structural repertoire discovery in compositional text-to-image generation: discovering as many distinct structural realizations of a prompt as possible under a fixed generation budget. It identifies a control–realization mismatch, where a specified layout control does not guarantee that the generated image realizes the intended structure. The proposed method, AGSI, prioritizes under-covered structural niches using control-derived proxies, but updates its archive only from the measured feasibility and realized structure of generated images. Experiments on GenEval spatial relations and counting, with transfer to T2I-CompBench++ and InstanceDiffusion, show that allocating budget by realized discoveries outperforms optimizing feasibility or balancing intended layouts alone.

Critical strengths
  1. Clear problem reframing with a relevant failure mode. The paper makes a useful distinction between the structure intended by a control and the structure actually realized by a stochastic generator. This is important for any search method that treats a controllable generator as a black-box mapping from layouts to images.
  2. A focused matched ablation and consistent empirical pattern. The Realized-Archive versus Proxy-Archive comparison isolates the proposed feedback signal more directly than a comparison against unrelated search methods. The reported gains appear across spatial relations, counting, benchmark transfer, and generator transfer, which supports the practical relevance of the principle.
Critical weaknesses
  1. The contribution is inference-time capability exploitation, not capability improvement. AGSI does not change model parameters, training data, or the generator’s representational capacity. It searches over controls and stochastic generations to find more diverse outputs that the frozen model can already produce. If a requested relation, object count, or structural configuration is outside the model’s realizable distribution, AGSI cannot generate it. Therefore the paper should not frame its result as improving compositional T2I capability in the stronger sense; its contribution is budget allocation and repertoire discovery under an existing model’s capability ceiling.
  2. The chosen tasks and examples are insufficiently novel or representative. Spatial relations and exact counting are standard compositional T2I evaluation dimensions that have already been targeted by many existing methods. Improving these benchmark scores, even with a well-designed search strategy, does not by itself establish a new research capability or address a previously unresolved problem. The paper does not show that AGSI succeeds on difficult cases that existing methods cannot solve, nor that its examples represent a meaningful new application or failure regime. This substantially weakens the paper’s informativeness and conference-level significance.
Questions for authors
  1. Can the authors evaluate AGSI with stronger text-to-image models, such as Qwen-Image or FLUX.2-dev, on challenging prompts from the Artificial Analysis Text to Image Leaderboard? For example: “In a high-tech startup office, a team of five is brainstorming around a digital whiteboard. A woman in her late 20s with a pixie cut and a smartwatch gestures towards a flowchart; a young man with a hoodie and messy hair is typing rapidly on a laptop; an older man with spectacles and a sweater vest is making notes on a tablet; a woman with long braids and a bright blazer is speaking passionately; and a fifth, a woman with a short bob and glasses, is drawing diagrams.” Demonstrating improved accuracy on such multi-person, attribute-rich prompts would substantially strengthen the paper’s practical significance and could affect the overall assessment.
  2. What are the total wall-clock time, GPU memory, and UNet/evaluator costs for AGSI and each baseline? How do the results change when methods are matched by total inference compute or latency rather than by the number of returned images?
Reviewer_ZExPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work focuses on control-realization mismatch in a text-to-image generation model, while you are also giving bounding boxes to the input. The work is mainly revolving around how to realize diverse generations for a fixed prompt, mostly focusing on positional relation between two objects. It first proposes candidates of the bounding boxes, and then proceeds with the image generation. If the sample generates the target object in a specific region, the algorithm updates the occupancy of that region and then iterates this process until the occupancy list is sufficiently diverse.

Critical strengths

Guiding the generative model to maintain diversity for a fixed prompt is an important problem, since there may be multiple plausible images corresponding to a prompt. The paper approaches this problem primarily by focusing on the positional relationships between two objects in an image. I think the problem itself is interesting but a more thorough analysis of the failure cases of existing generative models would have provided deeper insights into the underlying challenges and offered a more compelling motivation for the proposed approach.

Critical weaknesses

The main weakness of the proposed method is that the algorithm is simply exhaustively iterating over different bounding boxes (controls) and checks if the generated sample succeeds or not in expanding the diversity. So this does not include any training of models or reward streering methods that actually improves the generative model but it rather exhaustively search over different controls that satisfy the requirements.

Also, the work addresses a highly constrained version of the motivated problem, limiting spatial relationships to a 3×3 grid and considering only images containing two objects. Additionally, several terms (e.g., control, niche, archive, and repertoire) are difficult to follow and the authors should provide more concise definitions before being used in subsequent arguments.

Questions for authors

Described in the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.