acceptodds
Withdrawn from ICLR 2027

Graph2Scene: Relation-Aware Text-to-Image Generation with Graph-Guided Diffusion Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image diffusion models still struggle with complex prompts containing many elements and interactions. Existing attention-control methods mainly focus on controlling where elements appear, but often ignore how objects interact with each other. As a result, relation tokens describing interactions between elements may assign high attention to irrelevant regions, leading to missing or incorrect interactions. In this paper, we propose Graph2Scene, a training-free framework that treats scene graphs as a core control signal for diffusion generation. It represents elements as nodes and their relations as edges, and translates these element-relation dependencies into selective attention guidance. It grounds each element to its corresponding region and guides relational interactions among the involved elements. To achieve appropriate relation control without over-constraining the generation process, Graph2Scene further adapts the control duration to scene complexity and progressively relaxes graph guidance during denoising. Experiments on T2I-CompBench and ConceptMix show clear improvements in compositional generation, especially for object relations. On ConceptMix, Graph2Scene improves relationship accuracy from 0.59 to 0.84 over the strongest FLUX.1-based regional-control baseline. Results on LongBench-T2I further show its advantage on long and complex prompts. Code is available at https://anonymous.4open.science/r/GraphExecutor-CF2F.

Primary area:
generative models
Keywords:
Text-to-Image, Diffusion Models, Scene Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GMShOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

They introduce a training-free T2I method that uses a VLM to convert prompts into scene graphs and object layouts, then controls attention during denoising to improve consistency with these structures. Experiments report improvements over base FLUX and some other baselines.

Critical strengths

The idea of using object relations in the scene graph to guide attention between interacting regions is well-motivated.

The control-horizon ablation clearly shows the degradation caused by excessive control.

Critical weaknesses
  1. Given the cost of VLM-based prompt rewriting and graph construction, together with the warm-up procedure in Appendix A.5, reporting total computation and time per image is necessary but not reported.

  2. Further ablations are needed to show how much each component contributes. It is not clear that how much improvement comes from VLM-based prompt refinement, layout guidance, and relation attention control. In particular, comparisons should include FLUX with the same refined prompt, layout-only control using identical prompts and boxes, and the full method. Also, it is not clear how well the method transfers to other backbones (e.g. models with fewer sampling steps).

  3. The evaluation reports custom VLM-based scores on T2I-CompBench prompts. Results using the standard T2I-CompBench (or GenEval) evaluation protocol are not reported.

Minor notes (not affecting the score):

  • Writing: “Attend-and-Excitep” should be corrected at L133 and in Tables 1–2. Empty citations appear at L204 and L295. The method is sometimes named Graph2Scene and sometimes GraphExecutor.

  • Reproducibility: The correspondence between the manuscript and the released implementation is a bit unclear. The repo includes a complexity-based control-horizon policy and several attention modes, which makes it difficult to understand which parts correspond to the paper.

Questions for authors
  1. I could not find the name of the VLM in the manuscript, although Qwen-VL-Max appears as the default in the released code. How accurately does the planning VLM extract relations and predict layouts? How closely do the objects in the generated images follow the given boxes?

  2. Which methods in Table 1 use the same backbone? When the backbones are different, it is difficult to know how much improvement comes from the method itself.

Reviewer_vMgJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph2Scene, a training-free framework for compositional text-to-image generation with a frozen FLUX backbone. A VLM converts the prompt into a scene graph containing object descriptions, bounding boxes and relations. The graph guides attention between text tokens and image regions, and information exchange among interacting objects. A stage-wise strategy shifts control from layout towards relations and adapts the control duration. Experiments cover T2I-CompBench, an extension of ConceptMix with more concepts and an additional interaction dimension, and qualitative examples from LongBench-T2I, with additional COCO-Stuff comparisons in the appendix. The paper reports substantial improvements in relationship accuracy over FLUX-based Regional Prompting.

Critical strengths
  1. Associating relation tokens with the participating regions and routing information among interacting objects addresses a useful limitation of object-wise layout control. Figure 1 illustrates this distinction. The design reuses a pretrained generator without additional training, making it a potentially useful extension of regional generation methods.

  2. Tables 1 and 2 report relationship accuracy improvements over FLUX-based Regional Prompting from 0.73 to 0.89 on T2I-CompBench and from 0.59 to 0.84 on the extended ConceptMix setting, alongside better existence and attribute scores. These comparisons provide more relevant evidence than comparisons with older backbones. Figure 7(c) also supplies partial component evidence through the layout-only comparison.

Critical weaknesses

The central attention and scheduling rules are not specified consistently enough to identify the evaluated method. Equation (2) permits image-to-image attention only within an object region or between graph-connected regions. In the supplied anonymous code snapshot, the default graph_adaptive route in graphexecutor/attention_routing.py instead retains attention between unrelated image regions and adds typed relation biases; its unrelated-pair blocking acts on image-to-text connections. This changes the permitted information flow, beyond simply approximating negative infinity with a finite value. Similarly, Appendix A.5 determines the control duration from attention statistics collected during an unguided warm-up generation, whereas compute_control_policy uses object/relation density, bounding-box overlap and text similarity. Section 3.3 also leaves unspecified how its layout/relation weights enter Equation (3), and the code offers several execution paths. These differences may reflect different versions, but the path and configurations used for Tables 1–2 and Figure 7 are not identified. Consequently, the reported improvements cannot yet be tied to a reproducible specification of the two central contributions.

Questions for authors
  1. Which code path and configurations produced Tables 1–2 and Figure 7? Please reconcile the attention connectivity and duration estimator with the manuscript, specify how the layout/relation weights enter the attention update, and identify the tested diffusers version and processor. This would clarify which algorithm the results validate.

  2. Were Regional Prompting and the Figure 7(c) layout-only condition given the same VLM-generated text, bounding boxes and sampling settings? If Figure 7(c) already provides this matched comparison, clarifying its configuration would resolve part of the concern. Under the same layout-to-relation schedule, how does adaptive duration compare with a fixed duration selected on a validation set? This comparison would directly test the complexity-aware scheduling claim.

  3. Could the authors provide the extended ConceptMix prompts, sample counts per K, generation seeds and the evaluator model/version? Were evaluation checklists fixed per prompt and shared across methods? Per-image scores would help assess the uncertainty in the reported improvement.

Reviewer_VjuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Graph2Scene is a training-free framework that converts a prompt into a scene graph containing object descriptions, bounding boxes, and relations. It uses the graph to restrict text–image attention and communication between image regions in a frozen diffusion Transformer. An unguided warm-up estimates attention leakage and overlap, which determine the duration of a schedule transitioning from layout to relation control. Evaluation on T2I-CompBench and an interaction-enriched ConceptMix setting reports improved compositional accuracy.

Critical strengths
  1. Useful distinction between object placement and interaction correctness. The method explicitly addresses failures that remain after objects are placed in appropriate regions.
  2. Meaningful same-backbone improvements. Comparisons with FLUX.1 and Regional Prompting provide stronger evidence than comparisons across different generative backbones.
  3. The paper documents severe degradation under excessive control and motivates releasing constraints during later denoising.
Critical weaknesses
  1. The contribution of relation-aware attention is not sufficiently isolated from prompt and layout planning. The VLM rewrites descriptions and predicts bounding boxes, potentially improving generation independently of graph attention. Baselines should receive the same rewritten prompts, boxes, and graph-derived descriptions. A factorial ablation separating relation-token cross-attention, graph-connected image self-attention, and adaptive scheduling is needed.

  2. The central evaluation is insufficiently specified and departs from standard benchmark protocols. ConceptMix is extended to 8–20 concepts with an additional interaction dimension, and T2I-CompBench is evaluated using custom VLM-based accuracies.

  3. The training-free formulation has limited demonstrated practical usability. Avoiding training does not eliminate the difficulty of obtaining stable results. The reported sensitivity analysis shows substantial dependence on control duration and layout–relation weighting, with excessive intervention causing severe artifacts or complete generation collapse.

  4. End-to-end runtime and memory should include graph construction, the warm-up generation, and attention-mask overhead.

Questions for authors

Please refer to weakness 1 2 and 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.