acceptodds
Under review as a conference paper at ICLR 2027

Dependency-Aware Discrete Diffusion for Scene Graph Generation

Abstract

Scene graphs (SGs) represent objects and their relationships as structured graphs, enabling applications in image generation, robotics, and 3D understanding. Recent work suggests that conditioning image generation on scene graphs improves compositional fidelity compared to text-only prompting. However, since users typically provide text rather than structured graphs, a key challenge is to generate scene graphs from natural language. Prior work on discrete diffusion has demonstrated success in generating generic graphs such as molecules and encoding dependencies indirectly via asynchronous generation schedules, but does not account for the hierarchical structure and strong dependencies between objects, edges, and relations in scene graphs directly in the graph state. We address this limitation by introducing a dependency-aware, hierarchically constrained discrete diffusion model for scene graph generation. Our approach decouples structure and semantics across the forward and reverse processes, enabling the model to capture conditional dependencies. At inference time, we perform training-free conditioning to sample text-aligned scene graphs. We evaluate our method on standard SG benchmarks and demonstrate improvements over both continuous and discrete graph generation baselines across graph and layout metrics. When fed to downstream image generation, our approach yields improved compositional alignment compared to text-to-image models, particularly in multi-object scenarios.

Primary area:
generative models
Keywords:
Scene Graph generation, Hierarchical discrete diffusion, T2I image generation with structured prior

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_QcuWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiScGraph, a discrete diffusion framework for scene graph generation. The method factorizes a scene graph into objects, edge existence, and relation semantics, with the constraint (edge existence to relation). Based on this representation, the authors introduce an edge-aware forward corruption process and a dependency-aware reverse sampler following Node(V)-Edge(E)-Relation(R). Text-conditioned scene graphs are obtained at inference time via CLIP-based reward tilting and Sequential Monte Carlo sampling. The method is evaluated on Visual Genome, COCO, and LAION-SG, with experiments covering unconditional graph generation, graph completion, layout prediction, and downstream text-to-image generation.

Critical strengths
  1. Scene-graph-specific discrete diffusion formulation. The paper goes beyond directly applying generic graph diffusion by explicitly separating edge existence from relation semantics and incorporating this hierarchy into the diffusion formulation. This is well motivated for sparse scene graphs with long-tailed relation vocabularies, and the ablations indicate that both the factorized state and dependency-aware sampler improve semantic consistency.
  2. Strong empirical validation. The evaluation is broad and includes both graph-level and downstream image-generation tasks. In particular, under the same SDXL-SG backbone, DiScGraph-generated scene graphs achieve performance close to ground-truth scene graphs and substantially outperform DiGress and DiffuseSG, providing evidence that the learned structural prior is useful beyond graph-distribution metrics.
Critical weaknesses
  1. The forward diffusion process appears incomplete for edge reactivation. The edge process can produce $ e_{ij,t-1}=0\rightarrow e_{ij,t}=1$. Since the constraint implies rij,t−1=0r_{ij,t-1}=0, it is unclear how the active-edge transition $ q_R(r_{ij,t}\mid r_{ij,t-1}=0,e_{ij,t}=1)$ is defined when R+={1,…,Krel}R^+=\{1,\ldots,K_{\mathrm{rel}}\}. This also makes the claimed closed-form relation marginal unclear when an edge becomes inactive and later reactivates. Because the main novelty lies in redefining the discrete diffusion process itself, this issue directly affects the technical soundness of the paper.

  2. The claimed V→E→RV\rightarrow E\rightarrow R factorization is not fully reconciled with the implementation. The method defines $ p_\theta(E_0\mid V_0,x_t)$, but the implementation appears to predict object and edge logits from the same Graph Transformer representation, while explicit conditioning is only described for the relation head through edge logits. It is therefore unclear whether the implementation truly realizes the proposed sequential probabilistic dependency or a weaker shared-representation factorization. In addition, the large effect of relation-class reweighting makes it difficult to isolate how much of the long-tail improvement comes specifically from the proposed diffusion formulation.

Mathematical soundness hole should be solved.

Questions for authors

Please refer to the critical weaknesses above

2 comments
AuthorsOpenReview ↗

Response to reviewer QcuW (1/2)

[W1]

Thank you for pointing out the missing discussion on forward edge reactivation, we should have expanded on all scenarios. We have revised the manuscript to include the edge reactivation case and the corresponding marginal in Appendix A.1.2 (colored blue).

When an inactive edge becomes active during forward diffusion (eij,t−1=0→eij,t=1)(e_{ij,t-1}=0\rightarrow e_{ij,t}=1), its previous relation information is lost since (rij,t−1=0r_{ij,t-1}=0). However, the newly activated relation can simply be initialized from the prior (πR\pi_R), i.e., a positive relation is sampled:

qR(rij,t∣rij,t−1=0,eij,t−1=0,eij,t=1)=πR(rij,t).q_R(r_{ij,t}\mid r_{ij,t-1}=0,e_{ij,t-1}=0,e_{ij,t}=1)=\pi_R(r_{ij,t}). (Eqn 8)

Likewise, the closed-form relation marginal can be derived from the edge state via the edge–relation evolution (Eqn 59), QtER=QtE(et−1,et)qR(rt∣rt−1,et−1,et)Q_t^{ER} = Q_t^E(e_{t-1},e_t)q_R(r_t\mid r_{t-1},e_{t-1},e_t) in place of relation-only QtRQ_t^R (Eqn 55) defined when eij,t=1e_{ij,t} = 1.

For example, consider an edge initially labeled person riding horse. During forward diffusion, its state may evolve as (e,r):(1,riding)⟶(0,0)⟶(1,beside).(e,r): (1,\texttt{riding})\longrightarrow(0,0)\longrightarrow(1,\texttt{beside}). When the edge disappears, its relation is set to zero. Upon reactivation, a new positive relation is sampled from πR\pi_R, allowing the edge to reappear with a potentially different relation. This entire trajectory is captured by the cumulative transition Q‾tER=∏s=1tQsER\overline Q_t^{ER}=\prod_{s=1}^{t}Q_s^{ER} which accounts for edge disappearance and reactivation while preserving the constraint (eij,t=0⇒rij,t=0e_{ij,t}=0\Rightarrow r_{ij,t}=0). In particular, the relation marginal for an active edge is obtained by conditioning the coupled E,RE,R marginal on edge existence (Eqn 60):

$q(r_{ij,t}=a|z_{ij,0},e_{ij,t}=1) = \frac{[\overline Q_t^{ER}]{z_{ij,0},(1,a)}} {[\overline Q_t^E]{e_{ij,0},1}}, \qquad a\in\mathcal R^+.\qquad z = (e,r)$.

The object and edge corruption kernels remain unchanged. Thus, the edge reappearance is handled in DiScGraph’s factorized diffusion formulation and maintains its key advantages: decoupling structural sparsity from relation semantics, explicitly modeling object, edge, and relation dependencies in the forward and reverse processes, and preserving valid scene-graph states throughout diffusion.

AuthorsOpenReview ↗

Response to reviewer QcuW (2/2)

[W2]

(a.) V→E→RV \to E \to R factorization:

Thanks for catching this. We apologize for this error in the architectural description. We have corrected the equations and clarified the conditioning in Appendix A.4.2 (revisions colored in blue).

Particularly, the prediction heads are explicitly coupled in the order (V→E→RV\rightarrow E\rightarrow R). Object states are passed to the edge head, and the resulting edge states are passed to the relation head.

ℓijE=softmax(WE[hijE,L,ℓiV,ℓjV,EmbV(vi,t−1),EmbV(vj,t−1)])\ell_{ij}^E=\text{softmax}\left(W_E\left[h_{ij}^{E,L},\ell_{i}^V, \ell_{j}^V,Emb_{V}(v_{i,t-1}),Emb_{V}(v_{j,t-1})\right]\right) ℓijR=softmax(WR[hijR,L,ℓijE,EmbE(eij,t−1)])\ell_{ij}^R=\text{softmax}\left(W_R\left[h_{ij}^{R,L},\ell_{ij}^E,Emb_E(e_{ij,t-1})\right]\right)

Therefore, DiScGraph implements hierarchical conditioning at both the prediction-head and reverse-sampling levels, rather than relying solely on shared Graph Transformer representations.


(b.) Class Weighting:

We agree that relation-class reweighting improves long-tail relation modeling. However, the improvements cannot be attributed solely to reweighting because DiScGraph's factorized diffusion formulation addresses a key imbalance inherent to scene graphs, sparsity dominates relation semantics. To isolate its contribution from the proposed diffusion formulation, we extend Table 7 with a new Joint + Effective-num ablation. The original flat edge–relation diffusion formulation is retained in joint and effective-num relation weighting is applied in its training. Specifically, positive relation classes use the same weights as DiScGraph, wk=λ+1−β1−βfkw_k=\lambda_+\frac{1-\beta}{1-\beta^{f_k}}, where fkf_k denotes the frequency of relation class k∈R+k \in R^+, and λ+=1\lambda_+=1. Since the joint categorical representation also includes the dominant no-edge class (k=0k=0), we assign it a separate weight w0=λ0=0.1w_0=\lambda_0=0.1 to limit its influence on the training loss.

MethodN-MMD (↓\downarrow)R-MMD (↓\downarrow)ID-MMD (↓\downarrow)OD-MMD (↓\downarrow)TRIP-TV (↓\downarrow)R-K-TV (↓\downarrow)
Joint (row1)0.00770.00780.00680.00620.69100.8118
Joint + Effective_num (new)0.00760.00740.00770.00790.66530.7892
Factorized state (row2)0.00660.00660.00560.00560.58220.6099
Factorized state + dependency-aware sampler - DiScGraph (row3)0.00460.00580.00430.00460.48780.5199

The results show that reweighting alone does not obtain the benefits of factorized diffusion in joint. Joint improves Rare-K-TV and R-MMD with reweighting, but factorized formulation is better. These improvements arise from two complementary mechanisms. Conventional graph diffusion models edge existence and relation semantics using a single categorical variable {0,1,…,Krel}\{0,1,\ldots,K_{\mathrm{rel}}\}, where the overwhelmingly frequent no-edge category dominates the corruption and denoising distributions. DiScGraph instead separates structural sparsity from relation semantics, restricting relation diffusion to R+∈1,…,KrelR^+\in{1,\ldots,K_{\mathrm{rel}}} on active edges. Class reweighting only modifies the training loss to address frequency imbalance among relation classes R+R^+, without changing the diffusion dynamics.

Thus, the results support that factorized diffusion addresses structural sparsity to enable relation learning, while relation-class reweighting provides complementary improvements in long-tail learning.

Reviewer_aYDiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DiScGraph, a discrete diffusion model for scene graph generation that factorizes the graph state into objects, edge existence, and relations defined only on active edges. It introduces an edge-gated forward noising process and a dependency-aware reverse sampler that generates objects, edges, and relations in the order V→E→R, ensuring that inactive edges carry no relation. Text conditioning is handled at inference time through reward-tilted sequential Monte Carlo with CLIP similarity, and the generated scene graphs are used to condition downstream text-to-image generation. Experiments show improved graph metrics, long-tail relation modeling, and competitive downstream compositional image generation.

Critical strengths

DiScGraph makes a structural adaptation of discrete diffusion to scene graphs: it decouples edge existence from relation semantics, enforces valid graph constraints through edge-gated corruption and dependency-aware reverse sampling, and validates the learned prior on both graph-level metrics and downstream SG-conditioned image generation.

Critical weaknesses
  • The motivation is not fully validated against strong modern T2I models. The paper argues that scene graphs compensate for the weakness of plain text in compositional relations, but the downstream gains rely heavily on SDXL-SG and CLIP reward. Since modern T2I models are already strong at composition, the paper should better justify whether an SG prior is still necessary, ideally by analyzing this motivation with stronger T2I backbones.

  • The ablation supports the factorized state and dependency-aware V→E→R sampler, but not the optimality of this specific order. The experiments show that factorized modeling plus dependency-aware sampling helps, yet they do not rule out other dependency orders or prove that V→E→R is the best choice.

  • The text-conditioned comparison may not be budget-matched. DiScGraph uses reward-tilted SMC with D=16 particles, per-step propagation, reward computation, resampling, and split-Gibbs/soft-mask refinement. The paper states that DiffuseSG and DiGress w/ text-cond use the same training-free reward-tilted conditioning, but does not clearly specify whether they use the same number of particles and refinement steps, making the comparison potentially unfair.

  • There is no fair runtime comparison, and the training/evaluation share the same CLIP metric. The Graph Transformer scales quadratically, and DiScGraph additionally uses multiple particles and refinement, so its compute cost is likely higher, yet no matched-cost comparison is provided. In addition, using CLIP both for reward tilting and evaluation may bias results toward the proposed method.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to reviewer aYDi (1/2)

[W1]

Yes, we have analyzed DiScGraph's performance on a stronger T2I backbone, FLUX.1 dev, in Appendix A.6.4. We enable zero-shot SG-conditioning in FLUX.1 dev using Universal Guidance ( FLUX-SG-test) and evaluate on CompSGBench prompts against native text-conditioned FLUX.1-dev. Table 13 (below) reports the inference-time compositional gains in FLUX from enabling DiSCGraph’s SG conditioning. The results demonstrate that DiSCGraph’s SGs contain relevant structural information that further improves compositional performance of a stronger backbone.

MethodFIDSG-IoUE-IoUR-IoU
FLUX.1-Dev38.270.48160.77030.7922
FLUX-SG-test37.220.52660.79980.8375

Additionally, we also evaluate compositional performance on FLUX.1dev via an alternative conditioning modality, bounding-boxes or layouts. In Section 7, we report new experiments that compare LLM-generated layouts with DiScGraph’s SG-to-layouts on a native layout-to-image model, CreatiLayout-FLUX. The results reported in Table 9 (Section 7 & below) and Figure 19 (Appendix A.9.2) highlight the compositional gains from DiScGraph’s improved object and relational fidelity. DiScGraph’s contribution is the scene-graph prior (P(G∣T)P(G\mid T)) which can be plugged to any downstream image generator (P(I∣G)P(I\mid G)). The FLUX.1 experiments demonstrate that DiScGraph’s generated structural priors can also benefit stronger modern T2I models through several conditional generation pathways.

MethodSG-IoU (↑\uparrow)R-IoU (↑\uparrow)E-IoU (↑\uparrow)BLIP-VQA (↑\uparrow)ImageReward (↑\uparrow)Obj Recall (↑\uparrow)Hall. Obj Rate (↓\downarrow)
LLM-layout0.50190.81630.87950.63891.30740.80150.2396
DiScGraph SG-to-layout0.58220.93550.95640.7781.53260.95340.0373

[W2]

Thank you for this point. DiScGraph does not enforce an object-first (V→E→RV\rightarrow E\rightarrow R) or edge-first (E→R→VE \to R \to V) ordering in its formulation. We adopt this ordering because it follows the intuitive process of constructing a visual scene: identifying objects, establishing their connectivity, and assigning relation semantics.

To evaluate sensitivity to this choice, we provide new results for the alternative E→R→VE\rightarrow R\rightarrow V dependency-aware sampler, which determines connectivity and semantic relations before object identities. The reverse transition is factorized as pθ(xt−1∣xt)=pθ(Et−1∣xt)pθ(Rt−1∣Et−1,xt)pθ(Vt−1∣Et−1,Rt−1,xt)p_\theta(x_{t-1}\mid x_t)=p_\theta(E_{t-1}\mid x_t) p_\theta(R_{t-1}\mid E_{t-1},x_t) p_\theta(V_{t-1}\mid E_{t-1},R_{t-1},x_t). We modify the denoiser conditional prediction heads accordingly, for example, object predictions are ℓiV=WV[hiV,L,ciE,ciR]\ell_i^{V}=W_V\left[h_i^{V,L},c_i^E,c_i^R\right]where ciE,ciRc_i^E, c_i^R are edge and relation state information. The same factorized state space, forward diffusion process, and experimental settings are retained. The implementation details are provided in Appendix A.7.7.

The new results in Table 20 (below) show that object-first and edge-first sampling produce similar graphs that match distributionally. The object-first ordering achieves better Triplet-TV and Attach-TV, suggesting that conditioning relation predictions on object identities (V→E→RV \to E \to R) provides a useful inductive bias for semantic consistency.

OrderN-MMD (↓\downarrow)R-MMD (↓\downarrow)ID-MMD (↓\downarrow)OD-MMD (↓\downarrow)TRIP-TV (↓\downarrow)R-K-TV (↓\downarrow)AT-TV (↓\downarrow)
V→E→RV\rightarrow E\rightarrow R0.00460.00580.00430.00460.49200.51940.1766
E→V→RE \to V \to R0.00520.00570.00420.00470.53080.52330.2081

Relation-before-edge orderings, R→E→V,V→R→E,R→V→ER\rightarrow E\rightarrow V, \quad V\rightarrow R\rightarrow E, \quad R\rightarrow V\rightarrow E, are less suitable for the factorized scene-graph representation. Sampling RR before EE produces relations before determining connectivity, since relation semantics are defined only for active edges, it requires including the null relation class {0,1,…,Krel}\{0,1,\ldots,K_{\mathrm{rel}}\}. Under the constraint rij=0⇔eij=0r_{ij}=0\Leftrightarrow e_{ij}=0, edge existence is then trivially determined by eij=1[rij≠0]e_{ij}=\mathbb{1}[r_{ij}\neq0], effectively collapsing edge-existence and relations into a single categorical variable, like any general graph diffusion work. This reintroduces the dominant no-edge case that DiScGraph’s factorization explicitly avoids.

AuthorsOpenReview ↗

Response to reviewer aYDi (2/2)

[W3]

Thanks for pointing this out, we should have mentioned the baseline settings clearly. DiffuseSG and DiGress are unconditional graph generators, so we apply the same training-free reward-tilted SMC procedure described in Section 3.4 and Appendix A.3, using identical reverse diffusion steps (T=100T=100), particle counts (D=16D=16), and CLIP reward guidance. However, Split-Gibbs and soft-mask refinements are designed for DiScGraph's factorized sampler and are not applied to these baselines.

To address the concern about additional refinements, we have extended Tables 10 and 11 (colored blue in paper, also shown below) with new results for DiScGraph without sampler refinements but with text-conditioning. This provides a matched comparison across all three methods. Consistent with the unconditional graph-generation results of DiScGraph without sampler refinements in Table 14 (row 1) outperforming the baselines in Table 1, DiScGraph without refinements also outperforms DiffuseSG and DiGress in downstream T2I generation across structural metrics (SG/E/R-IoU) and perceptual/compositional metrics (FID, BLIP-VQA, ImageReward).

These results show that DiScGraph's improvements persist even without the additional sampler refinements, highlighting the contribution of its hierarchical factorized diffusion formulation. The refinements provide further gains but are not the only reason for the performance difference relative to the baselines.

SGBase P(I/G)P(I/G)SG-IoU (↑\uparrow)E-IoU (↑\uparrow)R-IoU (↑\uparrow)
GT SGSDXL-SG0.5690.85280.8536
DiScGraphSDXL-SG0.56290.83570.8299
DiScGraph no sampler refinementsSDXL-SG0.53810.81620.8204
DiffuseSG w/ text-condSDXL-SG0.27330.72810.6197
DiGress w/ text-condSDXL-SG0.1460.66050.6088
-SDXL0.32600.76200.7188

SG priorBaseFID (↓\downarrow)CLIP-I2T (↑\uparrow)CLIP-I2I (↑\uparrow)BLIP-VQA (↑\uparrow)ImageReward (↑\uparrow)
GT SGSDXL-SG33.130.74040.33540.79911.4653
DiScGraphSDXL-SG33.850.73990.33500.78541.4512
DiScGraph no sampler refinementsSDXL-SG33.830.73150.31930.75791.3666
DiffuseSG w/ text-condSDXL-SG38.670.62350.25990.66081.0848
DiGress w/ text-condSDXL-SG45.720.5980.24130.60750.9535
-SDXL31.34**0.74150.33610.64251.1286

[W4]

  • Yes, we provide matched runtime comparisons with baselines. We have extended Table 18 (colored blue in paper, also shown below) with matched sampling settings for DiScGraph, DiffuseSG, and DiGress using identical reverse steps (T=100T=100), particles (D=16(D=16), and no sampler refinements (R=0R=0). DiScGraph has comparable runtime and memory usage to both baselines. Furthermore, adding Split-Gibbs and soft-mask refinements increases DiScGraph's runtime by only ~8%, while providing further improvements. As demonstrated in Tables 10–11, DiScGraph also maintains its superior downstream T2I performance without these refinements. Thus, DiScGraph’s improvements are still observed with comparable inference-time computations.
MethodParticles DReverse Steps, TRefinement Steps (Split-Gibbs, soft-mask), RParticle EvaluationsMean ResamplesTime / 100 SG (s)Peak GPU (GB)
DiScGraph110020120042
DiScGraph161002019207725
DiScGraph2410020288071026
DiScGraph no sampler refinements161000160011665
DiffuseSG w/ text cond161000160016685
DiGress w/ text cond16100016005665
  • Although Graph Transformers have quadratic complexity in graph size, both DiGress and DiffuseSG also use Graph Transformer denoisers. Therefore, DiScGraph does not introduce a higher denoiser complexity relative to these baselines, as demonstrated by the comparable runtime and memory measurements.
  • CLIP is not used to train DiScGraph's scene-graph diffusion model; it is employed only for training-free inference-time text conditioning. We agree that evaluating CLIP-guided generation using CLIP metric may favor the guidance objective. Therefore, our evaluation extends beyond CLIP similarity to include distributional image quality (FID), perceptual and compositional alignment (BLIP-VQA and ImageReward), and fine-grained structural consistency (SG-IoU, E-IoU, R-IoU) metrics. DiScGraph outperforms DiffuseSG and DiGress on these complementary metrics all under the same CLIP-guided conditioning (Tables 10,11). Therefore, the observed improvements are not solely attributed to optimization of the CLIP reward.

Thank you for your response and for the additional experiments. My concerns regarding the ablation study and runtime have been addressed.

However, with respect to the motivation, the authors are still comparing the accuracy of the SG rather than its necessity. Table 13 does support this to some extent, but in fact the FID differences reported in Table 13 are not significant, and is the generation with 30+ FID even reliable? It seems a little too high. Moreover, I still find the qualitative comparison insufficient: only Fig. 3 is provided, but Fig. 3 does not compare against the strongest T2I baseline, and the improvement it shows seems to lie more in visual quality than in relations.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to reviewer RbwV

[W1]

Scene graphs provide explicit object–relation structure that can improve spatial consistency in complex compositional generation. To demonstrate their value beyond SG-specific backbones, we compare DiScGraph-derived layouts against LLM-generated layouts using CreatiLayout-FLUX, a modern FLUX.1-Dev-based layout-to-image generator backbone. The results show that DiScGraph’s structure-semantic modeling remains useful at improving modern T2I models. Please see our response to Q1 below.


[W2]

Thanks for highlighting EDGE's extension to attributed graphs. We agree that our previous discussion did not sufficiently acknowledge this extension. We have revised Appendix A.5 (revisions colored blue) to clarify how DiScGraph’s factorized process offers unique improvements and its importance for scene graph generation. We also briefly state it here,

The key distinction: In DiScGraph’s factorized formulation, the structure and semantics continuously evolve together through joint modeling of objects, edges and relations. This produces realistic and valid scene graphs that represent a coherent visual scene.

EDGE’s factorization for attributed graphs is p(V,E,R)=p(V)p(E∣V)p(R∣V,E)p(V,E,R)=p(V)p(E\mid V)p(R\mid V,E), generating the topology (structure) before relation (semantics) attributes. EDGE’s relation diffusion only refines labels on previously generated edges. Therefore, relation semantics cannot inform structure generation, and missing edges cannot be recovered during relation assignment.

In contrast, DiScGraph performs diffusion over xt=(Vt,Et,Rt)x_t=(V_t,E_t,R_t), with factorized predictions, pθ(V0,E0,R0∣xt)=pθ(V0∣xt)pθ(E0∣V0,xt)pθ(R0∣V0,E0,xt)p_\theta(V_0,E_0,R_0\mid x_t) =p_\theta(V_0\mid x_t) p_\theta(E_0\mid V_0,x_t) p_\theta(R_0\mid V_0,E_0,x_t). (Eqn 13 in paper)

Unlike EDGE, DiScGraph’s topology predictions can incorporate the evolving relation state (RtR_t), enabling structural and semantic information to interact across reverse diffusion steps. The constrained reverse posterior (Eqn 14) further handles edge reactivation (et=0,et−1=1)(e_t=0,e_{t-1}=1), requiring relations to be inferred when previously absent edges reappear.

This addresses a central challenge in scene graph generation: structural attributes and visual scene semantics are dependent and mutually informative, rather than topology being merely a prerequisite for assigning relations. Therefore, DiScGraph incorporating structural-semantic dependencies is suited for scene graph generation.


[W1, Q1]

Scene graphs and layouts provide structured representations of prompts for image generation, but from different perspectives. Layouts encode objects and their spatial placements without explicitly modeling inter-object connectivity or relations , whereas scene graphs additionally specify connectivity and semantic relations, therefore SGs have advantages over layouts that can potentially improve complex compositions.

To directly address this question, in Section 7, we provide new results using CreatiLayout-FLUX, a modern native layout-to-image generator built on FLUX.1-Dev, a SOTA generator. Briefly, we compare Sonnet 4.5-generated layouts against DiScGraph's SG-derived layouts on 100 complex CompSGBench prompts. The new results in Table 9 (below) show improved structural fidelity (SG/E/R-IoU) and compositional quality (BLIP-VQA and ImageReward) with DiScGraph-derived layouts. Qualitative examples in Figure 19 also show that the LLM-layouts exhibit excessive bounding-box overlaps in foreground, collapsing multiple objects, and impairing complex compositions. In contrast, DiScGraph better preserves spatial consistency through semantic relation modeling. DiScGraph’s SGs thus offer an alternative structural representation that remains an useful addition to modern image generators.

MethodSG-IoU (↑\uparrow)R-IoU (↑\uparrow)E-IoU (↑\uparrow)BLIP-VQA (↑\uparrow)ImageReward (↑\uparrow)Obj Recall (↑\uparrow)Hall. Obj Rate (↓\downarrow)
LLM-layout0.50190.81630.87950.63891.30740.80150.2396
DiScGraph SG-to-layout0.58220.93550.95640.7781.53260.95340.0373

We also report object recall and hallucinated object rate that measure if all entities in the prompt are captured in the layouts and no excessive extra layouts are generated. We observe that LLM-generated layouts tend to miss prompt-objects and introduce too many layout-objects compared to DiScGraph, also illustrated in Figure 19. Regarding computational cost, both approaches require upstream inference: graph diffusion with reward-tilted sampling for DiScGraph (Table 18 reports cost analysis) and LLM inference for Sonnet 4.5. While their computational costs are not equivalent, using the same downstream image generator isolates the effect of the corresponding layouts.

Thanks for the responce.

My main concern is whether explicit scene graphs are necessary for T2I generation and whether their benefits justify the additional complexity in practice.

I agree that scene graphs explicitly encode inter-object connectivity and semantic relations beyond what layouts alone represent. However, it remains unclear whether this explicit graph representation offers a substantial advantage over layouts combined with the original text prompt, which already describes those relations. For example, to generate “a man riding a horse,” one could provide bounding boxes for the man and the horse while retaining the relation “riding” in the text prompt.

The new CreatiLayout-FLUX experiments demonstrate that DiScGraph produces more effective layouts than the evaluated LLM baseline. However, since the downstream generator receives layouts and text rather than explicit scene graphs, these results primarily establish the effectiveness of DiScGraph as a layout generator; they do not isolate the benefit of explicit scene graph modeling over other approaches to generating relation-aware layouts.

From a practical perspective, directly generating layouts conditioned on text may also offer a simpler pipeline and potentially more scalable data collection than learning a scene graph prior that requires explicit relation annotations.

I also agree with Reviwer aYDi that modern T2I models are already strong at composition such as Flux3 Imgae and MOSS image, which prove the text + layout can express the relati lons well, so I still concerns about the motivation of this work.

Maybe the author can provide some practical use cases that scene graph based T2I generation is better than layout + relation description?

AuthorsOpenReview ↗

Response to Reviewer RbwV

Thank you for your response.

SG advantage:  Layouts+text does not explicitly model the dependencies between object identities, connectivity, and relations. They rely on the generator to learn the dependencies implicitly. In contrast, SGs explicitly model them via object-relation triplets. This offers a few advantages:

  • Precise relation-level control: Consider a scene containing two people and a dog. A user wants to change it from person A feeding the dog to person B feeding the dog, while preserving the other objects and their spatial arrangement. With text + layout, the model has to interpret the changed description, determine which object interaction changed and which did not. With an SG, the user can simply modify the relevant triplet:(person A,feeding,dog)⟶(person B,feeding,dog).(\text{person A},\text{feeding},\text{dog}) \longrightarrow(\text{person B},\text{feeding},\text{dog}).
  • Under-specification in prompts: An underspecified prompt or portion of a prompt such as “a dog playing with a ball and a table” can imply different relationships (dog beside table, dog on top of table, dog under table). DiScGraph explicitly models a distribution over such object–relation choices, enabling different semantic interpretations, while retaining the specificity in other parts.
  • Explicit relation constraints: Let a prompt be a worker is holding a tool, the tool is touching a pipe, and the pipe is connected to a valve. SGs can represent this directional interconnection as explicit graph constraints (pipe not touching worker) with possible localized modifications, whereas layout+text requires the implicit cues to be learned by the generator.

These examples show that the advantage of SGs is not additional information absent from the text, but a representation that offers stronger control over structure. Thus, explicit vs implicit difference really becomes useful in complex multi-object multi-relation scenes as observed in FLUX.1 dev experiments (Table 13), comparisons with composition works (Tables 4,5), and CreatiLayout (Table 9, Figure 19). Moreover, there is additional complexity/effort in incorporating either layout or SG modalities.

We also want to clarify some points,

  1. Contribution: Our main contribution in this work is a hierarchical factorized discrete diffusion framework designed for graphs with structural–semantic dependencies and long-tailed relation distributions, such as scene graphs. Compositional T2I generation is an important downstream application highlighting the utility of the learned graph prior. Beyond T2I, SGs also help applications such as VQA reasoning or image retrieval that benefit from complex visual scenes represented as graphs.
  2. Structural conditioning: Explicit layout or SG conditioning is probably not useful for simple prompts and if T2I models continue to improve then yes we agree any structural conditioning (layout or SGs) will reduce in its utility. However, complex composition is still a weakness in existing T2I models, addressed by various layout, SG, or text-based interventions.
  3. Scalability: Text-to-layout is not inherently simpler or more scalable than text-to-SG. Both require inferring and annotating the respective structured representations from text and training conditional image generators on them. We showed SG-layout as an alternative pathway in our CreatiLayout experiments but SGs can be consumed directly by a conditional generator as with layouts (FLUX: Table 13, SDXL: Tables 12, 5 and 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.