acceptodds
Withdrawn from ICLR 2027

ESG: Early Spatial Guidance for Compositional Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image models often generate the requested objects but fail to arrange them according to the prompt. We present Early Spatial Guidance (ESG), a training-free method for improving spatial composition in rectified flow transformers. ESG reads object locations from native joint attention and updates the latent state to satisfy relative spatial constraints during early sampling. The objective leaves absolute object positions unspecified and handles multiple objects by combining pairwise relations. Normalized updates control intervention strength, while attention-layer calibration adapts the spatial readout to each model. Gradients require only the transformer prefix leading to the selected readout. We analyze the sensitivity of this readout, the interaction of relation gradients, and how interventions propagate through the remaining sampling trajectory. Experiments on SD3 and SD3.5 show improved spatial alignment on public benchmarks and generalization to unseen object pairs. Comparisons with spatial-guidance methods and time-matched native sampling demonstrate a favorable accuracy–cost tradeoff. On controlled multi-object tasks, ESG outperforms the evaluated absolute-centroid objective at matched update budgets, with the clearest gains under repeated guidance. These findings show that early attention-based guidance can coordinate multiple spatial relations at competitive inference cost, without retraining the generator or requiring user-specified layouts.

Primary area:
generative models
Keywords:
Text-to-image generation, Spatial composition, Rectified flow, Attention guidance, Training-free generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_udmFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Early Spatial Guidance (ESG), a training-free method for improving spatial relations in text-to-image generation. ESG estimates object centers from native joint attention and uses their relative positions to guide latent updates during early sampling. The objective combines pairwise directional relations without specifying absolute object coordinates. Updates are normalized relative to the latent norm, and gradient computation uses only the transformer prefix needed for the selected attention readout. The paper analyzes readout sensitivity, interactions between relation gradients, and the limits of translating attention-level improvements into correct image layouts. Experiments on SD3 and SD3.5 cover GenEval position, T2I-CompBench spatial relations, unseen object pairs, and controlled multi-object scenes. The reported results show improved spatial alignment at competitive inference cost, supported by comparisons with existing controllers, time-matched native sampling, and matched objective controls.

Critical strengths
  1. ESG provides a simple, training-free approach to spatial control without requiring user-specified object coordinates. Normalized latent updates and gradient computation through the selected transformer prefix make the method practical, with reported improvements even from a single early intervention.

  2. The empirical evaluation includes paired comparisons under common computation budgets, time-matched native sampling, and controls that change only the spatial objective. These experiments provide meaningful evidence that the improvements go beyond simply spending more time on generation.

  3. The paper carefully distinguishes attention-level behavior from final image correctness. Its analysis identifies limitations of combining relation gradients, while the experiments report both average relation accuracy and all-relations success, alongside object-count and image-quality diagnostics. This makes the benefits and limitations easier to assess.

Critical weaknesses
  1. The paper does not adequately discuss related training-free layout-control methods, including BoxDiff, Layout Guidance, Attention Refocusing, R&B and MAC. Their relevance should be clarified, including whether a representative comparison is feasible under matched input and computation budgets.

  2. The multi-object absolute-position control can assign conflicting targets to shared objects, making it difficult to separate the benefit of relative constraints from weaknesses in the comparator’s construction.

  3. The summed relation objective cancels intermediate object coordinates in collinear chains. Although acknowledged, this limits direct control over individual relations; the experiments do not compare against an objective that specifically targets unsatisfied relations.

Questions for authors
  1. How does ESG relate to BoxDiff [1], Layout Guidance [2], Attention Refocusing [3], R&B [4], and MAC [5]? Could a representative method be compared using layouts derived from the same relation inputs and a comparable computation budget?

  2. Does ESG retain its advantage when the absolute-position control uses globally consistent targets for shared objects, with the same readout and update procedure?

  3. How does cancellation of intermediate coordinates affect chain compositions? Would an objective that specifically targets unsatisfied relations improve all-relations success under the same update budget?

Reference:

[1] Jinheng Xie et al. BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion. ICCV, 2023.

[2] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-Free Layout Control With Cross-Attention Guidance. WACV, 2024.

[3] Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded Text-to-Image Synthesis with Attention Refocusing. CVPR, 2024.

[4] Jiayu Xiao, Henglei Lv, Liang Li, Shuhui Wang, and Qingming Huang. R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation. ICLR, 2024.

[5] Huancheng Chen, Jingtao Li, Weiming Zhuang, Haris Vikalo, and Lingjuan Lyu. Training-Free Layout-to-Image Generation with Marginal Attention Constraints. CVPR Workshops, 2026.

Reviewer_uG73OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Early Spatial Guidance (ESG), a training-free method for improving spatial relations in text-to-image generation. ESG estimates object centers from native joint attention and optimizes a relative-position objective through normalized latent updates during early sampling. Guidance gradients are computed only through the transformer prefix needed for the selected attention readout. Experiments on SD3 and SD3.5 evaluate spatial accuracy and inference cost on public benchmarks and controlled multi-object tasks, alongside analyses of readout sensitivity and interactions between relation gradients.

Critical strengths
  1. The method requires no additional training or user-provided layouts.
  2. The evaluation includes measured inference costs, time-matched sampling baselines, and controlled ablations of the objective, intervention timing, and readout layer.
Critical weaknesses
  1. Limited methodological novelty beyond integrating established components. Most of ESG’s central design principles have clear precedents:

    ESG componentPrior work
    Attention-derived object centroids and gradient-based spatial guidanceSelf-Guidance (2023)
    Gradient normalization for spatial guidanceZestGuide (2023), although ESG uses a different update scale
    Early-stage spatial interventionZestGuide (2023); STORM (2025)
    Relative spatial control without user-specified absolute coordinatesSTORM (2025); InfSplign (2025)
    Spatial objectives based on directional centroid separationInfSplign (2025)

    Neither training-free operation nor the absence of user-provided coordinates distinguishes ESG from these predecessors. The remaining changes—its particular update scaling, native-attention readout, and selected-prefix computation—appear incremental. Moreover, the objective-only ablation compares against absolute anchors rather than the closest existing relative-centroid objective, leaving the substantive contribution of the proposed formulation insufficiently established.

  2. The multi-relation objective can cancel information needed to enforce individual constraints. For a horizontal chain requiring A left of B and B left of C, the summed objective becomes

=\mu_{C,x}-\mu_{A,x}.$$ The middle object's position disappears from the objective: configurations with identical endpoints receive the same score regardless of whether B lies between them. Shared latent coupling may still move B, but provides no assurance that either individual constraint improves. This is a structural limitation of the proposed composition rule, and the reported 11.25% all-relations success further indicates that reliable multi-object control remains unresolved. 3. **The theoretical analysis does not establish the connection to final-image spatial correctness.** The guarantees primarily concern attention-centroid sensitivity and conditional downstream response. However, attention centroids are proxies for object locations, and improving their ordering need not improve the geometry of the generated image. The assumptions connecting readout ascent to terminal spatial improvement are not verified on the evaluated models. Although the paper acknowledges this limitation, the analysis consequently provides limited justification for the central mechanism's image-level reliability.
Questions for authors
  1. What substantive methodological advance does ESG provide beyond the established components listed above? Can you isolate its benefit over InfSplign’s relative-centroid spatial objective using the same readout, update rule, and intervention schedule?

  2. How does ESG enforce intermediate-object constraints when centroid terms cancel in a relation chain? Can you report results specifically measuring whether intermediate objects satisfy both adjacent relations?

  3. What empirical evidence connects improvements in the attention-based objective to improvements in final-image geometry? How often does the readout score improve while the generated spatial relations remain incorrect or deteriorate?

Reviewer_4c4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ESG, a training-free method for improving spatial relations in text-to-image generation. It extracts entity locations from internal attention maps, converts textual left/right/above/below relations into relative centroid objectives, and updates the latent during early diffusion steps.

Critical strengths

1.The method is training-free and includes useful cost-aware comparisons, time-matched sampling controls, and multi-object evaluations.

2.Experiments on SD3 and SD3.5 show improved spatial-relation scores at comparable inference cost.

Critical weaknesses

1.Novelty is unclear. Early-stage spatial guidance is already explored by prior work such as STORM. The paper does not clearly show that its combination of relative objective, normalized update, and prefix gradient is substantially new.

2.The attention-grounding assumption is unproven. The method assumes attention centroids indicate real object locations, but does not quantitatively validate this.

3.Limited generality. Experiments only use SD3 and SD3.5, and the method relies on simple rule-based parsing rather than robust free-form language understanding.

Questions for authors

1.Do attention centroids match actual generated-object locations, including failure cases?

2.The paper partially compares the native joint-attention readout with a text-normalized alternative. However, it does not individually ablate the normalized update or selected-prefix gradient. Please provide these component-wise ablations and report whether the small joint-attention improvement is statistically robust.

3.Does the method transfer beyond SD3/SD3.5?

4.Does it work for more complex free-form prompts, paraphrases, and repeated entities?

5.Why is prompt-only relative control preferable to layout/reference-based generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.