acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Cross-modal Representation Alignment for Text-to-image Diffusion Models

Abstract

Modern text-to-image diffusion models inherit rich compositional semantics from pretrained text encoders, yet their denoising objectives provide limited direct supervision for binding phrases to distinct image regions. Building on the success of representation alignment with external models, we harness the generator's own language representations as a source of hierarchical supervision. We introduce Hyperbolic Intermodal Projection (HIP), a lightweight method for cross-modal alignment in text-to-image diffusion models. HIP promotes fine-grained cross-modal alignment, which is stabilized by anchoring local representations to their global semantic context. To preserve the distinctiveness of concepts sharing the same anchor, we introduce a shared hyperbolic space whose geometry supports local separation within a coherent scene neighborhood. Experiments on SD3.5-M and FLUX.1-dev demonstrate improvements over baselines: HIP improves GenEval macro accuracy by and relative to the respective pretrained baselines, and achieves the highest scores across all six evaluated categories under the official T2I-CompBench evaluator on both backbones.

Primary area:
generative models
Keywords:
alignment, diffusion model, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZJ3AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HIP, a hierarchical cross-modal representation alignment method for text-to-image diffusion models. Experiments show that HIP improves compositional generation on SD3.5-M and FLUX.1-dev without adding extra modules during inference.

Critical strengths
  1. This paper starts with a clear motivation and focuses on fine-grained text–image alignment in diffusion models. It identifies the lack of explicit phrase region supervision in standard flow matching training and proposes hierarchical alignment to address this problem.

  2. This paper proposes a good method that combines local phrase region alignment with global semantic anchoring. It uses the joint attention of the model itself to construct region representations without external teachers or region annotations.

Critical weaknesses
  1. Appendix E reports results averaged over three training seeds, while Appendix K describes four images per prompt and refers to 553 retained images per checkpoint during rescoring. However, the relationship between these settings for the reported GenEval scores remains unclear, making the results difficult to reproduce.

  2. Section 3.4 explains the benefits of hyperbolic space through geometric separation and asymptotic representation capacity. But, the HIP uses only a shallow two-level hierarchy. As a result, this paper does not clearly prove whether the learned representations benefit from these theoretical properties.

Questions for authors
  1. Please provide further analysis of the contribution of hyperbolic geometry.

  2. Please clarify the experimental details such as number of generated images per GenEval prompt, the final number of evaluated images, and so on.

Reviewer_e8SAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Intermodal Projection (HIP) to improve compositional generation in text-to-image diffusion models. HIP uses the generator’s internal language representations and attention to establish phrase–region correspondences, jointly learning these correspondences with whole–part constraints that anchor constituents to their sentence or image context in a shared hyperbolic space. Training requires neither external teacher models nor region annotations. At inference, the auxiliary modules are discarded and the LoRA updates are merged into the backbone, preserving the original generator’s computational structure. On SD3.5-M and FLUX.1-dev, HIP improves GenEval accuracy by 14.48% and 8.24%, respectively, relative to the pretrained baselines, and achieves the highest scores among the compared methods across all six evaluated T2I-CompBench categories.

Critical strengths
  • HIP turns the generator’s language representations into fine-grained phrase–region supervision, avoiding external teacher models and region-level annotations. Combining local correspondence with whole–part anchoring provides a coherent approach to preserving scene context while distinguishing individual concepts.

  • HIP improves GenEval accuracy on both SD3.5-M and FLUX.1-dev and achieves the highest scores among the evaluated methods across all six T2I-CompBench categories.

  • The component study shows that phrase–region alignment is the strongest individual auxiliary objective and that the full method performs best. The supplementary Euclidean-space control and representation diagnostics further support the combined alignment and geometric design.

Critical weaknesses
  • The contribution of hyperbolic geometry remains insufficiently isolated. The Euclidean control is valuable, but identical temperatures, margins, and loss weights do not ensure comparable distance scales or optimization dynamics across geometries. Geometry-specific tuning under equal search budgets and a factorial comparison of geometry and anchoring would strengthen the causal interpretation.

  • Grounding on unseen images needs stronger evidence. The unseen COCO evaluation measures internal consistency because the model’s own attention defines the matching regions. The annotated-box evaluation provides complementary evidence: box mass improves, although pointing accuracy does not. However, it uses training images and a separate set of checkpoints. Evaluating unseen images with box annotations, using the main generation checkpoints, would better establish whether HIP’s grounding improvements generalize.

  • The uncertainty of small generation gains is not reported. Appendix E reports averages over three training seeds, but the generation benchmark tables do not provide seed-level results or variation across runs. This matters for the FLUX GenEval difference between HIP (0.69) and InfoNCE without hierarchy (0.68). Seed-level generation scores and uncertainty estimates for the differences would help.

  • The main-text claim about global alignment is stronger than the appendix supports. Appendix B.5 correctly distinguishes the absolute distances in Equation (14) from the relative discrimination learned by InfoNCE. However, the main text states that HIP’s auxiliary loss bounds the sentence–image distance. Equation (14) alone does not establish this connection, because its right-hand side includes absolute phrase–region distances rather than InfoNCE. The main text should state the result conditionally or provide the additional argument needed for the stronger claim.

Questions for authors
  • How sensitive is the geometry comparison to distance scale and hyperparameter choices? Does HIP’s advantage over the Euclidean control remain when temperature, margin, and loss weights are tuned separately under equal search budgets? As an additional analysis, a Euclidean comparison with and without anchoring would help clarify how the two components interact.

  • Do the grounding improvements generalize to unseen images with box annotations? Could you evaluate box mass and pointing accuracy on a held-out set using the same checkpoints as the main generation experiments?

  • How consistent are the generation gains across training seeds? Could you report seed-level scores and uncertainty estimates for the performance differences, particularly the FLUX GenEval comparison between HIP (0.69) and InfoNCE without hierarchy (0.68)?

  • Under what conditions does HIP’s auxiliary loss control the sentence–image distance? Equation (14) depends on absolute phrase–region distances, whereas InfoNCE learns relative discrimination. Could you explain how the joint objective bridges this gap, or revise the main-text statement to reflect the conditional nature of the bound?

Reviewer_JPLyOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a fine-tuning method for text-to-image generators Stable Diffusion 3.5 and FLUX to help them avoid concept bleeding of specific parts of the prompt into unrelated parts of the image. During training it extracts the noun phrases from each caption. It finds the image region each phrase refers to by reading the model’s own attention. It then adds a loss that makes each phrase’s representation match its own region and not the other regions in the image. It also adds penalties that keep each phrase near the whole-caption embedding and each region near the whole-image embedding. Their losses are in hyperbolic space rather than Euclidean space.

Critical strengths

The method is simple and practical. It’s a LoRA fine-tune with auxiliary losses. At inference the projection heads are discarded and the LoRA merged, so there’s no added parameters or compute.

It needs no labels. Phrase regions come from the model’s own joint attention, with no boxes or external teacher.

The submission is transparent. The appendix is unusually detailed, the code is released, and the readme openly discloses its own limitations.

Critical weaknesses

The reported numbers can’t be tied to the method as described. The readme (hip_reproduction_bundle/readme.md line 100) says the retained run configs don’t match the paper’s loss weights, that the paper configs don’t claim to correspond to the reported checkpoints, and that the config for the main SD3.5 result is missing.

The paper contradicts itself on whether results are averaged over seeds. Line 1409 says there are 3 random seeds but lines 363–364 say a single checkpoint is used. And the provided configs in configs/original/ use seeds 42 and 123.

Closest prior work and baselines are missing. TokenCompose (CVPR 2024), which fine-tunes with token-level grounding losses, isn’t cited. Nor are inference-time binding methods (CONFORM, Attend-and-Excite, SynGen) used as baselines.

Questions for authors

Which config, seed and scorer produced each reported row in the paper?

I would like to see a comparison against Attend-and-Excite or another inference-time binding method adapted to MMDiT. Also tokencompose is another very related work so the differences to it should at least be discussed.

I would like to see an ablation on the curvature chosen for the model to see if the curved space is important for the performance of their method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.