acceptodds
Withdrawn from ICLR 2027

Zero-Shot Multi-Concept Image Generation via Decoupled Cross and Self-Attention in MMDiT

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating high-fidelity images with multiple reference concepts is an important but challenging task. Existing methods either rely on additional training or suffer from identity blending, attribute leakage, and structural inconsistency in complex multi-reference scenarios. We propose Decoupled Cross- and Self-Attention MMDiT (Deco-DiT), a training-free framework for multi-concept image generation. Deco-DiT decouples multi-reference conditioning into Multi-Reference Cross-Attention (MR-CA) for localized identity routing and Spatial-Consistent Self-Attention (SC-SA) for reference-target structural modeling. MR-CA routes encoder-level visual features to LLM-parsed spatial regions, while SC-SA enables target tokens to access structural and appearance cues from reference latents. Together with a stage-wise denoising schedule, Deco-DiT improves identity preservation, text alignment, and compositional consistency across challenging multi-reference settings without additional training.

Primary area:
generative models
Keywords:
Zero-Shot, Multi-Concept Image Generation, Diffusion Models, Training-Free

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_muprOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Deco-DiT, a training-free multi-concept generation method on FLUX.1-dev that combines LLM-predicted box masks over SigLIP reference features, polyptych self-attention with noised reference latents, and a two-stage prompt schedule, and reports the best CLIP-T/CLIP-I/DINO-I with three references on a 10-combination benchmark.

Critical strengths
  • Multi-concept composition without per-concept tuning is a practically relevant problem, and the method requires no additional training.
  • The ablation visualizations (Fig. 3) are intuitive: removing each component produces a distinct failure mode.
Critical weaknesses
  • Outdated methodology. The authors used a pure text-to-image (T2I) model and turned it into a text-image-to-image (TI2I) model by a training-free method. However, current foundation models, like the Qwen-Image family, the FLUX.2 family, and a lot of closed-source models, are born with TI2I abilities. One year ago, many training methods of adapter-based models were proposed, as mentioned in the related works of this paper, and paved the way to TI2I foundation models. For ICLR 2027, I hardly think this paper has enough academic value.
  • Missing key implementation details, like which LLM is used, how you project SigLIP space into T5 space and so on, making it hard to reproduce this work.
  • Limited novelty. Box-based attention masking, polyptych in-context reference conditioning, and timestep-dependent prompt switching are all established techniques; the paper combines them without new analysis of why this combination works.
  • Minor issue: please take care of the citation format.
Questions for authors
  • Referring to critical weakness 1, what is the academic contribution of this paper in 2027 from the authors' perspective? Is this method beneficial to newer base models, or is there any insight I missed?
  • It seems magical to me that you just turned a pure text-to-image model into a text-image-to-image model by a training-free method, and even introduced a new feature space. It is uncanny, unless you used Redux, is that right? Please explain how you project the SigLIP space into T5 space.
Reviewer_3QFcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Deco-DiT, a training-free method for multi-concept, reference-driven image generation built on FLUX.1-dev. An LLM first parses the prompt into per-concept bounding boxes and a background description. Multi-Reference Cross-Attention (MR-CA) then injects SigLIP features of each reference only into its assigned box, using an additive attention mask. Spatial-Consistent Self-Attention (SC-SA) tiles noised reference latents together with the target latent into a "polyptych" and runs joint self-attention over it. A two-stage text schedule applies the background prompt during the first 20% of steps and the full prompt afterwards. The method is evaluated on a self-built benchmark drawn from DreamBench++ and CustomDiffusion concepts (10 combinations × 10 prompts × 10 seeds for each of 1, 2 and 3 references), using CLIP-T and SAM2-segmented CLIP-I/DINO-I. The baselines are MS-Diffusion, MIP-Adapter, OmniGen2, DreamO and XVerse. Deco-DiT reports the best CLIP-T at every reference count and the best scores on all metrics in the three-reference setting. Ablations of the guidance stages, the layout boxes and SC-SA are provided.

Critical strengths
  1. A practical setting with a clear SC-SA effect: A training-free multi-concept method on a frozen MMDiT is useful in practice. The SC-SA ablation (Table 2) shows a large drop when it is removed (DINO-I 0.6705 →\to 0.5634), clearly larger than the gaps between methods in Table 1. This indicates that reference-latent self-attention is the main source of identity preservation in the pipeline.
  2. Competitive results in the three-reference setting: against trained FLUX-based methods (Table 1: CLIP-I 0.8531 vs. 0.8448, DINO-I 0.6705 vs. 0.6571 for XVerse), obtained without any additional training by the authors.
Critical weaknesses
  1. Limited novelty. The core design routes semantic image-encoder features through cross-attention for identity and feeds reference features into self-attention for fine detail. This is an established pattern: Animate Anyone (CVPR 2024) and IDM-VTON (2024) both use it. In the training-free FLUX setting, closely related mechanisms also already exist. Personalize Anything (AAAI 2026) injects reference tokens region by region and uses a timestep threshold to separate early identity formation from later global refinement. LAMIC (AAAI 2026) restricts attention within each reference–region group and defers fusion across regions. Diptych Prompting (CVPR 2025) performs joint self-attention over a tiled reference–target canvas. Deco-DiT's three components map closely onto these prior designs, yet the Related Work section does not discuss any training-free method. The paper does not make clear what it contributes beyond combining existing techniques.
  2. No comparison with methods of the same kind. The main selling point of the paper is that it is training-free, but every baseline in Table 1 (MS-Diffusion, MIP-Adapter, OmniGen2, DreamO, XVerse) is a trained model that is only tuning-free at inference. The closest competitors are training-free methods that the authors themselves cite (Personalize Anything, LAMIC, LoRAShop), and none of them is compared. OmniGen is listed as a baseline in Section 4.2 but is missing from Table 1. Deco-DiT also receives LLM-generated layouts, whereas the baselines receive none; this includes MS-Diffusion, which can take layout input. As a result, Table 1 does not show that Deco-DiT advances the state of the art within its own category.
  3. The method is underspecified, and the training-free claim may be inaccurate. Eq. 2 states that the 1152-d SigLIP features are projected with weights "inherited from the pretrained MMDiT block". These weights operate on 3072-d hidden states, so the dimensions do not match as described. The standard route from SigLIP features into FLUX is the trained FLUX.1 Redux projector. If that projector or a similar module is used, the paper should disclose it and qualify the "training-free" claim. Several other key details are also missing: which blocks use MR-CA and SC-SA, how the MR-CA output is combined with the native joint attention, how tokens outside every box are handled (Eq. 4 gives an all-−∞-\infty row), and how the polyptych is laid out for N≠3N \neq 3. There is no appendix and no code, so the method cannot be reproduced from the paper.
  4. The evaluation does not adequately support the claims. The benchmark is small and unreleased: 10 concept combinations per setting, with the 10 seeds per prompt adding samples but not diversity. The paper also does not explain how SAM2 regions are assigned to each reference, which directly affects the identity metrics. CLIP-I and DINO-I reward copying the reference, and several outputs indeed reproduce the reference pose and viewpoint almost unchanged (Fig. 2, row 3; Fig. 3). The "w/o global guid." variant produces a visible box collage, yet reaches nearly the same DINO-I as the full model. Finally, the claim that motivates the paper that shared attention in DiTs amplifies identity leakage is never examined, and MR-CA itself is never ablated. It therefore remains unclear which component is responsible for the gains, and why.
Questions for authors
  1. How are SigLIP features mapped into the input space of the FLUX attention projections? Is FLUX.1 Redux, or any other pretrained adapter, used?
  2. How are SAM2 regions obtained and assigned to reference concepts? Is the same procedure applied to all methods, and are Deco-DiT's own layout boxes used as prompts?
  3. Can you report comparisons with Personalize Anything and LAMIC (and with MS-Diffusion given the same LLM layouts), with mean ± std over seeds? Can you also add an ablation that removes MR-CA entirely?
  4. Can you report a human study or a VLM-based score (e.g., the DreamBench++ protocol) to show that the identity gains are not due to copying the reference?
Reviewer_xycXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Generating high-fidelity images with multiple reference concepts is an important but challenging task. Existing methods either rely on additional training or suffer from identity blending, attribute leakage, and structural inconsistency in complex multi-reference scenarios. We propose Decoupled Cross- and Self-Attention MMDiT (Deco-DiT), a training-free framework for multi-concept image generation. Deco-DiT decouples multi-reference conditioning into Multi-Reference Cross-Attention (MR-CA) for localized identity routing and Spatial-Consistent Self-Attention (SC-SA) for reference-target structural modeling. MR-CA routes encoder-level visual features to LLM-parsed spatial regions, while SC-SA enables target tokens to access structural and appearance cues from reference latents. Together with a stage-wise denoising schedule, Deco-DiT improves identity preservation, text alignment, and compositional consistency across challenging multi-reference settings without additional training.

Critical strengths
  1. Multi-reference composition without additional training has clear practical value. The combination of region-restricted encoder-feature injection, reference-latent interaction, and staged text guidance offers some novelty (Sections 3.2–3.5). As described, the method requires neither per-subject fine-tuning nor test-time optimization. If it reliably achieves the illustrated results, it would be a useful approach to multi-concept generation; modest novelty alone would not negate this practical contribution.

  2. The three-reference results and ablations provide concrete evidence of effectiveness. Table 1 reports CLIP-T/CLIP-I/DINO-I of 0.3662/0.8531/0.6705, respectively, exceeding the listed baselines in that setting. In Table 2, removing SC-SA reduces DINO-I from 0.6705 to 0.5634, while removing later global text guidance reduces CLIP-T from 0.3662 to 0.3115. Subject-region similarity also reduces confounding by whole-image backgrounds. These results support complementary component contributions, although they do not establish strict identity/structure disentanglement.

Critical weaknesses
  1. The qualitative evidence is too limited to establish applicability, particularly for human subjects. Figure 2 contains only five comparison examples, and Figure 3 contains only two ablation scenarios. The demonstrations predominantly show objects, also include animals, but contain no people. They therefore do not establish preservation of human identity, avoidance of identity blending between people, or reasonable scale and interaction between people and objects. Section 4.2 does not report the category composition or human-subset results, so it is unclear whether this capability was evaluated. Please add more diverse generated examples, including at least two distinct human-containing scenarios, preferably covering a person with objects and two people. Show the references, full prompts, and comparable baseline outputs. This validation matters for broad personalization utility; additional seeds of the same scene would not replace it.

  2. The SigLIP feature-integration path and its rationale are unspecified. Equation (2) uses K/V projections inherited from pretrained MMDiT blocks, while Section 4.1 identifies SigLIP as the reference encoder. The paper does not state which integration implementation is actually used or why it was selected. Please describe the complete path from SigLIP outputs to MR-CA conditioning, including mapping modules, feature dimensions at each stage, projections, output fusion, and pretrained weight sources. Explain how this path supports region-specific reference guidance without additional training and the design rationale relative to alternative integration approaches. Existing pretrained mappings or adapters offer possible implementations; the concern is the undisclosed choice and rationale, which limit reproducibility and assessment of the contribution. Section 3.2 also omits the LLM, prompt template, box format, relative-scale procedure, and manual-editing details; Section 3.4 leaves the two-reference arrangement and positional encoding unspecified.

  3. The scope of the attention operations within FLUX and the directionality of MR-CA are unclear. Section 3.3 only states that MR-CA is applied at selected MMDiT blocks, without identifying whether the operations apply to double-stream blocks, single-stream blocks, or both. The scope of SC-SA is also unclear. Please specify the block types, layer indices, and sampling stages for each mechanism, and explain their selection. Double- and single-stream blocks differ in token organization and projection structure, affecting reference injection, attention masks, and output fusion. A further ambiguity is that Equations (2)–(3) describe only target-image queries against reference keys/values, and Section 3.4 describes MR-CA as directional injection, whereas Figure 1 labels both cross-modal regions between condition and image tokens as MR-CA. Under the usual interpretation of rows as queries and columns as keys, the other region suggests condition-token queries. Please clarify whether this reverse attention is computed, whether condition tokens are updated, and how the diagram corresponds to the equations. These requests concern scope and directionality while recognizing the specified noising, attention interaction, and supporting SC-SA ablation.

Questions for authors
  1. Please add more generated examples, including at least two distinct human-containing scenarios, preferably a person with objects and two people. Show reference images, full prompts, and baseline comparisons. Are human identities preserved, are identities blended between people, and are person–object scale and interactions reasonable? Does the existing benchmark contain people? If so, please report their number and subset results. This is a request for validation relevant to the current assessment, not merely a suggestion for future expansion.

  2. Which LLM and version are used in Section 3.2? Does it receive only the full text prompt, or also the reference images? Please provide the layout-generation prompt, output format, and actual bounding-box examples. How is relative object size determined, and how are relations such as a vase on a table, perspective, and occlusion handled? Is there manual editing, post-generation validation, or retrying, and are the evaluated layouts generated fully automatically?

  3. Equation (6) illustrates only the three-reference 2×2 arrangement. How are two references and the target arranged? Is an empty panel added, and if so, how is it filled, masked, and assigned positional encoding? Please show the actual latent dimensions and layouts for one, two, and three references, and explain the per-step noising or updating of reference latents.

  4. How is positional encoding assigned to reference and target latents in SC-SA? Are the native RoPE coordinates of the concatenated canvas used, or are reference/target coordinates modified?

  5. Which SigLIP feature-integration implementation is actually used in Section 3.3, and why was this path selected? Please identify the SigLIP checkpoint and feature layer, and describe the complete path from encoder outputs to F_ref and K/V in Equation (2) and the conditioning stream in Figure 1, including mapping modules, dimensions at each stage, projection sources, MR-CA output fusion, and pretrained weight sources. How does this path support region-specific reference guidance without additional training, and what is its design rationale relative to alternatives? Identify the reused pretrained modules, frozen parameters, and any parameters additionally trained for this method.

  6. Are MR-CA and SC-SA applied to FLUX's double-stream blocks, single-stream blocks, or both? Which layers are meant by selected MMDiT blocks? Please give the block types and layer indices for each mechanism, state whether the same scope is used in both sampling stages, and explain why these locations were selected. How are reference tokens, masks, and output fusion handled in each block type?

  7. Section 3.3 describes only target-image tokens providing Q and reference features providing K/V, whereas Figure 1 labels both cross-modal regions as MR-CA. Does MR-CA also include reverse attention with condition tokens providing Q? If so, please describe its computation and update procedure; otherwise, clarify the meaning of the other cross-modal region in the figure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.