acceptodds
Withdrawn from ICLR 2027

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion-based image generation. Unlike prior approaches that train separate adapters for each condition, UNITY jointly learns shared semantics across multiple conditioning modalities and later specializes without modifying the architecture. The proposed two-stage training paradigm consists of a Universal Stage, which captures cross-modal joint representations across all conditioning types using half of the total training steps, followed by a Specialization Stage that refines modality-specific features with the remaining steps. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel-aware and spatially adaptive feature alignment through learnable flow fields and attention fusion. This constant-complexity formulation allows flexible operation under single or composite conditioning while significantly reducing inference latency and memory footprint. Extensive experiments across multiple datasets demonstrate that UNITY achieves state-of-the-art image fidelity and memory efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Controllable Image Generation, Adapter-Based Conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_FuZMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Many existing image-conditioned image generation methods treat conditioning modality as isolated components, overlooking the relationships among different conditions and consequently causing training cost and memory consumption to scale linearly with the number of conditions. To address this problem, this paper proposes UNITY, which divides the training process into two stages: a Universal Stage to capture cross-modal joint representations and a Specialization Stage to refine modality-specific features. The paper also designs a Morphable Attention Flow Network for condition injection. Experiments involving four conditioning modalities are conducted on SD1.5 and SDXL, where UNITY achieves the best FID and CLIP Score results.

Critical strengths
  1. The motivation of this work is well justified. Joint learning across multiple conditions can avoid the repeated computational costs incurred by training a separate model for each condition.
  2. The method is evaluated on both SD1.5 and SDXL across multiple conditioning modalities. The consistent improvements demonstrate its generalizability across different diffusion backbones.
Critical weaknesses
  1. The presentation of the proposed method is somewhat confusing and difficult to follow. In particular, the terminology related to “MorphWrapper,” “morphological wrapping,” and “warping” is not used consistently, which further affects the clarity of the method description.
  2. The memory column in Table 1 appears to report the number of 24GB GPUs rather than the measured memory consumption. Aggregating device capacities is not a reliable measure of training memory. The paper should report the peak allocated memory for each method instead.
  3. FID and text-image CLIPScore do not directly measure whether the generated image adheres to the given spatial condition. Image-image CLIP similarity and SSIM may be reported as complementary metrics where appropriate.
  4. Tables 3 and 4 report ablation results only for Sketch and Segmentation. Since the main experiments consider four conditioning modalities, the ablation results for Canny and Depth should also be provided to demonstrate that the conclusions generalize across all evaluated conditions.
  5. UNITYIndUNITY_{Ind} and UNITYPreUNITY_{Pre} first appear in Tables 1 and 2, whereas their precise definitions are provided only in the appendix. These two settings should be introduced in the main text when they are first used.
  6. The equal division between the Universal Stage and the Specialization Stage appears to be heuristic. The paper would benefit from an ablation study using different stage ratios to justify the current design.
  7. The current ablation study does not clearly isolate the contributions of the Universal Stage, the Specialization Stage, and the MAF Network. A more direct ablation would be: (1) Universal Stage only with ControlNet, (2) Universal Stage and Specialization Stage with ControlNet, and (3) Universal Stage and Specialization Stage with the proposed MAF Network. Such a comparison, under the same total training budget, would more clearly demonstrate the individual contributions of the two-stage training strategy and the proposed MAF architecture.
Questions for authors

The training-cost analysis assumes a cost of KNKN for baselines supporting KK conditions. However, this assumption may not apply to unified methods such as CtrLoRA and Uni-ControlNet. A unified baseline can be trained jointly with a cost of approximately NN, whereas UNITY requires N(K+1)2\frac{N(K+1)}{2} training steps. When K>1K>1, UNITY seems to incur a higher computational cost. Could the authors clarify the training costs of CtrLoRA, Uni-ControlNet, and the other baselines?

Reviewer_Qx2eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

UNITY introduces a diffusion adapter that uses learned spatial warping and attention to align conditioning features. It jointly pretrains on multiple control modalities before fine-tuning separate modality-specific adapters. Experiments on SD1.5 and SDXL report improved image quality and a 37.5% reduction in total training steps across four modalities.

Critical strengths
  • The paper is well structured, with a clear presentation of related work, the proposed method, and extensive experiments.
  • The experimental evaluation covers several established baselines and provides useful evidence of the method’s effectiveness, although the comparisons mainly involve older approaches.
Critical weaknesses
  • Limited comparison with recent methods. The baselines in Tables 1 and 2 were published in 2024 or earlier. Comparisons with more recent approaches are needed to establish the method’s competitiveness against the current state of the art.
  • Insufficient context for Figure 2. The quality–latency comparison does not adequately contextualize parameter counts, backbone architectures, and inference settings. Without this information, it is difficult to determine whether the reported trade-offs reflect the proposed method or differences in model configurations, making the figure potentially misleading.
  • Insufficient justification of the design. The methodological presentation primarily consists of equations describing individual network blocks. It provides limited conceptual or theoretical justification for why these particular components and their arrangement should improve conditioning. A more principled explanation would strengthen the contribution.
  • Limited evidence of scalability. The evaluation is restricted to SD1.5 and SDXL. Experiments on more recent, larger diffusion backbones, particularly DiT-based models, would help establish whether the method’s effectiveness and efficiency generalize beyond the evaluated architectures.
  • Unclear methodological novelty. The contribution appears to rely heavily on combining and modifying existing architectural components. The paper should more clearly distinguish its conceptual advances from prior adapter-based approaches, including IP-Adapter, while accounting for differences in their conditioning tasks. As presented, it is difficult to assess whether the method introduces a substantially new conditioning mechanism or primarily an incremental architectural refinement.

To the authors: I used an AI tool to help analyze the manuscript and improve my understanding. I formulated my own human-based concerns and used the tool to polish their wording. I carefully reviewed the revised text to ensure that it accurately reflected my original concerns.

Questions for authors

see weakness

Reviewer_D1agOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents UNITY, a conditioning adapter for controllable image generation with frozen Stable Diffusion backbones. The proposed training strategy first learns shared representations from multiple conditioning modalities and then initializes modality-specific fine-tuning from the resulting universal adapter. Architecturally, UNITY uses dual spatial encoders, a Morphable Attention Flow network, and attention-weighted spatial sampling to perform cross-feature alignment, self-refinement, and fusion. The resulting features are conditioned on text embeddings and injected into the diffusion UNet through additive fusion. Experiments on SD1.5 and SDXL report improvements in FID and CLIP scores for Canny, depth, sketch, and segmentation conditioning, with additional experiments on four other conditioning modalities. The paper also claims reduced training cost and resource requirements, together with flexible composite conditioning at constant complexity.

Critical strengths
  • Training separate adapters for different conditioning modalities introduces repeated optimization and requires maintaining multiple models. Learning a shared initialization from related conditions, followed by specialization for individual tasks, is a reasonable and practically useful direction. The proposed training strategy offers a plausible way to exploit common structure across conditioning modalities and reduce repeated training effort.
  • UNITYPreUNITY_{Pre} consistently achieves strong results across the evaluated conditioning modalities on both SD 1.5 and SDXL. Its improvements over UNITYIndUNITY_{Ind} provide useful evidence for the benefit of universal pretraining. The component ablations help assess the contributions of the proposed architecture, while Appendix A.1 provides reasonably clear descriptions of the training configurations and evaluation settings.
  • The experiments on Normal Map, OpenPose, Color Palette, and Dehaze show that the architecture can be applied to another set of conditioning tasks. Since these experiments involve a new universal training phase, they establish applicability to a different set of conditions rather than zero-shot generalization to unseen modalities. The authors acknowledge this limitation in Appendix A.3 and also disclose that deploying all specialized conditions requires retaining their corresponding fine-tuned weights. These disclosures help clarify the scope of the demonstrated capabilities.
Critical weaknesses
  • My main concern is the connection between the proposed training procedure and simultaneous conditioning at inference. Figure 1 presents generation with four visual conditions, but Algorithm 1 produces separate specialized parameters $\phi_i$ and concludes with ($\phi \leftarrow \bigcup_i \phi_i$), without explaining what this operation means computationally. Appendix A.1 further states that universal training uses a 12-channel input, whereas specialization reinitializes the first stage of each spatial encoder for a 3-channel input. Appendix A.3 then notes that the specialized weights must be retained for deployment. These descriptions leave it unclear which checkpoint generates the composite results in Figure 1 and how multiple conditions are processed together after specialization. Does inference use the universal checkpoint, combine several specialized adapters, or apply an additional merging procedure? Shared pretraining remains useful even if separate checkpoints are needed, but the paper should clearly distinguish its benefits for training from its claims about unified inference.
  • Tables 1, 2, and 5 evaluate the conditioning modalities separately. These results support improvements across several individual conditioning tasks, but they do not establish that a single generated image simultaneously follows multiple visual conditions. This distinction is particularly important because composite conditioning is central to the motivation and presentation of the paper, including Figure 1. Moreover, FID and CLIP Score do not directly measure adherence to each supplied spatial condition. The qualitative examples alone are insufficient to establish the claimed benefit of cross-condition spatial alignment. A quantitative evaluation of joint conditioning, together with condition-specific fidelity measurements, is needed to connect the experimental evidence to these claims.
  • Figure 2 shows that T2I-Adapter has lower inference latency, while UNITY achieves better FID. This is a quality–efficiency tradeoff, and the discussion should make that tradeoff explicit. Although Appendix A.1 specifies the general image-generation settings, the timing experiment should identify the hardware, inference batch size, numerical precision, and whether the same sampling settings are used for every method. It should also state whether the reported time covers the complete generation pipeline or only selected components. These details are necessary to interpret and reproduce the comparison.
  • UniControl (UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild) is a particularly relevant missing baseline because it learns across multiple conditioning tasks and introduces MultiGen-20M, which is used in UNITY’s experiments. Its inclusion would help assess the benefits of the proposed shared training strategy relative to an established unified model. The presence of Uni-ControlNet and UniCon does not address this omission, since they are different methods. More recent comparisons could also strengthen the paper’s positioning. For example, Nexus Adapters, published at WACV 2026 and already cited in the manuscript, are relevant to text-guided structural conditioning and adapter efficiency. UniCombine, published at ICCV 2025, is relevant to the composite-conditioning claim, although its diffusion-transformer backbone requires careful treatment when comparing it with UNITY’s UNet-based implementations.You should select more baselines for comparison in the relevant studies from 2025–2026; the current studies are all limited to 2023–2024.
  • Qin et al. (2023a) and Qin et al. (2023b) on page 12 refer to the same UniControl paper, with the same title, authors, venue, and DOI. The manuscript uses one entry when discussing UniControl in Related Work and the other when introducing MultiGen-20M in the experimental setup. These entries should be consolidated and the corresponding in-text citations unified.
Questions for authors
  • What is the exact inference mechanism for composite conditioning?

Please prioritize clarifying how multiple conditions are processed simultaneously after the specialization stage. What does (ϕ←⋃iϕi)(\phi \leftarrow \bigcup_i \phi_i) mean in Algorithm 1: a collection of independent checkpoints, a merged parameter set, or another construction? Which checkpoint and input configuration produce Figure 1? If multiple specialized adapters are active, how are their outputs combined, and what are the resulting parameter and memory requirements? A short inference algorithm showing how the 12-channel universal model relates to the 3-channel specialized models would resolve much of this ambiguity.

  • Can the comparison be strengthened with additional relevant methods, including work from 2025 and 2026?

In particular, could the authors compare against UniControl and suitable recent approaches that address shared training, structural conditioning, or composite control? Please explain the inclusion or exclusion of relevant methods such as Nexus Adapters and UniCombine. Comparisons should account for differences in backbone, supported conditions, training data, and computational budget. Where a controlled numerical comparison is impractical, a precise discussion of these differences would still improve the paper’s positioning.

  • Can the authors provide direct quantitative evidence of simultaneous adherence to multiple conditions?

Please evaluate representative combinations of two and four visual conditions supplied together to generate the same image. In addition to FID and CLIP Score, report fidelity to each supplied condition using appropriate measurements, such as edge agreement, depth consistency, or segmentation consistency. A comparison against suitable compositional baselines, together with an ablation of the proposed alignment mechanism under the same joint-conditioning setting, would help establish whether MAF improves simultaneous control and cross-condition spatial alignment.

  • What is the specific contribution beyond existing deformable attention and shared training approaches?

Which aspects of the Morph Wrapper introduce functionality beyond existing mechanisms based on learned sampling offsets and attention weights? Is the main contribution the operator itself, its use for condition fusion, or the arrangement of cross-modal alignment and self-refinement? Similarly, how does universal training followed by specialization differ from the shared learning already present in CtrLoRA and UniControl? In particular, the Introduction, lines 66–68, characterizes CtrLoRA as treating conditioning modalities independently, whereas its published formulation explicitly learns a shared Base ControlNet across multiple conditions alongside condition-specific LoRAs. Please clarify this distinction and, where feasible, provide a controlled comparison that isolates the benefit of UNITY’s particular design.

Reviewer_u162OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UNITY, a Universal-to-Specialized adapter for diffusion models to perform conditioned image generation. A universal stage jointly processes multiple conditioning modalities, after which separate specialization runs adapt the shared initialization to individual modalities. The architecture combines dual spatial encoders, multi-scale flow estimators, and attention-weighted feature warping through a Morphable Attention Flow network, followed by text-conditioned feature injection into a frozen diffusion backbone. Experiments on SD1.5 and SDXL report improved FID and CLIP scores for canny, depth, sketch, and segmentation conditioning. The paper also claims efficient composite conditioning and reduced training cost through shared pretraining.

Critical strengths
  1. Learning a shared initialization before modality-specific specialization addresses a practically relevant problem. The architecture combines cross-modal alignment, self-refinement, and text guidance while keeping the diffusion backbone frozen. Algorithm 1 also demonstrates a reduction in the total number of training steps relative to training each modality independently.
  2. The experiments cover two backbones and multiple conditioning modalities. In Tables 1, 2, and 5, UNITY_Pre consistently achieves lower FID than the corresponding UNITY_Ind configuration, supporting the effectiveness of universal pretraining.
Critical weaknesses
  1. The "constant-complexity" claim appears inconsistent with the described training procedure. Algorithm 1 gives a total of N(1+K)/2 steps, which still grows linearly with the number of modalities K. For K=4, the reported 37.5% saving is a reduction relative to independent training, which does not establish that total training cost remains constant as the number of modalities increases.

  2. The novelty of the Morph Wrapper relative to existing deformable operators is not sufficiently clear. Equations 10–13 predict spatial offsets and attention weights, perform bilinear sampling, and aggregate the sampled features. This mechanism appears closely related to deformable convolution and deformable attention: learned-offset sampling is central to deformable convolution, while deformable attention combines learned offsets with normalized attention weights.

  3. Tables 1, 2, and 5 primarily report single-condition results. This paper lacks a clear quantitative evaluation of generation with multiple conditions supplied jointly at inference time.

Questions for authors
  1. Could the proposed framework support multiple conditioning inputs simultaneously at inference time?
  2. Could the MSFE and Morph Wrapper modules be replaced with deformable convolution or deformable attention? What are the key differences between the proposed combination and these alternatives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.