acceptodds
Under review as a conference paper at ICLR 2027

Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models

Abstract

While many diffusion models perform well when controlling particular aspects such as style, attributes, and interaction, they struggle with fine-grained control due to dataset limitations and intricate architectural design. This paper introduces a novel training-free algorithm for fine-grained generation, called Aggregation of Multiple Diffusion Models (AMDM). The algorithm aggregates intermediate latent states from multiple compatible diffusion models within the same ecosystem into a target model, providing a training-free mechanism to transfer complementary control signals for fine-grained generation. Experimental results show that AMDM can improve fine-grained control under compatible model settings, e.g., raising instance-level attribute success rate from 34.06% to 54.78% on COCO-MIG and HOI detection score from 16.87 to 26.04 on FGAHOI, while preserving generation quality. AMDM provides a practical way to reuse specialized conditional diffusion models for fine-grained generation, reducing the need for complex multi-condition datasets, architectures, and additional training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HfunOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Aggregation of Multiple Diffusion Models (AMDM), an algorithm for improving fine-grained control over generations from diffusion models by combining complementary capabilities of models within the same “ecosystem”. A diffusion model ecosystem roughly translates to models sharing the same latent space and noise schedule. The key idea is to use spherical interpolation to interpolate between latents of different models during diffusion denoising. A deviation correction term is used to bring back the interpolated latent to the manifold of each model. The algorithm is training-free and supports combining more than two models, in principle. Main experimental results are on COCO-MIG and FGAHOI

Critical strengths

(1) AMDM provides a clean strategy to combine the capabilities of complementary models at test-time without additional training; experiments indeed demonstrate improved controllability of generations by combining strengths of two models.

(2) The idea of switching to higher quality model sampling after a fixed number of sampling steps is quite intuitive; the intuition that initial denoising steps dictate “broader” aspects of the image while later steps make finer adjustments is known in literature, and it has been effectively used in tandem with the AMDM algorithm

Critical weaknesses

(1) My biggest concern/ question is regarding the choice of two independent Gaussian samples as initial noise for the two models. Most of the developed machinery (the spherical interpolation, deviation correction) seems to stem from this assumption. However, is it not possible for all models to start from the same initial noise? I believe simple linear interpolation would work well in this case; with a shared trajectory, the method essentially reduces to averaging the noise predictions of the models.Could the authors clarify why this might not work/ justification for why they assume independent starting noise for each model?

(2) The compositional baselines (projective composition, FKC, SuperDiff) combine both models at every sampling step, whereas AMDM aggregates only in the first s steps and then switches to the target model. Since Figure 5a shows that aggregating in later steps significantly degrades both control and FID, it is unclear whether the gains come from the aggregation operator itself or simply from the early-window schedule. Simpler baselines with the same schedule would help, e.g., linearly averaging the noise predictions of both models in the first s steps, or simply sampling from MIGC for the first s steps and then switching to InteractDiffusion.

Questions for authors

Any clarifications/ experimental results addressing the two weaknesses would be helpful.

3 comments

Thank you for carefully reviewing our paper and recognizing that AMDM combines complementary control capabilities without additional training, as well as the effectiveness of the early aggregation strategy. Your concerns focus on two points, namely the necessity of independent initialization and the aggregation operator, and whether the performance gains mainly come from the early aggregation window. Below, we respond to each point and provide comparisons with shared initialization, first-20-step prediction averaging, and pure switching.

W1. Choice of Independent Initial Noise

The table below compares two settings that start from the same noise with AMDM. Shared initialization with linear aggregation performs on par with first-20-step prediction averaging in W2(a).

MethodInitial noiseISR ↑mIoU ↑
InteractDiffusionSingle model34.0630.40
Shared initialization + linear aggregationShared46.5640.41
First-20-step prediction averagingShared47.0940.75
AMDMIndependent54.7847.74

Shared initialization and first-20-step prediction averaging both essentially average the noise predictions of the two models, which is score composition. Original Section 2 notes that score composition suits mutually independent conditions, whereas layout, attributes, and interactions describe the same scene and depend on each other. Independent initialization gives the two branches richer latent information that is more likely to be complementary, but it also makes their directions nearly orthogonal. In Original Appendix E.2, their angle is about 1.58 rad, close to 90 degrees. Directly averaging two orthogonal vectors shrinks the norm to about 0.71 times, which falls outside the range the models are familiar with from training, so linear aggregation is hard to apply directly. Spherical aggregation preserves the norm, and deviation correction pulls the aggregated result back onto each model's own denoising trajectory. This is why AMDM adopts independent initialization and spherical aggregation. It outperforms the shared-initialization scheme by 8.22/7.33 percentage points, and the final results confirm the effectiveness of this design.

W2. Aggregation Operator versus Early Window

(a) First-20-Step Prediction Averaging

Following your suggested setting, we linearly average the noise predictions of MIGC and InteractDiffusion during the first 20 of 50 DDIM steps and run only InteractDiffusion for the remaining 30 steps, with the same prompts, seeds, and sampling settings as AMDM, as shown below.

MethodCOCO-MIG ISR/mIoU ↑HOI Default/KO ↑
MIGC55.46/48.5316.87/17.84
InteractDiffusion34.06/30.4029.53/30.99
First-20-step prediction averaging47.09/40.7523.12/24.58
AMDM54.78/47.7431.40/32.76

First-20-step prediction averaging is 7.69 and 6.99 percentage points below AMDM on ISR and mIoU, respectively. Its HOI Default also drops from 29.53 for InteractDiffusion to 23.12, whereas AMDM raises it to 31.40. Linear averaging introduces MIGC's attribute control but weakens the target model's original interaction capability, while AMDM preserves both controls.

(b) Pure Switching from MIGC to InteractDiffusion

Following your suggested setting, we first sample with MIGC for 20 steps and then pass the latent unchanged to InteractDiffusion for the remaining 30 steps, without aggregation or deviation correction. Results for MIGC, InteractDiffusion, and AMDM are taken from Original Tables 1 and 2, and pure switching is evaluated under the same COCO-MIG and HOI settings, as shown below.

MethodCOCO-MIG ISR/mIoU ↑HOI Default/KO ↑
MIGC55.46/48.5316.87/17.84
InteractDiffusion34.06/30.4029.53/30.99
Pure switching55.24/48.2619.27/17.19
AMDM54.78/47.7431.40/32.76

The attribute metrics of pure switching are comparable to MIGC and slightly higher than AMDM, showing that early MIGC sampling already transfers layout and attributes. However, its HOI is only 19.27/17.19, close to MIGC and far below InteractDiffusion and AMDM, so it loses most of InteractDiffusion's interaction control. First-20-step prediction averaging and pure switching both use the same early window as AMDM, yet neither preserves both attribute and interaction control, so the early window alone cannot explain AMDM's gains.

Thank you for your valuable suggestions. The above clarifications and controlled experiments respectively examine the roles of the initial noise, the aggregation operator, and the early window, and we will incorporate them into the revision to further improve the paper. We hope these clarifications and additional results address your concerns. Thank you again for your careful review.

I thank the reviewers for the detailed rebuttal as well as the additional experiments. I think the additional experiments contextualize the paper better; I am raising my score accordingly.

Thank you for your reply, and for recognizing our additional experiments and raising your score. We will incorporate these experiments and your suggestions into the revision to further improve the paper.

Thank you again for your careful review.

Reviewer_Fo5dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AMDM, a training-free inference method that aggregates intermediate latent states from conditional diffusion models sharing a compatible latent representation and diffusion process. It combines spherical interpolation with deviation correction during an initial sampling window, followed by target-model denoising, to transfer complementary control capabilities across models.

Critical strengths
  • The paper provides an interesting geometric perspective on model aggregation. The compatibility analysis offers an understandable motivation for combining intermediate states from specialized models without requiring identical denoising architectures. The distinction between shared latent-space compatibility and architectural identity is useful.
  • The empirical results clearly demonstrate complementary control transfer. The qualitative examples and quantitative evaluations show improvements in attribute binding, human–object interactions, and style-related alignment, supported by comparisons with composition baselines and component ablations.
Critical weaknesses
  • The connection between geometric compatibility and improved fine-grained control remains insufficiently established. Section 2.2 and the subsequent propositions primarily motivate the feasibility of aggregation and the preservation of local geometric compatibility. Although Appendices E.2–E.3 provide supporting measurements, they do not directly establish when the proposed operations improve joint condition satisfaction rather than merely keep model trajectories close. A stronger theoretical or controlled empirical connection between the compatibility quantities and the observed control gains would better justify the specific aggregation rule.
  • The comparative evidence beyond the main Stable Diffusion setting remains incomplete. The FLUX-based experiments are a useful extension, but Table 13 primarily compares each target model with its aggregated version on the capability supplied by the auxiliary model. It does not include the auxiliary model alone or competing composition approaches in that setting, nor quantify whether the target’s original strengths are preserved. These comparisons are important for establishing that AMDM provides an advantageous combination of capabilities, rather than mainly transferring a capability already available from another model.
  • The rationale for terminating aggregation early is not fully resolved by the stage ablations. Figure 4 varies aggregation duration up to 50 steps, whereas Figure 5 moves a fixed aggregation window between early, middle, and late stages. Poor performance from late-only aggregation does not establish that continuing aggregation after an early start would cause the same deterioration. A unified comparison of early-only and full-trajectory aggregation, reporting both control metrics and image quality under the same sampling setup, would clarify whether the cutoff primarily protects generation quality or reduces inference cost.
Questions for authors
  • Can the authors connect the measured geometric compatibility quantities to improvements in joint condition satisfaction, including cases where aggregation does not help? In particular, how can one determine whether the functional-proximity assumption is sufficiently strong to predict successful control transfer?
  • For the FLUX experiments, could the authors report both component models and relevant composition baselines under matched conditioning, together with metrics assessing preservation of the target model’s original capability?
  • Could the authors add full-trajectory aggregation to the T=20T=20 setting of Figure 5 and relate the results to the aggregation-length sweep in Figure 4? Does terminating aggregation early improve the control–quality trade-off, or is its main benefit computational?
3 comments

Thank you for carefully reviewing our paper and recognizing the value of our geometric perspective and our experimental results on complementary control transfer. Below, we respond to your concerns point by point.

W1/Q1. Connection between Geometric Compatibility and Control Gains

Geometric compatibility is a prerequisite for control transfer, not a criterion for per-image semantic success. dd is an upper bound on how far the aggregated variable deviates from the auxiliary model's conditional generation domain. Original Proposition 2.2 shows that a smaller dd yields a higher lower bound on the probability that the aggregated variable stays within this domain, so the auxiliary model's control information is more likely to reach the final result. Original Appendices E.2 and E.3 show that after the first aggregation, the angle drops to between 0 and 0.0312 rad and the latent distance stays between 1.31 and 2.82, supporting a small and stable dd. Original Tables 1 and 2 directly verify the final control gains, as shown below.

COCO-MIG (800 images, 3200 instances)InteractDiffusionAMDMGain
ISR34.06%54.78%+20.72 pp
Correct instances10901753+663

We computed the Spearman ρ\rho between per-image ISR and both the angle and the normalized distance on 800 images regenerated with seed 42, and both values are −0.002-0.002. This indicates that, within the compatible regime, small geometric fluctuations do not predict whether an individual image succeeds. Original Appendix H.9 examines the 47 outputs in which MIGC fails to satisfy all requested controls. Relative to InteractDiffusion, AMDM improves 23 of them, leaves 22 unchanged, and degrades 2. Original Table 10 shows that when the two models no longer share the diffusion process, AMDM's ISR drops from the target model's 34.06% to 18.13%. Geometric analysis therefore determines when aggregation is possible, while task metrics verify the resulting control.

W2/Q2. Comparisons beyond Stable Diffusion: FLUX Components and Capability Preservation

Original Appendix H.6/Table 13 compares each target model with its AMDM version under identical prompts, seeds, and T=28T=28 sampling. Under the same setting, we further report both component models, a relevant composition baseline, and AMDM, evaluating both the transfer of the auxiliary capability and the preservation of the target model's original capability, as shown below.

FLUX pairTypeMethodISR/mIoU ↑HOI Default/KO ↑
OneHOI + CreatiLayoutComponentOneHOI35.4/38.729.7/31.1
ComponentCreatiLayout62.0/56.05.0/5.8
Composition baselineProjective Composition52.8/49.425.2/26.9
AMDMOneHOI(+CreatiLayout)59.8/54.529.0/30.5
Regional Prompting + OneHOIComponentRegional Prompting50.0/48.54.6/5.3
ComponentOneHOI35.4/38.729.7/31.1
Composition baselineProjective Composition42.2/42.819.1/21.5
AMDMRegional Prompting(+OneHOI)49.2/47.822.9/25.1

Both groups show that AMDM acquires the auxiliary control while preserving the target model's original capability. OneHOI(+CreatiLayout) raises ISR/mIoU from 35.4/38.7 to 59.8/54.5, while its HOI changes only from 29.7/31.1 to 29.0/30.5. Regional Prompting(+OneHOI) raises HOI from 4.6/5.3 to 22.9/25.1, while its ISR/mIoU changes only from 50.0/48.5 to 49.2/47.8. Both AMDM results also outperform the corresponding Projective Composition. Each component model provides only one of the two capabilities, and AMDM is the only configuration that provides both.

W3/Q3. Rationale for Early Termination of Aggregation

Stopping aggregation early mainly reduces computation. Original Figure 4 shows that control metrics keep improving as the number of spherical aggregation steps increases and saturate after 20 steps. Original Figure 5 shows that, under the same 10-step aggregation budget, the early window outperforms the middle and late windows. Using the same prompts, seeds, and T=20T=20 sampling as Figure 5, we further compare early aggregation with full-trajectory aggregation directly, as shown below.

SettingAggregation stepsISR ↑HOI Default ↑FID ↓Time (s) ↓
Early aggregation1052.2730.0519.8610.40
Full-trajectory aggregation2053.5030.8019.4011.80

Full-trajectory aggregation is therefore slightly better in control and quality, while early stopping retains most of the gains at a lower computational cost.

Thank you for your valuable suggestions. We will incorporate the above additional experiments and clarifications into the revision to further improve the paper. We hope these clarifications and additional results address your concerns. Thank you again for your careful review.

I thank the authors for their thoughtful response. While W1 remains partially unresolved, the additional results sufficiently address W2 and W3. I appreciate these clarifications and will raise my rating from 2 to 3.

Thank you for your reply, and for recognizing our additional experiments and raising your score. We will incorporate these experiments and your suggestions into the revision to further improve our paper.

Thank you again for your careful review.

Reviewer_9ZA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMDM, a training-free approach for improving fine-grained image generation by aggregating complementary capabilities from multiple diffusion models. Its main contributions are spherical latent aggregation and deviation optimization, which integrate control signals while preserving generation quality. Experiments on attribute binding, human-object interaction, and style control demonstrate its effectiveness. AMDM improves instance success rate from 34.06% to 54.78% and HOI detection score from 16.87 to 26.04, outperforming existing diffusion composition methods. Ablation studies further validate the proposed components.

Critical strengths
  • Interesting and practical idea. The proposed training-free aggregation algorithm is interesting and well-motivated, as the need for effective aggregation is evident.

  • Theoretically grounded methodology. The proposed method is derived from theoretical analysis under explicit assumptions, which aligns well with the scope of ICLR.

Critical weaknesses

W1. The theoretical motivation is not fully convincing. The paper provides extensive geometric analysis to motivate latent aggregation, but the connection between geometric properties and improved controllability remains unclear. In particular, sharing a compatible latent space and preserving its geometric structure do not necessarily guarantee that interpolated latent states retain the control capabilities of both models. Overall, the analysis appears to justify the proposed operations rather than explain why they improve fine-grained controllability.

W2. Limited evaluation on contemporary generative models. The main experiments rely on relatively old Stable Diffusion-based models, including MIGC, InteractDiffusion, and IP-Adapter. Although additional FLUX-based experiments are provided, the evaluation remains limited to a few model families and control settings. Given recent advances in image generation, it is unclear whether AMDM offers meaningful advantages over stronger contemporary single-model approaches or generalizes to other modern architectures.

W3. Incomplete evaluation of joint controllability. The paper primarily evaluates attribute binding, human-object interactions, and style alignment separately, rather than measuring whether multiple conditions are simultaneously satisfied. In particular, COCO-MIG does not evaluate meaningful interaction control, while the three-model evaluation does not provide reference images to IP-Adapter. This weakens the evidence supporting the central claim of effective multi-condition aggregation.

W4. Insufficient isolation of the proposed method's contributions. AMDM aggregates models only during early denoising steps, whereas the main composition baselines combine models throughout sampling. Without comparing against early-stage-only variants of these baselines, it is difficult to distinguish the benefits of geometric aggregation from those of the sampling strategy. Furthermore, BoxDiff achieves a relatively competitive instance success rate (52.00% vs. 54.78%), suggesting that the advantage over simpler control methods may be smaller than the main comparisons indicate.

Questions for authors

Q1: Can the authors quantify the relationship between latent-space compatibility and control performance? Specifically, using the existing MIGC–InteractDiffusion experiments, please report whether smaller angular deviations or latent distances correlate with higher instance success rates. This would help establish whether the proposed geometric analysis explains the observed improvements.

Q2: Can the authors evaluate the three-model aggregation on a subset of 100–200 prompts containing simultaneous attribute, interaction, and reference-style constraints? Please report the percentage of images satisfying all three conditions, alongside individual success rates. The evaluation should include actual reference images for IP-Adapter and compare single-, two-, and three-model configurations.

Q3: Can the authors compare AMDM with an early-stage-only score composition baseline, where the model predictions are combined during the first 20 of 50 denoising steps, followed by target-only sampling? Using the same prompts, seeds, and sampling schedule would help clarify whether the improvements primarily come from spherical latent aggregation or the early-stage composition strategy.

Q4: Can the authors evaluate AMDM against at least one stronger contemporary single-model generator on the same COCO-MIG and HOI benchmarks, ensuring comparable conditioning inputs? Additionally, can they clarify whether the proposed aggregation is expected to work with pure single-stream diffusion transformers, and what architectural or sampling constraints would prevent its application?

3 comments

Thank you for carefully reviewing our paper and recognizing AMDM's practical value and theoretical grounding. Below, we respond to your concerns point by point.

W1/Q1. Theoretical Motivation: Geometric Compatibility and Control Performance

Fine-grained generation motivates aggregating models with complementary controls, while geometric analysis motivates how to aggregate them. Geometric compatibility is a condition for aggregation rather than a sufficient condition for semantic control success, so we do not treat smaller angular deviations or latent distances as monotonic predictors of per-image ISR. Results on 800 COCO-MIG images confirm this, as the Spearman ρ\rho between either quantity and per-image ISR is −0.002-0.002. The geometric analysis shows how to aggregate, and Original Section 2.2, Propositions 2.1 and 2.2, and Appendix D provide the analysis of the compatibility conditions, local spherical approximation, and deviation correction for this purpose. Original Appendices E.2 and E.3 show that, after the first two steps, φ\varphi ranges from 0 to 0.0312 rad, δ\delta ranges from 1.31 to 2.82, and the norm difference is at most 0.125 during aggregation. Original Tables 1 and 2 independently verify the ISR, mIoU, and HOI gains.

W2/Q4. Evaluation on Contemporary Generative Models

AMDM is an aggregation algorithm, not a single generative model. Original Appendix H.3/Table 9 and Appendix H.6/Table 13 apply AMDM to SDXL and FLUX-based DiT/flow-matching models, respectively. The FLUX comparison uses identical benchmark inputs, prompts, seeds, and T=28T=28 sampling. We also report Qwen-Image-2512 on 600 COCO-MIG requests as a text-conditioned single-model reference, as shown below.

Setting/metricTarget onlyAfter AMDM aggregation
SDXL: InteractDiffusion / +IP-Adapter, Style CLIP0.5450.597
FLUX: OneHOI / +CreatiLayout, ISR/mIoU35.37/38.7159.76/54.46
FLUX: Regional Prompting / +OneHOI, HOI Default/KO4.57/5.3222.91/25.05
Qwen ISR/mIoU7.44/20.79--

For pure single-stream DiTs, AMDM can operate directly on the sampling process as long as the models have compatible latents, timesteps and noise levels, solver updates, and prediction parameterizations, and expose intermediate sampling states.

W3/Q2. Evaluation of Joint Controllability

No unified benchmark currently annotates instance attributes, human-object interactions, and reference styles together. The three existing evaluations use different data and metrics, and neither HOI nor reference style has a unified per-image success threshold, so a joint accuracy based on manually chosen thresholds would not be reliable. Original Appendix H.5/Table 12 separately compares single-, two-, and three-model configurations using ISR, mIoU, HOI, and Style CLIP on the available benchmarks. We further construct a simple joint setting, shown in the new Figure 10 highlighted in red in revised Appendix H.5. Four prompt groups simultaneously include attribute, interaction, and reference-style constraints, and we use real reference images to show the joint generation of InteractDiffusion(+MIGC,+IP-Adapter).

W4/Q3. Isolating the Contributions of AMDM

Original Figure 4 shows that spherical aggregation improves with more aggregation steps and saturates after 20 steps. Original Figure 5 shows that early aggregation is most effective, so the default 20 steps balance performance and computation. Following your suggestion, we test linearly averaging the two model predictions for the first 20 steps, followed by 30 steps using only InteractDiffusion, as shown below.

MethodMIG ISR/mIoU ↑HOI Default/KO ↑
MIGC55.46/48.5316.87/17.84
InteractDiffusion34.06/30.4029.53/30.99
First-20-step prediction averaging47.09/40.7523.12/24.58
AMDM54.78/47.7431.40/32.76

With identical prompts, seeds, and 50-step DDIM sampling, first-20-step prediction averaging is 7.69 and 6.99 percentage points below AMDM in original Table 2 on ISR and mIoU, respectively. This shows that AMDM's aggregation rule provides additional gains under the same early window. Original Table 16 further compares BoxDiff under the same setting, as shown below.

MethodISR ↑mIoU ↑Time ↓
BoxDiff52.0044.3716.04 s
AMDM54.7847.7413.52 s

AMDM achieves higher ISR and mIoU than BoxDiff at lower latency, and it extends to multiple models that aggregate attribute, interaction, and style controls.

Thank you for your valuable suggestions. We will incorporate the above additional experiments and clarifications into the revision to further improve the paper. We hope these clarifications and additional results address your concerns. Thank you again for your careful review.

Thank you for the detailed rebuttal. The additional FLUX experiments and early-stage baseline address several of my concerns and strengthen the empirical evidence.

However, the theoretical justification and quantitative evaluation of joint controllability remain insufficient. Overall, my assessment is now closer to borderline, but I will maintain my Weak Reject rating.

Thank you for your reply and support, and for recognizing our additional FLUX experiments and early-stage baseline. We will incorporate your suggestions on the theoretical justification and the quantitative evaluation of joint controllability into the revision to further improve the paper.

Thank you again for your careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.