- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Aggregation of Multiple Diffusion Models (AMDM), an algorithm for improving fine-grained control over generations from diffusion models by combining complementary capabilities of models within the same “ecosystem”. A diffusion model ecosystem roughly translates to models sharing the same latent space and noise schedule. The key idea is to use spherical interpolation to interpolate between latents of different models during diffusion denoising. A deviation correction term is used to bring back the interpolated latent to the manifold of each model. The algorithm is training-free and supports combining more than two models, in principle. Main experimental results are on COCO-MIG and FGAHOI
Critical strengths
(1) AMDM provides a clean strategy to combine the capabilities of complementary models at test-time without additional training; experiments indeed demonstrate improved controllability of generations by combining strengths of two models.
(2) The idea of switching to higher quality model sampling after a fixed number of sampling steps is quite intuitive; the intuition that initial denoising steps dictate “broader” aspects of the image while later steps make finer adjustments is known in literature, and it has been effectively used in tandem with the AMDM algorithm
Critical weaknesses
(1) My biggest concern/ question is regarding the choice of two independent Gaussian samples as initial noise for the two models. Most of the developed machinery (the spherical interpolation, deviation correction) seems to stem from this assumption. However, is it not possible for all models to start from the same initial noise? I believe simple linear interpolation would work well in this case; with a shared trajectory, the method essentially reduces to averaging the noise predictions of the models.Could the authors clarify why this might not work/ justification for why they assume independent starting noise for each model?
(2) The compositional baselines (projective composition, FKC, SuperDiff) combine both models at every sampling step, whereas AMDM aggregates only in the first s steps and then switches to the target model. Since Figure 5a shows that aggregating in later steps significantly degrades both control and FID, it is unclear whether the gains come from the aggregation operator itself or simply from the early-window schedule. Simpler baselines with the same schedule would help, e.g., linearly averaging the noise predictions of both models in the first s steps, or simply sampling from MIGC for the first s steps and then switching to InteractDiffusion.
Questions for authors
Any clarifications/ experimental results addressing the two weaknesses would be helpful.
3 comments
I thank the reviewers for the detailed rebuttal as well as the additional experiments. I think the additional experiments contextualize the paper better; I am raising my score accordingly.
Thank you for your reply, and for recognizing our additional experiments and raising your score. We will incorporate these experiments and your suggestions into the revision to further improve the paper.
Thank you again for your careful review.
Thank you for carefully reviewing our paper and recognizing that AMDM combines complementary control capabilities without additional training, as well as the effectiveness of the early aggregation strategy. Your concerns focus on two points, namely the necessity of independent initialization and the aggregation operator, and whether the performance gains mainly come from the early aggregation window. Below, we respond to each point and provide comparisons with shared initialization, first-20-step prediction averaging, and pure switching.
W1. Choice of Independent Initial Noise
The table below compares two settings that start from the same noise with AMDM. Shared initialization with linear aggregation performs on par with first-20-step prediction averaging in W2(a).
Shared initialization and first-20-step prediction averaging both essentially average the noise predictions of the two models, which is score composition. Original Section 2 notes that score composition suits mutually independent conditions, whereas layout, attributes, and interactions describe the same scene and depend on each other. Independent initialization gives the two branches richer latent information that is more likely to be complementary, but it also makes their directions nearly orthogonal. In Original Appendix E.2, their angle is about 1.58 rad, close to 90 degrees. Directly averaging two orthogonal vectors shrinks the norm to about 0.71 times, which falls outside the range the models are familiar with from training, so linear aggregation is hard to apply directly. Spherical aggregation preserves the norm, and deviation correction pulls the aggregated result back onto each model's own denoising trajectory. This is why AMDM adopts independent initialization and spherical aggregation. It outperforms the shared-initialization scheme by 8.22/7.33 percentage points, and the final results confirm the effectiveness of this design.
W2. Aggregation Operator versus Early Window
(a) First-20-Step Prediction Averaging
Following your suggested setting, we linearly average the noise predictions of MIGC and InteractDiffusion during the first 20 of 50 DDIM steps and run only InteractDiffusion for the remaining 30 steps, with the same prompts, seeds, and sampling settings as AMDM, as shown below.
First-20-step prediction averaging is 7.69 and 6.99 percentage points below AMDM on ISR and mIoU, respectively. Its HOI Default also drops from 29.53 for InteractDiffusion to 23.12, whereas AMDM raises it to 31.40. Linear averaging introduces MIGC's attribute control but weakens the target model's original interaction capability, while AMDM preserves both controls.
(b) Pure Switching from MIGC to InteractDiffusion
Following your suggested setting, we first sample with MIGC for 20 steps and then pass the latent unchanged to InteractDiffusion for the remaining 30 steps, without aggregation or deviation correction. Results for MIGC, InteractDiffusion, and AMDM are taken from Original Tables 1 and 2, and pure switching is evaluated under the same COCO-MIG and HOI settings, as shown below.
The attribute metrics of pure switching are comparable to MIGC and slightly higher than AMDM, showing that early MIGC sampling already transfers layout and attributes. However, its HOI is only 19.27/17.19, close to MIGC and far below InteractDiffusion and AMDM, so it loses most of InteractDiffusion's interaction control. First-20-step prediction averaging and pure switching both use the same early window as AMDM, yet neither preserves both attribute and interaction control, so the early window alone cannot explain AMDM's gains.
Thank you for your valuable suggestions. The above clarifications and controlled experiments respectively examine the roles of the initial noise, the aggregation operator, and the early window, and we will incorporate them into the revision to further improve the paper. We hope these clarifications and additional results address your concerns. Thank you again for your careful review.