Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models

While many diffusion models perform well when controlling particular aspects such as style, attributes, and interaction, they struggle with fine-grained control due to dataset limitations and intricate architectural design. This paper introduces a novel training-free algorithm for fine-grained generation, called Aggregation of Multiple Diffusion Models (AMDM). The algorithm aggregates intermediate latent states from multiple compatible diffusion models within the same ecosystem into a target model, providing a training-free mechanism to transfer complementary control signals for fine-grained generation. Experimental results show that AMDM can improve fine-grained control under compatible model settings, e.g., raising instance-level attribute success rate from 34.06% to 54.78% on COCO-MIG and HOI detection score from 16.87 to 26.04 on FGAHOI, while preserving generation quality. AMDM provides a practical way to reuse specialized conditional diffusion models for fine-grained generation, reducing the need for complex multi-condition datasets, architectures, and additional training.