On Complementarity in Diffusion Transformers

Diffusion Transformers (DiTs) have achieved remarkable success in high-quality image generation by stacking multiple homogeneous self-attention layers. However, this homogeneous design may lead to functional redundancy, as different layers and attention heads can learn highly similar transformations. In this work, we investigate complementarity as a structural principle for reducing redundancy in DiTs, encouraging different network components to perform distinct yet cooperative computations. Specifically, we address three key questions: (1) what forms of complementarity can be introduced, (2) where complementarity can be instantiated within a transformer, and (3) how different structural designs affect its effectiveness. First, we explore two representative forms of functional complementarity: rotation complementarity, which introduces distinct rotation patterns across layers, and scale complementarity, which employs different convolutional receptive fields. Second, we implement complementarity from layers to individual attention heads, demonstrating its applicability at different architectural granularities. Third, we systematically examine different symmetry, grouping, and ordering designs, showing that the benefits of complementarity persist across diverse configurations. Experiments demonstrate that incorporating complementarity maintains comparable image generation quality while improving network efficiency. We hope this study can serve as a starting point for exploring underexamined structural properties of DiTs and inspire the development of more efficient and diverse architectures.