acceptodds
Under review as a conference paper at ICLR 2027

On Complementarity in Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have achieved remarkable success in high-quality image generation by stacking multiple homogeneous self-attention layers. However, this homogeneous design may lead to functional redundancy, as different layers and attention heads can learn highly similar transformations. In this work, we investigate complementarity as a structural principle for reducing redundancy in DiTs, encouraging different network components to perform distinct yet cooperative computations. Specifically, we address three key questions: (1) what forms of complementarity can be introduced, (2) where complementarity can be instantiated within a transformer, and (3) how different structural designs affect its effectiveness. First, we explore two representative forms of functional complementarity: rotation complementarity, which introduces distinct rotation patterns across layers, and scale complementarity, which employs different convolutional receptive fields. Second, we implement complementarity from layers to individual attention heads, demonstrating its applicability at different architectural granularities. Third, we systematically examine different symmetry, grouping, and ordering designs, showing that the benefits of complementarity persist across diverse configurations. Experiments demonstrate that incorporating complementarity maintains comparable image generation quality while improving network efficiency. We hope this study can serve as a starting point for exploring underexamined structural properties of DiTs and inspire the development of more efficient and diverse architectures.

Primary area:
generative models
Keywords:
Diffusion Transformers,Structural Complementarity,Architectural Redundancy,Attention Head Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_T5BaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies functional redundancy in homogeneous self-attention layers of Diffusion Transformers (DiTs) and proposes structural complementarity as a design principle for improving architectural efficiency. The paper instantiates complementarity through two mechanisms. The first is rotation complementarity, which rotates token maps before applying causal attention and restores them afterward, allowing different layers or heads to capture different directional dependencies. The second is scale complementarity, which applies different K/V downsampling ratios to induce multi-scale attention. Experiments show that the proposed designs achieve comparable image generation quality while reducing sampling latency.

Critical strengths
  1. Complementarity offers a novel perspective for unifying sparse attention, multi-scale K/V token compression, and heterogeneous head/layer designs within a common DiT framework.
  2. The proposed mechanisms are relatively simple to implement and do not require additional training objectives, distillation, or complex optimization strategies.
  3. Experiments show that incorporating these complementary strategies can maintain comparable or slightly improved generation quality relative to a homogeneous DiT baseline while improving network efficiency.
Critical weaknesses
  1. The paper suggests that the union of causal masks under different rotations covers the token pairs of dense attention. However, mask coverage does not guarantee information propagation equivalent to dense attention. The paper should explain how this structural coverage across multiple attention layers translates into expressivity or preserved generative quality.
  2. The paper mainly compares against its own homogeneous baseline and lacks fair comparisons with other efficient attention or DiT acceleration methods.
  3. The paper requires more comprehensive efficiency comparisons. It reports inference latency, but does not report FLOPs or memory cost. Efficiency comparisons across the different architectural configurations in Section 4 would also be useful.
  4. The presentation needs improvement. Figure 4(b) appears to have missing content, possibly the "Layer1 Head3" label?
Questions for authors
  1. With a single trained model, does the rotation method work for latent grids with multiple aspect ratios for the text-to-image task?
  2. The method section mainly describes the rotation-based complementarity design using the rotation angles [0, 90, 180, 270]. However, the rotation angles [0, 180] already satisfy the proposed complementarity condition, and the experiments show that the [0, 180] configuration performs slightly better than [0, 90, 180, 270]. What is the advantage of using four rotation angles over two opposite rotation angles?
1 comment

Thank you for the review and the thoughtful questions, to which we respond below.

The union of causal masks under different rotations covers the token pairs of dense attention, but mask coverage does not guarantee information propagation equivalent to dense attention. How does this structural coverage across multiple attention layers translate into expressivity or preserved generative quality?

We agree that coverage alone does not make the model equivalent to dense attention, and the paper uses it only as a structural precondition. Across layers, alternating 0° and 180° gives every pair of tokens a direct attention connection within any two consecutive layers, whereas a single direction leaves about half of the pairs unconnected at any depth and collapses (Table 3). Coverage is therefore necessary, but it does not by itself determine quality: Homogeneous Blocks uses the same four directions, so the union of its masks covers all token pairs as in the interleaved settings, yet clustering identical directions harms quality (Table 4). Whether quality is preserved is thus an empirical question, which Tables 1-3, the SiT results and Appendix C.1 answer. We have rewritten the corresponding passage in Sec. 2.2 to make this distinction explicit: structural coverage is now described as a precondition for global context, and the preserved quality is attributed to the experiments. The caption of Fig. 2 is adjusted in the same way.

The paper mainly compares against its own homogeneous baseline and lacks fair comparisons with other efficient attention or DiT acceleration methods.

The paper studies complementarity as a structural design principle rather than competing with efficient-attention or acceleration methods. Acceleration methods such as token merging and pruning, caching and distillation are applied to a trained model, whereas complementarity concerns the design of the architecture itself; as discussed in Appendix A.2, the two operate at different levels and could potentially be combined.

The paper reports inference latency but not FLOPs, and efficiency comparisons across the architectural configurations in Section 4 would also be useful.

FLOPs are counted analytically with the same convention as DiT:

Resolution (tokens)256 (256)512 (1,024)1024 (4,096)2048 (16,384)
Quadratic attention term, share of baseline FLOPs3.6%12.9%37.2%70.3%
LightningDiT-XL (G)118.6524.5290924623
Layer-Rot / Head-Rot (G)116.5 (-1.8%)490.7 (-6.4%)2367.8 (-18.6%)15965 (-35.2%)
Layer-Scale (G)116.2 (-2.0%)486.5 (-7.2%)2300.4 (-20.9%)14883 (-39.6%)

Rotation adds no parameters, and Layer-Scale adds about 18K (<0.01% of 675M). FLOPs are a theoretical count, so they may differ from measured latency, which also depends on the hardware and low-level implementation. Within Section 4, all rotation variants have identical FLOPs and parameters, and so do all scale variants, which use the same set of ratios; the FID differences there are therefore comparisons at equal cost.

Figure 4(b) appears to have missing content, possibly the "Layer1 Head3" label.

Thank you for spotting this. The empty position is meant to indicate the remaining heads in the layer, and the revised figure shows an ellipsis there.

With a single trained model, does the rotation method work for latent grids with multiple aspect ratios in text-to-image generation?

Structurally, the 0°/180° setting used in our main results applies to any aspect ratio, because a 180° rotation is equivalent to reversing the token sequence and does not depend on the grid shape. For 90° and 270°, an H×W grid becomes W×H, and the operation remains well defined. Our experiments use class-conditional generation on square grids, so evaluating text-to-image generation with multiple aspect ratios is left to future work.

The method section describes four rotation angles, but [0°, 180°] already satisfies the complementarity condition and performs slightly better than [0°, 90°, 180°, 270°]. What is the advantage of four angles over two opposite ones?

Two opposite angles are sufficient, and they are the setting used for Layer-Rot in Table 1; as you note, they already satisfy the complementarity condition, and the results of the two settings are comparable (Table 3). Four angles are the general form of the function set: we use them to present the framework, and they are needed for the symmetry, grouping and ordering analyses and for Head-Rot.

Reviewer_kNM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the source of redundancy from DiT backbone and introduces 2 main complementary forms to reduce such a redundancy: layer-wise rotation & multi-scale features, these two components are all operated on feature map, namely rotating features by a certain degree or using strided convolution layer to produce multi-scale features. These propose components are injected not just at layer level but also at attention head level. Then the method conducts experiments with different design choices, demonstrating generation quality and improved efficiency.

Critical strengths
  • The motivation of homogeneous design is clear and sound, with clear evidence in figure 1.
  • The papers have conducted a lot of experiments to support the arguments and claims.
Critical weaknesses
  • The concept of rotation is akin to scanning patterns of input features in Vision Mamba models [1], that allow to each block to learn different feature patterns.
  • Similarly, the concept of multi-scale features in attention layer
  • Paper completely drop related work section so no discussion of relevant work.
  • This paper is packed with a lot of results and experiments on two complimentary components: the feature rotation and multi-scale features. However, there is no clear take-away message on each of them. This is crucial part of this kind of paper.
  • In terms of paper story and structure, the current flow is What is complementary functions? Where to apply them? How sensitive of the design choices? I believe the authors should only go with the first two questions. The last question is just ablation studies of multiple design options. So, table 1 and 2 are the main results of the paper, but they are quite limited to just two generative backbones, LightningDiT-XL and SiT. It should also run on more.
  • There is no appendix section but the main manuscript does help reference to it. Exposition:
  • The abstract and intro need works. For example, "Third, we systematically examine different symmetry, grouping, and ordering designs, showing … " This is very vague since it is not clear what is symmetry, grouping and ordering, these are all new concepts that never explain before.
  • L131-132: "While the parameters θi vary across units, the structural invariance of ψbase constrains the model to a singular inductive bias, leading to functional redundancy in the latent representation space." Not clear what is structural invariance? How it leads to singilar inductive bias? (Need evidence for this claim)
  • The math notations are quire complex, since so many are introduced. This needs to be simplified.
  • It is not cleary what is Asymmetric (Imbalanced) setting.

[1] Li et al. "Mamba-nd: Selective state space modeling for multi-dimensional data." in ECCV, 2024. [2] Phung et al. "DiMSUM: Diffusion Mamba--A Scalable and Unified Spatial-Frequency Method for Image Generation." in NeurIPS, 2024.

Questions for authors
  • Is there result for more than 4 rotation degrees?
  • Why no rotation works so bad? Is it due to causal attention constraint?
  • Can we combine both rotation and scale?
Reviewer_v2RGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes complementarity as a structural design principle for Diffusion Transformers, arguing that stacking homogeneous self-attention layers causes functional redundancy. To realize this principle, it introduces two instantiations: rotation complementarity, which uses direction-varying causal attention, and scale complementarity, which uses ratio-varying K/V down-sampling. Both are applied at layer and head granularity, together with ablations on symmetry, grouping, and ordering. On LightningDiT-XL and SiT-S over ImageNet-256, the variants match the homogeneous baseline in FID and IS while cutting backbone sampling latency by about 9 to 12 percent.

Critical strengths
  1. The paper is well organized and easy to follow. Its conceptual framing rests on three clear conditions, namely functional diversity, additive approximation, and computational efficiency, and the research-question-driven structure makes the overall logic easy to trace. The method is also easy to reproduce, since the appendix provides pseudocode and the implementation only relies on simple causal attention and shared convolution blocks.
  2. The design ablations are systematic. By studying symmetry, grouping, and ordering, the paper arrives at a useful observation that local homogeneity harms rotation complementarity but benefits scale complementarity, and that the efficiency gains grow as resolution increases.
Critical weaknesses
  1. Some closely related work is not discussed, which makes the novelty hard to position. Rotation complementarity rotates the feature map, runs causal attention across layers, and aggregates the results to recover global context at a lower cost. This is close in spirit to the multi-direction scanning in VMamba and to the use of causal attention inside a Diffusion Transformer in LEDiT. Scale complementarity, which assigns different K and V down-sampling ratios across units, also overlaps with multi-scale attention-head designs such as Shunted Transformer. These works are neither cited nor contrasted, so the paper would benefit from a related-work discussion that clarifies how the proposed method differs from them.
  2. The experiments lack competitive baselines and the reported gains are not significant. All comparisons are made only against the authors' own baseline, with no quantitative comparison to the efficiency and token-reduction works the paper itself cites, nor to recent training-free attention methods. Under this narrow setup the improvements stay within noise, since the best cfg FID across variants is only 2.01 against the baseline's 2.11 on LightningDiT, with the Layer-Rot variant even landing at exactly 2.11, and 10.76 against 11.02 on SiT. The latency reduction is also modest and drops to about 2 percent for the head-wise variant.
Questions for authors

Please refer to the two concerns raised in the Weaknesses section, particularly regarding the positioning of the proposed complementarity against prior multi-direction and causal-attention designs that are not discussed, and the lack of competitive baselines together with the limited significance of the reported gains. We would appreciate the authors' response to these two points.

Reviewer_WRvhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether global modeling in DiTs can be distributed across components with different inductive biases. It introduces rotation-based causal attention and layer-wise K/V scaling, evaluated on ImageNet-256 with LightningDiT and SiT. The methods maintain competitive FID with modest latency gains, and ablations highlight the importance of alternating rotation directions.

Critical strengths

The methods are simple and supported by focused ablations. The 0∘/180∘ comparison clearly demonstrates why alternating information flow is necessary in the proposed sparse design.

The approach maintains competitive generation quality while reducing latency on two backbones, suggesting a useful quality–efficiency trade-off.

Critical weaknesses

The evidence does not establish complementarity as a general explanation of standard DiTs. The standard DiT baselines use dense bidirectional attention, whereas the key ablations concern deliberately restricted causal variants. The experiments therefore validate an efficient architecture more directly than they explain existing DiTs.

The effect of complementarity is not isolated from causal sparsity and K/V downsampling, and complementarity itself is not directly measured. Limited comparisons with other efficient attention methods also make the source and competitiveness of the gains unclear.

Questions for authors

Is the paper primarily intended to explain existing DiTs or propose an efficient architecture? If the former, what evidence supports complementarity in standard dense DiTs?

Which gains remain after controlling for sparsity and K/V downsampling, and how does the method compare with efficient-attention baselines at matched compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.