- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies functional redundancy in homogeneous self-attention layers of Diffusion Transformers (DiTs) and proposes structural complementarity as a design principle for improving architectural efficiency. The paper instantiates complementarity through two mechanisms. The first is rotation complementarity, which rotates token maps before applying causal attention and restores them afterward, allowing different layers or heads to capture different directional dependencies. The second is scale complementarity, which applies different K/V downsampling ratios to induce multi-scale attention. Experiments show that the proposed designs achieve comparable image generation quality while reducing sampling latency.
Critical strengths
- Complementarity offers a novel perspective for unifying sparse attention, multi-scale K/V token compression, and heterogeneous head/layer designs within a common DiT framework.
- The proposed mechanisms are relatively simple to implement and do not require additional training objectives, distillation, or complex optimization strategies.
- Experiments show that incorporating these complementary strategies can maintain comparable or slightly improved generation quality relative to a homogeneous DiT baseline while improving network efficiency.
Critical weaknesses
- The paper suggests that the union of causal masks under different rotations covers the token pairs of dense attention. However, mask coverage does not guarantee information propagation equivalent to dense attention. The paper should explain how this structural coverage across multiple attention layers translates into expressivity or preserved generative quality.
- The paper mainly compares against its own homogeneous baseline and lacks fair comparisons with other efficient attention or DiT acceleration methods.
- The paper requires more comprehensive efficiency comparisons. It reports inference latency, but does not report FLOPs or memory cost. Efficiency comparisons across the different architectural configurations in Section 4 would also be useful.
- The presentation needs improvement. Figure 4(b) appears to have missing content, possibly the "Layer1 Head3" label?
Questions for authors
- With a single trained model, does the rotation method work for latent grids with multiple aspect ratios for the text-to-image task?
- The method section mainly describes the rotation-based complementarity design using the rotation angles [0, 90, 180, 270]. However, the rotation angles [0, 180] already satisfy the proposed complementarity condition, and the experiments show that the [0, 180] configuration performs slightly better than [0, 90, 180, 270]. What is the advantage of using four rotation angles over two opposite rotation angles?
Thank you for the review and the thoughtful questions, to which we respond below.
We agree that coverage alone does not make the model equivalent to dense attention, and the paper uses it only as a structural precondition. Across layers, alternating 0° and 180° gives every pair of tokens a direct attention connection within any two consecutive layers, whereas a single direction leaves about half of the pairs unconnected at any depth and collapses (Table 3). Coverage is therefore necessary, but it does not by itself determine quality: Homogeneous Blocks uses the same four directions, so the union of its masks covers all token pairs as in the interleaved settings, yet clustering identical directions harms quality (Table 4). Whether quality is preserved is thus an empirical question, which Tables 1-3, the SiT results and Appendix C.1 answer. We have rewritten the corresponding passage in Sec. 2.2 to make this distinction explicit: structural coverage is now described as a precondition for global context, and the preserved quality is attributed to the experiments. The caption of Fig. 2 is adjusted in the same way.
The paper studies complementarity as a structural design principle rather than competing with efficient-attention or acceleration methods. Acceleration methods such as token merging and pruning, caching and distillation are applied to a trained model, whereas complementarity concerns the design of the architecture itself; as discussed in Appendix A.2, the two operate at different levels and could potentially be combined.
FLOPs are counted analytically with the same convention as DiT:
Rotation adds no parameters, and Layer-Scale adds about 18K (<0.01% of 675M). FLOPs are a theoretical count, so they may differ from measured latency, which also depends on the hardware and low-level implementation. Within Section 4, all rotation variants have identical FLOPs and parameters, and so do all scale variants, which use the same set of ratios; the FID differences there are therefore comparisons at equal cost.
Thank you for spotting this. The empty position is meant to indicate the remaining heads in the layer, and the revised figure shows an ellipsis there.
Structurally, the 0°/180° setting used in our main results applies to any aspect ratio, because a 180° rotation is equivalent to reversing the token sequence and does not depend on the grid shape. For 90° and 270°, an H×W grid becomes W×H, and the operation remains well defined. Our experiments use class-conditional generation on square grids, so evaluating text-to-image generation with multiple aspect ratios is left to future work.
Two opposite angles are sufficient, and they are the setting used for Layer-Rot in Table 1; as you note, they already satisfy the complementarity condition, and the results of the two settings are comparable (Table 3). Four angles are the general form of the function set: we use them to present the framework, and they are needed for the symmetry, grouping and ordering analyses and for Head-Rot.