acceptodds
Under review as a conference paper at ICLR 2027

SFA: A Statistical Physics-Inspired Diffusion Transformer for Efficient and High-Quality Image Generation

21 views

Abstract

Diffusion transformers have achieved remarkable success in image generation, but accelerating attention while preserving generation quality remains a major challenge at high resolutions. Inspired by statistical physics and the widespread occurrence of scale-free laws in nature, we introduce Scale-Free Attention (SFA), which preserves fine local interactions while summarizing distant regions at progressively coarser scales. For tokens, mean pooling, token-count correction, and a shared softmax over a scale-free spatial hierarchy give SFA complexity in both forward and backward passes. Our Triton implementation achieves forward and backward attention speedups over FlashAttention at 262,144 tokens. After 40 epochs of PixelFlow training from scratch on ImageNet at resolution, SFA achieves the best generation quality among five attention mechanisms, including full attention. Fr\'echet inception distance evaluated on 50,000 generated samples (FID-50k) improves from 13.25 for full attention to 12.51, and the advantage over full attention persists through 80 epochs. SFA also achieves the best generation quality among the compared efficient-attention methods on Flickr-Faces-HQ (FFHQ). Beyond training from scratch, Stable Diffusion 3.5 Medium with attention replaced by SFA retains the generation quality of its full-attention counterpart under matched fine-tuning budgets. We obtain similar results when fine-tuning pretrained PixelFlow on ImageNet. SFA establishes scale-free structure as an effective design principle for efficient, high-quality diffusion transformers.

Primary area:
generative models
Keywords:
Diffusion Model, Diffusion Transformer, DiT, Attention, Log-Linear, Scale-Free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LFArOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper draws on the property of low correlation between distant pixel patches in natural images, and proposes to pool distant token patches within the attention mechanism to reduce the computational cost of attention. To this end, the authors design a hierarchical pooling strategy and apply it to image generation, achieving a notable speedup compared with existing attention mechanisms.

Critical strengths

The phenomenon observed in the paper (the interaction strength between distant tokens is significantly lower than that between adjacent tokens) is intuitive and consistent with common sense. The proposed scheme is also intuitive and achieves noticeable performance improvements in some experiments.

Critical weaknesses
  1. The presentation of the paper is disorganized and appears to be AI-generated. The paper does not seem to be carefully prepared, organized or formatted. I even doubt whether the authors have read through their own manuscript. Some notable issues are listed below: a) Panels (b)/(c) of Figure 1 appear before Figure 2(c)/(d) in layout, but their citations in the main text come much later than those for Figure 2(c)/(d). Figure 2(c)/(d) is presented in the introduction section (page 2), while Figure 1(b)/(c) only appears in the work section (page 4). This arrangement is inconsistent with the reading flow for readers. b) Figure 1(b)/(c) contain numerous terms that are undefined in the main text and must be looked up in the appendix. Furthermore, one panel shows theoretical results and the other shows experimental measurements, yet they use entirely different terminology with substantially divergent definitions in the appendix. It is unclear why separate terms are introduced for theoretical and measured results, which violates standard writing conventions. c) The plot in Figure 1(d) is asymmetric. Neither the caption nor the main text provides a detailed description of the setup or the rationale for this asymmetry. d) There are excessive and abnormal appendix citations throughout the main text. Three or more citations pointing to different appendices appear on nearly every page, severely hindering readability. e) The paper contains an excessively large appendix packed with abundant terminology and descriptions, even though the core of the proposed method is not complicated and can be fully understood from Figure 1(e). This raises suspicion that most of the appendix content was generated by an LLM. f) The abstract cites experiments on SD3.5 as supporting evidence, yet these experiments are not included in the main manuscript.

  2. In addition, there are flaws in the authors’ experiments: a) Comparisons against state-of-the-art methods such as REPA and JiT are missing. b) The speed evaluation is problematic. The claimed 200×+ speedup for 256k resolution in forward/backward passes is misleading, as no practical model generates images at 256K resolution. The actual speed values shown in Table 1 have only minor differences compared with the full baseline.

Questions for authors

See above.

Reviewer_ErPAOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Scale-Free Attention (SFA). SFA preserves fine local interactions and represents distant regions with progressively coarser pooled keys and values. Token-count correction and a shared softmax combine the scales, with a Triton implementation giving O(Nlog⁡N)O(N \log N) forward and backward complexity. Experiments cover PixelFlow on ImageNet, DiT on FFHQ, and fine-tuning SD3.5 Medium. Reported results include ImageNet FID-50k of 7.11 versus 7.55 for full attention after 80 epochs, together with kernel and end-to-end speed measurements.

Critical strengths
  1. Generation quality is supported by substantive comparisons. On ImageNet, SFA reports FID-50k of 7.11 vs. 7.55 for full attention after 80 epochs; under matched SD3.5 fine-tuning budgets, it also matches the full-attention DPG score of 85.51.
  2. The fixed-partition reverse-KL characterization gives a concrete interpretation of the approximation. Component ablations support count correction and shared normalization: removing them raises FFHQ-128 FID from 22.91 to 25.91 and 27.99, respectively.
  3. The method includes a forward/backward Triton implementation with O(Nlog⁡N)O(N \log N) complexity. Reporting end-to-end runtime and memory alongside kernel benchmarks makes its practical benefits and remaining bottlenecks assessable.
Critical weaknesses
  1. The ablations do not isolate the contribution of the specific distance-dependent hierarchy from a comparably budgeted combination of local attention and pooled global context. The reverse-KL result holds for a fixed partition and therefore does not establish why this hierarchy is preferable.
  2. The learned-aggregation variant in Section 5.6 also removes token-count correction. Its worse FID (41.40 versus 40.66) therefore does not isolate the effect of the pooling operator, limiting the evidence for choosing mean pooling over learned aggregation.
  3. Quality evidence is less complete where the largest end-to-end speedups are reported. FFHQ-512 lacks a full-attention quality reference, and SD3.5 quality is evaluated at 1024 rather than 2048; the ImageNet and SD3.5-1024 settings with full-attention quality comparisons show more modest inference gains of about 1.09×1.09\times and 1.20×1.20\times.
Questions for authors
  1. Is there an existing mean vs. learned pooling comparison that retains token-count correction in both variants?
  2. Are distance-decay statistics available at other diffusion timesteps beyond the reported setting?
  3. Are any SD3.5 quality measurements at 2048 resolution available from the existing experiments?
Reviewer_oeHhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Scale-Free Attention (SFA), an efficient attention mechanism for diffusion transformers.

Its core idea is to preserve exact local interactions and partition the remaining image into progressively larger blocks, each represented by its mean key and value, i.e., fine grained attention up close and coarse pooled summaries far away.

SFA retains the original attention projections without additional parameters or hyperparameters and stays drop-in compatible with pretrained models.

The authors combine this design with a fast triton kernel and an approximation analysis, and run experiments both training from scratch (on ImageNet and FFHQ) and adapting pretrained models (PixelFlow and Stable Diffusion 3.5 Medium). The results are demonstrated the effectiveness of SFA, showing that it can match or exceed full attention in generation quality while being substantially faster.

Critical strengths
  1. SFA retains the original attention projections without additional parameters or hyperparameters, thus it can be dropped into a pretrained model and fine-tuned without changing the projections.
  2. The method is backed by an optimality result (SFA uniquely minimizes reverse KL for a fixed partition), and the authors report large kernel-level speedups over FlashAttention at long sequences.
Critical weaknesses
  1. ImageNet results at 40/80 epochs show improvements but do not distinguish faster convergence from better eventual quality. LLSA, SLA, and VSA stop at 40 epochs, leaving the longer-training comparison incomplete. FFHQ provides additional wins over efficient-attention baselines but lacks a full-attention quality comparison.

  2. Practical speedups are setting-dependent, and related baselines are missing. The headline 357×/265× gains concern isolated attention kernels at 262,144 tokens. End-to-end training gains are approximately 1.08× on PixelFlow and 1.21×/2.10× on SD3.5 at 1024/2048. The largest demonstrated gains occur in small pixel-space models with very long sequences, leaving broader applicability insufficiently tested. Direct comparisons with FMA and MRA are absent; although FMA’s implementation is unavailable, the provided ablations do not fully establish SFA’s advantage over related multiscale designs.

Questions for authors
  1. The from-scratch ImageNet comparison covers all five methods at 40 epochs, but only SFA/full are continued to 80. Can you also run LLSA/SLA/VSA to 80 epochs to test whether SFA's advantage persists under longer training?
  2. The headline 357× kernel speedup is measured at a much longer sequence length than PixelFlow uses, where end-to-end training gains are ~1.08×. Can you clarify the practical speedup regimes and report time-to-target-quality alongside the existing throughput measurements?
  3. Why aren't closely related multiscale methods such as FMA or MRA compared directly?
  4. The radial-attention analysis covers multiple heads and layers but only one backbone, dataset, and timestep. Does the power-law trend persist across other backbones, datasets, and denoising timesteps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.