acceptodds
Under review as a conference paper at ICLR 2027

AReA: Adaptive Receptive-Field Attention for Efficient High-Resolution Diffusion Transformers

Abstract

Generating images beyond the training resolution of Diffusion Transformers (DiTs) remains challenging, as the number of tokens grows rapidly with resolution, full attention becomes expensive, and outputs often suffer from structural distortions and degraded detail. We identify uniform full attention as the common source of both problems. Attending to all tokens at every step causes quadratic cost and exposes the model to more tokens and longer relative distances than seen during training, pushing it out of distribution. We observe that attention's effective receptive field narrows during denoising, from global in early steps that form structure to local in later steps that refine details. We propose AReA, a training-free attention mechanism whose receptive field evolves from coarse to fine, following the denoising progression. In early steps, AReA applies cross-scale coarse attention over downsampled keys and values, capturing global context, while in later steps it applies local attention within progressively shrinking windows. Without additional parameters or training, experiments show that AReA consistently outperforms state-of-the-art training-free baselines across target resolutions and different models, while also being faster and more efficient.

Primary area:
generative models
Keywords:
Diffusion Models, High-resolution Image Generation, Diffusion Transformers, Training-free Inference, Text-to-image generation, Efficient Attention, High-Resolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zuRnOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AReA, a training-free attention adaptation method for high-resolution diffusion transformers. Its central idea is to decouple output resolution from the resolution and spatial extent of the attention context. During early denoising, full-resolution queries attend to spatially subsampled keys and values to establish global composition. Later steps use progressively shrinking local windows, with wavelet-guided splitting further reducing computation in smooth regions. Overlapping aggregation, halo context, and a global-KV bridge mitigate boundary artifacts and preserve cross-region consistency.

Critical strengths

1.A coherent organizing principle. Adapting the attention context to the changing requirements of denoising provides a unified explanation for both computational savings and the balance between global composition and local detail.

2.A meaningful decoupling of resolutions. Maintaining full-resolution queries allows high-resolution states to evolve directly, rather than inheriting a fixed composition from an independently generated low-resolution image. Computation is reduced by restricting contextual access rather than reducing the resolution of the generation state.

3.Substantial design validation. The paper includes phase replacement, fixed-window, pooling, switching-time, component-removal, and low-resolution bootstrap comparisons. The DWT ablation provides particularly interpretable evidence of computational savings with approximately preserved quality.

Critical weaknesses

1.The distinctive contribution is not fully separated from the supporting components. The method combines subsampled KV, window scheduling, wavelet analysis, halo context, and global tokens. Existing removal ablations are useful, but compute-matched comparisons against alternative schedules and random or uniform splitting would better establish the value of the proposed temporal and content-adaptive principles.

2.Positional adaptation and receptive-field adaptation remain confounded. Section 3.2 explicitly rescales query and key coordinates, whereas the paper also describes the method as not altering positional encodings. Preserving the RoPE frequency formula does not preserve the resulting positional representation. Separate interventions on coordinates and KV counts are needed to identify the source of structural improvements. (pp. 5, 8)

3.The central structural-quality claims require more direct evaluation. Fixed windows achieve comparable or higher scores on some generic metrics despite visibly fragmented outputs. This suggests that duplication rates, counting errors, seam frequency, or blinded structural assessments would be more informative than additional aggregate quality scores. (Table 3; Figures 7 and 13)

4.The causal interpretation is stronger than the supporting evidence. Increasing attention locality motivates the method, but does not directly determine the proposed window schedule. The out-of-distribution exposure analysis provides useful associations without independently establishing the effects of token count and positional distance on generation failures

Questions for authors

1.Contribution of adaptive scheduling. Could you clarify whether the proposed window schedule and DWT-guided splitting outperform simpler alternatives at comparable computational budgets? Existing component-removal ablations change both the allocation strategy and the amount of computation, making these effects difficult to separate.

2.Role of positional coordinates. Section 3.2 rescales Q/K coordinates, whereas the paper also states that positional encodings are unchanged. Does “unchanged” refer specifically to the RoPE frequency formula? Are query coordinates continuously scaled or quantized to the coarse grid? Can the effects of coordinate rescaling and KV subsampling be distinguished?

3.Structural-quality assessment. Table 3 shows that some simplified variants remain competitive on generic quality metrics despite the structural failures illustrated in the figures. Do you have systematic evidence on duplication, counting errors, or window-boundary artifacts that establishes how frequently these failures occur?

4.Generality of the locality argument. Figure 3 aggregates attention across heads, blocks, and prompts. How heterogeneous is the observed global-to-local transition, and are there identifiable heads or prompt types that continue to require substantial long-range interaction during late denoising? How effectively does the global-KV bridge accommodate these cases?

6 comments

We thank the reviewer for the constructive feedback and are glad they found the core idea coherent, the design meaningful, and the ablations thorough. Please find our detailed responses below; we would be happy to engage in further discussion if needed.


W1 / Q1

Compute-Matched Comparison of Window Schedules and Splitting Strategies

To further establish the value of the proposed temporal and content-adaptive principles, we extended our ablations with compute-matched comparisons against alternative and simpler approaches for the window schedule and the splitting signal. We compare the window schedule and the splitting strategy against alternatives at equal or higher compute on FLUX.1 at 4096×4096 using all 195 Aesthetic-4K prompts.

Phase II window size schedule

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
reverse1511.276.4645.3923.050.29129.60151.5
linear1581.266.3445.3823.210.28729.42152.6
late shrinking1591.266.3445.3123.200.28729.44152.7
early shrinking (default)1511.326.3745.2223.310.29729.71148.50

The default schedule shrinks windows quickly at first and slowly near the end, following how attention narrows during denoising (Figures 3 and 11). We compare it with three alternatives: reverse, which uses the same window sizes in reverse order, so windows grow over time; linear, which shrinks windows by the same amount at every step; and late shrinking, which keeps windows large for longer and shrinks them quickly near the end. All three use the same or more compute but lower ImageReward and raise FID.

Phase II window splitting

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
variance1511.276.3445.3823.230.28829.47153.0
random1511.226.3345.1623.220.28729.47152.8
uniform1511.206.3145.0923.140.28129.44155.1
DWT (default)1511.326.3745.2223.310.29729.71148.50

All strategies run at the same cost and differ only in which windows are split. Uniform splitting selects a set of windows once and keeps splitting the same windows until the end of denoising. Random splitting selects new windows at random at each step. Variance splitting ranks windows by the variance of their tokens instead of the DWT map. Uniform and random splitting ignore image content and can split windows over detailed regions, while variance also reacts to smooth changes. DWT-guided splitting performs best overall.


W2 / Q2

Clarification on RoPE and Positional Coordinates

We thank the reviewer for this careful observation, which is correct. By “unchanged,” we refer specifically to the RoPE frequency formula. AReA does not rescale or recalibrate the rotary frequencies and requires no RoPE length extrapolation. However, in Phase I it does remap the token coordinates used by RoPE. To be more specific, in Phase I query coordinates are quantized to the coarse grid, not continuously scaled: each f × f block of queries is assigned and shares the position of its pooled key on the coarse grid, while in Phase II all tokens keep their original coordinates. We will fix and clarify this in the revised version of the paper.

Separating Coordinate Remapping and KV Subsampling

The two effects can be separated. To isolate the coordinate remapping, we ran Phase I with the same key–value pooling but without remapping the query coordinates, so that queries keep their full-resolution positions while attending to the pooled keys. Removing the remapping alone causes structural collapse (table below). The remapping is therefore needed: it places the queries on the same coarse grid as the pooled keys.

Phase I without query coordinate remapping (FLUX.1, 4096×4096)

ConfigIR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
without query remapping−1.285.4837.3519.520.25023.73248.5
AReA1.326.3745.2223.310.29729.71148.50

The KV subsampling follows from what the early denoising steps need. In these steps, attending to all high-resolution tokens raises attention entropy beyond the training range and degrades quality (see our response to Reviewer FzqH on attention entropy). These steps mainly form the low-frequency layout, which does not require every token, while full attention over all tokens is costly. Pooling the keys and values therefore keeps the number of keys close to the training range and reduces computation, while keeping the information needed to form the layout.


Our response continues in the next comment.

W3 / Q3

LLM-as-Judge Evaluation of Structural Consistency

We thank the reviewer for this suggestion. We agree that generic metrics miss structural failures; as the reviewer notes, fixed windows score well on them despite fragmented outputs. To measure these errors directly, we added a blind evaluation in which a vision-language model (Claude Opus 5.5) acts as a judge. It scores text rendering, object counts, spatial relations, and structural coherence, and reports duplication, seam, and anatomy-error rates on both backbones at three resolutions. AReA ranks first or second on text, count, spatial relations, aesthetics, detail, and structure in every setting, and has the lowest anatomy-error rate in all 6 settings. If the reviewer considers a user study necessary, we will add one. Please see our response to Reviewer AykN for the full protocol and results.


W4

Window Schedule Design

The window schedule is shaped by the compute budget, the model’s native resolution, and the observed locality of attention. The start window is the largest window still cheaper than full attention, and the final window is the model’s native size. Between them, the schedule follows attention locality. As Figure 3 shows, attention narrows quickly in early steps and more slowly later, so in our scheduler windows shrink the same way (Figure 11). To test this choice, we compared it with other schedules on FLUX.1 at 4096×4096:

Phase II window size schedule

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
reverse1511.276.4645.3923.050.29129.60151.5
linear1581.266.3445.3823.210.28729.42152.6
late shrinking1591.266.3445.3123.200.28729.44152.7
early shrinking (default)1511.326.3745.2223.310.29729.71148.50

Reverse uses the same window sizes in reverse order, so windows grow over time. Linear shrinks windows by the same amount at every step. Late shrinking keeps windows large for longer and shrinks them quickly near the end. Reverse improves some perceptual metrics, but all three alternatives lose structural coherence, with lower ImageReward and higher FID. Following how attention narrows gives the best results overall.

Separating Out-of-Distribution Causes

To separate the effects of token count and positional distance, we ran two controlled experiments on FLUX.1, each changing only the token count or only the distance. To test unseen token counts alone, we generate at 4096×4096 with NTK-scaled RoPE. This maps query–key distances back into the training range, while every query still attends to 16× more tokens than in training. To test unseen distances alone, we generate at the native 1024×1024 with inverse NTK scaling, which keeps the token count the same as in training but makes the model see distances as if the canvas were 4096×4096.

Unseen token counts only: NTK-scaled RoPE (FLUX.1, 4096×4096, all 195 Aesthetic-4K prompts)

ConfigIR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
Base−0.725.8022.5520.340.21125.72179.8
unseen tokens−0.295.8023.8620.820.19825.52182.3
AReA1.326.3745.2223.310.29729.71148.50

Unseen distances only: inverse NTK scaling (FLUX.1, 1024×1024, all 195 Aesthetic-4K prompts)

ConfigIR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
unseen distances0.996.3568.5922.490.28829.03156.5
Base1.406.4168.8523.580.30729.66151.5

Each cause alone degrades generation, with a distinct failure mode. With unseen token counts, attention spreads over too many keys, and outputs become washed out and lose high-frequency detail, even though distances are within range. With unseen distances, the model produces duplicated subjects and low-frequency artifacts, even at the native token count. AReA mitigates both causes, since in both phases it keeps the number of keys and the query–key distances close to the training range (Appendix G). We will add these experiments, along with qualitative comparisons, to the revised paper.


Our response continues in the next comment.

Q4

Consistency of Attention Locality and the Global-KV Bridge

We thank the reviewer for this question. To measure heterogeneity, we measured how attention locality changes during denoising separately for every head, block, and prompt instead of averaging them, using all 195 Aesthetic-4K prompts on FLUX.1 and Qwen-Image (see also our response to Reviewer AykN). For each (prompt, block, head), we take the attention-weighted query–key distance of all keys over all queries and compare it at the first and last denoising steps. If it is smaller at the last step, attention has become more local.

Change in attention locality per head, block, and prompt (1024×1024)

FLUX.1: became more localFLUX.1: shrinkQwen-Image: became more localQwen-Image: shrink
Heads (each head, in each block and prompt)93.0% (of 266,760)2.2×90.1% (of 280,800)2.6×
Blocks (mean over heads and prompts)57/571.9×59/602.3×
Prompts (mean over blocks and heads)195/1951.8×195/1952.0×

The behavior is consistent. 93.0% (FLUX.1) and 90.1% (Qwen-Image) of heads become more local, and so does every one of the 195 prompts in both models. The exceptions are few: 7.0% of FLUX.1 heads and 9.9% of Qwen-Image heads stay flat, and in Qwen-Image most of them are in the first block, which stays global. We will add these experiments and analysis, along with complementary plots, to the revised version.

Thus, no prompt type keeps long-range attention late in denoising, and the few heads that do are mostly in the first block of Qwen-Image and scattered across blocks in FLUX.1.

The global-KV bridge gives each window’s queries a summary of the most important tokens outside the window, to prevent structural divergence and duplicated content. A token’s importance is how far its key deviates from the mean key of the image. As Figure 3 and the analysis above show, attention becomes more local in most blocks, especially over dominant keys. In full attention, however, queries still attend to all tokens, only with much smaller weights than to their neighbors. The bridge approximates this at low cost. It also serves these remaining global heads, since every window keeps access to the most important tokens across the image; without it (ρ = 0), objects are duplicated (tables below). By default, it selects tokens in a stratified way over an 8×8 grid, so that the selected tokens do not all come from a small region. To ablate this, we compare it with top-k selection, which ranks all tokens outside the window by importance without stratification and keeps the same number of tokens. We also vary ρ, which sets how many bridge tokens each window receives.

Global-KV token selection (FLUX.1, 4096×4096)

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
top-k1501.316.3345.3923.260.29129.62149
stratified (default)1511.326.3745.2223.310.29729.71148.50

Global-KV bridge ratio ρ (FLUX.1, 4096×4096)

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
ρ = 01331.326.2545.3823.250.29429.11161.25
ρ = 0.251411.326.3544.5223.230.29629.70148.8
ρ = 0.5 (default)1511.326.3745.2223.310.29729.71148.50
ρ = 0.751681.246.3244.7623.140.29229.35154.2
ρ = 11891.216.3244.1623.040.29029.18154.4

Stratified selection performs better than top-k on most metrics, since it distributes the bridge budget across the canvas. Increasing ρ slows inference and softens detail, as each query spreads its attention over more tokens. Removing the bridge (ρ = 0) causes duplicated objects, raising FID from 148.5 to 161.3, while results barely change between ρ = 0.25 and 0.5. We will add these analyses to the revised paper.


Thank you again for your thoughtful and constructive comments. We sincerely appreciate the opportunity to clarify and discuss any further questions.

Reviewer_FzqHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that existing training-free high-resolution generation methods often fail to adequately consider the effective range of attention at different timesteps. Aiming at the characteristic of image generation where global structures are produced first and local details are refined subsequently, the authors propose reducing the number of attention tokens at high-noise timesteps, while focusing attention on local regions at low-noise timesteps to synthesize image details more favorably. Experiments demonstrate that this approach can produce high-quality high-resolution images and significantly reduce inference time.

Critical strengths
  1. The idea to perform attention mechanism differently in different timestep is straight-forward and seems effective.
  2. The experiments shows the method is effective and efficient, demonstrating the method valuable to the research field.
Critical weaknesses
  1. For training-free high-resolution image generation, prior works have pointed out that directly modifying the number of tokens in attention will cause entropy shift and thus degrade generation quality. Why does this phenomenon not occur in this paper? Reference: Training-free diffusion model adaptation for variable-sized text-to-image synthesis. NeurIPS 2024.

  2. In training-free high-resolution image generation, the structure-first-then-details paradigm is a widely discussed topic. The authors need to cite relevant papers for further discussion, especially the differences from them. References: a) Hidiffusion: Unlocking high-resolution creativity and efficiency in low-resolution trained diffusion models. ECCV 2024. b) FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling. ICLR 2024. c) Demofusion: Democratising high-resolution image generation with no $$$. CVPR 2024.

  3. The clarity of the paper needs to be improved, mainly in two aspects. First, the text in some figures is too small for human reading, such as Figure 6. Second, the description of ADAPTIVE WINDOWED FINE ATTENTION is overly rigid, and the schematic diagram in Fig4 fails to clearly illustrate how adaptive window partitioning is performed.

  4. Adaptive window partitioning for images is also a hot topic in image generation. The authors should discuss and summarize it in the related works section, and clarify the distinctions between all adaptive schemes proposed in this paper and existing ones. References: a) GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generation. NeurIPS 2025. b) MaskBit: Embedding-free Image Generation from Bit Tokens. ICLR 2025.

Questions for authors

Overall, this paper is generally good with solid ideas and experiments, yet it lacks discussion and citations of relevant prior works. Please respond with reference to the content in the weakness section.

5 comments

We thank the reviewer for the constructive feedback and are glad they found the idea straightforward and effective, the method efficient, and the work valuable to the field. Please find our detailed responses below; we would be happy to engage in further discussion if needed


W1

Attention Entropy Shift

As shown by Jin et al., attention entropy changes with the number of tokens being processed, which directly affects generation quality. At higher resolutions, the number of tokens grows beyond the training range, so each query’s attention spreads over many more keys, increasing the entropy and leading to attention dilution. Since entropy is computed over each query’s attention distribution, it is determined by the number of keys each query attends to. AReA addresses this by bringing this number back toward the training range (Figures 14 and 16 in the Appendix). In Phase I, queries attend to downsampled keys whose count stays close to the native training count. In Phase II, each query attends only to a limited number of tokens within its local window, far fewer than in full attention. As a result, AReA reduces attention dilution and keeps the entropy close to the training regime. This is how AReA enables high-resolution generation without rescaling the attention logits.

To further support this, we measure attention entropy on FLUX.1 at 4096 × 4096. For each query, we compute the entropy over the keys it attends to, average it over all blocks and heads, and report the shift relative to native 1024² generation (table below). Together with Figure 16, these results show that AReA stays closest to the model’s native behavior, consistent with Jin et al.’s findings. We will add this analysis and the complete plot of the entropy shift across all denoising steps to the revised version.

Attention entropy shift

MethodAverage Entropy ShiftPhase I, f = 4Phase I, f = 2Phase II startPhase II midLast step
Base, 4K+2.63+2.66+2.66+2.63+2.58+2.64
SEGA, 4K+1.45+1.70+1.57+1.49+1.38+1.28
AReA, 4K+0.76+0.02+1.12+1.56+0.93+0.22

W2

Discussion of Structure-First-Then-Details Methods (HiDiffusion, DemoFusion, and FreCaS)

We thank the reviewer for the suggested references. We would like to note that DemoFusion and HiDiffusion are already cited and discussed in the paper (Appendix I, Extended Related Work). We will add FreCaS and expand the discussion of the structure-first-then-details paradigm in the revised version.

In brief, DemoFusion and FreCaS both start at a low resolution and progressively upscale, reusing information from the low-resolution result (skip residuals and cross-attention maps, respectively) to preserve the global structure. In both methods, the layout is formed at low resolution and inherited by the high-resolution output, so the final image is anchored to the low-resolution result. HiDiffusion is specifically designed for U-Net architectures: it modifies the U-Net’s downsampling and upsampling layers so that its deeper blocks operate at the training resolution, and applies shifted-window attention. AReA instead generates directly at the target resolution in a single pass, without any low-resolution generation, and follows this paradigm by adapting the attention receptive field from global to local over denoising steps.


W3

Clarity of Figures and Description

We thank the reviewer for these suggestions and will improve the clarity of the figures and the method description in the revised version. Regarding Figure 4, we note that it is intended as a high-level overview of the pipeline rather than a detailed schematic. The complete illustration is provided in Figure 12 (Appendix), which shows the base windows, their overlaps, and the DWT-based splits at each Phase II step, with the full procedure described in Appendices B and C. We will clarify this in the caption of Figure 4 and refer to Figure 12 from the main text in the revised version.


W4

Discussion of Adaptive Partitioning Methods

We thank the reviewer for pointing us to these works. GPSToken and MaskBit change how images are tokenized. GPSToken partitions the image into variable-size regions by entropy and encodes each as a 2D Gaussian token, while MaskBit represents images with embedding-free bit tokens. Both require training a tokenizer and a generator. AReA instead keeps the pretrained model and its token grid unchanged, and adapts only which keys each query attends to at inference time, both across denoising steps and across image content, without any training. We will add a discussion of these and other adaptive partitioning methods to the extended related work section in the appendix, and clarify their distinctions from AReA in the revised version.


Thank you again for your thoughtful comments; we welcome any further questions

Thanks for the reviewing. We have no further concerns~

Thank you for your time and feedback. We are glad we were able to address your concerns, and we would be grateful if you would consider updating your rating accordingly.

Reviewer_AykNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AReA (Adaptive Receptive-Field Attention), a training-free attention mechanism for efficient high-resolution image generation with diffusion transformers. AReA uses pooled keys and values to capture global structure during early denoising, then progressively shrinks local attention windows to refine details. It also uses DWT-based window splitting, halo context, and a global-KV bridge to reduce computational cost, boundary artifacts, and duplicate objects.

Critical strengths
  • The coarse-to-fine attention schedule adapts the receptive field to the progression of diffusion denoising, balancing global structure and local detail.
  • AReA requires no additional training or parameters, making it practical for adapting pretrained diffusion transformers to high-resolution generation.
  • Quantitative and qualitative results demonstrate improvements in perceptual quality, prompt alignment, and structural coherence over competing training-free methods.
Critical weaknesses
  • The novelty of the proposed method is unclear. In Phase 1, the proposed components resemble KV compression in PixArt-Sigma [1] and context drop in Lumina-Next [2]. For Phase 2, the proposed components are similar to blocked local attention with halo of Halo [3] and Local Attention used in DiTFastAttn [4] and GRAT [5]. It would be better to refer to these papers and discuss the differences.

[1] PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation. ECCV 2024.

[2] Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT. NeurIPS 2024.

[3] Scaling Local Self-Attention for Parameter Efficient Visual Backbones. CVPR 2021.

[4] DiTFastAttn: Attention Compression for Diffusion Transformer Models. NeurIPS 2024.

[5] Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers. arXiv 2025.

  • It is unclear whether using a fixed attention layer design for all layers is the best choice. DiTFastAttn [4] emphasizes that attention locality differs for every layer. However, in Fig. 3, AReA only averages across heads and blocks and applies the same window and pooling schedule to all layers. It would be better to analyze layer-level behavior and evaluate the design choice using a dynamic window size per layer.

  • In the experiment, the validation of the core assumption is limited. The attention-locality analysis is based on vanilla FLUX.1 and 24 prompts. More evidence is needed to show that progressively localized attention consistently benefits performance across models and prompts. Also, as ResDiT also combines global attention and local attention, it would be better to empirically compare the effectiveness with ResDiT.

  • In addition, quantitative evaluations focus on FLUX.1 and Qwen-Image, and results for other models and extreme resolutions are primarily qualitative. It would be better to compare these models quantitatively as well.

  • Also, evaluation of structural consistency is limited. The reported metrics do not directly measure object-count accuracy, spatial-relation consistency, or text-rendering quality, despite these being central claims of the paper.

  • In terms of the efficiency, in addition to the reported substantial speedups, it would be better to also analyze peak GPU memory usage and the runtime overhead of auxiliary operations, such as DWT-based window splitting and global-KV selection.

  • Finally, the method relies on several empirically chosen hyperparameters and schedules. More extensive sensitivity analyses and cross-model ablations are needed to demonstrate robustness.

Questions for authors

Please refer to the weaknesses.

6 comments

We thank the reviewer for the constructive feedback and are glad they found the core idea well-motivated, the method practical, and the results convincing. Please find our detailed responses below; we would be happy to engage in further discussion if needed.


W1

Novelty Relative to Prior KV Compression and Local Attention Methods

We thank the reviewer for the references and agree that several of the attention operations used in AReA have been used in different prior work. We will cite and discuss all of them in the revised version. However, our contribution is not these operations, but the view that attention’s receptive field should adapt as denoising progresses, and an attention mechanism design that follows from it. We argue that at high resolution, full-resolution global attention at every denoising step is not only unnecessary but harmful. It is unnecessary because, on average, our analysis shows that influential attention becomes increasingly local during denoising. It is harmful because attending to a much larger number of tokens can dilute attention, while larger positional offsets introduce out-of-distribution interactions. AReA therefore matches the receptive field to the stage of denoising, moving from coarse global attention to fine local attention. This addresses structural and detail degradation while improving efficiency.

AReA’s Phase I differs from PixArt-Σ and Lumina-Next in its purpose, in how keys and values are reduced, and in how the coarse view evolves during denoising. Both methods reduce KV to save computation, and neither is designed to keep attention close to its training conditions or to track layout formation. Both also use merging, PixArt-Σ through a learned convolution and Lumina-Next through average pooling. We found that averaging removes the distinctive parts of the keys, which flattens attention and degrades quality. Subsampling performs better in our case (Table 4). The coarse grid then becomes gradually finer as the layout forms, before handing over to local attention instead of returning to full attention. Phase II differs from DiTFastAttn and GRAT in how its windows change over denoising. Both keep their window size or attention pattern fixed during denoising, and DiTFastAttn also computes full attention at some steps to recover long-range information. In contrast, our windows start large and shrink as attention becomes more local, ending at the model’s native size, and full attention is never computed. The windows also adapt to image content: windows over smooth regions, identified by a DWT map, are split into smaller ones to save computation. Our halo follows HaloNet’s idea of extending each window’s keys with neighboring tokens. We use it only to help reduce seams between windows, and our method still performs well without it (Table 3).

Importantly, the core of AReA is adapting the receptive field of attention to the stage of denoising, and the operations in both phases are the design choices we use to achieve this. We will discuss the referenced methods and clarify our contributions in the revised paper.


W4

Quantitative Results on Additional Models and Extreme Resolutions

Following the suggestion, we quantitatively evaluate AReA on three backbones at 6144 × 6144. Generation at this resolution is very time-consuming, so given the limited rebuttal period we use 20 randomly selected prompts from Aesthetic-4K. The results at 6K resolution are consistent with our main experiments (table below).

Quantitative comparison across three backbones at 6144×6144 (20 Aesthetic-4K prompts)

ModelMethodIR↑Aesthetic↑CLIP↑PS↑FID↓FID_p↓
FLUX.1Base−2.254.6512.4817.06459.9364.8
SEGA0.686.1727.5422.38232.2163.8
AReA0.836.3728.5822.47227.8156.9
Qwen-ImageBase−0.955.5423.9619.78282.7215.8
SEGA1.386.4228.8123.89219.5160.2
AReA1.566.5129.1224.17205.3153.7
Ideogram 4Base−0.595.8625.2020.69268.5229.0
SEGA0.826.0226.6422.31225.8160.9
AReA0.936.1327.9222.92235.9160.6

Our response continues in the next comment.

W2

Per-Head, Per-Block, and Per-Prompt Locality

Figure 3 averages attention locality over all heads, blocks, and prompts. To check that this behavior is consistent across them, we repeated the measurement for each one separately, on all 195 Aesthetic-4K prompts for FLUX.1 and Qwen-Image. For each (prompt, block, head), we take the attention-weighted query–key distance of all keys over all queries. We then compare this distance at the first and last denoising steps. If it is smaller at the last step, queries attend to closer keys at the end of denoising than at the start, so attention has become more local.

Change in attention locality per head, block, and prompt (1024×1024)

FLUX.1: became more localFLUX.1: shrinkQwen-Image: became more localQwen-Image: shrink
Heads (each head, in each block and prompt)93.0% (of 266,760)2.2×90.1% (of 280,800)2.6×
Blocks (mean over heads and prompts)57/571.9×59/602.3×
Prompts (mean over blocks and heads)195/1951.8×195/1952.0×

The behavior is consistent. 93.0% (FLUX.1) and 90.1% (Qwen-Image) of heads become more local, and so does every one of the 195 prompts in both models. The exceptions are few: 7.0% of FLUX.1 heads and 9.9% of Qwen-Image heads stay flat, and in Qwen-Image most of them are in the first block, which stays global. We will add these experiments and analysis, along with complementary plots, to the revised version.

Per-Block Window Schedule

Based on a per-block analysis of locality, we tested a per-block window schedule on FLUX.1 at 4096×4096, using a subset of 20 Aesthetic-4K prompts. Blocks that stay global or become less local than the others start with larger windows and shrink to a larger final size, while the more local blocks use smaller windows. We applied this to Phase II only.

ConfigLatency (s)↓IR↑Aesthetic↑MUSIQ↑PS↑HPS↑CLIP↑FID↓
Dynamic windows (per block)1551.296.1844.9923.120.28730.09297.9
Uniform windows (default)1511.306.2545.1723.250.29230.11286.7

The per-block schedule does not improve over the uniform one and is slightly slower. Only a few blocks behave differently, and most become local in a similar way, so a single schedule already fits them. We will add this experiment to the revised paper.


W3

Extended Attention-Locality Analysis

We thank the reviewer for this suggestion. To make the analysis more thorough, we expanded the same experiment to all 195 Aesthetic-4K prompts on both FLUX.1 and Qwen-Image. As in Figure 3 of the paper, for the center query, we measure the attention-weighted query–key distance at every denoising step. We measure it in two ways: over all keys, and over its dominant keys only. As shown in the table below, the behavior matches Figure 3 of the paper, and both models behave similarly. We will update Figure 3 in the revised paper.

Query–key distance in tokens, start → end of denoising (1024×1024)

ModelAll keysShrinkDominant keysShrink
FLUX.117.9 → 13.21.4×14.6 → 2.55.7×
Qwen-Image19.5 → 12.41.6×10.01 → 2.73.7×

The same behavior holds for every one of the 195 prompts in both models when measured separately (see our response to the previous point).

Regarding ResDiT, we agree that it is closely related, as it also combines global and local attention. Unfortunately, to the best of our knowledge, the code and implementation of ResDiT have not been publicly released, so we could not include it in our empirical comparison.


W5

LLM-as-Judge Evaluation of Structural Consistency

We thank the reviewer for this suggestion and agree that generic metrics do not capture structural failures well. They give a single overall score per image, so they do not measure specific errors such as wrong object counts, inconsistent spatial relations, duplicated objects, or poorly rendered text. To measure these errors directly, we added a blind evaluation in which a vision-language model (Claude Opus 5.5) acts as a judge. For each prompt, the judge sees the outputs of all methods side by side, blindly and in random order, so it cannot tell which method produced which image. Each image is shown in two forms. A downscaled view of the whole image is used to judge global composition and layout. Eight native-resolution crops, taken at the same fixed positions for every method, are used to judge fine detail. The judge scores each image from 1 to 10 against a fixed written rubric on aesthetic quality, detail fidelity, structural coherence, and absence of seams. It also scores object-count accuracy for prompts with at least two counted subjects, spatial-relation consistency for prompts that state a spatial relation, and text rendering for prompts that ask for text. In addition, it flags each image for object duplication, visible seams, and anatomy errors, which we report as rates.

We evaluate all 195 Aesthetic-4K prompts at 4096×2048, 4096×4096, and 5120×5120 on FLUX.1 and Qwen-Image, for Base, DyPE, SigMa, SEGA, and AReA (about 5,850 images). Scores are averaged over the prompts that all methods completed in each setting.

FLUX.1

Res.MethodText↑Count↑Spatial↑Aesth↑Detail↑Struct↑Dup%↓Seam%↓Anat%↓
4096×2048Base3.864.396.485.825.744.8164.62.615.9
DyPE4.006.078.537.527.056.8727.20.517.4
SigMa2.867.328.537.046.496.7014.43.624.1
SEGA3.718.168.747.767.057.6110.32.68.2
AReA6.298.348.628.137.387.995.62.63.7
4096×4096Base1.574.074.123.163.112.9556.937.431.3
DyPE3.005.688.306.425.516.5027.23.614.9
SigMa2.437.078.136.014.996.2211.813.818.5
SEGA2.867.338.597.366.577.3713.35.69.2
AReA5.148.648.747.576.887.6410.86.94.6
5120×5120Base1.001.972.151.651.561.7227.674.112.6
DyPE2.143.617.254.673.934.8441.431.635.1
SigMa1.436.006.032.912.114.2112.624.135.6
SEGA3.147.928.577.726.757.4715.52.312.1
AReA5.578.038.507.876.877.822.310.98.6

Qwen-Image

Res.MethodText↑Count↑Spatial↑Aesth↑Detail↑Struct↑Dup%↓Seam%↓Anat%↓
4096×2048Base5.714.097.316.416.125.2061.54.610.3
DyPE3.718.267.916.255.166.4812.819.512.3
SigMa5.867.658.677.626.227.4215.92.64.1
SEGA4.438.238.728.106.927.8810.20.54.1
AReA5.718.789.108.277.137.968.33.62.6
4096×4096Base5.144.055.544.274.153.6371.133.018.6
DyPE2.576.866.273.982.724.959.822.227.3
SigMa5.577.288.457.034.907.406.24.110.3
SEGA3.867.678.638.226.867.987.21.56.2
AReA6.008.589.168.156.888.095.76.23.2
5120×5120Base3.004.295.364.143.664.3553.628.210.9
DyPE1.003.923.532.061.352.8525.537.331.8
SigMa3.207.836.855.593.766.279.17.314.5
SEGA2.608.177.888.016.407.469.16.614.5
AReA6.208.668.927.836.387.498.210.74.9

To check that the judge’s scores are consistent and not random, we added a second, identical copy of one method’s images to the evaluation, under a different anonymous label. The judge gave both copies exactly the same scores. If the reviewer considers a user study necessary, we will add one. We will also add this evaluation and the full rubric to the revised paper.


Our response continues in the next comment.

Comments on the paper

1 comment

Dear AC and Reviewers,

We thank the reviewers for their time and constructive feedback. We have uploaded a revised paper that includes the changes from our responses, highlighted in the text, along with new figures and tables. Below, we summarize the main concerns and how we addressed them:

  1. Contribution and related work. We clarified AReA’s core contribution and its differences from the suggested prior work.
  2. Extended attention analysis. We extended the attention-locality analysis across models, heads, blocks, and prompts, and added controlled experiments separating the effects of token count and positional distance, along with an attention-entropy analysis.
  3. Extended evaluation. We added quantitative results at higher resolution on more backbones, and a blind LLM-as-judge evaluation that directly measures structural errors, such as duplication, seams, and counting errors, across methods and ablations.
  4. Design choices and robustness. We added compute-matched comparisons, a per-block window schedule, hyperparameter sweeps, cross-model ablations, and an ablation separating coordinate remapping from KV subsampling.
  5. Efficiency and clarity. We reported memory, compute, and runtime overhead, clarified how RoPE coordinates are handled, and improved the figures and method description.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.