- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AReA, a training-free attention adaptation method for high-resolution diffusion transformers. Its central idea is to decouple output resolution from the resolution and spatial extent of the attention context. During early denoising, full-resolution queries attend to spatially subsampled keys and values to establish global composition. Later steps use progressively shrinking local windows, with wavelet-guided splitting further reducing computation in smooth regions. Overlapping aggregation, halo context, and a global-KV bridge mitigate boundary artifacts and preserve cross-region consistency.
Critical strengths
1.A coherent organizing principle. Adapting the attention context to the changing requirements of denoising provides a unified explanation for both computational savings and the balance between global composition and local detail.
2.A meaningful decoupling of resolutions. Maintaining full-resolution queries allows high-resolution states to evolve directly, rather than inheriting a fixed composition from an independently generated low-resolution image. Computation is reduced by restricting contextual access rather than reducing the resolution of the generation state.
3.Substantial design validation. The paper includes phase replacement, fixed-window, pooling, switching-time, component-removal, and low-resolution bootstrap comparisons. The DWT ablation provides particularly interpretable evidence of computational savings with approximately preserved quality.
Critical weaknesses
1.The distinctive contribution is not fully separated from the supporting components. The method combines subsampled KV, window scheduling, wavelet analysis, halo context, and global tokens. Existing removal ablations are useful, but compute-matched comparisons against alternative schedules and random or uniform splitting would better establish the value of the proposed temporal and content-adaptive principles.
2.Positional adaptation and receptive-field adaptation remain confounded. Section 3.2 explicitly rescales query and key coordinates, whereas the paper also describes the method as not altering positional encodings. Preserving the RoPE frequency formula does not preserve the resulting positional representation. Separate interventions on coordinates and KV counts are needed to identify the source of structural improvements. (pp. 5, 8)
3.The central structural-quality claims require more direct evaluation. Fixed windows achieve comparable or higher scores on some generic metrics despite visibly fragmented outputs. This suggests that duplication rates, counting errors, seam frequency, or blinded structural assessments would be more informative than additional aggregate quality scores. (Table 3; Figures 7 and 13)
4.The causal interpretation is stronger than the supporting evidence. Increasing attention locality motivates the method, but does not directly determine the proposed window schedule. The out-of-distribution exposure analysis provides useful associations without independently establishing the effects of token count and positional distance on generation failures
Questions for authors
1.Contribution of adaptive scheduling. Could you clarify whether the proposed window schedule and DWT-guided splitting outperform simpler alternatives at comparable computational budgets? Existing component-removal ablations change both the allocation strategy and the amount of computation, making these effects difficult to separate.
2.Role of positional coordinates. Section 3.2 rescales Q/K coordinates, whereas the paper also states that positional encodings are unchanged. Does “unchanged” refer specifically to the RoPE frequency formula? Are query coordinates continuously scaled or quantized to the coarse grid? Can the effects of coordinate rescaling and KV subsampling be distinguished?
3.Structural-quality assessment. Table 3 shows that some simplified variants remain competitive on generic quality metrics despite the structural failures illustrated in the figures. Do you have systematic evidence on duplication, counting errors, or window-boundary artifacts that establishes how frequently these failures occur?
4.Generality of the locality argument. Figure 3 aggregates attention across heads, blocks, and prompts. How heterogeneous is the observed global-to-local transition, and are there identifiable heads or prompt types that continue to require substantial long-range interaction during late denoising? How effectively does the global-KV bridge accommodate these cases?
6 comments
W3 / Q3
LLM-as-Judge Evaluation of Structural Consistency
We thank the reviewer for this suggestion. We agree that generic metrics miss structural failures; as the reviewer notes, fixed windows score well on them despite fragmented outputs. To measure these errors directly, we added a blind evaluation in which a vision-language model (Claude Opus 5.5) acts as a judge. It scores text rendering, object counts, spatial relations, and structural coherence, and reports duplication, seam, and anatomy-error rates on both backbones at three resolutions. AReA ranks first or second on text, count, spatial relations, aesthetics, detail, and structure in every setting, and has the lowest anatomy-error rate in all 6 settings. If the reviewer considers a user study necessary, we will add one. Please see our response to Reviewer AykN for the full protocol and results.
W4
Window Schedule Design
The window schedule is shaped by the compute budget, the model’s native resolution, and the observed locality of attention. The start window is the largest window still cheaper than full attention, and the final window is the model’s native size. Between them, the schedule follows attention locality. As Figure 3 shows, attention narrows quickly in early steps and more slowly later, so in our scheduler windows shrink the same way (Figure 11). To test this choice, we compared it with other schedules on FLUX.1 at 4096×4096:
Phase II window size schedule
| Config | Latency (s)↓ | IR↑ | Aesthetic↑ | MUSIQ↑ | PS↑ | HPS↑ | CLIP↑ | FID↓ |
|---|---|---|---|---|---|---|---|---|
| reverse | 151 | 1.27 | 6.46 | 45.39 | 23.05 | 0.291 | 29.60 | 151.5 |
| linear | 158 | 1.26 | 6.34 | 45.38 | 23.21 | 0.287 | 29.42 | 152.6 |
| late shrinking | 159 | 1.26 | 6.34 | 45.31 | 23.20 | 0.287 | 29.44 | 152.7 |
| early shrinking (default) | 151 | 1.32 | 6.37 | 45.22 | 23.31 | 0.297 | 29.71 | 148.50 |
Reverse uses the same window sizes in reverse order, so windows grow over time. Linear shrinks windows by the same amount at every step. Late shrinking keeps windows large for longer and shrinks them quickly near the end. Reverse improves some perceptual metrics, but all three alternatives lose structural coherence, with lower ImageReward and higher FID. Following how attention narrows gives the best results overall.
Separating Out-of-Distribution Causes
To separate the effects of token count and positional distance, we ran two controlled experiments on FLUX.1, each changing only the token count or only the distance. To test unseen token counts alone, we generate at 4096×4096 with NTK-scaled RoPE. This maps query–key distances back into the training range, while every query still attends to 16× more tokens than in training. To test unseen distances alone, we generate at the native 1024×1024 with inverse NTK scaling, which keeps the token count the same as in training but makes the model see distances as if the canvas were 4096×4096.
Unseen token counts only: NTK-scaled RoPE (FLUX.1, 4096×4096, all 195 Aesthetic-4K prompts)
| Config | IR↑ | Aesthetic↑ | MUSIQ↑ | PS↑ | HPS↑ | CLIP↑ | FID↓ |
|---|---|---|---|---|---|---|---|
| Base | −0.72 | 5.80 | 22.55 | 20.34 | 0.211 | 25.72 | 179.8 |
| unseen tokens | −0.29 | 5.80 | 23.86 | 20.82 | 0.198 | 25.52 | 182.3 |
| AReA | 1.32 | 6.37 | 45.22 | 23.31 | 0.297 | 29.71 | 148.50 |
Unseen distances only: inverse NTK scaling (FLUX.1, 1024×1024, all 195 Aesthetic-4K prompts)
| Config | IR↑ | Aesthetic↑ | MUSIQ↑ | PS↑ | HPS↑ | CLIP↑ | FID↓ |
|---|---|---|---|---|---|---|---|
| unseen distances | 0.99 | 6.35 | 68.59 | 22.49 | 0.288 | 29.03 | 156.5 |
| Base | 1.40 | 6.41 | 68.85 | 23.58 | 0.307 | 29.66 | 151.5 |
Each cause alone degrades generation, with a distinct failure mode. With unseen token counts, attention spreads over too many keys, and outputs become washed out and lose high-frequency detail, even though distances are within range. With unseen distances, the model produces duplicated subjects and low-frequency artifacts, even at the native token count. AReA mitigates both causes, since in both phases it keeps the number of keys and the query–key distances close to the training range (Appendix G). We will add these experiments, along with qualitative comparisons, to the revised paper.
Our response continues in the next comment.
Q4
Consistency of Attention Locality and the Global-KV Bridge
We thank the reviewer for this question. To measure heterogeneity, we measured how attention locality changes during denoising separately for every head, block, and prompt instead of averaging them, using all 195 Aesthetic-4K prompts on FLUX.1 and Qwen-Image (see also our response to Reviewer AykN). For each (prompt, block, head), we take the attention-weighted query–key distance of all keys over all queries and compare it at the first and last denoising steps. If it is smaller at the last step, attention has become more local.
Change in attention locality per head, block, and prompt (1024×1024)
| FLUX.1: became more local | FLUX.1: shrink | Qwen-Image: became more local | Qwen-Image: shrink | |
|---|---|---|---|---|
| Heads (each head, in each block and prompt) | 93.0% (of 266,760) | 2.2× | 90.1% (of 280,800) | 2.6× |
| Blocks (mean over heads and prompts) | 57/57 | 1.9× | 59/60 | 2.3× |
| Prompts (mean over blocks and heads) | 195/195 | 1.8× | 195/195 | 2.0× |
The behavior is consistent. 93.0% (FLUX.1) and 90.1% (Qwen-Image) of heads become more local, and so does every one of the 195 prompts in both models. The exceptions are few: 7.0% of FLUX.1 heads and 9.9% of Qwen-Image heads stay flat, and in Qwen-Image most of them are in the first block, which stays global. We will add these experiments and analysis, along with complementary plots, to the revised version.
Thus, no prompt type keeps long-range attention late in denoising, and the few heads that do are mostly in the first block of Qwen-Image and scattered across blocks in FLUX.1.
The global-KV bridge gives each window’s queries a summary of the most important tokens outside the window, to prevent structural divergence and duplicated content. A token’s importance is how far its key deviates from the mean key of the image. As Figure 3 and the analysis above show, attention becomes more local in most blocks, especially over dominant keys. In full attention, however, queries still attend to all tokens, only with much smaller weights than to their neighbors. The bridge approximates this at low cost. It also serves these remaining global heads, since every window keeps access to the most important tokens across the image; without it (ρ = 0), objects are duplicated (tables below). By default, it selects tokens in a stratified way over an 8×8 grid, so that the selected tokens do not all come from a small region. To ablate this, we compare it with top-k selection, which ranks all tokens outside the window by importance without stratification and keeps the same number of tokens. We also vary ρ, which sets how many bridge tokens each window receives.
Global-KV token selection (FLUX.1, 4096×4096)
| Config | Latency (s)↓ | IR↑ | Aesthetic↑ | MUSIQ↑ | PS↑ | HPS↑ | CLIP↑ | FID↓ |
|---|---|---|---|---|---|---|---|---|
| top-k | 150 | 1.31 | 6.33 | 45.39 | 23.26 | 0.291 | 29.62 | 149 |
| stratified (default) | 151 | 1.32 | 6.37 | 45.22 | 23.31 | 0.297 | 29.71 | 148.50 |
Global-KV bridge ratio ρ (FLUX.1, 4096×4096)
| Config | Latency (s)↓ | IR↑ | Aesthetic↑ | MUSIQ↑ | PS↑ | HPS↑ | CLIP↑ | FID↓ |
|---|---|---|---|---|---|---|---|---|
| ρ = 0 | 133 | 1.32 | 6.25 | 45.38 | 23.25 | 0.294 | 29.11 | 161.25 |
| ρ = 0.25 | 141 | 1.32 | 6.35 | 44.52 | 23.23 | 0.296 | 29.70 | 148.8 |
| ρ = 0.5 (default) | 151 | 1.32 | 6.37 | 45.22 | 23.31 | 0.297 | 29.71 | 148.50 |
| ρ = 0.75 | 168 | 1.24 | 6.32 | 44.76 | 23.14 | 0.292 | 29.35 | 154.2 |
| ρ = 1 | 189 | 1.21 | 6.32 | 44.16 | 23.04 | 0.290 | 29.18 | 154.4 |
Stratified selection performs better than top-k on most metrics, since it distributes the bridge budget across the canvas. Increasing ρ slows inference and softens detail, as each query spreads its attention over more tokens. Removing the bridge (ρ = 0) causes duplicated objects, raising FID from 148.5 to 161.3, while results barely change between ρ = 0.25 and 0.5. We will add these analyses to the revised paper.
Thank you again for your thoughtful and constructive comments. We sincerely appreciate the opportunity to clarify and discuss any further questions.
We thank the reviewer for the constructive feedback and are glad they found the core idea coherent, the design meaningful, and the ablations thorough. Please find our detailed responses below; we would be happy to engage in further discussion if needed.
W1 / Q1
Compute-Matched Comparison of Window Schedules and Splitting Strategies
To further establish the value of the proposed temporal and content-adaptive principles, we extended our ablations with compute-matched comparisons against alternative and simpler approaches for the window schedule and the splitting signal. We compare the window schedule and the splitting strategy against alternatives at equal or higher compute on FLUX.1 at 4096×4096 using all 195 Aesthetic-4K prompts.
The default schedule shrinks windows quickly at first and slowly near the end, following how attention narrows during denoising (Figures 3 and 11). We compare it with three alternatives: reverse, which uses the same window sizes in reverse order, so windows grow over time; linear, which shrinks windows by the same amount at every step; and late shrinking, which keeps windows large for longer and shrinks them quickly near the end. All three use the same or more compute but lower ImageReward and raise FID.
All strategies run at the same cost and differ only in which windows are split. Uniform splitting selects a set of windows once and keeps splitting the same windows until the end of denoising. Random splitting selects new windows at random at each step. Variance splitting ranks windows by the variance of their tokens instead of the DWT map. Uniform and random splitting ignore image content and can split windows over detailed regions, while variance also reacts to smooth changes. DWT-guided splitting performs best overall.
W2 / Q2
Clarification on RoPE and Positional Coordinates
We thank the reviewer for this careful observation, which is correct. By “unchanged,” we refer specifically to the RoPE frequency formula. AReA does not rescale or recalibrate the rotary frequencies and requires no RoPE length extrapolation. However, in Phase I it does remap the token coordinates used by RoPE. To be more specific, in Phase I query coordinates are quantized to the coarse grid, not continuously scaled: each f × f block of queries is assigned and shares the position of its pooled key on the coarse grid, while in Phase II all tokens keep their original coordinates. We will fix and clarify this in the revised version of the paper.
Separating Coordinate Remapping and KV Subsampling
The two effects can be separated. To isolate the coordinate remapping, we ran Phase I with the same key–value pooling but without remapping the query coordinates, so that queries keep their full-resolution positions while attending to the pooled keys. Removing the remapping alone causes structural collapse (table below). The remapping is therefore needed: it places the queries on the same coarse grid as the pooled keys.
The KV subsampling follows from what the early denoising steps need. In these steps, attending to all high-resolution tokens raises attention entropy beyond the training range and degrades quality (see our response to Reviewer FzqH on attention entropy). These steps mainly form the low-frequency layout, which does not require every token, while full attention over all tokens is costly. Pooling the keys and values therefore keeps the number of keys close to the training range and reduces computation, while keeping the information needed to form the layout.
Our response continues in the next comment.