acceptodds
Withdrawn from ICLR 2027

Attention Residuals for Pixel-Space Flow Transformers: An Empirical Study

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Attention Residuals (AttnRes) replace fixed residual accumulation with content- dependent retrieval over earlier residual increments. We study the published Full AttnRes operator in class-conditional, pixel-space JiT flow Transformers under matched backbone and training settings. On ImageNet 256×256, Full AttnRes reduces six-encoder FDr6 from 15.80 to 14.72 at B/16 (6.8%) and from 11.92 to 10.81 at L/16 (9.3%), with lower ratios in all six representation spaces at both scales. Learned parameters increase by approximately 0.01% and profiled infer- ence FLOPs by below 1%, although full-history storage and mixing remain im- portant systems costs. We also insert zero-query Full AttnRes into a shared 100- epoch standard-residual checkpoint. Equal-budget continuation raises IS from 224.6 for unchanged continuation to 230.6 after conversion. We explain approx- imate function preservation under explicit pre-normalization assumptions, local- ize most of an ordered ablation’s endpoint gap to condition-token history, and observe flow-time-dependent depth routing. These results support studying Full AttnRes both from scratch and as a continued-training feature, while highlighting representation-dependent effect sizes and the need for repeated-run and systems evaluation.

Primary area:
generative models
Keywords:
Diffusion Model, Attention Residual, Architecture

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wMiROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies Full Attention Residuals in pixel-space JiT flow Transformers, replacing fixed residual accumulation with attention residuals. Matched ImageNet 256×256 experiments show improved FDr6 and modest FID gains. The authors also demonstrate zero-query checkpoint conversion followed by continued training, and analyze the role of condition-token history and flow-time-dependent depth routing. The contribution is an empirical evaluation of an existing operator in image generation.

Critical strengths
  1. Controlled evidence across model scales and evaluation representations. The paper isolates the residual aggregation mechanism through matched JiT backbones and training settings. Full AttnRes reduces FDr6 by 6.8% at B/16 and 9.3% at L/16, with improvements in all six feature spaces. Although the FID gains are modest, these consistent results provide useful evidence that depth-wise retrieval benefits pixel-space generation, supporting the value of the empirical contribution.

  2. A practical checkpoint-conversion strategy with a meaningful control. Zero-query initialization enables introducing Full AttnRes into an existing checkpoint, with approximate function preservation under explicit normalization assumptions. Starting from the same checkpoint, conversion outperforms equal-budget standard-residual continuation in both FID and IS. This strengthens the paper by showing that the method’s benefits can extend to continued training without requiring training from scratch.

Critical weaknesses
  1. Limited methodological novelty. The paper applies the existing Full AttnRes operator to pixel-space JiT models without introducing a new method. While the controlled experiments are useful, the findings primarily establish applicability to a new setting. The limited methodological novelty raises the bar for the empirical insights, which remain insufficient to make a strong case for acceptance.

  2. Incomplete evaluation of practical efficiency. The paper reports small parameter and FLOP overheads but does not measure actual runtime latency. These measurements are particularly consequential for Full AttnRes. Moreover, the absence of a Block AttnRes comparison leaves unclear whether full-history retrieval offers a favorable quality–efficiency trade-off or whether a more efficient variant could achieve similar gains.

Questions for authors
  1. What insights beyond applying Full AttnRes to a new setting distinguish this study from prior work?

  2. Could you report training/inference latency and compare Full AttnRes with Block AttnRes to clarify the quality–efficiency trade-off?

Reviewer_WXUHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the existing Full Attention Residuals operator in class-conditional, pixel-space JiT flow transformers. Instead of summing residual increments with fixed weights, each sublayer retrieves a learned, content-dependent mixture of earlier increments. Image and conditioning-prefix tokens maintain separate histories. Matched ImageNet-256 experiments show reductions in the six-encoder FDr₆ metric at B/16 and L/16, alongside smaller FID improvements. The paper also studies inserting zero-initialized attention-residual modules into a pretrained checkpoint, the contribution of conditioning-token history, and changes in depth-routing weights across flow time.

Critical strengths
  1. The principal comparisons control the backbone and training protocol, and the conversion experiment uses an appropriate continuation baseline. The from-scratch models receive matched 600-epoch training. For conversion, both variants start from the same 100-epoch checkpoint and receive another 100 epochs, including the same optimizer-reset protocol. Conversion improves FID from 5.22 to 5.06 relative to unchanged continuation. This is substantially more informative than comparing conversion only with the unmodified starting checkpoint.

  2. The improvement is observed across multiple evaluation representations. FDr₆ decreases from 15.80 to 14.72 at B/16 and from 11.92 to 10.81 at L/16, with lower scores in all six constituent spaces. The conditioning-history ablation also identifies a potentially useful architectural detail: retaining prefix-token history accounts for most of the improvement along the evaluated ablation sequence. The authors appropriately avoid treating this sequence as a causal decomposition.

Critical weaknesses
  1. The incremental contribution beyond existing depth-routing work is not sufficiently established. The paper explicitly applies the published Full AttnRes operator rather than introducing a new mechanism. Closely related "Diffusion-Adaptive Routing" (https://arxiv.org/abs/2605.20708) work already studies residual-history aggregation in diffusion transformers, query parameterizations, timestep dependence, and chunked aggregation. Nevertheless, the experiments compare only against standard residual addition. For an empirical contribution, matched comparisons with an existing diffusion-routing method, Block AttnRes, or simpler learned depth weights would help establish what is specifically valuable about full-history retrieval in pixel-space JiT. The analysis currently remains descriptive: changing routing weights with flow time does not establish that this adaptation causes the quality improvement. The condition-history experiment also omits a condition-only variant, preventing separation of main effects and interactions. These limitations are consequential because the study covers only one dataset, resolution, conditioning regime, and backbone family.

  2. The reliability and practical value of the gains remain unclear. FID improves from 4.08 to 3.95 at B/16, but only from 2.85 to 2.84 at L/16. The larger FDr₆ improvements are encouraging, yet all results are point estimates without uncertainty across independent training runs. Agreement among correlated encoders does not replace evidence of reproducibility. Meanwhile, Full AttnRes stores the complete residual history and repeatedly mixes it, introducing depth-dependent activation storage and memory traffic. The reported parameter increase and sub-1% profiled FLOP overhead do not establish low runtime or memory overhead. Without training throughput, peak memory, and sampling-latency measurements, it is difficult to judge whether the observed quality improvements justify the implementation cost. Comparison against cheaper history aggregation is particularly important here.

Questions for authors
  1. How reproducible are the B/16, L/16, and conversion improvements across independent training runs? Please distinguish variation from training seeds from variation caused by finite generated evaluation samples.

  2. Under the same JiT setup, how does Full AttnRes compare with Block AttnRes, simple learned depth weighting, and a diffusion-specific alternative such as DAR? What benefit requires full-history, content-dependent retrieval?

  3. What are the measured training throughput, peak memory, and sampling latency at matched hardware, precision, and batch size? How do these costs scale from B/16 to L/16?

  4. Does a condition-token-only history variant recover most of the improvement? This would help determine whether full image-token history is necessary.

  5. How closely does conversion preserve predictions immediately after insertion under the actual normalization epsilon and numerical precision? Please quantify prediction discrepancies before continued training.

Reviewer_HKMKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate the utility of AttnRes in JiT-style flow Transformers for image generation. Motivated by its reported benefits in NLP, they extend the operator to the visual domain and evaluate whether it improves generation quality. All experiments are conducted on class-conditional ImageNet generation.

Critical strengths

The authors aim to minimize experimental variability when comparing their method with existing approaches and provide several intuitive ablation studies.

Critical weaknesses

Below I list my main concerns:

  • Marginal improvements: FID remains practically unchanged, while FDr6 improves by only approximately one point on average. Given the inherent variability of these metrics, the results do not provide clear evidence of meaningful benefits from the AttnRes operator.

  • Limited novelty: The paper introduces no substantial technical innovations or new insights. Combined with the marginal empirical improvements, this makes the work appear insufficiently novel for an ICLR publication.

  • Restricted experimental scope: The evaluation is limited to ImageNet and JiT-style models, and the results appear to be based on single runs. Consequently, it is difficult to assess the generalizability and variability of the proposed approach.

  • Incomplete computational analysis: The computational analysis focuses on parameter count and FLOPs but does not report wall-clock runtime or memory footprint -- measurements particularly important from a practitioner’s perspective.

  • Writing and presentation: The paper would benefit from substantial improvements in writing and organization. The narrative does not flow smoothly, which makes it difficult for the reader to follow the work.

Questions for authors

Is there a particular reason why the evaluation is restricted to JiT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.