- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies Full Attention Residuals in pixel-space JiT flow Transformers, replacing fixed residual accumulation with attention residuals. Matched ImageNet 256×256 experiments show improved FDr6 and modest FID gains. The authors also demonstrate zero-query checkpoint conversion followed by continued training, and analyze the role of condition-token history and flow-time-dependent depth routing. The contribution is an empirical evaluation of an existing operator in image generation.
Critical strengths
-
Controlled evidence across model scales and evaluation representations. The paper isolates the residual aggregation mechanism through matched JiT backbones and training settings. Full AttnRes reduces FDr6 by 6.8% at B/16 and 9.3% at L/16, with improvements in all six feature spaces. Although the FID gains are modest, these consistent results provide useful evidence that depth-wise retrieval benefits pixel-space generation, supporting the value of the empirical contribution.
-
A practical checkpoint-conversion strategy with a meaningful control. Zero-query initialization enables introducing Full AttnRes into an existing checkpoint, with approximate function preservation under explicit normalization assumptions. Starting from the same checkpoint, conversion outperforms equal-budget standard-residual continuation in both FID and IS. This strengthens the paper by showing that the method’s benefits can extend to continued training without requiring training from scratch.
Critical weaknesses
-
Limited methodological novelty. The paper applies the existing Full AttnRes operator to pixel-space JiT models without introducing a new method. While the controlled experiments are useful, the findings primarily establish applicability to a new setting. The limited methodological novelty raises the bar for the empirical insights, which remain insufficient to make a strong case for acceptance.
-
Incomplete evaluation of practical efficiency. The paper reports small parameter and FLOP overheads but does not measure actual runtime latency. These measurements are particularly consequential for Full AttnRes. Moreover, the absence of a Block AttnRes comparison leaves unclear whether full-history retrieval offers a favorable quality–efficiency trade-off or whether a more efficient variant could achieve similar gains.
Questions for authors
-
What insights beyond applying Full AttnRes to a new setting distinguish this study from prior work?
-
Could you report training/inference latency and compare Full AttnRes with Block AttnRes to clarify the quality–efficiency trade-off?