- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Foveated Diffusion, which generates high-resolution tokens only within the foveation mask and low-resolution tokens for the peripheral. This token mixing strategy saves computation by exploiting human vision and is applicable to many interaction applications. The work proposes ways to combine mixed-resolution token sequences, to modify RoPE, and to post train on existing models. Experiments on FLUX and Wan models show efficacy of the proposed method.
Critical strengths
- Problem is well-defined and useful in practice.
- Method is sound and straightforward.
- Experimentation is conducted on both T2I and T2V generation.
Critical weaknesses
- Some parts of the method is underspecified. Please see my questions below.
Questions for authors
Questions on the technical implementation.
-
The method does not seem to add embeddings to DiT to differentiate high and low-resolution tokens. The foveation mask is not provided to the DiT either. The RoPE embedding only aligns the position of the tokens but does not indicate whether the token is suppose to be high-resolution or low. Then, how does the DiT model knows which token is supposed to be high or low-resolution at generation time? Have the author conducted any ablation studies on the architecture design? e.g. condition on the mask or include token embeddings?
-
The paper does not adequately explain how partial high/low-resolution tokens are passed through the VAE for decoding at inference time. Do you set missing tokens to zeros? Or do you repeat from low-resolution or pool from high-resolution? This part of the operation is very underspecified in the paper.
-
Can author specify what downsample/upsample method is used for training and inference? Bilinear? Bicubic?
Questions on the user experience.
- The visualization images/videos provided in the paper and supplementary materials all have a white circle indicating the mask location. However, I hides how smoothly high-resolution area blends with the low-resolution area. Can the authors comment on the if the current method blends well between different resolution areas for human perception?
The authors have addressed most of my concerns.
After carefully considering the rebuttal alongside other reviews, although generation quality remains a minor concern, I am inclined to raise my score. This direction might hold promising potential, particularly for extensions into streaming video generation.