acceptodds
Under review as a conference paper at ICLR 2027

Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation

Abstract

Diffusion and flow matching models have unlocked unprecedented capabilities for creative content creation with photorealistic image and video generation. The growing demand for higher resolutions, frame rates, and context lengths, however, makes efficient generation increasingly challenging, as computational complexity grows quadratically with the number of generated tokens. Our work seeks to optimize the efficiency of the generation process in settings where the user can specify the region of interest. In these settings, we leverage the eccentricity-dependent acuity of human vision: while a user perceives very high-resolution visual information in a small region around their gaze location (the foveal region), the ability to resolve detail quickly degrades in the periphery of the visual field. Our approach starts with a mask modeling the foveated resolution to allocate tokens non-uniformly, assigning higher token density to foveal regions and lower density to peripheral regions. An image or video is generated natively in a mixed-resolution token setting, yielding results perceptually indistinguishable from full-resolution generation, while drastically reducing the token count and generation time. To this end, we develop a principled mechanism for constructing mixed-resolution tokens directly from high-resolution data, allowing a foveated diffusion model to be post-trained from an existing base model while maintaining content consistency across resolutions. We validate our approach through extensive analysis and a carefully designed user study, demonstrating the efficacy of foveation as a practical and scalable axis for efficient generation

Primary area:
generative models
Keywords:
diffusion models, mixed-resolution diffusion, adaptive tokenization, foveation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qyydOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Foveated Diffusion, which generates high-resolution tokens only within the foveation mask and low-resolution tokens for the peripheral. This token mixing strategy saves computation by exploiting human vision and is applicable to many interaction applications. The work proposes ways to combine mixed-resolution token sequences, to modify RoPE, and to post train on existing models. Experiments on FLUX and Wan models show efficacy of the proposed method.

Critical strengths
  1. Problem is well-defined and useful in practice.
  2. Method is sound and straightforward.
  3. Experimentation is conducted on both T2I and T2V generation.
Critical weaknesses
  1. Some parts of the method is underspecified. Please see my questions below.
Questions for authors

Questions on the technical implementation.

  1. The method does not seem to add embeddings to DiT to differentiate high and low-resolution tokens. The foveation mask is not provided to the DiT either. The RoPE embedding only aligns the position of the tokens but does not indicate whether the token is suppose to be high-resolution or low. Then, how does the DiT model knows which token is supposed to be high or low-resolution at generation time? Have the author conducted any ablation studies on the architecture design? e.g. condition on the mask or include token embeddings?

  2. The paper does not adequately explain how partial high/low-resolution tokens are passed through the VAE for decoding at inference time. Do you set missing tokens to zeros? Or do you repeat from low-resolution or pool from high-resolution? This part of the operation is very underspecified in the paper.

  3. Can author specify what downsample/upsample method is used for training and inference? Bilinear? Bicubic?

Questions on the user experience.

  1. The visualization images/videos provided in the paper and supplementary materials all have a white circle indicating the mask location. However, I hides how smoothly high-resolution area blends with the low-resolution area. Can the authors comment on the if the current method blends well between different resolution areas for human perception?
Reviewer_KuKoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Foveated Diffusion, which employs a mask to model foveated resolution, allowing for the non-uniform allocation of tokens—assigning higher token density to foveal (central) regions and lower density to peripheral regions. Experiments demonstrate the effectiveness of this approach. While the concept is novel, I have significant concerns regarding human-computer interaction, as well as the method's scalability for large-scale data training.

Critical strengths
  1. An interesting aspect of this work is that it applies targeted enhancements to regions of human interest, while simultaneously accelerating the generation of images and videos.

  2. The authors provide experimental results for both images and videos to validate the effectiveness of the proposed method.

Critical weaknesses
  1. The premise of this approach is somewhat puzzling, as it appears to necessitate a two-stage generation process, which entails higher computational costs. Furthermore, since "regions of interest" vary from person to person, it is unclear how the method ensures both diversity and the overall effectiveness of the generation.

  2. Additionally, the proposed method requires fine-tuning on specific datasets, making it an unfair comparison against training-free approaches like ToMe (Token Merging).

  3. Some of the visualizations in Figure 14 (in the appendix) appear anomalous; for instance, in the second row, the "foveal" (high-density) region does not seem to be centered on the candle.

Questions for authors

How does the variability in human eye-movement patterns (i.e., different individuals focusing on different areas) affect the outcome? While the generation speed is improved, does this come at the cost of a decline in image quality? (Please also refer to the points raised under "Weaknesses.")

2 comments

The authors have addressed most of my concerns.

After carefully considering the rebuttal alongside other reviews, although generation quality remains a minor concern, I am inclined to raise my score. This direction might hold promising potential, particularly for extensions into streaming video generation.

We highly appreciate the reviewer's support to accept this paper. We agree that our method builds the foundation for streaming video generation, which we validate as a proof-of-concept using distilled image model experiments in Section C.3. Streaming video generation produces video frames chunk by chunk; image generation can be viewed as the special case where the chunk size is one. Our method accelerates distilled image models by up to 2x, with latency below one second, while maintaining comparable image quality:

MethodToken RatioImage-Reward ↑HPS v2.1 ↑HPS v3 ↑CLIP Score ↑LAION Aes. ↑FID ↓LPIPS ↓Time / Speedup
Full high-res100%1.0630.28811.460.8465.54712.340.7061.83s / 1.00×
Ours54%1.0270.28611.530.8535.55612.520.7171.30s / 1.41×
Ours32%1.0550.28612.800.8565.56113.210.7231.00s / 1.83×
Ours26%1.0740.28713.120.8575.57713.820.7250.92s / 1.99×
Reviewer_7R3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Foveated Diffusion for efficient image and video generation. Inspired by eccentricity-dependent nature of human vision, it uses more tokens in high-resolution regions, and fewer tokens in periphery. Mixed-resolution tokenization and post-training are also proposed to improve consistency. Experiments show that the propsed Foveated Diffusion achieves 2x speedup for images and 3x for videos

Critical strengths
  1. Reducing tokens is necessary in image and video generation, which can save computation.
  2. The proposed method can be applied to both image and video models. Experiments show the speedup for both tasks.
Critical weaknesses
  1. The method assumes that only the foveal region needs high-resolution generation, while the peripheral regions can be generated at lower resolution. However, important information can appear anywhere in the scene, not just near the gaze center. Losing details in these regions may affect the usefulness of the generated content. This is a fundamental limitation of the proposed method, and the paper does not provide enough experiments to study this trade-off.
  2. In the experiment part, there is no clear ablation of the main components, such as the modified RoPE. It is hard to know which part is important.In Table 2, video imaging quality drops from 0.6411 to 0.5906. Dynamic Degree also drops from 0.2400 to 0.1750. The effect on video quality needs more analysis.
Questions for authors

In Table 2, video imaging quality drops from 0.6411 to 0.5906. Dynamic Degree also drops from 0.2400 to 0.1750. The effect on video quality needs more analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.