acceptodds
Under review as a conference paper at ICLR 2027

Level of Token Diffusion

Abstract

Image and video diffusion models allocate equal computation to every region, even when the intended scene calls for varying levels of detail. The spatial distribution of detail can often be anticipated before generation, indicating where computation can be reduced. We introduce Level-of-Token (LoT) Diffusion, a framework that turns this knowledge into an explicit multiresolution token layout (Level-of-Token layout) for adaptive and efficient generation. Tokens represent rectangular patches of varying sizes and shapes, allocating finer tokens where detail is needed and coarser tokens elsewhere. We adapt pretrained diffusion transformers to LoT layouts through a patch-wise asymmetric flow parametrization and embeddings for multiresolution tokens, preserving full-resolution flow prediction at every denoising step while processing only a reduced token sequence. LoT Diffusion enables layout-adaptive generation while preserving pretrained generative priors. We demonstrate LoT with layouts derived from semantic masks, bounding boxes, texture variance, and depth-of-field cues, as well as agentic plans. Across image and video generation, LoT offers favorable quality–efficiency tradeoffs, with significant speedups determined by the layout’s token budget. Additional qualitative results and interactive demonstrations are available at https://iclr-submission7014.github.io/supplementary/

Primary area:
generative models
Keywords:
image generation, video generation, efficient diffusion model, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kyS2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper reduces the cost of image and video generation by allocating tokens according to the detail needed in different regions. It uses rectangular tokens of different sizes and an asymmetric flow parameterization to predict a full-resolution latent field from a shorter token sequence. The method is adapted to pretrained FLUX.2 and Wan2.1 models through fine-tuning. The experiments compare generation speed and quality at different token budgets and demonstrate layouts derived from bounding boxes, segmentation masks, and depth. The largest reported speedups are 2.04× for images and 3.53× for videos, excluding the cost of obtaining the layouts.

Critical strengths
  1. The method in Section 4 provides a practical way to support heterogeneous token layouts in pretrained models. It addresses patch embedding, full-resolution prediction, and initialization together while retaining the existing VAE. Being able to adjust the local token budget without training a new generative model from scratch makes the approach useful.

  2. Tables 1–2 show encouraging results on both image and video generation. At the evaluated budgets, LoT generally offers a better quality–speed tradeoff than the compared compression methods. These results, together with the layout-controlled examples in Figures 4–6, are the main reason I lean toward acceptance.

Critical weaknesses
  1. The main quantitative results and the demonstrations of flexible layout control use models trained under different settings. The experimental setup in Section 5 describes mixed-source training for layout-controlled generation, whereas Section 5.2 uses models trained exclusively on texture-variance layouts for Tables 1–2. The strongest quantitative evidence therefore comes from a specialized setting. The examples with other layout sources show that the method can handle them, but leave open whether the mixed-source model retains the reported quality–speed advantages. This limits the evidence for the breadth of the method.

  2. The ablation does not fully explain the benefit of asymmetric flow prediction in this setting. The Direct HR prediction variant in Table 3 performs substantially worse, but its target includes high-dimensional noise that may be difficult to predict from compressed inputs. A comparison with direct clean-latent prediction would help distinguish the benefit of the asymmetric parameterization from the more general benefit of avoiding full-dimensional noise prediction. This matters for understanding what the flow formulation contributes in the proposed multi-resolution setting.

Questions for authors
  1. In Appendix A.9, the layout-translation experiment shows that subject placement can follow the fine-token region. Beyond placement and coverage, how are subject detail and prompt adherence affected? In particular, are the objects and attributes required by the prompt preserved when the layout changes? Relatedly, for the FID improvements over the full-resolution model in Table 1, is there a control that receives the same layout information, or other evidence that separates the effects of layout conditioning from those of token allocation?

  2. For the direct full-resolution velocity variant in Table 3, are data scaling and effective loss weighting matched to the full model, and how is the output head initialized? More importantly, how would direct clean-latent prediction perform with the same compressed input and full-resolution output architecture? If this comparison is unavailable, can the existing results or analysis explain what the asymmetric parameterization gains over that alternative?

  3. The experimental setup in Section 5 describes mixed-source training for layout-controlled generation, while Section 5.2 and Tables 1–2 use models trained exclusively on texture-variance layouts. How does the mixed-source model perform quantitatively across the supported layout sources? Are the reported quality–speed advantages retained by the model used for flexible layout control?

  4. Eq. (7) describes a fixed projection AeTA_e^T to DD dimensions, whereas Eq. (12) uses an input head Win,eW_{in,e} that is fully fine-tuned. Is the fixed projection preserved throughout training, or is it only used to initialize an otherwise unconstrained input head? This affects how I understand the information retained by each token.

  5. The Foveated Diffusion baseline has already been extended to larger patch sizes. What additional technical difficulties arise when applying AsymFlow to such a model, and which LoT-specific changes address them? I would appreciate an explanation tied to the existing results, rather than necessarily another complete baseline implementation.

Reviewer_4C8pOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Level-of-Token (LoT) Diffusion, which allocates computation spatially through variable-sized tokens for efficient image and video generation. Fine tokens represent regions requiring more detail, while coarse tokens reduce computation elsewhere. The method adapts pretrained diffusion transformers using patch-wise asymmetric flow, extent dependent input/output heads, and patch size conditioning, while maintaining full-resolution latent predictions. Experiments on FLUX.2 and Wan2.1 demonstrate favorable quality efficiency trade-offs across multiple token budgets and layout sources. The reported speedups measure generation with a supplied layout, excluding layout construction.

Critical strengths
  1. The idea is intuitive and well motivated: different spatial regions require different levels of detail, making spatially adaptive computation a reasonable approach to accelerating generation.

  2. The reported experiments are reasonably solid within the evaluated setting. Comparisons at matched token budgets demonstrate favorable quality–efficiency trade-offs against several token-reduction baselines for both images and videos.

  3. The method integrates with pretrained models and supports multiple layout sources, suggesting practical flexibility.

Critical weaknesses
  1. Robustness to mismatches between the prescribed detail allocation and generated content is insufficiently evaluated, particularly when moving subjects leave fine-token regions. The consequences for visual quality and temporal consistency remain unclear.

  2. Although the direct-HR ablation supports AsymFlow empirically, the motivation for preferring AsymFlow could be explained more clearly.

  3. Table 2 reports six VBench custom-video metrics but lacks semantic and video–text alignment evaluation. Consequently, the results do not sufficiently establish whether compression preserves object counts, actions, spatial relationships, and prompt adherence.

  4. The training implementation is underspecified for variable-length token sequences. The paper does not explain how videos with different token counts are batched or how this affects computational efficiency and memory usage.

Questions for authors
  1. What happens when generated subjects fall outside the prescribed fine-token regions, especially during video generation? Could the authors provide failure cases or robustness evaluations?

  2. Beyond the direct-HR ablation, could the authors clarify the theoretical motivation for AsymFlow and its advantages over alternatives such as direct clean-data prediction?

  3. Could the authors supplement Table 2 with semantic or video–text alignment evaluations across compression budgets, including comparisons against the full-resolution baseline? The video (prompt is "The video opens with a young girl standing in a dimly lit underwater scene, surrounded by tall, slender purple plants. She is wearing a sleeveless top with horizontal stripes in shades of yellow and white. As the video progresses, a large, blue shark with white teeth and gills swims into the frame. The shark approaches the girl, who appears to be unaware of its presence. The shark's mouth opens wide, revealing its sharp teeth, and it moves closer to the girl. The girl's facial expressions change from surprise to fear as she realizes the shark's intentions. The shark continues to circle around her, and the girl's fear escalates. The video ends with the shark swimming away, leaving the girl alone in the underwater environment.") is not right in semantics.

  4. How are variable-length video token sequences handled within training batches? Are padding, sequence packing, or length-based bucketing used, and what are the resulting training throughput and memory costs?

Reviewer_eJ4pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper accelerates image and video generation by adaptively merging spatial tokens based on local detail requirements and using asymmetric flow to enable full-resolution denoising with reduced Transformer computation.

Critical strengths

The authors provide a detailed design for spatial token compression, including extent-dependent projections and data scaling to accommodate patches of different sizes, patch-wise asymmetric flow and extent-specific output heads to enable full-resolution predictions from compressed inputs, and patch-shape embeddings and RoPE aligned to the finest token grid to encode token geometry and preserve spatial alignment. It is interesting.

Critical weaknesses

I will focus on what I consider to be the major issues with this paper and leave aside the minor concerns.

1.The reported runtime does not include the computational cost of the algorithm used to partition the image into regions.

2.The method introduces local discontinuities. During region partitioning, different areas of the same object may be assigned to separate blocks, leading to inconsistent textures across those areas. When viewed in the context of the entire image or video, these inconsistencies manifest as noticeable spatial distortions and temporal flickering, which are completely unacceptable in practical applications. In practice, preserving visual quality must come first; acceleration should only be considered after that requirement is met. The videos in the authors’ supplementary material provide evidence of this problem.

3.What happens when the region-partitioning model is inaccurate? The proposed method depends on the accuracy of this model, so how does it handle errors in the partitioning?

4.Existing acceleration techniques, such as distillation, quantization, and caching, may also degrade overall quality, but they do not introduce this particular type of pronounced spatial distortion and inconsistency caused by block partitioning. Although the reported metrics may suggest only a modest degradation, I find the deterioration in visual quality to be severe.

Questions for authors

see weakness

3 comments

Dear Reviewer eJ4p, as the discussion period is ending soon, we would like to kindly follow up on our responses above. To address your concerns, we included the following set of experiments:

  1. We reported the layout construction runtime, which adds only 1.5% (image) and 0.19% (video) overhead.
  2. We added quantitative evaluations of subject coverage and temporal consistency at token-size transitions, which reviewer 4C8p has found to provide convincing evidence for LoT mitigating most of the temporal consistency compared to previous baselines.
  3. We clarified that the layout comes from simple heuristics rather than a trained model, and that our models are robust to arbitrary layouts because it is trained on a diverse set of token layout derived from various sources.
  4. We showed that LoT is complementary to distillation and TeaCache and distillation, with compounded speedups without significant quality degradation.

We would appreciate hearing whether these responses resolve your concerns, and we are happy to clarify further in the time remaining. If they do, we would be grateful if you would consider updating your score. Thank you again for your time and feedback.

Dear Reviewer eJ4p,

With one day left in the discussion period, we would greatly appreciate it if you could take a look at our responses and the additional results we provided to address your concerns. Please let us know whether they resolve them, or if anything remains that we can clarify in the time left. If so, we would be grateful if you would consider raising your score.

Thank you again for your time.

Dear Reviewer eJ4p,

As the discussion period closes today, this is a final gentle reminder regarding our rebuttal. We would be very grateful if you could look over our responses and the additional results we provided to address your concerns. Let us know whether any issues remain. If your concerns have been resolved, we would kindly ask you to consider raising your score. Thank you very much for your time and effort.

Comments on the paper

1 comment

We thank the reviewers again for their thoughtful comments and feedback. We appreciate that the reviewers found our work to provide a favorable efficiency-quality trade-off on open-source image and video generative models.

We would like to reiterate that the premise of our paper is that not all pixels need to be generated at full resolution. By finetuning image and video models using various LoT token layouts derived from prior knowledge such as semantic masks, bounding boxes, depth maps, or texture variance, they learn to adapt the generated content to a given LoT token layout, reducing the computational cost while preserving generation quality.

We responded to each reviewer’s questions individually, including additional experiments and clarifications. We are happy to provide further clarification or results in the time remaining, and we would kindly ask the reviewers to consider updating their assessments if our responses have resolved their concerns. Thank you again for your time and effort in reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.