acceptodds
Withdrawn from ICLR 2027

Spatially Adaptive Multi-Resolution Token Merging for Efficient Diffusion Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion transformers repeatedly process dense spatial token grids, making high-resolution generation computationally expensive. Existing training-free token-merging methods primarily use global token matching or ranking, overlooking that redundancy in generated images is spatially structured. We formulate diffusion token reduction as budgeted hierarchical spatial-resolution allocation and introduce **QuadMerge**, which converts classifier-free-guidance residuals into a multi-resolution partition of , , and regions. QuadMerge conservatively coarsens only uniformly low-importance regions, aggregates them with importance-weighted pooling, couples the plan with timestep-adaptive compression, and caches allocation plans across compatible transformer blocks. At a matched token budget, QuadMerge improves both generation quality and measured latency over global token-merging baselines, and the margin grows under aggressive compression. On PixArt-α, a 30% token reduction costs 0.09 FID (36.60 → 36.69) versus 1.05 for importance matching and 12.48 for ToMe; unlike those matchers, it stays below uncompressed wall-clock cost. At a 70% reduction the FID gap over importance matching grows to 7.92, and the same allocation transfers to Stable Diffusion 2.1. The method requires no retraining, no additional learned modules, and no modification to the diffusion sampler.

Primary area:
generative models
Keywords:
Diffusion Models, Token Merging, Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5E1iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes QuadMerge, a training-free token-reduction method for diffusion inference. It replaces global token matching with a budgeted multi-resolution partition: a CFG-residual importance map assigns 1x1, 2x2, or 4x4 groups, with a max-risk rule, importance-weighted pooling, timestep-adaptive compression, and plan caching. On PixArt-alpha, 30% token reduction costs 0.09 FID versus 1.05 for importance matching and 12.48 for ToMe; the gap grows at 70%.

Critical strengths

The formulation is clean. Image redundancy is spatially structured, and a budgeted multi-resolution partition is a sensible alternative to global matching. The max-risk rule keeps boundaries out of coarse groups, and the merge-attend-broadcast design leaves the dense interface untouched. The ablations over beta4, TAC, random maps, and scoring variants are thorough.

Critical weaknesses
  1. The central efficiency claim fails at the operating point the paper recommends. At r=.30, end-to-end latency is 2.500 s against 2.550 s for the uncompressed baseline, a 2% difference measured on five images with one warm-up and no variance. The 30% figure is a token ratio: full-transformer FLOPs drop by 12.8% because only blocks 7-20 are patched and every MLP stays dense. At r=.50 the numbers are 19.7% FLOPs and 8.2% latency; at r=.70 about 25% and 13.1%. Table 2 is worse for the claim: the 14-step baseline reaches 37.25 FID at 1.714 s and 90.9 TFLOPs, while QuadMerge at r=.30 reaches 36.93 at 2.322 s and 91.4 TFLOPs. The FID difference is 0.32, within noise, and the baseline is 35% faster. Composing 14 steps with r=.30 gives 37.20 at 1.694 s, a 1.2% latency gain. The method wins only in the r=.70 regime that the authors call robustness rather than deployment. The SD2.1 2.2x speedup is the exception, but it is a 2022 UNet.

  2. The planner and schedule are not isolated, and the evidence base is thin. At r=.70, QuadMerge without TAC is 57.29, close to importance matching at 56.91, and importance matching with TAC is 56.30; the large gain (48.99) appears only when both are combined. The two TAC instantiations use different caps (.75 versus .95), so this is not schedule-matched. There is no planner-only result at r=.30 or .50, the recommended settings. The only backbones are PixArt-alpha (2023, 0.6B) and SD2.1 (2022, UNet) with 20-step DDIM, and the structure claim rests on FID/CLIP plus qualitative panels, not a localized structural metric.

Questions for authors
  1. Why not use newer models?
  2. Can you report latency as mean and standard deviation over more than five images, with a breakdown into attention, MLP, text encoder, VAE, and planner overhead? At r=.30, K and TFLOPs are almost identical across QuadMerge, ToMe, and importance matching (2868.7 versus 2868.0, 113.16 versus 113.10), yet latency spans 2.500 to 2.690 s. The 2% gap against the baseline and the 2.2x SD2.1 speedup both need variance and a breakdown. On SD2.1 latency is also flat from r=.30 to .70 (0.727 versus 0.738 s), which does not fit a simple attention-scaling story.
Reviewer_j4JbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QuadMerge is a training-free token-merging method for diffusion transformers that reframes token reduction as spatial-resolution allocation. Using the CFG residual as an importance map, it partitions the latent grid into non-overlapping 1×1, 2×2, and 4×4 regions under a fixed removal budget, coarsening only regions whose tokens are all low-importance. Merged tokens pass through attention and are broadcast back to their original positions. A timestep-adaptive schedule compresses more in early steps, and the merge plan is reused across blocks within each step. On PixArt-α and SD 2.1, the method reports better FID/CLIP and lower latency than ToMe and importance-based merging. QuadMerge is a training-free token-merging method for diffusion transformers that reframes token reduction as spatial-resolution allocation. Using the CFG residual as an importance map, it partitions the latent grid into non-overlapping 1×1, 2×2, and 4×4 regions under a fixed removal budget, coarsening only regions whose tokens are all low-importance. Merged tokens pass through attention and are broadcast back to their original positions. A timestep-adaptive schedule compresses more in early steps, and the merge plan is reused across blocks within each step. On PixArt-α and SD 2.1, the method reports better FID/CLIP and lower latency than ToMe and importance-based merging.

Critical strengths

The idea of merging tokens at different resolutions according to their importance is simple and clear.

Critical weaknesses

[W1] Limited benefit

On PixArt-α, the speedup is at most 1.15×. This is a small benefit that could be achieved even with very simple caching or spatial acceleration. Moreover, despite a speedup of only 1.15×, FID increases by 34%, and the qualitative results show that image quality and level of detail clearly fall short of the baseline. While the results are better than ToMe, there is no comparison with more recent merging methods, which raises further doubts about the 1.15× figure.

[W2] Unclear meaning of importance

This paper claims that regions with a large difference between the conditional and unconditional predictions are the regions the model focuses on to reflect the text prompt, and that these regions should not be merged in order to maintain image quality. However, it is unclear why the regions the model focuses on to reflect the text prompt are the same as the regions that are relatively important and should not be merged.

[W3] Novelty Claim

The paper’s claim that formulating token reduction as spatial-resolution allocation is a contribution should be tempered. Region-adaptive training-free acceleration methods (RAS (CVPR 2026), RALU (CVPR 2026)) already allocate computation/resolution spatially. The novelty lies in the specific operator rather than the formulation.

Questions for authors

[Q1] Why are the regions the model focuses on to reflect the text prompt the regions that should not be merged? A related analysis is needed.

[Q2] Is a higher speedup possible?

[Q3] Is a comparison with more recent merging methods possible? I recommend ToMA (ICML 2025) and SDTM (CVPR 2025).

[Q4] Under similar speedup settings, can the authors compare performance with some caching methods (e.g., TaylorSeer (ICCV 2025)) and spatial acceleration methods (e.g., RALU (CVPR 2026))? Alternatively, can they show that the method can be applied complementarily with these methods to achieve additional speedup?

Reviewer_YSD9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces QUADMERGE, a training-free token-merging method that formulates diffusion inference acceleration as spatial-resolution allocation. A classifier-free-guidance residual guides a partition into fine and coarse regions, which are merged before attention and broadcast back afterward. The method combines this planner with timestep-adaptive compression and within-step plan reuse. Experiments on PixArt-α and Stable Diffusion 2.1 report favorable quality–efficiency trade-offs against global token-merging baselines, supported by realized-compute measurements and component ablations.

Critical strengths
  1. The central idea is intuitive and well motivated: spatially contiguous, low-importance regions can tolerate coarser representations, while prompt-sensitive content benefits from finer tokens. The method requires neither retraining nor additional learned modules.
  2. The PixArt comparisons provide useful evidence at closely matched realized compute. At removal ratios of 0.50 and 0.70, QUADMERGE achieves FIDs of 38.93 and 48.99, compared with 42.49 and 56.91 for Importance. Reporting actual sequence lengths and full-transformer FLOPs makes these comparisons more informative than matching configured ratios alone.
Critical weaknesses
  1. The largest reported speedup requires clarification. Section 3.5 and Appendix B state that Stable Diffusion merging starts at step 10 of 20 and affects only selected self-attention blocks, yet Table 4 reports latency decreasing from 1.582 to 0.727 seconds, approximately 2.18×. If the first ten steps remain unchanged and dense steps have comparable costs under the same execution setup, they should already account for roughly half the baseline latency, before the remaining steps are executed. The current description does not reconcile this result with the activation schedule.
  2. The practical gain on the primary DiT benchmark is modest, and comparison with stronger efficiency baselines is incomplete. At r = 0.30, PixArt latency decreases by only about 2%; at r = 0.50, the reduction is 8.2% with worse FID than the uncompressed model. ToMA, which is cited for locality and efficient merge–unmerge execution, is not evaluated. Table 2 includes a useful reduced-step baseline, but adding QUADMERGE to its 14-step trajectory provides only about 1.2% further latency reduction.
Questions for authors
  1. Could the authors provide the exact Stable Diffusion activation steps, per-step timings, and a no-merging control using the same attention backend, precision, CFG execution, and timing scope? How is the 0.727-second result reconciled with the uncompressed initial steps?
  2. Could the authors compare against ToMA under matched hardware, execution settings, and quality constraints, and report repeated timings with uncertainty? What incremental advantage remains over an appropriately shortened sampling trajectory?
  3. Could the spatial planner and global matcher be compared under the same feasible per-step token budgets? Please clarify whether max-risk is only a ranking heuristic or enforces a rejection rule, and whether larger-sample local-design ablations support the chosen scoring and pooling operations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.