acceptodds
Withdrawn from ICLR 2027

Spend-Where-It-Matters (SWIM): Content-Aware Efficient Diffusion Models for Image and Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models achieve state-of-the-art performance in image synthesis and editing, but remain costly due to long denoising trajectories and large backbone models. Existing accelerations reduce the number of sampling steps or reuse cached features on a static schedule, so every input receives the same computation, even though which parts of the backbone matter varies across inputs and timesteps. We introduce SWIM, a model-agnostic, content-aware diffusion framework that decides, at each denoising step, which backbone blocks to execute under a user-set compute budget, while leaving the backbone weights frozen. A lightweight recurrent gating module reads the current sample's intermediate features, the change in the predicted score across diffusion steps, and bypasses the blocks that contribute least to the current update, so compute is spent where the sample needs it rather than on a fixed schedule. Because per-step compute is no longer uniform, we report cumulative multiply–accumulate operations (CMACs) as the measure of compute cost, over the full trajectory. For SD3.5 T2I generation, SWIM uses under 50% of DeepCache's CMACs yet outperforms it across all T2I-CompBench++ benchmarks with a 38% relative mean score improvement. On FLUX 2, SWIM cuts CMACs by 23% while staying within 0.01 of the T2I-CompBench++ mean. On ImageNet-256, SWIM + PixelDiT uses 18% fewer CMACs and 19% fewer parameters than PixelDiT at competitive FID, and improves FID over DiT-XL with 79% fewer CMACs. SWIM also extends to video generation, reducing CogVideoX-5B inference time by 13%.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Diffusion models, Dynamic gating, Block Pruning, Content-Aware

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T6ZeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SWIM freezes a diffusion backbone and trains a lightweight recurrent controller to select which transformer blocks to execute at each denoising step. The controller conditions on the current latent, prompt embedding, timestep, and a trajectory-curvature signal. A hard top-k rule keeps the per-step block budget fixed while allowing the identities of executed blocks to vary by sample and timestep. The method is evaluated on SD3.5, FLUX 2, PixelDiT, and CogVideoX.

Critical strengths
  1. The same-backbone FLUX 2 result is promising: SWIM reduces reported CMACs from 23.8T to 18.32T while changing the T2I-CompBench++ mean from 0.74 to 0.73. If all costs are measured consistently, this is a meaningful efficiency-quality trade-off.
  2. The PixelDiT controls support the value of learned dynamic routing. At essentially identical compute, SWIM obtains FID 3.69, compared with 4.52 for static selection and 55.25 for random anchored selection. This shows that block identity and temporal adaptation matter, rather than only the number of executed blocks.
Critical weaknesses
  1. The curvature signal appears circular or substantially more expensive than reported. The gate uses κ~t\tilde{\kappa}_t to decide which blocks to execute at step tt, but Appendix A.1 defines this quantity through a Hessian-vector product of the model output energy, requiring gradient computation and a double backward pass. It is unclear how this signal can be obtained before deciding which blocks to execute. If it is computed from the previous step, the indexing and first-step behavior are not specified. More importantly, the reported CMAC and latency accounting does not state whether these backward passes are included.
  2. The efficiency comparisons do not isolate the contribution of SWIM reliably. Several baselines use different backbones, schedules, or numbers of evaluations. The paper reports "active parameters," although every frozen block must remain stored because it may be selected later. The gate is separately trained for each backbone, so "model-agnostic" describes the framework rather than transferable gate weights. In addition, the proposed QNC metric adds a 20–25-scale CLIP score to a 0–1 benchmark mean, making the latter nearly irrelevant.
Questions for authors
  1. How is κ~t\tilde{\kappa}_t computed causally at inference time? Does its computation require executing the current backbone or performing backward/HVP operations, and are those costs included in every reported latency and CMAC value?
  2. How is the GRU trained over complete denoising trajectories? Please specify hidden-state initialization, rollout construction, gradient truncation, and how STE gradients are obtained for skipped blocks without evaluating them.
  3. Can the acceleration baselines be rerun on the same checkpoint, sampler, NFE, resolution, prompts, and seeds under matched CMAC or matched quality?
  4. Please distinguish stored model parameters from active parameters and report batch-size sensitivity, since block skipping only saves computation when all samples in a batch skip the same block.
Reviewer_BPdDOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an approach to efficient diffusion generation by selecting backbone blocks to skip during inference. Using a GRU with inputs of latent image representations, calculated diffusion trajectory curvatures, and the denoising timestep, such selection could be content-aware. The results are reported on various tasks with significant CMACs drops, demonstrating its effectiveness in saving computational cost.

Critical strengths
  • This method achieves promising efficiency-quality trade-off on some strong pretrained models. As in Table 2 FLUX results, performance on benchmarks and CLIP score do not drop much while reducing MACs by around 23%.
  • The proposed SWIM can be adapted to various diffusion backbones like SD3.5, FLUX2, PixelDiT, and CogVideoX across various tasks, proving its applicability.
  • Evaluation on various benchmarks and tasks is appreciated. Additional human preference study is also a plus.
Critical weaknesses
  • One weakness is a clear mismatch between motivation and proposed solution. The authors use Figure 2 to provide an illustration for their motivation. However, information like how the dropped blocks are selected? Are the same blocks used for all tasks? are neglected. Without such information, jumping to conclusion like "nominal prompt complexity alone is insufficient to predict computational slack" seems unreasonable. Maybe the dropped blocks are just not suitable for task 2. The overall summary of Section 3.1 is also unsmooth. Figure 2 can only prove that (if all tasks drop the same static blocks), different prompts could be affected differently by block selections. But it does not necessarily prove that such optimal block selection changes with denoising state, and it cannot be directly predicted by a simpler combination of timestep+prompt (without latent history). To address this concern, I believe ablation study using a simple MLP without latent history to predict block drop is a must to validate the necessity of the GRU.
  • L218-221: This is still a concern regarding motivation. The authors explicitly mention that "prompts of similar complexity tolerate very different numbers of dropped blocks". To solve that, a natural next step would be to adaptively choose different numbers of dropped blocks for different prompts. But instead, the paper proposes to still drop the same amount of blocks (by a user-selected fraction), but selectively. I believe the motivation part of this paper needs to be reframed entirely.
  • The calculation of local curvature of the denoising trajectory in A.1 seems uncheap. I wonder whether such calculation is needed for each denoising step during inference, and whether it is counted within the reported computational cost.
  • Comparisons and discussion with more relevant baselines on dynamic computation/adaptive execution path [1][2][3] are missing.


[1] AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation. ECCV'24.
[2] Dynamic Diffusion Transformer. ICLR'25.
[3] Timestep-Aware Block Masking for Efficient Diffusion Model Inference. 2026.

Questions for authors
  • In Table 3, SWIM (only image) and SWIM perform similarly. With the cost of score calculation, I wonder whether such performance gain is worthy. Besides, the gain may not purely come from score input since timestep is also a source of variance. Additional ablation study on SWIM (image+timestep) would clearly separate the contribution of each part.
  • In Table 4, SWIM + CogVideoX's CLIP score dropped quite a lot (-2.72) compared to CogVideoX, with little gain in time saving (27s). More evaluation metrics on video quality would be appreciated to see whether such a gain does not hurt performance too much.
  • Line 269, SD3.5 mean is reported as 0.66. But in Table 2, the value is 0.7. Not sure what happened here. Is this a typo, or are there different experimental settings?
  • Please improve the layout of the paper around tables and figures. Some captions are too small to read and are too close to the text.
Reviewer_VToHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SWIM is a new diffusion framework that dynamically allocates computation at every time step by utilizing information gathered from earlier steps. It employs a gating recurrent network to determine the optimal compute allocation, deciding which backbone blocks to activate given a set compute budget. Although following the principles of ADSF, SWIM enhances efficiency by estimating the score field curvature, which helps detect stable update directions and identifies redundant computation. A key differentiator from ADSF is that SWIM provides a richer input to the scheduler, incorporating CNN features of the noisy latents, the text prompt embedding, and the timestep embedding, and substitutes the MLP with a recurrent gating network.

Critical strengths

SWIM targets one of the biggest limitations of diffusion-based image generators: the high computational cost and latency of the inference pass.

The approach presented reduces the computational cost using a lightweight model-agnostic module, tested on multiple backbones and benchmarks.

With this lightweight addition, the method obtains a substantial 23% MACs reduction on FLUX2 with comparable image quality scores.

Critical weaknesses

The main weakness of the paper is in the evaluation and comparison to prior work.

W1 Table 2 does not clarify under what setup each method is evaluated. Comparing values for other methods that have different backbones or parameters adds limited information. The only direct baseline used in Table 2 is TeaCache, a method introduced primarily for video model acceleration. It would be interesting to evaluate the method against text-to-image caching methods like TaylorSeer [1], or ToCa [2] [1] From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers, Liu et al., ICCV 2025 [2] Accelerating Diffusion Transformers with Token-wise Feature Caching, Zou et al., ICLR 2025

W2 Figure 6 presents a comparison to Deepcache, TGATE and ToMe, again without specifying what backbone each uses. Further, Deepcache is presented as a direct comparison in Figure 6, but as part of the "other competitive methods" category in Table 2. The "other competitive methods" section in Table 2 also includes baselines like Stable XL, Stable v2, raising the concerns that DeepCache is being used as a comparison with a different backbone, making it impossible to directly compare the acceleration methods.

W3 TGATE and ToMe are presented as main comparison in Figure 6 and the human study, but not in Table 2

Questions for authors

Q1 Can the authors discuss why in Table 4 TeaCache is reported as slower than the model it is meant to accelerate?

Q2 Can the authors expand on the observation in the ablations section that the majority of the gains come from the image features? Additionally, the static block selection appears to perform better than the score-based selection, questioning whether the score selection mechanism is adding useful information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.