acceptodds
Under review as a conference paper at ICLR 2027

Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions

Abstract

Diffusion transformers require repeated denoiser evaluations, making image and video generation computationally expensive. We propose a training-free framework that uses a few calibration conditions to construct a fixed, fine-grained module-reuse schedule without schedule search. The design is motivated by an empirical effect along deterministic sampling trajectories: with initial noise fixed, condition-dependent deviations in several module outputs evolve similarly across adjacent steps, so temporal differencing attenuates much of their variation. We term this effect Conditional Common-Mode Rejection (CCMR). Motivated by this observation, we rank timestep–layer–module locations using normalized ratios of consecutive feature displacements and construct a Cache Book through two-stage calibration. The second stage updates a scorer-only reference with each provisional bit after recording its indexed score, while retaining full-compute trajectories. At inference, the fixed schedule requires no per-input policy estimation and can be nested within fixed step-wise cache policies. Module caching alone achieves – speedup on the evaluated image models and – on the video models. Combining it with MagCache or SeaCache provides further acceleration at a model-dependent cost in paired fidelity.

Primary area:
generative models
Keywords:
Diffusion Transformers, Inference acceleration, Feature caching for diffusion transformers, Training free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xoc5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for accelerating image and video diffusion transformers through fixed module-level feature reuse. The method is motivated by Conditional Common-Mode Rejection (CCMR), an empirical observation that adjacent-step differencing reduces condition-dependent variation in module outputs when the initial noise is fixed. Stage I ranks timestep–layer–module locations using ratios of consecutive L1 feature displacements and constructs a provisional Cache Book through family-wise quantile selection. Stage II recomputes these priorities using a reference history influenced by the provisional book, while retaining full-compute trajectories.

Critical strengths

Given the reuse hyperparameters, the method constructs a fine-grained schedule from one or two calibration conditions without gradient-based optimization, combinatorial schedule search, or per-generation profiling. This simplicity could be useful when adapting caching to different model configurations.

The method produces measured latency reductions on both image and video models. The DiT operating point compares favorably with the tested ToCa settings on speed, FID, and paired fidelity, although not uniformly across all metrics.

Critical weaknesses

A small ratio can result from a large denominator and does not imply that reusing an earlier output at the indexed step is safe. The manuscript acknowledges this limitation, but does not directly evaluate whether the score predicts the consequences of individual reuse decisions. Table 3 provides encouraging comparisons with adjacent MSE and cosine distance, but the realized reuse fractions and FLOPs differ.

The main configurations use model- and module-specific quantiles, with separate settings for hybrid caching. Constructing a book after fixing these values is search-free, but choosing them may still require quality evaluation and experimentation. Appendix A reports the number of full-compute trajectories rather than measured calibration costs for the main configurations. Although a calibration time is provided for the eight-step distilled experiment, broader accounting of selection cost, calibration latency, and memory is needed to substantiate the practical advantage.

Questions for authors

How sensitive are the final quality metrics to independently sampled calibration conditions and random seeds? A repeated calibration experiment on one representative image model and one video model, together with broader motion-rich video evaluation, would help assess the reliability of the few-condition claim.

Can you provide quality–latency curves against independently tuned MagCache/SeaCache, together with a controlled comparison against ECAD or SODA on a shared model? In particular, does adding the module Cache Book improve quality at matched latency compared with increasing the baseline’s reuse aggressiveness?

Can you evaluate Stage I versus Stage II on an independent paired set with comparable executed compute? Please also explain the motivation for assigning the look-ahead ratio to step t and for updating the scorer reference after observing step t+1, rather than directly following the inference cache history.

Reviewer_jVS4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a probing method, CCMR, which motivates a diffusion acceleration method that requires a pre-caliberation. CCMR gain is large when the variation of δ(next activation,current activation)\delta(\text{next activation}, \text{current activation}) across prompts is small relative to that of the original activations. Stage I prioritizes cache reuse when the magnitude of δ(next activation,current activation)\delta(\text{next activation}, \text{current activation}) is small relative to that of δ(current activation,previous activation)\delta(\text{current activation}, \text{previous activation}). Stage II accounts for activations becoming stale during consecutive reuse. Experiments shows practical speedup across image generation and video generation models. The proposed method can be also integrated with existing step-wise acceleration methods.

Critical strengths

S1: The proposed method demonstrates practical inference acceleration across a broad range of models. The policy, once caliberated, is computationally lightweight during inference. The caliberation also follows a unified condition (two stages), which makes it easier and practical to implement and deploy.

Critical weaknesses

W1 [THEORETICAL]: The proposed method is not justified for reducing cache-reuse error. CCMR gain is large when the variation of δ(next activation,current activation)\delta(\text{next activation}, \text{current activation}) across prompts is small relative to that of the original activations. Stage I prioritizes cache reuse when the magnitude of δ(next activation,current activation)\delta(\text{next activation}, \text{current activation}) is small relative to that of δ(current activation,previous activation)\delta(\text{current activation}, \text{previous activation}). Stage II accounts for activations becoming stale during consecutive reuse. However, small variation across prompts and a small ratio across steps do not mean that the activation difference itself is small.

For example, scalar module activations A=(0,1000,1100)A=(0,1000,1100) and B=(10,1010,1110)B=(10,1010,1110) for two prompts at steps t−1,t,t+1t-1,t,t+1 yield strong CCMR and ρt≈0.1\rho_t\approx0.1, yet reusing the previous activation at step tt incurs an absolute error of 10001000 for both prompts.

W2 [EMPIRICAL]: The approximation design is insufficiently justified, and the reported speed–fidelity trade-off is not convincing. The method relies on direct reuse of intermediate module activations, without establishing why this design is preferable to higher-order numerical approximation or output-level approximation. Direct reuse may limit approximation accuracy, while module-level caching may require architecture-specific integration.

In fact, the reported paired fidelity reinforces this concern. The reported LPIPS values ranging from 0.3230.323 to 0.5710.571 across the four evaluated models. Furthermore, stronger baselines such as SADA, SparseVideoGen 1/2, and ZEUS are missing from the evaluation, as these methods report LPIPS usually substantially better (below 0.10.1) in their respective evaluation settings, motivating direct comparison.

Similarity metrics (LPIPS/SSIM/PSNR) are important for evaluating acceleration methods because they measure whether accelerated outputs remain similar to the unmodified baseline. Meanwhile, distributional metrics such as FID are NOT reliable metric for T2I acceleration since the training and evaluation distributions differ substantially, so even the FID of 2222 moves to 19 alone does not establish that the accelerated outputs remain faithful to the unmodified model.

Questions for authors

Q1 Following W2, could the author try higher order approximation (such as, xt-1 = xt + (xt - xt+1), a second order predictor) instead of direct reuse?

Q2 Following W2, could the author compare one of SADA, SVG1, SVG2, or ZEUS in the same setting? These methods are direct baselines for training free acceleration and are fully open-sourced.

Reviewer_qb1yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free feature-caching method for diffusion transformers that builds a fixed timestep–layer–module compute-or-reuse schedule, called a Cache Book, from one or two calibration conditions. The design rests on two measurements: "Conditional Common-Mode Rejection" (CCMR), i.e., with fixed initial noise, adjacent-step differencing suppresses condition-induced variance in module outputs; and the agreement of score rankings from few conditions with full-set rankings. Positions are ranked by the ratio of consecutive ℓ1\ell_1 displacements, selected by family-wise quantiles, and re-scored in a second full-compute pass against a stale, policy-dependent reference. On DiT-XL/2, FLUX.1-dev, Wan2.1, and HunyuanVideo-1.5, module caching alone yields 1.54–1.72× speedup and adds acceleration when nested in MagCache or SeaCache.

Critical strengths
  1. Search-free, low-cost schedule construction that works across architectures. For given family quantiles, a Cache Book needs only 2K2K full-compute trajectories with K≤2K \le 2, and no training, quality-feedback search, or per-input profiling, versus the 44–700 GPU-hours cited for ECAD. It runs on four DiTs, up to 720p, 121-frame HunyuanVideo-1.5, and stacks on step-wise caches, e.g., raising SeaCache from 2.08× to 2.36× on HunyuanVideo with similar VBench scores, which makes fine-grained caching practical to deploy.

  2. Measured support for few-condition calibration. Fig. 1(d) shows that rankings from a single condition already agree closely with full-set rankings (Spearman 0.87 on DiT and 0.90 on FLUX), with seed-level bootstrap intervals. This directly supports the title claim and gives the caching community an empirical basis for single-prompt calibration.

Critical weaknesses
  1. The experiments do not show a favorable speed–quality trade-off. Module caching alone is slower and worse than MagCache, e.g., 1.66× vs. 2.04× on FLUX with PSNR 16.29 vs. 24.89, and HunyuanVideo shows a similar pattern. The nesting gains are never compared with the step-wise policy alone at a stronger threshold and equal speedup, although adding the book costs 5–10 dB PSNR. On Wan2.1, adding the book removes about 8% of FLOPs yet 30% of latency, which points to implementation differences. Speed–quality Pareto curves for all methods in one code path, including TeaCache and a fine-grained schedule baseline such as ECAD or SODA, are needed.

  2. The motivation and components that distinguish the method from profiling-based static schedules, such as Cache Me if You Can and the uncited DiTFastAttn (Yuan et al., 2024), are not validated. The shuffled control gives −3.28 dB on DiT, essentially the ≈−3 dB obtained by differencing unrelated deviations, so it cannot separate CCMR from ordinary per-trajectory temporal smoothness. The score ρt=δt+1/δt\rho_t=\delta_{t+1}/\delta_t decides reuse at step tt, yet its denominator is the displacement δt\delta_t that this reuse incurs, and this indexing is not justified. Stage II shows no measurable gain: all three 95% intervals in Table S12 include zero. Matched-reuse ablations against random selection, relative-ℓ1\ell_1 displacement, δt/δt−1\delta_t/\delta_{t-1}, and a single global quantile are needed.

Questions for authors

Clarification:

  1. Were all methods timed in one code path with identical attention kernels, compilation, and offloading settings, and why does the same FLUX Full configuration take 8.084 s in Table 1 but 7.018 s in Table 3, which would put module-only speedup at about 1.43×?
  2. How were the family quantiles q\mathbf{q} (e.g., [.70,.70,.30,.23,.50,.02][.70,.70,.30,.23,.50,.02] on FLUX) and the prefix PP chosen for each model and hybrid: on which prompts, with or without inspecting evaluation outputs, and at what total compute cost?
  3. How much do FID, LPIPS, and VBench vary when the single calibration condition is swapped (e.g., across five classes or prompts) and when KK = 1, 2, 4, and 8?
  4. Which guidance scale does the main DiT table use, and given that both cached methods obtain lower FID-5K than Full (14.16 vs. 18.70) with lower IS, does caching weaken effective guidance, and what does FID-50K show at the standard scale of 1.5?
  5. In hybrid runs, a module reused right after a whole-step skip returns an output at least two steps old; does hybrid calibration account for the step-wise skip pattern, and how often does this case occur?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.