acceptodds
Withdrawn from ICLR 2027

Trajectory-Consistent Calibration for Cache-Accelerated Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion Transformers require repeated denoiser evaluations during iterative sampling, making inference computationally expensive. Cache-based acceleration reduces this cost by reusing intermediate representations across denoising steps, but the resulting cache-side features can deviate from those produced by fresh computation and degrade generation quality. We identify two coupled calibration challenges. First, a reused or approximately computed feature can differ from its full-computation counterpart at the current calibration site. Second, applying an earlier correction changes the subsequent cache-side trajectory, so a later calibration operator may encounter a feature distribution different from the one used to fit it. We refer to these challenges as direct cache mismatch and trajectory-induced calibration mismatch, respectively. To address them, we propose Trajectory-Consistent Calibration (TCC), which fits closed-form, site-specific calibration operators from paired cache-side and full-computation features without modifying the denoiser parameters or the base cache strategy. Rather than fitting all operators on a single uncorrected cache trajectory, TCC processes selected denoising timesteps sequentially: it probes cache-side features, fits the corresponding operators, and replays and commits the calibrated step before estimating any later operators. Experiments on PixArt-α and DiT-XL/2 report lower FID values across the evaluated cache-based acceleration settings. In a representative PixArt-α setting based on FORA, TCC reduces FID from 29.83 to 27.35, slightly surpassing the corresponding full-computation baseline in terms of FID.

Primary area:
generative models
Keywords:
Diffusion Transformers, Efficient Inference, Cache-Based Acceleration, Representation Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bPK4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Trajectory-Consistent Calibration (TCC) to correct feature errors in cache-accelerated diffusion models. TCC fits closed-form transformations from paired cached and full-computation features, estimating them sequentially so that later corrections account for trajectory changes caused by earlier ones. Experiments on PixArt-α and DiT-XL/2 show FID improvements over the evaluated cache baselines, with additional computation and latency overhead.

Critical strengths

The design reasonably accounts for the fact that earlier corrections alter the subsequent generation trajectory during calibration.

Critical weaknesses

[W1] Limited insight

The paper offers a limited take-home message.

a) Presentation is one contributing factor. Figure 2, which should explain the method, is difficult to follow. It primarily consists of conceptual blocks, whereas the main text relies heavily on equations, making the connection between the two unclear. The text in the figure is also too small. Analyses motivating the proposed method are either insufficient or not presented where they would best support the motivation, as in the case of Figure 3.

b) The insight gained over existing calibration methods is limited. My understanding of the central message is that earlier corrections alter the generation trajectory, so subsequent corrections should be estimated accordingly. However, the paper does not analyze when or to what extent this mismatch occurs, or how much it has limited the performance of existing methods.

[W2] Limited experimental evaluation

The evaluated models, caching baselines, and offline plug-in baselines are all limited, and the results raise questions about the practical utility of this work.

a) PixArt-α is outdated as a text-to-image model, making it difficult to assess the method’s effectiveness on current models. In the qualitative comparisons in Figures 1 and 6, even the full model’s outputs appear substantially worse than those of state-of-the-art models.

b) FORA and ToCa are not state-of-the-art caching baselines, and many subsequent methods have been published. In particular, Table 1 shows that ToCa’s latency reduction is much smaller than its FLOPs reduction, making it a weak baseline for demonstrating practical acceleration.

c) Comparisons with offline plug-in methods, which are the most directly relevant baselines, are scarce or absent. Only EOC is evaluated on DiT-XL/2, and no such baseline is evaluated on PixArt-α.

d) The experimental results raise questions about the need for this work. TCC increases TFLOPs by over 10% in some settings, yet it is difficult to determine whether the reduction in FID is substantial because there are no comparisons at similar speedups. For example, in Table 1, a fair comparison would require decreasing FORA’s N to match the speedup of FORA (N=3) + TCC. Conversely, the decrease in CLIP score appears substantial and concerning. For example, HarmoniCa (ICML 2025) reports a CLIP score decrease of less than 0.1 on PixArt-α, whereas applying TCC results in decreases of 0.23 for FORA and 0.58 for ToCa, indicating a critical loss of text alignment.

Questions for authors

[Q1] How much do earlier corrections change subsequent features, and to what extent do these changes increase the mismatch between the fitted calibration operators and the features they encounter during inference?

[Q2] Recent caching methods can also accelerate video generation models while maintaining high quality. Would TCC work well on video generation models?

[Q3] Could you evaluate the method on at least one model capable of generating high-quality images at higher resolutions? I suggest FLUX.

[Q4] Could you evaluate TCC on top of at least two recent state-of-the-art caching baselines? I suggest TeaCache (CVPR 2025) and TaylorSeer (ICCV 2025).

[Q5] Please add comparisons with offline plug-in baselines. At a minimum, a comparison with ICC (CVPR 2025) is needed.

[Q6] Please also evaluate metrics other than FID. Generating 30,000 images for each of the requested settings may be burdensome within the rebuttal period. I suggest ImageReward.

[Q7] The FLOPs values in Tables 1 and 2 appear implausibly low. Are they reported in TFLOPs?

[Q8] In Figures 1 and 6, TCC appears to improve image quality, but it also changes the images considerably, suggesting low SSIM between the outputs before and after calibration. I am not arguing that the corrected images should remain similar to the uncorrected ones; I would simply like to understand why such substantial changes occur.

Reviewer_9d1hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trajectory-Consistent Calibration (TCC), a closed-form, plug-in method that corrects the feature errors introduced by existing cache-based acceleration methods for Diffusion Transformers (e.g., FORA, ToCa, L2C) without modifying the denoiser or the underlying cache policy. The authors define two kinds of discrepancy in image quality that arise between the full-computation diffusion trajectory and the cache-accelerated trajectory, and attempt to reduce them with TCC.

Critical strengths
  1. The problem formulation is well written and easy to understand. It is not hard to follow the authors’ motivation for why TCC is needed and what limitations existing methods have.
  2. The method is also clearly written and easy to follow.
  3. The evaluation is candid: the one-shot vs. sequential ablation directly tests the central claim, main results are repeated over three paired seeds, and unfavorable results (CLIP drop, ToCa latency) are reported rather than omitted.
Critical weaknesses

W1. (Baselines are not described in detail) The motivation for cache-based acceleration and the proposed method are easy to follow, but the baselines themselves (FORA, ToCa, L2C) are not described anywhere in the paper; Appendix A lists citations rather than explaining what each method reuses and how. Since the tables are indexed by baseline-specific hyperparameters (e.g., N and R), and these are defined only in the last sentence of Appendix B.2, I would suggest adding to Appendix A a sentence or two per baseline that explains its reuse mechanism and what its hyperparameters mean.

W2. (Additional experimental analysis) (1) The paper defines (i) direct cache mismatch and (ii) trajectory-induced calibration mismatch but never reports their magnitude. Since the calibration procedure already collects AuA_u and BuB_u at every selected site, I think the authors could report, for a fixed configuration (model, guidance scale, number of steps, cache policy π\pi), feature-level statistics such as ∥Au−Bu∥\|A_u - B_u\| before and after applying Cu\mathcal{C}_u, and the distance between BuB_u collected on the uncorrected trajectory versus on the calibrated one, as a function of timestep and layer. (2) In Table 6, one-shot calibration degrades FID at α=0.1\alpha = 0.1 and collapses at α=0.5\alpha = 0.5, well inside the interpolation range. The paper only states that the operators are mismatched to the shifted trajectory, which explains the direction but not the magnitude of this failure. Since the one-shot and TCC operators coincide at the first calibration timestep by construction, could the authors show where the divergence begins, e.g., by enabling the one-shot operators one timestep at a time? Please also confirm that the one-shot variant uses the same 19–12 window as TCC; the caption of Table 6 (“all cached attention and MLP outputs”) leaves this ambiguous, and if it extends to late timesteps the comparison would be confounded by the late-step instability reported in Appendix B.1. (3) The stated goal of TCC is to align the cache-side trajectory with the full-computation trajectory, and the qualitative figures already use matched seeds. It would therefore be informative to report per-sample distances between TCC outputs and the corresponding full-computation outputs—e.g., LPIPS and a DINO-based structure distance.

W3. (Generalization assumption) (1) TCC fits Cu\mathcal{C}_u on KK representative trajectories and applies it to unseen prompts, which requires the mismatch au−bua_u - b_u to be dominated by a prompt-independent component. The authors state this as their “key modeling assumption” (Sec. 3.1) but never test it; Appendix B.1 analyzes activation dispersion for a single ImageNet class, not the mismatch. I would ask for a per-timestep, per-layer statistic of the direct cache mismatch on the evaluated prompt set (e.g., the ratio of the mean-mismatch norm to the average per-prompt deviation), together with the Procrustes residual on held-out vs. fitting prompts. This would show why a 1K-prompt operator should transfer and would ground the choice of calibration window. (2) Table 9 also needs clarification. More representative prompts should reduce the variance of the estimated statistics, yet 2K performs worse than both 1K and 5K. Since all settings use α=0.25\alpha = 0.25, the value selected for the 1K configuration, the comparison favors 1K by construction. Could the authors re-select α\alpha per prompt count, add 3K and 4K, and repeat each count over several random prompt subsets? The spread in Table 9 (~0.8 FID) is much larger than the seed-to-seed deviation in Table 5 (0.15), suggesting that it is driven by the prior estimate itself.

Questions for authors

See above weaknesses.

Reviewer_oakwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Trajectory-Consistent Calibration (TCC), a training-free method for improving image quality in cache-accelerated diffusion models. It uses closed-form feature transformations to align cached features with those obtained through full computation, sequentially estimating calibration operators along the denoising trajectory to reduce accumulated errors. Experiments on PixArt-α and DiT-XL/2 show that TCC improves generation quality when combined with existing caching methods, as measured by metrics such as FID. However, these improvements come with additional computational overhead, and the evaluation is limited to two relatively established DiT models, leaving its effectiveness on more recent, high-quality text-to-image models unclear.

Critical strengths

The paper proposes a reasonable and training-free approach to improving the quality of cache-accelerated diffusion models. By calibrating cached features toward their full-computation counterparts, the method aims to compensate for the errors introduced by feature reuse. Its trajectory-consistent calibration strategy is particularly interesting, as it accounts for the changes in subsequent denoising trajectories induced by earlier calibration steps.

Critical weaknesses
  1. Trade-off between generation quality and inference efficiency. Although TCC improves generation quality, it also introduces additional computational overhead, reducing the overall speedup. For example, the speedup of PixArt-α with FORA decreases from 2.15× to 1.82× after applying TCC. It would be helpful to provide a more comprehensive analysis of the quality-efficiency trade-off across different caching strategies and acceleration ratios, to clarify whether the quality improvements justify the additional computational cost.
  2. Limited evaluation on modern, high-quality generation models. The experiments are limited to PixArt-α and DiT-XL/2, without evaluations on more recent models such as FLUX or Stable Diffusion 3/3.5. Given the differences in model architectures and feature representations, it remains unclear whether the proposed calibration method generalizes to more capable, modern text-to-image systems. Additional experiments on these models would strengthen the evidence for the method's general applicability.
  3. Limited qualitative evaluation. The paper provides only a limited number of qualitative comparisons, making it difficult to fully assess how the proposed method affects perceptual quality across diverse prompts and challenging generation scenarios. More extensive visual comparisons, including failure cases, would help clarify whether the improvements in quantitative metrics translate into consistent perceptual benefits.
  4. [major concern] Limited Methodological Novelty The proposed method addresses a relevant problem in cache-accelerated diffusion models, but its methodological novelty appears limited. Feature calibration and cache-error compensation have already been explored in prior work, including ICC, EOC, and GOC. Moreover, FoCa, published at AAAI 2026, adopts a predictor–corrector framework to calibrate hidden-feature evolution and improve the stability of cache-accelerated sampling. The main contribution of TCC therefore appears to lie in its specific strategy of estimating calibration operators sequentially along the corrected denoising trajectory, rather than in a fundamentally new calibration mechanism. While this design is reasonable, it remains unclear whether it constitutes a sufficiently substantial advance over existing calibration and error-compensation methods. The ablation comparing trajectory-consistent estimation with one-shot calibration supports the proposed design, but does not establish its advantage over the most relevant prior approaches. A more explicit discussion of the relationship to existing predictor–corrector and calibration methods, together with direct comparisons under matched settings, would be important to establish the novelty and significance of TCC.
Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.