acceptodds
Under review as a conference paper at ICLR 2027

Closed-Form Least-Squares Forecasting for Fast Diffusion Transformers

Abstract

Diffusion transformers spend most of their inference budget re-evaluating a network whose outputs change slowly along the sampling trajectory. Cache-based accelerators exploit this redundancy by skipping selected steps, and their quality is governed by two levers—which steps to compute and how to fill in the skipped ones—that have mostly been studied in isolation. We build a framework that exposes both at once and use it as a controlled attribution study. Our predictor treats the sequence of whole-transformer outputs as the observable of a discrete-time system and forecasts a skipped output as a linear functional of previously computed outputs, fitted either per image by global least squares or, once for all images, by closed-form calibration on a handful of prompts requiring no gradient training. The attribution result is unambiguous and survives every control we apply: step selection dominates predictor design, and per-image adaptive gating is worse than committing to a schedule even at a lower skip rate. Repeating the whole protocol on SD3.5-Medium, a B MMDiT-X model with flow matching and a different text stack, shows the same ordering and a better operating point: wall-clock at CLIP below the uncached model with of its sharpness, and CLIP at or above the uncached model at a -step budget and at px. On PixArt- the method reaches – at alignment matching the uncached model on two benchmarks.

Primary area:
generative models
Keywords:
diffusion models, diffusion transformers, feature caching, training-free acceleration, closed-form forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jcarOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an acceleration method for diffusion transformers that requires no gradient-based training. It computes a fixed subset of denoising steps and approximates the skipped transformer outputs as linear combinations of previously computed outputs. The predictor uses either polynomial least-squares extrapolation based on the computed step indices or closed-form calibration over a small collection of full trajectories. The paper also develops gap-bounded schedules and studies the relative effects of step selection and predictor design.

The central empirical claim is that step selection has a greater influence on image-text alignment than the predictor variants tested, while forecasting and calibration can improve sharpness on PixArt-alpha. Experiments report approximately 2.3-2.9x wall-clock speedups on PixArt-alpha and a 3.39x operating point on SD3.5-Medium. The latter uses 4-bit weights and a 28-step flow-matching trajectory, whereas PixArt uses 20 DDIM steps. The paper includes probes across sampling budgets, resolutions and solvers, as well as calibration-size and seed ablations. Although CLIP alignment is largely preserved at the selected operating points, the sharpness trade-off is appreciably larger on PixArt than on SD3.5.

Critical strengths
  1. Simple method with low inference overhead.

The method leaves the pretrained model weights and the surrounding solver unchanged. A skipped step requires only a weighted sum of cached outputs, and calibration uses a small closed-form solve. This makes the approach practically attractive and provides a useful low-cost reference for studying diffusion caching.

  1. Useful controlled experiments across two architectures.

The paper separates scheduling from prediction and includes fixed-schedule comparisons of predictor variants, calibration-size studies, multiple seeds, and replication on SD3.5-Medium. These experiments provide useful evidence about where quality changes arise. The inclusion of sharpness diagnostics and ImageReward also helps expose degradation that CLIP alone may miss.

Critical weaknesses
  1. Limited methodological novelty and overly broad conclusions.

L2P already learns linear combinations of historical features. Closed-form fitting is a useful simplification, but the methodological advance appears incremental. This makes the attribution study particularly important. The experiments, including the comparison at matched wall-clock cost, support the selected schedules against the gates tested, but do not establish that fixed schedules generally outperform adaptive methods or that predictor design is secondary across caching approaches. Clearer positioning relative to recent methods such as TAP, SeaCache, SVD-Cache and LESA is needed to establish the scope and added contribution.

  1. Unclear calibration protocol for the attribution study.

Section 3.3 states that calibrated weights are shared across schedules, yet Eq. (3) depends on the selected steps and history indices. It is unclear whether the numerical weights are reused or refitted for each schedule. This distinction matters for the attribution study: refitting can change the predictor alongside the schedule, while reusing weights across different histories requires justification.

Questions for authors
  1. Calibration protocol (clarification): Were the numerical weights held fixed across schedules and computed-step budgets, or was Eq. (3) solved separately for each schedule using the same calibration trajectories? Please reconcile the schedule dependence of Eq. (3) with the reuse statement in Section 3.3. If the weights are reused, how are changes in the history indices handled?

  2. Schedule selection and data separation (clarification): The paper describes residual screening followed by direct rendering. What criterion determines the final operating point after rendering, and which prompts and seeds are used for this selection? Please distinguish the data used for coefficient fitting, schedule selection and final evaluation, and clarify whether these sets overlap.

  3. Scope of the attribution claim (evidence affecting the assessment): Could the authors provide a representative comparison with a recent adaptive method such as SeaCache or TAP at comparable wall-clock cost, or explain any adaptation barriers and narrow the general conclusion accordingly? Please also discuss how SVD-Cache and LESA affect the claim about the relative importance of predictor design, accounting for their different preparation costs.

  4. Benefit of the estimator (evidence affecting the assessment): Beyond the reported comparison of prediction residuals, under an identical schedule and prediction target, does an end-to-end comparison with Taylor-style forecasting show a meaningful difference in final image quality? Such a control would help distinguish the practical contribution of the estimator from that of the selected schedules; it need not demonstrate that least squares is superior if the intended contribution is primarily the attribution study.

Reviewer_hoP2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A few-step acceleration method is proposed. The claimed novelty is the step weights solved in close form.

Critical strengths
  1. The problem is decomposed well and I like!
  2. I also like the idea to precompute the weight and solve it as a independent problem
Critical weaknesses
  1. L2P [1] is somewhat too close to the method and limit the novelty
  2. Sould be compared also to more NFE samplers, should be compared to like 6-8 steps NFE samplers
  3. Table 1, TeaCache [2] has CLIP 30.18, sharpness 816.5, but this is 1.43× acc. should take a look, why it is like the same as unached? in MJHQ-30K?
  4. should really cite Spectrum [3] and it is similar in your Eq 2

[1] Zhirong Shen, Rui Huang, Jiacheng Liu, Chang Zou, Peiliang Cai, Shikang Zheng, Zhengyi Shi, Liang Feng, Linfeng Zhang. Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models. CVPR 2026. arXiv:2604.26365.

[2] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, Fang Wan. Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model. CVPR 2025. arXiv:2411.19108.

[3] Jiaqi Han, Juntong Shi, Puheng Li, Haotian Ye, Qiushan Guo, Stefano Ermon. Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration. CVPR 2026. arXiv:2603.01623.

Questions for authors

Is this essentally similar to NFE samplers?

Reviewer_gqGpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a caching method for diffusion transformers, accelerating inference by selectively executing model evaluations and approximating skipped outputs as linear combinations of previously computed outputs. Specifically, the authors propose least-squares extrapolation based on timestep indices and a prediction method calibrated in closed form using a small set of prompt trajectories. Neither approach requires gradient-based training or modifications to the pretrained models. These predictors are combined with fixed schedules that limit the number of consecutive skipped steps. Experiments cover PixArt-α and SD3.5-Medium on COCO, MJHQ, and PartiPrompts, demonstrating good acceleration with preserved CLIP scores.

Critical strengths
  1. This method adopts least-square forecasting to approximate the skipped full inference. The method does not relies on training and only requires a few calibration data/prompt. This makes the method very practical.
  2. The performance is good. On PixArt, the method achieves approximately 2x+ acceleration while maintaining comparable CLIP scores, although some fine detail is lost. On SD3.5, the configuration using eight model evaluations achieves 3+ acceleration with a slight decrease in CLIP score and retains approximately 97% of the baseline sharpness statistic.
Critical weaknesses
  1. In Table 1, the seven-step and six-step exps on PixArt-α with COCO largely preserve CLIP scores, but retain only approximately 29% and 16% of the uncached model’s sharpness. These results suggest that preserved text–image alignment may still come at a cost to visual quality. Additional perceptual quality evaluation would help readers assess the practical speed–quality trade-offs of these configurations.
  2. The additional benefit of the predictor over simple reuse could be clarified. In Table 14 (Appendix D.4), under the fixed eight-step schedule on SD3.5-Medium with COCO, verbatim reuse achieves a CLIP score of 31.61, compared with 31.53 for affine least-squares prediction, at essentially the comparable latency (1.09 vs. 1.10 seconds per image). Further clarification of when the predictor provides consistent benefits, and how these benefits compare with those from scheduling, would be helpful.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.