Closed-Form Least-Squares Forecasting for Fast Diffusion Transformers

Diffusion transformers spend most of their inference budget re-evaluating a network whose outputs change slowly along the sampling trajectory. Cache-based accelerators exploit this redundancy by skipping selected steps, and their quality is governed by two levers—which steps to compute and how to fill in the skipped ones—that have mostly been studied in isolation. We build a framework that exposes both at once and use it as a controlled attribution study. Our predictor treats the sequence of whole-transformer outputs as the observable of a discrete-time system and forecasts a skipped output as a linear functional of previously computed outputs, fitted either per image by global least squares or, once for all images, by closed-form calibration on a handful of prompts requiring no gradient training. The attribution result is unambiguous and survives every control we apply: step selection dominates predictor design, and per-image adaptive gating is worse than committing to a schedule even at a lower skip rate. Repeating the whole protocol on SD3.5-Medium, a $2.0$B MMDiT-X model with flow matching and a different text stack, shows the same ordering and a better operating point: $3.39\times$ wall-clock at $0.34$ CLIP below the uncached model with $97\%$ of its sharpness, and CLIP at or above the uncached model at a $30$-step budget and at $768$px. On PixArt-$\alpha$ the method reaches $2.6$–$2.7\times$ at alignment matching the uncached model on two benchmarks.