Efficient Residual-Centric Inference for Spacetime Autoregressive Video Generation

Spacetime autoregressive video models combine multiscale spatial generation within each clip with temporal autoregression across clips, avoiding repeated full-resolution denoising while enabling previously generated visual states to be reused as context. This design, however, introduces two complementary inference bottlenecks: attention computation becomes concentrated at late, high-resolution scales, while reusable history creates a persistent KV cache. A natural solution is to sparsify attention and compress historical KV, but the two are typically optimized with different proxy objectives. We observe that once omitted attention is replaced by a cheap pooled correction, the relevant quantity is no longer raw attention importance or standalone tensor reconstruction error, but the output residual left unexplained by the approximate attention operator. Based on this observation, we introduce residual-centric inference, which provides a shared post-correction residual view for compute allocation and cache representation. We instantiate this principle with Output-Sensitive Routing (OSR), which allocates token-level attention using a proxy for post-correction residual risk, and Residual-Interaction Quantization (RIQ), which preserves pooled statistics and fits low-bit history representations to residual K–V interactions. On InfinityStar, OSR achieves up to 1.86× end-to-end acceleration at native 720p/5s, while an orthogonal FFN FP8 optimization increases the complete-system speedup to 2.01×. At 480p/10s, OSR+RIQ achieves up to 1.50× speedup and 5.74× retained-history KV compression. Across both settings and tasks, these configurations largely preserve generation quality, with at most 0.31% relative degradation in VBench and VBench-I2V scores.