Smooth Residual Adaptation for Video Diffusion Models

Improving conditional video generation involves choosing the form of a correction as well as its strength. Conditional smoothness offers a structural bias for this choice, but imposing it on a model's existing response also restricts the conditional behavior it can express. We introduce smooth residual adaptation for video diffusion, which constrains the added correction while retaining the model's nonlinear response. The residual is linear in selected condition coordinates at a fixed sampling state, while its coefficients adapt to the evolving video. Gaussian projection characterizes the approximation cost of excluding higher-order responses, and finite-sampler sensitivity describes how successive corrections affect the output. Experiments across multiple tasks and datasets show that smooth residual adaptation can improve generative performance across different conditioning interfaces.