acceptodds
Withdrawn from ICLR 2027

Smooth Residual Adaptation for Video Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Improving conditional video generation involves choosing the form of a correction as well as its strength. Conditional smoothness offers a structural bias for this choice, but imposing it on a model's existing response also restricts the conditional behavior it can express. We introduce smooth residual adaptation for video diffusion, which constrains the added correction while retaining the model's nonlinear response. The residual is linear in selected condition coordinates at a fixed sampling state, while its coefficients adapt to the evolving video. Gaussian projection characterizes the approximation cost of excluding higher-order responses, and finite-sampler sensitivity describes how successive corrections affect the output. Experiments across multiple tasks and datasets show that smooth residual adaptation can improve generative performance across different conditioning interfaces.

Primary area:
generative models
Keywords:
video diffusion models, generative models, conditional generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_36BHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Smooth Residual Adaptation (SRA), a construction for improving conditional video diffusion models by adding a constrained learned correction on top of a frozen parent model. The central idea is a deliberate factorization of nonlinearity: the correction is a residual, where selected condition coordinates q enter through a linear path, while the operator obtains its coefficients from a reference evaluated at the current sampling state and is recomputed as denoising proceeds. The correction is thus linear in the condition coordinate at any fixed state/reference, but adapts nonlinearly to the evolving video; "smoothness" refers to bounded curvature of the added response in these coordinates. A zero-initialized readout makes the adapted model coincide with the parent at the start, and an optional learnable spatial/temporal mask weights the residual.

Critical strengths
  1. A single, well-motivated construction that unifies several conditioning interfaces with low deployment risk. The reference-gated-operator / linear-condition-path factorization is applied, essentially unchanged, to action prediction, editing, and text/image generation. This is consequential because unification of this kind is rare and valuable: zero-initialization and the frozen parent make the method a low-risk add-on, and the shared construction suggests the insight is structural rather than task-specific. This generality is the primary reason the paper merits serious consideration despite its marginal numbers.

  2. Intellectual honesty and statistical discipline. The paper explicitly delimits what its theory does and does not establish (characterization vs. distributional guarantee; the Gaussian measure as analytical, not empirical; derivative bounds unestimated at the deployed dose), and the empirical protocol uses held-out evaluation, bootstrap confidence intervals, paired comparisons, and multi-fit means with dispersion. This candor and rigor are above the norm for the area and make the paper's claims unusually easy to assess at face value — a non-trivial virtue that materially increases my trust in the (modest) results.

Critical weaknesses
  1. The empirical gains are marginal, and their practical significance is not established. On the formal held-out task (Table 2, LIBERO), LPIPS moves 0.28174 → 0.28020 and PSNR 21.754 → 21.753 (effectively unchanged); the clearest movement is FVD (640.4 → 612.1). Editing improves Overall 3.566 → 3.616 on a 1–5 scale (+0.05). These may be statistically distinguishable, but the paper never demonstrates that they correspond to a perceptible improvement. For a venue that generally expects either a clear empirical win or a deployable theoretical advance, "statistically significant but plausibly imperceptible" is the single biggest obstacle to acceptance.

  2. No comparison against the learned-conditioning methods the paper positions itself against. The related-work section frames SRA relative to ControlNet, T2I-Adapter, IP-Adapter, and guidance variants, but the experiments compare only to the frozen parent and an energy-matched random residual. We therefore learn that SRA beats doing nothing (and beats random corrections of equal energy), but not whether it is competitive with, or preferable to, the established adapters/guidance it claims to improve upon. Without this, the contribution's standing relative to the state of the art cannot be judged.

  3. The theory is characterization-only and, by the authors' own statements, disconnected from the deployed regime. Lemma 1 relies on a Gaussian coordinate measure the paper says "is not an established model" of the encoded actions/videos/text, and Proposition 1's bounds "have not been estimated for the deployed models," leaving "the approximation at the deployed dose η=1 unverified." The analyses thus motivate the design but neither predict nor explain the observed gains; the heavy theoretical apparatus and the thin empirical payoff are badly mismatched in weight.

  4. Several central design choices are asserted rather than empirically justified against simpler alternatives. The paper notes that "direct guidance and nonlinear heads using the same reference information show no stable ordering" on development inputs — i.e., SRA's restricted linear-in-condition form is not shown to beat an unrestricted nonlinear head or a directly calibrated contrast. Combined with a rank-8 operator retaining 99.26% of the Frobenius norm (suggesting the learned correction is very low-rank / near-trivial), this raises the concern that the specific constrained parameterization may not be doing much work beyond what a far simpler correction would.

Questions for authors

(→ W1, practical significance) Can you demonstrate that the measured gains are perceptible, not just significant? Concretely: (a) a human preference / side-by-side study on the editing and generation outputs with win-rate and CIs; and (b) for prediction, an FVD-centered analysis (your most-moved metric) with qualitative panels isolating cases where sub-0.002 LPIPS/PSNR changes do or do not correspond to visible differences. If gains remain imperceptible, please reframe the contribution accordingly.

(→ W2, missing baselines) Please add head-to-head comparisons against at least one learned-conditioning method per interface that you cite (e.g., a ControlNet/IP-Adapter/T2I-Adapter-style conditioning pathway, and a directly calibrated/scheduled guidance baseline), trained and evaluated under your shared inputs, noise, and sampler settings. The key deliverable is evidence that SRA is competitive with — ideally better than — these established methods, not only better than the frozen parent and random residuals.

(→ W3, theory–practice gap) Can you empirically connect the theory to the deployed setting? Specifically: (a) estimate or bound the finite-sampler constant C / derivative magnitudes for at least one deployed model so Proposition 1 says something at η=1; and (b) report the measured curvature / higher-order residual term from Lemma 1 on real condition coordinates, so readers can see how far the Gaussian-measure characterization is from the actual data. If the bounds cannot be made informative, please state explicitly that the theory is motivational and separate it from the empirical claims.

(→ W4, is the constraint load-bearing?) Please provide a controlled comparison that isolates the value of the constrained SRA form: (a) an unrestricted nonlinear head and a directly calibrated contrast using identical reference information, reported with the same statistics (not just "no stable ordering"); and (b) given that rank-8 retains 99.26% of the operator norm, an explicit low-rank/near-identity ablation showing what minimal correction recovers most of the gain. The deliverable is a demonstration that the specific smooth/linear-in-condition parameterization provides a measurable benefit over simpler corrections — or, if it does not, a revised claim.

Reviewer_JGT1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Smooth Residual Adaptation (SRA), an add-on that enhances video diffusion models while keeping the original model frozen. SRA adds a correction that scales linearly with the condition and adapts to the video as it is generated. The video prediction, editing, and generation show modest improvements over the original models.

Critical strengths
  1. Simple way to adapt existing video models.

SRA learns a small correction while keeping the model fixed, which can handle actions, text, and image inputs, making it useful for adapting pretrained models to different video tasks.

Critical weaknesses
  1. Benefit of making the correction linear is unclear

SRA does not consistently outperform simpler guidance methods or nonlinear corrections. The results do not show the main design choice is responsible for the gains.

  1. Evidence across different tasks is limited

Text generation and image generation results use only 8 development examples, and unclear whether the improvement extends to new inputs. Also editing improvement is smalland confidence interval includes zero.

Questions for authors
  1. Do the generation improvements persist on inputs excluded from configuration selection?
Reviewer_QD5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Smooth Residual Adaptation (SRA), a lightweight correction added to a frozen video diffusion model's conditional prediction at selected sampling steps. The correction is a matrix computed from the model's features under a reference condition (e.g., empty text or a neutral action), multiplied by a "condition coordinate": the difference between the actual and reference conditions in some representation, such as an action or text-embedding difference, or the conditional-minus-reference prediction. The correction is therefore linear in the condition difference at a fixed sampling state, while the matrix adapts to the evolving video; with prediction-difference coordinates, SRA is a learned linear transformation of the guidance contrast. The paper analyzes the approximation cost of this linear restriction under a Gaussian measure and the first-order effect of inserted corrections on a finite sampler's output. Experiments add SRA heads (50K–1.3M trainable parameters) to frozen AdaWorld (action-conditioned prediction on LIBERO, Franka and Jaco), LucyEdit (editing on OpenVE-Bench), CogVideoX-2B and Wan2.2-I2V-A14B (generation), and report small improvements over the parents.

Critical strengths
  1. Extensive experiments with detailed settings. SRA is evaluated on four frozen parents spanning action-conditioned prediction, text-guided editing, and text- and image-conditioned generation, and the appendices document data roles, calibration, compute, and implementation for each (Tables 7 and 25), including a held-out LIBERO test on 30 disjoint tasks × 50 clips.
  2. Consistent, though small, gains over the frozen parents with small heads. For example, on held-out LIBERO, LPIPS improves from 0.28174 to 0.28020 and FVD from 640.4 to 612.1; on Jaco, FVD improves from 537.1 to 503.7; on OpenVE-Bench, SRA has a higher mean score than LucyEdit in all eight categories (Overall 3.566 → 3.616); and on CogVideoX-2B, five VBench quality dimensions improve on average.
Critical weaknesses
  1. Presentation is hard to follow and lacks visual aids. The abstract and main text rely on unfamiliar terms ("conditional smoothness", "condition coordinate", "Gaussian projection", "finite-sampler sensitivity") and abstract descriptions, without concrete video-generation examples or visual illustrations of what they mean. For example, the abstract's "Gaussian projection characterizes the approximation cost of excluding higher-order responses, and finite-sampler sensitivity describes how successive corrections affect the output" does not indicate that these refer to the paper's own theoretical analyses (Lemma 1 and Proposition 1). As a reader working on video generation, I could not understand what the method does from the abstract alone; the concrete head for each task only becomes clear in Appendix C. Qualitative evidence is also limited: there are no videos, only frames (Figures 5 and 6), and in Figure 5 I cannot see a difference between AdaWorld and SRA, despite the caption's statement that "SRA shows more distinct basket texture and object boundaries than AdaWorld".
  2. Gains are small, and the central design choice is not supported by the paper's own comparisons. Improvements are on the order of 10⁻³ in LPIPS and VBench (PSNR is unchanged on LIBERO: 21.754 vs. 21.753), and the OpenVE-Bench Overall gain of +0.049 on a 1–5 scale has a 95% interval of [−0.012, 0.111] (Table 24). More importantly, the comparisons in Appendix D.10 do not show that the linear (smooth) correction helps: on Wan text-to-video, the nonlinear head has lower FVD than the linear head in both blocks (3834.5 vs. 3999.2; 3923.6 vs. 3935.2), and on AdaWorld there is no stable ordering between linear and nonlinear heads. The restriction that gives the method its name therefore lacks empirical support.
Questions for authors
  1. (Clarification) Given that the nonlinear head matches or outperforms the linear head in Appendix D.10, what evidence beyond Lemma 1 supports restricting the correction to be linear in the condition coordinates?
  2. (Experiment) Could you provide videos comparing each parent with SRA, and illustrate in the video-generation setting what "conditional smoothness" means (e.g., how the output changes along a condition coordinate)?
  3. (Experiment) For editing, can you show whether the gain is significant with more judge repetitions or a human evaluation, given that the Overall interval includes zero?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.