acceptodds
Under review as a conference paper at ICLR 2027

High-Order Drifting Models

Abstract

Drifting models enable efficient one-step generation by evolving the model distribution through a kernel-induced transport field during training. Existing drifting methods, however, typically use only the local field value at sampled anchors, ignoring the local response structure of the same field. This leaves local curvature and higher-order structural mismatch unused, limiting convergence and generation fidelity. To address this problem, we propose Higher-Order Drifting Models, which exploit the local response hierarchy of the original kernel transport field. We show that the response derivatives of the field form a hierarchy of local cumulants; for Gaussian kernels, these cumulants correspond to spatial derivatives of the smoothed log-density ratio, extending the score-matching interpretation of drifting to higher orders. We further show that, under a local linearization, the higher-order responses act as a spectral reweighting that enhances the effective damping of transition-region modes without altering the asymptotic high-frequency behavior of the kernel. The experiments show that these responses provide complementary optimization signals while preserving one-step inference, accelerating convergence and improving generation accuracy, with diminishing returns as the order increases.

Primary area:
generative models
Keywords:
Generative Model, One-step Generation, High-order Drifting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YkBCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an extension of drifting models that considers higher-order statistics of the model and data distributions. This paper introduces the tilted kernel-local expectation, which is a generalization of the mean shift term in drifting models; its derivative corresponds to the cumulant tensors associated with the kernel-localized distributions. The proposed training loss is constructed from the discrepancy between the cumulant tensors, enabling models to take higher-order statistics into account. In addition, this paper provides a theoretical analysis of the correction effect of the proposed method from the perspective of spectral analysis of the drifting dynamics. Experimental evaluations are conducted on a Gaussian mixture dataset and image datasets, and the proposed method achieves better scores than the baseline drifting models in most cases.

Critical strengths
  1. This paper is well-organized. The motivation and the technical background are clearly explained.
  2. The proposed method is supported by theoretical foundations. The construction of the tilted kernel-local expectation is sophisticated and successfully achieves considering higher-order statistics. In addition, theoretical analyses of its behavior, such as Proposition 3, are provided, which deepen the understanding of the proposed method and drifting models.ss.
Critical weaknesses
  1. Some parts lack sufficient explanation. Some examples are listed as follows.
  • In Eq. (12), the derivative of Em\mathcal{E}_m with respect to XX is taken. However, it is unclear what XX representes, as XX does not appear in the quantities involved in the computation of Em\mathcal{E}_m involves.
  • Is waiqw_{ai}^q missing from Eq. (13)?
  • In Eqs (11) and (13), summation over anchor points aa is taken. However, how the anchor point set is constructed remains unclear.
  • There seem to be incomplete sentences around Line 268:
    • "only in the optional extension of" what?
    • "The stop-gradient operator is denoted by" what?
  • In experiments, explanations of methods denoted by "Parallel-only" and "Orthogonal-only" are largely omitted, alghough it can be presumed that these are related to the cosine gate in Eq. (18).
  1. This paper mainly focuses on the Gaussian kernel and ignores the Laplacian kernel, which is practically used [1]. In addition, the spectral analysis of [2] also suggests that the Laplacian kernel might be favored for drifting models. While Proposition 3 ensures the correction effect of the proposed method, it seems that the cut-off issue of the Gaussian kernel still remains, as the exponential bottleneck in the high-frequency region is unchanged. One critical concern is that changing the kernel might have a greater impact on model performance than considering higher-order statistics. Experiments regarding the Laplacian kernel are not conducted. Further experimental comparison using the Laplacian kernel or further discussion on this point would enhance the demonstration of the effectiveness of the proposed method.

[1] M. Deng et al. "Generative modeling via drifting." arXiv preprint arXiv:2602.04770 (2026).

[2] E. Turan et al. "Generative drifting is secretly score matching: a spectral and variational perspective." arXiv preprint arXiv:2603.09936 (2026).

Questions for authors

Regarding Weakness 2, could further experiments using the Laplacian kernel or further discussion on the Laplacian kernel be provided?

4 comments
AuthorsOpenReview ↗

Response part 1 to Reviewer YkBC

W1: Clarifications on formulas, terminology, and missing explanations

Thank you for identifying these ambiguities. We agree that the particle dependence, anchor construction, and directional ablations require clearer explanations. We have revised the relevant passages and marked the changes in blue. Equation numbers below refer to the original submission.

1. Meaning of XX in Eq. (12)

We now define X=(x1,…,xn)∈Rn×dX=(x_1,\ldots,x_n)\in\mathbb R^{n\times d} as the generated particle coordinates. The derivative acts on the finite-particle surrogate Em(X)\mathcal E_m(X), whose generated-side local cumulants depend on XX. Anchor locations, normalized weights, local scales ℓa\ell_a, and data-side statistics are held fixed during this differentiation; the generated-side means and cumulants retain their dependence on XX. This makes explicit the transition from the population hierarchy to the particle correction. We also retain the distinction between this surrogate and its independent-fold estimator in Appendix B.3.

2. Weight waiqw^q_{ai} in Eq. (13)

No additional factor is missing: the weight is already included in the weighted covariance,

\mu_{a,q}(X)&=\sum_j w^q_{aj}x_j,\\ C_{a,q}(X)&=\sum_j w^q_{aj}(x_j-\mu_{a,q}(X))(x_j-\mu_{a,q}(X))^\top. \end{aligned}$$ Writing $e_{ai}=x_i-\mu_{a,q}(X)$, fixed-weight differentiation gives $$dC_{a,q}=w^q_{ai}\left(dx_i e_{ai}^{\top}+e_{ai}dx_i^{\top}\right),$$ which produces the explicit $w^q_{ai}$ in Eq. (14). We have added these definitions and the differentiation step to make the role of the weight clear. ### 3. Anchor construction The revised finite-batch formulation explicitly specifies the indexed anchor collection $\mathcal A=(a_b)_{b=1}^{n}$, with $a_b=\operatorname{sg}(x_b)$. Anchors are refreshed at each training step and held fixed when differentiating the local surrogate. This distinguishes their role as neighborhood centers from the differentiable particle coordinates. The baseline affinity normalization and applicable self-pair masking are retained when defining the local weights. ### 4. Incomplete sentences We have completed the reference to the optional third-order extension in Section 4.3 and explicitly defined $\operatorname{sg}(\cdot)$ as the stop-gradient operator, which preserves its input value while blocking gradients through that input. ### 5. Parallel-only and Orthogonal-only We have added their definitions in the training-objective subsection and cross-referenced them in the experimental description. Let $H=\alpha_2G^{(2)}$, with $\alpha_2$ computed from the full correction. Using the batch Frobenius inner product, for $V\neq0$ we define $$H_{\parallel}=\frac{\langle V, H\rangle_F}{\|V\|_F^2}V, \qquad H_{\perp}=H-H_{\parallel}.$$ Parallel-only uses $V+H_{\parallel}$; Orthogonal-only uses $V+H_{\perp}$. The gate and scale are computed before projection and are not recomputed for the retained component. In particular, applying the cosine gate again to the orthogonal component would suppress it. We have also replaced the ambiguous cosine notation in Eq. (18) with an explicit normalized inner product.
AuthorsOpenReview ↗

Response part 2 to Reviewer YkBC

W2: Kernel choice / Theoretical discussion

We thank the reviewer for raising this important point. We agree that the Laplace kernel is widely used in drifting models and that changing the kernel can have a significant impact. We have addressed this in both theory and experiments.

  1. Theoretical analysis

Our original theoretical discussion in Section 4.5/4/6 was mainly developed for the Gaussian case, where the covariance discrepancy admits a clean interpretation as a spatial derivative of the smoothed log-density ratio. However, the method itself is kernel-agnostic: Eq. (14) is defined through kernel-weighted local moments and does not rely on Gaussianity. To address this, we have completely rewritten Section 4.6 and Proposition 3 to cover general normalized nonnegative radial kernels with finite covariance. We explicitly provide the spectral rate and the covariance multiplier for both the Gaussian kernel and the radial Laplace kernel (Appendix A.5.2 and Figure 4). We show that for the Laplace kernel, the covariance correction strengthens finite-frequency damping while preserving the leading algebraic tail, providing a theoretical basis for why the method is not limited to the Gaussian case.

In normalized frequency u=σK∥ξ∥2u=\sigma_K\Vert\xi\Vert_2, with k^(ξ)=ψk(u)\widehat{k}(\xi)=\psi_k(u), the revised result separates the baseline rate from the covariance multiplier:

\lambda_{1,k}(u)=-u\psi_k'(u),$$ $$W_{2,k}(u)=1+c_2\bigl[\psi_k''(u)+\psi_k(u)\bigr],\qquad c_2=\frac{\alpha_2\sigma_K^2}{\ell}.$$ We now provide closed-form instances for both kernels used in the experiments. For the Gaussian kernel $k=G_h$, $\sigma_K=h$ and $\psi_G(u)=e^{-u^2/2}$, giving $$\lambda_{1,G}(u)=u^2e^{-u^2/2},\qquad W_{2,G}(u)=1+c_2u^2e^{-u^2/2},\qquad c_2=\frac{\alpha_2h^2}{\ell}.$$ For the radial Laplace kernel $k_b$, $\sigma_K^2=(d+1)b^2$ and, with $z=u^2/(d+1)$, $$\psi_L(u)=(1+z)^{-(d+1)/2},\qquad \lambda_{1,L}(u)=\frac{(d+1)z}{(1+z)^{(d+3)/2}},$$ $$W_{2,L}(u)=1+c_2\frac{z(z+d+4)}{(1+z)^{(d+5)/2}}.$$ In both cases $W_{2,k}(u)\to1$ as $u\to\infty$, so the correction strengthens finite-frequency damping while preserving the leading high-frequency tail of the underlying kernel. These instances show that the benefit is not specific to the Gaussian case. We have also added a scope statement in Section 4.6 and Remark 1 clarifying which proposition applies to which experiment. **2. Laplace kernel in the ImageNet experiments.** We would like to clarify that our ImageNet experiments **already use the Laplace kernel**. We continue training from the publicly released Drifting checkpoint, which originally used the Laplace kernel. We deliberately kept the kernel unchanged to isolate the effect of our higher-order correction. Therefore, the ImageNet improvements (FID 1.75 to 1.61) are obtained under the Laplace kernel, directly demonstrating the effectiveness of the covariance correction in the Laplace setting at scale.

I thank the authors for their reply and revised manuscript. Since the Critical Weaknesses I previously raised have been adequately addressed, I'll raise my score accordingly. However, I think that several passages in the revised text still require writing improvements, and I would ask the authors to address them in the final version. For example:

  • The revised definition of the stop-gradient operator in Line 273 is appreciated. However, since the notation sg(⋅)sg(\cdot) is already used in Section 3, it would be more appropriate to introduce this definition earlier. Please reconsider the placement of this explanation.

  • I am satisfied with the additional explanation about the weight waiqw_{ai}^q in Eq. (13). That said, I feel that the description in Line 231: "Here waiqw_{ai}^q denotes the normalized generated-side kernel weight for anchor aa and particle ii" remains difficult to understand. I would suggest that provinding the explicit form of waiqw_{ai^q} or adding a cross-reference to Eqs. (3) and (4) would make this part easier to understand.

  • In Line 303, the current text states "... extend the local-homogeneity spectral analysis of Turan et al. (2026) to a class of radial kernels". However, the assumption on the kernel κ∈L1(Rd)\kappa \in L^1 (\mathbb{R}^d) in the spectral analysis of Turan et al. (2026) appears to be that it is an even kernel, and radial kernels already satisfy this assumption. Therefore, I'm not sure that it is accurate to claim that the analysis has been extended to radial kernels. Rather, extending it to higher-order statistics would be a more suitable characterization of the provided theorem.

  • The current manuscript appears to heavily rely on Turan et al. (2026) for the definition and explanations of the linearized McKean-Vlasov dynamics and the spectral rates λeff,k\lambda_{eff,k} and λ1,k\lambda_{1,k}. For the paper to be more self-contained, I would recommend adding brief but sufficient explanations of these concepts, either in the main text or in the appendix, rather than leaving these details to Turan et al. (2026).

AuthorsOpenReview ↗

Thank you very much for your thorough feedback

Thank you very much for raising your score and for your thorough feedback. We value all your suggestions and will revise the main text to improve readability and manuscript self-containedness.

Reviewer_9QrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Higher-Order Drifting, an extension of Drifting Models that adds to the usual drift (the difference of kernel-weighted local means of data and generated samples) a correction based on the difference of local covariances, with an optional third-order term. The authors show that the higher derivatives of the local mean under exponential tilting are the local cumulants, and that for Gaussian kernels these cumulant differences are higher spatial derivatives of the smoothed log-density ratio. Under a linearization of the dynamics, the added term increases the damping of intermediate-frequency modes. The correction is used inside the same stop-gradient regression, with a cosine gate, normalization and trust-region scaling, so one-step inference is unchanged. Experiments on a 16-dimensional Gaussian mixture, MNIST, and fine-tuning of pretrained ImageNet 256 Drifting checkpoints show improvements: FID 1.73 →\to 1.59 in latent space and 1.74 →\to 1.64 in pixel space.

Critical strengths

The idea of exploiting more of the local structure of the kernel field is natural, and the method is simple to implement on top of existing drifting code, with no inference cost. The Gaussian identities give a clean and unified reading of the additional terms. The ImageNet results are the main asset. The improvement is consistent in both pixel and latent space, and the control showing that continued training with the original loss leaves FID unchanged is well designed and convincing as far as it goes. The directional ablation on the Gaussian mixture (rescaled drift, parallel-only and orthogonal-only components) is a good attempt to separate the effect of the correction from a mere change of step size. The paper is clearly written.

Critical weaknesses

The motivation overstates what the correction adds. The first-order drift already vanishes exactly when the smoothed densities, and hence the distributions, coincide, so no higher-order mismatch is “left unused” at the level of the objective. The added term can change the training dynamics, not the target. Under a Gaussian kernel, the covariance difference is proportional to the spatial derivative of the first-order field, so the correction essentially extrapolates the existing drift from the anchors to the particles, weighted by a tunable gain. That is a reasonable refinement of how the field is estimated, but it should be presented as such. The theory is largely standard: Proposition 1 is the definition of a cumulant-generating function, and Proposition 2 is the higher-order Tweedie identity (Efron, 2011; Meng et al., NeurIPS 2021), which should be cited. Proposition 3 is heuristic, with constants determined only up to proportionality and a sign that is assumed, and it does not cover the gated, normalized and clipped update actually used.

The link between theory and experiments is also loose. The Gaussian interpretation does not apply to the patch-restricted covariances used for images, and it is unclear whether the ImageNet runs, which keep the official Drifting configuration, use a Gaussian kernel at all. The method also adds several hyperparameters (trust-region ratio, warm-up, gate, patch size) on top of an already kernel-sensitive approach, and a comparison with simply tuning or mixing bandwidths is missing. Experimentally, the Gaussian mixture results come from a single seed, the MNIST FIDs (62 vs. 53) are far from what is achievable on that dataset and come from single runs, and the ImageNet standard deviations reflect evaluation sampling rather than independent training runs. The number of fine-tuning steps is not reported, and since only continuation from pretrained checkpoints is tested, the claim of faster convergence is not supported at scale.

B. Efron. Tweedie’s Formula and Selection Bias. Journal of the American Statistical Association 106(496):1602–1614, 2011.

C. Meng, Y. Song, W. Li, S. Ermon. Estimating High Order Gradients of the Data Distribution by Denoising. NeurIPS 2021.

Questions for authors

Which kernel is used in the ImageNet and MNIST experiments, and if it is not Gaussian, how do Sections 4.5–4.6 apply?

How large is the ImageNet improvement across independent fine-tuning runs, and how many fine-tuning steps were used?

Does the gain persist when training from scratch, and does it accelerate convergence as claimed?

How does the second-order correction compare with simply tuning or mixing kernel bandwidths at matched cost?

Can you comment on the reading of the correction as a local-linear extrapolation of the first-order field, and on whether a larger step size along the first-order drift, tuned at scale, recovers part of the gain?

4 comments
AuthorsOpenReview ↗

Response part 1 to Reviewer 9Qrc

We thank the reviewer for the constructive comments.

W1/Q1: Kernel choice / Theoretical discussion

We thank the reviewer for this question. We clarify the kernel choice and the scope of Sections 4.5–4.6 below. MNIST uses a Gaussian kernel, while ImageNet continues training from the released Drifting checkpoint and therefore uses a Laplace kernel; we keep the kernel unchanged to isolate the effect of the covariance correction.

Our original theoretical discussion in Section 4.5/4/6 was mainly developed for the Gaussian case, where the covariance discrepancy admits a clean interpretation as a spatial derivative of the smoothed log-density ratio. However, the method itself is kernel-agnostic: Eq. (14) is defined through kernel-weighted local moments and does not rely on Gaussianity. To make this explicit, we have rewritten the spectral proposition in Section 4.6 for general normalized nonnegative radial kernels with finite covariance. In normalized frequency u=σK∥ξ∥2u=\sigma_K\Vert\xi\Vert_2, with k^(ξ)=ψk(u)\widehat{k}(\xi)=\psi_k(u), the revised result separates the baseline rate from the covariance multiplier:

\lambda_{1,k}(u)=-u\psi_k'(u),$$ $$W_{2,k}(u)=1+c_2\bigl[\psi_k''(u)+\psi_k(u)\bigr],\qquad c_2=\frac{\alpha_2\sigma_K^2}{\ell}.$$ We now provide closed-form instances for both kernels used in the experiments. For the Gaussian kernel $k=G_h$, $\sigma_K=h$ and $\psi_G(u)=e^{-u^2/2}$, giving $$\lambda_{1,G}(u)=u^2e^{-u^2/2},\qquad W_{2,G}(u)=1+c_2u^2e^{-u^2/2},\qquad c_2=\frac{\alpha_2h^2}{\ell}.$$ For the radial Laplace kernel $k_b$, $\sigma_K^2=(d+1)b^2$ and, with $z=u^2/(d+1)$, $$\psi_L(u)=(1+z)^{-(d+1)/2},\qquad \lambda_{1,L}(u)=\frac{(d+1)z}{(1+z)^{(d+3)/2}},$$ $$W_{2,L}(u)=1+c_2\frac{z(z+d+4)}{(1+z)^{(d+5)/2}}.$$ In both cases $W_{2,k}(u)\to1$ as $u\to\infty$, so the correction strengthens finite-frequency damping while preserving the leading high-frequency tail of the underlying kernel. These instances show that the benefit is not specific to the Gaussian case. We have also added a scope statement in Section 4.6 and Remark 1 clarifying which proposition applies to which experiment. ## Q2: ImageNet improvement across independent fine-tuning runs and fine-tuning budget Previously, we trained a single model and performed five evaluation samples from that same checkpoint. We have now conducted two additional independent training runs for Drifting B/2 due to the time constraint. Each of the three checkpoints was evaluated five times (15 evaluations total), and the results are aggregated as mean $\pm$ standard deviation: FID $= 1.61 \pm 0.04$, IS $= 288.1 \pm 0.8$. For the standard Drifting model trained under the same setting, we obtain FID = $1.75 \pm 0.03$ and IS = $262.4 \pm 1.2$.
AuthorsOpenReview ↗

Response part 2 to Reviewer 9Qrc

Q3: Does the gain persist when training from scratch, and does it accelerate convergence?

Yes. The from-scratch evidence is in the GMM study in Appendix: Tables 5–6 and Figures 5 and 8 compare the controlled second-order correction with ordinary Drifting under identical settings. Across these runs the correction lowers both final SWD and MMD2\mathrm{MMD}^2, and the convergence curves reach a given metric value in fewer steps, supporting both claims.

We also found that a short warmup helps at the start of from-scratch training: when generated samples are still near noise, the covariance estimate is noisy and the higher-order direction can be unreliable. Early-phase directional decomposition shows this instability, and the warmup ablation in Appendix C.3 is consistent with it. MNIST is trained from scratch and uses warmup. ImageNet is not trained from scratch; it continues from a pretrained Drifting checkpoint, where the early noise-dominated phase has passed, so we did not use warmup there. Due to compute constraints, training an ImageNet generator from scratch is not feasible, and we state this limitation explicitly.

Q4: Comparison with tuning or mixing kernel bandwidths at matched cost

We thank the reviewer for this question. We agree that a matched-cost comparison is important. Our main comparisons are at matched optimization steps; we have now added a matched wall-clock analysis and will state explicitly that matched-step gains are not automatically matched-cost gains.

On low dim dataset GMM and MNIST, the additional covariance overhead is below 1%1\% in both time and memory, so matched wall-clock is essentially matched steps. Under the same wall-clock budget, ordinary Drifting with tuned or mixed bandwidths (MNIST already uses a multi-bandwidth mixture, {0.02,0.05,0.2}\{0.02,0.05,0.2\}) remains worse than Higher-Order Drifting.

On ImageNet, the per-step overhead is larger (10.63%10.63\% wall-clock time and 3.19%3.19\% memory). At matched wall-clock, we compare the Higher-Order checkpoint at 90%90\% of the baseline training time with the full-budget ordinary Drifting continuation; Higher-Order Drifting still achieves better FID. We also note that, with LoRA, ordinary Drifting first degrades slightly and then returns to the baseline, whereas the new HO term changes the update direction and continues to improve. We will add these matched-cost results and the corresponding limitation statement to the revised manuscript.

Q5: Local-linear extrapolation and larger first-order steps

We do not interpret the correction as a local-linear extrapolation of the first-order field, and our controls indicate that a larger step along the first-order drift does not recover the gain.

First, during training we tracked the angle between the ordinary drift field and the second-order term in space. For most particles this angle exceeds 60∘60^\circ, so the correction is not a small perturbation along the first-order direction. Intuitively, a component parallel to the first-order drift corresponds to a global translation of a misaligned model distribution, whereas the covariance term carries additional local shape information and acts in directions that the first-order field does not cover.

Second, the directional decomposition in Table 1 supports this reading at matched settings. Keeping only the component of the second-order term parallel to the first-order drift (parallel-only) gives little or no gain over ordinary Drifting, and a magnitude-matched rescaled drift behaves similarly. Keeping only the orthogonal component (orthogonal-only) recovers most of the gain, and the full correction is close to orthogonal-only. Thus the benefit is not reproduced by amplifying the first-order step; it comes from the directional, shape-related component of the covariance correction.

We therefore do not expect a tuned larger first-order step at scale to recover the gain. The matched-RMS rescaled-drift control in Table 1 is exactly such a scalar amplification, and it remains clearly worse than the full controlled second-order update.

I thank the authors for their reply. It did clarify some aspects of the implementation and calibration of the method. Still, my overall impression remains that the method proposed is a fairly incremental improvement on Drifting and does not really shed new theoretical light on it. I will keep my score. .

AuthorsOpenReview ↗

Thank you for your constructive feedback

We sincerely thank the reviewer for the careful follow-up and constructive feedback.

We understand the reviewer's remaining concern regarding methodological novelty. While our method is technically simple, we believe that its effectiveness and theoretical insights are valuable. In particular, we have strengthened the theoretical analysis in the revised manuscript by deriving the kernel-dependent spectral response of the proposed covariance correction, with explicit results for both Gaussian and Laplace kernels. This provides a more concrete understanding of how the correction affects drifting dynamics beyond the existing first-order analysis.

We will further clarify the positioning of our contribution and explicitly acknowledge the scope and limitations of the theoretical results.

We appreciate the reviewer's thoughtful discussion and consideration of our work.

Reviewer_mpY3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds higher-order information to drifting updates. The main method uses differences between local data and model covariances to construct an additional training direction. The authors also consider a third-order correction, estimated using random tensor contractions. Neither changes the generator or its one-step inference procedure.

The theory connects derivatives of local, exponentially tilted means to cumulants. For Gaussian kernels, these correspond to derivatives of the smoothed log-density ratio. The paper then uses a spectral analysis to explain why the corrections should improve convergence. Experiments on a Gaussian mixture, MNIST, and ImageNet report improvements from the covariance correction. The third-order term gives smaller additional gains in the GMM experiments.

Critical strengths

Strengths

  1. The covariance correction is straightforward and easy to interpret. It expands or contracts generated neighborhoods according to how their covariance differs from the data. The response–cumulant connection explains where this information comes from, and the correction can be added to the existing drifting objective while keeping one-step generation.
  2. The experiments include useful controls. The comparisons with rescaled drift and parallel/orthogonal corrections help isolate what the new direction contributes. On ImageNet, the authors compare against continued training with the original loss from the same checkpoint. The reported FID gains are 1.74±0.031.74 \pm 0.03 to 1.64±0.021.64 \pm 0.02 in pixel space and 1.73±0.021.73 \pm 0.02 to 1.59±0.031.59 \pm 0.03 in latent space. MNIST also improves, from 61.87 to 52.89, at the same stated training budget. Together, these results give some evidence that the correction is useful beyond the synthetic example.
Critical weaknesses

Weaknesses

  1. The experimental evidence is still quite limited. For an ICLR submission claiming improvements in both convergence and generation quality, I would expect a more thorough experimental study. The GMM and MNIST experiments are fine as initial toy tests, but they give a limited picture of when the method helps. Moreover, the main GMM comparison uses a single seed, and several tables and figures disagree about the results or training budget. For example, Table 5 states 1,000 steps while Figure 6 extends to 1,500, and the same results are assigned different rmax⁡r_{\max} values in Tables 1 and 5. These need to be reconciled.

    The ImageNet results are a useful next step, but are limited to continued training of pretrained models using LoRA. Comparing against continuation with the original loss is a good control. Still, I would like a clearer description of the setup, including whether both methods train identical LoRA parameters, and more evidence that the gains hold across training runs. The five repeated evaluations in Table 2 do not establish this. A few ablations in the image setting, for example, on the correction strength and the sparse covariance approximation, would also help explain what matters beyond the GMM example.

    Finally, the additional cost of the default covariance correction is not reported. Since faster training is one of the main claims, a comparison at matched wall-clock time seems important. Overall, these are useful first results, but I do not think they yet establish how reliable the gains are or whether they justify the extra computation.

  2. I am unsure about the spectral argument in Section 4.6. I had limited time to check the derivation in detail, so I may be missing something here. However, this seems important because it is the main theoretical explanation for why the higher-order correction should improve convergence.

    My confusion comes from comparing Eqs. (20) and (14). The covariance discrepancy in Eq. (20) already involves Gaussian-smoothed densities. Equation (14) then averages this discrepancy over anchors using a Gaussian kernel. I would therefore expect this average to introduce another smoothing step. Under the assumptions used in the analysis, with fixed coefficients and local variance scale, this would give two factors of e−h2∣ξ∣2/2e^{-h^2|\xi|^2/2}, resulting in e−h2∣ξ∣2e^{-h^2|\xi|^2}. Equation (41) instead contains only one.

    In the proof, the Taylor expansion in Eqs. (38)–(39) seems to be where this additional smoothing is approximated away. I can see how that gives a low-frequency approximation, but I am unsure why it also justifies the claims about transition and high frequencies in Proposition 3. Could the authors explain which frequencies this approximation covers?

    There is also a related point I could not reconcile: the proof says that λeff/λGauss→1\lambda_{\mathrm{eff}}/\lambda_{\mathrm{Gauss}}\to1, while Eq. (24) gives a polynomial ratio that grows for positive coefficients. These issues may leave the qualitative conclusion intact, but the stated formula and the proof need to agree.

Questions for authors
  1. Could you clarify which runs and checkpoints produced Tables 1 and 5 and Figures 2 and 6? In particular, please explain the different rmax⁡r_{\max} labels and the 1,000- versus 1,500-step budgets, and confirm which results were obtained at the same training budget.

  2. For ImageNet, did both continuation methods train the same LoRA parameters? Please provide the training and evaluation settings, steps, learning rate, batch size, and guidance scale, for both pixel and latent models. How many independent continuation runs were performed, and how much did the results vary between them?

  3. How much extra time per training step does the covariance correction add? Do the gains hold at a matched wall-clock budget, or have the methods only been compared at a matched number of optimization steps?

  4. Could you explain how the kernel averaging in Eq. (14) is accounted for in the spectral calculation? Since the covariance discrepancy already involves Gaussian-smoothed densities, I would expect another Gaussian factor. Is the result intended as a low-frequency approximation, and does it apply to the adaptive coefficient in Eq. (18), or only to fixed coefficients?

3 comments
AuthorsOpenReview ↗

Response part 1 to Reviewer mpY3

We thank the reviewer for the careful assessment and identifying the experimental reporting inconsistencies and the missing kernel factor in our spectral derivation. We address these concerns below.

1. Consistency of GMM results and training budgets (W1/Q1)

We apologize for the confusion. Specifically, in the original version:

  • All are 16-D Gaussian mixture from-scratch runs.
  • Table 1: final metrics at 1,000 steps.
  • Table 5: wide r_max sweep with default 100-step warmup, final metrics at 1,500 steps.
  • Figure 2: convergence curves over 1,000 steps, recorded every 10 steps.
  • Figure 6: full trajectories of the same sweep, plotted to ~1,500 steps. Only its step-1,000 point is comparable to Table 5.

Label issue

In Table 1, the row labeled r_max = 5.0, which exactly matches the r_max = 4 row in Table 5. The actual r_max = 5 row in Table 5 is 0.056687 / 0.000521. We have rerun the model and correted it in the revised manuscript.

Correction Note

  • The original submission mixed 1,000-step and 1,500-step results due to experiments run across different periods.
  • The revised version unifies Table 1, Table 5, Figure 2, and Figure 6 to 1,500 steps.
  • The experimental conclusions remain unchanged.

2. ImageNet continuation protocol and independent runs (W1/Q2)

Yes. For a given base generator, the ordinary Drifting continuation and the higher-order (HO) continuation trained exactly the same LoRA adapters. Both methods started from the same pretrained checkpoint and inserted LoRA modules into attn.qkv, attn.proj, mlp.w1, mlp.w2, and mlp.w3 in every DiT block, with rank r=16, scaling factor alpha=32, dropout 0, and bias="none". All original DiT weights and the frozen feature extractor remained fixed. The HO method only changes the training objective by adding the local covariance correction; it does not introduce additional trainable parameters or change the LoRA parameterization.

The ImageNet continuation settings were as follows: For additional training hyper-parameters, we used AdamW with learning rate 5e-4 betas (0.9, 0.95), gradient clipping at norm 2.0, and EMA decay 0.999 for 10k steps.

During training, the classifier-free guidance scale was sampled from the same range as in the original Drifting implementation. For evaluation, we kept the original Drifting evaluation protocol unchanged: the same ImageNet validation split, 50,000 generated samples, preprocessing, FID reference statistics, Inception-based FID/IS computation, and guidance-scale. We did not retune the guidance scale separately for the HO and ordinary continuation methods. The reported checkpoint-specific guidance scales were 1.0 for Pixel-B, 1.1 for Latent-B, matching the corresponding original Drifting models.

Previously, we trained a single model and performed five evaluation samples from that same checkpoint. We have now conducted two additional independent training runs for Drifting B/2 due to the time constraint. Each of the three checkpoints was evaluated five times (15 evaluations total), and the results are aggregated as mean ±\pm standard deviation: FID =1.61±0.04= 1.61 \pm 0.04, IS =288.1±0.8= 288.1 \pm 0.8. For the standard Drifting model trained under the same setting, we obtain FID = 1.75±0.031.75 \pm 0.03 and IS = 262.4±1.2262.4 \pm 1.2.

3. Computational overhead and local covariance (W1/Q3)

We thank the reviewer for highlighting the missing computational-cost analysis.

The overhead depends strongly on the representation space. In the ImageNet latent model, the covariance correction adds negligible compute because the latent dimensionality is low. In pixel space, a full covariance estimate is infeasible due to high dimensionality, memory blow-up, long compute time, and ill-conditioning; we therefore compute the covariance only over neighbors. In fact, in our subsequent latent Drifting experiments and in the MNIST experiments, the covariance matrix is also computed only over the neighborhood. Under this configuration, adding the Higher-Order correction to one full ImageNet pixel training step increases wall-clock time by 10.63% and memory by 3.19%. On MNIST, the additional time and memory are both below 1%. Because the newly introduced covariance correction is negligible in the low-dimensional latent space, matched-step optimization can be regarded as approximately matched training resource in that setting. For pixel Drifting, where the overhead is non-negligible, we additionally report a matched wall-clock comparison: at matched wall-clock, we compare the Higher-Order checkpoint at 90% of the baseline training time with the full-budget ordinary Drifting continuation; Higher-Order Drifting still achieves better FID.

AuthorsOpenReview ↗

Response part 2 to Reviewer mpY3

4. Corrected spectral interpretation (W2/Q4)

We thank the reviewer for this careful check. The reviewer is correct on both counts: the outer kernel average in Eq. (14) does contribute an additional smoothing factor, and the original proof effectively Taylor-expanded this factor away, turning the result into a low-frequency approximation that was then used beyond its validity range. We have revised Section 4.6, Proposition 3, and Appendix A.4 accordingly. In the process we also generalized the result from Gaussian kernels to normalized nonnegative radial kernels with finite covariance, and we added explicit Gaussian and radial Laplace instances in Appendix A.5 and full-frequency plots in Appendix A.6 (Figure 4).

How the outer kernel average is accounted for:

In the revised derivation the outer anchor average is retained exactly, not Taylor-expanded. In Fourier space it contributes an extra factor ∇ξΦ\nabla_{\xi}\Phi, where Φ=k^\Phi=\widehat{k} is the kernel Fourier transform; this is the source of the additional smoothing factor the reviewer anticipated.

For the Gaussian kernel k=Ghk=G_h, Appendix A.5.1 makes the two smoothing operations explicit. Writing fh=Gh∗ff_h=G_h*f, the first variation of the covariance discrepancy is

=-h^4\nabla^2 f_h(a)+O(\epsilon^2).$$ The covariance action in Eq. (14)/(29) then gives $$\dot{G}_f(x) = \frac{1}{\ell}\int G_h(x-a)D_f(a)(x-a)\,da = -\frac{h^2}{\ell}\nabla\cdot(G_h*D_f)(x) = \frac{h^6}{\ell}\nabla\Delta(G_h*G_h*f)(x),$$ where we used $(x-a)G_h(x-a)=-h^2\nabla_xG_h(x-a)$. In Fourier space, $$\widehat{\dot G_f}(\xi) =-\frac{i h^6}{\ell}\,\xi|\xi|^2\,e^{-h^2|\xi|^2}\widehat f(\xi).$$ The exponent is now $-h^2|\xi|^2$, i.e. two Gaussian factors, exactly as the reviewer expected. The full additive rate is therefore $$\lambda_{\mathrm{eff},G}(u) =h^2|\xi|^2e^{-h^2|\xi|^2/2} +\frac{\alpha_2h^6}{\ell}|\xi|^4e^{-h^2|\xi|^2} =\lambda_{1,G}(u)\,W_{2,G}(u),$$ with $$W_{2,G}(u)=1+c_2u^2e^{-u^2/2},\qquad c_2=\frac{\alpha_2h^2}{\ell}.$$ The original Eq. (41)/(24) contained only one Gaussian factor because the outer average was expanded as $\int k_h(u)u\,e^{i\xi u}du\approx -ih^2\xi$, i.e. $e^{-h^2|\xi|^2/2}\approx1$. That step is valid only for $h|\xi|\ll1$; we no longer use it beyond that regime. ### Frequency range and the high-frequency limit: The revised Proposition 3 is stated for a normalized, nonnegative radial kernel with finite covariance $\sigma_K^2I$, and the proof in Appendix A.4 retains the outer anchor average exactly. In normalized frequency $u=\sigma_K\|\xi\|_2$, with $\widehat{k}(\xi)=\psi_k(u)$, $$\lambda_{\mathrm{eff},k}(u)=\lambda_{1,k}(u)W_{2,k}(u),\qquad \lambda_{1,k}(u)=-u\psi_k'(u),$$ $$W_{2,k}(u)=1+c_2\bigl[\psi_k''(u)+\psi_k(u)\bigr],\qquad c_2=\frac{\alpha_2\sigma_K^2}{\ell}.$$ This expression is exact within the stated homogeneous population linearization and is not restricted to low frequencies. Proposition 3(ii) establishes $$W_{2,k}(0)=1,\qquad W_{2,k}(u)\to1\quad(u\to\infty),$$ using $\psi_k(0)=1$, $\psi_k'(0)=0$, $\psi_k''(0)=-1$, and the Riemann–Lebesgue lemma for $\phi''(r)=-\widehat{x_1^2k}(re_1)$. For the Gaussian instance this gives the closed form above; for the radial Laplace kernel $k_b$ it gives $$\psi_L(u)=(1+z)^{-(d+1)/2},\quad z=\frac{u^2}{d+1},$$ $$W_{2,L}(u)=1+c_2\frac{z(z+d+4)}{(1+z)^{(d+5)/2}},$$ and again $W_{2,L}(u)\to1$ as $u\to\infty$. Thus the correction strengthens finite-frequency damping, while the leading Gaussian exponential tail and the leading Laplace algebraic tail are preserved; the earlier inconsistency between the polynomial multiplier and the limit $\lambda_{\mathrm{eff}}/\lambda_{\mathrm{Gauss}}\to1$ is resolved by the additional Gaussian factor. We also removed the claim that the result holds for arbitrary order $M$. Proposition 3 is now restricted to the covariance action $M=2$, and Corollary 2 states the finite-horizon attenuation-threshold interpretation only for the Gaussian and radial Laplace instances. We hope this addresses the reviewer's concerns. The revised Section 4.6, Proposition 3, Appendix A.4, the new Gaussian and radial Laplace instances in Appendix A.5, and the full-frequency plots in Appendix A.6 (Figure 4) now make the scope and the additional smoothing factor explicit.

Dear Reviewer mpY3,

We sincerely thank you for the time and effort you have dedicated to reviewing our submission. As the discussion period will end on Nov 18 AOE, we would greatly appreciate any feedback you may have regarding our response. Please let us know if there are any remaining concerns or points that would benefit from further clarification.

Thank you for your time and consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.