acceptodds
Under review as a conference paper at ICLR 2027

Learning the Geometry of Drifting Models: Persistent Representations for One-Step Generation

13 views

Abstract

Recently proposed Drifting Models shift iterative distribution refinement from inference to training, enabling effective one-step generation. However, their performance on complex image datasets depends strongly on the representation used to construct the drifting field: pixel-space drifting performs poorly, whereas pretrained feature spaces substantially improve sample quality for reasons that remain unclear. We trace this gap to the discriminative geometry of the representation, which determines sample weighting in kernel density estimation (KDE) and, consequently drift. We introduce persistent representation learning, which continuously learns a more discriminative representation geometry as the generator evolves across batches. We further establish a current-step gradient equivalence between the KDE ratio loss and drift regression loss under matched conditions, connecting density-ratio-based generator optimization to empirical drifting and motivating direct control of the drifting velocity. Across multiple datasets, our method learns effective discriminative representations directly from pixels and reduces FID by approximately -% over the original pixel-space Drifting Models, without pretrained encoders. Adapting pretrained representations and applying velocity clipping provide further gains.

Primary area:
generative models
Keywords:
Drifting Models, Generative Adversarial Networks, Wasserstein Gradient Flow, Lipschitz Regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u28dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is an interesting paper which studies the dynamics of latent space in drifting models. It is well-known that drifting works far better in latent space than in raw pixel space. This work studies the impact of training the encoder together with training the drifting directly. This is done by training a discriminative loss through Eq. (7), then alternating training of the generative and discriminative parts of the model. It's important to note that the drifting affects the dynamics of the encoder, as the discrimination is done between the generated samples and the true ones, so this is truly a novel framework. Some empirical studies are done showing the usefulness of the framework, in particular demonstrating that the method works considerably better than simply training directly in pixel space.

Critical strengths

This is a very reasonable thing to do, the results are broadly positive, and it is certainly important to obtain a detailed understanding of how the latent space interacts with drifting dynamics, which this work does.

Critical weaknesses

My primary concern is that it is not clear to me that the work here really demonstrates that this method is significantly better than using an existing pretrained encoder (which is standard in the drifting literature, and is thus the more natural baseline); it seems to me that really one should be comparing to that rather than comparing to direct pixel space comparisons. Some work towards this is done in the empirical section, but in my eyes this should be argued for slightly more forcefully and clearly than it currently is, which at the moment is really only done in the second experiment in Section 5.1. I would be happy to increase my rating if this could be strengthened.

Questions for authors
  1. Suggestion for further development: It seems that Section 5.1 is the primary comparison between the new method and drifting with a fixed pretrained encoder. Here indeed it is shown in Figure 2 that the new method is slightly better on FID, and also exhibits lower discriminative loss (and correspondingly higher discriminative accuracy). I think it would be good to strengthen this part. Here I have a few concrete suggestions: 1a. It would be good to show visually some samples from drifting with a frozen encoder and drifting with the new method (i.e. add the frozen encoder samples to Figure 3); if these look very different to the eye then this would be strong evidence for the superiority of the new method, perhaps more convincing than the marginal gain on FID on CIFAR10 in Figure 2. 1b. It is not entirely clear to me how to interpret that the KDE accuracy in the right panel of Figure 2 is highest for new method; I understand that the generator and discriminator are playing somewhat adversarial roles, but one might have expected that if this worked perfectly, then eventually the discriminator would fail to tell apart the generated and data images, which is not what we see. Could the authors spend a bit of time explaining this? I.e. is this expected from experience with GANs? 1c. In general any other ablations etc. that could demonstrate the superiority of this method over a frozen encoder would be appreciated. E.g. perhaps one could do experiments in section 5.2 where you train the encoder and the drifting mode separately, and also train them together using the method of the paper while keeping the total compute fixed, and demonstrate that the new method is better? This would conclusively show the benefits arising from the new paradigm.

  2. Clarification request: It's instructive to study a simple synthetic dataset as in Figure 1. Nevertheless, in Figure 1 (b), the "vanilla" raw space drifting baseline seems to really do very poorly to the extent that one might worry that it is actually mistuned -- could you elaborate on exactly how the reference field is constructed? (Appendix A.1 states how many samples are used but doesn't make quite clear what this reference field means) In particular, what should we interpret from the fact that the alignment gets smaller and smaller as the training proceeds? (Is the generator distribution getting farther and farther away from the target? Surely not -- but then is this alignment really the right measure?)

5 comments
AuthorsOpenReview ↗

Response to Reviewer u28d [1/3]

We sincerely thank Reviewer u28d for the thoughtful and constructive feedback. We are encouraged that the reviewer finds the study of representation dynamics interesting and the empirical results broadly positive. We particularly appreciate the concrete suggestions for strengthening the comparison with frozen pretrained encoders and clarifying the diagnostics. We address each point below.

W1 & Q1c The advantage over a frozen pretrained encoder, the standard setting in the drifting literature, needs stronger evidence, for example by comparing separate and joint encoder–generator training under a fixed total compute budget.

We agree that comparisons with frozen pretrained encoders are important and deserve greater prominence. Our compute-matched experiments show that persistent representation learning continues to improve generation quality after separate pretraining and frozen-encoder training plateau. Intuitively, a pretrained encoder is optimized for its pretraining objective rather than specifically for generation, so its fixed representation may not capture all the discrepancies that remain as the generator evolves. In contrast, persistent representation learning allows the feature geometry to adapt to the current generated distribution, potentially providing more informative drifting fields throughout training. Following the reviewer's suggestion, we conducted a compute-matched comparison of separate and joint encoder–generator training on CIFAR-10.

Setup. For separate training, we pretrained our residual CNN encoder on CIFAR-10 using MoCo-v2+, following the solo-learn recipe [1] adapted to our architecture with the flattened encoder output and the generator's input normalization. Pretraining for 1,000 epochs required 1.4 GPU-hours and achieved 80.8% linear-probe accuracy. We then froze the pretrained encoder and trained the generator using Drifting Models. For joint training, our method learns the same encoder architecture from identity initialization alongside the generator. Both settings use the same generator architecture, encoder architecture, batch size, kernel temperature, velocity clipping, and evaluation protocol, while differing in their encoder training strategies.

Results. The table reports the best FID over 50k generated samples achieved within each total compute budget on GH200 GPUs, including encoder pretraining for separate training.

Total compute (GPU-hours)24405672Best
Separate (pretrained, frozen)19.4218.4218.4218.4218.42
Joint (ours)21.5717.2114.9413.6912.81

Separate training initially performs better but plateaus at an FID of 18.42 after 19.5k generator steps, with no further improvement over the following 20k steps. Joint training continues to improve, overtakes separate training at approximately 35 GPU-hours, and ultimately reaches an FID of 12.81. This suggests that, beyond learning a discriminative representation through pretraining, persistently adapting the representation to the evolving generator provides additional benefits for drift-based generation.

Frozen ImageNet encoder. Section 5.1 provides complementary evidence with an ImageNet-pretrained MoCo-v2 encoder: adapting its final residual block improves FID from 8.90 to 6.23 under the same generator and training schedule (see Q1a).

We have included the compute-matched comparison and its pretraining details in the revised manuscript.

Q1a Add visual comparisons between frozen-encoder Drifting Models and the proposed method alongside the FID comparison.

We agree that a visual comparison with the frozen encoder would complement the FID comparison. Our new visual comparison confirms the improvement from adapting the pretrained representation, consistent with the reduction in FID from 8.90 to 6.23. We added uncurated samples of Drifting Models with the frozen MoCo-v2 encoder and with our adapted encoder to Appendix E of the revised version (Figure 13). Both runs follow the CIFAR-10 setting of Figure 2 and use the same generator and training schedule for 10k steps, and both generators sample from the same 64 latent noise vectors. The FID improvement represents a 30% relative reduction over the frozen encoder. The samples also show a qualitative improvement, with more objects appearing in clear and recognizable shapes.

AuthorsOpenReview ↗

Response to Reviewer u28d [2/3]

Q1b Explain why higher KDE discrimination accuracy accompanies better generation, given the expected indistinguishability at adversarial equilibrium.

We thank the reviewer for this insightful question about the interpretation of KDE discrimination accuracy. Higher KDE discrimination accuracy during training is consistent with better generation because it reflects the representation's ability to detect remaining discrepancies, rather than the indistinguishability expected only at equilibrium. The discriminator fails to tell apart generated and data images only at the global optimum, where the generated distribution equals the data distribution [2]. During training, the discriminator keeps identifying the remaining discrepancy between the two distributions, and this discrepancy provides the learning signal for the generator. BigGAN [3] reports that the discriminator separates real and generated training images almost perfectly throughout successful training. The authors regard this as in line with the role of the discriminator, which is to distill the training data and provide a useful learning signal for the generator.

The near-chance KDE accuracy of pixel-space drifting in Figure 2 reflects a weak ability of the pixel representation to detect the discrepancies that remain. Its FID stays poor, so the generated distribution remains far from the data distribution. Each KDE accuracy in Figure 2 is computed in the representation used by the corresponding method, so it depends on how well that representation detects the remaining discrepancy. In pixel space, the kernel cannot separate real and generated samples even though the two distributions differ substantially, and the accuracy stays close to chance.

Compared with pixel space, the persistent representation detects the discrepancies that remain for the current generator. This gives the drifting field a discriminative signal, and the FID improves substantially. We measure generation quality by FID and use KDE accuracy as a diagnostic of how well a representation detects the remaining discrepancy. We have added this explanation to the revised version.

AuthorsOpenReview ↗

Response to Reviewer u28d [3/3]

Q2 Clarify the synthetic reference field, the tuning of pixel-space Drifting Models, and the interpretation of decreasing field alignment during training.

We thank the reviewer for this clarification request. The reference field is constructed using the informative semantic coordinates, the raw-space baseline uses matched training settings, and decreasing alignment reflects inaccurate drift directions rather than increasing distance from the target distribution. We explain these points below.

Construction of the reference field. The reference field is the drift field that the same KDE estimator produces when its kernel uses only the two semantic coordinates. The eight Gaussian modes are defined in these coordinates, and the 30 nuisance coordinates carry no information about the target distribution. For each of the 1,024 generated evaluation samples xx, we compute

Vref(x)=∑iwi+(x) s(yi)−∑jwj−(x) s(x~j)V_{\mathrm{ref}}(x)=\sum_i w_i^{+}(x)\,s(y_i)-\sum_j w_j^{-}(x)\,s(\tilde x_j),

where s(⋅)s(\cdot) extracts the two semantic coordinates, yiy_i are 4,096 fresh samples from the target distribution, x~j\tilde x_j are 4,096 fresh samples from the current generator, and the weights are softmax normalized over −∥s(x)−s(⋅)∥2/τ-\|s(x)-s(\cdot)\|^2/\tau with the training temperature τ=0.5\tau=0.5. Both anchor sets are drawn independently of the evaluation samples. VrefV_{\mathrm{ref}} is therefore the drift field under an ideal representation that keeps the semantic coordinates and discards the nuisance coordinates, estimated from 4,096 real and 4,096 generated anchors. The estimated field uses the same formula on the 32-dimensional observations, with 1,024 real samples and the 1,024 generated samples themselves as anchors (each sample excluded from its own repulsion term). Its kernel distances are computed in the raw space for the raw-space baseline and in the learned representation for our method. We project this field onto the two semantic coordinates and report the cosine similarity with VrefV_{\mathrm{ref}} at each sample, averaged over the 1,024 samples.

Tuning of the raw-space baseline. The raw-space baseline and our method share the generator, optimizer, learning rate, batch size, number of steps, KDE temperature, and random seed, and differ only in the space where kernel distances are computed. The raw-space baseline computes them on the 32-dimensional observations, and our method computes them in the persistent representation, which is initialized as the identity map and updated during training.

Decreasing alignment and its interpretation. The decreasing alignment in raw space is caused by nuisance-dominated kernel distances, which prevent the estimated drift from accurately tracking the reference field. The alignment is the right measure for the question that Figure 1(b) addresses: whether the drift field computed in a representation points in the direction of the drift field under the ideal representation. Its decrease does not indicate deteriorating generation quality, as the W2W_2 distance in semantic space decreases throughout training. While the reference field's average magnitude decreases from 0.87 to 0.23, the raw-space drift's semantic projection remains large (1.33-1.70) due to nuisance dimensions, causing alignment to fall from 0.40 to 0.03. With the learned representation, the estimated drift magnitude decreases alongside the reference field (1.70 to 0.23), and alignment improves to 0.81. These results show that the learned representation produces drift directions that more accurately reflect the remaining discrepancy between the generated and target distributions. We have clarified this interpretation in the revised appendix.

References

[1] V. G. Turrisi da Costa, E. Fini, M. Nabi, N. Sebe, E. Ricci. solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR 2022.

[2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative Adversarial Nets. NeurIPS 2014.

[3] A. Brock, J. Donahue, K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR 2019.

Reviewer_wZt4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Drifting Models, which train a one-step generator by regressing it toward targets displaced by a kernel-based drift between real and generated samples. The authors argue that the poor performance of pixel-space drifting comes from the geometry in which the kernel is evaluated. They propose to learn this geometry with an encoder (initialized at the identity) trained alongside the generator to discriminate real from generated samples through a KDE log-density ratio. They also give a gradient identity relating the drift regression loss to a KDE ratio loss, and introduce clipping of the drift norm. On SVHN, CIFAR-10 and CIFAR-100, the method greatly improves over pixel-space drifting without pretrained encoders, and it also helps when adapting a pretrained MoCo-v2 encoder.

Critical strengths

The diagnosis is clear and useful: the quality of drifting is governed by the geometry in which the kernel is evaluated. The KDE classification accuracy (Fig. 2) is a simple, informative diagnostic of this. The toy example with nuisance dimensions illustrates the point well. Removing the dependence on pretrained encoders is a relevant practical goal for this class of methods. The improvements over pixel-space drifting are large and consistent, and the ablations separating the effects of the learned representation and of clipping are helpful. The paper is well written and easy to follow.

Critical weaknesses

A key weakness of Drifting Models is that their performance depends strongly on the choice of kernel, with no principled guideline. This paper displaces that problem rather than solving it. The bandwidth τ\tau is fixed. Since the kernel depends only on distances in feature space divided by τ\tau, the encoder effectively learns the bandwidth, and a position-dependent one at that. However, it learns it through a discrimination objective, which favors ever sharper kernels and hence nearest-neighbor attraction toward training samples. This is never examined. The representation-similarity diagnostic used (linear CKA) is scale-invariant and cannot detect it, and velocity clipping at a budget far below the typical drift norm discards the drift magnitude that would reveal it. No memorization check is provided, which matters given the very low SVHN FID.

The positioning with respect to prior work is also insufficient. Learning the feature map of a kernel adversarially to discriminate real from generated samples is the core of MMD-GANs [1], and KDD-GAN [2], cited here only for the KDE score, is the closest existing method and should be the primary baseline. More broadly, absorbing an iterative particle update into a one-step generator through a stop-gradient regression predates Drifting Models [3], and the connection the paper establishes between density-ratio-based generator updates and drifting is precisely the construction of [4, 5], where samples are displaced along the gradient of a critic-estimated density ratio and the generator is refit by regression. Proposition 1 follows directly from the classical mean-shift identity [6] and the definition of a stop-gradient regression, so presenting it as a main contribution overstates it. Experimentally, the main baseline (pixel-space drifting) is known to fail, the only other baseline is an off-the-shelf SNGAN, and results are small-scale and far from current one-step methods. The clipping study shows nearly flat results over four orders of magnitude of the budget, from single runs.

[1] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, B. Póczos. MMD GAN: Towards Deeper Understanding of Moment Matching Network. NeurIPS 2017.

[2] A. Lemkhenter, A. Bielski, A. E. Sari, P. Favaro. Generative Adversarial Learning via Kernel Density Discrimination. arXiv:2107.06197, 2021.

[3] Y. Feng, D. Wang, Q. Liu. Learning to Draw Samples with Amortized Stein Variational Gradient Descent. UAI 2017.

[4] R. Johnson, T. Zhang. Composite Functional Gradient Learning of Generative Adversarial Models. ICML 2018.

[5] Y. Gao, Y. Jiao, Y. Wang, Y. Wang, C. Yang, S. Zhang. Deep Generative Learning via Variational Gradient Flow. ICML 2019.

[6] K. Fukunaga, L. Hostetler. The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition. IEEE Trans. Inf. Theory, 1975.

Questions for authors

How does the effective scale of the learned representation (e.g., mean pairwise feature distance relative to τ\sqrt{\tau}) evolve during training, and how sensitive is the method to τ in the learned space?

Can you provide nearest-neighbor comparisons with the training set to rule out memorization?

Since almost all drifts are clipped at the best budget, is clipping effectively a per-sample normalization of the drift direction, and how does the method behave without the learned encoder but with such normalization?

How are the multiple feature groups normalized and weighted in the kernel?

How does the method compare with MMD-GAN-type approaches with learned kernels, and why does the persistent scalar critic perform so poorly?

How does the method compare empirically with KDD-GAN [2] and with critic-driven particle methods such as [4, 5]?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wZt4 [1/4]

We appreciate Reviewer wZt4 for your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns.

W1 & Q1 & Q2 & Q3 Learning the representation may displace the kernel-choice problem: the encoder may act as a learned bandwidth that sharpens the kernel, which calls for evidence on feature scale, sensitivity to τ\tau, memorization, and the role of clipping.

We appreciate the reviewer’s careful consideration of kernel sensitivity. Our results show that learning representation geometry addresses a limitation that bandwidth selection over a fixed representation cannot resolve, while the generator-side normalization prevents global feature scaling from sharpening its kernel weights. A radial kernel and its bandwidth determine how strongly each neighbor is weighted, while the representation determines which samples are neighbors. Learning this geometry therefore changes the drift in ways that selecting a radial kernel and tuning its bandwidth over a fixed representation cannot.

Kernel choice and representation geometry. Tuning the bandwidth of a fixed kernel leaves the raw-space drift poorly aligned, while the learned representation aligns it. To test this, we repeat the toy experiment of Section 5.1 with the same Gaussian kernel and vary only the bandwidth τ\tau of raw-space drifting. We report the drift alignment of Figure 1(b) at the end of training.

Raw spaceOurs
τ\tau0.050.10.20.51250.5
Alignment ↑0.0150.0230.0280.0290.0470.0350.1250.812

Across two orders of magnitude of τ\tau, the alignment of raw-space drifting stays below 0.13, while the learned representation reaches 0.81 at the original τ\tau. Appendix B.3 reports the corresponding result on CIFAR-10, where varying the bandwidth over three orders of magnitude leaves pixel-space drifting poor.

Feature scale and sensitivity to τ\tau. Global feature-scale growth does not sharpen the generator-side kernel, and removing the encoder objective’s scale dependence preserves generation quality. We agree with the reviewer that the encoder's KDE discrimination objective uses unnormalized features, and on CIFAR-10 the mean pairwise feature distance at the encoder output relative to τ\tau rises from 5.3 at initialization to 23.0 after 50k steps.

The generator is unaffected because the generator-side drift normalizes feature distances by their batch mean, which makes its kernel weights invariant to a global rescaling of the features. At every checkpoint of the 50k-step run, its attraction weights spread over at least 105 real images, so the drift does not reduce to nearest-neighbor attraction.

To confirm this, we added the same batch-scale normalization to the encoder loss and trained for the full 50k steps. The best FID is 13.23, compared with the original method’s best FID of 12.81. Removing the scale dependence of the encoder loss therefore maintains comparable generation quality. We have implemented the normalized encoder loss in the revised implementation.

For the sensitivity to τ\tau, we trained the CIFAR-10 configuration of Table 2 for 10k steps with τ\tau varied over two orders of magnitude.

τ\tau0.020.050.22
FID ↓22.7921.5736.1256.95

The bandwidth affects the result and remains a hyperparameter that requires tuning in our method. This is consistent with the temperature ablation of Drifting Models (Deng et al., 2026), where the FID also varies with τ\tau.

Memorization. Our quantitative and qualitative checks find no evidence of training-set copying under the evaluated criteria. We added a memorization analysis to Appendix F, following two established protocols.

  • Quantitative criterion. We adopt the memorization criterion (Gu et al., 2023): a generated sample counts as memorized when its ℓ2\ell_2 distance to the nearest training image is smaller than one third of its distance to the second nearest training image. For each dataset, we draw 10k samples from the checkpoint with the best FID and search the full training set in pixel space. None of the samples meets this criterion on CIFAR-10, CIFAR-100, and SVHN, which yields a memorization ratio of 0% on all three datasets.

  • Nearest-neighbor retrieval. Following the nearest-neighbor analysis of Drifting Models (Deng et al., 2026, App. B.4), we retrieve the ten nearest training images of the samples displayed in the paper using CLIP ViT-L/14 features (Figures 14 and 15). The retrieved images share the class and global appearance of each generated sample and differ in object shape, pose, font, and background.

AuthorsOpenReview ↗

Response to Reviewer wZt4 [2/4]

Clipping and drift normalization. Velocity clipping is distinct from per-sample normalization and is the more effective form of velocity control, with and without the learned encoder.

  • Clipping and per-sample normalization. On CIFAR-10 with the best threshold Vmax⁡=0.005V_{\max}=0.005, the fraction of clipped drifts varies with the training step. Over the first 10k steps, which correspond to Table 2, it ranges from 99.68% to 99.998% across steps logged every 100 iterations, and the median drift RMS before clipping ranges from 0.59 to 0.76. At Vmax⁡=0.05V_{\max}=0.05, the fraction over the same steps ranges from 99.22% to 99.93%. We compared velocity clipping with per-sample drift normalization under the configuration of Table 2 for 10k steps. Clipping reduces the best FID from 27.41 without clipping to 21.57, a 21.3% reduction, whereas normalization yields 28.42.

    SettingFID ↓
    Ours without clipping27.41
    Ours with clipping21.57
    Ours with normalization28.42
  • Normalization without a learned encoder. We also compared clipping (Vmax⁡=1.5V_{\max}=1.5) with per-sample drift normalization in standard drifting under identical training settings. Clipping reduces the best FID from 150.49 to 126.33, a 16.1% reduction, while normalization yields 137.93. Thus, clipping remains effective without a learned encoder and achieves the best FID among the evaluated variants.

    SettingFID ↓
    Standard drifting without clipping150.49
    Standard drifting with clipping126.33
    Standard drifting with normalization137.93

Illustration of velocity control. Velocity clipping and per-sample normalization differ in how they modify smaller drifts, even though both can reduce large drift magnitudes. Figure 11 in Appendix C visualizes this distinction using the toy example of Figure 1. Clipping bounds each sample's target displacement while preserving its original direction and leaving smaller drifts unchanged. In contrast, normalization rescales all nonzero drifts to a fixed magnitude, potentially lengthening those that were originally small. This direct control over per-sample drift magnitudes provides an additional degree of flexibility compared with KDD-GAN's direct KDE density-ratio optimization, allowing excessively large drifts to be bounded before their contributions are aggregated into the generator update.

We have incorporated these additional analyses and visualizations into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer wZt4 [3/4]

W2 Prior learned-kernel and particle-regression methods require clearer positioning and direct comparisons; the theorem novelty, baseline strength, experimental scale, and single-run clipping study need stronger support.

We thank the reviewer for the thoughtful comments on related work, the role of Proposition 1, and the experimental evaluation. Our contribution is persistent representation learning for Drifting Models, rather than adversarial kernel learning, particle regression, or the underlying KDE score identity; the accompanying experiments substantiate its benefit in the evaluated settings. We cite and discuss the related works and clarify these distinctions.

Relation to MMD-GAN and KDD-GAN. We recognize the connection to MMD-GAN and KDD-GAN, particularly their use of learned representations and kernel-based distribution comparisons. Our contribution, however, is not the introduction of adversarial kernel learning. We investigate a specific limitation of Drifting Models: the strong dependence of their empirical drifting field on the representation geometry. Our method learns this geometry persistently across batches while reconstructing the KDE density-ratio estimate and drift from the current real and generated samples at each step. KDD-GAN is therefore an important related baseline, and we have strengthened its discussion and clarified the distinction between the methods.

Relation to earlier particle-based methods. We acknowledge the connection to earlier approaches that incorporate particle updates into generator training through regression. Drifting Models provide a useful framework in which the update field is explicitly constructed through nonparametric interactions between current real and generated samples, making its dependence on representation geometry directly accessible. Our work builds on this framework to investigate and address its representation bottleneck. The general particle-regression mechanism originates in prior work.

Role of Proposition 1. The proposition formalizes a current-step generator-gradient identity between KDE density-ratio optimization and drift regression under matched conditions. Its significance for our method lies in the underlying relation V^φ(u)=τ2∇uℓφ(u)\widehat V_\varphi(u)=\frac{\tau}{2}\nabla_u\ell_\varphi(u), which connects the spatial gradient of the density-ratio potential to the direction and magnitude of the drifting field. This connection motivates direct control of per-sample velocity, analogous to Lipschitz control in GANs. Unlike spectral normalization (Miyato et al., 2018), which constrains the critic and affects generator updates indirectly, velocity clipping directly bounds each sample's drift magnitude while preserving its direction. We have clarified that the contribution is this connection and its role in motivating algorithm design. The constituent mean-shift identity is established in prior work.

Experimental scope. Drifting Models (Deng et al., 2026) are themselves a competitive contemporary one-step generative framework, and our work addresses their dependence on pretrained representations. Our controlled comparisons and three-seed benchmarks substantiate the representation-learning improvement, while the clipping ablation supports velocity control within the evaluated setting. Across SVHN, CIFAR-10, and CIFAR-100, our method consistently outperforms pixel-space drifting, with mean FID improvements exceeding 54 points on all three datasets. We also provide direct comparisons with KDD-GAN and learned-kernel MMD-GANs in our responses to Q5 and Q6. Regarding velocity clipping, Table 2 shows improvements across all evaluated thresholds, reducing CIFAR-10 FID from 27.41 without clipping to 21.57 at the best threshold. Together with the three-seed benchmark results obtained using velocity clipping, these findings support its effectiveness as part of our training framework.

Q4 How are multiple feature groups normalized and weighted in the kernel?

Each feature group is normalized independently and weighted equally, following Drifting Models (Deng et al., 2026) and DriftXpress (Falahati et al., 2026). For group jj with CjC_j channels, features are divided by the stop-gradient batch distance scale SjS_j, computed as the mean distance from generated samples to real and generated samples, divided by Cj\sqrt{C_j}. All groups use the kernel exp⁡(−∥⋅∥/(τCj))\exp(-\|\cdot\|/(\tau\sqrt{C_j})) with a shared τ=0.05\tau=0.05. Each group's drift is normalized to unit RMS before velocity clipping, and the group losses are summed with equal weights. No group-specific hyperparameters are tuned. We have added these details to the revised appendix.

Reviewer_Sv1wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes persistent representation learning for drifting generative models, jointly adapting the feature geometry and generator without pretrained encoders. It establishes a local gradient equivalence between KDE density-ratio optimization and drift regression, and evaluates representation learning and velocity clipping on three image datasets.

Critical strengths

The method significantly improves pixel-space drifting across SVHN, CIFAR-10, and CIFAR-100. On CIFAR-10, FID decreases from 124.94 to 12.81, also outperforming the reported SNGAN baseline. These results support the feasibility of learning effective drifting representations without external pretraining. Although, I believe an FID of 12.81 on CIFAR-10 remains far behind established SOTA one-step GAN approaches. Without comparisons to stronger generative baselines or closely related KDE-based methods, the practical significance of the proposed framework remains insufficiently demonstrated.

The experiments distinguish persistent representation learning from persistent scalar critics and velocity clipping. The controlled geometric analysis and equal-training-time comparison further support the central hypothesis that representation quality affects drifting optimization.

Critical weaknesses

KDD-GAN (Lemkhenter et al., 2021) already learns discriminative feature representations using KDE-based density-ratio objectives. The proposed representation-learning mechanism is conceptually similar, while the gradient equivalence largely follows from established KDE score identities and the chain rule.

A direct comparison with KDD-GAN under matched conditions is essential to establish the distinct contribution of the proposed training framework.

The strongest comparisons rely on poorly performing pixel-space drifting and an older SNGAN baseline. Modern generative baselines and stronger pretrained drifting configurations are not directly compared under matched resources.

Furthermore, the main benchmark lacks multi-seed uncertainty estimates, and the gradient-equivalence theorem applies only without velocity clipping, whereas clipping contributes to the final method's performance.

Need stronger results on more complex data other than CIFAR100.

Questions for authors

How does the method compare with KDD-GAN under matched architectures, training budgets, and evaluation protocols?

Can the authors report multi-seed variability and stronger contemporary generative baselines?

What algorithmic benefit remains beyond direct KDE density-ratio generator optimization, particularly when velocity clipping is enabled?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Sv1w [1/3]

We thank Reviewer Sv1w for the thoughtful feedback and for considering our work. Below, we provide point-by-point responses to your questions and concerns.

W1 The representation objective overlaps with KDD-GAN, and the gradient equivalence appears to follow from established KDE score identities and the chain rule.

We appreciate the reviewer’s careful consideration of prior KDE-based methods. Our contribution is to learn persistent representation geometry within the Drifting Models framework, whose explicit drift construction and stop-gradient generator update differ from KDD-GAN’s direct density-ratio optimization.

Relation to KDD-GAN. We recognize the conceptual connection to KDD-GAN, particularly its use of discriminative representation learning with KDE-based objectives. However, the two methods differ in how they train the generator: KDD-GAN directly optimizes a KDE density-ratio loss, whereas Drifting Models explicitly construct a sample-dependent drifting field and update the generator through stop-gradient regression, allowing direct control of the drifting velocity. Our work is motivated by a central limitation of Drifting Models: the quality of the drifting field depends strongly on the underlying representation, making effective generation difficult without pretrained feature encoders. We address this limitation through persistent representation learning, allowing the feature geometry to evolve alongside the generator while reconstructing the KDE density ratio and drifting field from the current samples at each step. This enables effective generation directly from pixels without relying on pretrained encoders.

Gradient equivalence. The gradient equivalence builds on established KDE score identities and the chain rule under the stated assumptions. Its significance lies in making explicit how the representation-dependent KDE density ratio determines the generator gradient in drift regression. In particular, the underlying identity V^φ(u)=τ2∇uℓφ(u)\widehat V_\varphi(u)=\frac{\tau}{2}\nabla_u\ell_\varphi(u) connects the direction and magnitude of the drifting field to the spatial gradient of the density-ratio potential. This connection further motivates direct per-sample velocity control, whose empirical benefits are demonstrated in Table 2. As established in Sections 4.1–4.2 and Appendix I, this connection provides the theoretical basis for understanding representation-dependent generator supervision and motivating velocity control.

W2 & Q1 A direct comparison with KDD-GAN under matched architectures, training budgets, and evaluation protocols is needed to establish the distinct contribution.

We appreciate the reviewer's emphasis on the importance of comparing our method with KDD-GAN. Our matched CIFAR-10 comparison shows a substantial improvement over KDD-GAN, with FID decreasing from 24.34 to 12.81.

Matched setting. We implemented KDD-GAN following the original paper (vMF kernel with τ=0.05\tau=0.05, hinge loss) and trained it with the same U-Net generator, the same residual CNN feature network, the same batch size of 6,000, and the same 50k training steps as our method. Both methods are evaluated with the same FID protocol on 50k generated images.

Hyperparameter search for KDD-GAN. Since the networks and batch size differ from the original KDD-GAN setting, we searched its hyperparameters in the matched setting. Starting from the original configuration (learning rate 2×10−42\times10^{-4}, 4 discriminator steps per generator step), we varied the learning rate from 10−610^{-6} to 8×10−48\times10^{-4} over nine values and the number of discriminator steps from 1 to 5, which gives 13 configurations. Each configuration was trained for 10k steps and ranked by its best FID within these steps. The best configuration (learning rate 5×10−55\times10^{-5}, 4 discriminator steps) was then trained for the full 50k steps.

MethodBatch sizeStepsFID ↓
KDD-GAN6,00050k24.34
Ours6,00050k12.81

In the matched setting, our method reduces FID from 24.34 to 12.81, representing a 47.4% relative reduction. KDD-GAN optimizes the generator adversarially through the KDE density-ratio loss, while our method regresses the generator onto drift targets with velocity clipping. The matched comparison demonstrates the advantage of our overall training framework under the evaluated settings. We have added these results to the revised version.

AuthorsOpenReview ↗

Response to Reviewer Sv1w [2/3]

W3 Comparisons with modern generative baselines and stronger pretrained Drifting Models under matched resources are missing.

We appreciate the reviewer's concern about the strength of the baselines. Drifting Models (Deng et al., 2026) are themselves a competitive contemporary one-step generative framework, achieving state-of-the-art results under the evaluation settings reported in their paper. The original work already compares against contemporary generative methods, including GigaGAN, StyleGAN-XL, MeanFlow, AdvFlow, and SiT. Reproducing these methods under fully matched architectures and training budgets would require substantial additional computational resources. Our work addresses a specific limitation of Drifting Models: their reliance on pretrained representations to construct effective drifting fields. We therefore prioritize controlled comparisons within the Drifting Models framework to directly evaluate the contribution of persistent representation learning.

Pretrained Drifting Models under matched resources. We compared our method with pretrained-encoder Drifting Models in two settings.

  • In-domain pretrained encoder at a fixed compute budget. We pretrained the same encoder with MoCo-v2+ on CIFAR-10, froze it, and trained the generator with Drifting Models, counting the pretraining in the total compute. This baseline saturates at an FID of 18.42, while our method overtakes it at about 35 GPU-hours and reaches 12.81.

  • Frozen ImageNet encoder. With the frozen ImageNet MoCo-v2 encoder of Section 5.1, adapting the final residual block with our method reduces FID from 8.90 to 6.23 under the same generator and training schedule, representing a 30.0% relative reduction.

We have presented these comparisons together in the revised version.

W4 The main benchmark lacks multi-seed uncertainty, and the gradient equivalence excludes velocity clipping used in the final method.

We thank the reviewer for raising these important points about uncertainty estimates and the scope of the gradient equivalence. Multi-seed results confirm the improvements over the evaluated baselines, while the gradient equivalence motivates velocity clipping without claiming equivalence for the clipped update.

Multi-seed uncertainty. We trained our method and standard drifting three times on each benchmark with the configuration of Table 1 and report the mean and standard deviation of the best FID over training. The SNGAN reference trained under its standard protocol is included for comparison. Our method obtains 3.09 ± 0.23 on SVHN, 15.73 ± 3.46 on CIFAR-10, and 22.62 ± 1.74 on CIFAR-100. The improvement over standard drifting exceeds 54 FID points on every dataset, and our method also improves on SNGAN across all three benchmarks. We have added these results to the main benchmark table of the revised version.

MethodSVHNCIFAR-10CIFAR-100
Standard drifting57.49 ± 3.78119.62 ± 6.83121.10 ± 11.45
Ours3.09 ± 0.2315.73 ± 3.4622.62 ± 1.74
SNGAN12.42 ± 0.7429.42 ± 0.9529.69 ± 0.59

Scope of the gradient equivalence. Proposition 1 serves as the theoretical basis for velocity clipping. Under its conditions, the drift regression gradient coincides with the gradient of the KDE density-ratio loss, and the underlying identity V^φ(u)=τ2∇uℓφ(u)\widehat V_\varphi(u)=\frac{\tau}{2}\nabla_u\ell_\varphi(u) shows that the drift of each generated sample is determined by the spatial gradient of the density-ratio potential. This relation identifies the drift magnitude as a quantity that governs the generator update, and velocity clipping controls this quantity directly. The proposition therefore provides the motivation for clipping, and Table 2 provides its empirical support, as detailed in our response to Q3.

AuthorsOpenReview ↗

Response to Reviewer Sv1w [3/3]

W5 Evaluation on more complex datasets beyond CIFAR-100 is needed.

We thank the reviewer for this suggestion. Our additional Tiny ImageNet results show that the benefit of persistent representation learning extends beyond CIFAR-100: our method achieves an FID of 75.58, compared with 176.84 for standard drifting, on a dataset with 200 classes at 64×6464\times64 resolution.

Setting. Both methods use the same U-Net generator, a batch size of 6,000, and 10k training steps without pretrained encoders, and FID is computed on 50k generated images. Standard drifting computes the drift in pixel space, and our method computes it in the persistent representation with velocity clipping (Vmax⁡=0.5V_{\max}=0.5).

Result. Our method reduces FID from 176.84 to 75.58, representing a 57.3% relative reduction. It also improves on the SNGAN reference trained on the same dataset (85.79). The benefit of persistent representation learning over pixel-space drifting therefore carries over to a dataset with more classes and a higher resolution than CIFAR-100. While this result is not yet state of the art, our contribution is to address the representation bottleneck in Drifting Models rather than to establish a new generative benchmark. We have added these results to the revised version.

Q2 Can multi-seed variability and stronger contemporary generative baselines be reported?

We appreciate the reviewer's request for stronger evidence. We provide both multi-seed uncertainty estimates and comparisons with pretrained-encoder Drifting Models, a competitive contemporary one-step generative framework. The multi-seed results are reported in W4, while W3 details the resource-matched pretrained-encoder comparisons: our method reaches an FID of 12.81 versus 18.42 for a frozen in-domain pretrained encoder, and adapting a frozen ImageNet MoCo-v2 encoder improves FID from 8.90 to 6.23.

Q3 What algorithmic benefit remains beyond direct KDE density-ratio generator optimization when velocity clipping is enabled?

We appreciate the question about the algorithmic benefit of explicit drift regression. Its benefit is direct per-sample control of the generator’s regression targets through velocity clipping, even though the unclipped gradient is related to KDE density-ratio optimization. Proposition 1 makes this connection explicit through the identity V^φ(u)=τ2∇uℓφ(u)\widehat V_\varphi(u)=\frac{\tau}{2}\nabla_u\ell_\varphi(u). The direction of the drift follows the spatial gradient of the density-ratio potential, and its magnitude is proportional to the norm of this gradient. A potential with large local gradients therefore produces large displacements of the regression targets, which motivates bounding the drift of each sample, in analogy with Lipschitz control of GAN critics.

Velocity clipping implements this control as a radial projection of each drift onto a ball of radius Vmax⁡V_{\max}. The projection preserves the drift direction given by the density-ratio potential and bounds the feature-space displacement of every generated sample at every feature location. Spectral normalization of the encoder constrains the representation and affects the potential only indirectly. The theoretical connection motivates this design, and we evaluate its benefit empirically.

Table 2 supports this design. Velocity clipping improves FID over the unconstrained model (27.41) for every evaluated budget, reaching 21.57 at Vmax⁡=0.005V_{\max}=0.005, whereas spectral normalization of the encoder degrades FID to 31.82. We have added this discussion to the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.