acceptodds
Withdrawn from ICLR 2027

Drifting through Annealed Clustering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Drifting models generate in a single forward pass, but learning coarse structure and fine detail together remains challenging. We turn soft clustering into a novel formulation of drifting that learns the data one scale at a time. Replacing the finite centroids of deterministic-annealing clustering with a neural generator yields a new likelihood objective: explain the data through a kernel-smoothed generated distribution. We characterize its optimum, showing that when exact matching is possible, the generator represents a sharpened version of the data while the kernel gives the remaining variation. Annealing the bandwidth progressively transfers this variation to the generator, turning broad clusters into increasingly detailed distributions. We derive a simple regression algorithm with expectation-maximization targets, in which generated samples compete to explain the data without explicit repulsion or manual cluster splitting, and use it to generate reliable samples on both synthetic multi-scale data (where drifting struggles) and realistic large datasets.

Primary area:
generative models
Keywords:
generative, drifting, anneal, clustering, expectation-maximization, one-shot, temperature

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mRMsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an alternative way of computing a drifting field for drifting models. The performance of the methods is tested on MNIST and CIFAR-10 datasets.

Critical strengths

None

Critical weaknesses

Novelty and presentation My major concern is that it is difficult to understand what exactly the paper proposes. Equations 11–13 suggest that the main difference compared to Deng et al. (2026) is an alternative way of estimating the vector field VV, which is given in Equations 12–13. It is not clear how this estimator relates to clustering. Moreover, Deng et al. (2026) mention that V−V^{-} is necessary as it improves the performance of the algorithm. The question is discussed briefly in lines 221–223, as well as in footnote 4. However, I find the discussion insufficient. Besides, the claim "... MSE loss both is much quicker and works well regardless on real, high-dimensional data" is not fully supported by the experimental results, which I talk about below. Finally, I find the theoretical justification presented in Section 3 difficult to follow. I detail my questions about the exposition of the theory in Inconsistencies.

Experiments I have the following concerns about the experiments in the paper:

  1. No comparisons with baselines, such as Deng et al. (2026), Gretton et al. (2026), and Han et al. (2026).
  2. Figure 3, which supposedly motivates the proposed algorithm, is a little questionable. Drifting models are capable of learning such a distribution if the temperature τ\tau is tuned. It should be noted that the proposed algorithm also has τ\tau as a hyperparameter that possibly requires tuning.
  3. The FID scores on CIFAR-10 are high, which suggests that the algorithm hasn't fully converged or requires substantial improvements.
  4. It has been shown (Deng et al., 2026; Han et al., 2026) that drifting models can be successfully scaled to high-resolution images. The question remains whether the proposed method can achieve the same results.

Inconsistencies

  1. Line 86 and Equation 1: VpV_p is used, but pp is never defined;
  2. The term "positive normalised kernel" in line 84 is not defined. I suspect it should have been "positive definite".
  3. Equation 11 and Equation 5 define optimisation objectives, yet it is unclear with respect to which variables these objectives are defined.
  4. It is not clear why qτ(y)q^\tau(y) is a valid density. Moreover, it is not clear how qτ(y)q^\tau(y) depends on τ\tau.
Questions for authors

Clarifications Please clarify what the main advantages of the proposed way of computing the drifting field are, as opposed to Deng et al. (2026) and Han et al. (2026). I venture to suggest presenting your method as an algorithm similar to Algorithm 2 in Deng et al. (2026) or Algorithm 1 in Han et al. (2026).

Suggestions

  1. For the low-dimensional experiments, I suggest comparing your method against the baselines discussed in Gretton et al. (2026).
  2. For experiments on images, I suggest comparing your method against Deng et al. (2026) and Han et al. (2026). I appreciated the attempts to do so in Tables 4 and 5; however, I believe the results can be substantially improved on both CIFAR-10 and MNIST.
1 comment
AuthorsOpenReview ↗

Response to reviewer mRMs

We thank the author for reading our paper and their questions and constructive criticisms. Though we will be withdrawing this submission we would nevertheless like to address their questions below.

Weaknesses:

My major concern is that it is difficult to understand what exactly the paper proposes. Equations 11–13 suggest that the main difference compared to Deng et al. (2026) is an alternative way of estimating the vector field , which is given in Equations 12–13. It is not clear how this estimator relates to clustering.

We agree that the connection between the clustering loss and the MSE proxy loss was not fully emphasised and made clear. Appendix B has now been added with a step-by-step derivation of the two losses, showing that they are equivalent up to a weighting by total responsibility per centroid point, and have equal zeroes in terms of the generated points xix_i.

Figure 3, which supposedly motivates the proposed algorithm, is a little questionable. Drifting models are capable of learning such a distribution if the temperature is tuned. It should be noted that the proposed algorithm also has τ\tau as a hyperparameter that possibly requires tuning. We thank the reviewer for bringing this to our attention, and work is ongoing on tuning our ordinary drifting approach. In our proposed algorithm, τ\tau needs only to anneal from a high value to a low value - we never tune it instantaneously. The shape of the anneal is simple and exponential, as discussed and illustrated on page 18 in Appendix E.

Clarifications:

Please clarify what the main advantages of the proposed way of computing the drifting field are, as opposed to Deng et al. (2026) and Han et al. (2026). I venture to suggest presenting your method as an algorithm similar to Algorithm 2 in Deng et al. (2026) or Algorithm 1 in Han et al. (2026).

The proposed way of computing the drifting field, while similar to Algorithm 2 in Deng et al. (2026), differs in several key respects.

Algorithm 2 in Deng et al. takes a geometric mean of two softmaxes of the kernel |x_i-y_j|/\tau, one softmax over i (the generated points) and the other over j (the data points). This is an effective but unexplained method to prevent collapse during learning. Our method is equivalent to a single softmax of the kernel over generated points for each data point. There is no row normalisation. Algorithm 2 in Deng et al. is not an exact gradient of a potential, whereas ours is the gradient of a free energy defined from classical clustering theory. Algorithm 2 in Deng et al., along with all related drifting algorithms, makes use of a repulsive term to ensure coverage of the data and sample diversity. For us, generated points compete to describe the data under a clustering objective, and spread out as a result of this competition.

For the low-dimensional experiments, I suggest comparing your method against the baselines discussed in Gretton et al. (2026). The low-dimensional datasets in Gretton et al. (2026) are not fully defined in the paper, but can be inferred from the visual results. Our method was tested against Deng’s drifting algorithm on a selection of similar targets, and the results may be seen in Appendix F. Both methods achieve excellent, comparable performance on these simple targets.

Reviewer_ZDGhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a new formulation of drifting generative models based on soft cluster assignments, combining ideas from annealed clustering and drifting models. The authors propose a clustering-inspired loss and compare it with an ensemble approach that combines multiple drifting fields and losses across a range of temperatures. Experiments are conducted on toy clustering datasets, MNIST, Fashion-MNIST, and CIFAR-10.

Critical strengths
  1. The paper presents an interesting combination of ideas from different areas. It provides a clear reinterpretation of drifting models through the lens of classical clustering with annealing. While the individual ideas are established, the connection between them is novel.
Critical weaknesses
  1. Insufficient experimental validation. The paper relies on simplified toy datasets, and the generated samples on MNIST, Fashion-MNIST, and CIFAR-10 are of poor visual quality. The results on MNIST are particularly concerning for a submission to a top ML conference. The baseline comparison is also limited to temperature ensembles of the original drifting method, without comparisons to other generative clustering approaches. Furthermore, the drifting baseline performs substantially worse than the results reported in the original paper, raising concerns about the implementation and the reliability of the comparisons. Overall, the experimental evidence is insufficient to support publication in its current form.
  2. Lack of clarity. The paper lacks a related work section and a clear description of the experimental setup, making it difficult to follow and assess. Abbreviations should be defined upon first use. The definitions of precision and recall, and how they are computed for this generation task, are unclear. In Figures 4 and 5, it is unclear which temperature values, τ\tau, correspond to which generated samples. Furthermore, the introduction promises a temperature schedule, but this is not revisited in the main text.
Questions for authors
  • The generator resembles that of a GAN: it takes sampled noise as input and generates new data. How does the proposed approach avoid mode collapse, and how well does it cover the modes of the data distribution?
  • How is the number of clusters, KK, selected? Is it a hyperparameter that requires tuning, and how does its choice affect the performance of the drifting model?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZDGh

We thank the reviewer for reading our paper, appreciating our ideas, and raising questions. Though we will be withdrawing this submission we would like to address their questions below.

Weaknesses:

Insufficient experimental validation. The paper relies on simplified toy datasets, and the generated samples on MNIST, Fashion-MNIST, and CIFAR-10 are of poor visual quality. The results on MNIST are particularly concerning for a submission to a top ML conference.

Since the paper was submitted, the experiment on MNIST was run and tuned with a UNet network architecture instead of the basic MLP used before, and the updated results, of a much higher quality, may be seen in Figure 4.

We note that FID where it is measured in the current draft is measured at 32px with 50k samples, with a standard Inception net, while the previous draft’s FID numbers were measured with a much lower number of samples and a nonstandard Inception net.

The baseline comparison is also limited to temperature ensembles of the original drifting method, without comparisons to other generative clustering approaches.

While it would be interesting to compare our method against existing generative approaches, the aim of presenting this annealed clustering reformulation of drifting is presenting an alternative, principled method to specifically ensure learning of different scales (a problem drifting is known to struggle with on some data).

Furthermore, the drifting baseline performs substantially worse than the results reported in the original paper, raising concerns about the implementation and the reliability of the comparisons. Overall, the experimental evidence is insufficient to support publication in its current form.

The experiments in this paper were carried out intentionally with basic, small-scale architecture across both ordinary drifting and our clustering method as a controlled test (hence less high-quality results than the original drifting paper). Work is underway on running our loss with vastly up-scaled architecture.

Lack of clarity. The paper lacks a related work section and a clear description of the experimental setup, making it difficult to follow and assess. Abbreviations should be defined upon first use. The definitions of precision and recall, and how they are computed for this generation task, are unclear. In Figures 4 and 5, it is unclear which temperature values, (\tau), correspond to which generated samples. Furthermore, the introduction promises a temperature schedule, but this is not revisited in the main text.

In Figures 4 and 5, all images marked ‘multi-tau drifting’ are results from drifting applied to an ensemble of multiple taus, with the ensembles detailed in Appendix E for each dataset. Generally, the ensemble used is {0.05crit,crit,3crit} (where crit is the first splitting tau in RGF clustering). This was found on all three of the image datasets to yield best or equal-best performance upon a coarse sweep of values. The temperature schedule is a simple exponential anneal, which is laid out in detail in Appendix E on page 18, and visualised in Figure 8.

Question 1:

The generator resembles that of a GAN: it takes sampled noise as input and generates new data. How does the proposed approach avoid mode collapse, and how well does it cover the modes of the data distribution?

The generator learning a map from noise to new data is shared with other modern generative models such as diffusion and flow matching models. Unlike GANs, there is no adversarial training principle. Indeed, the core strength of this annealed clustering method is that it ensures mode coverage of the entire dataset, because the algorithm starts by learning at large, coarse scales, and progressively learns final detail.

Quantitatively, Figure 3 shows an example of clustering covering all modes of the data distribution in progressively finer detail.

Question 2:

How is the number of clusters, K, selected? Is it a hyperparameter that requires tuning, and how does its choice affect the performance of the drifting model?

The novelty of this approach that there is a set number of ‘cluster points’ which never changes, and is simply as large as computationally feasible. This is therefore not a parameter that is tuned or selected. Unlike conventional clustering, in which a fixed number of points converge to a description of the data at a set scale, under the objective (7), this set of many clustering points first all gather at a single location, then split across 2 locations, then 3, and so forth, as seen in Figure 2 in snapshots.

Reviewer_ZDGhOpenReview ↗

Acknowledgment of Author Response

Thank you for taking the time to address my concerns and questions, despite having decided to withdraw the submission. I appreciate the additional clarifications.

Wishing you all the best with your future research!

Reviewer_UmmkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new approach to one-step generative modeling by connecting drifting models with deterministic-annealing clustering. The main idea is to replace discrete cluster centers with a distribution represented by a neural generator and train it using a kernel-smoothed likelihood objective. This leads to an EM-inspired update rule that does not require the explicit repulsive term used in standard drifting. By gradually decreasing the kernel bandwidth, the model learns the data distribution from coarse to fine scales. The authors also introduce a locally adaptive smoothing procedure to improve sample quality and diversity. The method is evaluated on synthetic checkerboard datasets, including a challenging multiscale setting, as well as MNIST, FashionMNIST, and CIFAR-10. The results show clear improvements over the tested multi-temperature drifting baseline on the synthetic task and generally better image-generation metrics, although the latter rely substantially on the additional smoothing step.

Critical strengths

(1) The main strength of the paper is the connection it establishes between deterministic-annealing clustering and drifting models. I find this perspective interesting, particularly because it leads to a different training objective that does not require an explicit repulsive term. The idea of progressively learning the data distribution from coarse to fine scales is well motivated and provides a useful alternative to combining multiple kernel bandwidths, as typically done in drifting models.

(2) The experimental results are also encouraging, especially on the multiscale synthetic benchmark, where the proposed method clearly outperforms the standard drifting baseline across the reported metrics. I find this experiment particularly relevant because it directly illustrates the problem the paper aims to address. The improvements on the image datasets, while more limited and dependent on additional smoothing, further suggest that the proposed approach has practical potential.

Critical weaknesses

(1) My first concern is the gap between the theoretical formulation and the training procedure actually used. The paper motivates the method through a kernel-smoothed likelihood objective, but the practical implementation relies on a stop-gradient MSE update with an additional normalization. It is not entirely clear to me how this affects the optimization dynamics or to what extent the theoretical properties of the original objective carry over to the implemented algorithm. Since this connection is central to the paper's contribution, I think it deserves a more careful justification.

(2) My second concern is that the experimental results do not fully isolate the benefits of the proposed learning objective. In particular, the improvements on image datasets seem to depend quite strongly on the locally adaptive smoothing procedure, as illustrated by the ablations in Table 5. I would have liked to see a comparison where the same smoothing is also applied to standard drifting, to better understand where the gains actually come from. Moreover, this additional step appears to require access to real data at inference time, which raises questions about the practical efficiency of the proposed one-step generator. Finally, the overall image-generation quality remains relatively modest, making it difficult to judge how well the approach would scale to more challenging settings.

Questions for authors

(1) Could the authors clarify the relationship between the proposed likelihood objective and the stop-gradient MSE update used in practice? In particular, I would like to understand whether the additional normalization in Equation 12 changes the stationary points of the optimization or mainly affects the update dynamics. This seems important given how central the theoretical motivation is to the paper.

(2) How much of the improvement over standard drifting can be attributed to the proposed learning objective rather than the locally adaptive smoothing? The results in Table 5 suggest that smoothing plays a substantial role in sample quality and diversity. I think a comparison where the same smoothing procedure is also applied to the drifting baseline would help clarify this point and could affect my assessment of the empirical contribution.

(3) Could the authors clarify the computational cost of locally adaptive smoothing at inference time? My understanding is that it requires access to real data samples to estimate a covariance matrix for each generated sample. Is this indeed the case, and if so, how does it affect the practical efficiency of the method compared with standard one-step generation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1

We thank the reviewer for their careful reading of our paper, appreciation of the perspectives we present, and detailed questions, which helped us to make our work clearer. Though we will be withdrawing this submission we would nevertheless like to address their questions below.

Weaknesses:

(1) We agree that the connection between the clustering loss and the MSE proxy loss was not fully emphasised and made clear. Appendix B has now been added with a step-by-step derivation of the two losses, showing that they are equivalent up to a weighting by total responsibility per centroid point, and have equal zeroes in terms of the generated points xix_i.

(2) We address the efficacy of the local smoothing procedure in the questions. The image-generation quality is indeed modest, with the caveat that these tests were all carried out with very rudimentary architecture (MLPs, a UNet for the updated MNIST results, and a basic DiT for CIFAR10). Experiments are underway to refine the method, to implement Phase II effectively, and expand the architecture.

We note that FID where it is measured in the current draft is measured at 32px with 50k samples, with a standard Inception net, while the previous draft’s FID numbers were measured with a much lower number of samples and a nonstandard Inception net.

Question 1:

We have now added an Appendix B in which the gradients of the two losses are worked out in explicit detail and compared next to each other. This reformulation into an MSE regression is familiar from other areas of Machine Learning; for example, VQ-VAE. The gradients have the same zeroes in terms of the generated points x_i–in other words, the two losses have identical stationary points. They differ in the update in parameter space, since the clustering objective loss weights each datapoint’s contribution to the loss by a factor N_i, corresponding to the total responsibility of the generated cluster point x_i. In practice, on low-dimensional targets, both losses display virtually identical performance.

Question 2:

We thank the reviewer for drawing attention to this important practical point. The locally adaptive smoothing exists because the optimum solution to the clustering-style loss, at any nonzero τ\tau, is not the data distribution. It is a sharpened version of the data distribution. The smoothing needs to be added to achieve the data distribution. On the other hand, the solution of the ordinary drifting objective is exactly the data distribution, so adding smoothing to it shifts it to a new distribution, which is not the data distribution. To prove the point empirically in any case, we add the same noise from the covariance matrix defined from the posterior distribution on the ordinary drifting samples. We use the value of tau in ordinary drifting to define this posterior distribution. The results from this ablation on MNIST may be seen in Appendix D.1, clearly worsening the quality of the samples (and though not reproduced in the appendix, the sliced W^2 distance increases too).

Question 3:

First, to compute the responsibilities, kernels are computed over the generated batch of samples and the points of the entire dataset. Note that the denominator in the responsibility is (the batch size times) the expectation of the smoothed sample distribution, which means we compute it by adding noise of width \tau to the generated samples as a batch and computing their average.

Once the responsibilities are computed, the easiest way to sample from a Gaussian with a given covariance matrix (in this case, from the responsibilities) is to multiply a vector sampled from a normal Gaussian (no computational cost) with the square root of the weights that define the covariance we need. Overall, the cost is O(NPD) (there are only N*P matrices to multiply).

It is a practical constraint of our method that access to the full dataset is needed at generation time after the forward pass of the network.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.