acceptodds
Under review as a conference paper at ICLR 2027

Categorical Drifting Models

12 views

Abstract

We propose CatDrift, a categorical extension of Drifting Models for one-step generation in discrete spaces. We reinterpret drifting as a local alignment problem: each generated sample is corrected in the direction of a data-dependent field while remaining close to its current value. In the original formulation, this constraint is Euclidean, yielding additive updates in sample space. This choice is natural for continuous data but ill-suited to categorical variables, whose states lie on the simplex. Replacing the Euclidean penalty with a Bregman divergence turns the drifting update into a mirror-descent step, a standard object in optimization with a natural information-geometric interpretation. For negative entropy, the categorical update has a closed form: build a kernel-weighted positive-minus-negative field from real samples and the generator’s own outputs, add it to logits, project with softmax, and regress to the frozen target by cross-entropy. The result is a simple categorical drifting algorithm that obtains competitive-to-state-of-the-art one-step results on discrete image generation, molecular generation, and DNA sequence generation, without distilling diffusion models.

Primary area:
generative models
Keywords:
Drifting models, discrete data, bregman divergence, mirror descent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DTH6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work builds on extending drifting models to categorical or discrete features. Drifting models are designed specifically for one step generation. The key idea of this work is that (continuous) drifting models are designed for Euclidean space, and they are ill-suited for discrete categorical data over a probability simplex. The authors replace the Euclidean penalty terms in continuous drifting models with a Bregman divergence term, and from a mathematical view, the authors drive a closed-form algorithm for categorical drifting models. So the model constructs a local log-evidence field comparing real vs. generated neighbor distributions first, and then additively tilts the logits, and then trains the generator using cross-entropy against this frozen soft target. This proposed method is evaluated on discretized images, DNA sequences, SMILES strings, and good performance is shown.

Critical strengths
  • This is a novel or the frist extension of drifting models to categorical data as far as I know (also with the help from AI to do a literature review)-
  • The motivation is mathematically sound, the resulting closed-form softmax tilt is intuitive, simple to implement, and naturally preserves the simplex constraints for categorical data
Critical weaknesses
  • I do have some questions, I won't frame them as weaknesses for now since it is possible that I misunderstood. Please refer to questions.

The paper positions CatDrift as an alternative to methods that require "expensive training pipelines, pretrained teachers, or additional adaptation stages." However, the local improvement field heavily relies on an external, pre-trained Masked Autoencoder (MAE) to compute the feature map (ϕ\phi) for the locality kernel. This somewhat undercuts the "trained from scratch" claim, as the success of the local neighborhood lookup is bottlenecked by the quality of this separate representation-learning phase.

Questions for authors
  1. The authors claim CatDrift as a alternative to methods that require expensive training pipelines, pretrained teacher models, or additional adaptation stages. However, based on my understanding, the local improvement field in this work still relies on a pretrained MAE? Then how is this work an alternative?

  2. Why is the eval on CIFAR-10 on only 1,000 examples? If computational cost is not a constraint (which I don't think it would be since CIFAR-10 is pretty small), can you provide an updated result with a larger eval size? Also, I understand that this work is not to compete with standard continuous diffusion/flow models on image generation, but it would be greatly helpful to provide some results just a reference to contextualize the results in this work

  3. How critical is the pre-trained MAE for your proposed method? If you have a less well-trained MAE, would CatDriff fail?

1 comment

Thank you so much for the thoughtful and constructive review, specifically your questions about the role of pretrained representations and the CIFAR-10 evaluation. We found these suggestions genuinely helpful and took some additional time to investigate them through new experiments. The resulting findings have helped us better understand CatDrift and strengthen the paper, so we're grateful that you raised these points.

Q1. Dependence on a pretrained MAE

We agree that our original wording could be more precise. By teacher-free generation, we mean that CatDrift requires no pretrained generative teacher, distillation, or subsequent generative adaptation. Our original implementation does use a separately trained MAE for locality estimation, as is also common in continuous Drifting Models (Deng et al., 2026).

However, learned representations are not required by the categorical update itself.

To investigate this, we removed the feature extractor entirely for promoter DNA generation. Using a direct locality kernel with temperatures 0.01, 0.05, and 0.1, and further tuning the kernel parameters, we obtained:

  • MSE: 0.030 ± 0.002 (MAE) → 0.029 (no extractor).
  • 6-mer correlation: 0.941 ± 0.040 (MAE) → 0.905 (no extractor).

These results, evaluated over ten runs, remain in a comparable performance regime without a separate representation-learning stage, although the 6-mer correlation is somewhat lower.

We will clarify the role and computational cost of representation pretraining in the revised paper.

Q2. CIFAR-10 evaluation

We agree that evaluating FID on only 1,000 samples was limiting.

Our experiments use the categorical CIFAR-10 formulation considered by PairFlow (Park et al., 2026), where each RGB channel is modeled directly as a 256-way categorical variable, rather than generating continuous RGB values.

Following your suggestion, we substantially expanded our evaluation and investigated whether a separately trained MAE could be replaced by a general-purpose representation.

Using an off-the-shelf pretrained DINOv3 encoder (Siméoni et al., 2025), CatDrift now achieves FID 4.8 ± 0.85, over five independent training seeds, each evaluated using 50,000 argmax-decoded samples against CIFAR-10 training-set reference statistics. Generation requires only one network evaluation.

For comparison, the original MAE-based result was FID 58 ± 2.9 using 1,000 samples across ten seeds. Since both the feature representation and evaluation sample count changed, these values should not be interpreted as a controlled MAE-versus-DINOv3 comparison.

Nevertheless, the new result shows that CatDrift can achieve strong one-step categorical image generation using an existing, general-purpose encoder without separately training a task-specific MAE.

For context, continuous one-step generators on unconditional CIFAR-10 report FID 2.83 (iCT; Song & Dhariwal, 2024), 2.92 (MeanFlow; Geng et al., 2025), and 3.60 (ECT; Geng et al., 2024), using FID-50K with one network evaluation. These comparisons are contextual, as the continuous pixel-space formulations and training protocols differ from ours.

We will update the evaluation section to clarify these distinctions.

Q3. How critical is the pretrained MAE?

Our additional experiments suggest that representation quality matters, particularly for high-dimensional image generation, consistent with findings in continuous Drifting Models (Deng et al., 2026; Zhang et al., 2026).

However, the DNA experiments suggest that appropriate kernel tuning can make learned features unnecessary in some categorical domains. The DINOv3 experiment further shows that CatDrift is not tied to a specific MAE or task-specific representation-learning procedure.

These experiments do not establish insensitivity to MAE quality across all settings. They instead clarify the distinction between locality estimation, which depends on the kernel and representation, and the categorical update, whose geometry is defined by the Bregman potential.

We will incorporate these findings and make the corresponding limitations more explicit.

References:

Deng et al. (2026), Generative Modeling via Drifting;

Park et al. (2026), PairFlow;

Siméoni et al. (2025), DINOv3;

Zhang et al. (2026), Learning Discriminative Geometry for Drifting Models;

Geng et al. (2025), Mean Flows for One-step Generative Modeling.

Reviewer_bhAyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an extension of drifting models considering the geometry of data space, especially for categorical data. This paper reformulates the typical drifting update as a local optimization problem and generalizes it as mirror descent utilizing Bregman divergence. Using KL divergence as the Bregman divergence, a specific algorithm of the drifting model for categorical data is proposed. Experiments on discretized images, DNA sequences, and SMILES-based molecules demonstrate superior performance to baselines.

Critical strengths
  1. This paper is well-organized and easy to understand. The motivation and the technical background are clearly explained. The proposed method is explained adequately in most parts and is supported by theoretical foundations. In addition, this paper provides further discussion related to information-geometric topics such as Fisher-Rao geometry and Wasserstein geometry.
  2. The proposed method is evaluated on image, DNA, and molecule benchmarks, which can be considered sufficiently broad to verify its performance.
Critical weaknesses

While this issue is considered to be within an acceptable range, I would like to raise a weakness. Although this appears to represent a steady advance in drifting models, mirror descent is a well-known classical approach, and applying mirror descent to adapt it for categorical data might be a fairly natural idea, since there is a related paper that applied mirror descent to diffusion models for data on constrained sets [1].

[1] G.-H. Liu et al. "Mirror diffusion models for constrained and watermarked generation." Advances in Neural Information Processing Systems 36 (2023): 42898-42917.

Questions for authors

To construct the logit-space signal V, this paper proposes to use model particles hard-sampled from the model probabilities. However, using the model logits directly would also be a possible approach to construct the negative term for the positive-minus negative drifting field. Have you considered this design choice, and if so, what motivated the current choice?

2 comments

Thank you for the thoughtful and encouraging review. We particularly appreciate the connection you draw to Mirror Diffusion Models and your question about hard versus soft model particles. Both are interesting points that deserve a more careful discussion, and your suggestions have helped us better articulate the motivation and contribution of our approach.

W1. Novelty relative to Mirror Diffusion Models

We agree that mirror descent is a classical framework, and that Mirror Diffusion Models (Liu et al., 2023) provide relevant prior work on using mirror maps for constrained generation. We should have covered this more in depth and will do so in the next version of the paper.

The main distinction is where and how mirror geometry enters the generative process. Mirror Diffusion Models use a mirror map to transform the state space and then construct an iterative diffusion process in the resulting dual space. In contrast, CatDrift uses mirror descent only during training to construct improved targets for a generator that produces samples in a single step.

This distinction is particularly relevant for categorical data. With the negative-entropy potential, the mirror map involves log⁡r\log r, which is undefined at the boundary of the simplex, where one-hot categorical data lies. CatDrift avoids this difficulty by applying the mirror update to the generator's soft probability predictions, which lie in the interior of the simplex. Real categorical samples are never transformed by the mirror map; instead, they contribute to the local evidence field used to update those predictions.

Our contribution is the resulting categorical drifting algorithm, combining local data-model evidence, entropy-regularized updates, and frozen-target regression, without constructing a diffusion process or using a pretrained generative teacher.

Q1. Why use hard model particles rather than logits?

Our motivation is to estimate the distribution of actual generated categorical objects. Hard model particles preserve the attraction–repulsion interpretation of drifting: the negative term measures which categories are already represented by the model's generated population.

Using the query's own probabilities instead changes the update. Ignoring smoothing, setting p−=rθp^- = r_\theta yields

r+∝rθ1−η(p+)η.r^+ \propto r_\theta^{1-\eta}(p^+)^\eta.

This becomes a geometric interpolation toward the local data frequencies. At η=1\eta=1, we obtain r+=p+r^+=p^+, and the population-level repulsion disappears.

Averaging soft probabilities across other generated particles is a valid alternative that retains population-level repulsion and may reduce variance. Our hard-particle construction directly estimates the argmax-decoded distribution used in our experiments, while keeping neighborhood lookup consistent with the hard categorical inputs used to train the encoder.

We will clarify this distinction and discuss soft population estimates as an alternative.

Reference: Liu et al. (2023), Mirror Diffusion Models for Constrained and Watermarked Generation, NeurIPS.

Thank you for the thorough responses. I appreciate the clarifications the authors have provided. The responses regarding the pretrained MAE provided to the other reviewers were also helpful. I had a similar concern, though I did not raise it, as the use of MAE is common practice in the original drifting models. Based on the authors' responses, I am pleased to maintain my positive score.

Reviewer_ob78OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CatDrift, a categorical extension of Drifting Models for teacher-free and one-step generation models. In this work, the authors formulate the existing drifting model's approach as a Bregman-regularized local optimization problem. For this, applying the negative entropy to the derive mirror-descent. For generator training, they use KL-divergence with frozen soft targets, along with a categorical improvement field built from a kernel-weighted positive-minus-negative field on real samples. Experiments on MNIST and CIFAR-10 benchmarks show consistent FID gains and better results on ZINC250k.

Critical strengths

S1. The CatDrift shows large performance gains in the image generation and molecule generation tasks compared to the existing methods and the original Drifting Model under the controlled training budget.

S2. The CatDrift method is a teacher-free, one-step generation method that is efficient in terms of training and inference, and the mirror-descent formulation gives the logit update with the categorical extension.

Critical weaknesses

W1. The given method is relying on the MAE feature extractor. The local improvement field relies on the pretrained MAE features. So, it is important to perform an ablation study on the sensitivity of the feature extractor design. Also, the main image experiments are evaluated on small FID benchmarks such as MNIST and CIFAR-10. For generalization and robustness, extensions should be conducted.

W2. The proposed contributions of the methods are not sufficiently isolated. For CatDrift, many things change compared with the original drifting models, such as the geometry structure of the update, the improvement field, and the training objective. Although the controlled experiments demonstrate the effectiveness of the sum of method, it should be much clearer which parts of the method and its contributions are responsible for the which improvement.

Questions for authors

Q1. How many runs were conducted for this work? The main table includes standard deviation statistics, but it does not specify the number of runs.

2 comments

Thank you for the questions about representation dependence and the contributions of individual components. We spent additional time investigating these issues, and the resulting experiments have helped clarify the method.

W1. Feature extractor dependence and robustness

This is a fair point, and we have not made this clear in the manuscript. Indeed, our method, and drifting in general, does not need a feature extractor, but in practice benefits from one, especially in high-dimensional settings. E.g., for high-dimensional images, learned representations help construct meaningful neighborhoods, as also observed in continuous Drifting Models (Deng et al., 2026) and subsequent work on representation geometry (Zhang et al., 2026).

To illustrate the effect of feature extractions, we investigated two alternatives:

  • DNA: We removed the learned feature extractor entirely and used a direct locality kernel, combining temperatures 0.01, 0.05, and 0.1. After further tuning, we obtained MSE 0.029 and 6-mer correlation 0.905, compared to 0.030 ± 0.002 and 0.941 ± 0.040 with MAE features. This shows that in such low-dimensional settings a kernel is not needed per se.
  • CIFAR-10: With an off-the-shelf DINOv3 encoder (Siméoni et al., 2025), CatDrift achieves FID 4.8 ± 0.85, evaluated using 50,000 samples across five independent training seeds. The original MAE result was 58 ± 2.9 using 1,000 samples across ten seeds. These are different evaluation protocols as asked by another reviewer, but this shows that a ' standard ' pretrained model also suffices.

These results suggest that pretrained representations are particularly useful for high-dimensional data, while meaningful locality can also be constructed without learned features in some categorical domains.

We will clarify this distinction and the associated representation-learning costs.

W2. Isolating individual contributions

We agree that separating the effect of the update geometry from the rest of the training pipeline is important.

Our existing comparison shares the generator architecture, MAE features, categorical improvement field, neighborhood construction, and training/tuning budgets. The difference lies in how the improvement signal updates the generator and how the resulting frozen target is matched: Euclidean drifting uses an additive update with squared-error regression, whereas CatDrift uses an entropy-mirror update with KL regression.

We also evaluated whether simply constraining Euclidean drifting to the simplex would recover the improvement:

MethodCIFAR-10 FID ↓
Euclidean Drifting210 ± 10.4
Euclidean Drifting + softmax constraint~150
CatDrift58 ± 2.9

All three use the original MAE-based locality pipeline.

The softmax-constrained baseline improves on unconstrained drifting but remains substantially worse than CatDrift. This suggests that the improvement is not explained simply by enforcing valid categorical outputs.

We will clarify these implementation details and distinguish the roles of the update geometry and regression objective in the revised manuscript.

Q1. Number of runs

Thank you for catching this omission. The original image experiments were evaluated over 10 independent training seeds, with the reported ± values indicating standard deviation across runs.

The new DINOv3 experiment uses five independent training seeds, each evaluated with 50,000 generated samples, yielding FID 4.8 ± 0.85.

We will make the number of runs and evaluation protocol explicit for each experiment.

References:

Deng et al. (2026), Generative Modeling via Drifting;

Zhang et al. (2026), Learning Discriminative Geometry for Drifting Models;

Siméoni et al. (2025), DINOv3.

Thank you for the additional experiments and analysis. Some of my concerns are resolved, so I'll keep my positive overall assessment.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. Apologies for the delayed response; we wanted to spend time conducting additional experiments addressing the questions raised, rather than relying solely on conceptual arguments.

Across the reviews, three themes emerged: the dependence on pretrained representations, the role of the categorical update geometry, and the robustness of our evaluation.

Following these suggestions, we obtained several additional results:

  • Feature-free DNA generation: Removing the learned feature extractor and further tuning the kernel, CatDrift obtains MSE 0.029 and 6-mer correlation 0.905, compared to 0.030 ± 0.002 and 0.941 ± 0.040 with the original MAE representation.
  • Improved CIFAR-10 generation: Using an off-the-shelf DINOv3 encoder (Siméoni et al., 2025), CatDrift now achieves FID 4.8 ± 0.85, evaluated over five independent seeds with 50,000 samples per seed. Our original MAE-based result was 58 ± 2.9 using 1,000 samples.
  • Geometry and constraints: We additionally evaluated Euclidean drifting with softmax-constrained updates, obtaining approximately 150 FID, compared to 210 for unconstrained Euclidean drifting and 58 for CatDrift under the original evaluation setting.

We will clarify the corresponding experimental protocols, distinguish the effects of representation choice and evaluation sample count, and sharpen the presentation of the categorical update.

We will also clarify that teacher-free generation refers to the absence of a pretrained generative teacher or distillation, while learned feature representations may still be used for locality estimation.

These additional experiments have helped us considerably strengthen the paper, and we appreciate the reviewers' suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.