acceptodds
Under review as a conference paper at ICLR 2027

Acceleration of Diffusion Language Model through Discrete Average Generator

Abstract

Discrete diffusion models and flow matching have emerged as powerful frameworks for generative modeling over discrete state spaces, yet efficient few-step generation remains a fundamental challenge. In this work, we introduce the Discrete Average Generator, a principled extension of MeanFlow to Continuous-Time Markov Chains (CTMCs). Analogously to how MeanFlow defines an average velocity field over a time interval in continuous spaces, we define an average generator as the normalized increment of the transition kernel over a time interval. We show that this average generator satisfies a self-consistency identity, which provides the foundation for our training objective. We further develop training strategies that align with the standard training paradigm of diffusion language models while keeping the resulting objective tractable. When projected onto per-coordinate marginals, the self-consistency identity admits a closed-form expression, enabling efficient training and inference. In Potts model simulations, our objective reduces the total variation distance of the -step sampler by up to 67%. On OpenWebText, our method achieves the lowest generative perplexity among the evaluated methods for 8 to 64 sampling steps while enabling a acceleration, and achieves comparable performance to existing methods on ImageNet.

Primary area:
generative models
Keywords:
Diffusion Language Model, Continuous-Time Markov Chains

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ghi4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the Discrete Average Generator, an extension of MeanFlow to CTMC for few-step discrete diffusion generation. The authors define an average generator on an interval basis, derive the self-consistency identity, and develop the training objective. Validation includes Potts model simulation, text generation on OpenWebText, and image generation on ImageNet. The experimental results show lower distributional error, improved few-step generation perplexity of text, and satisfactory image generation performance.

Critical strengths

The most substantial contribution of this study is extending MeanFlow to the discrete state space, which gives a mathematically grounded method. The proposed average generator contains a training objective made tractable via per-coordinate projection. Experimental results also demonstrate its effectiveness in both text and image generation tasks.

Critical weaknesses

The newly added contribution in terms of methodology is not sufficiently evaluated. The Kolmogorov backward equation directly gives the average generator identity, so the main technical contribution seems to be the tractable training objective and the corresponding estimator. However, the reported experiments lack sufficient ablation results to show whether the performance gains come from the proposed method or simply from further training.

Another concern is that the manuscript in its current form is somewhat verbose and could be better organized. For example, the Preliminaries section is much too long, and some mathematical details could be moved to the appendix.

Questions for authors

Have the authors compared the proposed training objective with simply continuing training with the original loss, starting from the same weights and using the same training budget? How much improvement does the newly proposed objective provide?

2 comments

We thank the reviewer for the thoughtful comments and suggestions. We appreciate the time you spent on the paper and your recognition of our contributions and strengths, including the novelty of the discrete average generator and the performance of the proposed method. Below we address the concerns and comments that you have provided.

W1. The newly added contribution in terms of methodology is not sufficiently evaluated. The Kolmogorov backward equation directly gives the average generator identity, so the main technical contribution seems to be the tractable training objective and the corresponding estimator. However, the reported experiments lack sufficient ablation results to show whether the performance gains come from the proposed method or simply from further training.

Q1. Have the authors compared the proposed training objective with simply continuing training with the original loss, starting from the same weights and using the same training budget? How much improvement does the newly proposed objective provide?

Thank you for your suggestion. We provide a detailed comparison on various backbones.

Potts

K = 4K = 8K = 16
Pretrained base0.1980.0570.029
+ baseline objective0.199 ± 0.0000.057 ± 0.0000.029 ± 0.000
+ ours0.117 ± 0.0030.032 ± 0.0020.022 ± 0.002

OpenWebText

K = 4K = 16K = 64
none (released Di4C+ReDi¹ checkpoint)47.524.820.3
continued ReDi49.024.420.1
Ours56.122.917.0

ImageNet

FIDIS
ReDi²7.86240
+ continued ReDi8.94 ± 0.02268.3
+ ours7.82 ± 0.06252.3

Our objective achieves better performance than continued discrete flow matching and ReDi, which verifies the effectiveness of the proposed method.

We further conduct experiments on two setups: text8 (a character-based discrete flow matching model trained from scratch) and DiffusionGemma (a pre-trained 26B parameters discrete flow matching model)

text8, bits per character (BPC) of the samples ↓K = 4K = 8K = 16K = 32K = 64valid words, K = 64
discrete flow matching4.583 ± 0.0053.442 ± 0.0082.796 ± 0.0102.550 ± 0.0062.428 ± 0.00388.3%
our objective4.525 ± 0.0073.216 ± 0.0072.551 ± 0.0032.341 ± 0.0052.257 ± 0.00490.8%

We train from scratch with the same number of steps with our objective with the vanilla discrete flow matching objective, our objective lowers the bits per character (BPC) of the samples and raises the share of valid words, which verifies the effectiveness of the proposed method.

GSM8K, accuracy (%)K = 16K = 32
DiffusionGemma96.296.6
+ standard fine-tuning96.0 ± 0.296.4 ± 0.1
+ fine-tuning with our objective96.4 ± 0.396.6 ± 0.1

We fine-tuned LoRA adapters on GSM8K with the standard objective and with ours (3 seeds). We achieve better accuracy on math reasoning tasks, which demonstrates the effectiveness of the proposed method.

W2. Another concern is that the manuscript in its current form is somewhat verbose and could be better organized. For example, the Preliminaries section is much too long, and some mathematical details could be moved to the appendix.

Thank you for your suggestion. We moved the detailed discussion of the discrete flow matching and the MeanFlow parameterizations to the Appendix in the revised version.

As the discussion period is coming to a close, please feel free to let us know if you have any further questions or concerns regarding our responses. We would be more than happy to clarify any points or discuss them further. Thank you very much for your time.

Reviewer_kpLvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Discrete Average Generator (DAG) for accelerating discrete diffusion models. The main idea is to define an average generator over a finite time interval using the CTMC transition kernel, derive a self-consistency identity like MeanFlow, and project this identity onto per-coordinate marginals to obtain a tractable training objective. The method uses a pretrained denoiser to estimate instantaneous rates and a stochastic estimator for the transport term, which requiresan additional network evaluation during training. The authors report up to 67% reduction in sampler TV distance in the Potts experiment, improved generative perplexity with 8–64 sampling steps on OpenWebText, and comparable FID with slightly improved IS on ImageNet.

Critical strengths

[S1] Interesting formulation with a useful theoretical insight. The idea of extending MeanFlow to CTMCs through an average generator is natural and potentially useful. I particularly liked the per-coordinate projection argument (S3.3), which makes the otherwise intractable generator identity computationally feasible. The deterministic identity and the uniqueness result for its projected solution appear mathematically sound, although I didn't check rigorously. This gives a reasonably clean theoretical motivation for learning finite-step transition marginals without explicitly simulating multi-step teacher trajectories.

[S2] Promising empirical results, especially for few-step text generation. The Potts experiment is a nice controlled testbed since the sampler distribution and TV distance can be computed exactly. OWT results are also encouraging, showing improvement in generative PPL with similar reported unigram entropy (although there's recent paper by Amin et al. ("Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported") showing that this may not translate to "real" gains). The fact that the method can be applied to existing pretrained models is practically appealing. The results suggest that the proposed training objective may provide a useful way to improve few-step generation.

Critical weaknesses

[W1] The theoretical guarantee does not seem to extend to the actual stochastic training objective. This is my main concern. Proposition 5 establishes that the deterministic projected identity uniquely determines the exact finite-step marginals. However, in practice, the transport term is replaced by an unbiased single-sample estimator, which enters inside the logarithm of the generalized-KL loss (Eq. 21). An unbiased estimate of the correction does not generally give an unbiased gradient, since the objective is nonlinear in this correction. In fact, even with an exact denoiser, one can construct a simple two-coordinate correlated binary example where the true finite-step marginals are not a stationary point of the expected stochastic update.

There is also a related positivity issue: Eq. 20 guarantees that the shifted probabilities sum to one, but not that they are nonnegative. The supplementary code handles this by clipping the correction and flooring the shifted probabilities, neither of which is discussed in the theoretical analysis. These modifications can change the fixed point. I don't think this necessarily invalidates the empirical method, but it considerably weakens the claim that the proposed practical objective learns exact finite-step marginals. Also, the Potts experiment uses exact rather than stochastic transport, so it does not really validate this aspect of the method.

[W2] The novelty and empirical gains are not sufficiently isolated from closely related work and additional fine-tuning. The paper should better position itself relative to Discrete MeanFlow (Khan et al., 2026), which already defines the normalized increment of a CTMC transition kernel and derives a related MeanFlow identity. The backward identity and marginal projection here may still be meaningful contributions, but the novelty is narrower than the current presentation suggests. Relatedly, for standard masked diffusion with an exact posterior denoiser, the analytic bridge already gives the correct finite-step coordinate marginals. This suggests that the practical benefit may largely come from improving an imperfect pretrained denoiser rather than correcting an inherent finite-step marginal error. That could still be a useful contribution, but the experiments do not isolate it.

In particular, the OpenWebText and ImageNet models are initialized from strong Di4C+ReDi and ReDi² checkpoints, respectively, yet there is no matched additional-training baseline using the original objective. Thus it is difficult to know how much of the improvement comes specifically from the average-generator correction.

Questions for authors
  1. Can you clarify the theoretical justification for the stochastic objective? In particular, why should the exact projected marginals remain a fixed point when the unbiased transport estimator is placed inside the logarithmic loss? How is nonnegativity of the shifted probability guaranteed, and what is the effect of the clipping used in the supplementary implementation?

  2. How much improvement comes from the proposed objective rather than additional fine-tuning? A matched training-budget comparison using the original denoising/distillation objective would be important, especially given the strong pretrained checkpoints. An ablation comparing exact and single-sample transport on Potts, and removing the transport correction, would also help establish the mechanism.

  3. Can the authors provide further evidence for the practical acceleration claims? Measured inference latency and an additional text-quality evaluation beyond generative PPL/entropy would help (e.g. via GroupEval framework in Amin et al., 2026).

5 comments

We thank the reviewer for the thoughtful comments and suggestions. We appreciate the time you spent on the paper and your recognition of our contributions and strengths, including the interesting formulation with theoretical insight and the promising empirical results. Below we address the concerns and comments that you have provided.

W1. The theoretical guarantee does not seem to extend to the actual stochastic training objective. This is my main concern. Proposition 5 establishes that the deterministic projected identity uniquely determines the exact finite-step marginals. However, in practice, the transport term is replaced by an unbiased single-sample estimator, which enters inside the logarithm of the generalized-KL loss (Eq. 21). An unbiased estimate of the correction does not generally give an unbiased gradient, since the objective is nonlinear in this correction. In fact, even with an exact denoiser, one can construct a simple two-coordinate correlated binary example where the true finite-step marginals are not a stationary point of the expected stochastic update.

There is also a related positivity issue: Eq. 20 guarantees that the shifted probabilities sum to one, but not that they are nonnegative. The supplementary code handles this by clipping the correction and flooring the shifted probabilities, neither of which is discussed in the theoretical analysis. These modifications can change the fixed point. I don't think this necessarily invalidates the empirical method, but it considerably weakens the claim that the proposed practical objective learns exact finite-step marginals. Also, the Potts experiment uses exact rather than stochastic transport, so it does not really validate this aspect of the method.

Q1. Can you clarify the theoretical justification for the stochastic objective? In particular, why should the exact projected marginals remain a fixed point when the unbiased transport estimator is placed inside the logarithmic loss? How is nonnegativity of the shifted probability guaranteed, and what is the effect of the clipping used in the supplementary implementation?

Thank you very much for this insightful question. Stationarity and nonnegativity are guaranteed only when the transport term is computed exactly. As the reviewer correctly points out, we employ an unbiased single-sample estimator to approximate the transport term. However, when this estimator is incorporated into the generalized KL loss, which involves a logarithm, the resulting gradient estimator is no longer unbiased. The estimator remains unbiased only for the U-prediction loss (Eq. 9), whose gradient is affine in the estimated transport term. We present an ablation study to evaluate the effect of the clipping operation. We incorporate the discussion in the revised version (the clipped form is a biased, lower-variance surrogate).

TV at K = 4
Baseline objective0.199
Eq. 21, clipped0.117
Eq. 21, not clipped0.178
Eq. 9 (linear), no clip0.117

The clipped formulation improves the performance as expected, and we also find that U-prediction loss also achieves good performance because of its unbiased property.

W2. The novelty and empirical gains are not sufficiently isolated from closely related work and additional fine-tuning. The paper should better position itself relative to Discrete MeanFlow (Khan et al., 2026), which already defines the normalized increment of a CTMC transition kernel and derives a related MeanFlow identity. The backward identity and marginal projection here may still be meaningful contributions, but the novelty is narrower than the current presentation suggests. Relatedly, for standard masked diffusion with an exact posterior denoiser, the analytic bridge already gives the correct finite-step coordinate marginals. This suggests that the practical benefit may largely come from improving an imperfect pretrained denoiser rather than correcting an inherent finite-step marginal error. That could still be a useful contribution, but the experiments do not isolate it.

In particular, the OpenWebText and ImageNet models are initialized from strong Di4C+ReDi and ReDi² checkpoints, respectively, yet there is no matched additional-training baseline using the original objective. Thus it is difficult to know how much of the improvement comes specifically from the average-generator correction.

Thank you very much for introducing Discrete MeanFlow. Khan et al. define the same objective and derive an identity. We cite this paper in the revised version. What remains ours is p~\tilde p-reparametrization and marginal projection. We believe both are essential for achieving strong performance in real-world applications. In our experiments, we found that although the UU-prediction parameterization works well for simulations with a Bregman divergence loss, it is difficult to optimize for text generation. In contrast, the p~\tilde p-reparametrization enables us to leverage pretrained language model checkpoints, which is crucial for practical text generation tasks. Furthermore, marginal projection provides a tractable formulation that scales to large vocabulary sizes and long sequence lengths.

For standard masked diffusion, our objective has the same optimum as the denoising loss. The two checkpoints we fine-tune are not this optimum: Di4C+ReDi¹ combines one round of rectification on model-generated noise–text pairs with distillation into a few-step student, and ReDi² has been rectified twice on noise–image pairs, with an initial state in which 70% of the mask tokens are replaced by uniformly random tokens. Consequently, the optimum of our objective may differ from that of the corresponding checkpoint. An inherent finite-step error does exist on the mixture path with a non-absorbing source, which is the setting of the Potts simulations and of the new text8 experiment. There the optimum of our objective differs from the denoiser, and the ablations below show that the objective improves on continued training with the original loss.

Q2. How much improvement comes from the proposed objective rather than additional fine-tuning? A matched training-budget comparison using the original denoising/distillation objective would be important, especially given the strong pretrained checkpoints. An ablation comparing exact and single-sample transport on Potts, and removing the transport correction, would also help establish the mechanism.

Thank you for your question. The reason why we start from a pretrained model is a design we learned from [1]. In this paper, the authors show that the MeanFlow objective is the sum of a trajectory flow-matching term and a trajectory-consistency term whose gradients conflict, and that optimizing the flow-matching term first converges better. Our objective has the same structure: the gradient of Eq. 9 is the gradient of the standard denoising loss for the (t, r)-conditioned network, plus a consistency term weighted by r − t that vanishes at r = t. A pretrained denoiser has already solved the first part, and it supplies the rate Q_t that the second part needs. To alleviate the reviewer's concern, we continue training checkpoints with the discrete flow matching objective or the ReDi objective for the same number of iterations.

Potts

K = 4K = 8K = 16
Pretrained base0.1980.0570.029
+ baseline objective0.199 ± 0.0000.057 ± 0.0000.029 ± 0.000
+ ours0.117 ± 0.0030.032 ± 0.0020.022 ± 0.002

OpenWebText

K = 4K = 16K = 64
none (released Di4C+ReDi¹ checkpoint)47.524.820.3
continued ReDi49.024.420.1
Ours56.122.917.0

ImageNet

FIDIS
ReDi²7.86240
+ continued ReDi8.94 ± 0.02268.3
+ ours7.82 ± 0.06252.3

Our objective achieves better performance than continued discrete flow matching and ReDi, which verifies the effectiveness of the proposed method. In the following, we provide the ablation for the transport correction.

K = 4K = 8K = 16
Pretrained base0.1980.0570.029
+ baseline objective0.199 ± 0.0000.057 ± 0.0000.029 ± 0.000
+ ours, without the transport term0.175 ± 0.0160.080 ± 0.0130.057 ± 0.007
+ ours, exact transport0.133 ± 0.0020.035 ± 0.0010.023 ± 0.002
+ ours, one-sample transport0.117 ± 0.0030.032 ± 0.0020.022 ± 0.002

Continuing the baseline objective for the same number of steps leaves the sampler unchanged, so the gain is not due to further training. Without the transport term the sampler is worse than the base for K ≥ 8. The one-sample estimator is even better than the exact transport term. We do not have a full explanation for the advantage of the one-sample estimator. One possibility is that the noise of the estimate acts as a regularizer in this small problem.

We further conduct experiments on two setups: text8 (a character-based discrete flow matching model trained from scratch) and DiffusionGemma (a pre-trained 26B parameters discrete flow matching model)

text8, bits per character (BPC) of the samples ↓K = 4K = 8K = 16K = 32K = 64valid words, K = 64
discrete flow matching4.583 ± 0.0053.442 ± 0.0082.796 ± 0.0102.550 ± 0.0062.428 ± 0.00388.3%
our objective4.525 ± 0.0073.216 ± 0.0072.551 ± 0.0032.341 ± 0.0052.257 ± 0.00490.8%

We train from scratch with the same number of steps with our objective with the vanilla discrete flow matching objective, our objective lowers the bits per character (BPC) of the samples and raises the share of valid words, which verifies the effectiveness of the proposed method.

GSM8K, accuracy (%)K = 16K = 32
DiffusionGemma96.296.6
+ standard fine-tuning96.0 ± 0.296.4 ± 0.1
+ fine-tuning with our objective96.4 ± 0.396.6 ± 0.1

We fine-tuned LoRA adapters on GSM8K with the standard objective and with ours (3 seeds). We achieve better accuracy on math reasoning tasks, which demonstrates the effectiveness of the proposed method.

[1] Zhang et al. AlphaFlow: Understanding and Improving MeanFlow Models.

Reviewer_oawAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends MeanFlow-style average velocity ideas to continuous-time Markov chains on discrete state spaces. It defines an average generator as the normalized finite-interval transition-kernel increment, derives a self-consistency identity, and projects it onto per-coordinate marginals to obtain a tractable objective. The method is tested on an exactly enumerable Potts model, OpenWebText generation, and ImageNet token generation. It reports improved text generative perplexity at 8-64 steps and competitive four-step ImageNet results

Critical strengths
  • The average-generator definition and the projected identity are a coherent technical contribution. The paper clearly explains why the full state-space operator is intractable and why per-coordinate marginals are the relevant objects for a factorized sampler.

  • The evaluation spans a controlled setting with exact total variation, text generation, and image generation. On OpenWebText the gains over the listed strong baseline at 8–64 steps are consistent and the reported entropy makes the perplexity result more credible than perplexity alone.

Critical weaknesses
  • The empirical attribution is incomplete. The text model is initialized from Di4C+ReDi1 and the image model from ReDi2, but there is no ablation that separates the proposed correction, the initialization, the boundary fraction, and the single-draw transport estimator. Thus the experiments do not establish which part of the reported gain comes from the new objective rather than a strong inherited model or tuning choices.

  • The acceleration claim is not backed by matched compute measurements. On OpenWebText, baseline numbers are imported from prior work while the proposed model is newly trained, and the comparison reports sampling steps rather than end-to-end latency, memory, training cost, or equal-quality compute. The 16x claim compares 64 proposed steps against a 1024-step baseline point, but the paper does not show a matched-quality runtime comparison or uncertainty across training seeds. ImageNet also reports only four-step results and is not best in FID.

Questions for authors
  1. Please provide a controlled ablation from the same pretrained initialization: baseline objective, average-generator objective without projected transport, with exact versus one-sample transport where feasible, and the final method. What are mean and variation across independent training runs?
  2. Can the authors report measured end-to-end generation latency and memory at matched generative perplexity/FID, including the cost of the additional neighbor-state forward pass during training?
  3. Does the projected objective retain its advantage for a weaker or differently parameterized base diffusion model, rather than only the selected ReDi/Di4C initializations?
3 comments

We thank the reviewer for the thoughtful comments and suggestions. We appreciate the time you spent on the paper and your recognition of our contributions and strengths, including the novelty of the discrete average generator and the soundness of the experiments. Below we address the concerns and comments that you have provided.

W1. The empirical attribution is incomplete. The text model is initialized from Di4C+ReDi1 and the image model from ReDi2, but there is no ablation that separates the proposed correction, the initialization, the boundary fraction, and the single-draw transport estimator. Thus the experiments do not establish which part of the reported gain comes from the new objective rather than a strong inherited model or tuning choices.

Q1. Please provide a controlled ablation from the same pretrained initialization: baseline objective, average-generator objective without projected transport, with exact versus one-sample transport where feasible, and the final method. What are mean and variation across independent training runs?

Thank you for your question. The reason why we start from a pretrained model is a design we learned from [1]. In this paper, the authors show that the MeanFlow objective is the sum of a trajectory flow-matching term and a trajectory-consistency term whose gradients conflict, and that optimizing the flow-matching term first converges better. Our objective has the same structure: the gradient of Eq. 9 is the gradient of the standard denoising loss for the (t, r)-conditioned network, plus a consistency term weighted by r − t that vanishes at r = t. A pretrained denoiser has already solved the first part, and it supplies the rate Q_t that the second part needs. To alleviate the reviewer's concern, we continue training checkpoints with the discrete flow matching objective or the ReDi objective for the same number of iterations.

Potts

K = 4K = 8K = 16
Pretrained base0.1980.0570.029
+ baseline objective0.199 ± 0.0000.057 ± 0.0000.029 ± 0.000
+ ours0.117 ± 0.0030.032 ± 0.0020.022 ± 0.002

OpenWebText

K = 4K = 16K = 64
none (released Di4C+ReDi¹ checkpoint)47.524.820.3
continued ReDi49.024.420.1
Ours56.122.917.0

ImageNet

FIDIS
ReDi²7.86240
+ continued ReDi8.94 ± 0.02268.3
+ ours7.82 ± 0.06252.3

Our objective achieves better performance than continued discrete flow matching and ReDi, which verifies the effectiveness of the proposed method. In the following, we provide the ablation for the boundary fraction and the single-draw transport estimator on simulations.

Boundary fraction

Boundary fraction with r = tK = 4K = 32
100% (baseline objective)0.1990.0206
75%0.1340.0195
50%0.1220.0192
25%0.1170.0195
0% (no pair with r = t)0.1140.0214

The fewer boundary pairs, the lower the error at K = 4. With no boundary pairs at all, the error at K = 32 rises.

Transport estimator

K = 4K = 8K = 16
Pretrained base0.1980.0570.029
+ baseline objective0.199 ± 0.0000.057 ± 0.0000.029 ± 0.000
+ ours, without the transport term0.175 ± 0.0160.080 ± 0.0130.057 ± 0.007
+ ours, exact transport0.133 ± 0.0020.035 ± 0.0010.023 ± 0.002
+ ours, one-sample transport0.117 ± 0.0030.032 ± 0.0020.022 ± 0.002

Continuing the baseline objective for the same number of steps leaves the sampler unchanged, so the gain is not due to further training. Without the transport term the sampler is worse than the base for K ≥ 8. The one-sample estimator is even better than the exact transport term. We do not have a full explanation for the advantage of the one-sample estimator. One possibility is that the noise of the estimate acts as a regularizer in this small problem.

[1] Zhang et al. AlphaFlow: Understanding and Improving MeanFlow Models.

W2. The acceleration claim is not backed by matched compute measurements. On OpenWebText, baseline numbers are imported from prior work while the proposed model is newly trained, and the comparison reports sampling steps rather than end-to-end latency, memory, training cost, or equal-quality compute. The 16x claim compares 64 proposed steps against a 1024-step baseline point, but the paper does not show a matched-quality runtime comparison or uncertainty across training seeds. ImageNet also reports only four-step results and is not best in FID.

Q2. Can the authors report measured end-to-end generation latency and memory at matched generative perplexity/FID, including the cost of the additional neighbor-state forward pass during training?

Thank you for your question. In the following tables, we compare the latency, memory, and training costs.

Latency

Klatencyppl
Di4C+ReDi¹64487.9 ms20.32
Our64488.7 ms16.97
Di4C+ReDi¹10247797.5 ms18.44

Our method achieves a smaller PPL at K=64 than Di4C+ReDi¹ with K = 1024 with 15.95× less time.

Memory

Di4C+ReDi¹Ours
31.1 GB31.1 GB

The proposed method does not differ in Memory.

Training.

Di4C+ReDi¹ours
1.482.29 (1.5×)

The proposed method needs one extra forward pass at a neighbour state, without gradient, making it 1.5× slower during training.

Q3. Does the projected objective retain its advantage for a weaker or differently parameterized base diffusion model, rather than only the selected ReDi/Di4C initializations?

Thank you for your suggestion. We further conduct experiments on two setups: text8 (a character-based discrete flow matching model trained from scratch) and DiffusionGemma (a pre-trained 26B parameters discrete flow matching model)

text8, bits per character (BPC) of the samples ↓K = 4K = 8K = 16K = 32K = 64valid words, K = 64
discrete flow matching4.583 ± 0.0053.442 ± 0.0082.796 ± 0.0102.550 ± 0.0062.428 ± 0.00388.3%
our objective4.525 ± 0.0073.216 ± 0.0072.551 ± 0.0032.341 ± 0.0052.257 ± 0.00490.8%

We train from sketch with the same number of steps with our objective with the vanilla discrete flow matching objective, our objective lowers the bits per character (BPC) of the samples and raises the share of valid words, which verifies the effectiveness of the proposed method.

GSM8K, accuracy (%)K = 16K = 32
DiffusionGemma96.296.6
+ standard fine-tuning96.0 ± 0.296.4 ± 0.1
+ fine-tuning with our objective96.4 ± 0.396.6 ± 0.1

We fine-tuned LoRA adapters on GSM8K with the standard objective and with ours (3 seeds). We achieve better accuracy on math reasoning tasks, which demonstrates the effectiveness of the proposed method.

As the discussion period is coming to a close, please feel free to let us know if you have any further questions or concerns regarding our responses. We would be more than happy to clarify any points or discuss them further. Thank you very much for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.