- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes the Discrete Average Generator, an extension of MeanFlow to CTMC for few-step discrete diffusion generation. The authors define an average generator on an interval basis, derive the self-consistency identity, and develop the training objective. Validation includes Potts model simulation, text generation on OpenWebText, and image generation on ImageNet. The experimental results show lower distributional error, improved few-step generation perplexity of text, and satisfactory image generation performance.
Critical strengths
The most substantial contribution of this study is extending MeanFlow to the discrete state space, which gives a mathematically grounded method. The proposed average generator contains a training objective made tractable via per-coordinate projection. Experimental results also demonstrate its effectiveness in both text and image generation tasks.
Critical weaknesses
The newly added contribution in terms of methodology is not sufficiently evaluated. The Kolmogorov backward equation directly gives the average generator identity, so the main technical contribution seems to be the tractable training objective and the corresponding estimator. However, the reported experiments lack sufficient ablation results to show whether the performance gains come from the proposed method or simply from further training.
Another concern is that the manuscript in its current form is somewhat verbose and could be better organized. For example, the Preliminaries section is much too long, and some mathematical details could be moved to the appendix.
Questions for authors
Have the authors compared the proposed training objective with simply continuing training with the original loss, starting from the same weights and using the same training budget? How much improvement does the newly proposed objective provide?
2 comments
As the discussion period is coming to a close, please feel free to let us know if you have any further questions or concerns regarding our responses. We would be more than happy to clarify any points or discuss them further. Thank you very much for your time.
We thank the reviewer for the thoughtful comments and suggestions. We appreciate the time you spent on the paper and your recognition of our contributions and strengths, including the novelty of the discrete average generator and the performance of the proposed method. Below we address the concerns and comments that you have provided.
Thank you for your suggestion. We provide a detailed comparison on various backbones.
Potts
OpenWebText
ImageNet
Our objective achieves better performance than continued discrete flow matching and ReDi, which verifies the effectiveness of the proposed method.
We further conduct experiments on two setups: text8 (a character-based discrete flow matching model trained from scratch) and DiffusionGemma (a pre-trained 26B parameters discrete flow matching model)
We train from scratch with the same number of steps with our objective with the vanilla discrete flow matching objective, our objective lowers the bits per character (BPC) of the samples and raises the share of valid words, which verifies the effectiveness of the proposed method.
We fine-tuned LoRA adapters on GSM8K with the standard objective and with ours (3 seeds). We achieve better accuracy on math reasoning tasks, which demonstrates the effectiveness of the proposed method.
Thank you for your suggestion. We moved the detailed discussion of the discrete flow matching and the MeanFlow parameterizations to the Appendix in the revised version.