- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies whether discrete diffusion models can learn efficiently when only a small fraction of all possible sequences occur in the data. The paper introduces a smoothing method that assigns probability to unseen sequences, where the probability is determined by the number of distinct training sequences. The paper shows that masked and uniform diffusion can achieve statistically optimal estimation rates without knowing the true number of possible sequences.
Critical strengths
- Useful theoretical insight.
This work shows that both masked and uniform diffusion can learn sparse distributions under ideal conditions. This helps clarify what models can theoretically achieve.
Critical weaknesses
- Limited practical significance.
The paper does not show why diffusion is beneficial or whether trained neural networks achieve the same guarantees. The practical benefit remains unclear.
Questions for authors
- How can these results guide the training of diffusion models in the real world?
- Is there any evidence that the proposed regularization improves diffusion models trained on real-world datasets?
2 comments
Thank you for clarifying the training recipe and theoretical scope.
My practical-significance concern remains unresolved, and related concerns were also raised by Reviewers H4TK and 7xtj.
The explicit neural construction uses weights computed from sample counts and does not establish guarantees for networks learned through SGD. Also, the response describes promising text8 behavior but does not yet provide quantitative results.
As the practical benefit cannot be assessed, which is my main concern, I maintain my score.
We thank you for your review. The object of this paper is the theoretical study of discrete diffusion models, and we focused on the case of sparse distribution estimation, a class that seems realistic for many structured discrete data. Valid sentences, syntactically correct programs, valid molecules and functional protein sequences are all a vanishing fraction of the strings of the same length.
We agree that why diffusion is beneficial in practice, and whether trained neural networks achieve the same guarantees, are important questions (we are adding a result in which, for masked diffusion, we also show that the minimax rate is attained by an explicit ReLU--softmax denoiser whose size grows linearly with the sparsity, see our answer to Reviewer H4TK, although this construction does not cover SGD training). Our results address a question that comes before them: what discrete diffusion models can achieve statistically, through their own training objective and sampler. A trained network that approximately minimizes LM therefore targets a minimax-optimal law, and any remaining gap is due to approximation or optimization. Quantifying this gap for specific architectures is a natural next step, which we listed as future work.
Although our work does not aim to beat any benchmark, we believe it can help to better design these models. For example, it shows that the standard, unregularized objective is minimized by the empirical score, whose sampler only reproduces the training sequences and assigns zero probability to every unseen one. Regarding your questions, we answer point by point below.