- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper formulates a machinery for training discrete diffusion models for variable . Their basic strategy is as follows:
-
Suppose data distribution as ,
-
Corrupt it by a kernel as where is "round-robin" forward noising defined as
-
Consider its reversal:
-
Make model estimating for under the assumption: and differ at only one coordinate , which gives and the model approximating
-
Run the inference based on reverse process with to mimic sampling from the target distribution .
The training model is designed as: where if , and otherwise.
They validate their proposals in the numerical experiments.
Contributions
The contributions of this paper are
- formulation of the discrete diffusion model based on modeling energy function of single-site conditional distributions given the remaining coordinates,
- reformulation of the autoregressive models from this viewpoint.
Validation
The authors validate the formulation by running some numerical experiments:
- Test case1: Binary data:
- Ising model
- the NeuRISE-based denoising estimator shows the sharpest decay in total variation (TV) distance with increase in sample size, and performs better than the GGM based estimator of Varma et al. (2024), the SEDD approach proposed in Lou et al. (2024) and the discrete flow matching method of Gat et al. (2024).
- MNIST
- the NeuRISE based learning of conditionals achieve the lowest MMD and cross-correlation error.
- D-wave dataset
- Results in Table 1b show the advantage of NeurISE Diffusion in all metrics.
- Ising model
- Test case2: Multi-valued data:
- -state potts model
- the TV error decreases as the number of training samples are increased.
- -state potts model
- Test case3: Multi-valued data:
- quantum tomography of GHZ state
- the cross-correlation error goes down significantly after samples, indicating that the model is able to learn a faithful generative model for this quantum state.
- quantum tomography of GHZ state
Critical strengths
- While I do not believe this goes beyond the framework of standard discrete diffusion models, there may be some value in reframing it as the learning of a local Hamiltonian.
Critical weaknesses
- The claim that "numerical experiments demonstrate the superiority of NeuroISE diffusion models across all metrics" seems too good to be true. We can't identify any decisive reason why such remarkably favorable results would emerge.
- As previously noted, the theoretical component of this paper, i.e., applying round-robin noise modeling and its inverse, appears to be a standard approach in discrete diffusion models. Replacing this with energy function learning merely involves taking the logarithm of the standard model, suggesting little novelty in this aspect.
Questions for authors
Question regarding the aforementioned weakness
- Does simply modifying the conventional discrete diffusion model configuration to employ learned local energy function produce such dramatic performance improvements as the author emphasized? Additionally, while hyperparameters were reportedly automatically tuned, was there proper trial-and-error conducted regarding the neural network architecture and other model design aspects?
- If this formulation is novel, what specifically makes it a new idea compared to existing methods?
General questions
-
is defined for pairs , but how are the vector-valued components like defined?
-
The rationale for introducing remains unclear for me. Ultimately, only produces differences of nearly constant multiplicative factors within the portion defining , making that portion unnecessary and better handled by . Wouldn't that make the original portion redundant?
-
At various points, and appeared in similar contexts, but is there a deliberate distinction between them? If so, what is its meaning?
-
The authors write as follows:
In order to implement the reverse dynamics, we need to estimate the ratio from samples of the forward process. Suppose differ at only one coordinate , i.e., for all , . Then, for any distribution over , we have:
The part seems to derive from the forward process in a round-robin fashion, but does it need to be assumed?
-
At line 281, the final character before the closing bracket should be a comma instead of a period?