- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper gives approximation and estimation error bounds for masked diffusion models with a sliding-window Transformer. It assumes that each token mainly depends on a few nearby tokens, with exponentially decaying dependence on distant ones. Under this assumption, a window size growing as is enough, where is the sample size. The assumption is checked on Ising and GTR models.
Critical strengths
A first end-to-end error analysis for masked diffusion trained with the NELBO loss.
Critical weaknesses
I am not familiar with the approximation theory of Transformers or the theoretical analysis of masked diffusion models. I therefore lack the expertise to properly assess the value and novelty of the theoretical contributions, and I did not check the proofs in detail. My comments below are only based on the more general aspects of the paper, and I would ask the AC to give my assessment less weight.
- Strong Assumptions: The locality assumption holds for Ising models at high temperature, but already fails at low temperature (Figure 1a). Natural language has long-range dependencies, so it is unclear what the results mean for practical masked diffusion language models.
- No Validation of the Bounds: The experiments only check the assumption. No model is trained to check the predicted window scaling.
Questions for authors
Could you train sliding-window Transformers on the Ising or GTR models to verify the predicted window scaling?
3 comments
Thank you for the response. As noted in my review, I am not familiar with the theoretical aspects of this area. My main remaining concern is the lack of experiments validating the bounds, so I will keep my score.
Dear reviewer,
Thank you for your reply. We have done experiments training a sliding window transformer on sequences of length 64. The data we used is generated from a one dimensional ising model with open boundary conditions.
Changes to the manuscript
We have added a subsection I.3 to the appendix describing the experimental setup. The results are in I.3.1
Implications
The main results of interest are Figures 36, 37b and 38. Figure 36 shows the exponential decay of approximation error with K. Figure 37b shows estimation error obeys a power law decay in . A regression fit shows the exponent to be close to (so that the decay is ) for .
Figure 38 shows the plot of sliding window size vs . A least squares fit has , validating the linear fit.
Locality assumption
We thank the reviewer for recognizing our contribution to the error analysis for masked diffusion models. We address the concern regarding the strong assumptions below.
We agree that for certain large values of β, the assumption fails for the Ising model at low temperature. However, Figure 1a shows that even at low temperature, the assumption can be satisfied by choosing a lower β. Here, β is not explicitly present in the definition of the data distribution (Ising in this case). To understand the effect of choosing a lower β, consider the window size Km∼2βlog2logm derived in the appendix section “Tradeoff between Approximation and Finite Sample statistical errors” (A.9 in the current version). Thus, whenever the spatial-decay condition holds for some fixed β>0, the logarithmic scaling in m is preserved, albeit with a larger coefficient proportional to 1/β. This illustrates that weaker spatial decay does not necessarily change the main scaling result.
We also agree that exponential decay is not a good condition for distributions with long-range dependencies. We acknowledge this limitation at the end of Section 5.2.1 of the main text. To address a broader class of distributions, the appendix section “Changes under Polynomial Spatial Weights” (Appendix H in the current version of the manuscript) extends our analysis by replacing exponential with polynomial weights. This extension is motivated in part by observations of power-law correlations in natural language data.