- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work tackles the problem of errors that arise from parallel decoding from masked diffusion models (MDMs) owing to their independent factorization of tokens. This paper proposes a novel tensor decomposition framework for parameterizing the conditional joint distribution of masked tokens, which leads to an efficient sampling procedure that can be enabled with lightweight fine-tuning of existing MDMs. The authors demonstrate that their method leads to lower text Gen PPL and improved molecular validity, especially for TTD, in few-step settings.
Critical strengths
- The paper addresses an important limitation of masked diffusion models.
- The application of tensor decomposition to incorporating cross-token dependencies is novel and represents an interesting approach.
- The efficient sampling algorithm is interesting.
Critical weaknesses
Improved presentation
- I think the presentation can be made more readable by better motivating how these decompositions introduce inter-position dependencies. For example, I found it useful to conceptualize CPD as a single shared categorical latent variable that can be marginalized and TTD as a chain of auxiliary states with first-order dependencies (somewhat akin to a bigram model on this latent chain).
- Moreover, throughout, I believe it would be useful to provide tensor shapes more prominently. This will also clarify what the overall memory overhead is for each approach and how it compares to full dependencies on one extreme vs. vanilla MDLM on the other.
Limited Experimental Results
- Currently I found the OWT Gen PPL vs entropy results to not be very compelling. While gen ppl improves for both MDLM and VADD this comes at the expense of diversity, and there is not a clear dominance in terms of the entropy vs gen ppl tradeoff (and additionally I have questions about the entropy computation used as the reported values seem significantly different than those previously reported in the literature). I think it is also important to supply example generated samples to better understand the qualitative nature of the model outputs.
- I was confused why there was not a VADD + CPD/TTD variant reported for the QM9 results. In my opinion novel sample count (which I assume consists of novel and unique samples) is the most important metric, and VADD is markedly superior here for >10 steps.
- One suggestion would be to explore additional applications. I think results on the popular Sudoku task and tinyGSM dataset would be quite well suited for the proposed method and for highlighting its low-NFE performance.
I am open to adjusting my score with more compelling empirical results.
Questions for authors
- The Entropy values in Tables 1, for example, seem very high relative to those typically reported for these models which is closer to < 5.5. What might be causing this discrepancy?