- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies an interesting phenomenon exhibited by several representative continuous diffusion language models: the coverage of rare words in generated text is lower than that in human-written text. The paper hypothesizes that, because the denoising head is trained to predict the mean of possible clean embeddings, the embeddings predicted during generation tend to bias toward more common words, with rare words being too insignificant to be recovered by the final decoding step. The paper then quantitatively verifies the hypothesis and proposes a sampling method to mitigate the issue. The proposed sampling method detects the potentially missing rare words at an early stage of generation, commits to the most likely rare word, and then reruns the generation from scratch with the committed rare word being fixed. The results show that the proposed sampling method can improve the rare-word usage
Critical strengths
- The findings of the paper are an interesting phenomenon of several representative continuous diffusion language models. This paper quantitatively verifies the proposed hypothesis and proposes a sampling algorithm that effectively mitigates this issue.
Critical weaknesses
- The proposed sampling method depends on many heuristics, e.g., the read time, the threshold to determine the strongest rare candidate, the number of committed rare words, and it seems many of them can not be determined without first checking the actual predicted distribution of the model. Is there a more principled way to determine these design options? Furthermore, they may also depend on other hyperparameters such as sampling steps and sequence length, which seem to be fixed in the experiments
- The paper only reports metrics regarding word frequencies, but people would be naturally interested in whether the early committed rare words might affect the quality of final samples. Moreover, since the re-generation inevitably incurs overhead, it would be better to more comprehensively profile the quality-efficiency trade-off
Questions for authors
Please see Weaknesses