- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a new sampling mechanism for uniform discrete diffusion where the "context" can be made persistent by giving it to the network, while using the current state in the reverse transition's conditioning. They show that a certain distillation procedure does not preserve the LOO structure of the model and that this procedure has a different expressivity than top-p sampling. They design confidence strategies to choose the selected context and investigate empirically the effect of the proposed method on natural language.
Critical strengths
The proposed persistent context mechanism is simple and training free, contrary to similar ideas like self-conditioning, which makes it very easy to test and implement in a trained model. The DCD result is interesting and can guide the choice of tokens in the context, although not really surprising.
Critical weaknesses
The theoretical results are mostly toy-ish and do not give a lot of insights into the proposed sampler. For instance the top-p one is relatively artificial, since they just show that top-p cannot reproduce some distribution but they do not show the need to target such distribution or the fact that this is specifically an issue.
The sampler introduces a bias such that the model is not exact anymore (in the infinite time limit) which is not very well motivated or interpretable. One could argue that temperature or top-p sampling does the same, but in this case, it is much more interpretable.
The experiments are narrow in terms of their scope (only natural language and a single model comparison).
Questions for authors
Do you have a principled explanation for the proposed sampler, apart from the toy-example considered in the appendix ? An analysis that could show the benefit of this method or a general condition (maybe on p_0) that could show the advantage of introducing a bias ? Could you explain the experimental results that you get ? For instance, why does your method sometimes seem to help at matched entropy but not at matched repetition ? Why does it seem to not improve for GIDD ? What would help the paper be more convincing would be to have an underlying predictive interpretation for these results.