- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Parallel drafters like DFlash and DSpark uses bidirectional attention with a fixed block size. A naive way to extend the block size would be by doing the bidirectional attention on the whole extended block. The paper raises the problem that top-1 accuracy of original block positions drops significantly because the model was not trained for longer blocks. To mitigate that, the paper suggests to mask the tokens beyond the original block, thus keeping the original predictions for the earlier tokens, while possibly benefiting from additional accepted tokens. The experiments show speedup using that simple method, compared to the naive extension.
Critical strengths
- The paper is well written, and the motivation is clear. The figures are informative and relevant.
- The method is simple to understand and implement, and is training-free.
- Experiments show speedup compared to using the naive extension.
Critical weaknesses
- There is limited novelty and the method is quite predictable in retrospect. This could also be seen as a strength.
- I think there is another ablation that is missing: training the drafter with causal attention for all positions. If bidirectional attention is not necessary, causal attention could work better for both shorter and longer block sizes.
- The experiments were conducted only with batch size 1, and not in a high performance serving framework like vLLM or SGLang. It could be useful to see the speedup on these.
Questions for authors
Do you think it's better to train DFlash/DSpark with causal attention to avoid such complications?