- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SPDLMs model dependencies among tokens generated in the same denoising step, using a lightweight decoder and one backbone evaluation. Experiments on TinyStories, GIDD-3B, and Sumi-7B show improved few-step generation and quality–latency trade-offs.
Critical strengths
-
The method models within-step dependencies without extra backbone evaluations. The controlled comparisons in Appendix C.6 support the benefit of higher conditional order.
-
The results show a useful quality–latency trade-off. On TinyStories, one-step W32K32 achieves comparable matched-entropy GenPPL to the 32-step Product baseline with an 8.9x speedup. The Sumi-7B results also show higher GSM8K accuracy with little added decoding cost.
Critical weaknesses
The larger-model experiments use W16K2 and 128-token outputs. It is unclear whether higher conditional order improves the quality–latency trade-off at this scale, or whether the gains persist with longer outputs.
Questions for authors
-
At few-step budgets, how does W16K16 compare with W16K2 on GIDD or Sumi in quality and latency?
-
Do the quality–latency gains over Product persist with longer outputs? A GSM8K comparison at 256 answer tokens under matched training and decoding settings would help.
2 comments
I thank the authors for the detailed response and additional experiments, which address most of my concerns. Based on my overall assessment of the paper's contributions, I will maintain my original score.
Author answer
We thank the reviewer for recognizing the within-step dependency mechanism and the studies of different structures. The two questions identify valuable tests beyond TinyStories, and we have directly addressed both on Sumi-7B: independent Product/W16K2/W16K16 adaptations across three seeds, matched 256-token targets, and all 1,319 GSM8K test questions per setting.
Question 1: higher order at 7B scale
We report in the table above the mean / standard deviation across training seeds, along with batch-1 time. At 256 answer tokens, higher order improves mean accuracy by 0.83 points at eight steps and 3.18 points at 32, at essentially the same measured time. Both width-16 paths have depth 16: the intervention adds clean conditioning information, not head rounds or backbone calls. K16 wins two of three seeds at each 256-token budget. This extends the matched comparison of conditional orders to 7B scale and demonstrates a better mean quality–cost point in the tested setting.
Question 2: 256-token answers
Yes, the gains persist under matched ancestral decoding. We changed the training targets as well as the evaluation window, rather than only extending old 128-trained checkpoints. K2 and K16 beat independent Product in every seed at both 256-token budgets above; at eight steps, accuracy increases from 4.12% to 12.36%/13.19% with modest added generation time. All cleaned test references fit 256 tokens.
The additional 128-token evaluation uses these same 256-trained adapters; K2 is approximately tied with Product there, while K16 improves the mean. We will report these results alongside the native-confidence comparisons, keeping the sampler settings explicit.