- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to choose the number of draft tokens in speculative decoding, arguing that a fixed length cannot adapt to changing prediction confidence.
It proposes CSD, which uses the draft model’s confidence and recent changes in probability margins to choose an initial draft length, then permits a short extension when confidence and margins remain high.
Experiments report speedups over autoregressive decoding and improvements over speculative decoding baselines.
Critical strengths
- A simple method with practical integration potential.
CSD uses probability statistics already available during drafting, requires no additional training, and retains the underlying verification procedure.
- The experiments include meaningful component ablations.
Table 2 shows that round-level control and late extension each improve throughput, with their combination performing best across four datasets.
Critical weaknesses
The manuscript’s presentation makes it difficult to connect the motivation, decision rules, and experimental evidence.
- Figure 1 occupies substantial space but leaves its “difficulty score” undefined and does not clarify whether the curve is empirical or schematic (lines 54–107). More substantively, “stability” denotes historical margin variation in round-level control but a current margin exceeding its historical median in late extension, without explaining the distinction (lines 197–262).
- The relationship and update rules of the two history structures are unclear, and the serial and parallel descriptions do not consistently explain the signals available when draft length is selected (lines 246–293).
- Tables also fail to distinguish planned from actual draft lengths, while Tables 3 and 5 report different throughputs and reverse fixed-length rankings without explaining configuration differences.
- Repeated citation-formatting errors, an uncited reference to SpecDec++, and disruptive figure/table placement further impede reading. Together, these issues prevent the reader from confidently reconstructing the method and interpreting the evidence supporting its claims.
Questions for authors
-
Evidence for the motivation: Is Figure 1 an empirical plot or a schematic illustration? What exactly is its “difficulty score,” and how does it relate to the confidence and stability statistics used by CSD? For the analysis in lines 99–107, how was confidence controlled when assessing the additional information provided by stability?
-
Algorithm specification: Round-level control measures variation in historical margins, whereas late extension compares the current margin against its historical median. What justifies interpreting the latter as stability? Please clarify the relationship, initialization, and update rules of (B_t) and (H_t), and which current confidence value is available when parallel decoding selects its planned length.