- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes T2D, which constructs draft models for speculative decoding through learnable structured pruning of the target model. It jointly optimizes cross-entropy on target-generated data and predicted latency, with a reference-based penalty to limit alignment degradation. On Llama-2-7B-Chat and Vicuna-13B-v1.5, the resulting drafts achieve overall speedups of 1.74× and 1.89×, outperforming the evaluated pruning baselines.
Critical strengths
Jointly choosing the draft architecture and aligning it with the target is a sensible and practically relevant direction. The improvements in Table 1 hold across all 11 domains, rather than being driven by a few tasks. The method also produces compact dense models that can be used directly for inference.
Critical weaknesses
- The benefit of automatic architecture selection is not yet fully established. In Appendix D, the main comparison uses the five-layer Wide architecture adopted from prior work, whereas T2D produces three- and four-layer drafts. The ShortGPT sweep in Section 5.2 shows that the pruning ratio substantially affects speedup. However, a baseline architecture selected on validation data is not directly compared with T2D under consistent training settings. The results support improvement over the selected baselines, but the additional benefit and construction cost relative to a simple architecture search remain unclear.
- The central assumption behind the surrogate objective is not directly validated. T2D optimizes CE and predicted single-step latency, whereas the actual objective depends on consecutive acceptance length and end-to-end speedup. Figure 3 illustrates the trade-off for ShortGPT, and Figure 4 shows training dynamics. Neither establishes whether T2D's surrogate reliably ranks candidate architectures. Appendix H also provides no latency prediction error.
Questions for authors
- How much advantage remains after tuning the depth and width of Wide/ShortGPT on validation data under matched data and training budgets? How does a baseline that fixes T2D's discovered architecture before pruning and distillation perform?
- Could the authors report CE, predicted and measured latency, acceptance length, and actual speedup for the same candidate architectures? Ablations removing the latency term or reference penalty would also help establish which components improve architecture selection.
3 comments
Thank you for the additional experiments. This shows that giving the 3-layer architecture by itself does not recover the full gain. It largely addresses my first concern. For the second concern, I have one remaining question: how many draft architectures are used to compute the correlation (−0.93 ) between CE and acceptance length? Overall, the new results have improved my assessment of the paper.
Dear Reviewer 9gwq,
Thank you for your feedback and for increasing your assessment score. We are happy to hear that your first concern has been addressed.
Regarding your second concern:
- We have used 8 draft architectures to compute the correlation (−0.93) between CE and acceptance length.
- Overall, the correlation reinforces that decreasing CE on target-model-generated responses can increase the acceptance length.
Thank you once again for your thoughtful feedback. We truly appreciate the time and care you have put into reviewing our work. We hope your remaining concern is addressed, and we are happy to discuss further or provide any additional information if needed.
Kind regards,
Authors
Dear Reviewer 9gwq,
Thank you for sharing your observations on our paper. We ran additional experiments on matched architectures, latency/speedup consistency, and the CE-reference ablation, to address your queries. We describe our efforts in detail below.
W-1: The benefit of automatic ... architecture search remain unclear.
Q-1: How much advantage remains ... pruning and distillation perform?
WR-1 & QR-1: Thank you for the excellent observation. In the main results, the baselines use the 5-layer architecture chosen by the Wide authors' own architecture search, while T2D discovers its architecture in a single run. We ran the baselines with T2D's 3-layer architecture under the same data and training budget:
W-2: The central assumption behind the surrogate objective ... no latency prediction error.
Q-2: Could the authors report ... establish which components improve architecture selection.
WR-2 & QR-2: Thank you for your observation. We provide the following clarifications: