Speculative Decoding Has Disparate Impacts
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
The practice of speculative decoding, whereby inference is probabilistically supported by a smaller, cheaper, drafter model, has become a standard technique for systematically reducing the decoding time of large language models. This paper conducts an analysis of speculative decoding through the lens of its potential disparate speed-up rates across tasks. Crucially, the paper shows that speed-up gained from speculative decoding is not uniformly distributed across tasks, consistently diminishing for under-fit, and often underrepresented tasks. To better understand this phenomenon, we derive an analysis to quantify this observed “unfairness” and draw attention to the factors that motivate such disparate speed-ups to emerge. Further, guided by these insights, the paper proposes a mitigation strategy designed to reduce speed-up disparities and validates the approach across several model pairs, revealing up to a 76.7% improvement in our fairness metric.
- Keywords:
- Speculative Decoding, Fairness, Knowledge Distillation