At What Cost? Evaluating the Security Impact of Lossy Speculative Decoding

Lossy Speculative Decoding (LSD) accelerates LLM inference beyond the practical limits of vanilla Speculative Decoding by trading controlled quality loss for greater speedup. However, the security costs of this quality-speed trade-off remain largely unexplored. We present the first security analysis of LSD and uncover a preference manipulation vulnerability: malicious third-party draft models can exploit LSD's tolerance for quality loss to steer final outputs toward attacker-desired outcomes. We observe that generation positions with multiple plausible continuations of comparable quality are particularly amenable to manipulation. This observation motivates a dual-objective optimization that amplifies attacker preferences at these decisive positions while promoting draft-token acceptance. Formulating this optimization is challenging because the attacker controls only the draft model and has access to neither the victim's LSD acceptance rule nor its relaxation parameter. We propose BadDrafter, a preference manipulation attack that requires no knowledge of the victim's LSD implementation. It minimizes draft-target discrepancy in generation behavior as a rule-agnostic surrogate for promoting acceptance and derives an Advantage-Inducing Shift Lower Bound (AISL) to determine the minimum distributional shift required for effective preference manipulation. We further propose CLSD, a training-free LSD method that dynamically calibrates acceptance relaxation based on token entropy to constrain the attack. Experiments across LSD methods, configurations, model pairs, decision-making tasks, and public benchmarks show that BadDrafter is effective and stealthy, while CLSD substantially suppresses the attack with minimal utility loss while retaining acceleration.