acceptodds
Withdrawn from ICLR 2027

At What Cost? Evaluating the Security Impact of Lossy Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Lossy Speculative Decoding (LSD) accelerates LLM inference beyond the practical limits of vanilla Speculative Decoding by trading controlled quality loss for greater speedup. However, the security costs of this quality-speed trade-off remain largely unexplored. We present the first security analysis of LSD and uncover a preference manipulation vulnerability: malicious third-party draft models can exploit LSD's tolerance for quality loss to steer final outputs toward attacker-desired outcomes. We observe that generation positions with multiple plausible continuations of comparable quality are particularly amenable to manipulation. This observation motivates a dual-objective optimization that amplifies attacker preferences at these decisive positions while promoting draft-token acceptance. Formulating this optimization is challenging because the attacker controls only the draft model and has access to neither the victim's LSD acceptance rule nor its relaxation parameter. We propose BadDrafter, a preference manipulation attack that requires no knowledge of the victim's LSD implementation. It minimizes draft-target discrepancy in generation behavior as a rule-agnostic surrogate for promoting acceptance and derives an Advantage-Inducing Shift Lower Bound (AISL) to determine the minimum distributional shift required for effective preference manipulation. We further propose CLSD, a training-free LSD method that dynamically calibrates acceptance relaxation based on token entropy to constrain the attack. Experiments across LSD methods, configurations, model pairs, decision-making tasks, and public benchmarks show that BadDrafter is effective and stealthy, while CLSD substantially suppresses the attack with minimal utility loss while retaining acceleration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Lossy Speculative Decoding, Speculative Decoding, Large Language Model Security, Preference Manipulation Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AP52OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that lossy speculative decoding (LSD) lets a malicious third-party draft model steer outputs. BadDrafter fine-tunes a draft to promote attacker-preferred tokens at high-entropy decisive positions, within a KL budget (AISL) just large enough to make them Top-1, while distilling the target elsewhere. The defense, CLSD, falls back to lossless verification at low-entropy or out-of-top-ρ positions, and otherwise relaxes acceptance with a per-token budget equal to the minimum KL needed to make the draft token Top-1.

Critical strengths
  1. Deployment-agnostic BadDrafter training. The proposed objective balances preference amplification with draft–target alignment without requiring knowledge of the victim’s LSD configuration. Experiments on the constructed recommendation benchmarks and standard utility benchmarks show that BadDrafter shifts recommendations toward attacker-preferred outcomes while largely preserving general task performance.
  2. A useful insight into evaluation blind spots. The experiments show substantial preference shifts while largely preserving standard benchmark accuracy, highlighting that utility preservation alone does not establish resistance to preference manipulation.
Critical weaknesses
  1. The proposed defense mechanisms are not very practical. CLSD’s advantage over lossless speculative decoding is unclear. Although it reduces manipulation, its approximately 1.15× speedup over lossless SD on standard benchmarks leaves a limited efficiency benefit while retaining residual risk.
  2. The broader applicability of the attack could be better characterized. The evaluation mainly focuses on constrained single-choice tasks with chain drafting, leaving its effectiveness under tree-based verification unclear. In the brand-recommendation setting, it would also be helpful to understand how attack effectiveness varies for brands with lower baseline recommendation rates.
Questions for authors

Many speculative decoding methods outside the DFlash family use tree-based verification to increase the mean number of accepted tokens. Does preference manipulation remain effective when lossy verification is combined with tree-based drafting? Can CLSD be extended to this setting while retaining its mitigation effectiveness and speedup?

Reviewer_hEeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies vulnerabilities in lossy speculative decoding (LSD), showing how malicious third-party draft models can manipulate target-model preferences through relaxed token verification. The authors propose BADDRAFTER, an attack guided by an Advantage-Inducing Shift Lower Bound (AISL), and CLSD, a defense based on entropy-aware verification and a Conservative Acceptance Bound (CAB).

Critical strengths
  • Novel attack vector - The paper identifies a previously underexplored attack surface via relaxing the traditional distribution-preservation guarantee of lossless SD.
  • Broad experimental coverage - The attack is evaluated across multiple LSD algorithms, model families, decoding configurations, and an additional tool-selection setting, supporting its generality across the tested methods.
  • Useful defense evaluation - The authors include approximately speed-matched comparisons and a defense-aware attack.
Critical weaknesses
  • Unclear practical significance - This is one of my biggest concerns. Lossless SD is commonly used, but lossy speculative decoding, particularly with untrusted third-party draft models, is much less established as far as I know. The paper does not provide sufficient evidence that this deployment scenario is common or that LSD offers compelling advantages over lossless alternatives in security-sensitive applications.
  • Limited real-world security impact - The main eval focuses on synthetic, brand-neutral smartphone recommendation tasks where multiple outcomes are reasonable and the attacker-preferred brand already has substantial prior support.
  • Theoretical parts are not sufficiently isolated - AISL is a soft training constraint rather than a guarantee of token acceptance, and its benefit over a tuned constant divergence budget is not evaluated. Similarly, the contribution of CAB is difficult to separate from CLSD's entropy gate and filter.
  • Limited defense robustness and acceleration benefits. - The adaptive attack only replaces AISL with CAB rather than directly optimizing against CLSD's full verification. CLSD also achieves only approximately 1.15× speedup over lossless SD, so it’s not a very strong motivation to me.
  • Limited statistical analysis - No confidence intervals, training-seed variance, and independent validation of the LLM-based preference evaluator makes some of the reported differences difficult to assess reliably.
Questions for authors

Can the authors provide evidence of practical adoption of LSD with third-party draft models/explain when it would be preferable to existing lossless speculative decoding methods?

Can the authors compare against a benignly distilled draft and evaluate whether simple behavioral probing can detect the malicious model?

How do AISL and CAB compare against tuned constant-budget alternatives, and can CLSD be evaluated against a stronger defense-aware attacker?

Reviewer_wC6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the security of Lossy Speculative Decoding (LSD). The authors show that a malicious third-party draft model can manipulate the target model’s outputs by exploiting LSD’s tolerance for quality loss. They propose BadDrafter, an attack that does not require knowledge of the victim’s LSD configuration. The attack encourages the draft model to remain close to the target model to maintain a high acceptance rate, while using AISL to estimate the minimum distribution shift needed to promote an attacker-preferred token. The authors also propose CLSD, a training-free defense that adjusts token acceptance based on token entropy. Experiments demonstrate that the attack is effective and difficult to detect, while CLSD can mitigate it with limited overhead.

Critical strengths
  1. Exploiting LSD’s tolerance for quality loss as an attack vector is an interesting and novel research direction.

  2. The experimental evaluation is comprehensive, covering five LSD methods, three draft–target model pairs, and an adaptive attack against the proposed defense.

  3. CLSD requires no training and can directly replace the verification rule in existing LSD methods, making it easy to deploy.

Critical weaknesses
  1. Practical relevance and scope. The proposed attack is limited to LSD, which is not currently the dominant speculative decoding paradigm and appears to have limited adoption. Could the authors discuss LSD’s practical adoption and the real-world relevance of the threat model? In addition, the evaluation only considers frameworks that use a standalone small draft model. Could the authors extend the evaluation to frameworks without a standalone draft model, such as EAGLE or Medusa, or clarify whether the proposed attack applies to these settings?

  2. Method overview. The paper lacks a framework figure illustrating the proposed method. Adding an overview diagram would help readers understand the attack pipeline and the roles of its main components.

  3. AISL budget enforcement and validation. AISL is described as the minimum distribution shift required to make the attacker-preferred token the top-ranked token. However, in Eq. (5), it is incorporated only as a soft penalty rather than enforced as a hard constraint. Consequently, the trained draft model may introduce a substantially larger shift, yet the paper does not report the actual shift. At the selected attack positions, how does the observed KL divergence compare with the AISL lower bound? Does the learned shift remain close to the estimated minimum, or does it substantially exceed it?

  4. Missing simple baselines. The evaluation would benefit from straightforward attack and defense baselines. Could the authors include such baselines to clarify the gains attributable to the proposed methods?

Questions for authors

Please see the questions raised in the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.