acceptodds
Withdrawn from ICLR 2027

Why Freeze the Target Model? Aligning the LLM Toward Its Small Drafter for Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding accelerates large language model (LLM) inference by using a small language model (SLM) to draft several tokens that the LLM verifies in parallel. Existing methods typically focus on training the SLM towards LLM to fully imitate the LLM’s distribution. However, we observe that, in open-ended generation, many rejected SLM outputs are semantically correct but differ from the LLM in wording or style. For a capacity-limited SLM, matching the LLM’s wording and style in addition to generating correct content remains challenging. This raises a key question: can the LLM instead accommodate the SLM’s expression style while retaining its own capabilities? Building on this idea, we propose SpecL2S, a two-stage mutual alignment framework. We first use content-aware distillation to train the SLM to prioritize content accuracy over stylistic imitation. We then freeze the trained SLM and optimize a lightweight soft prompt for the LLM, adapting its expression preferences to the SLM while preserving its content-critical behavior. Together, these two stages align the SLM toward the LLM’s capabilities and the LLM toward the SLM’s expression style. The resulting pair uses standard parallel verification without an additional semantic judge. Across five open-ended benchmarks, SpecL2S increases the mean number of accepted tokens per target call from 3.05 to 4.57 and achieves a 2.43× decoding speedup, while quality remains nearly unchanged

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative Decoding, Large-Small model Collaborate

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wc3zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies that speculative decoding often prematurely rejects correct-content drafts due to superficial discrepancies between the draft and the target model. To address this, we propose SpecL2S, an asymmetric mutual-alignment framework. In the first stage, we employ content-aware distillation to focus the small model on critical content-related decisions. In the second stage, we freeze the models and train only the soft prompts of the target LLM, enabling it to remain compatible with drafts despite minor, low-risk variations in expression, while incorporating a distribution-preservation constraint to prevent the acceptance of incorrect content. Experiments demonstrate an approximately 2.43x speedup, with open-ended generation quality remaining essentially unchanged.

Critical strengths
  • The authors point out that a significant number of rejections in speculative decoding stem from superficial differences—such as paraphrasing, word order, or formatting—rather than actual content errors. Based on this, they propose an asymmetric alignment strategy: the content is learned from the draft model, while the expression is aligned with the target model. This perspective is novel and well-motivated.

  • The approach involves training only short, soft prompts while keeping both models frozen. By incorporating content-aware distillation and distribution-preserving constraints, it achieves approximately 2.43x speedup across various model pairs and open-ended benchmarks.

Critical weaknesses
  • It changes the served target distribution. SpecL2S is lossless only for the adapted target pθ,ϕp_{\theta,\phi} , not the original pθp_\theta. Quality must therefore be re-evaluated rather than assumed preserved.

  • The criticality router is costly and imperfect. It still falsely accepts 3.40% of content-critical errors and may miss subtle factual or safety issues, while conservative thresholds reduce speedup.

  • The soft prompt is pair- and domain-specific. It must be retrained when models or traffic change, and gains shrink under high concurrency, short generation, or small target models.

  • Evaluation is still narrow. It excludes exact-match math, executable code, and safety-critical tasks, so quality retention is shown only for the tested open-ended benchmarks and model pairs.

Questions for authors
  1. The criticality router still falsely accommodates 3.40% of content-critical mismatches and depends on costly human/LLM-assisted labels. How do you bound the risk of subtle factual or safety-critical errors in deployment, and can the router be made more robust without giving up the reported 2.43x speedup?

  2. SpecL2S is lossless only with respect to the adapted target pθ,ϕp_{\theta,\phi}, not the original pθp_\theta, and the soft prompt is pair- and domain-specific. How should practitioners detect distribution drift in a new domain, and what fallback or retraining protocol do you recommend before serving the adapted target?

Reviewer_nbhVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to train the target model in speculative decoding, so it's more aligned with the draft model and increases the acceptance rate. The target-side training adopts a prompt-tuning approach, which appends some trainable prompt tokens to the question.

Critical strengths
  1. The proposed method is easy to understand and implement.
  2. According to Table 2, the final performance is only slightly decreased in the evaluated benchmarks.
Critical weaknesses
  1. The evaluation benchmarks are intentionally cherry-picked. According to Lines 313-318, this paper evaluates only the easy tasks, excluding mathematics and code tasks, which is where degradation is most likely. Therefore, the claim "quality remains nearly unchanged" is questionable.
  2. Weak and missing baselines. EAGLE-2 is an old and weak baseline since EAGLE-3 has proven to be a stronger baseline, and there are publicly available EAGLE-3 checkpoints.
  3. Missing results of baselines. Table 2 shows results of SpecL2S on many benchmarks, but the results of other baseline methods are missing. Therefore, I can't see the evidence that SpecL2S is a better method.
Questions for authors
  1. Are there any motivations to use prompt-tuning instead of other fine-tuning methods, like LoRA?
Reviewer_ZphyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpecL2S for speculative decoding. It first trains the small drafter with more weight on content-critical decisions. Then it learns a soft prompt to make the target model better match the drafter’s expression style. The paper reports higher acceptance and faster decoding, with small quality changes on selected open-ended benchmarks. The method preserves the adapted target distribution, rather than the original target distribution.

Critical strengths

The method does not need an extra semantic judge during inference. This is useful for keeping the inference process simple. The paper also includes ablations for the two training stages and the preservation loss, and clearly acknowledges that the target distribution is changed.

Critical weaknesses

I think the main idea needs a better comparison with earlier work. Blockwise Parallel Decoding (Stern et al., 2018) already discussed freezing versus fine-tuning the base model and studied the trade-off between accepted block length and quality. Medusa-2 also fine-tunes the backbone together with its prediction heads. So changing the target to improve decoding efficiency is not a new question. The specific content-aware training and soft-prompt adaptation may still have novelty, but the paper should explain this narrower contribution more clearly. It should also specify whether the Medusa baseline uses Medusa-1 or Medusa-2.

The observation that good draft tokens can be rejected because they differ from the verifier’s preferred tokens is also already studied. Aggressive Decoding discusses this in its GAD++ variant. Judge Decoding addresses the same problem by learning to recognize valid but non-aligned continuations, yet it is missing from the discussion and experiments. FLy is discussed in the paper but is not reported as a separate baseline. These methods change verification rather than adapt the target, but they are relevant comparisons for the problem this paper wants to solve.

This also makes the comparison setting important. SpecL2S changes the original target distribution, so its benefit should be evaluated against other methods that allow output changes, at comparable quality levels. Table 1 gives only one generic “relaxed verification” result. This is not enough to show whether the proposed adaptation gives a better speed–quality trade-off. A curve over different preservation strengths and baseline acceptance settings would be more informative than comparing a few selected operating points.

My bigger concern is still the quality evaluation. Changing the target distribution does not necessarily hurt quality, but it removes the guarantee relative to the original model. However, the evaluation mainly uses tasks within the small drafter’s capability range. Some subsets exclude mathematics and coding, and IFEval is restricted to single-constraint prompts. These experiments largely avoid the difficult cases where the target may be much stronger than the drafter. OOD evaluation with the learned prompt kept fixed is also needed. The current results support an optimization for selected workloads, but do not establish that the broader abilities of the original model are kept. The main quality tables also do not report uncertainty intervals, even though the paper mentions five seeds and bootstrap evaluation. Small score differences alone are not enough to establish negligible degradation.

The criticality evaluator needs more detail. The method depends on this evaluator to distinguish harmless wording changes from content errors, but its model identity and judging prompt are not specified clearly. The quality judges are also not identified clearly. The paper reports human calibration data, but the full cost of collecting counterfactual continuations and repeatedly judging mismatch pairs is unclear. Since these labels determine which target behavior is allowed to change, this is an important part of the method to make reproducible.

There are also some numbers that need explanation. Averaging the five adapted-to-original score ratios in Table 12 gives about 99.55%, while the paper reports 99.21% retention. Weighting the surface-category recovery rates by their shares in Table 3 gives about 70.51%, rather than 69.80%. These differences may come from another aggregation procedure, but that procedure should be stated. Table 12 also gives identical rounded scores for adapted-target AR and SpecL2S on all five benchmarks. This is possible, but matching aggregate scores does not establish distributional equivalence. The authors should explain how these outputs were generated and whether randomness or generated responses were shared between the two evaluations.

Questions for authors
  1. What is the main new contribution compared with the freezing/fine-tuning experiments in Blockwise Parallel Decoding and the compatibility ideas in early Speculative (Aggressive) Decoding?

  2. With the learned prompt fixed, how does the method perform on difficult reasoning, executable code, and domains not covered by the adaptation data?

  3. Can the authors report paired confidence intervals for quality differences and specify what amount of degradation the experiments can rule out?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.