acceptodds
Under review as a conference paper at ICLR 2027

RelaySpec: Adapting Frozen Speculative Drafters with Linear Maps

34 views

Abstract

Speculative decoding accelerates autoregressive LLM inference by using a lightweight drafter to propose tokens that the target verifies in parallel. Recent high-performance drafters improve proposal quality by conditioning on intermediate features from the target model. This coupling, however, makes the drafter difficult to reuse when the target changes. LoRA adaptation shifts these features, and replacing the target can even change their dimensionality. We introduce RelaySpec, a method that adapts a pretrained drafter to a new target by learning one linear map per target feature stream while keeping the drafter's transformer frozen. On Qwen3-4B with GSM8K, KiCad and NanoCoder LoRAs, RelaySpec improves DFlash throughput by 53% on average over the unadapted drafter. When transferring a Qwen3-4B drafter to Qwen3-8B with in-domain fitting data, the adapted drafter reaches 94-100% of native-drafter throughput across math, code, and chat. The reverse transfer, from Qwen3-8B to Qwen3-4B, reaches 88-97%. A single fit on mixed data reaches 99.4% of native throughput on Spec-Bench. In vLLM serving, three LoRA-fine-tuned targets share a single frozen drafter, achieving 77% higher aggregate throughput than the unadapted drafter at 32 concurrent clients. RelaySpec also generalizes to EAGLE-3, improving throughput by up to 105% over the unadapted drafter across three LoRA-fine-tuned targets. It also outperforms SD by 3.6-10.2% even when SD also fine-tunes the drafter, while training 19 fewer parameters, and outperforms OmniDraft by 1.9-52.4%. RelaySpec adapts an existing drafter with thousands of target-generated trajectories instead of training a new one from scratch, which for DFlash takes roughly 800,000 examples.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Drafter Model Adaptation, Efficient LLM Serving, Cross-Model Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_24zdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RelaySpec adapts a pretrained speculative drafter to a different target model by learning linear maps between the target’s hidden features and the feature space expected by the drafter. Both models remain frozen during adaptation, and the maps can be absorbed into the drafter’s existing fusion projection for inference. The paper uses accept-until-fail supervision for LoRA-adapted targets and feature reconstruction for transfer between models. Experiments cover DFlash and EAGLE-3, several adaptation settings, and concurrent serving. The results show substantial gains over an unadapted drafter, with performance approaching that of a native drafter in several same-family transfers.

Critical strengths

The paper identifies a useful way to reuse a feature-conditioned drafter. Its strongest contribution is the finding that linear adaptation at the target–drafter interface can recover substantial performance while leaving the drafter’s transformer frozen. Absorbing the maps into the existing projection also makes the approach straightforward to deploy. This is particularly relevant when several adapted targets need to share a drafter.

The experimental support goes beyond an isolated improvement over an unadapted baseline. The paper evaluates two drafter families, compares against adaptation methods and alternative trainable components, and includes concurrent serving. The reported 1.77× throughput improvement over the unadapted drafter at 32 concurrent clients is useful evidence that the gains survive a serving setting. The appendix also provides training-cost accounting and repeated fits, which make the practical claims more credible.

Critical weaknesses

The ablations leave some uncertainty about what specifically makes the proposed recipe effective. Appendix I.1 includes a direct fusion-projection update, but it uses the same learning rate and update budget as the maps. Since the maps ultimately produce a modified fusion projection, a separately tuned projection baseline matters: the observed difference may depend substantially on optimization and parameterization. Likewise, Table 22 compares complete training recipes rather than varying only the objective. It supports the selected recipes, but provides less direct support for the paper’s explanation of why LoRA adaptation and cross-model transfer require different supervision. These issues do not undermine the measured speedups, but they weaken the interpretation of the ablations.

The independent validation of transfer is narrower than I would like. Near-native performance is most convincing for same-family transfers; for Llama-3.1-8B, Table 3 reports only 66.5% of native-drafter throughput on chat, although other workloads fare considerably better. Appendix A.2 also states that the evaluation prompts in Tables 3 and 4 were used during method development. The fresh-prompt check is helpful, but covers only one transfer direction and two workloads. This leaves uncertainty about whether the broader transfer results hold beyond the settings used to develop the method.

Questions for authors

Parameterization comparison: Was the direct fusion-projection baseline tuned with its own learning rate? How sensitive is its gap to RelaySpec to that choice? A matched-budget comparison after separate tuning would help establish the benefit of learning per-stream maps.

Objective comparison: Which training settings differ between the objective recipes in Table 22? Do you have a comparison that holds the data, initialization, and optimization schedule fixed while changing only the loss? This would help distinguish evidence for the proposed explanation from evidence that the complete recipes work well.

Independent evaluation: Which method or hyperparameter choices were made using the evaluation prompts in Tables 3 and 4? Is fresh-prompt evaluation available for the reverse same-family transfer or the cross-family setting? Those results would materially affect my confidence in the generality of the transfer claims.

Reviewer_aBh8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on a practical issue with feature-conditioned drafters: because the drafter reads the target’s hidden states, it degrades when the target is LoRA-fine-tuned and becomes incompatible when the target is replaced by a model with a different hidden dimension. To handle this, the authors propose RelaySpec, which keeps the drafter frozen and trains only one linear map for each target feature stream. The maps are merged into the drafter’s fusion projection before inference, so there is no runtime overhead. The proposed method is verified on Qwen3-4B/8B with DFlash and EAGLE-3 drafters.

Critical strengths
  • The motivation is reasonable and the method design is straightforward. Merging the maps into the existing projection means the adapted drafter runs exactly as before, and a single frozen drafter can serve several LoRA targets.
  • The choice of a different loss for the two settings is well argued and backed by ablation.
Critical weaknesses
  • For the contribution to be significant, the comparison that matters is against simply LoRA-tuning the drafter rather than against the unadapted drafter. A drafter LoRA can also be merged for single-target deployment, and in multi-target serving it can be swapped per request just like RelaySpec’s maps (this is what methods like S-LoRA do), so the “frozen drafter” property does not buy much on its own. The paper does run this controlled comparison, but in Appendix Table 21, which I found somewhat odd given how important it is. There, drafter LoRA comes close to the proposed maps (HM 1.418 vs. 1.445), and outside of KiCad there is no clear advantage for the maps at all.

  • RelaySpec seems to help little in domains the drafter already handles reasonably well (e.g., NanoCoder), which is not too bad in itself and somewhat expected. But since KiCad contributes most of the average throughput gain on the LoRA targets (2.09× versus 1.09–1.41× elsewhere), I wonder whether the map approach only works well on particular tasks.

Questions for authors
  • Were the baselines in Table 6 trained with the same loss? The ablations suggest the loss has an effect on speedup, and it seems OmniDraft is trained with a reverse-KL loss while RelaySpec uses AUF.
Reviewer_1TGTOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to reuse a pretrained drafter when the target model is fine-tuned or replaced. Specifically, the RelaySpec proposed by the author learns one linear map per target feature stream while keeping the drafter's Transformer backbone frozen. These maps are merged into existing input projections for inference. Moreover, the authors use token supervision for LoRA-fine-tuned targets and feature reconstruction for cross-model transfer, and evaluate the method in multiple models, workloads, and service settings.

Critical strengths
  1. Reusing an existing drafter after updating the target model is a meaningful way to reduce training and maintenanc costs. This paper provides evidence that changing the feature interface alone can recover substantial performance without updating the drafter backbone.

  2. The linear maps can be absorbed into existing fusion projections, avoiding a separate mapping operation during inference.

Critical weaknesses
  1. This paper uses feature reconstruction when replacing the target model, but the replacement may change output preferences and hidden features. When is feature alignment sufficient to maintain good acceptance, and when would direct supervision on the output of new targets also be needed?

  2. Since the drafter receives only the fused features, what additional benefit does matching each feature stream provide beyond matching the fused context? Is this mainly intended for stable training or to improve transfer performance? (Equation (5))

  3. RelaySpec assumes access to a pretrained drafter for a related model. When several drafters are available, what should guide the choice for a new target? Is a drafter trained for the same model family a reasonable default, or does selecting a suitable drafter require adapting and evaluating each candidate?

Questions for authors

Has a mixed domain fitting evaluation been conducted on the Qwen-to-Llama transfer at the same data? Does it substantially reduce the gap in Table 4?

Reviewer_UbxBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reuse a feature-conditioned speculative drafter when its target model is LoRA-adapted or replaced. RelaySpec freezes both the target and the pretrained drafter, and learns a linear map for each incoming target-feature stream. The maps are folded into the drafter’s existing fusion projection before inference. The training objective depends on the setting: accept-until-fail token supervision for LoRA targets, and reconstruction of the original target’s features and fused context for cross-model transfer. With DFlash, the reported greedy-decoding gains over an unadapted drafter range from 1.05× to 2.09× across five LoRA workloads (Table 2). For Qwen3-8B ← Qwen3-4B, in-domain fitting recovers 93.6–99.9% of native-drafter throughput under greedy decoding (Table 3). The paper also evaluates EAGLE-3, competing adaptation methods, and multi-LoRA serving.

Critical strengths

The main contribution is a practically useful adaptation interface. Learning upstream of the frozen fusion blocks permits the maps to be merged exactly into the inference projection (Eqs. 1–2), without inserting an additional operation into every decoding step. This matters particularly for serving multiple LoRA variants with one drafter: Fig. 4 reports 1.77× aggregate throughput over the unadapted-drafter setup at 32 concurrent clients.

The empirical study is more substantial than the simplicity of the method might suggest. It covers greedy and sampled decoding, different target widths, an EAGLE-3 extension, and comparisons with SD2 and OmniDraft. In the fresh-prompt comparison (Table 6), RelaySpec achieves a harmonic-mean gain of 1.406 versus 1.332 for SD2, while training 32.8M rather than 626.6M parameters. These are meaningful efficiency gains.

Critical weaknesses

First, the benefit of the specific map parameterization is not yet isolated from optimization effects. For the same-width Qwen3-4B LoRA setting, each learned product is F_i W_i, where both factors are square. If F_i has full rank, this has the same function class as directly updating the corresponding fusion block. Yet Table 21 reports +107.5% for per-stream maps versus +36.7% for a direct fusion-projection update on KiCad. The two variants share a learning rate and schedule, but their optimization geometry is different. Without tuning the direct-update control or characterizing F_i, this ablation does not establish that feature-space mapping, rather than parameterization-induced preconditioning, explains the gap.

Second, the strongest cross-model claims need more independent evaluation. Appendix A.2 explicitly states that the evaluation prompts in Tables 3–4 were used during method development. The fresh-prompt check is reassuring, but it covers only MATH and Chat for Qwen3-8B ← Qwen3-4B; it does not independently test code or the reverse/cross-family transfers. The limitation matters because transfer is sensitive to the fitting domain: for example, a NuminaMath fit achieves 0.777× native throughput on CodeAlpaca for that pair (Table 4), while cross-family transfer can fall substantially further below native. Thus, the near-native result is well supported for matched fitting conditions, but less so as a general claim about transfer.

Questions for authors
  1. In Table 21, how sensitive is the direct fusion-projection baseline to its learning rate or optimizer settings? Can you report a tuned comparison and, ideally, the ranks or conditioning of the frozen F_i blocks? This would clarify what advantage is specific to learning W_i.

  2. Can you provide a fully untouched evaluation cohort for code and at least one additional transfer direction, with the fitting recipe fixed in advance? It would also help to specify which method-design choices were informed by the prompts used in Tables 3–4. These results would materially affect my confidence in the transfer claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.