- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RelaySpec adapts a pretrained speculative drafter to a different target model by learning linear maps between the target’s hidden features and the feature space expected by the drafter. Both models remain frozen during adaptation, and the maps can be absorbed into the drafter’s existing fusion projection for inference. The paper uses accept-until-fail supervision for LoRA-adapted targets and feature reconstruction for transfer between models. Experiments cover DFlash and EAGLE-3, several adaptation settings, and concurrent serving. The results show substantial gains over an unadapted drafter, with performance approaching that of a native drafter in several same-family transfers.
Critical strengths
The paper identifies a useful way to reuse a feature-conditioned drafter. Its strongest contribution is the finding that linear adaptation at the target–drafter interface can recover substantial performance while leaving the drafter’s transformer frozen. Absorbing the maps into the existing projection also makes the approach straightforward to deploy. This is particularly relevant when several adapted targets need to share a drafter.
The experimental support goes beyond an isolated improvement over an unadapted baseline. The paper evaluates two drafter families, compares against adaptation methods and alternative trainable components, and includes concurrent serving. The reported 1.77× throughput improvement over the unadapted drafter at 32 concurrent clients is useful evidence that the gains survive a serving setting. The appendix also provides training-cost accounting and repeated fits, which make the practical claims more credible.
Critical weaknesses
The ablations leave some uncertainty about what specifically makes the proposed recipe effective. Appendix I.1 includes a direct fusion-projection update, but it uses the same learning rate and update budget as the maps. Since the maps ultimately produce a modified fusion projection, a separately tuned projection baseline matters: the observed difference may depend substantially on optimization and parameterization. Likewise, Table 22 compares complete training recipes rather than varying only the objective. It supports the selected recipes, but provides less direct support for the paper’s explanation of why LoRA adaptation and cross-model transfer require different supervision. These issues do not undermine the measured speedups, but they weaken the interpretation of the ablations.
The independent validation of transfer is narrower than I would like. Near-native performance is most convincing for same-family transfers; for Llama-3.1-8B, Table 3 reports only 66.5% of native-drafter throughput on chat, although other workloads fare considerably better. Appendix A.2 also states that the evaluation prompts in Tables 3 and 4 were used during method development. The fresh-prompt check is helpful, but covers only one transfer direction and two workloads. This leaves uncertainty about whether the broader transfer results hold beyond the settings used to develop the method.
Questions for authors
Parameterization comparison: Was the direct fusion-projection baseline tuned with its own learning rate? How sensitive is its gap to RelaySpec to that choice? A matched-budget comparison after separate tuning would help establish the benefit of learning per-stream maps.
Objective comparison: Which training settings differ between the objective recipes in Table 22? Do you have a comparison that holds the data, initialization, and optimization schedule fixed while changing only the loss? This would help distinguish evidence for the proposed explanation from evidence that the complete recipes work well.
Independent evaluation: Which method or hyperparameter choices were made using the evaluation prompts in Tables 3 and 4? Is fresh-prompt evaluation available for the reverse same-family transfer or the cross-family setting? Those results would materially affect my confidence in the generality of the transfer claims.