- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents a serving system for in-flight adaptation of speculative-decoding draft models SERVAL. It combines domain specific LoRA adapters and online updates from verification feedback collected while requests are still running. Its system design supports multi-adapter drafting and adapter-version switching without recomputing prefix KV caches. Experiments on three models and several domains show improved acceptance length and lower time per output token than the evaluated baselines.
Critical strengths
System-level contribution. The combination of domain-specific LoRA adapters, in-flight updates based on verification feedback, and efficient adapter management is a meaningful systems contribution. In particular, supporting multiple adapters within a decode batch and switching adapter versions without recomputing prefix KV caches addresses practical challenges in deploying online adaptation.
The paper addresses the distribution mismatch between draft models and online inference traffic. The reported results show improvements in acceptance length and time per output token across the evaluated models and workloads.
Critical weaknesses
1)The motivation and presentation of LoRA adapters need improvement. The introduction does not sufficiently motivate the introduction of low-rank adapters in the context of closely related work. The use of enumerated statements and a “first, second, third” structure makes the discussion feel more like a list of design choices than a progressively developed research argument. The introduction would benefit from a more coherent, step-by-step narrative that derives the proposed design from the limitations of prior work. The key differences from closely related approaches should also be explicitly stated in the introduction, rather than left primarily to the related-work section. 2)The paper does not provide comprehensive empirical comparisons with several closely related online-adaptation systems, including Aurora and EvoSpec. 3)SERVAL appears to combine several existing ideas, including online adaptation, LoRA, domain-specific specialization, and serving-time model updates. While their integration may constitute a useful systems contribution, the paper should clearly distinguish the components that are technically novel from those adopted from prior work.
Questions for authors
The authors should evaluate the robustness of SERVAL to domain-routing errors and discuss how the system would handle requests that span multiple domains or do not belong to any predefined domain.
2 comments
Thanks for your response. The authors have addressed some of my main concerns. Considering the overall technical contributions of the paper, I will maintain my original score.
Reply to Reviewer e69u
We have revised Section 1 so that it derives the design step by step and separates new from adopted components (W1, W3).
W1. Introduction and LoRA. The revised Section 1 derives the design from what a running request needs in order to benefit from its own feedback: new draft weights must reach it mid-request without invalidating its draft KV cache. OSD and Aurora install new weights only between requests, and AdaFlash reloads one global drafter for all running requests. In DFlash the cached keys and values are projections of the target's hidden states, so updating only the query and output projections keeps every prefix cache valid. Serving several domains and versions in one batch then needs small updates: a rank-32 LoRA on those two projections is 7.8 MB per version, against about 1.8 GB for full training with the key and value projections frozen.
W2. Aurora and EvoSpec. We have not evaluated either system end to end: EvoSpec has no public code, and Aurora's released code trains an EAGLE-3 draft, so a comparison would also differ in draft architecture. Inside SERVAL, two ablations isolate Aurora's choices. The first applies Aurora's version timing at SERVAL's update rate, so that only the timing differs: one global adapter whose new versions reach only requests admitted afterwards. Its AL is 3.0% below that of the same adapter updated in flight under abrupt domain shifts (3 runs), and 0.9% below on stationary traffic. The second delays feedback to the end of each request, as Aurora does, and lowers AL by 0.8% (stationary stream). SERVAL-shared follows EvoSpec's design of a single continually trained LoRA but is not a reproduction of EvoSpec. Per-domain adapters match its AL in aggregate and exceed it by 0.9% when traffic returns to an earlier domain.
W3. New and adopted. Adopted: training the draft from verification feedback and specializing it by domain (OSD), LoRA for online draft training (EvoSpec), loading new draft weights during serving (Aurora, AdaFlash), and batching many LoRA adapters in one forward pass (S-LoRA, Punica). New: (1) each running request switches to a new version of its adapter at its next decode step, independently of other requests, while its prefix KV cache stays valid; (2) several continuously trained draft adapters, and several versions of each, in one decode batch; (3) one trainer that updates several adapters concurrently from the hidden states the server exports, without running the target's backbone. The closest system is AdaFlash (Section 5), whose paper describes hot-reloading one global drafter between scheduling steps: its new weights also reach running requests, but all requests share one drafter, so it has neither (1) nor (2).
Q. Routing errors, multi-domain and unknown requests. On a 1,200-request stream (2 runs), misrouting 10% and 30% of requests lowers AL by 0.5% and 1.1%, and the shared adapter, which needs no labels, is within 0.3% of oracle routing in AL. Requests outside every domain, or spanning several, can be served by the shared adapter. We have not measured per-domain adapters with such a fallback, or requests that span domains.
We would be grateful if you would reassess the paper with these changes.