acceptodds
Under review as a conference paper at ICLR 2027

SERVAL: Efficient In-Flight Draft Adaptation for LLM Serving

Abstract

Speculative decoding accelerates large language model inference: a draft model proposes tokens and the target model verifies them. Its speedup depends strongly on the acceptance length, the number of tokens each verification step accepts, which often falls when a fixed draft encounters out-of-distribution traffic. We introduce SERVAL (SERVing-time Adaptive LoRA), a speculative-decoding serving system that adapts the draft (e.g., DFlash) to the traffic it serves using persistent per-domain LoRA adapters. At each decode step, the server sends the target model's hidden states and token-level acceptance outcomes to a trainer on separate GPUs, which updates the adapters asynchronously. The adapters are applied only to the query and output projections of a DFlash draft, whose prefix key–value (KV) cache is computed from the target's hidden states, so running requests switch to new adapter versions at the next decode step after they arrive, without recomputing that cache. Starting from a general adapter it trained earlier, SERVAL has a higher acceptance length and a lower mean time per output token than the frozen DFlash draft for three models on every out-of-distribution stream and benchmark we evaluate. On a mixed stream of code, finance, and medical requests, it raises the acceptance length by 12–57% and lowers the time per output token by 12–51%. The server's overhead for sending feedback and loading new adapter versions is only 0.6–2.1% of the decode step. The per-domain adapters it learns can be saved for reuse: served frozen by another server, they keep a higher acceptance length than DFlash and the general adapter on Qwen3-8B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, LLM serving, Online draft adaptation, Knowledge distillation, Parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e69uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a serving system for in-flight adaptation of speculative-decoding draft models SERVAL. It combines domain specific LoRA adapters and online updates from verification feedback collected while requests are still running. Its system design supports multi-adapter drafting and adapter-version switching without recomputing prefix KV caches. Experiments on three models and several domains show improved acceptance length and lower time per output token than the evaluated baselines.

Critical strengths

System-level contribution. The combination of domain-specific LoRA adapters, in-flight updates based on verification feedback, and efficient adapter management is a meaningful systems contribution. In particular, supporting multiple adapters within a decode batch and switching adapter versions without recomputing prefix KV caches addresses practical challenges in deploying online adaptation.

The paper addresses the distribution mismatch between draft models and online inference traffic. The reported results show improvements in acceptance length and time per output token across the evaluated models and workloads.

Critical weaknesses

1)The motivation and presentation of LoRA adapters need improvement. The introduction does not sufficiently motivate the introduction of low-rank adapters in the context of closely related work. The use of enumerated statements and a “first, second, third” structure makes the discussion feel more like a list of design choices than a progressively developed research argument. The introduction would benefit from a more coherent, step-by-step narrative that derives the proposed design from the limitations of prior work. The key differences from closely related approaches should also be explicitly stated in the introduction, rather than left primarily to the related-work section. 2)The paper does not provide comprehensive empirical comparisons with several closely related online-adaptation systems, including Aurora and EvoSpec. 3)SERVAL appears to combine several existing ideas, including online adaptation, LoRA, domain-specific specialization, and serving-time model updates. While their integration may constitute a useful systems contribution, the paper should clearly distinguish the components that are technically novel from those adopted from prior work.

Questions for authors

The authors should evaluate the robustness of SERVAL to domain-routing errors and discuss how the system would handle requests that span multiple domains or do not belong to any predefined domain.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer e69u

We have revised Section 1 so that it derives the design step by step and separates new from adopted components (W1, W3).

W1. Introduction and LoRA. The revised Section 1 derives the design from what a running request needs in order to benefit from its own feedback: new draft weights must reach it mid-request without invalidating its draft KV cache. OSD and Aurora install new weights only between requests, and AdaFlash reloads one global drafter for all running requests. In DFlash the cached keys and values are projections of the target's hidden states, so updating only the query and output projections keeps every prefix cache valid. Serving several domains and versions in one batch then needs small updates: a rank-32 LoRA on those two projections is 7.8 MB per version, against about 1.8 GB for full training with the key and value projections frozen.

W2. Aurora and EvoSpec. We have not evaluated either system end to end: EvoSpec has no public code, and Aurora's released code trains an EAGLE-3 draft, so a comparison would also differ in draft architecture. Inside SERVAL, two ablations isolate Aurora's choices. The first applies Aurora's version timing at SERVAL's update rate, so that only the timing differs: one global adapter whose new versions reach only requests admitted afterwards. Its AL is 3.0% below that of the same adapter updated in flight under abrupt domain shifts (3 runs), and 0.9% below on stationary traffic. The second delays feedback to the end of each request, as Aurora does, and lowers AL by 0.8% (stationary stream). SERVAL-shared follows EvoSpec's design of a single continually trained LoRA but is not a reproduction of EvoSpec. Per-domain adapters match its AL in aggregate and exceed it by 0.9% when traffic returns to an earlier domain.

W3. New and adopted. Adopted: training the draft from verification feedback and specializing it by domain (OSD), LoRA for online draft training (EvoSpec), loading new draft weights during serving (Aurora, AdaFlash), and batching many LoRA adapters in one forward pass (S-LoRA, Punica). New: (1) each running request switches to a new version of its adapter at its next decode step, independently of other requests, while its prefix KV cache stays valid; (2) several continuously trained draft adapters, and several versions of each, in one decode batch; (3) one trainer that updates several adapters concurrently from the hidden states the server exports, without running the target's backbone. The closest system is AdaFlash (Section 5), whose paper describes hot-reloading one global drafter between scheduling steps: its new weights also reach running requests, but all requests share one drafter, so it has neither (1) nor (2).

Q. Routing errors, multi-domain and unknown requests. On a 1,200-request stream (2 runs), misrouting 10% and 30% of requests lowers AL by 0.5% and 1.1%, and the shared adapter, which needs no labels, is within 0.3% of oracle routing in AL. Requests outside every domain, or spanning several, can be served by the shared adapter. We have not measured per-domain adapters with such a fallback, or requests that span domains.

We would be grateful if you would reassess the paper with these changes.

Thanks for your response. The authors have addressed some of my main concerns. Considering the overall technical contributions of the paper, I will maintain my original score.

Reviewer_PBdTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SERVAL continuously adapts draft models using verification feedback from running requests and allows in-flight adapter updates without recomputing prefix KV caches. Experiments show consistent improvements in acceptance length and TPOT.

Critical strengths
  • Sound systems design: The KV-cache-preserving adapter updates are a useful architectural contribution, and the combination of multi-adapter batching, fused Triton kernels, and asynchronous training is technically well motivated.

  • Consistent results: SERVAL improves acceptance length and TPOT across three target models and multiple workloads, with little overhead.

  • Useful ablations and analysis, and good reproducibility details.

Critical weaknesses
  • Unaccounted training cost: SERVAL requires a separate trainer using as many GPUs as the serving process, but the efficiency comparisons only fix the serving GPU budget. Without throughput/comparisons under equal total hardware resources, it is unclear whether the latency gains justify the additional compute. This is the biggest weakness to me.

  • Limited incremental gains from online adaptation: Much of the improvement over DFlash comes from the general adapter trained offline on 11,200 requests. A compute-matched offline fine-tuning baseline is missing, which is necessary to know if the continual learning is the important piece. This is also a major weakness for me.

  • Unrealistic traffic changes: The experiments use stationary mixed-domain streams with known domain labels, despite the paper's motivation around continuously changing traffic. Experiments with abrupt distribution shifts, shorter requests, incorrect domain routing, or unknown domains is necessary. This is also a major concern, given how the paper is framed.

  • Small benefits from the design choices: Per-domain adapters provide only marginal aggregate improvements over a shared adapter, and allowing running requests to switch adapter versions improves acceptance length by approximately 1% in the reported ablation. These results make the additional complexity seem less worth it.

  • Confounded comparison with OSD: The OSD baseline differs from SERVAL in feedback timing, loss function, and update schedule, so the performance difference cannot be attributed solely to in-flight training.

Questions for authors
  • How does SERVAL compare against DFlash under the same total GPU budget, particularly in throughput per GPU and latency under different loads?

  • How does SERVAL compare with a draft fine-tuned offline on similar target-generated traffic using a matched training budget?

  • How does SERVAL perform under non-stationary traffic, especially when the domain distribution changes abruptly during serving?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer PBdT

W1 and Q1. Training cost, and DFlash at the same total budget.

"Without throughput/comparisons under equal total hardware resources, it is unclear whether the latency gains justify the additional compute."

We agree that the submission did not count the trainer GPUs. At a fixed six-GPU budget on Qwen3-8B (Table A), two SERVAL servers with their trainers have 18-20% lower mean TPOT than three DFlash replicas at both arrival rates, and more of their requests meet the SLO. General-frozen replicas are faster than both, and replicas with frozen SERVAL adapters, which need no trainer, are the fastest. Table B gives output tokens/s per GPU with the trainer GPU counted.

W2 and Q2. Offline baseline.

"Much of the improvement over DFlash comes from the general adapter trained offline on 11,200 requests."

Yes. On five of the paper's six streams it accounts for 63-89% of SERVAL's TPOT reduction over DFlash. As an offline baseline matched to the 600-request stream, we trained per-domain adapters before deployment on 200 other requests per domain, with SERVAL's trainer and loss, and served them frozen. Their AL is 5.0% above general-frozen's. SERVAL's AL is 1.2% higher than theirs (3 runs) and its throughput the same (within 0.5%), with the same number of serving GPUs. In this matched comparison, most of the gain over general-frozen comes from target-generated domain data, which online training collects from live traffic instead of in advance.

This is useful when suitable training data or a matched adapter is not available before deployment.

  • A target without a matched draft. On DeepSeek-R1-0528-Qwen3-8B and Klear-Reasoner-8B, both post-trained from the Qwen3-8B base model, the general adapter's AL is 21% and 33% lower than on Qwen3-8B. Over its first 600 requests SERVAL, with one shared adapter, is 17% and 43% above general-frozen in AL and 13% and 29% lower in mean TPOT, with the same number of serving GPUs (2 runs and 1 run). Frozen after these 600 requests and served without a trainer, its adapter keeps AL gains of 13% and 43% and has 12% and 30% lower mean TPOT on unseen prompts.
  • No general adapter. When SERVAL starts from DFlash, its AL over the first 600 requests is 3.0-4.9% above general-frozen's on Qwen3-8B, 4.4-5.6% on gpt-oss-120b and 0.5-0.6% on MiniMax-M2.7 (3, 2 and 2 runs), without the 11,200 offline requests behind the general adapter.

W3 and Q3. Non-stationary traffic, short requests, routing, unknown domains.

  • Abrupt shifts (200 code, then 200 finance, then 200 medical requests, 3 runs): SERVAL is 6.3% above general-frozen in AL (5.1% in throughput).
  • Short requests (thinking mode off, the same 600 prompts, median about 820 output tokens, 3 runs): SERVAL's AL is 3.6% above general-frozen's over the requests that end before the length cap, and 10.9% over all requests: the 1-8 capped requests per run carry up to 30% of the output tokens. Its mean TPOT is between 0.4% lower and 3.9% higher than general-frozen's.
  • Routing errors (1,200-request stream, 2 runs): misrouting 10% and 30% of requests lowers AL by 0.5% and 1.1%.
  • Unknown domains: the shared adapter needs no labels, but we have not evaluated per-domain adapters with a shared fallback.

W4. Complexity.

"These results make the additional complexity seem less worth it."

In-flight version switching raises AL by 3.0% under abrupt shifts (SERVAL-shared). Per-domain adapters match SERVAL-shared in aggregate AL and exceed it by 0.9% when traffic returns to an earlier domain. The measured overhead of batching per-domain adapters is 0.001-0.016 ms per decode step.

W5. OSD comparison.

"the performance difference cannot be attributed solely to in-flight training."

We agree. On the same stream (2 runs each), the AL of OSD-analog, the paper's OSD-style variant, is 4.5% below SERVAL's. OSD's update schedule alone lowers AL by 3.7% and its reject-only loss alone by 2.7%, and the two effects are not additive. A request-end feedback ablation, which tests the third difference by delaying feedback to the end of each request, lowers AL by 0.8% on a stationary stream. Most of the gap therefore comes from OSD's update schedule and loss, not from feedback timing. The revision describes OSD-analog as differing in all three respects and reports this decomposition.

Thank you for the detailed rebuttal and additional experiments.

"General-frozen replicas are faster than both, and replicas with frozen SERVAL adapters, which need no trainer, are the fastest."

The fixed-budget experiments address my main concern about unaccounted GPU costs. However, the results also show that online SERVAL is less resource-efficient than frozen adapted models. Its practical advantage therefore appears limited to scenarios where a suitable adapter is unavailable.

"SERVAL's AL is 1.2% higher than theirs (3 runs) and its throughput the same (within 0.5%)."

The matched offline baseline is a valuable addition, but it reinforces my concern that most gains come from domain-specific training data rather than in-flight adaptation itself. The experiments on targets without matching drafts provide a stronger motivation, but the resource-normalized benefits in that setting is not clear.

"Its mean TPOT is between 0.4% lower and 3.9% higher than general-frozen's."

The non-stationary and routing experiments are helpful, but the lack of meaningful improvement on shorter requests limits the generality of the efficiency claims. Unknown-domain and mixed-domain serving also remain untested.

"In-flight version switching raises AL by 3.0% under abrupt shifts (SERVAL-shared)."

This strengthens the justification for in-flight switching. Nevertheless, per-domain adapters still provide little aggregate improvement over a shared adapter. The technical contribution is clear, but the necessity of the additional system complexity remains less convincing.

"Most of the gap therefore comes from OSD's update schedule and loss, not from feedback timing."

This decomposition satisfactorily addresses my concern about the confounded OSD comparison. I appreciate the clarification.

Overall, the rebuttal has improved my assessment. My remaining concern is that the practical benefits of continuous adaptation over simpler frozen-adapter alternatives are not yet sufficiently compelling across realistic serving scenarios.

I am inclined to raise my rating to a 2.

Reviewer_KUtPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SERVAL, a speculative-decoding serving system that continuously adapts the draft model to online traffic. SERVAL maintains persistent per-domain rank-32 LoRA adapters and updates them asynchronously using feedback already produced by target-model verification. Requests using different adapters can remain in the same continuous decode batch through fused multi-adapter kernels. The adapters are applied only to the query and output projections of DFlash. Since DFlash constructs its prefix keys and values from target-model hidden states through unchanged key and value projections, running requests can switch to newly trained adapter versions without recomputing their prefix KV caches. Experiments on Qwen3-8B, gpt-oss-120b, and MiniMax-M2.7 show improved acceptance length and lower mean TPOT over frozen DFlash and several online-adaptation variants.

Critical strengths
  1. SERVAL addresses an important practical problem. A fixed draft model can lose acceptance under domain and request-level distribution shifts. Adapting the draft during serving is a natural way to use the supervision already produced by target verification.

  2. The system design is thoughtful and technically interesting. In particular, restricting adaptation to the query and output projections enables in-flight adapter-version changes without invalidating the prefix KV cache. The fused multi-adapter implementation also allows requests with different domain adapters to share one continuous batch.

  3. The empirical evaluation is relatively broad. The paper evaluates three substantially different model systems, multiple mixed-domain streams and benchmarks, and controlled variants for persistence, domain specialization, and in-request updating. The improvements in acceptance length and mean TPOT are consistent across the evaluated settings.

Critical weaknesses

The main serving-efficiency claim is not established under a fixed total GPU budget: SERVAL uses as many additional H200 GPUs for training as for serving: 2 trainer GPUs for the 2-GPU Qwen3-8B server and 4 trainer GPUs for each 4-GPU gpt-oss-120b or MiniMax-M2.7 server. The paper compares methods on the same number of serving GPUs, but excludes the trainer GPUs from the resource budget.

This is particularly concerning because the Qwen3-8B adapter has only 3.9M trainable parameters. The trainer also performs draft replay and backpropagation and stores a frozen draft and target LM head, so some additional compute is clearly required. However, matching the server’s tensor-parallel degree appears to be an implementation choice rather than a demonstrated resource requirement. The paper does not report trainer utilization, the minimum trainer resources required to keep up with serving, or whether one trainer can support multiple serving replicas.

More importantly, the additional H200 GPUs could instead be used to deploy another frozen serving replica. Under the same total budget, the relevant comparison for Qwen3-8B is therefore:

  • one 2-GPU SERVAL server plus two trainer GPUs, versus
  • two independent 2-GPU DFlash or general-adapter servers.

A second serving replica could potentially nearly double aggregate capacity, while the incremental benefit of online adaptation over the frozen general adapter is much smaller. On the code+finance+medical stream, TPOT changes from:

  • 2.24 to 1.99 ms for Qwen3-8B, approximately an 11% reduction;
  • 3.83 to 3.65 ms for gpt-oss-120b, approximately a 5% reduction;
  • 10.09 to 9.80 ms for MiniMax-M2.7, approximately a 3% reduction.

The larger headline improvements relative to DFlash include the benefit of the previously trained general adapter and therefore overstate the marginal value of in-flight training itself.

The current results may still establish a single-request TPOT benefit, which an additional replica does not necessarily provide. However, the paper does not report request-level tail latency, maximum stable throughput, SLO goodput, or per-GPU efficiency. Consequently, it is impossible to determine whether SERVAL is a better use of the additional GPUs than straightforward serving replication.

Questions for authors
  1. Why must the trainer use the same tensor-parallel degree and number of H200 GPUs as the server? Can the authors report trainer utilization and evaluate 1-, 2-, and 4-GPU trainer configurations, including update lag and scheduler blocking?

  2. Under a fixed total GPU budget, how does SERVAL compare with using the trainer GPUs as additional DFlash or frozen-general-adapter serving replicas? Please report aggregate requests/s, output tokens/s, p95/p99 latency, and SLO goodput.

  3. Can one trainer be shared across multiple serving replicas? If so, how many replicas can it support before update lag or scheduler blocking materially affects serving performance?

  4. What is the marginal benefit of in-flight adaptation after excluding the offline general adapter? Please separately report the resource-normalized improvement from general-frozen to SERVAL rather than attributing the entire DFlash-to-SERVAL gain to online adaptation.

  5. Can the authors report cost-normalized metrics, including requests/s/GPU, output tokens/s/GPU and SLO goodput/GPU?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer KUtP

"The main serving-efficiency claim is not established under a fixed total GPU budget ... excludes the trainer GPUs from the resource budget."

We agree. Counting the trainer GPU, general-frozen produces more output tokens/s per GPU than SERVAL in all 12 model-benchmark pairs (Table B) and has lower mean TPOT at six GPUs on Qwen3-8B (Table A). At six GPUs, SERVAL's mean TPOT is 18-20% below that of DFlash replicas, and replicas with frozen SERVAL adapters, which need no trainer, are the fastest.

Q1. Trainer size, utilization, update lag, scheduler blocking. The trainer need not match the server's tensor-parallel degree (general response). We measure the trainer's busy time as the share of wall-clock time it spends in training steps and in publishing versions. With the trainer of the submission (2 GPUs for Qwen3-8B, 4 for the TP4 models), it is 42-43% (Qwen3-8B), 36-40% (gpt-oss-120b) and 19-20% (MiniMax-M2.7). With one trainer GPU it is 27-49% across the 12 model-benchmark pairs; the limit is its host thread (Q3). Serving waits on the one-GPU trainer for 0-1.8% of each run. The update lag, from when the server exports a feedback message to when an adapter version trained on it starts serving, is logged in decode steps. Converted with the server's mean decode-step time, its 90th percentile is below 1 s in 9 of the 12 pairs; Appendix E of the revised PDF gives the other three.

Q2 and Q5. Fixed budget, per-GPU metrics. Tables A and B give these metrics. The offered request rates per GPU in Table A are 0.11 and 0.19 requests/s. At 1.125 requests/s (Table A):

DFlashgeneral-frozenfrozen SERVAL adaptersSERVAL
output tokens/s (six GPUs)6,5046,8316,9626,987
end-to-end p95 / p99 (s)93 / 11047 / 5242 / 4751 / 59
TPOT p95 / p99 (ms)4.13 / 4.592.69 / 2.942.44 / 2.743.12 / 3.54
SLO goodput per GPU (tokens/s)3701,1371,1601,136

Q3. Sharing one trainer. We ran one trainer GPU with two fully loaded Qwen3-8B servers (one GPU and 32 concurrent requests each, 2 runs), which together produce about 2,400 messages/s. The trainer is limited by its single Python host thread, which processes about 1,710 messages/s. Mean TPOT rises from 5.04 ms with one trainer GPU per server to 7.28 ms (+44%), and each server blocks on the trainer's input queue for 240-254 s per run. With the current trainer, each fully loaded server therefore needs its own trainer GPU.

Q4. Marginal benefit over general-frozen. With the trainer GPU counted, SERVAL's output tokens/s per GPU is below general-frozen's (Table B). With the same number of serving GPUs, its mean TPOT is 3-5% lower than general-frozen's on Qwen3-8B and between 2.2% lower and 5.5% higher on the TP4 models. The benefit is larger where the general adapter fits the target less well. On the two targets it was not trained for, SERVAL's mean TPOT is 13% and 29% lower over the first 600 requests, with the same number of serving GPUs (reply to PBdT, W2). When SERVAL starts from DFlash without the general adapter, its AL over the first 600 requests is 0.5-5.6% above general-frozen's on the three models.

We would be grateful if you would reassess the paper with the revised claim.

Thank you for the detailed and candid response. I appreciate the authors’ willingness to acknowledge the limitations of the original resource accounting and to provide additional experiments, including results that do not favor SERVAL. These measurements substantially clarify the trade-offs and address several of my questions.

However, I will retain my original score. My main concern was not simply the absence of a fixed-budget comparison, but whether online adaptation is a better use of the additional GPUs than serving replication. The new results do not establish this advantage: when trainer GPUs are included, general-frozen achieves higher output tokens/s per GPU in all 12 model–benchmark pairs. In the reported six-GPU comparison, SERVAL also has worse tail latency than both general-frozen and frozen SERVAL adapters, with essentially unchanged SLO goodput relative to general-frozen. The small increase in aggregate output throughput at this offered load does not demonstrate a higher maximum sustainable serving capacity.

The comparison with frozen SERVAL adapters is particularly informative. It suggests that much of the benefit can be retained without a continuously running trainer. Similarly, the marginal TPOT improvement over general-frozen is modest or inconsistent in the main evaluated settings, even before accounting for training resources. The larger gains on targets for which the general adapter was not trained are useful evidence for adaptation under mismatch, but they are reported with equal serving resources rather than equal total resources, so they do not resolve the efficiency concern.

The one-GPU trainer experiments address my concern about the original trainer configuration. However, the sharing experiment also reveals a practical scaling limitation of the current implementation: one trainer cannot keep up with two fully loaded servers without substantial blocking and a 44% increase in mean TPOT. This limits the opportunity to amortize training costs across serving replicas.

I continue to find the cache-compatible adapter updates and fused multi-adapter serving design technically interesting. A revised claim focused on these capabilities and on adaptation under distribution mismatch would be better supported by the evidence. Nevertheless, narrowing the claim does not establish the resource-normalized serving advantage that motivated my original concern. For that reason, while I appreciate the stronger evaluation and the authors’ transparency, I do not find sufficient grounds to raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We agree with Reviewers KUtP and PBdT that our efficiency comparisons left the trainer's GPUs out of the budget. The new experiments count every trainer GPU.

One trainer GPU per server is enough in the tested configurations. The trainer runs the draft and the target's LM head on hidden states the server exports, not the target's backbone, so the GPUs it needs depend on the feedback rate rather than on the server's tensor-parallel degree. In the submission, the trainer mirrors the server's sharding and is busy 19-43% of the time. During the rebuttal we consolidated it onto one GPU, with the same loss and hyperparameters (reply to KUtP, Q1 and Q3).

Table A. Six GPUs under load (Qwen3-8B; every configuration serves the same 600 code, finance and medical prompts with the same Poisson arrival times; one run each). Entries are mean / p99 time per output token (TPOT) in ms and the share of requests meeting the SLO (first token ≤ 2 s including queueing, TPOT ≤ 3 ms).

requests/sDFlash (6 serving GPUs)general-frozen (6 serving GPUs)frozen SERVAL adapters (6 serving GPUs)SERVAL (4 serving + 2 trainer GPUs)
0.6752.16 / 3.48, 89.5%1.71 / 2.55, 100%1.51 / 2.31, 100%1.78 / 2.66, 99.8%
1.1252.79 / 4.59, 61.8%1.93 / 2.94, 99.3%1.68 / 2.74, 100%2.24 / 3.54, 91.5%

Table B. Output tokens/s per GPU, trainer GPU counted (32 concurrent requests per server, one run per pair).

modelbenchmarkSERVALgeneral-frozenDFlash
Qwen3-8BFinQA283940233674
AIME206429961871
LCB-hard171224051498
code_contests170824881534
gpt-oss-120bFinQA125115721695
AIME100612961153
LCB-hard7941026810
code_contests8801024827
MiniMax-M2.7FinQA678837713
AIME491601545
LCB-hard392494450
code_contests374482442

Counting the trainer GPU, SERVAL produces more output tokens/s per GPU than DFlash in 4 of the 12 model-benchmark pairs (3 of 4 on Qwen3-8B) and 14-31% fewer than general-frozen in all 12.

The trainer can be removed after adaptation. SERVAL's per-domain adapters, frozen after 4,000 requests per domain and served without a trainer, have 12-13% lower mean TPOT than the general-frozen replicas at each rate of Table A, on prompts that are not among their training requests. Training them takes about 3.6 trainer GPU-hours.

Online training helps most where the initial draft does not match the target. On two targets the general adapter was not trained for, SERVAL raises acceptance length (AL) by 17% and 43% and lowers mean TPOT by 13% and 29% relative to general-frozen over its first 600 requests, with the same number of serving GPUs. Details and the offline baseline are in the reply to PBdT (W2), and non-stationary traffic, short requests and routing errors in the replies to PBdT and e69u.

The revised PDF (changes in blue) narrows the claim as follows. SERVAL trains the draft from live traffic while requests run, without invalidating their caches, and serves several adapters and versions in one batch. With one additional trainer GPU per server in the tested configurations, SERVAL has lower mean TPOT than DFlash in 11 of the 12 model-benchmark pairs with the same number of serving GPUs, and at a fixed six-GPU budget on Qwen3-8B it is faster than DFlash replicas and slower than replicas serving frozen adapters, including the adapters it trained.

It also states in Section 1 the differences from OSD, Aurora and AdaFlash, adds a limitations paragraph, and reports these results in Section 4.4 and Appendices E and F (Tables A and B are Tables 16 and 15).

We thank all three reviewers and are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.