acceptodds
Under review as a conference paper at ICLR 2027

InvisibleLoRA: Fast Multi-LoRA Serving Without Per-Token LoRA Decoding

Abstract

Low-rank adaptation (LoRA) specializes a single base model into many fine-tuned variants, and production systems now serve hundreds of adapters on a shared base model. Serving many adapters concurrently, however, introduces substantial inference overhead. Because requests are partitioned across active adapters, each low-rank computation covers only a small fraction of the batch and runs as a small, GPU-inefficient operation. Consequently, per-step adapter overhead grows with the number of active adapters even though each adapter holds only a small fraction of the base-model parameters. Existing multi-LoRA systems reduce the cost of each adapter execution through specialized kernels and scheduling, but every active adapter still executes at every token. We propose InvisibleLoRA, a speculative decoding method that uses the shared base model as the common drafter for every served adapter. A LoRA-off forward pass proposes the next token for every request in the batch, and a single LoRA-on forward pass verifies all proposals. InvisibleLoRA therefore applies adapter computation once per decoding cycle rather than once per token, while speculative verification preserves the output distribution of standard multi-LoRA decoding. InvisibleLoRA further introduces a hybrid KV cache: the LoRA-off base model writes speculative KV entries into the KV cache of the LoRA-on target model, and the target model overwrites the speculative entries during verification, so the drafter needs no separate KV cache and reads LoRA-conditioned KV states for the committed prefix. InvisibleLoRA also adopts a run-time gate that skips drafting on steps where the adapter cost cannot repay a draft pass. InvisibleLoRA reaches median draft acceptance rates of 92–97% across 456 publicly available LoRA adapters on five base models. Implemented in vLLM, InvisibleLoRA delivers lossless acceleration over an optimized multi-LoRA baseline, improving throughput by up to 1.84× and reducing per-token latency by up to 48%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, multi-LoRA serving, LoRA, LLM inference, efficient inference, KV cache, lossless acceleration, serving systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6g5zOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

overall score raised from 2 (weak reject) to 3 (weak accept) after the rebuttal


The paper proposes a new solution for speculative decoding specifically for multi-LoRA serving. The core problem is that during decoding each adapter only acts on a single or few tokens and thus runs very inefficiently. In particular when many adapters are used, this can dominate latency over the base models backbone forward pass. The proposed solution uses the base model as a shared drafter for all adapters. This is a neat and elegant solution and seems of practical value. The idea does not require much theory and to me is very easy to understand mostly based on the ideas of LoRA and Speculative Decoding. The authors ablate their results on many models and adapters. A nice insight is that the backbone model can use the existing KV cache of the adapters and thus does not introduce any further memory overheads (that a seperate drafter would ad). I think the experimental validation might not be the most representative, however. All adapters and the acceptance rates are evaluated on generic tasks. In a realistic scenario, each adapter would be run on the task it was fine-tuned for. To me it is plausible that for the generic tasks, the adapters do not add much and hence the acceptance rates reported in the paper are inflated.

Overall imo opinion this is a nice idea that has practical use. Tough I'd say it is also not groundbreaking in any way.

I want to state that the main part of the paper reads well and I was very happy to read it. However, the appendix reads LLM generated with many LLM-ish phrases like "load-bearing", "the two arms".... This is very tiring to read as a human, hard to extract value from it, and i was a bit disappointed when I got there...

In the current state, I am reluctant to accept the paper. but if the additional experiments can be provided, code is released and the appendix gets a human rewrite, I am happy to support acceptance

Critical strengths

The paper very clearly formulates a crisp problem, the decoding overhead in multi LoRA serving, and provides a simple and elegant solution to it. The theory is simple and mostly follows from previous work. The main paper is cleanly written and I would attest that it includes honest human effort in writing.

Besides the runtime speedups, the method also circumvents the need for additional KV cache (or weights) of a drafter model, thus not adding HBM pressure. I think the authors could even call out one more advantage: Invisible Draft does not need any additional prefil of the drafter model. Other speculative decoding methods need to fill the drafters KV cache and thus add TTFT (time-to-first-token) overheads. InvisbileLoRA does not, which is great in my opinion.

Critical weaknesses
  • I think the experiemts for the acceptance rate are not representative. In practice, each LoRA adapter is used on a different task, the one which it is fine-tuned for. Thus, I'd think that on those tasks the difference in the distribution vs the baseline model is larger, than on a generic tasks on which the base model is already good. I think the results are those overly optimistic. I would encourage the authors to try to run experiments as well where each adapter is evaluated on its own task and report those as well.
  • code is not provided but the papers value hinges a lot on it in my opinion. Thus acceptance should only be given if the code is provided. i presume the method will be upstreamed to vLLM once published, but it would be nice as a reviewer to take a quick look and potentially play with it for a bit.
  • the appendix seems to AI-written to me and is not very readable. On the positive side, the authors disclose that they used AI for writing draft text -- but in the appendix it could have used more editing efforts.
Questions for authors
  • How much of the appendix did you write yourself vs how much AI generated is it?
  • Can you provide the code?
  • Can you please run some experiements that evaluate the adapter on its task? In particular it would be nice to see how the acceptance rate behaves as the accuracy difference between base model and LoRA model vary on that task. I think this could be done with a single family of adapters that vary in size or quality of their training.
  • I think it could be a bit clearer where the latency from the LoRA adapters comes from. I presume it is memory-bw bound, or is it actually launch overheads? Surely it is not the actual compute, but it would be nice if this could be clarified and analyzed separately
8 comments
AuthorsOpenReview ↗

Response to Reviewer 6g5z (1/5): W1 and Q3

W1. I think the experiments for the acceptance rate are not representative. … I would encourage the authors to try to run experiments as well where each adapter is evaluated on its own task and report those as well.

Q3. Can you please run some experiments that evaluate the adapter on its task? In particular it would be nice to see how the acceptance rate behaves as the accuracy difference between base model and LoRA model vary on that task. I think this could be done with a single family of adapters that vary in size or quality of their training.

Response. We thank the reviewer for this insightful comment. We agree that each adapter should also be evaluated on the task it was trained for, since that is where it is mainly used. In short, acceptance stays high on the own task of each adapter, and it drops when an adapter writes differently from the base model, not when it becomes more accurate.


Acceptance on each adapter's own task.

We had measured this, evaluating every math, code and chat adapter only on prompts of its own domain, but the result was only in an appendix of the submission, now Table 8 of Appendix D.

Table R1.1. Average acceptance on all prompts and on the own domain of each adapter.

promptsLlama-3.1-8BQwen3-8BMistral-7BQwen3-14BQwen3-32B
all prompts96.9%97.1%92.4%97.4%95.7%
own domain only95.5%97.4%89.0%96.6%92.1%

As you expected, acceptance is slightly lower on the own tasks, but it stays high. The average acceptance falls by at most 3.6 points, as shown in Table R1.1.


What acceptance measures.

Acceptance and accuracy measure different things. Acceptance is the share of tokens drafted by the base model that the adapter accepts, so it is high when the adapter would write almost the same text as the base model. Accuracy checks only the final answer. An adapter can therefore fix an answer while keeping the rest of the text, or rewrite a whole solution without fixing any answer. The three measurements below test which of the two lowers acceptance.


Acceptance against the accuracy gap.

Following Q3, we scored 24 public adapters, 12 for math and 12 from other domains, on the same 200 GSM8K problems, on which the base model scores 0.89.

Table R1.2. Median acceptance by the accuracy gap to the base model on 200 GSM8K problems.

accuracy gap to the base modeladaptersmedian acceptance
within 0.05799.2%
0.05 or more below1795.0%

Adapters with a larger accuracy gap have only slightly lower acceptance. As shown in Table R1.2, the median acceptance falls from 99.2% to 95.0%, about four points, when the accuracy of an adapter is 0.05 or more below the base model.


Following adapters through training.

We followed three adapter families over five checkpoints on the same GSM8K problems: a public GSM8K training run, and two runs that we trained on the same MATH problems with the same settings. Our two runs differ only in what the adapter learns to write: solutions written by people, or correct solutions written by the base model itself.

Table R1.3. Acceptance on GSM8K over five checkpoints of three adapter families.

checkpointpublic GSM8K runours, solutions by peopleours, solutions by the base model
1st94.7%94.1%98.5%
2nd92.2%92.2%98.1%
3rd91.4%90.9%97.9%
4th90.0%87.9%97.6%
5th88.6%87.1%97.1%

Acceptance falls mainly when the adapter learns to write differently from the base model. Learning solutions written by people lowers acceptance by 1.8 points per checkpoint, and the public run by 1.5 points, whereas learning the solutions of the base model lowers it by only 0.3 points, as shown in Table R1.3. Even after five checkpoints, the lowest acceptance is 87%.


An adapter that answers more questions correctly.

None of the three families clearly beats the base model in accuracy under our scoring, so we also evaluated a public math adapter reported to do so, following its model card: all 1,319 GSM8K test problems, scored by exact match of an answer tag.

An adapter can become more accurate and still keep acceptance high. Under its model card, this adapter raises accuracy from 0.647 to 0.748, and its acceptance is 99.5%, as high as the adapters closest to the base model. Under the scoring used above, it scores 0.860 against 0.890, and it is the only reported gain we could test.


We are grateful for this suggestion, which led us to a clearer understanding of how acceptance and accuracy are related. The revision states the own-task result in Section 4.2, keeps the full table as Table 8 of Appendix D, and adds the accuracy and checkpoint results as Tables 10 and 11 of Appendix D.

AuthorsOpenReview ↗

Response to Reviewer 6g5z (2/5): W2 and Q2

W2. code is not provided but the papers value hinges a lot on it in my opinion. Thus acceptance should only be given if the code is provided. I presume the method will be upstreamed to vLLM once published, but it would be nice as a reviewer to take a quick look and potentially play with it for a bit.

Q2. Can you provide the code?

Response. We thank the reviewer for stressing this. We agree that the value of the method depends on the code. The code is now attached to this submission as supplementary material, in the archive invisiblelora.zip.

The archive is ready for a quick look and a short run. It contains the patch for vLLM 0.11.2, the scripts that reproduce the figures and the manifests of the adapter pools. On one H100 GPU, the quick start reproduces the Llama-3.1-8B result with 64 adapters in about ten minutes.

AuthorsOpenReview ↗

Response to Reviewer 6g5z (3/5): W3 and Q1

W3. the appendix seems to AI-written to me and is not very readable. On the positive side, the authors disclose that they used AI for writing draft text – but in the appendix it could have used more editing efforts.

Q1. How much of the appendix did you write yourself vs how much AI Generated is it?

Response. We thank the reviewer for the frank feedback and for the kind words on the main text. You are right about the appendix, and we apologize that it was hard to read. As disclosed in our LLM usage statement, we used an LLM only to turn our experiment notes and logs into draft text, and we then revised that text ourselves, as we did for the main text. When we merged the final versions before submission, however, the unrevised draft of the whole appendix replaced our revised text, and we did not catch this before the deadline. For the revision, we first restored our revised text and then went through the whole appendix again ourselves, paragraph by paragraph. We read every sentence and rewrote it or kept it, replaced the phrases you point out, such as "load-bearing" and "the two arms", with plain wording, turned sentence-like paragraph headings into plain titles, and checked every number against the logs.

Because the appendix was hard to read, we also brought together material that had been scattered across sections and reorganized the appendix substantially.


We thank the reviewer again for this comment, which led us to go through the appendix once more and organize it more systematically.

Reviewer_8f66OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents InvisibleLoRA, a system that optimizes for multi-lora serving. The core observation that the authors have is that the LoRA execution time is proportional to the number of unique adapters. Hence, this paper uses base model as the draft model, and loRA patched model as verifier model to reduce the LoRA IO loading times. This paper also proposes a method that results in higher draft acceptance rate.

Critical strengths

This paper proposes amortizing the LoRA loading cost across different LLM invocations. This idea is interesting, and in theory, it should reduce decode execution time.

Critical weaknesses

Most of this paper's weakness are from its online evaluation. The current evaluation is not enough to show that the system works in standard serving scenarios.

  1. The evaluation relies solely on artificial workloads, including fixed input/output token counts and a mock arrival pattern. Many publicly available traces capture both token and arrival distributions and could have been used instead.
  2. The request-per-second (RPS) rate is not reported in the serving evaluation. Without it, the reported numbers are difficult to interpret.
  3. The testbed is poorly described, and the actual serving deployment parameters are not stated in the paper.
  4. Figure 13 shows that the system queued incoming requests. However, this queueing results from an artificial cap on the maximum batch size rather than from actual system load. It reflects an artificial system upper bound, not real demand.
Questions for authors
  1. How are the serving parameters configured? For example, what are the chunk size and maximum batch size? Online serving systems are extremely sensitive to these values. In addition, is prefix caching, PD disaggregation enabled in this setting, or is it still dealing with single turn conversation?
  2. What happens if the artificial trace is replaced with real traces that have much longer input and output lengths than 128/256?
  3. Different LoRAs adapt different parts of the transformer. How did you choose which LoRA to use, and how would the system be affected if I chose a LoRA that applies only to the attention module, or one that applies only to the FFN module?
  4. What is the acceptance length distribution for each dataset?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 8f66 (1/6): Summary, W1 and Q2

We thank the reviewer for the careful summary and for the positive assessment of the idea. As you note, the paper amortizes the cost of LoRA across forward passes. We would like to add one detail on where that cost comes from, since the answers below build on this detail: the adapters stay resident in GPU memory, and the cost is the LoRA kernels that run at every decode step, which take 78% of a step with 64 adapters. InvisibleLoRA runs those kernels once per draft-verify cycle, and each cycle commits about two tokens.

W1. The evaluation relies solely on artificial workloads with fixed token counts and a mock arrival pattern, although public traces could have been used.

Q2. What happens with real traces whose inputs and outputs are much longer than 128/256?

Response. We appreciate this valuable suggestion. We fully agree that fixed lengths and synthetic arrivals are not enough to show that the system works in standard serving scenarios. Following your suggestion, we replayed a public LLM serving trace and measured longer inputs and outputs.


Replay of a public serving trace.

We replayed the Azure LLM inference conversation trace (Patel et al., 2024), which supplies the arrival gaps and the input and output lengths of real requests, and rescaled only the arrival rate to the target load. The median input is about 1,000 tokens and the median output about 200 tokens, and outputs are not capped at 256, reaching up to 1,000 tokens.

Table R1.1. Azure LLM trace replay on Llama-3.1-8B with 64 resident adapters and uniform adapter choice, baseline → InvisibleLoRA.

loadbatch cap 64: throughputspeedupSLO metbatch cap 256: speedupSLO met
4 req/s719 → 804 tok/s1.12×95% → 100%1.09×77% → 100%
5 req/s845 → 990 tok/s1.17×71% → 100%1.10×55% → 100%
6 req/s947 → 1182 tok/s1.25×48% → 100%1.11×43% → 99%
7 req/s1009 → 1284 tok/s1.27×40% → 98%1.10×36% → 98%
8 req/s1064 → 1379 tok/s1.30×33% → 71%1.09×28% → 97%
10 req/s1150 → 1530 tok/s1.33×24% → 36%1.11×17% → 35%
12 req/s1198 → 1615 tok/s1.35×17% → 26%1.16×9% → 23%

Load served within the SLO.

Below saturation, both systems serve every arriving request, so the gain appears not as throughput but as the load that can be served within the SLO. We therefore report the SLO capacity, the highest measured load at which 90% of requests meet the SLO of the paper, a TTFT within 6 s and a TPOT within 50 ms; each value averages two runs that differ by under 1%.

InvisibleLoRA serves a higher load within the SLO. With a batch cap of 64, the SLO capacity rises from 4 to 7 req/s, 1.75 times as much, and at 7 req/s, 98% of requests meet the SLO with InvisibleLoRA against 40% with the baseline, as shown in Table R1.1. With a batch cap of 256, the difference is larger: InvisibleLoRA serves 8 req/s, whereas the baseline already falls below 90% at 4 req/s, the lowest load we measured.


Longer inputs.

The trace already has long inputs. To isolate input length, we fixed the batch at 64 with 64 adapters and varied only the input length, with prefix caching off so that every request pays for its prefill.

Table R1.2. Speedup by input length with a batch of 64.

input tokens1285122048
speedup1.76×1.60×1.33×

The gain shrinks with longer inputs only because decoding becomes a smaller share of each request. With the hybrid KV cache, InvisibleLoRA prefills once, as standard decoding does, so the gain comes only from decoding. Counting decoding alone, the speedup stays at 1.62× with 2048 input tokens, as shown in Table R1.2.


Longer outputs.

Most outputs in the trace are short, with a median of about 200 tokens, so we also ran outputs of up to 1024 tokens, with a median of about 320, under Poisson arrivals at 2 to 6 req/s.

With longer outputs, the gain appears as faster token generation. These loads stay below saturation, so both systems meet the SLO for every request, and the gain shows mainly as a median TPOT 28 to 52% lower, while throughput rises by 4 to 16%.


Very short outputs.

Conversely, the same public dataset also has a code-completion trace with a median output of 13 tokens.

With very short outputs, little decoding remains to speed up, so the gain does not reach throughput. The decoding steps still become faster, with a TPOT 3 to 22% lower, but throughput is within 1% of the baseline, and the share of requests meeting the SLO differs by at most 3 percentage points.


The trace replay showed that the gain appears most clearly as SLO capacity. We have added the trace replay and the long-output and code-completion results to Section 4.4 and to Tables 13 and 14 of Appendix F.

AuthorsOpenReview ↗

Response to Reviewer 8f66 (2/6): W2

W2. The request-per-second (RPS) rate is not reported in the serving evaluation. Without it, the reported numbers are difficult to interpret.

Response. We thank the reviewer for pointing this out. We agree that serving results cannot be interpreted without the request rate, and we apologize that the request rates in the submission were hard to find. The load axis appeared only in an appendix figure, now Figure 12 of Appendix F, and Figure 6, which was Figure 5 in the submission, stated the rule for choosing the load but not the loads themselves. The revision states the request rate behind every serving result, and we summarize these rates below.


Figure 6 compares the two systems at loads past saturation.

Figure R2.1. Peak throughput against the number of resident adapters.

In Figure R2.1, which is Figure 6 of the revised paper, each point is the peak throughput, measured at the offered load that maximizes the throughput of InvisibleLoRA. Both systems are past saturation at this load. With 64 resident adapters, the load is 40 req/s for uniform adapter choice, 32 req/s for skewed adapter choice and 32 req/s for trace-driven adapter choice. With 128 resident adapters, the load is 24 req/s for all three patterns.


InvisibleLoRA raises the SLO capacity under every adapter-choice pattern.

Figure R2.2. Throughput, latency and SLO attainment along the load axis.

Figure R2.2, which is Figure 12 of Appendix F, reports throughput, latency and the share of requests that meet the SLO at every measured load. The SLO capacity, the highest load at which 90% of requests meet the SLO, rises from 10.3 to 15.3 req/s with uniform adapter choice, from 14.2 to 18.4 req/s with skewed choice and from 16.2 to 18.9 req/s with trace-driven choice.

Below the SLO capacity of the baseline, the gain appears in latency, and above that capacity, the gain appears in throughput. With uniform adapter choice at 8 req/s, both systems carry the offered load, so the throughput ratio is only 1.05×, while the time per output token already decreases by more than half, from 37.2 to 16.6 ms. At 12 and 16 req/s, the throughput gain grows to 1.30× and 1.56×.


All new serving results are reported by request rate.

Table R2.1 lists the main results with their request rates. For example, in the trace replay with a batch cap of 64, the share of requests that meet the SLO at 7 req/s rises from 40% to 98%.

Table R2.1. New serving results by request rate.

setting (Llama-3.1-8B, 64 resident adapters)load (req/s)throughput speeduprequests meeting the SLO (baseline → InvisibleLoRA)
Azure LLM trace replay, batch cap 644 / 7 / 121.12× / 1.27× / 1.35×95 → 100% / 40 → 98% / 17 → 26%
uniform, batch cap 12824 / 321.41× / 1.45×30 → 100% / 18 → 37%
uniform, batch cap 25632 / 40 / 481.12× / 1.25× / 1.25×100 → 100% / 35 → 100% / 25 → 42%

Changes in the revision.

The caption of Figure 6 now states these loads and points to the load-axis figure, Figure 12 of Appendix F, to which we have added a throughput column. Appendix F also reports the trace-replay and batch-cap results by request rate in Tables 13 and 15.

AuthorsOpenReview ↗

Response to Reviewer 8f66 (3/6): W3 and Q1

W3. The testbed is poorly described, and the actual serving deployment parameters are not stated in the paper.

Q1. How are the serving parameters configured? For example, what are the chunk size and maximum batch size? Online serving systems are extremely sensitive to these values. In addition, is prefix caching, PD disaggregation enabled in this setting, or is it still dealing with single turn conversation?

Response. We thank the reviewer for these specific questions, and we fully agree that online serving results are sensitive to these settings. We apologize: the appendix went through many revisions, and the deployment settings ended up scattered across Section 4.1, Section 4.4 and two appendices of the submission, with several of the ones you ask about missing altogether.


Settings you asked about.

In the serving results of the paper, the chunk size is 2048 tokens and the maximum batch is 64; prefix caching is off, there is no prefill-decode disaggregation, and all requests are single-turn. The chunk size is the max_num_batched_tokens setting of vLLM, and the runs with batch caps of 128 and 256 in our response to W4 are new.

The fixed-batch runs differ in one respect: they measure decoding only. The fixed-batch runs of Figure 5 and of the appendix were measured after a warm-up pass with the default prefix caching of vLLM, so they report decode throughput without prefill. The revision states this protocol in Section 4.1 and reports the input-length study both with and without prefill in Figure 11 of Appendix E. For this reason, the input-length results in our response to W1 were measured with prefix caching off.


Complete serving configuration.

Table R3.1 gives the complete configuration.

Table R3.1. Serving configuration.

settingvalue
hardwareone NVIDIA H100 80GB SXM per engine, two for Qwen3-32B with tensor parallelism of 2; host Intel Xeon Platinum 8462Y+, 128 logical CPUs visible to the container
enginevLLM 0.11.2, V1 engine, full and piecewise CUDA graphs
schedulerchunked prefill on, max_num_batched_tokens 2048, max_num_seqs equal to the batch cap: 64, and 128 or 256 in the new runs
caching, servingprefix caching off, since prompts are reused cyclically and cache hits would distort TTFT
caching, fixed batchvLLM default, on; measured after a warm-up pass, so prefill is excluded
disaggregationnone; single-turn requests
decodinggreedy, EOS allowed, at most 256 output tokens; about 57% of requests reach the cap for Llama-3.1-8B and about 45% for Mistral-7B; trace replay uses the output lengths of the trace
arrivalsopen-loop Poisson over 60-second windows, or 90 s for the long-output and code-completion runs at 2 req/s; adapter choice uniform, Zipf with s=1, or trace-driven by the adapter popularity of the Azure Functions trace
LoRA kernelsPunica/Triton kernels, slot width 64; split_k fixed to 8 in both systems for a fair comparison, the value at which the baseline is fastest

We have collected this configuration in one place, Table 2 of Appendix A, and Section 4.1 now points to it.

Reviewer_ayjdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents InvisibleLoRA, a speculative decoding method for serving multiple LoRA adapters on a shared base model. It drafts tokens with LoRA disabled and verifies them in a batched forward pass with each request’s adapter enabled, spreading adapter execution costs across multiple output tokens. A shared KV cache avoids maintaining a separate draft cache, while a runtime gate disables drafting when it is unlikely to help. Verification preserves the target output distribution in exact arithmetic. Experiments cover five base models and 456 public adapters. Serving tests with adapters screened for high acceptance report up to 1.84 times the throughput and 48% lower time per output token than a tuned vLLM baseline.

Critical strengths
  1. The method addresses a clear serving bottleneck without extra draft training or separate draft weights and KV caches. This makes it easier to use with existing LoRA adapters while preserving the target output distribution.

  2. The experiments use a tuned baseline and carefully control kernel settings that could otherwise inflate the speedup. Tests under different request arrival patterns also show gains in service capacity, making the practical benefits more convincing.

Critical weaknesses
  1. The main serving results use adapters screened for at least 90% draft acceptance, excluding about 43% of the Mistral pool. Gains under mixed traffic that also includes adapters falling back to standard decoding remain unclear. This limits how broadly the reported speedups apply.

  2. Alternative drafters are compared mainly by acceptance rate, without throughput and latency comparisons. A cheaper drafter could provide greater speed despite lower acceptance. This leaves the practical advantage of the proposed drafting choice insufficiently tested.

Questions for authors
  1. Do the online serving results include traffic for adapters below the 90% acceptance threshold, with those requests using standard decoding? If excluded, what do the existing results tell us about gains for a mixed adapter pool?

  2. Are throughput and latency measurements available for the alternative drafters in Table 1? What evidence supports the proposed drafter when both acceptance rate and drafting cost are considered?

  3. How reliable is the acceptance screen when prompt domains or sampling temperatures change? Can an admitted adapter fall back to standard decoding if its acceptance drops during serving?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ayjd (1/5): W1 and Q1

W1. The main serving results use adapters screened for at least 90% draft acceptance, excluding about 43% of the Mistral pool. Gains under mixed traffic that also includes adapters falling back to standard decoding remain unclear. This limits how broadly the reported speedsups apply.

Q1. Do the online serving results include traffic for adapters below the 90% acceptance threshold, with those requests using standard decoding? If excluded, what do the existing results tell us about gains for a mixed adapter pool?

Response. We thank the reviewer for this question. The online results of the submission do not include traffic for adapters below 0.90. Figures 6 and 12 serve only the adapters that pass the 0.90 screen, which keeps 41 of the 72 Mistral-7B adapters. We have therefore measured the mixed traffic you describe on the Mistral-7B pool. On an unscreened pool of 64 adapters with 27 below 0.90, InvisibleLoRA raises throughput by 1.11 to 1.46× over standard decoding when every adapter drafts, close to the 1.14 to 1.54× of the screened pool. Falling back to standard decoding for those 27 adapters lowers the gain to 1.00 to 1.14×. The reported speedups therefore do not rest on excluding these adapters.

The gain depends on how many drafted tokens an adapter accepts, not on whether it passes the screen. The adapters below 0.90 still accept 74% of their drafts on average, so drafting still commits about 1.7 tokens per step for them. A request that falls back commits one token per step instead, holds its batch slot longer, and delays the requests that arrive after it.


The existing results already indicate the gain of a mixed pool. Table 1 covers every adapter of each pool, so the 92.4% reported for Mistral-7B already includes the adapters below 0.90. Since the gain scales with 1+α1+\alpha, the gain of a mixed pool follows its mean acceptance, and Equation (12) of Appendix B predicts 1.36× and 1.47× for the unscreened Mistral pool, against the measured 1.39× and 1.46× in Table R2.1. The gain also holds with a looser screen, as fixed batches on pools audited only at 0.70 reach 1.39 to 1.74× at the largest N across the five base models, as shown under Admission floor in Appendix E.

AuthorsOpenReview ↗

Response to Reviewer ayjd (2/5): W1 and Q1 (continued)

Online serving on the Mistral-7B pool confirms the prediction. We repeated the serving experiment on Mistral-7B, the base model with the widest acceptance tail, with three versions of the pool, each with 64 resident adapters (batch cap 64, Poisson arrivals, outputs up to 256 tokens):

  • unscreened: the first 64 adapters of the pool, 27 of them below 0.90, all drafting;
  • fallback: the same 64 adapters, with the 27 below 0.90 decoded without drafting, the configuration of your question;
  • screened: the 32 of the 41 adapters that pass 0.90, each loaded twice as separate adapters so that the adapter cost per step is the same.

Table R2.1. Online throughput in output tokens per second on Mistral-7B, K=1, means of two 60-second runs. The standard-decoding, fallback and every-adapter-drafts columns serve the same 64 adapters, 27 of them below 0.90, and the parentheses give the speedup over standard decoding. The last column gives the speedup of the screened pool, which holds different adapters and is measured against its own standard-decoding run.

adapter choiceloadstandard decodingfallbackevery adapter draftsscreened pool (speedup)
uniform16 req/s1,8172,031 (1.12×)2,526 (1.39×)1.43×
uniform24 req/s1,8512,101 (1.14×)2,710 (1.46×)1.54×
skewed (Zipf)20 req/s2,7102,705 (1.00×)3,113 (1.15×)1.18×
skewed (Zipf)24 req/s2,7362,763 (1.01×)3,256 (1.19×)1.21×
trace-driven (Azure Functions)20 req/s2,7602,761 (1.00×)3,074 (1.11×)1.14×
trace-driven (Azure Functions)24 req/s2,8152,823 (1.00×)3,237 (1.15×)1.18×

Letting every adapter draft also raises the share of requests that meet the SLO. On the same 64 adapters, at 16 req/s with uniform choice, the share of requests meeting the SLO rises from 34% under standard decoding to 100%, and at 20 req/s from 51% and 81% to 100% with skewed and trace-driven choice, as shown in Table R2.2.

Table R2.2. Share of requests meeting the SLO (%), a first token within 6 s and a mean time per output token within 50 ms, in the runs of Table R2.1. The first three columns serve the same 64 adapters, 27 of them below 0.90. The last column gives the screened pool, standard decoding → InvisibleLoRA.

adapter choiceloadstandard decodingfallbackevery adapter draftsscreened pool
uniform16 req/s345310030 → 100
uniform24 req/s13173213 → 33
skewed (Zipf)20 req/s515510059 → 100
skewed (Zipf)24 req/s27294829 → 59
trace-driven (Azure Functions)20 req/s8110010071 → 100
trace-driven (Azure Functions)24 req/s25244023 → 45

Falling back loses most of the gain. With fallback, the speedup drops to 1.12 to 1.14× with uniform choice and to 1.00 to 1.01× with skewed and trace-driven choice, and at 16 req/s with uniform choice only 53% of requests meet the SLO, against 100% when every adapter drafts, as shown in Tables R1.1 and R1.2.

We had set the 0.90 threshold for offline fixed batches, where one slow request holds the whole batch, as Appendix E explains, and your question led us to find that it is conservative for online serving at K=1. The revision keeps 0.90 for the fixed-batch results, states the screen in Section 4.1, and reports the unscreened and fallback results alongside the screened ones in Section 4.4 and in Table 16 of Appendix F.

AuthorsOpenReview ↗

Response to Reviewer ayjd (3/5): Q2

Q2. Are throughput and latency measurements available for the alternative drafters in Table 1? What evidence supports the proposed drafter when both acceptance rate and drafting cost are considered?

Response. We thank the reviewer for this question. We agree that a drafter with cheaper draft tokens can be faster despite lower acceptance, so acceptance alone cannot rank drafters whose draft tokens differ in cost. We measured throughput and latency for both model-based drafters of Table 1. Against the small drafter, the standard speculative decoding with four draft tokens, InvisibleLoRA reaches a shorter time to first token, TTFT, at every input length, by up to 16%. The small drafter reaches a lower time per output token, TPOT, but its throughput lead narrows for longer inputs, from 30% at 1,024 input tokens to 17–21% at 2,048 and 4,096.


Setup. Every drafter serves Llama-3.1-8B with 64 LoRA adapters in a fixed batch of 64 requests with 256 output tokens on one H100 GPU, with the prefix cache off. The small drafter, Llama-3.2-1B, uses its best draft length of four tokens, and InvisibleLoRA and the base model drafting from a separate KV cache draft one token, the setting of the paper.

Table R3.1. TTFT, TPOT and throughput speedup over standard multi-adapter decoding. TTFT is the time from sending the batch to the first token of each request, averaged over the requests.

input tokensdrafterTTFT (s)TPOT (ms)throughput speedup
1,024standard decoding2.00541×
small drafter2.1721.31.93×
separate-KV base model2.9038.91.05×
InvisibleLoRA2.0633.21.49×
2,048standard decoding3.67631×
small drafter4.0429.91.57×
separate-KV base model7.8233.4†0.88×
InvisibleLoRA3.7642.21.34×
4,096standard decoding7.31821×
small drafter8.8042.7†1.46×
separate-KV base model16.4230.5†0.86×
InvisibleLoRA7.4162.01.21×

†Part of the batch waits for KV cache space: 30 of the 64 requests for the separate-KV base model at 2,048 input tokens, and 17 for the small drafter and 47 for the separate-KV base model at 4,096. Fewer requests decode at once, so the waiting time appears in the TTFT and the throughput rather than in the TPOT.


InvisibleLoRA keeps the TTFT close to that of standard decoding because the drafter of InvisibleLoRA reads the KV cache of the target model. A drafter with its own KV cache must first fill that cache with a second prefill, which grows with the input. InvisibleLoRA accordingly adds only 1–3% to the TTFT of standard decoding, the time of one draft step, whereas the small drafter adds 9–20% and the separate-KV base model adds 45–125%, as shown in Table R3.1. The second KV cache also leaves less room, so part of the batch waits for cache space with longer inputs, and the TTFT lead of InvisibleLoRA over the small drafter grows from 5% at 1,024 input tokens to 16% at 4,096.


At the same draft cost, the higher acceptance of InvisibleLoRA makes InvisibleLoRA faster. The base model drafting from a separate KV cache runs the same model as InvisibleLoRA, so each draft token costs the same, and the hybrid KV cache raises acceptance over this drafter by 1.3–4.5% on every base model, as shown in Table 1. InvisibleLoRA accordingly reaches a 41–52% higher throughput than this drafter at every input length, as shown in Table R3.1.


The small drafter owes its lower TPOT to drafting more tokens per verification, and this advantage shrinks for longer inputs. Each draft token of a 1B model is cheap, so the small drafter drafts four tokens per verification. At the same draft length of one token, the small drafter reaches 1.48×, 1.35× and 1.26× at the three input lengths, within 4% of InvisibleLoRA, so the advantage comes from the draft length rather than from the drafter model. As inputs grow, reading the KV cache takes a growing share of each decoding step, so cheap draft tokens save a smaller share of the step, and the throughput lead of the small drafter falls from 30% at 1,024 input tokens to 17–21% at 2,048 and 4,096, as shown in Table R3.1.


We have added these measurements to Appendix G, Tables 18 and 19, and a TTFT figure to Section 4.2.


A fixed batch, however, measures each drafter at one batch of 64 requests, whereas in serving, requests keep arriving until the memory is full, so the KV capacity decides how many requests run at once. The practical advantage that the reviewer raises in W2 is therefore better measured by the goodput under a service-level objective, which we report in our response to W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.