- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
overall score raised from 2 (weak reject) to 3 (weak accept) after the rebuttal
The paper proposes a new solution for speculative decoding specifically for multi-LoRA serving. The core problem is that during decoding each adapter only acts on a single or few tokens and thus runs very inefficiently. In particular when many adapters are used, this can dominate latency over the base models backbone forward pass. The proposed solution uses the base model as a shared drafter for all adapters. This is a neat and elegant solution and seems of practical value. The idea does not require much theory and to me is very easy to understand mostly based on the ideas of LoRA and Speculative Decoding. The authors ablate their results on many models and adapters. A nice insight is that the backbone model can use the existing KV cache of the adapters and thus does not introduce any further memory overheads (that a seperate drafter would ad). I think the experimental validation might not be the most representative, however. All adapters and the acceptance rates are evaluated on generic tasks. In a realistic scenario, each adapter would be run on the task it was fine-tuned for. To me it is plausible that for the generic tasks, the adapters do not add much and hence the acceptance rates reported in the paper are inflated.
Overall imo opinion this is a nice idea that has practical use. Tough I'd say it is also not groundbreaking in any way.
I want to state that the main part of the paper reads well and I was very happy to read it. However, the appendix reads LLM generated with many LLM-ish phrases like "load-bearing", "the two arms".... This is very tiring to read as a human, hard to extract value from it, and i was a bit disappointed when I got there...
In the current state, I am reluctant to accept the paper. but if the additional experiments can be provided, code is released and the appendix gets a human rewrite, I am happy to support acceptance
Critical strengths
The paper very clearly formulates a crisp problem, the decoding overhead in multi LoRA serving, and provides a simple and elegant solution to it. The theory is simple and mostly follows from previous work. The main paper is cleanly written and I would attest that it includes honest human effort in writing.
Besides the runtime speedups, the method also circumvents the need for additional KV cache (or weights) of a drafter model, thus not adding HBM pressure. I think the authors could even call out one more advantage: Invisible Draft does not need any additional prefil of the drafter model. Other speculative decoding methods need to fill the drafters KV cache and thus add TTFT (time-to-first-token) overheads. InvisbileLoRA does not, which is great in my opinion.
Critical weaknesses
- I think the experiemts for the acceptance rate are not representative. In practice, each LoRA adapter is used on a different task, the one which it is fine-tuned for. Thus, I'd think that on those tasks the difference in the distribution vs the baseline model is larger, than on a generic tasks on which the base model is already good. I think the results are those overly optimistic. I would encourage the authors to try to run experiments as well where each adapter is evaluated on its own task and report those as well.
- code is not provided but the papers value hinges a lot on it in my opinion. Thus acceptance should only be given if the code is provided. i presume the method will be upstreamed to vLLM once published, but it would be nice as a reviewer to take a quick look and potentially play with it for a bit.
- the appendix seems to AI-written to me and is not very readable. On the positive side, the authors disclose that they used AI for writing draft text -- but in the appendix it could have used more editing efforts.
Questions for authors
- How much of the appendix did you write yourself vs how much AI generated is it?
- Can you provide the code?
- Can you please run some experiements that evaluate the adapter on its task? In particular it would be nice to see how the acceptance rate behaves as the accuracy difference between base model and LoRA model vary on that task. I think this could be done with a single family of adapters that vary in size or quality of their training.
- I think it could be a bit clearer where the latency from the LoRA adapters comes from. I presume it is memory-bw bound, or is it actually launch overheads? Surely it is not the actual compute, but it would be nice if this could be clarified and analyzed separately
8 comments
Response to Reviewer 6g5z (2/5): W2 and Q2
W2. code is not provided but the papers value hinges a lot on it in my opinion. Thus acceptance should only be given if the code is provided. I presume the method will be upstreamed to vLLM once published, but it would be nice as a reviewer to take a quick look and potentially play with it for a bit.
Q2. Can you provide the code?
Response. We thank the reviewer for stressing this. We agree that the value of the method depends on the code. The code is now attached to this submission as supplementary material, in the archive invisiblelora.zip.
The archive is ready for a quick look and a short run. It contains the patch for vLLM 0.11.2, the scripts that reproduce the figures and the manifests of the adapter pools. On one H100 GPU, the quick start reproduces the Llama-3.1-8B result with 64 adapters in about ten minutes.
Response to Reviewer 6g5z (3/5): W3 and Q1
W3. the appendix seems to AI-written to me and is not very readable. On the positive side, the authors disclose that they used AI for writing draft text – but in the appendix it could have used more editing efforts.
Q1. How much of the appendix did you write yourself vs how much AI Generated is it?
Response. We thank the reviewer for the frank feedback and for the kind words on the main text. You are right about the appendix, and we apologize that it was hard to read. As disclosed in our LLM usage statement, we used an LLM only to turn our experiment notes and logs into draft text, and we then revised that text ourselves, as we did for the main text. When we merged the final versions before submission, however, the unrevised draft of the whole appendix replaced our revised text, and we did not catch this before the deadline. For the revision, we first restored our revised text and then went through the whole appendix again ourselves, paragraph by paragraph. We read every sentence and rewrote it or kept it, replaced the phrases you point out, such as "load-bearing" and "the two arms", with plain wording, turned sentence-like paragraph headings into plain titles, and checked every number against the logs.
Because the appendix was hard to read, we also brought together material that had been scattered across sections and reorganized the appendix substantially.
We thank the reviewer again for this comment, which led us to go through the appendix once more and organize it more systematically.
Response to Reviewer 6g5z (1/5): W1 and Q3
W1. I think the experiments for the acceptance rate are not representative. … I would encourage the authors to try to run experiments as well where each adapter is evaluated on its own task and report those as well.
Q3. Can you please run some experiments that evaluate the adapter on its task? In particular it would be nice to see how the acceptance rate behaves as the accuracy difference between base model and LoRA model vary on that task. I think this could be done with a single family of adapters that vary in size or quality of their training.
Response. We thank the reviewer for this insightful comment. We agree that each adapter should also be evaluated on the task it was trained for, since that is where it is mainly used. In short, acceptance stays high on the own task of each adapter, and it drops when an adapter writes differently from the base model, not when it becomes more accurate.
Acceptance on each adapter's own task.
We had measured this, evaluating every math, code and chat adapter only on prompts of its own domain, but the result was only in an appendix of the submission, now Table 8 of Appendix D.
Table R1.1. Average acceptance on all prompts and on the own domain of each adapter.
As you expected, acceptance is slightly lower on the own tasks, but it stays high. The average acceptance falls by at most 3.6 points, as shown in Table R1.1.
What acceptance measures.
Acceptance and accuracy measure different things. Acceptance is the share of tokens drafted by the base model that the adapter accepts, so it is high when the adapter would write almost the same text as the base model. Accuracy checks only the final answer. An adapter can therefore fix an answer while keeping the rest of the text, or rewrite a whole solution without fixing any answer. The three measurements below test which of the two lowers acceptance.
Acceptance against the accuracy gap.
Following Q3, we scored 24 public adapters, 12 for math and 12 from other domains, on the same 200 GSM8K problems, on which the base model scores 0.89.
Table R1.2. Median acceptance by the accuracy gap to the base model on 200 GSM8K problems.
Adapters with a larger accuracy gap have only slightly lower acceptance. As shown in Table R1.2, the median acceptance falls from 99.2% to 95.0%, about four points, when the accuracy of an adapter is 0.05 or more below the base model.
Following adapters through training.
We followed three adapter families over five checkpoints on the same GSM8K problems: a public GSM8K training run, and two runs that we trained on the same MATH problems with the same settings. Our two runs differ only in what the adapter learns to write: solutions written by people, or correct solutions written by the base model itself.
Table R1.3. Acceptance on GSM8K over five checkpoints of three adapter families.
Acceptance falls mainly when the adapter learns to write differently from the base model. Learning solutions written by people lowers acceptance by 1.8 points per checkpoint, and the public run by 1.5 points, whereas learning the solutions of the base model lowers it by only 0.3 points, as shown in Table R1.3. Even after five checkpoints, the lowest acceptance is 87%.
An adapter that answers more questions correctly.
None of the three families clearly beats the base model in accuracy under our scoring, so we also evaluated a public math adapter reported to do so, following its model card: all 1,319 GSM8K test problems, scored by exact match of an answer tag.
An adapter can become more accurate and still keep acceptance high. Under its model card, this adapter raises accuracy from 0.647 to 0.748, and its acceptance is 99.5%, as high as the adapters closest to the base model. Under the scoring used above, it scores 0.860 against 0.890, and it is the only reported gain we could test.
We are grateful for this suggestion, which led us to a clearer understanding of how acceptance and accuracy are related. The revision states the own-task result in Section 4.2, keeps the full table as Table 8 of Appendix D, and adds the accuracy and checkpoint results as Tables 10 and 11 of Appendix D.