- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a label-free calibration method for KV cache sharing in multi-LoRA systems, building on DroidSpeak, which also serves as the baseline. AuLoRA selects a consecutive layers to recompute with calibration that does not require labeling or per-candidate decoding, resulting in stable and fast selection. The paper also presents an analysis of the sharing error, decomposing the error into two parts (weight term, hidden-state term) and estimating the contribution of each.
Critical strengths
-
AuLoRA introduces two-stage label-free profiling to select the recomputed blocks faster. It requires neither labeling nor per-candidate decoding during calibration, while maintaining or exceeding the accuracy of DroidSpeak.
-
AuLoRA decomposes the KV difference between the base and the expert models into a weight term and a hidden-state term. This addresses an aspect that had been overlooked in prior work. With the analysis, The authors argue that the recomputation of hidden state is unavoidable in multi-LoRA systems.
Critical weaknesses
-
AuLoRA's accuracy gain appears to come mainly from the tail compensation, not the layer selection. In common context-sharing scenarios in serving systems, it is natural for other methods (including DroidSpeak) to share only context (e.g. user history, user profile, system prompt, RAG context) without questions (e.g. user query). It is therefore hard to regard the tail compensation as a contribution of AuLoRA. It is rather a natural design choice for KV sharing methods.
-
AuLoRA covers only the base-to-expert KV sharing case. For expert-to-expert cases, AuLoRA does not propose a method for maintaining the base model’s KV cache for tokens generated by an expert or a method for sharing KV cache between experts. In multi-LoRA-agent scenario where multi agents take turns in a trajectory, AuLoRA may not be applicable or may not retain its TTFT gain.
Questions for authors
-
In the conclusion, the statement that AuLoRA is the first training-free recipe that shares a base model's KV cache with LoRA experts fine-tuned independently of it appears to be overclaimed. What distinguishes AuLoRA from prior work (LRAgent, DroidSpeak) in a way that supports this claim?
-
Regarding the accuracy improvement, how would you attribute the gain among the three components (1. two-stage label-free profiling, 2. per-expert profiling, 3. tail compensation)? Specifically, taking DroidSpeak’s published protocol (labeled search, profiled once per model family, M = 1) as the reference, please toggle each component independently while keeping everything else fixed, including the expert, the test set, the profiling pool, and the recomputation budget (e.g., b = 10).
-
Have you evaluated AuLoRA on multi-agent trajectories in which several agents take turns within a single trajectory, or at least an explanation of how to apply AuLoRA to such scenarios?
9 comments
Thank you for the additional experiments and the effort revising the paper to address my concerns. I have carefully read the main text of the revised paper again, focusing on the descriptions of tail compensation and of the baselines (LRAgent and DroidSpeak). Despite the authors’ efforts, I would like to maintain my score, for the following reasons.
- Tail compensation could be a natural choice or extension in serving systems, which weakens the contribution. The accuracy gain from tail compensation can easily be inferred from sliding-window-like designs found in many approximate KV methods (sharing, compression, and pruning). AuLoRA offers only an empirical, outcome-based interpretation of tail compensation. To claim the contribution, There might need an original analysis of its underlying mechanism at least.
- The superiority over LRAgent comes from the experimental setting and does not demonstrate the practical effectiveness of the proposed method. LRAgent targets scenarios where multiple LoRA agents take turns sequentially on the same trajectory (e.g., ReAct-style LoRA agents), that is, multi-turn expert-to-expert sharing. LRAgent’s strength lies in splitting an expert’s cache into a Base Cache and an LR Cache. AuLoRA’s setting, by contrast, is routing base-to-expert, where the sender’s cache does not need to be split into a base cache and an LR cache. The natural way to apply LRAgent in this setting is therefore a full re-prefill of the expert, including its Base Cache. In that case, LRAgent in Table 2 would achieve the same accuracy as full prefill with a comparable TTFT. Conversely, in a multi-turn expert-to-expert scenario, AuLoRA must always run a base-model prefill separately from the expert decode and repeat the recomputation at every turn. Its superiority over LRAgent in that scenario therefore needs to be shown experimentally.
For these two reasons, I understand the contribution of AuLoRA to be as follows:
- It makes DroidSpeak’s calibration label-free, fast, and accurate.
I consider this contribution meaningful. However, the overall structure and claims of the paper are insufficient or overstated in places, and I therefore maintain my negative score.
Second response to Reviewer JAeU (1/3)
We thank the reviewer for reading the revision again. We disagree with both conclusions. The reviewer's reading of the tail holds for the methods the reviewer mentions, where one model both writes and reads the cache, but it does not carry over to cross-model sharing, where the receiver's weights differ from the sender's. On LRAgent, we disagree with the reviewer's reading of the setting, of how LRAgent applies to it, and of what Table 2 shows. We answer the two points in turn.
1. Tail compensation.
Within one model, attention concentrates on the most recent tokens, which is why keeping them exact is a standard device in all three areas the reviewer names. KIVI quantizes the KV cache to 2 bits but keeps the most recent tokens in full precision; StreamingLLM evicts everything except a few initial tokens and a window of recent ones; SnapKV keeps an observation window at the end of the prompt; and in sharing, RelayCaching recomputes the last ten tokens of a cache reused between agents of the same model. All three kinds of methods the reviewer mentions work this way, and for them the point is right: when one model both writes and reads the cache, the device is not new. None of them is what the paper measures.
Unlike those designs, which keep or recompute a model's own recent entries, the tail touches nothing the sender cached: the receiver, a different model, computes only its own question with its own weights. In cross-model sharing this has stayed a setting: DroidSpeak always has the receiver compute a single suffix token, while LRAgent and KVComm always have it prefill its own tokens. None of them varies the tail apart from its repair, so none reports what we measured: a LoRA expert that computes only the few dozen tokens of its own question, over 12–16k tokens of context cached by the base model, recovers 43% of the accuracy drop, with no training and no alignment step (Table 3).
Nor does attention to recent tokens predict the effect. If recency were the reason, the tail would help for any pair of models. It does not:
- on the fully fine-tuned pair DroidSpeak uses, where the two models differ in every weight, the tail recovers 16% and adds nothing measurable on top of recomputation (−2.30 [−5.72, +1.25]; Appendix N);
- on our adapters it adds to recomputation (+1.90 [+0.94, +3.17], ten experts), while a low-rank correction adds nothing measurable on top of it (−0.19 [−0.54, +0.12], 48 adapters; Table A5).
Which pairs the tail helps, and which repairs it composes with, could only be found by measuring it in isolation. To our knowledge, no prior work has measured or explained this effect in cross-model sharing, and the comment names none. If the reviewer knows of one, we ask for the reference, and we will withdraw the claim.
The reviewer asks for a mechanism before the tail can count as a contribution. We read it the other way: a large, reproducible effect that no prior work measured and no current account explains is the contribution, and its mechanism is the open question Section 7 poses. What is certain is arithmetic: at the positions the receiver computes for itself, the weight term of Equation 1 is zero. What turns that into accuracy is not that those entries are exact; they attend to the shifted context at every layer, so they differ from the entries a full prefill would produce.
Response to Reviewer JAeU
We thank the reviewer for the precise questions; the reviewer's reading of where the accuracy gain comes from is correct. In short: W1/Q2, the toggle table below uses DroidSpeak's published protocol as the reference you specified, and what we add on the tail is its isolated measurement and its scope; W2/Q3, the method applies to relays unchanged, and its hand-offs are timed in vLLM; Q1, the claim is removed.
W1 / Q2 (tail vs. selection; toggle table). As the reviewer suspected, most of the accuracy gain comes from the tail; the table toggles each component with DroidSpeak's published protocol as the reference. The second profiling stage is removed in the revision (General Response, point 2), so the first component is now single-stage label-free profiling. Ten experts, 10 layers, the same experts, test sets and profiling pool throughout (2,000 questions per question-answering expert):
Paired differences over the ten experts: the tail alone adds +2.32 [+1.31, +3.80]; label-free selection alone adds +0.81 [+0.20, +1.69]; with the tail on both sides, AuLoRA vs. DroidSpeak's published protocol is +0.40 [−0.03, +1.04] (P = 96%), and vs. the labeled search run for every expert +0.16 [−0.38, +0.82] (P = 71%). So once both use the tail, our selection is comparable to the labeled search; its value is that it needs no labels and costs 3.0–5.7× (QA) and 6.6–9.3× (summarization) less than DroidSpeak's published protocol, and 52–73× and 1,795–2,636× less than the per-expert search.
On the tail itself, we agree that letting a receiver compute its own question is a natural design choice; prior protocols, including KVComm's and LRAgent's, already make it. That is also why its effect went unmeasured: because it is part of the setting, each protocol fixes it, and to our knowledge none reports what it contributes on its own. Measured in isolation, it turns out to be a method with a definite scope (General Response, point 3):
W2 / Q3 (multi-agent trajectories). Yes for latency, and the method applies unchanged: as in PrefillShare's hand-off, the base model prefills each turn's new tokens, and the next expert recomputes its block from the stored hidden state and prefills its own instruction. Each hand-off is therefore the single read of the base cache that our accuracy tables evaluate, on a context that now ends with the previous expert's output. Measured with the full method in vLLM over a relay of four experts (General Response, point 4; Table A23), every hand-off after the first reaches its first token 1.6–2.7× sooner for 4k–28k-token prefixes, whether each expert generates 32, 128 or 256 tokens; the first reader also pays for the base prefill. Expert-to-expert reuse remains outside our scope, because the tail's advantage holds only from the base model to an adapter.
Q1 ("first"). We agree that the claim was overstated and removed it from the conclusion. What distinguishes our work is how it relates to the two: DroidSpeak and LRAgent disagree on whether recomputation is needed at all, DroidSpeak recomputing a block of layers and LRAgent arguing that a low-rank correction suffices. We resolve that disagreement rather than take a side. The decomposition shows that a low-rank correction reaches only the weight term, a small part of the KV difference, so recomputation is needed wherever direct reuse causes an accuracy drop (Section 3). The method then keeps what works in each and drops what does not: DroidSpeak's recomputation, with the block chosen without labels rather than by a labeled search, and the question tail that LRAgent's protocol already includes, without the low-rank correction, which adds nothing once the tail is in place.