acceptodds
Under review as a conference paper at ICLR 2027

AuLoRA: An Automatic, Label-Free Framework for KV Cache Sharing in Multi-LoRA Systems

Abstract

In modern multi-agent and multi-expert systems, KV heterogeneity across models prevents a successor from inheriting its predecessor's KV cache: every handoff re-prefills the entire prefix context, inflating time-to-first-token (TTFT). This paper targets KV cache sharing in multi-LoRA systems, where one base model serves many domain-specific LoRA experts—a memory-efficient multi-expert setup. Here the base model is the natural sender: its KV cache is adapter-agnostic and can be reused by every expert. Existing training-free sharing methods, however, target either a pair of fine-tuned models or agents collaborating on a single task, leaving the base-to-adapter direction largely unexamined. Reusing the base model's KV cache directly leads to substantial performance degradation. We decompose the difference between the base model's and the expert's KV caches into a weight term and a hidden-state term, which correspond to the two existing families of training-free remedies, a low-rank correction and partial recomputation. Experiments show that the two terms are nearly orthogonal, and the hidden-state term dominates, so an exact low-rank correction removes only 1–3% of the KV difference. Recomputation is thus unavoidable in multi-LoRA systems. State-of-the-art recomputation methods rely on offline calibration: they search over consecutive-layer combinations for the one with the best task accuracy on a labeled calibration set, at a cost that grows quadratically with model depth. We present AuLoRA, a training-free KV cache sharing framework for multi-LoRA systems with two components. (i) Two-stage profiling: a label-free algorithm that reduces the search cost complexity from to , where is the model depth, without per-candidate decoding and with few calibration samples; its profiling result matches the current state of the art, with about faster end-to-end profiling on question answering and over an order of magnitude faster on summarization. (ii) Tail compensation: a LoRA-only mechanism that lets each expert prefill the tail of its own question on top of the shared context KV cache. With negligible compute overhead, it improves quality at a fixed recomputation budget and lowers the budget required for a target quality. Overall, with roughly one third of the layers recomputed, the two components recover 53–95% of the quality lost to direct reuse and reduce the expert's prefill latency by –.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache sharing, KV cache reuse, multi-LoRA systems, low-rank adaptation (LoRA), efficient LLM inference, selective layer recomputation, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JAeUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a label-free calibration method for KV cache sharing in multi-LoRA systems, building on DroidSpeak, which also serves as the baseline. AuLoRA selects a consecutive layers to recompute with calibration that does not require labeling or per-candidate decoding, resulting in stable and fast selection. The paper also presents an analysis of the sharing error, decomposing the error into two parts (weight term, hidden-state term) and estimating the contribution of each.

Critical strengths
  • AuLoRA introduces two-stage label-free profiling to select the recomputed blocks faster. It requires neither labeling nor per-candidate decoding during calibration, while maintaining or exceeding the accuracy of DroidSpeak.

  • AuLoRA decomposes the KV difference between the base and the expert models into a weight term and a hidden-state term. This addresses an aspect that had been overlooked in prior work. With the analysis, The authors argue that the recomputation of hidden state is unavoidable in multi-LoRA systems.

Critical weaknesses
  • AuLoRA's accuracy gain appears to come mainly from the tail compensation, not the layer selection. In common context-sharing scenarios in serving systems, it is natural for other methods (including DroidSpeak) to share only context (e.g. user history, user profile, system prompt, RAG context) without questions (e.g. user query). It is therefore hard to regard the tail compensation as a contribution of AuLoRA. It is rather a natural design choice for KV sharing methods.

  • AuLoRA covers only the base-to-expert KV sharing case. For expert-to-expert cases, AuLoRA does not propose a method for maintaining the base model’s KV cache for tokens generated by an expert or a method for sharing KV cache between experts. In multi-LoRA-agent scenario where multi agents take turns in a trajectory, AuLoRA may not be applicable or may not retain its TTFT gain.

Questions for authors
  • In the conclusion, the statement that AuLoRA is the first training-free recipe that shares a base model's KV cache with LoRA experts fine-tuned independently of it appears to be overclaimed. What distinguishes AuLoRA from prior work (LRAgent, DroidSpeak) in a way that supports this claim?

  • Regarding the accuracy improvement, how would you attribute the gain among the three components (1. two-stage label-free profiling, 2. per-expert profiling, 3. tail compensation)? Specifically, taking DroidSpeak’s published protocol (labeled search, profiled once per model family, M = 1) as the reference, please toggle each component independently while keeping everything else fixed, including the expert, the test set, the profiling pool, and the recomputation budget (e.g., b = 10).

  • Have you evaluated AuLoRA on multi-agent trajectories in which several agents take turns within a single trajectory, or at least an explanation of how to apply AuLoRA to such scenarios?

9 comments
AuthorsOpenReview ↗

Response to Reviewer JAeU

We thank the reviewer for the precise questions; the reviewer's reading of where the accuracy gain comes from is correct. In short: W1/Q2, the toggle table below uses DroidSpeak's published protocol as the reference you specified, and what we add on the tail is its isolated measurement and its scope; W2/Q3, the method applies to relays unchanged, and its hand-offs are timed in vLLM; Q1, the claim is removed.

W1 / Q2 (tail vs. selection; toggle table). As the reviewer suspected, most of the accuracy gain comes from the tail; the table toggles each component with DroidSpeak's published protocol as the reference. The second profiling stage is removed in the revision (General Response, point 2), so the first component is now single-stage label-free profiling. Ten experts, 10 layers, the same experts, test sets and profiling pool throughout (2,000 questions per question-answering expert):

ConfigurationQA (6)Summ. (4)
DroidSpeak published protocol (labeled, once per family, M=1)70.2337.41
+ per-expert profiling (128 labeled prompts)70.5638.38
label-free profiling instead (per expert)71.3637.75
DroidSpeak published + tail73.2838.63
per-expert 128 + tail73.3939.07
AuLoRA (label-free profiling + tail)73.6939.02

Paired differences over the ten experts: the tail alone adds +2.32 [+1.31, +3.80]; label-free selection alone adds +0.81 [+0.20, +1.69]; with the tail on both sides, AuLoRA vs. DroidSpeak's published protocol is +0.40 [−0.03, +1.04] (P = 96%), and vs. the labeled search run for every expert +0.16 [−0.38, +0.82] (P = 71%). So once both use the tail, our selection is comparable to the labeled search; its value is that it needs no labels and costs 3.0–5.7× (QA) and 6.6–9.3× (summarization) less than DroidSpeak's published protocol, and 52–73× and 1,795–2,636× less than the per-expert search.

On the tail itself, we agree that letting a receiver compute its own question is a natural design choice; prior protocols, including KVComm's and LRAgent's, already make it. That is also why its effect went unmeasured: because it is part of the setting, each protocol fixes it, and to our knowledge none reports what it contributes on its own. Measured in isolation, it turns out to be a method with a definite scope (General Response, point 3):

  • it stacks with recomputation, raising the peak block's score by +1.90 points [+0.94, +3.17] on the ten experts, but not with a low-rank correction, which adds −0.19 [−0.54, +0.12] on top of it over 48 multi-hop adapters (Table A5);
  • it accounts for nearly all of what LRAgent's BaseShared recovers in Table 2 (43.1% of 43.9%; Table A1);
  • it holds only from a base model to its LoRA experts: alone it recovers 43% of the accuracy drop on our experts but 16% on the fully fine-tuned pair DroidSpeak uses, where it adds nothing measurable on top of recomputation (−2.30 [−5.72, +1.25]; Appendix N). At the tail's positions the weight term is zero by construction; why that turns into accuracy on LoRA experts and not on a fully fine-tuned pair, we measure rather than explain (Section 7).

W2 / Q3 (multi-agent trajectories). Yes for latency, and the method applies unchanged: as in PrefillShare's hand-off, the base model prefills each turn's new tokens, and the next expert recomputes its block from the stored hidden state and prefills its own instruction. Each hand-off is therefore the single read of the base cache that our accuracy tables evaluate, on a context that now ends with the previous expert's output. Measured with the full method in vLLM over a relay of four experts (General Response, point 4; Table A23), every hand-off after the first reaches its first token 1.6–2.7× sooner for 4k–28k-token prefixes, whether each expert generates 32, 128 or 256 tokens; the first reader also pays for the base prefill. Expert-to-expert reuse remains outside our scope, because the tail's advantage holds only from the base model to an adapter.

Q1 ("first"). We agree that the claim was overstated and removed it from the conclusion. What distinguishes our work is how it relates to the two: DroidSpeak and LRAgent disagree on whether recomputation is needed at all, DroidSpeak recomputing a block of layers and LRAgent arguing that a low-rank correction suffices. We resolve that disagreement rather than take a side. The decomposition shows that a low-rank correction reaches only the weight term, a small part of the KV difference, so recomputation is needed wherever direct reuse causes an accuracy drop (Section 3). The method then keeps what works in each and drops what does not: DroidSpeak's recomputation, with the block chosen without labels rather than by a labeled search, and the question tail that LRAgent's protocol already includes, without the low-rank correction, which adds nothing once the tail is in place.

Thank you for the additional experiments and the effort revising the paper to address my concerns. I have carefully read the main text of the revised paper again, focusing on the descriptions of tail compensation and of the baselines (LRAgent and DroidSpeak). Despite the authors’ efforts, I would like to maintain my score, for the following reasons.

  • Tail compensation could be a natural choice or extension in serving systems, which weakens the contribution. The accuracy gain from tail compensation can easily be inferred from sliding-window-like designs found in many approximate KV methods (sharing, compression, and pruning). AuLoRA offers only an empirical, outcome-based interpretation of tail compensation. To claim the contribution, There might need an original analysis of its underlying mechanism at least.
  • The superiority over LRAgent comes from the experimental setting and does not demonstrate the practical effectiveness of the proposed method. LRAgent targets scenarios where multiple LoRA agents take turns sequentially on the same trajectory (e.g., ReAct-style LoRA agents), that is, multi-turn expert-to-expert sharing. LRAgent’s strength lies in splitting an expert’s cache into a Base Cache and an LR Cache. AuLoRA’s setting, by contrast, is routing base-to-expert, where the sender’s cache does not need to be split into a base cache and an LR cache. The natural way to apply LRAgent in this setting is therefore a full re-prefill of the expert, including its Base Cache. In that case, LRAgent in Table 2 would achieve the same accuracy as full prefill with a comparable TTFT. Conversely, in a multi-turn expert-to-expert scenario, AuLoRA must always run a base-model prefill separately from the expert decode and repeat the recomputation at every turn. Its superiority over LRAgent in that scenario therefore needs to be shown experimentally.

For these two reasons, I understand the contribution of AuLoRA to be as follows:

  • It makes DroidSpeak’s calibration label-free, fast, and accurate.

I consider this contribution meaningful. However, the overall structure and claims of the paper are insufficient or overstated in places, and I therefore maintain my negative score.

AuthorsOpenReview ↗

Second response to Reviewer JAeU (1/3)

We thank the reviewer for reading the revision again. We disagree with both conclusions. The reviewer's reading of the tail holds for the methods the reviewer mentions, where one model both writes and reads the cache, but it does not carry over to cross-model sharing, where the receiver's weights differ from the sender's. On LRAgent, we disagree with the reviewer's reading of the setting, of how LRAgent applies to it, and of what Table 2 shows. We answer the two points in turn.

1. Tail compensation.

Within one model, attention concentrates on the most recent tokens, which is why keeping them exact is a standard device in all three areas the reviewer names. KIVI quantizes the KV cache to 2 bits but keeps the most recent tokens in full precision; StreamingLLM evicts everything except a few initial tokens and a window of recent ones; SnapKV keeps an observation window at the end of the prompt; and in sharing, RelayCaching recomputes the last ten tokens of a cache reused between agents of the same model. All three kinds of methods the reviewer mentions work this way, and for them the point is right: when one model both writes and reads the cache, the device is not new. None of them is what the paper measures.

Unlike those designs, which keep or recompute a model's own recent entries, the tail touches nothing the sender cached: the receiver, a different model, computes only its own question with its own weights. In cross-model sharing this has stayed a setting: DroidSpeak always has the receiver compute a single suffix token, while LRAgent and KVComm always have it prefill its own tokens. None of them varies the tail apart from its repair, so none reports what we measured: a LoRA expert that computes only the few dozen tokens of its own question, over 12–16k tokens of context cached by the base model, recovers 43% of the accuracy drop, with no training and no alignment step (Table 3).

Nor does attention to recent tokens predict the effect. If recency were the reason, the tail would help for any pair of models. It does not:

  • on the fully fine-tuned pair DroidSpeak uses, where the two models differ in every weight, the tail recovers 16% and adds nothing measurable on top of recomputation (−2.30 [−5.72, +1.25]; Appendix N);
  • on our adapters it adds to recomputation (+1.90 [+0.94, +3.17], ten experts), while a low-rank correction adds nothing measurable on top of it (−0.19 [−0.54, +0.12], 48 adapters; Table A5).

Which pairs the tail helps, and which repairs it composes with, could only be found by measuring it in isolation. To our knowledge, no prior work has measured or explained this effect in cross-model sharing, and the comment names none. If the reviewer knows of one, we ask for the reference, and we will withdraw the claim.

The reviewer asks for a mechanism before the tail can count as a contribution. We read it the other way: a large, reproducible effect that no prior work measured and no current account explains is the contribution, and its mechanism is the open question Section 7 poses. What is certain is arithmetic: at the positions the receiver computes for itself, the weight term of Equation 1 is zero. What turns that into accuracy is not that those entries are exact; they attend to the shifted context at every layer, so they differ from the entries a full prefill would produce.

Reviewer_xrFLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets training-free KV cache sharing in multi-LoRA serving system. The authors decompose the KV cache error into a weight term and a hidden-state term. They show that cumulative hidden-state drift dominates and makes partial layer recomputation necessary. Inspire by this, they propose AuLoRA, a two-stage and label-free profiling method to select the layers to recompute and share the kv cache from based model to each expert. Experiments on two LLMs show that, recomputing 1/3 layers can recover performance from 53% to 95% to direct cache reuse while achieve a speedup of 2.5 ~ 2.7x.

Critical strengths
  1. This paper targets a important problem in multi-LoRA serving, as avoiding redundant prefill is important for reducing latency when using multiple models.

  2. The error decomposition provides a clear explanation of why simple low-rank cache corrections fail and why partial layer recomputation is needed.

  3. The idea of tail compensation is simple and effective

Critical weaknesses
  1. The evaluation does not show the end-to-end benefit of the mentioned serving setting. The experiments only evaluate single-turn QA and summarization, and the reported 2.5–2.7× speedup measures only the expert’s latency after the base cache is already available. Under such scenairo, directly prefilling the long-context input using a specific expert is more reasonable to me.

  2. The authors claim label-free profiling as key contribution. However, labels are already available for these tasks, so using them for calibration cost little. The benefit of profiling without labels is unclear.

  3. Using Stage 2 only brings marginal improvements over Stage 1 (Table 3), but largely increasing the profiling time (10x). It is hard to see why Stage 2 is worthy in standard deployments.

  4. If different experts select different starting layers for the save prefix, the whole framework need to keep a large amount of tensors in the memory. However, this additional memory overhead is not analysized.

Questions for authors
  1. What is the additional GPU memory overhead when serving multiple experts simultaneously?

  2. Have the authors tried batched serving in vLLM or SGlang? Does the speedup still holds under high-concurrency scenario?

  3. Why does tail compensation hurt performance on fully fine-tuned model?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xrFL

We thank the reviewer for the positive assessment and the concrete questions.

W1 (end-to-end benefit). We agree that the 2.5–2.7× leaves out the base prefill: it is the receiver's TTFT once the shared context is cached, the accounting DroidSpeak uses, chosen to compare with it on equal terms rather than to flatter our method. With the base prefill counted and the full method running in vLLM, eight experts reading one context reach their first token 2.21× sooner and finish 1.37–1.95× sooner, and in a relay of four experts, the multi-turn case, each hand-off after the first is 1.6–2.7× faster (General Response, point 4). The saving starts from a context's second reader, the multi-LoRA setting we target; for a single expert reading once, prefilling directly remains the right choice.

W2 (value of label-free profiling). The labels are cheap; the decoding is not: a labeled search must decode an answer for every candidate block, and it is accurate only if repeated for every expert. DroidSpeak's published protocol avoids the cost by profiling once per family, and is then less accurate: it recovers 44% of the accuracy drop against our 51%, a difference of +0.81 [+0.20, +1.69] points (Table 4). Label-free profiling needs two forward passes per prompt and no decoding: 0.01–0.02 GPU-hours per expert, 52–73× (QA) and 1,795–2,636× (summarization) cheaper than a labeled search run for every expert on 128 prompts, at comparable accuracy.

W3 (Stage 2). We agree and removed it from the method; it remains only as an optional output probe (Appendix M; General Response, point 2).

W4 / Q1 (memory). The overhead is one layer's hidden state over the shared context per distinct starting layer. For the ten experts of the paper, at ten layers, label-free profiling gives all five experts of a family the same peak block (layers 8–17 on Llama, 12–21 on Qwen), so one stored hidden state serves every expert: 6.25% (Llama) and 12.5% (Qwen) of the base KV cache for that context. Each expert served concurrently additionally holds the KV of its recomputed layers (10 of 32 or 28 layers). In vLLM, with Llama experts reading one 13k-token context, the KV memory, including the stored hidden state, is 1.13×, 1.62× and 2.08× smaller than in stock vLLM, where each expert holds its own cache, with 2, 4 and 8 experts (0.70×, i.e. larger, with a single expert). With 4 contexts × 8 experts on one H100, each expert generating 256 tokens, stock vLLM fills its entire 51.3 GiB KV cache and preempts a request in every run, whereas ours uses 25.3 GiB of the same capacity and is never preempted. In a sequential relay the trade-off reverses: we keep the base cache alongside the current expert's recomputed layers, so the peak KV memory is 1.43× that of an expert that re-prefills the prefix and then discards its cache, in exchange for 1.6–2.7× faster hand-offs.

Q2 (batched serving). Yes, in vLLM: in the rebuttal period we built a research prototype of the full method on vLLM 0.11 (Appendix O); it suffices to measure time and memory but is not yet a serving system. The base model prefills each context once and keeps on the GPU the hidden state at the start of the selected block; the adapters of all experts reading that context recompute the ten selected layers from it in one batch; and the experts' questions are then served together as ordinary requests. Where recomputation changes the answer, vLLM's answers match our reference implementation's, so the timings measure the full method rather than direct reuse. With 13k-token contexts on one H100, against stock vLLM with the same engine settings and the same KV memory, the batch reaches its first token 1.27–1.32×, 1.78–1.83× and 2.21–2.24× sooner with 2, 4 and 8 experts per context, across one to four contexts; this counts the base prefill and the recomputation. At the highest load we ran, four contexts × eight experts, stock vLLM preempts requests and ours does not (W4). Being a prototype, it leaves three costs, all against us: a driver script, not vLLM's scheduler, issues the steps; contexts are prepared one after another rather than in one batch; and our cache layout holds 2.9–3.1% fewer tokens than stock vLLM in the same memory. Li et al. (2025) first reused base prefix-cache blocks across adapters, for Activated LoRA.

Q3 (tail on the fully fine-tuned pair). The tail does not measurably hurt there: stacking it on recomputation changes the score by −2.30 [−5.72, +1.25], an interval that contains zero, so we say it does not help rather than that it hurts. Why it does not help is measured rather than explained, but what we can say is consistent with it: on that pair the hidden-state drift is much larger (ten recomputed layers recover 20% of the accuracy drop there, against 51% on our LoRA experts, and 22 of 32 layers are needed for 95%), so the error left at the tail positions is dominated by the shared context, which the tail does not touch.

AuthorsOpenReview ↗

Follow-up to Reviewer xrFL

Thank you again for the positive assessment. As the discussion period closes today, we would like to point briefly to where our responses address the reviewer's concerns.

  • W1 (end-to-end benefit). We agree that the 2.5–2.7× left out the base prefill. General Response, point 4, and Appendix O measure TTFT on vLLM
  • W2 (value of label-free profiling). The labels are cheap, but a labeled search must decode an answer for every candidate block and is accurate only if repeated for every expert; our reply compares the two in accuracy and in cost.
  • W3 (Stage 2). We agree, and it is removed from the method (General Response, point 2).
  • W4 / Q1 (memory). Our reply gives the overhead per distinct starting layer and the KV memory measured in vLLM against stock vLLM.

We would be glad to clarify anything before the deadline. If these responses address the reviewer's concerns, we would be grateful if the reviewer took them into account in the final assessment.

Thanks for your clarification. My concerns are now addressd, and I will keep my original positive score

AuthorsOpenReview ↗

Response to Reviewer xrFL's follow-up

Thank you for the follow-up and for the positive assessment. We are glad the clarifications addressed the concerns, and the points the reviewer raised have improved the paper.

Reviewer_AdSQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AuLoRA combines label-free layer profiling and expert-side question-tail prefill to enable KV-cache sharing from a base model to independently trained LoRA experts.

Critical strengths

The paper provides a useful cache-error decomposition and evaluates ten experts across two model families with component ablations.

Critical weaknesses
  1. Unsupported necessity claim. The analysis measures the residual of one specific low-rank correction; it does not prove that recomputation is unavoidable. Experiments use only rank-32, all-projection adapters, and two additional experts show no statistically distinguishable loss from direct reuse.

  2. Incomplete latency accounting. The reported 2.5–2.7× speedup excludes base-model prefill and cache preparation. Since each request is routed to one expert, receiver-side savings alone do not establish an end-to-end advantage over directly prefilling with that expert.

  3. Weak justification for the second profiling stage. Table 3 shows only a 0.31-point improvement over Stage 1, with a confidence interval crossing zero, while profiling cost increases by roughly an order of magnitude. The additional complexity lacks a convincing quality–cost justification.

  4. Poor presentation. The roughly 360-word abstract buries the contribution under background, analysis and results. Baseline discussions repeat across sections, while Figures 1, 2 and 4 compress important labels and annotations into small text. The manuscript needs substantial restructuring and visual editing.

Questions for authors

please refer the weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer AdSQ

We thank the reviewer for the detailed critique; we agree with several points and changed the paper accordingly.

W1 (necessity claim). The claim compares repairs: where direct reuse causes an accuracy drop, a low-rank correction cannot take the place of recomputation; the introduction and Section 3 state this condition. On the three points raised:

  • One specific low-rank correction. We test two: the exact correction, the most any correction of the weight term can recover, and the correction LRAgent itself uses (BaseShared). On the 48 multi-hop adapters of the grid, neither adds anything measurable on top of the tail, whereas recomputation adds +3.67 points [+2.97, +4.45], and does so under every choice of target module (Table A5, Section 5.1). BaseShared's score in Table 2 comes from the question tail its protocol includes: 43.1% of the accuracy drop for the tail alone, 43.9% with the correction (Table A1).
  • Only rank-32, all-projection adapters. We added 120 adapters spanning ranks 8–64 and the three choices of target module (all-projection, q/k/v/o, q/v) that account for 93–98% of the LoRA adapters we surveyed on HuggingFace (Appendices E and P). The exact correction closes 1.7%, 3.3% and 9.0% of the KV difference on average. Restricting the adapter shifts more of the difference into the weight term, as the reviewer suggests, but never enough to replace recomputation, and raising the rank eightfold moves 9.0% only to 9.1%. Downstream, AuLoRA is more accurate than DroidSpeak on 59 of the 66 question-answering adapters and than BaseShared on 41, and worse than neither on any (Appendix E).
  • The two Qasper experts. Direct reuse costs them nothing we can measure (+1.90 [−0.39, +4.08] and +1.14 [−0.63, +2.84]). Every KV-sharing method, DroidSpeak and LRAgent included, starts from an accuracy drop to repair; where there is none, no method has anything to do, so these experts are the case the condition excludes rather than a counterexample to it. We set them aside by a rule fixed before evaluation (Appendix I.2); on the other ten experts the loss is distinguishable from zero.

W2 (latency accounting). We agree. The 2.5–2.7× is the receiver's TTFT once the shared context is cached, the accounting DroidSpeak uses, chosen to compare with it on equal terms rather than to flatter our method; it leaves out the base prefill, and Section 5.1 now says so. The revision measures the full method on vLLM with the base prefill and the recomputation counted (General Response, point 4; Appendix O). The reviewer is also right about a request routed to a single expert once: its first token is 0.84× as fast, and direct prefill remains the right choice. The saving starts from a context's second reader, the multi-LoRA setting AuLoRA targets and the one ICaRus and PrefillShare also evaluate: eight experts reading one 13k-token context reach their first token 2.21× sooner (1.27× with two, 1.78× with four), and in a relay of four experts every hand-off after the first is 1.6–2.7× sooner for 4k–28k-token prefixes. End to end, eight experts finish 1.37–1.95× sooner and the relay 1.01–1.21× at 128 tokens per turn, because only prefill is shortened. Unlike methods that retrain the experts to read the base cache, AuLoRA serves existing adapters unchanged.

W3 (second stage). We agree and removed it. Re-measured on 72 adapters at five budgets (360 paired comparisons), it helps at ten layers and hurts at eight, for roughly ten times the profiling cost (General Response, point 2; Appendix M). Label-free profiling alone is more accurate than DroidSpeak's published protocol, +0.81 points [+0.20, +1.69], at 3.0–5.7× (QA) and 6.6–9.3× (summarization) lower profiling cost (Table 4).

W4 (presentation). We agree and restructured the paper (General Response, point 5). The abstract now introduces AuLoRA in its sixth sentence rather than its tenth, after four sentences of background and one of analysis; each baseline protocol is defined once, in a glossary (Appendix A); and every label in Figures 1–4 is set at 8 pt or larger.

What the paper contributes. The summary describes AuLoRA as a combination of profiling and a question tail. Its contributions are:

  • How much a low-rank correction can do. A closed form, approximately 1−√(1−ρ²) of the KV difference, measured on 120 adapters, where it is 1.7–9.0% on average (Section 3, Appendix E).
  • Label-free profiling. Two forward passes per prompt, with no labels and no decoding; it is more accurate than DroidSpeak's published protocol at 3.0–5.7× (QA) and 6.6–9.3× (summarization) lower profiling cost (Table 4).
  • The tail, measured on its own. To our knowledge its first isolated measurement: it adds to recomputation but not to a low-rank correction, it accounts for LRAgent's accuracy, and it holds only from a base model to its LoRA experts (General Response, point 3).
AuthorsOpenReview ↗

Follow-up to Reviewer AdSQ

As the discussion period closes today, we would like to point briefly to where our responses address each of the reviewer's concerns.

  • W1 (necessity claim). General Response, point 1, extends the analysis beyond rank-32, all-projection adapters to 120 adapters across ranks and target modules, with downstream accuracy measured on 48 of them.
  • W2 (latency accounting). We agree that the 2.5–2.7× left out the base prefill. General Response, point 4, and Appendix O measure TTFT on vLLM with the base prefill counted.
  • W3 (second stage). We agree, and it is removed from the method (General Response, point 2).
  • W4 (presentation). The whole paper is revised, with a restructured abstract and introduction, a glossary and larger figure text (General Response, point 5).

We would be glad to clarify anything before the deadline. If these responses address the reviewer's concerns, we would be grateful if the reviewer took them into account in the final assessment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all three reviewers for their careful reading. We have uploaded a revised PDF; a pdfdiff against the submission is available. In brief:

  • Why recomputation (point 1). Extended from ten rank-32 all-projection experts to 120 adapters across ranks and target modules, an exact low-rank correction still closes only 1.7–9.0% of the KV difference on average and adds nothing on top of the tail downstream, whereas recomputation does.
  • A simpler method (point 2). On 72 adapters the second profiling stage helped at one budget and hurt at another, for about ten times the cost, so it is removed; label-free profiling alone remains more accurate than DroidSpeak's published protocol, at lower profiling cost.
  • Why the tail (point 3). Prior work treats the question tail as a setting. Measured in isolation for the first time we know of, it turns out to be a method: it adds to recomputation but not to a low-rank correction, it accounts for LRAgent's accuracy, and it holds only from a base model to its LoRA experts.
  • Serving (point 4). The 2.5–2.7× TTFT left out the base prefill. A research prototype on vLLM now counts it, and AuLoRA pays off wherever several experts read one context: eight experts reach their first token 2.21× sooner.
  • Writing (point 5). The whole paper is revised, with a restructured abstract and introduction, one statistical standard, a glossary and larger figure text.

1. Why recomputation is needed (AdSQ W1). The submission derived the decomposition of the KV difference and its closed form, and measured on ten experts that a low-rank correction closes little of it. As Reviewer AdSQ noted, those experts all carry rank-32 adapters on every projection, the configuration in which a correction has the least to work with. The revision therefore extends the measurement:

  • 120 adapters in the common configurations. They span ranks 8–64 and the three choices of target module (all-projection, q/k/v/o, q/v) that account for 93–98% of the LoRA adapters we surveyed on HuggingFace (Appendices E and P). The exact correction, the most any correction of the weight term can recover, closes 1.7%, 3.3% and 9.0% of the KV difference on average, and raising the rank eightfold moves 9.0% only to 9.1%.
  • Downstream, on 48 of them. On the multi-hop question-answering adapters (2,000 questions each; Table A5), the exact low-rank correction is worth +0.84 points [+0.56, +1.15] on its own but −0.19 [−0.54, +0.12] once the tail is in place, and −0.80 [−1.53, −0.11] on q/v adapters, where it is largest.
  • Recomputation works on its own, and better with the tail. On the ten experts, recomputing ten layers raises the score over direct reuse by +5.12 points [+2.99, +8.37] on its own and by +7.02 [+4.29, +11.36] with the tail (Table 3).

2. A simpler method: label-free profiling alone (AdSQ W3; xrFL W3). As Reviewers AdSQ and xrFL pointed out, the second profiling stage added only +0.31 points [−0.16, +1.03] in the submission, for roughly ten times the profiling cost. We re-measured it on 72 adapters (ranks 8–64; all-projection, q/k/v/o, q/v) at five budgets, 360 paired comparisons on 2,000 questions each, and the result is mixed: it helps at ten layers (+0.26 [+0.08, +0.46], concentrated in Qwen multi-hop QA), hurts at eight (−0.20 [−0.34, −0.07]), and is comparable elsewhere (Appendix M). We therefore removed it, and AuLoRA now profiles with the first stage alone. On the grid, AuLoRA is better than DroidSpeak on 59 of the 66 question-answering adapters and worse on none, even with DroidSpeak profiled for each adapter (Appendix E). Comparing layer selection on its own, with both sides computing a single suffix token so that only the choice of block differs (ten experts, Table 4):

  • Against DroidSpeak's published protocol, label-free profiling is more accurate by +0.81 points [+0.20, +1.69], at 3.0–5.7× (QA) and 6.6–9.3× (summarization) lower profiling cost.
  • Against DroidSpeak strengthened. DroidSpeak's search becomes more accurate when each expert is profiled separately and on more prompts, so we strengthened it to give it its best chance. Profiled for each expert, its distance to us shrinks from +0.81 to +0.47 [−0.27, +1.23] (Appendix L), and with 128 prompts per expert to +0.23 [−0.54, +0.98], where the two are comparable, but at 52–73× (QA) and 1,795–2,636× (summarization) our profiling cost.

Having matched the strongest labeled search, we asked how far our block is from the best any selection could reach. The oracle chooses the block with the test labels of the very questions it is scored on, and receives the tail as well; even so, our block trails it by only 1.0–3.2 percentage points of recovery at every budget from 2 to 10 layers (four multi-hop experts, Figure 4).

AuthorsOpenReview ↗

General Response (2/3)

3. Why the tail: the first isolated measurement we know of (JAeU W1/Q2; AdSQ W1). Letting each expert prefill its own question arises naturally in multi-agent systems, and prior work treats it as a setting rather than a method; as Reviewer JAeU notes, serving systems routinely share the context but not the question. Measured in isolation, it turns out to be a method: it has an effect of its own, it adds to one repair and not to another, and it pays only in one setting.

  • Why it has gone unmeasured. Because the tail is part of the setting, every prior protocol fixes it: DroidSpeak always computes a single suffix token, while LRAgent and KVComm always let the receiver prefill its own tokens. None varies it, so none can say what it contributes. Separating it takes the same block on the same experts with and without the tail, crossed with recomputation and with a low-rank correction (Table 3), and at every budget (Figure 4).
  • It adds to recomputation, not to a low-rank correction. Alone it recovers 43% of the accuracy drop on the ten experts, and recomputation takes that to 68% (Table 3). A low-rank correction adds nothing measurable on top of it (−0.19 [−0.54, +0.12] on the 48 multi-hop adapters; point 1), because the two repair the same part of the KV difference.
  • LRAgent's accuracy comes from the tail. BaseShared scores close to DroidSpeak in Table 2 (70.3 vs. 70.2 on QA) because its protocol has the expert prefill its own question. Separating it into a tail and a low-rank correction (Table A1, ten experts) shows that it recovers 2.0% of the accuracy drop with a single suffix token and 43.9% with the tail, against 43.1% for the tail alone: once the tail is held fixed, the correction contributes 0.8 points of recovery.
  • It is specific to the base-to-LoRA setting. On the fully fine-tuned pair DroidSpeak itself evaluates, the tail recovers only 16% and adds nothing measurable on top of the block (−2.30 [−5.72, +1.25]; Appendix N). We therefore present the tail as a base-to-LoRA method.

4. Serving: TTFT with the base prefill counted, measured on vLLM (AdSQ W2; JAeU W2/Q3; xrFL W1, W4, Q1, Q2). The reviewers are right that the 2.5–2.7× in Table 2 leaves out the base model's prefill. It is the receiver's TTFT once the shared context is cached, the accounting DroidSpeak uses, chosen to compare with it on equal terms rather than to flatter our method. The revision answers this and the request for a real inference engine together: a research prototype of the full method on vLLM 0.11 (one H100, Llama experts; Appendix O) measures time and memory with the base prefill and the recomputation counted, against stock vLLM at the same engine settings and KV memory. Its time and memory are measurements of the full method, but it is not a deployable serving system. We measure two workloads.

A relay of experts taking turns (JAeU W2/Q3). Following PrefillShare's hand-off, the base model prefills the shared context once. At each hand-off it prefills the tokens the previous expert generated, and the next expert recomputes its block from the stored hidden state and prefills its own instruction, so every hand-off is the single read of the base cache that our accuracy tables evaluate. With four experts and 128 generated tokens per turn:

Prefix length (tokens)4k8k13k16k28k
First hand-off, × sooner0.810.840.830.840.84
Each later hand-off, × sooner1.662.102.342.432.69
All four hand-offs, × sooner1.331.531.621.661.74
Whole relay, with decoding, × sooner1.011.041.091.111.21

The first hand-off also pays the base prefill; later hand-offs gain about as much at 32 and 256 generated tokens per turn (Appendix O, Table A23).

Several experts reading one document. The base model prefills a 13k-token context once, and 2, 4 or 8 experts read it at the same time, each with its own question:

Experts per context1248
First token, × sooner0.841.271.782.21
KV memory, stock ÷ ours0.701.131.622.08

End to end, the eight experts finish 1.37–1.95× sooner, and the speedups hold with one to four such contexts served at once. The saving thus starts from a context's second reader: AuLoRA pays off where several experts read the same context, the multi-LoRA setting it targets, and for a request that reaches a single expert once, direct prefill remains the right choice.

AuthorsOpenReview ↗

General Response (3/3)

5. Writing and presentation (AdSQ W4; JAeU Q1). Following the reviewers' suggestions, we revised the whole paper, not only the abstract and the introduction, which are shortened and fully restructured:

  • Sections 3–5 are rebuilt around the simplified method: Section 3 tests the decomposition on 120 adapters, Section 4 describes single-stage label-free profiling and when the tail is worth adding, and Section 5 re-scores the main tables on 2,000 questions per question-answering expert and moves the ablation into the main text (Table 3; the submission's Table 3 is now Table 4).
  • One statistical standard replaces several: the ±0.5-point equivalence region with its win/equivalent/lose counts and the second oracle definition are gone, and every cross-expert comparison reports the difference with its 95% interval and P(better) from one two-level Bayesian bootstrap; "comparable" means only that the interval contains zero.
  • Section 6 now explains, quoting LRAgent, why BaseShared is the variant that applies to independently trained adapters such as ours, Section 7 now sets out limitations and open questions, and the claim to be the first training-free recipe of its kind is removed (JAeU Q1).
  • A glossary defines each term and baseline protocol once (Appendix A), every label in Figures 1–4 is set at 8 pt or larger, and the appendix is reorganized around the simplified method, with tables in place of long prose.

Where to find the changes: glossary (Appendix A), BaseShared (C), target modules and rank (E), statistics (I.3), optional output probe (M), fully fine-tuned pair (N), vLLM (O).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.