- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PReCache, a training-free KV cache sharing framework for existing multi-LoRA agent systems. It comprises two designs. PreLRShared, the first time a shared context segment is processed, uses the hidden states produced at that moment to perform low-rank down-projections and precompute each agent's LR cache, so subsequent agents no longer need to re-run a full backbone prefill over the accumulated context to construct their own LR caches. ReBaseShared further disables the LoRA adapters and rebuilds a neutral base cache from adapter-free hidden states, reducing the dependence of the shared base cache on the previous agent's representations. The authors also propose lazy prefill and double-batching schedules for single-stream inference and concurrent serving, respectively. Experiments cover two 8B models, HotpotQA and ScienceQA, single-GPU single-stream inference, and vLLM concurrent serving. Results show PreLRShared achieves up to 3.1x TTFT speedup and 2.3x single-request throughput improvement over non-shared inference; ReBaseShared incurs on average about 1.1-point accuracy drop relative to non-shared execution while maintaining memory usage close to FullShared.
Critical strengths
- The problem is practically meaningful: although multi-LoRA agents share a backbone, different roles repeatedly prefill over the growing shared trajectory, causing significant redundant computation and memory use.
- The incremental contribution over existing LRAgent/BaseShared is clear. PreLRShared targets the issue that BaseShared still requires repeated backbone processing, eliminating the main redundant computation by precomputing compact LR caches for all agents.
- The motivation for ReBaseShared is sound: even when base projection weights are shared, the input hidden states are still influenced by the previous agent's adapter; rebuilding the base cache from adapter-free hidden states mitigates this contamination.
- The LP and DB schedules address different deployment settings. LP keeps reconstruction out of the next agent's TTFT; DB folds the neutral path into continuous batching to reduce queueing delay caused by separate reconstruction under high concurrency.
- The experimental dimensions are fairly comprehensive, covering accuracy, TTFT, throughput, tail latency, and memory, plus ablations on LoRA rank, QKVO adapter configurations, number of agents, and evaluation on natural generated trajectories.
- The efficiency gains are substantial in long-context settings: PreLRShared achieves up to 3.1x TTFT speedup and 2.3x throughput - ReBaseShared is overall the closest to NonShared across the four backbone-dataset combinations, dropping only about 1.1 points on average and clearly outperforming FullShared.
- The paper is well organized, and the notation, cache construction procedure, and the distinction between single-stream and concurrent schedules are conveyed fairly completely.
- The authors provide an anonymized code package and report full numbers in the appendix, standard deviations over 20 accuracy evaluations, and several ablations, showing good reproducibility awareness.
Critical weaknesses
- External validity is limited. The accuracy evaluation covers only LLaMA-3.1-8B and Ministral-8B, HotpotQA and ScienceQA, and a single three-role plan-action-reflect AutoAct workflow; this is not yet enough to support broad conclusions about general multi-LoRA agent systems.
- The main experiments use LoRA that acts only on query/value projections with rank 8. Although the appendix provides rank and QKVO efficiency ablations, it does not report end-to-end task accuracy under the QKVO configuration.
- The selective-recompute baseline is fixed at a 30% recompute ratio. The paper claims this point is close to the practical Pareto frontier, but it does not show accuracy-latency-memory Pareto curves across multiple budgets for DroidSpeak, CacheBlend, and RelayCaching separately, so the comparison may be affected by the chosen operating point.
- PreLRShared must down-project for every registered agent and maintain each agent's LR cache, so its computation and memory overhead grow linearly with the number of agents. The paper only scales to eight agents; it does not evaluate multi-tenant pools with dozens or hundreds of adapters, nor discuss how historical LR caches would be built when adapters are added dynamically.
- ReBaseShared requires one extra adapter-free backbone pass. In single-stream experiments its throughput is clearly lower than PreLRShared (e.g., ~1006 tok/s vs 1516 tok/s in the number-of-agents experiment); the paper lacks a clear quality-cost selection criterion for when neutral reconstruction is worth enabling.
Questions for authors
- Could you sweep multiple recompute ratios for DroidSpeak, CacheBlend, and RelayCaching separately and provide accuracy-TTFT-throughput-memory Pareto curves? Would the current fixed-30% comparison change the main ordering?
- How does PreLRShared scale to large adapter pools? Please report the down-projection cost, LR cache memory, and scheduling overhead across all A_k when the number of agents exceeds 8, e.g., 16, 32, or higher.
- If a previously unseen LoRA adapter is registered dynamically at runtime, it has no LR cache for the historical context. Would the system re-process the entire trajectory, save hidden states per layer, or use some other fallback mechanism?
- On Ministral-8B/ScienceQA, ReBaseShared is still 2.60 points below NonShared. Could the authors analyze the main failure types and explain why the reduced geometric error of the neutral base cache does not translate into a commensurate recovery of task accuracy?
- Please report task accuracy under the QKVO configuration, not just system efficiency. Do the additional key LR cache and the pre-RoPE expansion enlarge the cache approximation error?
3 comments
Response to Weakness 3 and Question 1
Pareto comparison across recomputation ratios: We swept the recomputation ratio of DroidSpeak, CacheBlend, and RelayCaching separately, and no ratio of any method achieves both higher accuracy and lower TTFT than PreLRShared or ReBaseShared (Table D). Increasing the ratio recovers accuracy but brings TTFT and peak memory toward those of NonShared, since each recomputed layer or token requires full-dimensional computation and agent-specific KV cache storage, whereas decreasing it brings accuracy toward that of FullShared. Overall, the 30% ratio lies near the accuracy-efficiency Pareto frontier of the selective recomputation baselines, and the main conclusion does not depend on the fixed 30% ratio.
Table D. HotpotQA accuracy (%) on LLaMA-3.1-8B averaged over the three difficulty levels, and p50 TTFT (s), latency (s), and memory (GB) under concurrent serving at 2 QPS with the 17.3k-token trajectory, across recomputation ratios. Latency denotes the sum of agent-call latencies per trajectory.
| Method | Ratio | Accuracy | TTFT | Latency | Memory |
|---|---|---|---|---|---|
| DroidSpeak | 10% | 34.30 | 0.32 | 22.21 | 0.88 |
| DroidSpeak | 30% | 35.23 | 0.40 | 25.21 | 0.93 |
| DroidSpeak | 50% | 35.97 | 0.41 | 25.76 | 0.98 |
| CacheBlend | 10% | 34.42 | 0.20 | 10.63 | 0.61 |
| CacheBlend | 30% | 34.93 | 0.24 | 12.43 | 0.72 |
| CacheBlend | 50% | 36.48 | 0.42 | 17.10 | 0.87 |
| RelayCaching | 10% | 35.75 | 0.39 | 14.30 | 0.67 |
| RelayCaching | 30% | 36.30 | 0.42 | 20.71 | 0.78 |
| RelayCaching | 50% | 36.68 | 0.51 | 29.02 | 0.97 |
| PreLRShared | - | 36.32 | 0.08 | 7.31 | 0.55 |
| ReBaseShared | - | 36.95 | 0.21 | 12.28 | 0.56 |
| NonShared | - | 37.52 | 0.43 | 29.00 | 1.11 |
Response to Weakness 4 and Question 2
Per-agent overhead: The overhead of PreLRShared grows linearly with the number of agents, but with a small constant. Each agent adds one rank-$ r $ down-projection, about 0.015% of the backbone computations per token, and one LR cache, about 0.4% of a full KV cache, for LLaMA-3.1-8B with QV adaptation at $ r = 8 $. Even with 64 agents, all LR caches together amount to about a quarter of one agent's full KV cache, whereas NonShared stores a full KV cache per agent.
Scaling to 16, 32, and 64 agents: In addition to the agent-number experiment of Table 16, we evaluated up to 64 registered agents on vLLM, where eight agents act in round robin and the rest never act. This setting also reflects a larger agent pool with more dynamic context sharing, in which only a subset of the agents, such as subagents or subsystems, reads a shared context. Although this is the least favorable case for precomputation, since most precomputed LR caches are never used, PreLRShared remains robust (Table E). From 8 to 64 agents, the peak KV cache memory grows only by the size of the added LR caches, and the TTFT and throughput change only gradually, although they include the down-projection and scheduling overhead of all registered agents.
Table E. Summed TTFT (s), throughput (tok/s), and peak KV cache memory (GB) of PreLRShared on vLLM with a single A100 GPU, on a 16.9k-token, 16-turn trajectory in which eight of the registered agents act in round robin.
| Agents | 8 | 16 | 32 | 64 |
|---|---|---|---|---|
| TTFT | 2.45 | 2.49 | 2.61 | 2.91 |
| Throughput | 1590 | 1567 | 1508 | 1377 |
| Peak KV cache memory | 2.16 | 2.22 | 2.36 | 2.62 |
Multi-tenant adapter pools: Precomputation does not need to cover every registered adapter. The LR caches of a trajectory are read only by the agents of the same workflow, so PreLRShared precomputes them only for the agents that can be invoked on that trajectory, regardless of the size of the adapter pool. For an adapter registered during execution, we describe the fallback in our response to Question 3.
We added these results to the revised paper.
Response to Weakness 5
Quality-cost selection criterion: Despite the throughput difference between the two designs, both PreLRShared and ReBaseShared achieve lower TTFT and higher throughput than all baselines except FullShared (Table 16), with accuracy comparable to or higher than the selective recomputation baselines (Table 2). Between the two, ReBaseShared further reduces the accuracy gap to NonShared at the cost of the adapter-free pass, while PreLRShared achieves higher throughput. Neutral reconstruction is therefore worth enabling when accuracy is the priority and the pass can run off the critical path, as in single-stream inference with tool calls (Table 5). PreLRShared is preferable when throughput is the binding constraint, as in concurrent serving near saturation (Table 12). Both designs use the same adapters and cache layout, so the choice is a runtime configuration.
We state the explanation more clearly in the revised paper.
Response to Question 3
Fallback for a newly registered adapter: A newly registered adapter constructs its LR cache over the history with a one-time pass, processing the accumulated trajectory with its own adapter while reusing the shared base cache. This is the procedure that BaseShared uses to construct each agent's LR cache over the context the agent has not processed, and it costs about one prefill over the history. In PReCache, this pass is needed only once before the agent's first turn, after which the adapter joins precomputation.
Storing per-layer hidden states: Storing the hidden states of every layer would allow a new LR cache to be built with down-projections only. However, they require twice the memory of a full KV cache for LLaMA-3.1-8B, which would offset the memory reduction of KV cache sharing on every trajectory. Since registering a new adapter is rare compared with agent turns, we use the one-time pass instead
Response to Question 4
We observe that Ministral-8B on ScienceQA is the most challenging condition for all KV cache sharing methods (Table 2). The role adapters of Ministral-8B shift the representations more strongly than those of LLaMA-3.1-8B, lowering the cosine similarity of the caches across agents (Table 1 and Appendix A.1 of [1]), and this is most pronounced on ScienceQA (Table F). On ScienceQA, the neutral base cache is also closer to the current agent's base cache for fewer tokens than on HotpotQA (Table F). A shared cache therefore deviates more from each agent's own cache in this setting. ReBaseShared is affected in the same way as the other methods, while remaining comparable to BaseShared and more accurate than the other sharing methods (Table 2).
The main failure type in this setting is out-of-function (OOF), where the agents keep repeating turns and fail to produce a final answer before reaching the maximum number of iterations (Table 18 of [1]). Such cases lose the chance to answer correctly at all, so the task accuracy depends on whether each trajectory reaches a final answer rather than on the average cache error, and the reduced geometric error relatively less translate proportionally into accuracy.
Table F. Average cosine similarity of the full and base caches across agent pairs on the same context, and the fraction of tokens with $ \Gamma_{\text{err}} > 1 $, for which the neutral base cache is closer to the current agent's base cache than the previous agent's base cache. All values are measured on Ministral-8B with independently trained adapters over the test-split trajectories of each benchmark.
| Benchmark | Full cache similarity | Base cache similarity | $ \Gamma_{\text{err}} > 1 $ (%) |
|---|---|---|---|
| HotpotQA | 0.9375 | 0.9664 | 89.9 |
| ScienceQA | 0.9218 | 0.9453 | 75.3 |
[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026.
Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.
We agree that broader evaluation would strengthen our conclusions. Open-source datasets for fine-tuning multi-LoRA agents are scarce, and we followed AutoAct (Qiao et al., 2024), a representative open-source dataset of multi-LoRA agent trajectories. Its trajectories are generated by LLaMA models and train role adapters stably on a few backbones, such as LLaMA and Mistral. Still, PReCache relies only on decomposing each adapted projection into its frozen and low-rank components, which holds for any LoRA-based agent regardless of the workflow.
Within these limits, we added Qwen3-1.7B, a backbone of a different model family and size, evaluated on HotpotQA easy, where its role adapters trained on these trajectories remain reliable. ReBaseShared is 1.00 point below NonShared and the most accurate sharing method (Table A). We added this result to the revised manuscript and plan to extend the evaluation as more multi-LoRA agent datasets become available.
Table A. HotpotQA easy accuracy (%) on Qwen3-1.7B with QV adaptation at $ r = 8 $.
Task accuracy under QKVO adaptation: We evaluated task accuracy on HotpotQA (hard) and ScienceQA (grades 9–12) under QKVO adaptation at $ r = 4 $, which matches the parameter count of QV adaptation at $ r = 8 $ (Table B). ReBaseShared remains within 0.8 points of NonShared and comparable to or more accurate than BaseShared, and the methods keep the same ordering as under QV adaptation. NonShared itself was less accurate under QKVO adaptation than under QV adaptation, which is why we use QV as our default configuration.
Table B. Accuracy (%) on LLaMA-3.1-8B HotpotQA (hard) and ScienceQA (grades 9–12) with QV adaptation at $ r = 8 $ (Table 2) and parameter-matched QKVO adaptation at $ r = 4 $.
Effect of the key LR cache and the pre-RoPE expansion: The pre-RoPE expansion is exact. The key LR cache stores the rank-$ r $ projection of the hidden states and expands it with the up-projection before RoPE, which reproduces the adapter contribution to the key exactly. RoPE only prevents the associativity-based reordering of Flash-LoRA-Attention, which affects only the computational cost (Appendix D.2). The approximation of PReCache comes only from the hidden states used to construct the shared caches, which under QKVO adaptation also affect the key cache through its LR component. Table C reports the relative errors of the key and value caches against those constructed by the current agent, over the agent transitions of Appendix A.2. Under QKVO adaptation, the key error stays at the level of the QV setting, where the key is not adapted, and at about half the value error, so the additional key LR cache does not enlarge the approximation error. ReBaseShared also shows slightly lower errors than PreLRShared in both settings.
Table C. Relative error (%) of the key and value caches against those constructed by the current agent, averaged over layers, tokens, and agent transitions on LLaMA-3.1-8B HotpotQA.
We added these results to the revised paper.