acceptodds
Under review as a conference paper at ICLR 2027

PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

Abstract

Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present **PReCache**, a training-free KV cache sharing framework with two designs, namely **PreLRShared** and **ReBaseShared**, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a TTFT speedup and a improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only points relative to inference without cache sharing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LoRA, Multi-LoRA, LLM agent, KV cache, KV cache sharing, Agentic serving, Efficient serving, Efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WqvnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PReCache, a training-free KV cache sharing framework for existing multi-LoRA agent systems. It comprises two designs. PreLRShared, the first time a shared context segment is processed, uses the hidden states produced at that moment to perform low-rank down-projections and precompute each agent's LR cache, so subsequent agents no longer need to re-run a full backbone prefill over the accumulated context to construct their own LR caches. ReBaseShared further disables the LoRA adapters and rebuilds a neutral base cache from adapter-free hidden states, reducing the dependence of the shared base cache on the previous agent's representations. The authors also propose lazy prefill and double-batching schedules for single-stream inference and concurrent serving, respectively. Experiments cover two 8B models, HotpotQA and ScienceQA, single-GPU single-stream inference, and vLLM concurrent serving. Results show PreLRShared achieves up to 3.1x TTFT speedup and 2.3x single-request throughput improvement over non-shared inference; ReBaseShared incurs on average about 1.1-point accuracy drop relative to non-shared execution while maintaining memory usage close to FullShared.

Critical strengths
  • The problem is practically meaningful: although multi-LoRA agents share a backbone, different roles repeatedly prefill over the growing shared trajectory, causing significant redundant computation and memory use.
  • The incremental contribution over existing LRAgent/BaseShared is clear. PreLRShared targets the issue that BaseShared still requires repeated backbone processing, eliminating the main redundant computation by precomputing compact LR caches for all agents.
  • The motivation for ReBaseShared is sound: even when base projection weights are shared, the input hidden states are still influenced by the previous agent's adapter; rebuilding the base cache from adapter-free hidden states mitigates this contamination.
  • The LP and DB schedules address different deployment settings. LP keeps reconstruction out of the next agent's TTFT; DB folds the neutral path into continuous batching to reduce queueing delay caused by separate reconstruction under high concurrency.
  • The experimental dimensions are fairly comprehensive, covering accuracy, TTFT, throughput, tail latency, and memory, plus ablations on LoRA rank, QKVO adapter configurations, number of agents, and evaluation on natural generated trajectories.
  • The efficiency gains are substantial in long-context settings: PreLRShared achieves up to 3.1x TTFT speedup and 2.3x throughput - ReBaseShared is overall the closest to NonShared across the four backbone-dataset combinations, dropping only about 1.1 points on average and clearly outperforming FullShared.
  • The paper is well organized, and the notation, cache construction procedure, and the distinction between single-stream and concurrent schedules are conveyed fairly completely.
  • The authors provide an anonymized code package and report full numbers in the appendix, standard deviations over 20 accuracy evaluations, and several ablations, showing good reproducibility awareness.
Critical weaknesses
  • External validity is limited. The accuracy evaluation covers only LLaMA-3.1-8B and Ministral-8B, HotpotQA and ScienceQA, and a single three-role plan-action-reflect AutoAct workflow; this is not yet enough to support broad conclusions about general multi-LoRA agent systems.
  • The main experiments use LoRA that acts only on query/value projections with rank 8. Although the appendix provides rank and QKVO efficiency ablations, it does not report end-to-end task accuracy under the QKVO configuration.
  • The selective-recompute baseline is fixed at a 30% recompute ratio. The paper claims this point is close to the practical Pareto frontier, but it does not show accuracy-latency-memory Pareto curves across multiple budgets for DroidSpeak, CacheBlend, and RelayCaching separately, so the comparison may be affected by the chosen operating point.
  • PreLRShared must down-project for every registered agent and maintain each agent's LR cache, so its computation and memory overhead grow linearly with the number of agents. The paper only scales to eight agents; it does not evaluate multi-tenant pools with dozens or hundreds of adapters, nor discuss how historical LR caches would be built when adapters are added dynamically.
  • ReBaseShared requires one extra adapter-free backbone pass. In single-stream experiments its throughput is clearly lower than PreLRShared (e.g., ~1006 tok/s vs 1516 tok/s in the number-of-agents experiment); the paper lacks a clear quality-cost selection criterion for when neutral reconstruction is worth enabling.
Questions for authors
  • Could you sweep multiple recompute ratios for DroidSpeak, CacheBlend, and RelayCaching separately and provide accuracy-TTFT-throughput-memory Pareto curves? Would the current fixed-30% comparison change the main ordering?
  • How does PreLRShared scale to large adapter pools? Please report the down-projection cost, LR cache memory, and scheduling overhead across all A_k when the number of agents exceeds 8, e.g., 16, 32, or higher.
  • If a previously unseen LoRA adapter is registered dynamically at runtime, it has no LR cache for the historical context. Would the system re-process the entire trajectory, save hidden states per layer, or use some other fallback mechanism?
  • On Ministral-8B/ScienceQA, ReBaseShared is still 2.60 points below NonShared. Could the authors analyze the main failure types and explain why the reduced geometric error of the neutral base cache does not translate into a commensurate recovery of task accuracy?
  • Please report task accuracy under the QKVO configuration, not just system efficiency. Do the additional key LR cache and the pre-RoPE expansion enlarge the cache approximation error?
3 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1

We agree that broader evaluation would strengthen our conclusions. Open-source datasets for fine-tuning multi-LoRA agents are scarce, and we followed AutoAct (Qiao et al., 2024), a representative open-source dataset of multi-LoRA agent trajectories. Its trajectories are generated by LLaMA models and train role adapters stably on a few backbones, such as LLaMA and Mistral. Still, PReCache relies only on decomposing each adapted projection into its frozen and low-rank components, which holds for any LoRA-based agent regardless of the workflow.

Within these limits, we added Qwen3-1.7B, a backbone of a different model family and size, evaluated on HotpotQA easy, where its role adapters trained on these trajectories remain reliable. ReBaseShared is 1.00 point below NonShared and the most accurate sharing method (Table A). We added this result to the revised manuscript and plan to extend the evaluation as more multi-LoRA agent datasets become available.

Table A. HotpotQA easy accuracy (%) on Qwen3-1.7B with QV adaptation at $ r = 8 $.

SettingNonSharedFullSharedRelayCachingBaseSharedPreLRSharedReBaseShared
Qwen3-1.7B, rank 837.1032.5534.9036.0535.8036.10

Response to Weakness 2 and Question 5

"Although the appendix provides rank and QKVO efficiency ablations, it does not report end-to-end task accuracy under the QKVO configuration."

"Please report task accuracy under the QKVO configuration, not just system efficiency."

Task accuracy under QKVO adaptation: We evaluated task accuracy on HotpotQA (hard) and ScienceQA (grades 9–12) under QKVO adaptation at $ r = 4 $, which matches the parameter count of QV adaptation at $ r = 8 $ (Table B). ReBaseShared remains within 0.8 points of NonShared and comparable to or more accurate than BaseShared, and the methods keep the same ordering as under QV adaptation. NonShared itself was less accurate under QKVO adaptation than under QV adaptation, which is why we use QV as our default configuration.

Table B. Accuracy (%) on LLaMA-3.1-8B HotpotQA (hard) and ScienceQA (grades 9–12) with QV adaptation at $ r = 8 $ (Table 2) and parameter-matched QKVO adaptation at $ r = 4 $.

MethodHotpotQA hard (QV)HotpotQA hard (QKVO)ScienceQA 9–12 (QV)ScienceQA 9–12 (QKVO)
NonShared30.9530.6577.5877.35
FullShared26.1525.1072.7572.02
BaseShared30.1529.6576.8876.17
PreLRShared28.2527.9076.7976.05
ReBaseShared30.2529.8577.1776.58

"Do the additional key LR cache and the pre-RoPE expansion enlarge the cache approximation error?"

Effect of the key LR cache and the pre-RoPE expansion: The pre-RoPE expansion is exact. The key LR cache stores the rank-$ r $ projection of the hidden states and expands it with the up-projection before RoPE, which reproduces the adapter contribution to the key exactly. RoPE only prevents the associativity-based reordering of Flash-LoRA-Attention, which affects only the computational cost (Appendix D.2). The approximation of PReCache comes only from the hidden states used to construct the shared caches, which under QKVO adaptation also affect the key cache through its LR component. Table C reports the relative errors of the key and value caches against those constructed by the current agent, over the agent transitions of Appendix A.2. Under QKVO adaptation, the key error stays at the level of the QV setting, where the key is not adapted, and at about half the value error, so the additional key LR cache does not enlarge the approximation error. ReBaseShared also shows slightly lower errors than PreLRShared in both settings.

Table C. Relative error (%) of the key and value caches against those constructed by the current agent, averaged over layers, tokens, and agent transitions on LLaMA-3.1-8B HotpotQA.

AdaptationCachePreLRSharedReBaseShared
QV ($ r = 8 $)Key12.2711.11
QV ($ r = 8 $)Value23.8421.70
QKVO ($ r = 4 $)Key12.3111.79
QKVO ($ r = 4 $)Value24.3123.52

We added these results to the revised paper.

Response to Weakness 3 and Question 1

Pareto comparison across recomputation ratios: We swept the recomputation ratio of DroidSpeak, CacheBlend, and RelayCaching separately, and no ratio of any method achieves both higher accuracy and lower TTFT than PreLRShared or ReBaseShared (Table D). Increasing the ratio recovers accuracy but brings TTFT and peak memory toward those of NonShared, since each recomputed layer or token requires full-dimensional computation and agent-specific KV cache storage, whereas decreasing it brings accuracy toward that of FullShared. Overall, the 30% ratio lies near the accuracy-efficiency Pareto frontier of the selective recomputation baselines, and the main conclusion does not depend on the fixed 30% ratio.

Table D. HotpotQA accuracy (%) on LLaMA-3.1-8B averaged over the three difficulty levels, and p50 TTFT (s), latency (s), and memory (GB) under concurrent serving at 2 QPS with the 17.3k-token trajectory, across recomputation ratios. Latency denotes the sum of agent-call latencies per trajectory.

MethodRatioAccuracyTTFTLatencyMemory
DroidSpeak10%34.300.3222.210.88
DroidSpeak30%35.230.4025.210.93
DroidSpeak50%35.970.4125.760.98
CacheBlend10%34.420.2010.630.61
CacheBlend30%34.930.2412.430.72
CacheBlend50%36.480.4217.100.87
RelayCaching10%35.750.3914.300.67
RelayCaching30%36.300.4220.710.78
RelayCaching50%36.680.5129.020.97
PreLRShared-36.320.087.310.55
ReBaseShared-36.950.2112.280.56
NonShared-37.520.4329.001.11

Response to Weakness 4 and Question 2

Per-agent overhead: The overhead of PreLRShared grows linearly with the number of agents, but with a small constant. Each agent adds one rank-$ r $ down-projection, about 0.015% of the backbone computations per token, and one LR cache, about 0.4% of a full KV cache, for LLaMA-3.1-8B with QV adaptation at $ r = 8 $. Even with 64 agents, all LR caches together amount to about a quarter of one agent's full KV cache, whereas NonShared stores a full KV cache per agent.

Scaling to 16, 32, and 64 agents: In addition to the agent-number experiment of Table 16, we evaluated up to 64 registered agents on vLLM, where eight agents act in round robin and the rest never act. This setting also reflects a larger agent pool with more dynamic context sharing, in which only a subset of the agents, such as subagents or subsystems, reads a shared context. Although this is the least favorable case for precomputation, since most precomputed LR caches are never used, PreLRShared remains robust (Table E). From 8 to 64 agents, the peak KV cache memory grows only by the size of the added LR caches, and the TTFT and throughput change only gradually, although they include the down-projection and scheduling overhead of all registered agents.

Table E. Summed TTFT (s), throughput (tok/s), and peak KV cache memory (GB) of PreLRShared on vLLM with a single A100 GPU, on a 16.9k-token, 16-turn trajectory in which eight of the registered agents act in round robin.

Agents8163264
TTFT2.452.492.612.91
Throughput1590156715081377
Peak KV cache memory2.162.222.362.62

Multi-tenant adapter pools: Precomputation does not need to cover every registered adapter. The LR caches of a trajectory are read only by the agents of the same workflow, so PreLRShared precomputes them only for the agents that can be invoked on that trajectory, regardless of the size of the adapter pool. For an adapter registered during execution, we describe the fallback in our response to Question 3.

We added these results to the revised paper.

Response to Weakness 5

Quality-cost selection criterion: Despite the throughput difference between the two designs, both PreLRShared and ReBaseShared achieve lower TTFT and higher throughput than all baselines except FullShared (Table 16), with accuracy comparable to or higher than the selective recomputation baselines (Table 2). Between the two, ReBaseShared further reduces the accuracy gap to NonShared at the cost of the adapter-free pass, while PreLRShared achieves higher throughput. Neutral reconstruction is therefore worth enabling when accuracy is the priority and the pass can run off the critical path, as in single-stream inference with tool calls (Table 5). PreLRShared is preferable when throughput is the binding constraint, as in concurrent serving near saturation (Table 12). Both designs use the same adapters and cache layout, so the choice is a runtime configuration.

We state the explanation more clearly in the revised paper.

Response to Question 3

Fallback for a newly registered adapter: A newly registered adapter constructs its LR cache over the history with a one-time pass, processing the accumulated trajectory with its own adapter while reusing the shared base cache. This is the procedure that BaseShared uses to construct each agent's LR cache over the context the agent has not processed, and it costs about one prefill over the history. In PReCache, this pass is needed only once before the agent's first turn, after which the adapter joins precomputation.

Storing per-layer hidden states: Storing the hidden states of every layer would allow a new LR cache to be built with down-projections only. However, they require twice the memory of a full KV cache for LLaMA-3.1-8B, which would offset the memory reduction of KV cache sharing on every trajectory. Since registering a new adapter is rare compared with agent turns, we use the one-time pass instead

Response to Question 4

We observe that Ministral-8B on ScienceQA is the most challenging condition for all KV cache sharing methods (Table 2). The role adapters of Ministral-8B shift the representations more strongly than those of LLaMA-3.1-8B, lowering the cosine similarity of the caches across agents (Table 1 and Appendix A.1 of [1]), and this is most pronounced on ScienceQA (Table F). On ScienceQA, the neutral base cache is also closer to the current agent's base cache for fewer tokens than on HotpotQA (Table F). A shared cache therefore deviates more from each agent's own cache in this setting. ReBaseShared is affected in the same way as the other methods, while remaining comparable to BaseShared and more accurate than the other sharing methods (Table 2).

The main failure type in this setting is out-of-function (OOF), where the agents keep repeating turns and fail to produce a final answer before reaching the maximum number of iterations (Table 18 of [1]). Such cases lose the chance to answer correctly at all, so the task accuracy depends on whether each trajectory reaches a final answer rather than on the average cache error, and the reduced geometric error relatively less translate proportionally into accuracy.

Table F. Average cosine similarity of the full and base caches across agent pairs on the same context, and the fraction of tokens with $ \Gamma_{\text{err}} > 1 $, for which the neutral base cache is closer to the current agent's base cache than the previous agent's base cache. All values are measured on Ministral-8B with independently trained adapters over the test-split trajectories of each benchmark.

BenchmarkFull cache similarityBase cache similarity$ \Gamma_{\text{err}} > 1 $ (%)
HotpotQA0.93750.966489.9
ScienceQA0.92180.945375.3

[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026.

Reviewer_iDCgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PReCache studies KV-cache reuse among agents sharing a backbone but using different LoRA adapters. PreLRShared precomputes agent-specific low-rank caches to avoid repeated backbone computation over shared history. ReBaseShared additionally uses adapter-free hidden states to reconstruct a neutral cache. Lazy prefill and double batching provide alternative schedules for this reconstruction.

Critical strengths
  1. The paper addresses a practical bottleneck in multi-LoRA agent systems. Repeated processing of shared context introduces redundant computation, and the method supports adapters with different down-projection weights.

  2. The proposed variants offer useful quality–efficiency trade-offs. PreLRShared reduces repeated computation, while ReBaseShared adds neutral reconstruction to recover quality. The LP/DB comparison further examines how scheduling affects performance under different loads.

  3. The evaluation combines natural tool-use trajectories with controlled efficiency and concurrent-serving experiments. Rank and agent-count studies provide additional evidence, including positive relative efficiency scaling from four to eight agents.

Critical weaknesses
  1. The benefit of the additional complexity is not sufficiently isolated. Although the paper compares its variants, it remains unclear when neutral reconstruction justifies its extra computation and scheduling cost over PreLRShared. ReBaseShared changes the hidden-state sources for both the base and low-rank caches, without separately identifying their contributions. Its theoretical advantage is also conditional. More targeted ablations and component-level profiling would clarify which changes are necessary and when their costs are justified.

  2. The evaluation leaves important generalization questions open. Both backbones are 8B models, and the main system measurements use controlled, extended, prefill-heavy traces; concurrent serving uses random token IDs. Natural tool-use tasks and rank experiments are useful, but do not fully establish performance across newer models, heterogeneous adapter configurations, or more dynamic interactions. The four-to-eight-agent experiment also fixes the total number of calls and does not report corresponding quality scaling.

  3. Cache management under interrupted execution is insufficiently specified. The paper enables chunked prefill, paged KV management, and prefix caching, but does not explain how shared base and per-agent low-rank caches are evicted and recovered when agents wait for tools under memory pressure. Important vLLM execution settings are also missing from the description. Consequently, the reported serving gains are difficult to assess when cache residency is interrupted.

Questions for authors
  1. Can you independently vary the hidden-state sources for the base and low-rank caches and profile the additional computation, to identify when ReBaseShared is preferable to the simpler PreLRShared?

  2. How do quality and relative efficiency scale with larger agent populations, deeper interactions, and heterogeneous LoRA configurations, particularly when only a subset of the precomputed agent caches is subsequently used?

  3. What vLLM version and execution settings are used? How are shared cache dependencies and recovery handled after eviction, and how much benefit remains under controlled tool delays and memory pressure?

5 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1 and Question 1

"The benefit of the additional complexity is not sufficiently isolated. Although the paper compares its variants, it remains unclear when neutral reconstruction justifies its extra computation and scheduling cost over PreLRShared."

Role of PreLRShared and ReBaseShared: Both PreLRShared and ReBaseShared preserve accuracy and eliminate the repeated prefill, reducing TTFT and thus end-to-end latency, a combination that no prior method achieves for existing adapters (Table 1). Between the two, ReBaseShared further reduces the accuracy gap to NonShared at the cost of one adapter-free pass, while PreLRShared achieves higher throughput. ReBaseShared is therefore preferable when accuracy is the priority and the pass can run off the critical path, as in single-stream inference with tool calls (Table 5). PreLRShared is preferable when throughput is the binding constraint, as in concurrent serving near saturation (Table 12). Both designs use the same adapters and cache layout, so the choice is a runtime configuration.


"ReBaseShared changes the hidden-state sources for both the base and low-rank caches, without separately identifying their contributions. … More targeted ablations and component-level profiling would clarify which changes are necessary and when their costs are justified."

"Can you independently vary the hidden-state sources for the base and low-rank caches and profile the additional computation, to identify when ReBaseShared is preferable to the simpler PreLRShared?"

Accuracy gain from the base cache and LR cache sources: Following the reviewer's suggestion, we varied the base cache and LR cache sources independently, where PreLRShared and ReBaseShared correspond to two of the four combinations (Table A). Reconstructing the base cache from the adapter-free hidden states accounts for most of the accuracy gain, while the LR cache source adds a smaller gain, since the adapter output is much smaller in magnitude than the base cache [1].

Table A. HotpotQA accuracy (%) on Ministral-8B for each combination of hidden-state sources. Each cache is constructed from either the previous agent's adapted hidden states or the adapter-free hidden states.

Base cache sourceLR cache sourceAdapter-free passAccuracy (%)
Previous agentPrevious agent (PreLRShared)✗34.77
Previous agentAdapter-free✓34.80
Adapter-freePrevious agent✓35.87
Adapter-freeAdapter-free (ReBaseShared)✓36.08

Component-level profiling: While the main paper reports TTFT, throughput, and end-to-end latency (Tables 5, 9, and 12), Table B further breaks down the execution time per trajectory to show where the additional cost arises. The down-projections take 3.1% of the execution time in PreLRShared and 0.9% in ReBaseShared, which computes only the current agent's down-projection in the current agent pass and those of the other agents within the adapter-free pass. The added cost of ReBaseShared is therefore the adapter-free pass, which processes each newly added segment once and costs about one prefill over the trajectory, regardless of the number of agents. With lazy-prefill, this pass runs after decoding, so ReBaseShared closely follows PreLRShared in TTFT, while its throughput is lower due to the longer total execution time (Table 9). With double-batching, the adapter-free path shares the serving batch with the adapted path, so its cost appears as a queueing delay once the server saturates (Table 12).

Table B. Execution time (s) per trajectory on the 33.7k-token single-stream trace of LLaMA-3.1-8B.

ComponentPreLRSharedReBaseShared (LP)
Current agent prefill and decoding20.6321.18
Down-projections in the current agent pass0.670.28
Adapter-free reconstruction–9.53
Total21.3030.99
TTFT12.4412.53

"Its theoretical advantage is also conditional."

Empirical validity of the condition: Although the advantage is conditional by design, Appendix A.2 shows that the condition holds in practice. The layer-wise average of $ \Gamma_{\text{err}} $ exceeds one at every layer on both backbones (Figure 7), and the neutral base cache is closer to the current agent's base cache than the previous agent's base cache for 75–90% of the tokens. This follows from role adapters shifting the shared context in weakly aligned directions, consistent with the near-orthogonality of adapters for different tasks reported in prior work [1, 2].

We added these analyses to the revised paper.

[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026. [2] Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.

Response to Weakness 2 and Question 2


"Both backbones are 8B models, and the main system measurements use controlled, extended, prefill-heavy traces; concurrent serving uses random token IDs."

Scope of the evaluation: We acknowledge that broader generalization would strengthen our work. However, open-source datasets for fine-tuning multi-LoRA agents are scarce, and we observed that the available LLaMA-generated trajectories train role adapters stably mainly on backbones such as LLaMA and Mistral. We therefore measured efficiency on controlled traces following prior work [1], as the system cost depends on the token counts and agent schedules rather than the token contents, and address each point below with additional experiments.


"The four-to-eight-agent experiment also fixes the total number of calls and does not report corresponding quality scaling."

"How do quality and relative efficiency scale with larger agent populations, … particularly when only a subset of the precomputed agent caches is subsequently used?"

Larger agent populations: In addition to the agent-number experiment of Table 16, we evaluated larger agent populations on vLLM to emulate the case where only a subset of the precomputed agent caches is subsequently used. This is the least favorable case for PreLRShared, as it precomputes the LR caches of all agents (Table C). We use up to 64 registered agents, of which eight act in round robin and the rest never act. Even in this worst case, the TTFT, throughput, and peak KV cache memory of PreLRShared change only gradually up to 64 agents, since each additional agent adds one rank-$ r $ down-projection and an LR cache of about 0.4% of a full KV cache. We note that evaluating quality with more agents requires additional role adapters and training trajectories, which existing datasets do not provide beyond the three available roles.

Table C. Summed TTFT (s), throughput (tok/s), and peak KV cache memory (GB) of PreLRShared on vLLM with a single A100 GPU, on a 16.9k-token, 16-turn trajectory in which eight agents act in round robin and the remaining agents are registered but never act.

Agents8163264
TTFT2.452.492.612.91
Throughput1590156715081377
Peak KV cache memory2.162.222.362.62

"Natural tool-use tasks and rank experiments are useful, but do not fully establish performance across newer models, heterogeneous adapter configurations, or more dynamic interactions."

Deeper interactions: We grouped the HotpotQA questions on Ministral-8B by the number of agent turns in their NonShared trajectories (Table D). ReBaseShared is the most accurate among the sharing methods in every group and stays within 1 point of NonShared, and its accuracy loss does not grow with the number of turns. The efficiency over longer interactions is evaluated with controlled traces of up to 66.4k tokens (Table 9).

Table D. HotpotQA accuracy (%) on Ministral-8B grouped by the number of agent turns in the NonShared trajectory.

Agent turnsQuestions (%)NonSharedFullSharedPreLRSharedReBaseShared
≤ 1051.750.5546.3448.3449.66
11–1228.725.9723.4623.7925.37
> 1219.716.0813.7515.1115.98

Heterogeneous adapters: Adapters trained for different roles already differ substantially, as adapters for different tasks produce nearly orthogonal weight updates and outputs on the same context [1, 2], and [1] observed the same cache similarity structure for debate-style agents. At higher ranks, which let each adapter deviate further from the backbone, ReBaseShared still stays within 1 point of NonShared from $ r = 4 $ to $ r = 32 $, with a smaller gap than PreLRShared at every rank (Table 19). We also note that PReCache supports mixed ranks or configurations by design, since each agent constructs and applies its LR cache with its own projections.

Newer models: We added Qwen3-1.7B evaluated on HotpotQA easy. ReBaseShared is 1.00 point below NonShared and the most accurate sharing method (Table E).

Table E. HotpotQA easy accuracy (%) on Qwen3-1.7B with QV adaptation at $ r = 8 $.

SettingNonSharedFullSharedRelayCachingBaseSharedPreLRSharedReBaseShared
Qwen3-1.7B, rank 837.1032.5534.9036.0535.8036.10

We added these results to the revised paper.

[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026.

[2] Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.

Response to Weakness 3 and Question 3


"Important vLLM execution settings are also missing from the description."

"What vLLM version and execution settings are used?"

vLLM version and execution settings: We used vLLM 0.28.0 on a single A100 80GB GPU with chunked prefill (8,192 tokens per step), paged KV cache management with a block size of 256 required by Flash-LoRA-Attention, prefix caching with LRU eviction, a GPU memory utilization of 0.4, at most 256 concurrent sequences, and first-come-first-served scheduling, for all methods.


"The paper enables chunked prefill, paged KV management, and prefix caching, but does not explain how shared base and per-agent low-rank caches are evicted and recovered when agents wait for tools under memory pressure."

"How are shared cache dependencies and recovery handled after eviction, …"

Cache dependencies and recovery after eviction: The LR caches of a segment are valid only with the shared base cache of the same segment and are produced in the same pass, so PReCache manages them as a unit. The LR caches are stored with the base cache blocks of the same token positions, so they are allocated, cached, and evicted together, and an LR cache never outlives its base cache. When an evicted segment is needed again, a single prefill restores its base cache and the LR caches of all agents, through the adapted path in PreLRShared and the adapter-free path in ReBaseShared, whereas NonShared recomputes the segment separately for each agent that reads it. While agents wait for tools, PReCache holds one shared base cache and compact LR caches of about 0.4% of a full KV cache per agent, which reduces the memory pressure that causes eviction.


"Consequently, the reported serving gains are difficult to assess when cache residency is interrupted."

"… how much benefit remains under controlled tool delays and memory pressure?"

Benefit under tool delays and memory pressure: We provide evaluation of concurrent serving with a 0.5 s tool delay between the turns of each trajectory, and a KV cache pool reduced to a GPU memory utilization of 0.35 (Table F). All methods slow down under this setting, but PreLRShared and ReBaseShared degrade the least and their advantage over NonShared widens, since NonShared holds an agent-specific KV cache for every agent and recomputes each evicted segment for each agent that reads it.

Table F. p50 TTFT (s) and throughput (tok/s) under concurrent serving at 2 QPS with the 17.3k-token trajectory, in the default setting (Table 12) and with a 0.5 s tool delay per turn and a KV cache pool reduced to a GPU memory utilization of 0.35.

MethodTTFT (default)TTFT (delay + reduced pool)Throughput (default)Throughput (delay + reduced pool)
NonShared0.432.10596270
CacheBlend0.240.851391760
BaseShared0.431.10758500
PreLRShared0.080.1523671950
ReBaseShared (DB)0.210.4214081080

We added the settings, the cache management, and these results to the revised paper.

Reviewer_rpNFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PReCache extends low-rank KV-cache sharing to LoRA adapters with different down-projections. PreLRShared precomputes agent-specific low-rank states to avoid repeated processing of shared history, while ReBaseShared additionally constructs shared states through an adapter-free path. The paper evaluates task accuracy and execution efficiency and introduces two schedules for managing reconstruction overhead.

Critical strengths
  1. The paper identifies a concrete computational limitation of BaseShared: constructing an agent-specific low-rank cache still requires substantial backbone processing. Precomputing the low-rank states removes this repeated computation without imposing a shared down-projection. This is a technically useful extension of the previous framework.

  2. The evaluation includes controlled efficiency measurements, naturally generated trajectory latency, repeated accuracy evaluations, and scheduling comparisons. These provide evidence of execution benefits within the evaluated setting. Reconstruction overhead is included in total execution time.

Critical weaknesses
  1. The practical significance of relaxing the shared-training constraint is insufficiently established.

The main extension over BaseLRShared is support for existing adapters with different down-projections. However, broader compatibility does not automatically establish a consequential practical benefit. The role-specific adapters in this paper are trained by the authors, yet the evaluation does not establish what would be lost by adopting LRAgent’s shared-A design, or what benefit of retaining independent down-projections justifies the additional computation and approximation error. Excluding BaseLRShared as a direct baseline under fixed weights is reasonable, but this evaluation condition does not itself establish the importance of the added flexibility. A representative workload involving existing adapters, or a system-level comparison with shared-A training that considers task quality, inference cost, and retraining cost, would strengthen the contribution. The current results primarily establish execution benefits conditional on independently trained adapters, leaving the significance of removing the training constraint insufficiently supported.

  1. Neutral reconstruction lacks a sufficiently characterized mechanism and validity boundary.

Appendix A provides a correct geometric condition for when the neutral state is closer to the target state. However, this condition is an elementary reformulation of the distance comparison; it does not explain why independently trained adapters should satisfy it. The averaged measurements provide empirical support, but do not characterize the frequency or severity of regressions. ReBaseShared also changes both the shared base-cache source and the other agents’ low-rank-cache source, leaving their individual contributions unresolved. Furthermore, the longest controlled efficiency traces extend beyond the naturally generated accuracy-evaluation trajectories, while the QKVO extension primarily evaluates efficiency. Thus, the results support an empirical heuristic within the tested configurations, but do not sufficiently characterize its reliability in the regimes motivating the method. This concern does not require a universal theoretical guarantee; it requires stronger empirical boundary and attribution evidence.

Questions for authors
  1. For the adapters trained in this study, what benefit is obtained by retaining independent down-projections rather than using shared-A training? Alternatively, can the authors demonstrate a representative workload where preserving existing adapters is an important deployment requirement?

  2. Can the authors separately evaluate changing the base-cache source and changing the other agents’ low-rank-cache source? What is the distribution of reconstruction error ratios, including cases where neutral reconstruction is worse?

  3. Can task quality and total execution cost be jointly evaluated on longer, semantically meaningful trajectories? Does the reported quality preservation also hold under QKVO adaptation?

5 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1 and Question 1

The benefit lies in removing the co-training requirement, and PReCache also applies to shared-A adapters. When all role adapters can be trained jointly, shared-A training is an effective choice, achieving higher accuracy (Table A). PReCache also applies to these adapters, where PreLRShared becomes identical to BaseLRShared by design since the shared down-projection produces the same LR cache for all agents, and ReBaseShared still achieves higher accuracy than BaseLRShared. PReCache, however, targets the cases where such joint training is not possible, sharing the KV cache across adapters as they are given, whereas shared-A training requires all adapters to be trained together around one down-projection.

Shared-A training is impractical in common multi-LoRA deployments, because its co-training requirement cannot be met in the following cases. First, adapters from different sources: role adapters from other teams or tenants, including those released on public hubs, are trained independently, often without access to their training data and recipes. Second, evolving systems: adding or updating a role requires either retraining all adapters jointly or constraining the new adapter to the existing down-projection. Third, heterogeneous configurations: BaseLRShared requires all adapters to share one down-projection and thus the same rank, which restricts the configuration of each role adapter, whereas PReCache allows each agent its own rank and configuration.

For Table A, we retrained the role adapters with a shared down-projection following the shared-A design of LRAgent and evaluated all methods in our inference framework. BaseLRShared loses substantial accuracy on independent adapters, as it expands an LR cache computed with another agent's down-projection. In contrast, ReBaseShared on independent adapters already reaches accuracy higher than BaseLRShared on shared-A adapters, without retraining. Additionally, each time an agent role is added, retraining requires 3.1 GPU-hours.

Table A. Comparison with shared-A training on LLaMA-3.1-8B. Accuracy is averaged over HotpotQA, and TTFT and throughput are measured on the 33.7k-token single-stream trace. Retraining denotes the GPU-hours on an A6000.

AdaptersMethodAccuracy (%)TTFT (s)Throughput (tok/s)Retraining (GPU-h)
IndependentNonShared37.5233.72768None
IndependentBaseLRShared30.2312.311607None
IndependentPreLRShared36.3212.441581None
IndependentReBaseShared36.9512.531087None
Shared-ANonShared38.1033.587723.1
Shared-ABaseLRShared = PreLRShared36.6712.1716303.1
Shared-AReBaseShared37.5012.2111053.1

We added this comparison to the revised paper.

Response to Weakness 2 (former part) and Question 2


"Appendix A provides a correct geometric condition for when the neutral state is closer to the target state. However, this condition is an elementary reformulation of the distance comparison; it does not explain why independently trained adapters should satisfy it."

Condition on the orthogonality of role adapters: The condition in Appendix A.1 builds on the property that heterogeneous role adapters are weakly aligned. When the offsets of the previous and current agents (the difference between each agent's base cache and the neutral base cache on the same context) are weakly aligned, the neutral base cache is closer to the current agent's base cache than the previous agent's base cache is. Previous works, which we cite in Appendix A.2, show that the weight updates and adapter outputs of different tasks are nearly orthogonal, with cosine similarities close to zero [1, 2]. Following the multi-agent setting of [1], we reproduced this property for our independently trained role adapters, and further found that the resulting base cache offsets remain weakly aligned (Figure 7), and thus satisfy the condition (Appendix A.2 and Table B below).


"The averaged measurements provide empirical support, but do not characterize the frequency or severity of regressions."

"What is the distribution of reconstruction error ratios, including cases where neutral reconstruction is worse?"

Token-wise comparison of neutral reconstruction: Following the reviewer's suggestion, we report the per-token distribution of $ \Gamma_{\text{err}} $ for the base cache (Table B), where $ \Gamma_{\text{err}} > 1 $ means that the neutral base cache approximates the current agent's base cache better than the base cache produced by the previous agent. The neutral base cache is better for 75–90% of the tokens on both backbones and benchmarks. Even at the 5th percentile, $ \Gamma_{\text{err}} $ stays above 0.7. Therefore, the offsets of different role adapters are weakly aligned for most tokens, and even where they are aligned, the alignment remains limited. As shown in Figure 7 of Appendix A.2, these cases are relatively more frequent in later layers.

Table B. Per-token distribution of $ \Gamma_{\text{err}} $ for the base cache over layers, tokens, and agent transitions. $ \Gamma_{\text{err}} > 1 $, or equivalently $ \rho > 2\cos\theta $, indicates that the neutral base cache is closer to the current agent's base cache than the previous agent's base cache.

BackboneBenchmark$ \Gamma_{\text{err}} > 1 $ (%)p5p50
LLaMA-3.1-8BHotpotQA86.20.831.24
LLaMA-3.1-8BScienceQA79.00.731.13
Ministral-8BHotpotQA89.90.841.25
Ministral-8BScienceQA75.30.741.19

"ReBaseShared also changes both the shared base-cache source and the other agents' low-rank-cache source, leaving their individual contributions unresolved."

"Can the authors separately evaluate changing the base-cache source and changing the other agents' low-rank-cache source?"

Accuracy gain from the base cache and LR cache sources: For further examination, we varied the base cache and LR cache sources independently, where PreLRShared and ReBaseShared correspond to two of the four combinations (Table C). Reconstructing the base cache from the adapter-free hidden states accounts for most of the accuracy gain, while the LR cache source adds a smaller gain. This is because the adapter output is much smaller in magnitude than the base cache [1], so changing the hidden states that feed each agent's own adapter has a limited effect, whereas the same change in the base cache directly affects the dominant part of the KV cache.

Table C. HotpotQA accuracy (%) on Ministral-8B for each combination of hidden-state sources. Each cache is constructed from either the previous agent's adapted hidden states or the adapter-free hidden states.

Base cache sourceLR cache sourceAdapter-free passAccuracy (%)
Previous agentPrevious agent (PreLRShared)✗34.77
Previous agentAdapter-free✓34.80
Adapter-freePrevious agent✓35.87
Adapter-freeAdapter-free (ReBaseShared)✓36.08

We added this discussion and analysis to Appendix A.2 of the revised paper.

[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026. [2] Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.

Response to Weakness 2 (latter part) and Question 3


"Furthermore, the longest controlled efficiency traces extend beyond the naturally generated accuracy-evaluation trajectories …"

"Can task quality and total execution cost be jointly evaluated on longer, semantically meaningful trajectories?"

Quality and cost on longer trajectories: We extended the naturally generated trajectories by removing the limit on the retrieved text, which previously kept a single representative passage per observation. This keeps the task semantics while the trajectories reach up to 30.9k tokens and 12.1k tokens on average (Table D). ReBaseShared remains within 1 point of NonShared and is the most accurate among the sharing methods. PreLRShared and ReBaseShared also achieve the lowest end-to-end latency with lower TTFT than NonShared. Hence, in longer contexts, where the accuracy gap between NonShared and FullShared widens, our methods retain similar effectiveness over the representative prior methods, as in Table 2.

Table D. Accuracy (%), TTFT (s), and model and end-to-end latency (s) per trajectory on LLaMA-3.1-8B ScienceQA 9–12 with longer retrieved contexts.

MethodAccuracy (%)TTFT (s)Model Latency (s)E2E Latency (s)
NonShared77.7313.5422.0033.72
FullShared71.698.6020.6835.59
RelayCaching75.1212.5522.2433.46
BaseShared76.5812.4220.5031.94
PreLRShared76.2210.7018.1128.69
ReBaseShared76.7610.9118.5929.21

"… while the QKVO extension primarily evaluates efficiency."

"Does the reported quality preservation also hold under QKVO adaptation?"

Quality under QKVO adaptation: We additionally evaluated task accuracy on HotpotQA (hard) and ScienceQA (grades 9–12) under QKVO adaptation at $ r = 4 $, which matches the parameter count of QV adaptation at $ r = 8 $ (Table E). ReBaseShared remains within 0.8 points of NonShared and comparable to or more accurate than BaseShared, and the methods keep the same ordering as under QV adaptation. NonShared itself was less accurate under QKVO adaptation than under QV adaptation, which is why we use QV as our default configuration.

Table E. Accuracy (%) on LLaMA-3.1-8B HotpotQA (hard) and ScienceQA (grades 9–12) with QV adaptation at $ r = 8 $ (Table 2) and parameter-matched QKVO adaptation at $ r = 4 $.

MethodHotpotQA hard (QV)HotpotQA hard (QKVO)ScienceQA 9–12 (QV)ScienceQA 9–12 (QKVO)
NonShared30.9530.6577.5877.35
FullShared26.1525.1072.7572.02
BaseShared30.1529.6576.8876.17
PreLRShared28.2527.9076.7976.05
ReBaseShared30.2529.8577.1776.58

We added both analyses to the revised paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to the Reviewers

We sincerely thank all reviewers for their thoughtful feedback and insightful questions. We kindly ask the reviewers to refer to the updated manuscript for all table, figure, section, and appendix numbers in our responses.

We summarize here two points that appeared across the reviews: (a) the roles of PreLRShared and ReBaseShared, and (b) the generality of PReCache beyond the default configuration. We have updated the manuscript with clearer explanations and additional experimental results.

Roles of PreLRShared and ReBaseShared

Both PreLRShared and ReBaseShared preserve accuracy while eliminating the repeated prefill, a combination that remains challenging for prior methods with existing adapters (Table 1). Between the two, PreLRShared achieves higher throughput, while ReBaseShared further reduces the accuracy gap to NonShared with one adapter-free pass. Below, we show where the accuracy gain of ReBaseShared comes from, how the cost of its pass appears, and which design suits which deployment.

Source of the accuracy gain: Table G1 varies the hidden-state sources of the shared base cache and the other agents' LR caches independently, where PreLRShared and ReBaseShared correspond to two of the four combinations. Reconstructing the base cache from the adapter-free hidden states accounts for most of the accuracy gain, as the neutral base cache is closer to the current agent's base cache than the previous agent's base cache for 75–90% of the tokens (Appendix A.2). Changing only the LR cache source leaves the accuracy nearly unchanged, since the adapter output is much smaller in magnitude than the base cache [1].

Table G1. HotpotQA accuracy (%) on Ministral-8B for each combination of hidden-state sources. Each cache is constructed from either the previous agent's adapted hidden states or the adapter-free hidden states. NonShared achieves 36.72%.

Base cache sourceLR cache sourceAdapter-free passAccuracy (%)
Previous agentPrevious agent (PreLRShared)✗34.77
Previous agentAdapter-free✓34.80
Adapter-freePrevious agent✓35.87
Adapter-freeAdapter-free (ReBaseShared)✓36.08

Cost of the adapter-free pass and selection criterion: The adapter-free pass processes each newly added segment once, which costs about one prefill over the trajectory regardless of the number of agents. With lazy-prefill, it runs after decoding, so ReBaseShared keeps a TTFT close to that of PreLRShared, while its throughput is lower due to the longer total execution time (Table 9). ReBaseShared is therefore preferable when accuracy is the priority and the pass can run off the critical path, as in single-stream inference with tool calls (Table 5), whereas PreLRShared is preferable when throughput is the binding constraint, as in concurrent serving near saturation (Table 12).

We added the source ablation, a component-level breakdown of the execution time, and this selection criterion to the revised paper.

[1] Jeon et al. LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents. ICML 2026.

Generality beyond the Default Configuration

We further evaluated the accuracy preservation of PReCache beyond the default QV adaptation and the two 8B backbones. PReCache relies only on decomposing each adapted projection into its frozen and low-rank components, so it does not assume a particular set of adapted projections or a particular backbone.

QKVO adaptation: Under parameter-matched QKVO adaptation, ReBaseShared remains within 0.8 points of NonShared and comparable to or more accurate than BaseShared, and the methods keep the same ordering as under QV adaptation (Table G2).

Table G2. Accuracy (%) on LLaMA-3.1-8B HotpotQA (hard) and ScienceQA (grades 9–12) with QV adaptation at $ r = 8 $ (Table 2) and parameter-matched QKVO adaptation at $ r = 4 $.

MethodHotpotQA hard (QV)HotpotQA hard (QKVO)ScienceQA 9–12 (QV)ScienceQA 9–12 (QKVO)
NonShared30.9530.6577.5877.35
FullShared26.1525.1072.7572.02
BaseShared30.1529.6576.8876.17
PreLRShared28.2527.9076.7976.05
ReBaseShared30.2529.8577.1776.58

Backbone of a different family and size: On Qwen3-1.7B evaluated on HotpotQA easy, ReBaseShared is 1.00 point below NonShared and the most accurate sharing method (Table G3).

Table G3. HotpotQA easy accuracy (%) on Qwen3-1.7B with QV adaptation at $ r = 8 $.

SettingNonSharedFullSharedRelayCachingBaseSharedPreLRSharedReBaseShared
Qwen3-1.7B, rank 837.1032.5534.9036.0535.8036.10

We added these results to the appendix of the revised paper.


We thank the reviewers again for their constructive feedback and hope that our responses resolve their concerns.

Area_Chair_7ZvbOpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best,

Your AC

We sincerely thank the reviewers for their detailed feedback and the AC for guiding the discussion. We would be grateful if the reviewers could check our responses within the discussion period and let us know if any concern remains. We believe that a few critical gaps may still remain between some of the concerns and our presentation of the actual contributions, and we have clarified them in the discussion. The main points are summarized below.

Strengths noted by the reviewers: The reviewers found the problem practically meaningful (iDCg, Wqvn) and noted that precomputing the low-rank caches removes the repeated backbone processing without imposing a shared down-projection (rpNF, iDCg, Wqvn). They also found that the two designs and their schedules offer useful quality–efficiency trade-offs for different deployments (iDCg, Wqvn), and that the evaluation is comprehensive across natural trajectories, controlled traces, and concurrent serving (rpNF, iDCg, Wqvn). After the rebuttal, Reviewer iDCg found that the additional experiments "addressed my main concerns" and updated the rating, and Reviewer rpNF noted that PReCache "reduces end-to-end latency with a small accuracy loss on longer task trajectories".

Significance and mechanism

  • Independently trained adapters (rpNF): ReBaseShared on independent adapters is more accurate than BaseLRShared on shared-A adapters, without the joint retraining that shared-A training requires whenever a role is added (Table 24). Such adapters already arise in practice, as public hubs host agentic adapters from many independent developers, and enterprise workflows switch among task-specific adapters over the same context.

  • Why neutral reconstruction helps (rpNF, iDCg): The adapter outputs of different roles are nearly orthogonal, and our measurements confirm that their base cache offsets remain weakly aligned. As a result, the neutral base cache is closer to the current agent's base cache than the previous agent's for most tokens (75–90%) on both backbones and benchmarks (Figure 7 and Table 3).

  • Source of the accuracy gain (rpNF, iDCg): Varying the base cache and LR cache sources separately shows that reconstructing the base cache accounts for most of the gain, as the base cache dominates the KV cache while the adapter output is much smaller (Table 4).

  • Cost and choice of ReBaseShared (iDCg, Wqvn): The added cost of ReBaseShared is one adapter-free pass over each new segment, about one prefill over the trajectory regardless of the number of agents (Table 10). With lazy prefill, its TTFT stays close to that of PreLRShared, so ReBaseShared suits accuracy-critical single-stream inference, while PreLRShared suits throughput-bound concurrent serving.

Empirical validation

  • Longer trajectories (rpNF): On naturally generated trajectories of up to 30.9k tokens, ReBaseShared stays close to NonShared and is the most accurate sharing method, while our two designs achieve the lowest end-to-end latencies (Table 6).

  • Generality (rpNF, iDCg, Wqvn): ReBaseShared remains the most accurate sharing method under parameter-matched QKVO adaptation and on a smaller backbone from a different model family (Tables 22 and 8). Its accuracy loss also does not grow with the number of agent turns.

  • Larger agent pools (iDCg, Wqvn): With up to 64 registered agents, the TTFT, throughput, and memory of PreLRShared change only gradually, since each agent adds only a compact LR cache (Table 17). A newly registered adapter joins after a one-time pass over the history.

  • Recomputation budget (Wqvn): Across recomputation ratios, no configuration of DroidSpeak, CacheBlend, or RelayCaching achieves both higher accuracy and lower TTFT than PreLRShared or ReBaseShared (Table 18).

  • Tool delays and memory pressure (iDCg): The LR caches are allocated and evicted with the base cache blocks of the same tokens, and one prefill restores all of them. With tool delays and a reduced KV cache pool, our two designs degrade the least, and their advantage over NonShared widens (Table 11).

Results addressing the major concerns are included in the revised manuscript. We are grateful for the reviewers' engagement and hope this summary helps the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.