acceptodds
Under review as a conference paper at ICLR 2027

KVLease: Dynamic KV Memory Sharing for Mixture-of-Models Inference

Abstract

Language-model applications can combine several models to answer one request. In this Mixture of Models (MoM) setting, workflows connect parallel and sequential model calls through their intermediate outputs. Stage transitions shift key–value (KV) cache demand, but overlapping requests keep earlier-stage engines busy. A downstream model can therefore have its inputs ready yet wait for capacity that those engines repeatedly reuse, delaying the complete answer. Fixed partitions strand capacity; lending idle memory alone cannot recover pages from busy borrowers. The challenge is to serve the next stage without interrupting existing uses. We present KVLease, a KV-memory sharing system designed for MoM workflows. It couples lending and per-model allocation guarantees with targeted return: selected borrowed pages stop accepting new allocations, then return after existing references and GPU accesses finish. Capacity is shared per GPU; model-specific KV contents stay private. Our analysis characterizes how workflow stages correlate memory demand, separates partition imbalance from temporal sharing, and establishes conditional return bounds. Replicated evaluations across heterogeneous GPU platforms, serving engines (e.g., vLLM and SGLang), and diverse MoM workloads show 28–53% lower end-to-end p95 latency than equal-token fixed allocations at the reported memory-constrained operating points, over 50% higher completed-request throughput, and up to 35 percentage points higher service-level objective (SLO) attainment. Against kvcached at matched capacity, p95 falls a further 24–31% at high load. These gains preserve model computation, with identical outputs in paired correctness checks.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture of Models, LLM Serving, KV Cache, Dynamic Memory Sharing, Multi-Model Serving, GPU Memory Management, Resource Allocation, Inference Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Tz1JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents KVLease, a memory sharing system for mixture of models workflows. In the mixture of models setting, a request is processed by several models, e.g., an aggregator aggregates results from several proposers. The paper characterizes memory usage for mixture of models workflows. Based on the findings, it designs KVLease which coordinates memory lending and return across different models. The evaluation compares KVLease with a couple of baselines.

Critical strengths
  • The paper presents KVLease for memory sharing in the mixture of models setting. KVLease coordinates memory lending and return across different models.
  • The evaluation compares KVLease with a few baselines. The results show the benefits of KVLease.
Critical weaknesses
  • The evaluation uses synthetic workloads. It is unclear whether the workloads are representative of realistic scenarios.
Questions for authors
  1. In Section 2, the paper says "In a single-replication matched-volume diagnostic at two arrival rates, fixed-slice MoM workflows have 16–43% higher p95 than independent-call controls" and "Across the three primary evaluation settings, pooling raises generated tokens per engine step from 35.9 to 47.1, 36.6 to 45.7, and 32.6 to 53.1: 20–39% fewer steps per generated token". Can you clarify what the setup is? What hardware, models, and benchmarks do you use?

  2. The paper says that KVLease is a "KV-memory sharing system". Does KVLease only share the GPU memory of KV Cache? When there are multiple models, the total size of the model parameters can be very large. Does KVLease dynamically allocate the GPU memory for model parameters, and swap model parameters between GPU memory and host memory?

  3. For the workloads in Section 5.1, the paper mentions that "The main collaborative mix combines a single-model service, two-round three-model debate, and three-proposer aggregation on GSM8K, MATH-500, MMLU-Pro, and GPQA". How does the mix combine the tasks? Does the mix randomly choose these tasks?

  4. The SLO attainment is set as "the fraction of arrivals completing within 60 seconds". Why do you pick 60 seconds?

  5. Can you clarify the number of GPUs used in each experiment? Does B200 experiments use one B200 GPU and RTX experiments use two RTX GPUs? For the two RTX GPU experiments, how do the models share the two GPUs? How do you decide which models share which GPU?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Tz1J (1/3)

Thank you for the review, and for identifying the setup details that need clarification. We evaluated KVLease with replays of public production traces and with long real documents. KVLease keeps its latency benefit in these tests. Below we also explain the workload construction, the scope of memory sharing, the latency target and the GPU placement.

W1. Workload realism

Question. Are the workloads representative of realistic serving scenarios, and not only synthetic?

What we have done and where. We added replays of two public production traces: the Azure LLM inference trace and BurstGPT. In the Azure tests, both the arrival times and the service mix come from the trace. We also added long real inputs, which are questions about QMSum meeting transcripts. The results are in Section 5.6 and Appendices B.6 and B.21 of the revised paper. The deployment with larger models is in Appendix B.20.

Answer. KVLease's benefit persists on the production-trace workloads and on long inputs. The service demand in these traces changes over time, and this is the case where sharing memory helps.

Evidence and experiments. In the submitted paper, every prompt is a real benchmark question executed through an actual workflow, and two of the three settings of Table 1 already use Azure-derived arrival times. The new tests also take the service mix from the traces. The first test replays the Azure trace on the paper's main setting (one B200, five models, 32 GiB), with the paper's workflows. The second replays the same trace with questions about QMSum meeting transcripts on one B200. The third replays BurstGPT at its original timestamps, in the window with the largest mix change, on two RTX PRO 6000 GPUs.

Workload and replication  Offered workflows/s  Fixed slices p95  KVLease p95  Paired p95 change and range  SLO attainment
Azure trace with the paper's workflows, 6 periods2.0257.1 s64.1 s−64% [−83%, −4%]0.51 → 0.88
Azure trace with long documents, 18 sets1.6–2.8226.0 s47.4 s−72% [−84%, −23%]0.33 → 0.98
BurstGPT, 3 seeds2.0219.8 s116.1 s−47% [−52%, −43%]0.57 → 0.68

The range is across trace periods, request sets or seeds. SLO attainment is the fraction of workflows that finish within 60 s. KVLease has a lower p95 in every reported period, request set and seed. Every system replays identical arrivals. The two-B200 deployment with tensor-parallel models additionally finishes 96.3% of workflows within 60 s, against 28.7% for a static split profiled on earlier traffic.

Analysis. In the Azure trace, the share of requests that goes to each service shifts by a median of 24 percentage points over twelve hours. Changing demand strands capacity in fixed slices: busy models wait while idle models hold memory. This is the setting where lending and return are useful. So the benefit of KVLease persists under trace replay.

Commitment to revise. We will report the trace results in Section 5.6. We will document the trace periods, the mapping of services to workflows and the arrival rates in Appendix B.21, and the BurstGPT replay in Appendix B.6.

AuthorsOpenReview ↗

Response to Reviewer Tz1J (2/3)

Q1. Setup of the motivating measurements

Question. What hardware, models and benchmarks produce the 16–43% higher p95 of dependent workflows and the tokens-per-engine-step results in Section 2?

What we have done and where. We clarified the setup of both results in Section 2 and Appendix B.1 of the revised paper.

Answer. The two results come from two different experiments. Both use the five models of Section 5.1 (Gemma-2-9B-it, gpt-oss-20b, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct and Granite-3.3-8B-Instruct) and questions from GSM8K, MATH-500, MMLU-Pro and GPQA. The 16–43% result is a diagnostic on one B200 with 32 GiB of KV memory in fixed slices, at 1.5 and 2 workflows/s, with one run at each rate. It sends the same model calls either as workflows, where a call starts only when the calls it depends on have finished, or as independent calls. The tokens-per-step result compares fixed slices with KVLease in the three main settings of Table 1, with three seeds each: the collaborative workload on one B200, the routing workload on one B200, and the collaborative workload on two RTX PRO 6000 GPUs.

Commitment to revise. We will state the setup of both results in Section 2 and Appendix B.1.

Q2. Scope of KV memory sharing

Question. Does KVLease share only the GPU memory of the KV cache, or does it also allocate GPU memory for model parameters and swap them between GPU and host memory?

What we have done and where. We clarified the scope of memory sharing and the assumption that the model weights stay on the GPU. The clarification is in Section 4 and in the setup table of Section 5.1 of the revised paper.

Answer. KVLease shares only KV-cache memory. The model weights are loaded once and stay on the GPU, and KVLease does not swap them between GPU memory and host memory. In a workflow, the KV demand moves from one model to another within seconds, while every model stays loaded. Sharing the KV memory avoids tying every model to a fixed KV reservation. The KV contents of each model stay private, and only the capacity is shared. Weight swapping addresses a different capacity problem. It could be combined with KVLease, but it is not implemented or evaluated here.

Commitment to revise. We will state KV-only sharing, the resident-weight requirement and the tested model placements explicitly in Section 4 and in the setup table of Section 5.1.

Q3. Construction of the workflow mix

Question. How does the mix of Section 5.1 combine the datasets and workflow types, and are the tasks chosen randomly?

What we have done and where. We specified the class schedule, the random selection, the question sampling and the arrival process in Section 5.1 and Appendix B.1 of the revised paper.

Answer. Yes. Requests are sampled with a fixed seed inside a controlled sequence of dominant workflow classes, so every compared system receives identical requests.

Evidence and experiments. The four classes are:

Dataset   Workflow
GSM8KSingle-model service (Gemma)
MATH-500   Two-round debate among gpt-oss, Granite and Gemma
GPQAThe same two-round debate
MMLU-Pro   Llama, Mistral and Granite propose; gpt-oss aggregates

A run consists of 60-second phases with one dominant class. In each phase, 80% of arrivals select the dominant class and the remainder select uniformly across all four classes. A question is then drawn uniformly from the recorded questions of the selected dataset. Arrival times are Poisson in the B200 collaborative rows of Table 1 and Azure-derived in its routing and RTX rows.

Analysis. Different workflow classes exercise different model groups. Known phase transitions show how lending and return follow those changes, and seeded sampling makes the comparison reproducible. The trace-derived mix in W1 provides a separate test without the designed phase schedule.

Commitment to revise. We will give the sampling probability, the phase schedule, the question selection, the seeding and the arrival sources in Section 5.1 and Appendix B.1.

AuthorsOpenReview ↗

Response to Reviewer Tz1J (3/3)

Q4. Choice of the 60-second target

Question. Why is SLO attainment defined as the fraction of arrivals completing within 60 seconds?

What we have done and where. We explained the target using the light-load workflow latency, and we evaluated three other targets. The explanation is in Section 5.1 of the revised paper, and the results at the other targets are in Appendix B.3.

Answer. Sixty seconds is a queue-sensitive target for these multi-call workflows. It sits just above the latency of a workflow that is served without extra queueing. The advantage of KVLease also appears at the other tested targets.

Evidence and experiments. At 1 workflow/s on the B200, the median workflow latency is 13–15 s and the p95 is 44–48 s. The table gives the share of workflows that finish within each target, fixed slices → KVLease, in the five-seed runs of Table 1. Every arrival, including timeouts, remains in the SLO denominator.

Offered workflows/s   Within 30 s   Within 60 s   Within 120 s   Within 300 s
1.549% → 69%   82% → 98%   100% → 100%   100% → 100%
2.036% → 53%   50% → 86%   84% → 100%   100% → 100%
2.535% → 41%   41% → 61%   61% → 97%100% → 100%

The RTX comparisons also have KVLease at or above fixed slices at every reported target.

Analysis. Sixty seconds is above the light-load p95, so additional queueing becomes visible. The other targets show an improvement where the target is discriminating, and equality when both systems finish everything within a loose target. We also report p95, so the conclusion does not rest on one threshold.

Commitment to revise. We will explain the target, the SLO denominator and the treatment of timeouts in Section 5.1 and Appendix B.3.

Q5. GPU counts and model placement

Question. How many GPUs are used in each experiment, and how are models assigned to them?

What we have done and where. We consolidated the GPU counts, the model placements and the KV budgets of all experiments in a setup table in Section 5.1 of the revised paper. The new tensor-parallel deployment is detailed in Appendix B.20.

Answer. The original B200 experiments use one GPU. The original RTX experiments use two GPUs, with a fixed model placement and one pool per GPU. The revision adds a separate two-B200 deployment with tensor-parallel models.

Evidence and experiments. On the B200, all five models share one 32 GiB pool in the main setting; experiments with other budgets are labeled separately. On the two RTX PRO 6000 GPUs, Gemma, gpt-oss and Mistral run on the first GPU, and Llama and Granite on the second, with 16 GiB of KV capacity on each. Each of these engines has TP=1, and sharing is local to its GPU. The placement was chosen manually before measurement and kept identical across systems. The new deployment places Llama-3.3-70B and gpt-oss-120b across two B200s, with Gemma-2-9B on the first and Mistral-7B and Granite-3.3-8B on the second. Its pools use the memory left after weights and buffers.

Analysis. Keeping the placement, the memory capacity, the model settings and the arrivals identical isolates the allocator in the original comparisons. In the new deployment, we report KVLease per GPU separately from KVLease across GPUs, so the additional capability is visible.

Commitment to revise. The setup table of Section 5.1 will identify the GPU type and count, the model placement, the pool capacity, the workload and arrival process, and the repetitions of each experiment.

Reviewer_TSeWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache memory management for Mixture-of-Models (MoM) serving, where several heterogeneous LLMs are co-located on one GPU, each with its own engine (vLLM/SGLang), and user requests execute as DAGs of model calls (debate, proposer–aggregator, routing). Stage transitions shift KV demand across models, so fixed per-model partitions strand capacity. The authors propose KVLease: a per-GPU shared ledger over CUDA-VMM-backed KV pools that (i) lends idle capacity, (ii) assigns per-model guarantees that prioritize new allocations, and (iii) performs "targeted return," marking selected borrowed page groups as unavailable for new allocations and unmapping them once existing references and GPU accesses finish, without evicting live requests. An analytical section models workflow-induced demand covariance (Campbell's formula), decomposes pooling gains into partition-shape and temporal-multiplexing components, and gives a conditional page-return bound. Experiments on one B200 and two RTX PRO 6000 GPUs with five 7–20B models report 28–53% lower p95 than equal-token fixed slices and 24–31% lower p95 than kvcached at high load.

Critical strengths
  1. The core idea—letting co-located models elastically expand and shrink their KV capacity, and making borrowed capacity recoverable from busy borrowers rather than only lending idle memory—is interesting and well-motivated by the stage-shifting demand of MoM workflows.
  2. The implementation is practical: adapters for both vLLM and SGLang without a maintained fork, with prefix caching, chunked prefill, async scheduling and CUDA graphs kept enabled.
  3. Memory-safety is treated seriously (atomic accounting, reference + GPU-completion checks, finite-state checking, output-equivalence tests).
  4. The evaluation is transparent: paired seeds, per-seed tables, and several results unfavorable to the method are reported (e.g., profiled static splits, 88 GiB setting).
Critical weaknesses
  1. Unclear practical regime: the gains depend on many small models co-located on one GPU under an artificially constrained KV budget.

KVLease is scoped to a single GPU: all engines run with TP=PP=1, each GPU has an independent pool, and capacity does not migrate across devices (B.1). This restricts the evaluation to 7–20B models. For complex workflows, several small models sharing one GPU may be inferior to a single larger model on the same GPU (e.g., a 70B-class FP8 model fits on a B200), but the paper evaluates only serving efficiency on replayed traces and provides no quality or cost evidence for this deployment choice. Extending the mechanism beyond one GPU is non-obvious, because KV must reside with the model's weights: TP-sharded engines would need consistent multi-rank return, and cross-GPU or cross-node sharing would require KV or request migration. The paper's own analysis also implies diminishing returns at scale (under Proposition 1, the mean grows with Λ while the deviation grows with √Λ, so the sizing ratio approaches 1).

The headline numbers come from an operating point chosen to create memory pressure. The primary B200 experiments cap KV at 32 GiB, although about 94 GiB is available (B.14). Gains shrink from 40–50% at 32 GiB to 0–22% at 64 GiB (Table 4, one seed per point), and at 88 GiB KVLease is 1–4% worse (Table 16). Moreover, the near-full-pool setting is reported only at 1–1.5 workflows/s, omitting the 2–2.5 workflows/s rates where the main gains are claimed.

  1. The necessity of the central mechanism (fine-grained lending and targeted return) is not demonstrated against appropriate baselines.

The workflows are known DAGs, and the paper itself derives the optimal static split (A.2), yet the primary baseline is an untuned equal-token partition. The paper's measured sizing ratios are only 1.07–1.13 on B200 (A.6), implying that most of the 28–53% gain comes from a poorly shaped baseline partition rather than from dynamic sharing. Indeed, at the highest stable-mix load a profiled static split beats default KVLease by 12–19% on both platforms, and KVLease given the same profile does not surpass it (83.6 vs 82.4 s; B.13, Table 13). KVLease wins only in a single-replication shifted-mix diagnostic, under a workload whose dominant class switches every 60 seconds by design, which is far faster than typical production mix shifts. Periodic quota recomputation every 10–30 seconds is within 0–7% of the full policy (B.12), and adding targeted return worsens B200 p95 in the cumulative ablation (64.7 to 68.8 s, Fig. 5).

Why it is consequential: the claimed contribution is the lending-and-return contract, but the paper's own evidence suggests that a profiled or periodically updated static allocation captures most of the benefit at lower complexity and overhead.

Questions for authors
  1. Full-pool (~94 GiB) results on B200 at all rates (including 2–2.5 workflows/s) with multiple seeds, and the load at which the system becomes compute-bound.
  2. The analytically optimal (A.2) or profiled static split as the primary baseline, plus a periodically re-profiled static baseline.
  3. A quality–latency (or cost) comparison between the MoM workflows and a single larger model under the same GPU budget, or evidence of real deployments of this configuration.
  4. Discussion of TP-sharded models and multi-GPU deployments.
  5. Isolated overhead measurements: map/unmap latency, ledger lock contention vs. number of engines, reclaimer CPU, and prefix-cache loss from marking.
6 comments
AuthorsOpenReview ↗

Response to Reviewer TSeW (1/5)

Thank you for the review. This review helped us demonstrate the practicality and necessity of KVLease more clearly. We extended KVLease to large tensor-parallel models, and we evaluated it with the full memory pool, without any memory budget, and against the baselines that the review suggests.

Q1. Full-pool evaluation and the practical operating regime

Question. Do the gains depend on an artificially constrained 32 GiB KV budget? What happens with the full pool at all rates, including 2–2.5 workflows/s, and at which load does the system become compute-bound?

What we have done and where. We ran a long-document workload that uses all the memory left after weights and engine buffers, with no KV budget (Appendices B.20 and B.21 of the revised paper). We also ran the paper's main workload with the full pool at all four rates, with five seeds (Appendix B.14).

Answer. KVLease's benefit is not restricted to an artificially capped pool. It keeps substantial gains when long contexts create memory pressure with all the available GPU memory. On the paper's short-input workload with the full pool, memory pressure is low and KVLease performs close to fixed slices, as expected.

Evidence and experiments. Full memory, long documents. KV memory grows with the length of the context, and production contexts can be long. We used QMSum meeting transcripts as requests, with the timestamps and service mix of the Azure LLM inference trace. Each GPU's pool is all the memory that the weights and engine buffers leave. QMSum has long contexts, so memory pressure occurs naturally without us having to set a limit. The table gives the one-GPU setup (the paper's five models, 18 request sets).

Full-memory long-document workload   Equal-token slices   Profiled static split   KVLease
Finished within 60 s33.0%79.4%97.9%
p95 latency226.0 s82.9 s47.4 s

In the two-GPU setup of our answer to Q4, KVLease finishes 96.3% of workflows within 60 s, against 28.7% for the profiled static split and 52.8% for periodic re-profiling.

Full pool, the paper's workload. Table 16 of the paper is one seed of the catalogue experiment of Appendix B.14, which assigns the workflow roles to different models. So we ran the paper's main workload with an 88 GiB pool at all four rates, with five seeds. This is the largest pool that fits in practice: the 94 GiB of Appendix B.14 is measured at start-up, and the engines need a few more GiB once they serve requests. Each cell gives fixed slices → KVLease.

Offered workflows/s   p95 (s)Finished within 60 s   Pool used by running requests
1.043.7 → 43.9   99.4% → 99.4%8%
1.550.1 → 50.2   98.3% → 98.1%13%
2.058.9 → 57.7   95.6% → 96.4%19%
2.571.5 → 69.4   83.5% → 85.6%24%

The two are within 0.5% at 1.0 and 1.5 workflows/s, and KVLease is 2–3% lower at 2.0 and 2.5. Both finish 90% of workflows within 60 s at 2.0 workflows/s and not at 2.5. Running requests still use under a quarter of the pool there, and the time per output token rises from 14 ms at 1 workflow/s to 21 ms at 2.5. So the system becomes compute-bound between 2.0 and 2.5 workflows/s.

Analysis. Lending and return matter only when KV memory is scarce. When memory is plentiful, every model's KV fits in its own share. Nothing is lent and nothing is returned, so the mechanism stays inactive. This is the case with the full pool on the paper's workload, because its benchmark questions have only about 210 tokens each. Table 4 shows the same pattern: as the budget grows, the workload fills a smaller share of the pool, and there is less to lend. So this region is not suitable for evaluating KVLease. The memory budget is what lets the paper evaluate lending and return.

The review also notes that Proposition 1 implies diminishing returns at scale. This holds for one of its two terms. The first term is the random variation from individual requests. Its deviation grows with Λ\sqrt{\Lambda}, so it averages out at scale, as the review says. The second term is the variation from the traffic mix changing over time. Its deviation grows with Λ\Lambda, so it keeps pace with the mean and does not average out. The gain of KVLease at scale comes from this second term.

Commitment to revise. We will report the long-document results without a KV budget in Appendices B.20 and B.21 and the five-seed full-pool sweep in Appendix B.14. We will explain the two terms of Proposition 1 in Section 3.

AuthorsOpenReview ↗

Response to Reviewer TSeW (2/5)

Q2. Stronger baselines and the need for targeted return

Question. Does KVLease improve on a profiled or analytically optimized static split and on periodic quota recomputation? Is targeted return necessary?

What we have done and where. We evaluated the A.2 split under a changed mix, periodic re-profiling every 10 and 30 seconds, and targeted return on and off. The results are in Section 5.6 and Appendices B.5, B.13 and B.20 of the revised paper.

Answer. Yes. Under changing demand, KVLease improves on both the profiled static split and periodic re-profiling, and targeted return adds to this when a lender needs its memory back. A profiled split works well when the traffic matches the traffic it was computed from. But production traffic changes, and then neither baseline captures the benefit of lending and return.

Evidence and experiments. First, in the paper's main setting (one B200, five models, 32 GiB), the A.2 split is computed from the paper's mix and evaluated on a steady mix led by different models, with no 60-second switching. KVLease receives the same profile as its guarantees. This is a profile-transfer test, not an oracle split optimized for the evaluation mix. Five paired seeds are used.

Offered workflows/s  A.2 split p95  KVLease p95  Paired p95 change and seed range  SLO attainment
1.540.0 s37.1 s−6% [−21%, −1%]1.00 → 1.00
2.054.9 s38.9 s−26% [−48%, −9%]0.95 → 1.00
2.580.4 s40.6 s−48% [−62%, −37%]0.75 → 1.00
3.0110.7 s41.6 s−62% [−70%, −56%]0.51 → 1.00

With five seeds, the shifted-mix comparison of Table 13 gives the same picture: at 3 workflows/s the profiled split has a p95 of 108.1 s and KVLease with the same profile 43.9 s.

Second, we used the two-B200 deployment of our answer to Q4, with large tensor-parallel models. Requests are questions about QMSum meeting transcripts, with the timestamps and service mix of the Azure LLM inference trace, at 1.8 workflows/s over 18 request sets. The static profile comes from earlier traffic of the same service. Periodic re-profiling uses KVLease's live memory-adjustment mechanism, so it can change quotas without charging a model restart to the baseline.

SystemSharing scope  Mean p95  Finished within 60 s
A.2 profiled static splitPer GPU161.5 s28.7%
Re-profile every 10 sPer GPU97.8 s52.8%
Re-profile every 30 sPer GPU102.1 s49.8%
KVLeasePer GPU71.9 s80.1%
KVLease with peer allocation  Across GPUs53.5 s96.3%

The per-GPU comparison improves SLO attainment from 52.8% to 80.1% against 10-second re-profiling. Updating every 10 s instead of every 30 s makes little difference. The across-GPU variant additionally uses idle memory on the peer GPU, and its benefit is reported separately. Its p95 is lower than every baseline's in all 18 request sets.

For targeted return, the revised trigger waits for a shortage to persist for about one second. In the Figure 5 setting, p95 is 61.1 s with return and 61.3 s without it. In a separate long-document experiment over nine request sets, with one pool per GPU, return lowers p95 from 136 to 88 s and the lending model's queueing from 42 to 2 s per call, with an improvement in every request set.

Analysis. When traffic matches the profile, no model has idle memory to lend. So the profiled split and KVLease with the same profile perform the same (Table 13). The sizing ratios of 1.07–1.13 in A.6 describe this case, because they are defined for a mix that does not change. But traffic changes: in the Azure trace, the share of requests going to each service shifts by a median of 24 percentage points over twelve hours. A fixed profile then becomes mismatched to the current demand, and a quota stays unchanged between two periodic updates. Lending reacts to the capacity that is available, and targeted return makes borrowed capacity recoverable when its lender needs it. Periodic re-profiling itself needs this mechanism to move memory between running engines; the periodic quotas of Appendix B.12 already ran on it. The on/off experiment isolates the effect of return. The small difference in the Figure 5 setting shows that return is useful when persistent reclamation pressure occurs, rather than guaranteeing a latency improvement in every workload. The per-GPU row establishes the policy benefit before the separate peer-memory capability is added.

Commitment to revise. We will place the per-GPU KVLease row beside periodic re-profiling in the comparison of Appendix B.20, label the profile-transfer experiment explicitly in Appendix B.13, retain the stable-profile result of Table 13, and document the revised return trigger in Section 4.3 and Appendix B.5. We will keep the nine-set return experiment separate from the 18-set comparison.

AuthorsOpenReview ↗

Response to Reviewer TSeW (3/5)

Q3. Quality and latency relative to a single larger model

Question. Do the MoM workflows offer a useful quality–latency or quality–cost trade-off relative to a single larger model under the same GPU budget, or is there evidence of real deployments of this configuration?

What we have done and where. We compared a debate among gpt-oss-20b, Granite-3.3-8B and Gemma-2-9B with Llama-3.3-70B on 3,017 test questions of the paper's four benchmarks, each system on one B200. The comparison is added to Appendix B.23 of the revised paper. Below we also give published results and deployed systems that combine several models.

Answer. Yes. On the same GPU, the debate among the three small models is more accurate than the single 70B model. Its mean accuracy over the four benchmarks is 4.5 percentage points higher, and the gain is about 10 points on the harder benchmarks. The debate makes six model calls, so an answer takes longer, and 98% of its answers still finish within 60 s. Systems that combine several models are also deployed in production today.

Evidence and experiments. Measured comparison. The debate is the paper's two-round debate, with six model calls per answer. Llama-3.3-70B runs with FP8 weights, so that it fits on one B200. Both systems answer the GSM8K test set, MATH-500, a 1,000-question test subset of MMLU-Pro and GPQA Diamond, 3,017 questions in all, with the same prompts, greedy decoding and the same grader. A request that does not complete counts as wrong. Latency is measured by serving each system at 2 requests/s with identical arrivals from the four benchmarks (1,202 requests); the debate is served with KVLease.

BenchmarkQuestions   Accuracy: 70B   Accuracy: debate   Difference [95% interval]   Mean / p95 latency: 70B   Mean / p95 latency: debate
GSM8K1,31995.9%95.7%−0.2 [−1.2, +0.8]3.8 / 5.8 s12.4 / 20.9 s
MATH-50050065.4%75.8%+10.4 [+6.7, +14.1]8.9 / 23.9 s28.9 / 54.9 s
MMLU-Pro1,00070.7%69.4%−1.3 [−4.2, +1.6]9.0 / 20.9 s22.7 / 46.0 s
GPQA Diamond19849.5%58.6%+9.1 [+1.0, +17.2]12.8 / 23.6 s33.4 / 51.7 s
Mean of the four   70.4%74.9%+4.5 [+2.1, +6.8]8.6 / 20.7 s24.4 / 48.1 s

The difference is paired over questions, in percentage points. The last row gives the mean accuracy of the four benchmarks and the latency over all requests. The gain is on the harder benchmarks, 10.4 points on MATH-500 and 9.1 points on GPQA Diamond; on GSM8K and MMLU-Pro the two systems are within their intervals.

Published results and deployments. Published results show that a combination of models can match or exceed a single stronger model. MoA, built only from open-source models, scores 65.1% on AlpacaEval 2.0, against 57.5% for GPT-4o (Wang et al., 2024). Archon combines several models in one workflow and outperforms o1, GPT-4o and Claude 3.5 Sonnet by 15.1% on average (Saad-Falcon et al., 2025). FrugalGPT matches GPT-4 at up to 98% lower cost by combining models of different sizes (Chen et al., 2024). Such systems are also deployed today. The GPT-5 system card describes a fast model, a deeper reasoning model and a real-time router between them (OpenAI, 2025). Microsoft Foundry offers a model router that sends each prompt to one of several models behind a single deployment (Microsoft, 2026). GitHub Copilot Fusion, which is generally available, orchestrates multiple models for one task: one model drafts a response and other models review it (GitHub, 2026).

Analysis. Model composition is an application choice. The quality result provides a reason to serve this mixture, and the latency shows the cost of obtaining that improvement. KVLease's contribution is to reduce the serving overhead of the chosen workflow without altering its model computations. With the tensor-parallel extension of Section 4.4, KVLease also serves large models together with small models in one workflow, so the provider is free to choose the mixture.

Commitment to revise. We will report the matched quality–latency comparison in Appendix B.23, with the tested models, the weight precision, the decoding settings and the load.

Reviewer_G5vdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the critical memory bottleneck in Mixture-of-Models inference, where overlapping requests across parallel and sequential models lead to severe KV cache fragmentation and downstream latency. The authors propose KVLease, a dynamic memory sharing framework that enables cross-engine capacity lending while maintaining strict per-model performance guarantees. Extensive evaluation on NVIDIA B200 and dual-RTX platforms across diverse workloads and model catalogs shows significant improvements over baselines.

Critical strengths

1 The problem formulation is timely, targeting the increasingly relevant MoM inference paradigm where traditional single-model KV cache management fails.

2 The system design exhibits remarkable elegance in balancing flexibility with safety. The combination of atomic capacity accounting with per-model guarantees ensures that borrowing never starves any individual engine, while the targeted return mechanism provides a practical solution to the challenging problem of safe memory reclamation in asynchronous GPU environments.

3 The experimental evaluation is comprehensive and rigorous. Testing on both high-end (B200) and consumer-grade (RTX) hardware across multiple model scales and workload types provides strong evidence of generality.

Critical weaknesses

1 The comparison with related work, while covering kvcached and Fixed-slice vLLM, lacks depth in positioning against other emerging memory management approaches. Specifically, the paper does not discuss how KVLease compares to recent work on disaggregated memory architectures, CPU-GPU KV offloading strategies.

2 The adaptive quota management relies on the bm/fm ratio, but the paper does not adequately address how this metric behaves under highly variable or bursty workloads that deviate from the tested patterns. Real-world MoM deployments may exhibit workload characteristics not captured by the collaborative and routing benchmarks, such as sudden model failures, uneven request distributions, or non-stationary arrival patterns.

3 The evaluation focuses primarily on latency metrics and does not adequately address throughput or resource efficiency trade-offs. While p95 latency improvements are impressive, the paper does not report maximum sustainable throughput, tokens-per-second per GPU, or memory utilization efficiency under different sharing configurations.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer G5vd (1/3)

Thank you for the constructive feedback. KVLease can work together with KV offloading strategies. Recent works on disaggregated memory architectures and CPU–GPU KV offloading are orthogonal research directions, and they can be used together with KVLease. We also tested KVLease with bursty, uneven and non-stationary workloads, and it remains strong in those cases because of its dynamic memory allocation. Based on the suggestions, we added new metrics: maximum sustainable throughput, output tokens per second per GPU, and memory utilization.

W1. Disaggregation and CPU KV offloading

Question. How does KVLease relate to disaggregated memory architectures and CPU–GPU KV offloading, and can they be used together?

What we have done and where. We added this positioning to the related-work section of the revised paper. To confirm it, we ran KVLease together with vLLM's CPU offload, with offload enabled in every engine of both systems. The results are added to Appendix B.15.

Answer. KVLease's allocation contract is complementary to decisions about where KV is stored. We have demonstrated the compatibility of KVLease with CPU offloading experimentally. Disaggregated memory architectures decide where the KV of a model is placed, for example on another GPU or in an external store. KVLease controls how the models on one GPU share its KV memory. These are separate decisions, so by design KVLease can be used together with such architectures.

Evidence and experiments. DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2024) separate prefill and decode across GPUs and send the KV between them. Mooncake (Qin et al., 2025) uses an external KV store. These designs choose where the KV of one model is stored. They do not control how several models on the same GPU share its KV memory. KVLease controls this sharing.

CachedAttention (Gao et al., 2024), LMCache (Liu et al., 2025) and vLLM's CPU offload retain KV outside the serving GPU for later use. This helps when the same prefix is used again. A request that is running still needs its KV on the GPU. KVLease controls how the models on one GPU share this memory and how a model recovers its guaranteed share.

To test the combination, we ran five models on two RTX PRO 6000 GPUs, with 4 GiB of host KV storage per model and CPU offload enabled in every engine of both systems. We used two paired seeds.

Offered workflows/s   Fixed slices + CPU offload, p95   KVLease + CPU offload, p95   Change [range]   Completed workflows/s: fixed slices → KVLease
1.0259.3 s130.8 s−50% [−52, −47]   0.54 → 0.79
1.5407.2 s190.1 s−53% [−53, −53]   0.59 → 0.97

Change is the paired p95 change, with its range across seeds.

All workflows complete. The model outputs are identical with offload enabled and disabled. Between 3.0% and 9.5% of prompt tokens are restored from host memory, which confirms that the offload tier is in use. The paper's Table 18 additionally reports 42–48% lower p95 for KVLease than for fixed slices with CPU offload at the same GPU capacity.

Analysis. In the evaluated offload mode, host memory preserves reusable KV while active inference still requires GPU capacity. Lending available GPU pages across engines addresses that concurrent capacity demand, so the benefit persists with the offload tier enabled. The combination lowers p95 by 50–53% and raises completed workflows per second by approximately 45–64% in these runs.

Commitment to revise. We will distinguish storage placement, stage disaggregation and cross-engine capacity allocation in the related work, and include the joint-offload measurements and configuration in Appendix B.15.

AuthorsOpenReview ↗

Response to Reviewer G5vd (2/3)

W2. The adaptive signal under bursty workloads, and realistic deployment conditions

Question. (a) How does the waiting-to-allowance signal (bm/fmb_m/f_m) of the adaptive quota management behave under highly variable or bursty workloads? (b) Does KVLease still work under real-world conditions that the benchmarks do not cover, such as sudden model failures, uneven request distributions and non-stationary arrival patterns?

What we have done and where. For (a), we traced the signal through a sudden change of the workload, and we compared the adaptive rule on and off under a bursty trace. For (b), we ran each condition that the review names: a bursty trace, an uneven workload, a non-stationary production trace and a model failure under load. The results are added to Appendices B.6, B.21 and B.22 of the revised paper.

Answer. (a) The signal changes the allowances slowly and within fixed limits, so a burst cannot make it overreact. The fast reaction to a burst comes from lending and targeted return, and they do not depend on the signal. (b) KVLease keeps its benefit under all the named workload conditions, with a lower p95 than fixed slices in every seed and request set. Under a model failure, memory is never over-committed and the workflows that do not use the failed model all complete.

Evidence and experiments. (a) The signal. For each model, the signal divides the accumulated waiting of its workflows (bmb_m) by its allowance (fmf_m). The allowance is how much idle cache the model may keep before it lends. Every 10 s the rule moves a little allowance toward the models with the highest signal. One update moves at most 2% of the pool, and no allowance can fall below half of its starting value.

We followed the signal through one sudden change of the workload (one B200, 2 workflows/s). Targeted return starts 4 s after the change. The rule raises Mistral's allowance at 29 s and Llama's at 49 s. By 40 s, 2.35 GiB has been returned to the two models.

We also switched the rule off under the bursty trace of (b). With the rule on, p95 is equal or lower in every seed: 99.7 against 102.7 s, 112.1 against 112.6 s, and 136.4 against 148.5 s.

(b) Realistic conditions. The table compares KVLease with fixed slices under three workload conditions, with the adaptive rule on. The bursty workload replays BurstGPT at its original timestamps, in the four-minute window where its traffic mix changes most (two RTX PRO 6000 GPUs). In the uneven workload, most workflows belong to one class at a time, and the class changes every 60 s (one B200). The non-stationary workload follows the arrival times and the service mix of the Azure LLM inference trace, with questions about long meeting transcripts (one B200).

Condition and replicationOffered workflows/sFixed slices p95KVLease p95Paired p95 change and rangeSLO attainment
Bursty: BurstGPT replay, 3 seeds2.0219.8 s116.1 s−47% [−52%, −43%]0.57 → 0.68
Uneven demand, 5 seeds3.067.5 s43.8 s−35% [−42%, −27%]0.88 → 1.00
Non-stationary: Azure long-document replay, 18 sets1.6–2.8226.0 s47.4 s−72% [−84%, −23%]0.33 → 0.98

The range is across seeds or request sets. SLO attainment is the fraction of workflows that finish within 60 s. KVLease has a lower p95 than fixed slices in every seed and request set.

For a sudden model failure, we killed one model under load and restarted it about five minutes later, in two runs (one RTX PRO 6000 with Gemma, Mistral and Granite, rule on). Memory was never over-committed. All 538 workflows that did not use the killed model completed. Workflows that needed it received an error within 0.4 s until it was back. After the restart, the model had its guaranteed memory again 36–47 s after it received load.

Analysis. (a) The limits bound each step of the rule, so a burst can change the allowances only a little at a time. In the traced change, targeted return starts at 4 s and the rule makes its first change at 29 s. So the fast response comes from return and not from the signal. In the on/off comparison, p95 with the rule on is equal or lower in every seed, so the rule stays safe under bursts. (b) Bursty, uneven and non-stationary traffic all move the memory demand between models. Fixed slices leave memory idle in that case and KVLease lends it, so KVLease keeps its benefit.

Commitment to revise. We will report the signal trajectory, the update limits, the on/off comparison, the workload-selection rules and the failure protocol in Appendices B.6, B.21 and B.22.

AuthorsOpenReview ↗

Response to Reviewer G5vd (3/3)

W3. Throughput and memory efficiency

Question. What are the maximum sustainable throughput, the output tokens per second per GPU, and the memory utilization efficiency under different sharing configurations?

What we have done and where. We evaluated the throughput and the memory efficiency of KVLease in the paper's main setting (one B200, five models, 32 GiB, three seeds). We raised the arrival rate from 1.0 to 2.5 workflows/s to find the maximum sustainable throughput. In the same setting we measured the output tokens per second per GPU and the memory utilization of each sharing configuration. Section 5.1 of the revised paper lists these metrics, and the results are in Appendix B.10.

Answer. KVLease improves throughput and memory efficiency as well as latency. The three metrics improve for the same reason. With fixed slices, the memory that a busy model needs is locked inside the slices of idle models. KVLease puts this memory to work, so each engine runs more requests together. Its maximum sustainable throughput is 2.0 workflows/s, against 1.25 for fixed slices. It produces 26–31% more output tokens per second on the same GPU. While requests wait, fixed slices use 42% of the pool for running requests, and KVLease uses 75%.

Evidence and experiments. Maximum sustainable throughput. This is the highest arrival rate at which 90% of workflows finish within 60 s.

Offered workflows/s   Finished within 60 s: fixed slices → KVLease   p95 (s): fixed slices → KVLease
1.099.6% → 99.6%45.2 → 42.8
1.2596.5% → 99.0%56.1 → 46.8
1.579.2% → 98.2%75.3 → 52.6
1.7564.5% → 97.4%102.0 → 55.9
2.054.9% → 94.3%128.6 → 61.1
2.2548.3% → 83.0%144.1 → 73.1
2.541.1% → 67.2%170.8 → 92.4

KVLease stays above 90% up to 2.0 workflows/s. Fixed slices stay above 90% up to 1.25 workflows/s.

Output tokens per second per GPU.

Offered workflows/s   Output tokens/s per GPU: fixed slices → KVLease   Change
2.03,579 → 4,500+26%
2.53,978 → 5,219+31%

Memory utilization under different sharing configurations (2.0 workflows/s), measured at the moments when some requests are waiting for memory.

Sharing configurationMemory used by running requests (of 32 GiB)   Memory utilization
Fixed slices, no sharing13.4 GiB42%
Lending of idle memory18.7 GiB58%
+ retention allowances19.7 GiB62%
+ guarantees20.8 GiB65%
+ targeted return (full KVLease)   23.9 GiB75%

Analysis. With fixed slices, a large part of the pool serves no running request while other requests wait: running requests use only 42% of it. Each sharing step of KVLease puts more of this memory to work, up to 75%. Each engine then runs more requests together: tokens per engine step rise from 35.9 to 47.1. So the same GPU produces more tokens per second, and it stays within the 60 s target up to a higher rate.

Commitment to revise. We will list the three metrics in Section 5.1, and report the rate sweep, the output tokens per second per GPU and the memory utilization of each sharing configuration in Appendix B.10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.