- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents KVLease, a memory sharing system for mixture of models workflows. In the mixture of models setting, a request is processed by several models, e.g., an aggregator aggregates results from several proposers. The paper characterizes memory usage for mixture of models workflows. Based on the findings, it designs KVLease which coordinates memory lending and return across different models. The evaluation compares KVLease with a couple of baselines.
Critical strengths
- The paper presents KVLease for memory sharing in the mixture of models setting. KVLease coordinates memory lending and return across different models.
- The evaluation compares KVLease with a few baselines. The results show the benefits of KVLease.
Critical weaknesses
- The evaluation uses synthetic workloads. It is unclear whether the workloads are representative of realistic scenarios.
Questions for authors
-
In Section 2, the paper says "In a single-replication matched-volume diagnostic at two arrival rates, fixed-slice MoM workflows have 16–43% higher p95 than independent-call controls" and "Across the three primary evaluation settings, pooling raises generated tokens per engine step from 35.9 to 47.1, 36.6 to 45.7, and 32.6 to 53.1: 20–39% fewer steps per generated token". Can you clarify what the setup is? What hardware, models, and benchmarks do you use?
-
The paper says that KVLease is a "KV-memory sharing system". Does KVLease only share the GPU memory of KV Cache? When there are multiple models, the total size of the model parameters can be very large. Does KVLease dynamically allocate the GPU memory for model parameters, and swap model parameters between GPU memory and host memory?
-
For the workloads in Section 5.1, the paper mentions that "The main collaborative mix combines a single-model service, two-round three-model debate, and three-proposer aggregation on GSM8K, MATH-500, MMLU-Pro, and GPQA". How does the mix combine the tasks? Does the mix randomly choose these tasks?
-
The SLO attainment is set as "the fraction of arrivals completing within 60 seconds". Why do you pick 60 seconds?
-
Can you clarify the number of GPUs used in each experiment? Does B200 experiments use one B200 GPU and RTX experiments use two RTX GPUs? For the two RTX GPU experiments, how do the models share the two GPUs? How do you decide which models share which GPU?
3 comments
Response to Reviewer Tz1J (2/3)
Q1. Setup of the motivating measurements
Question. What hardware, models and benchmarks produce the 16–43% higher p95 of dependent workflows and the tokens-per-engine-step results in Section 2?
What we have done and where. We clarified the setup of both results in Section 2 and Appendix B.1 of the revised paper.
Answer. The two results come from two different experiments. Both use the five models of Section 5.1 (Gemma-2-9B-it, gpt-oss-20b, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct and Granite-3.3-8B-Instruct) and questions from GSM8K, MATH-500, MMLU-Pro and GPQA. The 16–43% result is a diagnostic on one B200 with 32 GiB of KV memory in fixed slices, at 1.5 and 2 workflows/s, with one run at each rate. It sends the same model calls either as workflows, where a call starts only when the calls it depends on have finished, or as independent calls. The tokens-per-step result compares fixed slices with KVLease in the three main settings of Table 1, with three seeds each: the collaborative workload on one B200, the routing workload on one B200, and the collaborative workload on two RTX PRO 6000 GPUs.
Commitment to revise. We will state the setup of both results in Section 2 and Appendix B.1.
Q2. Scope of KV memory sharing
Question. Does KVLease share only the GPU memory of the KV cache, or does it also allocate GPU memory for model parameters and swap them between GPU and host memory?
What we have done and where. We clarified the scope of memory sharing and the assumption that the model weights stay on the GPU. The clarification is in Section 4 and in the setup table of Section 5.1 of the revised paper.
Answer. KVLease shares only KV-cache memory. The model weights are loaded once and stay on the GPU, and KVLease does not swap them between GPU memory and host memory. In a workflow, the KV demand moves from one model to another within seconds, while every model stays loaded. Sharing the KV memory avoids tying every model to a fixed KV reservation. The KV contents of each model stay private, and only the capacity is shared. Weight swapping addresses a different capacity problem. It could be combined with KVLease, but it is not implemented or evaluated here.
Commitment to revise. We will state KV-only sharing, the resident-weight requirement and the tested model placements explicitly in Section 4 and in the setup table of Section 5.1.
Q3. Construction of the workflow mix
Question. How does the mix of Section 5.1 combine the datasets and workflow types, and are the tasks chosen randomly?
What we have done and where. We specified the class schedule, the random selection, the question sampling and the arrival process in Section 5.1 and Appendix B.1 of the revised paper.
Answer. Yes. Requests are sampled with a fixed seed inside a controlled sequence of dominant workflow classes, so every compared system receives identical requests.
Evidence and experiments. The four classes are:
| Dataset | Workflow |
|---|---|
| GSM8K | Single-model service (Gemma) |
| MATH-500 | Two-round debate among gpt-oss, Granite and Gemma |
| GPQA | The same two-round debate |
| MMLU-Pro | Llama, Mistral and Granite propose; gpt-oss aggregates |
A run consists of 60-second phases with one dominant class. In each phase, 80% of arrivals select the dominant class and the remainder select uniformly across all four classes. A question is then drawn uniformly from the recorded questions of the selected dataset. Arrival times are Poisson in the B200 collaborative rows of Table 1 and Azure-derived in its routing and RTX rows.
Analysis. Different workflow classes exercise different model groups. Known phase transitions show how lending and return follow those changes, and seeded sampling makes the comparison reproducible. The trace-derived mix in W1 provides a separate test without the designed phase schedule.
Commitment to revise. We will give the sampling probability, the phase schedule, the question selection, the seeding and the arrival sources in Section 5.1 and Appendix B.1.
Response to Reviewer Tz1J (3/3)
Q4. Choice of the 60-second target
Question. Why is SLO attainment defined as the fraction of arrivals completing within 60 seconds?
What we have done and where. We explained the target using the light-load workflow latency, and we evaluated three other targets. The explanation is in Section 5.1 of the revised paper, and the results at the other targets are in Appendix B.3.
Answer. Sixty seconds is a queue-sensitive target for these multi-call workflows. It sits just above the latency of a workflow that is served without extra queueing. The advantage of KVLease also appears at the other tested targets.
Evidence and experiments. At 1 workflow/s on the B200, the median workflow latency is 13–15 s and the p95 is 44–48 s. The table gives the share of workflows that finish within each target, fixed slices → KVLease, in the five-seed runs of Table 1. Every arrival, including timeouts, remains in the SLO denominator.
| Offered workflows/s | Within 30 s | Within 60 s | Within 120 s | Within 300 s |
|---|---|---|---|---|
| 1.5 | 49% → 69% | 82% → 98% | 100% → 100% | 100% → 100% |
| 2.0 | 36% → 53% | 50% → 86% | 84% → 100% | 100% → 100% |
| 2.5 | 35% → 41% | 41% → 61% | 61% → 97% | 100% → 100% |
The RTX comparisons also have KVLease at or above fixed slices at every reported target.
Analysis. Sixty seconds is above the light-load p95, so additional queueing becomes visible. The other targets show an improvement where the target is discriminating, and equality when both systems finish everything within a loose target. We also report p95, so the conclusion does not rest on one threshold.
Commitment to revise. We will explain the target, the SLO denominator and the treatment of timeouts in Section 5.1 and Appendix B.3.
Q5. GPU counts and model placement
Question. How many GPUs are used in each experiment, and how are models assigned to them?
What we have done and where. We consolidated the GPU counts, the model placements and the KV budgets of all experiments in a setup table in Section 5.1 of the revised paper. The new tensor-parallel deployment is detailed in Appendix B.20.
Answer. The original B200 experiments use one GPU. The original RTX experiments use two GPUs, with a fixed model placement and one pool per GPU. The revision adds a separate two-B200 deployment with tensor-parallel models.
Evidence and experiments. On the B200, all five models share one 32 GiB pool in the main setting; experiments with other budgets are labeled separately. On the two RTX PRO 6000 GPUs, Gemma, gpt-oss and Mistral run on the first GPU, and Llama and Granite on the second, with 16 GiB of KV capacity on each. Each of these engines has TP=1, and sharing is local to its GPU. The placement was chosen manually before measurement and kept identical across systems. The new deployment places Llama-3.3-70B and gpt-oss-120b across two B200s, with Gemma-2-9B on the first and Mistral-7B and Granite-3.3-8B on the second. Its pools use the memory left after weights and buffers.
Analysis. Keeping the placement, the memory capacity, the model settings and the arrivals identical isolates the allocator in the original comparisons. In the new deployment, we report KVLease per GPU separately from KVLease across GPUs, so the additional capability is visible.
Commitment to revise. The setup table of Section 5.1 will identify the GPU type and count, the model placement, the pool capacity, the workload and arrival process, and the repetitions of each experiment.
Response to Reviewer Tz1J (1/3)
Thank you for the review, and for identifying the setup details that need clarification. We evaluated KVLease with replays of public production traces and with long real documents. KVLease keeps its latency benefit in these tests. Below we also explain the workload construction, the scope of memory sharing, the latency target and the GPU placement.
W1. Workload realism
Question. Are the workloads representative of realistic serving scenarios, and not only synthetic?
What we have done and where. We added replays of two public production traces: the Azure LLM inference trace and BurstGPT. In the Azure tests, both the arrival times and the service mix come from the trace. We also added long real inputs, which are questions about QMSum meeting transcripts. The results are in Section 5.6 and Appendices B.6 and B.21 of the revised paper. The deployment with larger models is in Appendix B.20.
Answer. KVLease's benefit persists on the production-trace workloads and on long inputs. The service demand in these traces changes over time, and this is the case where sharing memory helps.
Evidence and experiments. In the submitted paper, every prompt is a real benchmark question executed through an actual workflow, and two of the three settings of Table 1 already use Azure-derived arrival times. The new tests also take the service mix from the traces. The first test replays the Azure trace on the paper's main setting (one B200, five models, 32 GiB), with the paper's workflows. The second replays the same trace with questions about QMSum meeting transcripts on one B200. The third replays BurstGPT at its original timestamps, in the window with the largest mix change, on two RTX PRO 6000 GPUs.
The range is across trace periods, request sets or seeds. SLO attainment is the fraction of workflows that finish within 60 s. KVLease has a lower p95 in every reported period, request set and seed. Every system replays identical arrivals. The two-B200 deployment with tensor-parallel models additionally finishes 96.3% of workflows within 60 s, against 28.7% for a static split profiled on earlier traffic.
Analysis. In the Azure trace, the share of requests that goes to each service shifts by a median of 24 percentage points over twelve hours. Changing demand strands capacity in fixed slices: busy models wait while idle models hold memory. This is the setting where lending and return are useful. So the benefit of KVLease persists under trace replay.
Commitment to revise. We will report the trace results in Section 5.6. We will document the trace periods, the mapping of services to workflows and the arrival rates in Appendix B.21, and the BurstGPT replay in Appendix B.6.