- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Janus, a cache-management system for speculative decoding with offloaded KV caches. It addresses repeated transfers and incompatible cache organization across drafting and verification. Janus uses a unified physical page abstraction. It enables cross-phase reuse and allocates cache capacity across execution phases. The evaluation targets consumer hardware and agent workloads. Validation uses consumer GPUs, three language models and three agent benchmarks. It compares against autoregressive offloading and a speculative baseline in the same engine. Janus reduces token latency against both evaluated baselines. Reported token-latency reductions are 38–78% against autoregressive offloading and 26.2–53% against the speculative baseline. The benefit also extends to end-to-end latency.
Critical strengths
- The paper connects a concrete mismatch in phase-specific cache requirements to redundant transfers and layout reconstruction. Sharing physical pages across layers and phases directly addresses this mismatch. This is a stronger motivation than simply applying speculative decoding to an offloading runtime.
- The reuse mechanism in Janus has direct evidence. Cross-phase reuse improves both token and end-to-end latency. Enabling cross-phase reuse yields 1.40× token-latency speedup and 1.24× end-to-end speedup. It also reduces time spent transferring cache data to the GPU. This control links reuse to the observed benefit.
Critical weaknesses
- The experimental description is incomplete. The setup does not adequately specify request counts, length distributions or cache-offloading fractions. Host/PCIe details and repeated-run variability are also unclear. These details affect the transfer bottleneck and the reproducibility of the speedups. It limits the confidence that the results transfer to other deployments.
Questions for authors
What are the request counts and context/output-length distributions? What fraction of the cache is offloaded? What host/PCIe configuration is used?
2 comments
Dear Reviewer, Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.
1. Experimental Details and Offloading Behavior (Weakness 1 & Key Question 1)
Thank you for pointing out which experimental details would benefit from further explanation. We report the workload sizes, length distributions, offloading fractions, hardware configuration, and measurement variability below.
Workload sizes and length distributions. The workloads contain 200 BFCL, 100 API-Bank, and 78 ToolTalk sessions. We reserve 10 sessions from each workload for calibration and evaluate the remaining 190, 90, and 68 sessions, respectively. Peak context length includes the system prompt, tool schemas, user–assistant messages, and tool results. Output length counts all model-generated tokens in a complete session and excludes tool results. The length statistics are averaged across model–GPU configurations.
Context growth is driven primarily by tool results, followed by subsequent user inputs and model-generated tokens.
Offloading behavior. All evaluated sessions trigger offloading. On average, 26.3%, 20.9%, and 70.0% of the KV-cache is offloaded for Qwen3-8B/RTX 4090, Llama3.1-8B/RTX 4090, and Qwen3-14B/RTX 5090, respectively.
System configuration. All models use BF16 weights and BF16 KV-cache. vLLM automatically profiles the available GPU KV-cache capacity after model loading and runtime-memory reservation.
Measurement protocol and variability. For each session, we perform one warm-up run followed by three measured runs. E2E latency includes the initial prefill, decoding, local tool execution, and subsequent multi-turn prefills. Across the nine model–workload configurations, the coefficients of variation are 1.2%–2.3% for TPOT and 1.2%–2.2% for E2E latency. We will add these details to the revised manuscript.