acceptodds
Desk rejected at ICLR 2027

Janus: Breaking Layer-Bound KV-Cache Layouts to Accelerate Local Agent Inference

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

As local agent conversations grow longer, the key-value (KV)-cache can exceed commodity GPU memory, making KV-cache offloading necessary but introducing substantial host-to-device (H2D) transfer latency. Sparse self-speculative decoding (SSP) is a promising way to reduce transfer latency because the sparse KV-cache can be small enough to fit on GPU during drafting, while the full KV-cache is fetched only to verify multiple candidate tokens together. However, the straightforward combination of SSP and offloading (naive SSP) incurs redundant H2D transfers and layout rematerialization. This is because existing layer-bound layouts cannot accommodate the different KV-cache requirements of draft and verification, preventing reuse of overlapping KV-pages. We propose Janus, a local agent inference system whose layer-decoupled KV-page abstraction manages physical KV-pages in a unified GPU pool to enable cross-phase reuse. To keep large-scale metadata lookups and updates efficient, a coalesced metadata manager processes them through batched and fused primitives. To use the limited GPU page budget efficiently, a hierarchical GPU KV-page allocator first selects the verification staging depth and then calibrates the sparse draft budget offline. We implement Janus on vLLM V1 and evaluate three LLMs on two commodity GPUs across three agent workloads. Compared with baselines, Janus reduces time per output token (TPOT) by 26.2%-78.0% and end-to-end latency by 18.8%-61.8%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agent Inference, Long-context Inference, KV-cache Offloading, Sparse Self-speculation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_meEKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Janus, a cache-management system for speculative decoding with offloaded KV caches. It addresses repeated transfers and incompatible cache organization across drafting and verification. Janus uses a unified physical page abstraction. It enables cross-phase reuse and allocates cache capacity across execution phases. The evaluation targets consumer hardware and agent workloads. Validation uses consumer GPUs, three language models and three agent benchmarks. It compares against autoregressive offloading and a speculative baseline in the same engine. Janus reduces token latency against both evaluated baselines. Reported token-latency reductions are 38–78% against autoregressive offloading and 26.2–53% against the speculative baseline. The benefit also extends to end-to-end latency.

Critical strengths
  1. The paper connects a concrete mismatch in phase-specific cache requirements to redundant transfers and layout reconstruction. Sharing physical pages across layers and phases directly addresses this mismatch. This is a stronger motivation than simply applying speculative decoding to an offloading runtime.
  2. The reuse mechanism in Janus has direct evidence. Cross-phase reuse improves both token and end-to-end latency. Enabling cross-phase reuse yields 1.40× token-latency speedup and 1.24× end-to-end speedup. It also reduces time spent transferring cache data to the GPU. This control links reuse to the observed benefit.
Critical weaknesses
  1. The experimental description is incomplete. The setup does not adequately specify request counts, length distributions or cache-offloading fractions. Host/PCIe details and repeated-run variability are also unclear. These details affect the transfer bottleneck and the reproducibility of the speedups. It limits the confidence that the results transfer to other deployments.
Questions for authors

What are the request counts and context/output-length distributions? What fraction of the cache is offloaded? What host/PCIe configuration is used?

2 comments

1. Experimental Details and Offloading Behavior (Weakness 1 & Key Question 1)

Thank you for pointing out which experimental details would benefit from further explanation. We report the workload sizes, length distributions, offloading fractions, hardware configuration, and measurement variability below.

Workload sizes and length distributions. The workloads contain 200 BFCL, 100 API-Bank, and 78 ToolTalk sessions. We reserve 10 sessions from each workload for calibration and evaluate the remaining 190, 90, and 68 sessions, respectively. Peak context length includes the system prompt, tool schemas, user–assistant messages, and tool results. Output length counts all model-generated tokens in a complete session and excludes tool results. The length statistics are averaged across model–GPU configurations.

WorkloadSessionsPeak context, P25 / P50 / P75 / P95Output tokens/session, Mean / P50 / P75
ToolTalk6845.9K / 48.7K / 52.2K / 56.9K322.7 / 290 / 396
API-Bank9044.8K / 47.6K / 50.9K / 55.6K243.3 / 220 / 298
BFCL19050.3K / 53.2K / 56.3K / 59.4K229.1 / 190 / 287

Context growth is driven primarily by tool results, followed by subsequent user inputs and model-generated tokens.

Offloading behavior. All evaluated sessions trigger offloading. On average, 26.3%, 20.9%, and 70.0% of the KV-cache is offloaded for Qwen3-8B/RTX 4090, Llama3.1-8B/RTX 4090, and Qwen3-14B/RTX 5090, respectively.

System configuration. All models use BF16 weights and BF16 KV-cache. vLLM automatically profiles the available GPU KV-cache capacity after model loading and runtime-memory reservation.

Model / GPUCPUPCIeGPU KV-cache capacity
Qwen3-8B / RTX 4090Dual Intel Xeon Silver 4210RPCIe 3.0 ×1637.4K full-KV tokens
Llama3.1-8B / RTX 4090Dual Intel Xeon Silver 4210RPCIe 3.0 ×1640.1K full-KV tokens
Qwen3-14B / RTX 5090Dual Intel Xeon Platinum 8470PCIe 5.0 ×1615.2K full-KV tokens

Measurement protocol and variability. For each session, we perform one warm-up run followed by three measured runs. E2E latency includes the initial prefill, decoding, local tool execution, and subsequent multi-turn prefills. Across the nine model–workload configurations, the coefficients of variation are 1.2%–2.3% for TPOT and 1.2%–2.2% for E2E latency. We will add these details to the revised manuscript.

Dear Reviewer, Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.

Reviewer_XezPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets local agent application (concurrency 1), and improves the performance of SSP by reusing KV cache from draft budget. The framework of reusing KV cache (removing layer-boundedness of the cache) is novel, at least to the best of my knowledge. The paper presents the result in a systematically robust way: the baseline, the theoretical optimum, and the best implementation result with proper ablation study.

Critical strengths

This paper clearly defines the target application, and addresses the one of the bottlenecks by focusing on inefficiencies coming from framework. Not only the solution but also how they approach the problem was great. This work will improve the performance of local agent application and to my understanding very easy to deploy.

Critical weaknesses

I don't see any critical weakness in the paper. However, I generally disagree with the assumption that concurrency is 1 for local agent: users can ask multiple questions at a time, either by themselves or through subagents. And higher concurrency may affect how we should manage the memeory for JANUS, its effectiveness, etc.

Questions for authors
  • (major) I think the paper assumed processing one batch at a time, but that may not be true, even for local agents: Local agents may spawn subagents, for example, so the effective concurrency can go up. Please describe and update on how this would affect the result and conclusion.
  • Could you comment on the strength of SSP coupled with JANUS with respect to other speculative decoding techniques? I know the paper confines the discussion within SSP, but for as a potential user I would like to understand what kind of options we have for speculative decoding (n-gram, small draft model, MTP, etc.), when SSP is a good choice, and whether adding JANUS not only improves the SSP performance, but also changes which speculative decoding technique we choose.
  • The effectiveness of JANUS will depend on hardware specs, such as memory size and speed of H2D. Could you briefly touch on that and describe how it will shift the conclusion? This would help extrapolate the result for GPUs other than the tested ones.
  • (if necessary) Could you explain what need to be done differently if we want to apply this to SGLang?
  • (minor) Figure 10a: how about a solid line box or something instead of a small star? At first I thought the star was a typo, but I think it indicates the selected config?
4 comments

Thank you for the constructive questions. Due to OpenReview's 5,000-character limit, we respond in two parts: (1) multi-request concurrency and speculative-decoding choices (W1/Q1–2); and (2) hardware generalizability, SGLang porting, and Figure 10a (Q3–5). We will add these results and clarifications to the revised manuscript.

Response 1/2: Concurrency and Speculative-Decoding Choices

1. Multi-Request Concurrency (Weakness 1 & Key Question 1)

Thank you for raising this important setting. We extended JANUS to support BS > 1 by managing KV-pages across multiple concurrent requests. Each request maintains its own forward and reverse mappings, while all requests share the layer-decoupled KV-page pool. JANUS allocates and evicts pages across requests while preserving page ownership, allowing its cross-phase reuse to remain effective under concurrent inference.

We evaluate BS = 2, 4, and 6 using Llama3.1-8B on RTX 4090 and Qwen3-14B on RTX 5090 across all three workloads. Each cell reports TPOT for AR / Naive SSP / JANUS in ms/token.

Model / GPUWorkloadBS = 2BS = 4BS = 6
Llama3.1-8B / RTX 4090BFCL632.98 / 286.74 / 166.60718.07 / 272.15 / 136.62622.75 / 260.31 / 145.77
Llama3.1-8B / RTX 4090API-Bank712.96 / 276.63 / 140.25572.11 / 260.88 / 154.44627.03 / 253.72 / 132.95
Llama3.1-8B / RTX 4090ToolTalk623.57 / 314.28 / 162.48661.44 / 302.94 / 145.11607.29 / 286.64 / 127.84
Qwen3-14B / RTX 5090BFCL169.59 / 153.88 / 83.18169.20 / 133.55 / 82.52171.19 / 123.12 / 83.76
Qwen3-14B / RTX 5090API-Bank167.00 / 137.45 / 75.38170.25 / 118.32 / 77.25167.33 / 105.59 / 74.00
Qwen3-14B / RTX 5090ToolTalk241.55 / 196.57 / 150.43212.20 / 170.97 / 119.33216.61 / 169.65 / 104.14

JANUS is the fastest method in all 18 settings. It reduces TPOT by 23.5%–55.4% over Naive SSP and by 37.7%–81.0% over AR. These results show that extending KV-page management across concurrent requests preserves JANUS's cross-phase reuse and performance advantage under BS > 1. We will add the multi-request design and these results to the revised manuscript.

2. SSP + JANUS versus Other Speculative-Decoding Options (Key Question 2)

Thank you for raising this question. N-gram methods require neither training nor additional model weights, but their drafts rely on repeated token patterns in the prompt or generation history. Drafting with a small model requires a compatible auxiliary checkpoint and additional memory for its weights and KV-cache. MTP requires a checkpoint equipped with trained MTP modules. SSP instead reuses the deployed model's weights and sparse KV-cache. It requires no auxiliary drafter training or checkpoint, while its draft tokens are still generated by the deployed model itself using sparse attention.

Beyond this deployment advantage, SSP is particularly attractive when long contexts make KV-cache access and transfer the dominant cost, as in agent inference, long document inference, and long code generation. During drafting, SSP accesses only selected KV-pages to generate multiple draft tokens, substantially reducing long-context KV-cache access. It then verifies these tokens together with full attention, amortizing the full-KV access over multiple accepted tokens while preserving the standard inference output. In contrast, a small drafter uses a smaller KV-cache, but its cache still grows with the full context. SSP therefore directly targets the KV-cache bottleneck that becomes increasingly important as the context grows.

In local deployment with KV-cache offloading, Naive SSP loses part of this advantage because switching between sparse drafting and full verification causes redundant KV-page transfers and layout rematerialization. JANUS reuses the overlapping KV-pages across the two phases and removes this overhead without changing draft generation or verification. Therefore, JANUS can influence the practical choice among speculative-decoding methods. For long-context, memory-constrained inference where KV-cache transfer dominates, SSP + JANUS becomes a practical training-free option. We will add this comparison and deployment guidance to the revised manuscript.

Response 2/2: Hardware Generalizability and Porting

3. Hardware Generalizability Across Memory Sizes and H2D Bandwidths (Key Question 3)

We thank the reviewer for this insightful comment. First, our evaluation in Section 5.1 already spans two GPU tiers with distinct VRAM capacities and interconnect bandwidths: an NVIDIA RTX 4090 (24 GB) and an RTX 5090 (32 GB), showing consistent speedups across both (Fig. 7).

More broadly, our empirical latency breakdowns (Fig. 2b, Figs. 8–9, and Appendix A.2) allow us to extrapolate how hardware specs affect performance bottlenecks and JANUS's speedups:

  1. H2D Bandwidth:

    • Lower H2D bandwidth or larger transfer demand (e.g., older PCIe or ultra-long contexts): Inference is H2D-transfer-bound. JANUS delivers its largest relative speedup by reusing the 46.5%–49.7% overlapping KV-pages across phases, cutting H2D time by 28.5% (Appendix A.2).
    • Higher H2D bandwidth (e.g., faster PCIe or NVLink-C2C): As H2D transfer time decreases, layout rematerialization and metadata preparation account for a larger share of the remaining overhead. JANUS eliminates rematerialization and cuts metadata latency by 32.4%–97.6% (Fig. 9), reducing KV-page preparation time by 68.0%–80.0% (Fig. 8).
  2. GPU Memory Size:

    • Smaller VRAM: Conversations trigger offloading at shorter contexts, immediately hitting redundant H2D transfers and layout rematerialization over a larger fraction of generation, amplifying JANUS's gains.
    • Larger VRAM: The offloading crossover shifts to longer contexts. Once GPU capacity is exceeded, the same layout-mismatch bottlenecks re-emerge. More broadly, as agents tackle longer-horizon reasoning and multi-turn tool use, context lengths keep expanding while VRAM remains limited, keeping KV-cache offloading relevant for these workloads, where JANUS directly applies.

We will add this concise extrapolation guideline to the revised manuscript.

4. Porting JANUS to SGLang (Key Question 4)

We thank the reviewer for this question. Because SGLang shares a similar paged KV-cache management paradigm with vLLM, JANUS can be adapted to SGLang through three focused runtime changes:

  • Unified KV-cache Pool: Add JANUS's layer-decoupled physical pool alongside SGLang's KV management backend so GPU page capacity can be shared across layers and KV-pages can be reused across drafting and verification.
  • Metadata and Budget Management: Add JANUS's forward/reverse mapping tables, coalesced Triton metadata primitives, and hierarchical budget allocator to track per-layer KV-pages and reserve the verification staging depth and sparse draft budget.
  • SSP Execution Pipeline: Adapt SGLang's speculative decoding worker from separate draft-model execution to dual-mode SSP, running sparse attention for drafting and layer-staged full KV-cache H2D streaming for verification.

We will add this porting discussion to the revised manuscript and, after further integration and validation, hope to contribute the implementation to SGLang and the broader open-source LLM community.

5. Selected Configuration in Figure 10a (Key Question 5)

Thank you for this constructive suggestion. In the revised manuscript, we will replace the small star in Figure 10a with a black solid-line box to clearly mark the selected configuration.

Dear Reviewer, Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.

Dear Authors, thank you for your detailed response. The response indeed resolved many, if not all, of my concerns. However, at this point I am keeping my score as it it. Not because I have more concerns, but I believe the scope, the alignment with the conference, and the impact and novelty of this research fits the score.

Reviewer_CL3jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JANUS, a system for accelerating long-context local agent inference with KV-cache offloading. The authors identify a layout mismatch in sparse self-speculative decoding: drafting requires sparse KV-cache across all layers, whereas verification requires full KV-cache for a small number of layers at a time. Conventional layer-bound layouts hinder cross-phase reuse, causing redundant host-to-device transfers and layout rematerialization. JANUS addresses this through a layer-decoupled GPU KV-page pool, a coalesced metadata manager with batched and fused operations, and a hierarchical allocator that selects verification staging depth and calibrates the sparse draft budget offline. Implemented on vLLM V1, JANUS is evaluated on three LLMs, two commodity GPUs, and three agent workloads. The reported results show reductions of 26.2%–53.0% in time per output token and 18.8%–46.7% in end-to-end latency relative to the naive SSP baseline.

Critical strengths

The paper explains how the different KV-cache requirements of drafting and verification interact poorly with layer-bound layouts, leading to redundant transfers and layout rematerialization. This provides a well-motivated target for optimization beyond simply combining sparse self-speculation with offloading.

JANUS combines a unified, layer-decoupled page pool with efficient metadata management and memory-budget allocation. These components address both the opportunity for cross-phase reuse and the overhead of enabling it.

Experiments cover three LLMs, two commodity GPUs, and three agent workloads, reporting substantial reductions in TPOT and end-to-end latency over both autoregressive offloading and naive SSP.

Critical weaknesses

The paper does not report workload sizes, context-length distributions, weight/KV-cache precision, available GPU KV-cache budgets, or CPU/PCIe configurations. Moreover, the predominantly normalized results obscure absolute latency, and the definition of end-to-end latency does not clarify whether tool execution and multi-turn prefilling are included.

Comparisons with existing systems are insufficient. Direct comparisons with relevant offloaded speculative decoding systems, or validation of the reimplementation against their reported behavior, would strengthen the evidence. The claim that layer-bound layouts must be abandoned also needs a clearer discussion of alternative designs that preserve resident sparse pages during verification.

The forward and reverse mappings explain page lookup, but the paper provides limited detail on eviction policies, synchronization with asynchronous transfers, partially filled pages, and rollback of rejected speculative tokens. It should also distinguish historical KV-pages eligible for reuse from candidate-token KV computed during sparse drafting.

The allocator’s generality is not sufficiently established. The argument for a staging depth of two assumes approximately uniform per-layer transfer and computation times, whereas actual transfer costs depend on residency and fragmentation. The detailed evaluation uses only one model–workload configuration. Similarly, the comparison with one online probing strategy does not establish that offline calibration remains preferable under changing context lengths or workload distributions.

Detailed component ablations are restricted to Llama3.1-8B on ToolTalk, and several large improvements are measured against deliberately inefficient variants, such as unbatched metadata operations or forward-table scans. These establish implementation necessity more clearly than comparative novelty.

Questions for authors

Could you report the number of requests, context-length distributions, weight/KV-cache precision, GPU KV-cache budgets, and CPU/PCIe configurations? What fraction of requests triggers offloading, and does end-to-end latency include multi-turn prefilling and tool execution?

How does the naive SSP implementation compare with existing offloaded speculative decoding systems such as TriForce? Could you provide a direct comparison or evidence that your reimplementation captures their relevant performance characteristics?

How does JANUS handle page eviction, synchronization with asynchronous transfers, partially filled pages, and rejected draft tokens? In particular, how are candidate-token KV entries produced during sparse drafting distinguished from those recomputed during full verification?

Do the experiments use greedy decoding or stochastic sampling? What acceptance and correction procedure is implemented, and what validation supports the claimed preservation of standard inference outputs?

How sensitive are the choices of staging depth and sparse draft budget to context length, available GPU memory, and transfer bandwidth? Does calibration on ten requests generalize to different workload distributions, and what online probing strategy is used in the comparison?

Could you report absolute latencies, measurement variability, acceptance rates, and H2D transfer volumes across configurations? Please also clarify how “ideal SSP” is constructed and how the cross-phase overlap ratio in Figure 2 is defined.

5 comments

Thank you for the constructive comments. Due to OpenReview's 5,000-character limit, we respond in three parts: (1) Experimental Configuration and Measurements (W1/Q1/Q6); (2) TriForce, page management, and output consistency (W2–3/Q2–4); and (3) allocator generality and component ablations (W4–5/Q5). We will add these results and clarifications to the revised manuscript.

Response 1/3: Experimental Configuration and Measurements

1. Detailed Experimental Configuration (Weakness 1 & Key Question 1)

We report workload sizes, context distributions, hardware, GPU KV-cache capacity, offloading behavior, absolute latency, variability, acceptance, and H2D volume below.

Workloads and context. We reserve 10 sessions from each workload for calibration. From the total 200 BFCL, 100 API-Bank, and 78 ToolTalk sessions, we evaluate the remaining 190, 90, and 68 complete sessions, respectively. Peak context retains the full history, including the system prompt, tool schemas, user–assistant messages, and tool results. The table reports percentiles averaged across model–GPU configurations.

WorkloadP25P50P75P95
ToolTalk45.9K48.7K52.2K56.9K
API-Bank44.8K47.6K50.9K55.6K
BFCL50.3K53.2K56.3K59.4K

These long contexts arise naturally as multi-turn interactions and tool outputs accumulate throughout a session.

System configuration. All models use BF16 weights and BF16 KV-cache. After model loading and runtime reservation, vLLM profiles the available GPU KV-cache capacity, which is identical across all methods. Offloading starts only when required pages exceed it.

Model / GPUCPUPCIeGPU KV-cache capacity
Qwen3-8B / RTX 4090Dual Xeon Silver 4210R3.0 ×1637.4K full-KV tokens
Llama3.1-8B / RTX 4090Dual Xeon Silver 4210R3.0 ×1640.1K full-KV tokens
Qwen3-14B / RTX 5090Dual Xeon Platinum 84705.0 ×1615.2K full-KV tokens

All evaluated sessions exceed the corresponding capacity and trigger offloading, yielding a 100% session-level rate. Each session has one warm-up and three measured runs. E2E latency includes initial prefill, decoding, local tool execution, and subsequent multi-turn prefills.

2. Absolute Performance, Variability, Acceptance Rate, and H2D Transfer Volume (Key Question 6)

For each repetition, we average all sessions and report the mean ± sample standard deviation across the three repetition-level means. Latency entries follow AR / Naive SSP / JANUS. Q8, L8, and Q14 denote Qwen3-8B, Llama3.1-8B, and Qwen3-14B.

ModelWorkloadTPOT (ms/token)E2E (s/session)JANUS acceptance
Q8BFCL510.74±8.42 / 315.26±5.17 / 203.97±3.06166.38±2.91 / 123.44±2.21 / 91.66±1.3760.92%
Q8API-Bank518.06±7.91 / 346.86±5.82 / 207.84±3.14247.63±4.38 / 217.75±3.46 / 155.89±2.7165.57%
Q8ToolTalk537.95±10.76 / 357.67±7.21 / 209.99±3.68262.41±5.64 / 204.84±3.37 / 139.22±2.4263.45%
L8BFCL613.93±9.48 / 252.80±3.69 / 137.47±1.84183.72±2.83 / 106.84±1.58 / 70.17±1.1183.78%
L8API-Bank591.55±8.27 / 249.09±4.16 / 135.78±2.09277.86±5.07 / 171.99±2.89 / 115.02±1.8784.78%
L8ToolTalk580.65±13.54 / 272.15±5.43 / 127.91±2.03273.48±5.96 / 214.96±3.74 / 114.58±1.6186.29%
Q14BFCL177.05±2.18 / 153.40±2.41 / 102.63±1.3676.34±0.94 / 68.01±1.14 / 48.13±0.6948.8%
Q14API-Bank176.17±2.69 / 151.62±2.07 / 91.29±1.22138.67±2.63 / 131.55±2.14 / 95.65±1.4347.3%
Q14ToolTalk177.15±2.35 / 148.92±1.76 / 109.90±1.58111.26±1.62 / 94.51±1.71 / 76.75±1.0652.2%

Across nine configurations, coefficients of variation are 1.2%–2.3% for TPOT and 1.2%–2.2% for E2E, showing stable performance across repeated runs.

H2D volume. Values are MiB of CPU-to-GPU KV-page transfers per output token; reductions are relative to Naive SSP.

ModelWorkloadARNaive SSPJANUSReduction
Q8BFCL5200.61348.41129.516.2%
Q8API-Bank5306.81222.81044.214.6%
Q8ToolTalk5181.91426.11201.215.8%
L8BFCL5969.91181.9956.019.1%
L8API-Bank6008.01209.0937.222.5%
L8ToolTalk5842.61145.2862.024.7%
Q14BFCL2368.71067.4724.132.2%
Q14API-Bank2467.4806.5561.830.3%
Q14ToolTalk2993.61097.8862.321.5%

Ideal SSP. It uses the same drafting, full-attention verification, sparse budget, acceptance procedure, and GPU capacity as Naive SSP, but assumes perfect cross-phase reuse: shared pages remain on GPU and only missing pages transfer. It therefore gives the minimum page movement under the same inference procedure. JANUS approaches this ideal through its layer-decoupled pool, which reuses the same physical pages rather than reconstructing phase-specific layouts.

Cross-phase overlap. For each KV-cache preparation during drafting or verification, we divide the number of required KV-pages already present on GPU by the total number of required KV-pages. Figure 2 reports this ratio averaged across both phases.

Response 2/3: TriForce KV Selection, Page Management, and Output Consistency

3. Comparison with TriForce KV Selection (Weakness 2 & Key Question 2)

We thank the reviewer for suggesting a further comparison with existing offloaded speculative decoding systems. As discussed in the appendix, our Naive SSP captures the offloading behavior relevant to these systems. To further validate this, we compare Naive SSP and JANUS under TriForce's target-level KV-selection policy.

TriForce's target-level path also performs sparse self-speculation using the target model. It changes how historical KV pages are scored and selected for sparse drafting, but does not address the repeated H2D transfers identified in our paper or enable cross-phase KV-page reuse. Thus, for the offloading behavior studied here, it retains the same layer-bound KV-cache management as Naive SSP. We therefore implemented TriForce's KV-selection policy in both Naive SSP and JANUS for comparison.

ModelWorkloadNaive SSP with TriForce KV selection TPOTJANUS with TriForce KV selection TPOTSpeedup
Qwen3-8BBFCL415.42 ms179.33 ms2.32×
Qwen3-8BAPI-Bank445.74 ms165.36 ms2.70×
Llama3.1-8BBFCL228.59 ms136.88 ms1.67×
Llama3.1-8BAPI-Bank222.09 ms138.30 ms1.61×

With TriForce's KV-selection policy, JANUS reduces TPOT by 37.7%–62.9%. These results show that JANUS's KV-cache-management benefit persists under this selection policy. We will add this comparison and clarify its scope in the revised manuscript.

4. Page Management Details (Weakness 3 & Key Question 3)

Thank you for pointing out the page-management details that would benefit from further explanation. We clarify each point below and will add these clarifications to the revised manuscript.

  • Page eviction. JANUS keeps the pages required by the current draft and the prefetched verification layers. If more GPU pages are needed, it prefers to evict a sparse page from a layer that will be used later. A page being used by a forward pass or transfer cannot be evicted. Before a physical page is assigned to another logical page, JANUS clears its old ownership and updates both mappings.
  • Asynchronous transfers. KV-cache transfers use a separate CUDA stream. A completion event is recorded after each transfer, and computation waits for this event before reading the transferred page. A page cannot be released or assigned to another logical page until the transfer using it has completed.
  • Partially filled pages. JANUS uses the current sequence length to determine which positions in the last page contain valid KV. Only these positions can be read or reused. After verification, the positions corresponding to accepted tokens become valid, while rejected positions remain invalid and can be overwritten later.
  • Rejected draft tokens. After verification, the sequence length advances only by the tokens committed to the output. Rejected positions remain outside the sequence length and are ignored by subsequent attention. Pages containing only rejected KV can be released, while rejected positions that share a page with committed tokens can be overwritten in the next iteration.
  • Historical KV versus candidate-token KV. Previously verified KV-pages may be reused. JANUS records KV produced during sparse drafting as unverified, so it is not reused as history. Full verification recomputes the candidate tokens with full attention, and only the KV of accepted tokens is then treated as verified history.

Together, these mechanisms keep transfers and page reassignment synchronized and prevent unverified KV from becoming reusable history.

5. Decoding and Output Consistency (Key Question 4)

Thank you for raising this question. All reported experiments use greedy decoding. We compared the generated token sequences and verified that AR, Naive SSP, and JANUS produced identical output tokens across all evaluated sessions.

SSP changes only how draft tokens are generated, while JANUS changes only KV-page management. These changes are independent of the verification rule, which remains identical to standard speculative decoding. Under greedy decoding, full-attention verification accepts consecutive matching drafts, replaces the first mismatch with the full-attention output, and discards the remaining drafts. Under stochastic sampling, it uses the standard acceptance-and-correction procedure.

Therefore, JANUS provides the same output-consistency guarantees as standard speculative decoding: token-level identity with AR under greedy decoding and the same output distribution as AR under stochastic sampling. We will add the decoding mode, verification procedure, and validation results to the revised manuscript.

Response 3/3

6. Allocator Generality and Sensitivity (Weakness 4 & Key Question 5)

Thank you for raising this question. We expanded the S/B sweep from Llama3.1-8B/ToolTalk to Qwen3-8B and Llama3.1-8B on RTX 4090 and Qwen3-14B on RTX 5090 across three workloads, covering 24/32 GB and PCIe 3.0/5.0. For each model–workload pair, we calibrate B on ten requests from the same workload, exclude them from evaluation, and fix B. We evaluate S={1,2,4,6,8} and B={2K,4K,6K,8K,10K}. Selected points are S=2, B=2K for Qwen3-8B/Qwen3-14B and S=2, B=6K for Llama3.1-8B.

Cells give TPOT for B=2K/4K/6K/8K/10K; ★ marks selection. Q8/L8/Q14 denote models.

ModelSBFCLAPI-BankToolTalk
Q81224.95/228.46/237.60/250.04/298.56226.90/233.59/240.01/246.13/256.32308.59/313.83/322.60/335.91/366.10
Q82203.97★/207.47/212.39/221.21/237.57207.84★/212.29/216.19/221.64/229.16209.99★/214.06/220.44/230.67/250.42
Q84201.69/206.15/212.80/220.98/238.74204.64/210.21/215.35/220.74/230.58209.90/214.22/220.97/231.22/252.99
Q86203.19/207.90/214.70/224.17/241.39206.05/211.55/215.99/223.01/233.11210.31/215.33/222.09/233.22/257.94
Q88206.47/209.27/215.14/224.45/244.75208.07/212.12/217.77/225.08/236.02219.62/222.64/226.95/235.09/264.14
L81227.96/209.31/209.43/191.95/175.04231.39/198.87/188.66/190.45/185.86217.56/191.79/181.87/230.45/293.96
L82161.83/146.25/137.47★/137.08/146.63166.57/145.69/135.78★/133.06/141.89155.56/138.60/127.91★/159.18/193.39
L84159.21/146.06/136.77/138.23/150.66163.96/144.23/135.01/132.49/146.77154.48/136.57/127.70/170.12/241.32
L86160.51/146.15/137.43/136.83/155.87163.77/143.47/137.11/134.09/151.17156.22/133.29/128.99/181.34/238.28
L88159.02/146.36/138.95/136.07/161.47165.71/142.42/135.14/134.66/155.81155.12/132.32/133.70/193.32/244.41
Q141122.34/128.74/135.70/149.78/162.21105.63/107.04/113.71/121.88/133.92132.01/132.44/141.29/153.45/166.06
Q142102.63★/105.88/110.64/117.00/132.6191.29★/94.07/95.89/103.49/112.56109.90★/108.24/115.48/124.51/141.02
Q144100.82/106.03/109.27/119.17/133.8092.92/93.37/96.60/105.49/116.40111.41/111.51/117.53/129.41/144.37
Q146102.41/106.16/111.59/121.45/137.1191.26/93.93/98.62/106.64/122.67109.05/112.80/120.52/134.10/155.16
Q148102.66/108.09/114.48/125.32/147.3991.78/93.88/99.03/108.14/128.89109.86/114.82/122.50/137.74/162.43

Selected points are within 2.48% of the grid optimum. Increasing S from 1 to 2 reduces TPOT by 8.4%–34.4%; all S>=2 settings at selected B remain within 4.6%. Every point runs the complete JANUS path, including actual residency, transfer fragmentation, asynchronous H2D transfers, and metadata costs. Thus, S=2 reaches the overlap point without unnecessary GPU pages, while larger S/B can increase page pressure. Across the three workloads and the reported 44.8K–59.4K context percentiles, ten-request calibration remains near-optimal.

Online probing. With S=2 and other settings unchanged, the baseline tests each B for one complete draft–verification cycle and uses the lowest-TPOT B for the rest of the session. To test generality, we extend the evaluation. Across BFCL/API-Bank/ToolTalk, offline calibration reduces TPOT by 12.2%/9.8%/11.6% for Qwen3-8B and 12.0%/9.5%/10.6% for Llama3.1-8B, respectively.

7. Generalized Component Ablations and Design Novelty (Weakness 5; also W2/Q2)

We extend component ablations from Llama3.1-8B/ToolTalk to six Qwen3-8B/Llama3.1-8B configurations. Values are TPOT (ms/token); parentheses show JANUS's reduction.

ModelWorkloadJANUSw/o ReuseBatched+UnfusedUnbatched+Unfusedw/o Reverse
Q8BFCL203.97271.96 (25.0%)243.11 (16.1%)3578.42 (94.3%)643.44 (68.3%)
Q8API-Bank207.84277.86 (25.2%)249.81 (16.8%)3778.91 (94.5%)702.16 (70.4%)
Q8ToolTalk209.99300.85 (30.2%)254.84 (17.6%)4117.45 (94.9%)846.73 (75.2%)
L8BFCL137.47186.53 (26.3%)164.44 (16.4%)2593.77 (94.7%)492.72 (72.1%)
L8API-Bank135.78197.07 (31.1%)163.99 (17.2%)2715.60 (95.0%)565.75 (76.0%)
L8ToolTalk127.91179.07 (28.6%)156.05 (18.0%)2623.43 (95.1%)492.45 (74.0%)

Across the six configurations, JANUS reduces TPOT by 25.0%–31.1% without reuse, 16.1%–18.0% versus batched but unfused operations, 94.3%–95.1% versus unbatched operations, and 68.3%–76.0% versus forward-table scans. No-reuse results show that cross-phase reuse generalizes; the other ablations quantify its implementation support.

JANUS's novelty is the shift from fixed per-layer reservations to a shared pool where any free physical slot can serve any layer. This supports all-layer sparse drafting and a rotating dense verification window while reusable pages remain in place. Reverse mappings and batched metadata maintain this organization. Attention Merge retains layer-bound storage and combines resident and fetched page sets; JANUS directly accesses them as one logical set. It reduces TPOT by 19.7%–22.4% versus Attention Merge.

Reviewer_adHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JANUS targets local agents that offload the KV-cache to CPU and decode with sparse self-speculation (SSP). It observes that layer-bound KV layouts make draft and verification re-transfer the 46.5–49.7% of pages they share, and replaces them with a unified GPU page pool (forward/reverse tables), fused metadata kernels, a fixed staging depth and an offline-calibrated draft budget. On vLLM V1 with three 8–14B models, two consumer GPUs and three agent benchmarks, it reports 26.2–53.0% lower TPOT than naive SSP.

Critical strengths
  1. Real, well-diagnosed problem. Fig. 2 pins down the cost of the layout mismatch (naive SSP is 44.4% slower than ideal), and the gains are consistent across all nine configurations against a careful in-engine baseline.
  2. Solid engineering. About 19K lines on vLLM V1, with internally consistent ablations.
Critical weaknesses
  1. The operating point is never shown. Every result is normalized to AR; there are no context lengths, page budget, absolute latency, accepted lengths or output-consistency checks. An 8B model on a 24 GB GPU holds tens of thousands of tokens of KV, so it is unclear whether these benchmarks need offloading at all or whether it is induced by an unstated budget; neither the premise nor the practical value can be judged.
  2. The contribution is narrower than claimed. The 78% headline is relative to AR and largely reflects SSP itself; cross-phase reuse alone gives 1.40× (App. A.2, one configuration). The claim that one "must break" the layer-bound layout is not tested against fetching only the missing pages and merging attention, and a page table with a reverse map is standard virtual-memory bookkeeping.
Questions for authors
  1. Per configuration: context length, page budget, absolute TPOT/E2E and accepted length — and how much of the naive-to-ideal gap does JANUS close?
  2. Why is a layer-decoupled pool needed rather than fetching only non-resident pages and merging attention over the two page sets?
  3. Was token-level identity with AR verified?
2 comments

1. Operating Point and Absolute Performance (Weakness 1 & Key Question 1)

Thank you for pointing out that normalized results obscure the operating points.

Built on vLLM, JANUS uses its startup profiling for GPU KV-cache capacity after model loading and runtime reservation. We do not reduce it or use request/dataset-specific thresholds; all methods use the same capacity. Offloading begins only when required KV pages exceed it.

Model / GPUFull-KV tokens16-token pages/layer
Qwen3-8B / RTX 409037.4K2.34K
Llama3.1-8B / RTX 409040.1K2.51K
Qwen3-14B / RTX 509015.2K0.95K

We run complete agent sessions with full history. Peak context includes the system prompt, tool schemas, user–assistant messages, and tool results. The percentiles below are averaged across configurations.

WorkloadP25P50P75P95
ToolTalk45.9K48.7K52.2K56.9K
API-Bank44.8K47.6K50.9K55.6K
BFCL50.3K53.2K56.3K59.4K

These long contexts arise naturally as multi-turn interactions and tool outputs accumulate. The same sessions are used for every configuration, and the measured session-level offloading rate is 100% for each configuration.

Absolute performance. Each performance entry is reported as AR / Naive SSP / JANUS. The last column reports JANUS's average number of accepted draft tokens per speculative iteration. Q8, L8, and Q14 denote Qwen3-8B, Llama3.1-8B, and Qwen3-14B.

ModelWorkloadTPOT (ms/token)E2E (s/session)Accepted
Q8BFCL510.74/315.26/203.97166.38/123.44/91.663.66
Q8API-Bank518.06/346.86/207.84247.63/217.75/155.893.93
Q8ToolTalk537.95/357.67/209.99262.41/204.84/139.223.81
L8BFCL613.93/252.80/137.47183.72/106.84/70.175.03
L8API-Bank591.55/249.09/135.78277.86/171.99/115.025.09
L8ToolTalk580.65/272.15/127.91273.48/214.96/114.585.18
Q14BFCL177.05/153.40/102.6376.34/68.01/48.132.93
Q14API-Bank176.17/151.62/91.29138.67/131.55/95.652.84
Q14ToolTalk177.15/148.92/109.90111.26/94.51/76.753.13

Finally, at the Figure 2 operating point, JANUS closes 93.2% of the E2E gap from Naive SSP to Ideal SSP: (0.5935-0.4235)/(0.5935-0.4110)=93.2%. Naive SSP is 44.4% slower than Ideal SSP, whereas JANUS is only 3.0% slower.

2. Layer-Decoupled Pool vs. Attention Merge (Weakness 2 & Key Question 2)

Thank you for suggesting this alternative. We agree this is more efficient than reloading a full layer. We therefore implemented Attention Merge, which retains resident pages, fetches only missing pages, and merges attention over the two page sets. Results are averaged over all sessions.

ModelWorkloadAM TPOTJANUSReductionAM E2EJANUSReduction
Q8API-Bank259.15207.8419.8%182.54155.8914.6%
Q8BFCL262.85203.9722.4%109.5191.6616.3%
L8API-Bank171.66135.7820.9%132.36115.0213.1%
L8BFCL171.20137.4719.7%81.2270.1713.6%

Attention Merge avoids retransferring resident pages, but still materializes missing pages in layer-local storage, processes resident and fetched pages separately, merges attention over the two page sets, and updates both sets as residency changes. JANUS instead lets resident and fetched pages form one logical page set in its layer-decoupled pool. Each page still represents one layer and token block, but any free GPU slot can store it. Page tables and reverse maps support this allocation; the contribution is the layer-decoupled allocation and cross-phase reuse they enable. Thus, Attention Merge still incurs materialization and the cost of merging attention; JANUS eliminates them through its layer-decoupled pool.

To clarify the source of the overall gains, we separate JANUS's contribution from that of SSP. The reported 78% TPOT reduction compares JANUS with AR. Against Naive SSP under the same drafting and verification procedure, JANUS provides 1.36–2.13× TPOT and 1.23–1.88× E2E speedups across nine configurations. Appendix A.2 isolates reuse within the layer-decoupled layout: reuse alone provides 1.40× TPOT and 1.24× E2E speedups. These comparisons separate JANUS's KV-cache-management gains from SSP's algorithmic gains.

3. Output Consistency (Key Question 3)

Thank you for raising this question. All experiments use greedy decoding. We verified identical output tokens among AR, Naive SSP, and JANUS across all evaluated sessions. SSP changes draft generation and JANUS changes KV-page management, while verification remains standard: full attention accepts matching drafts until the first mismatch, retains the full-attention output, and discards the remaining drafts. JANUS therefore has the same guarantees as standard speculative decoding: token-level identity with AR under greedy decoding and, with standard acceptance-and-correction, the same output distribution under stochastic sampling.

We will incorporate the above results and clarifications into the revised manuscript.

Dear Reviewer, Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.