acceptodds
Under review as a conference paper at ICLR 2027

DynamicKV: Predictive and Reversible KV-Cache Precision Allocation for Agentic LLM Inference

38 views

Abstract

Agent workflows and long-chain reasoning continually retain and access growing histories, making the KV cache a major GPU memory cost during inference. Existing KV quantization methods typically determine precision when historical KV states enter the cache or reduce bit widths unidirectionally during generation. Analysis of real generation traces shows that earlier KV states can be reused after a period of limited attention; moreover, the effect of demoting the same KV region on subsequent inference can increase or decrease across generation stages. To anticipate changes in future demand and adapt quantization under a limited budget, we introduce DynamicKV, a predictive and reversible KV precision allocation method. DynamicKV partitions historical KV states into regions aligned with underlying storage blocks and periodically estimates two signals: , a score for significant next-interval attention, and , a score for significant damage from demotion to low-bit precision. Their product, , serves as a gated ranking score for dynamic precision allocation under a fixed proportional GPU KV-byte budget. Each region retains a low-bit base on the GPU and a separable residual in host memory. When the region becomes important again, activating the residual restores an approximately high-bit representation, enabling repeated demotion and restoration. On agent workflow tasks (-bench and BFCL V4), DynamicKV improves success rates by 5.0–10.5 percentage points over the strongest compression baseline for each model. On AIME 2024/2025 and MATH-500, it improves accuracy by up to 3.3 percentage points over the strongest compression baselines. On a single A100-80GB GPU, DynamicKV supports the concurrency of BF16 at 32K and improves aggregate output throughput by 4.7–44.3% at sequence lengths of 16K–32K.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache quantization, mixed precision, long-chain reasoning, LLM agents, predictive control

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_Rf2oOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

a KV-cache precision allocation method for autoregressive reasoning and agent workloads. Historical KV states are divided into regions, each retaining a 4-bit GPU base and a quantized residual in host memory. At decision boundaries, a GRU predicts next-interval attention demand and an MLP predicts damage from reducing a region’s precision. Their product determines which residuals are loaded under a proportional GPU memory budget, allowing regions to be repeatedly demoted and restored.

Critical strengths
  • Earlier information can become important again, particularly during tool interactions. Preserving a residual makes later precision restoration plausible.
  • The reported improvements on τ²-bench and BFCL suggest that reversible precision allocation may be useful for workflows that revisit earlier constraints and observations.
Critical weaknesses
  • The experiments do not sufficiently establish the contribution over closely related methods. ThinKV combines thought-dependent precision with evolving retention decisions, and R-KV repeatedly selects tokens during reasoning using importance and redundancy. Neither is evaluated, to judge the unique advantage this work provides.
Questions for authors
  • How does DynamicKV compare with ThinKV, R-KV, and QEvict at matched GPU KV-byte budgets?
  • Relation to Polestar-Cache: Polestar-Cache also uses GPU-resident summaries, CPU-resident quantized residuals, and selective retrieval to refresh cached states [1]. Although it addresses representation staleness in diffusion LLMs, whereas DynamicKV addresses quantization damage in autoregressive inference, could you clarify the novelty of DynamicKV’s residual-storage and recovery mechanism relative to this design? Which aspects extend beyond adapting selective residual retrieval, and which contributions arise specifically from the learned demand and demotion-risk predictors?
  • Do the throughput measurements include attention-statistic collection, prediction, sorting, residual encoding, and CPU–GPU transfers? What are the measured host-memory footprint, transfer volume, and throughput at a common batch size?

[1] Mingyu Lee, Akshat Ramachandran, Souvik Kundu, and Tushar Krishna. Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs, 2026. https://arxiv.org/abs/2607.14107

1 comment

Thank you for highlighting precision restoration in agent workflows.

Q1. Matched GPU KV budgets

On Qwen3-8B, all methods target 28.3% of BF16 GPU KV bytes and use a 32,768-token output cap. For your comparison, we show accuracy or task success (%).

MethodMATH-500BFCL V4
R-KV32.008.50
QEvict26.0012.00
ThinKV50.001.00
DynamicKV61.0055.50

DynamicKV leads the best listed baseline by 11.0/43.5 percentage points on MATH-500/BFCL V4.

With no public code, we reimplemented ThinKV using tertile thresholds after trimodal calibration failed. Repetition exhausted context, so results may not reflect the original.


Q2. Polestar-Cache and ablations

Both methods use compact GPU storage, host residuals, and selective retrieval.

Polestar-Cache stores hidden-state deviations from cluster centers and retrieves them for sparse forward computation to refresh stale KV in diffusion LLMs.

DynamicKV reconstructs higher-precision historical KV as follows.

B=Q4(X),R=Q4(Q8(X)−B).B=Q_4(X),\qquad R=Q_4(Q_8(X)-B).

QQ denotes quantization followed by dequantization. Attention uses BB at low precision and B+RB+R after restoration to approximate Q8. Regions can be demoted and restored repeatedly without KV recomputation.

For precision allocation, a GRU predicts next-interval attention demand π\pi. An MLP estimates demotion risk dd, supervised by single-region precision interventions. The controller ranks regions by πd\pi d per additional GPU byte.

Predictor ablation.

At the same 28.3% budget without residual refinement, MATH-500 accuracy is 40.00% with π\pi, 36.60% with dd, and 51.20% with πd\pi d.

With refinement, replacing previous-interval attention with learned π\pi in the ranking improves AIME 2025 from 43.33% to 56.67% and MATH-500 from 36.60% to 61.00%.

Recovery ablation.

All rules use the Q4 base, B+RB+R hot representation, host residuals, πd\pi d ranking, and the same budget, without an extra BF16 or Q8 copy.

Accuracy/success (%)

RuleMATH-500τ²-benchBFCL V4
Frozen10.0017.008.00
Demote-only51.0020.0044.50
Full DynamicKV61.0022.0055.50

Frozen fixes the initial allocation. Demote-only and full DynamicKV start with the same high-precision regions. Allowing restoration after demotion improves all three scores.


Q3. End-to-end costs and dynamic workloads

E2E timing includes attention statistics, prediction, sorting, residual encoding and validation, both transfer directions, scheduling, and synchronization. Model loading, warmup, and tokenization precede timing.

We choose Qwen3-8B for performance tests due to the RTX 4090's VRAM capacity limit.

All methods use the same RTX 4090 with a non-standard VRAM configuration (47.40 GiB visible to CUDA). We use Qwen3-8B BF16 weights and vLLM 0.22.1rc1.dev300+g59401ac9f.cu129.

FP8: E4M3, FlashInfer paged attention. Throughput: five-run medians (sample SD) at 32K input/512 output tokens and B4.

KVE2E tok/sPeak RSS GiBPinned GiBGPU→CPU MiBCPU→GPU MiB
BF1650.26 (0.05)1.7160.00000
FP857.06 (0.03)1.6850.01600
DynamicKV61.69 (0.04)9.8914.8554953.38425.53

Sampled peak RSS includes pinned memory. Transfer volumes sum residual copies across prefill and decode.

DynamicKV gains 22.75%/8.12% E2E throughput over BF16/FP8, with most of the gain over FP8 before the first token. Final GPU KV usage is 5.225 GiB, 71.42%/42.84% below BF16/FP8. At 8K/B2 it reaches 85.78 tok/s, between BF16 and FP8.

Selected decode overheads.

At 32K/B4 and 512 output tokens, 12 decode checkpoints (4 requests × 3) take cumulative CPU times of 5.719 ms for statistics finalization/readback, 6.386/1.247 ms for demand/risk, 2.436 ms for sorting/allocation, 3.379 ms for joint K/V encoding, and 2.923/4.519 ms for GPU→CPU/CPU→GPU submission.

Stages can overlap, and submission times measure CPU work. Single-layer attention takes 233.70 μs/call with reconstruction, tail handling, and full-region statistics.

Poisson arrivals.

All methods use the same 48-request traces, 32K input/512 output tokens per request, a 40 GiB GPU budget, and first-come, first-served scheduling capped at B4.

Throughput covers the full run. E2E p95 includes queueing and uses the middle 32 arrivals. Measurements exclude HTTP/IPC and vLLM V1 serving. Cells report tok/s and p95 seconds.

req/sBF16FP8DynamicKV
0.0841441.76 / 90.5342.17 / 70.7142.30 / 56.99
0.1165049.34 / 118.7954.59 / 86.7256.35 / 78.94

Against BF16/FP8, DynamicKV lowers p95 by 37.05%/19.40% at 0.08414 req/s. At 0.11650 req/s, it raises throughput by 14.20%/3.21% and lowers p95 by 33.55%/8.98%.

Long decode.

B4, fixed output lengths; one full-length warmup and one timed run. Decode tok/s (including policy/transfer waits):

Input/outputBF16FP8DynamicKV
32K/8K99.68136.42138.80
8K/32K122.23157.23155.52
Reviewer_9ychOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents DynamicKV for dynamic KV-cache precision allocation. It predicts future demand and demotion risk, and uses residuals in host memory to restore higher precision. Experiments cover mathematical reasoning, agent tasks, and system efficiency.

Critical strengths

S1) The motivation is clear. Historical regions can regain importance, and their sensitivity to quantization changes during generation. These observations support dynamic precision allocation and recovery. S2) The ablations support the main components. Repeated precision recovery improves task performance with the cache representation and memory budget held fixed. S3) On the tested agent tasks, success rates improve by 5–10.5 percentage points over the strongest compression baseline in each setting. These gains suggest practical value.

Critical weaknesses

W1) The risk predictor needs more direct validation under the mixed-precision cache states used at inference. (D1) W2) Different cache budgets and missing uncertainty estimates make the performance gains harder to assess. (D2, D3) W3) System costs and some details needed for reproducibility are not fully reported. (D4, D5)

Questions for authors

Q1) Risk labels are generated by demoting one region to 4 bits while keeping all other regions at approximately 8 bits. At inference, many regions are already at 4 bits. Please evaluate whether the risk scores remain reliable under these mixed-precision conditions. (Sections 4.3–4.5.) Q2) DynamicKV uses 28.3% of the BF16 GPU KV-cache memory, compared with 26.6% for KIVI and PM-KVQ. Please compare methods at similar cache budgets, or show how task performance changes with the budget. (Section 5.1.) Q3) Each AIME set contains only 30 questions. A gain of 3.33 percentage points corresponds to one additional correct answer. Please report the number of evaluation runs and uncertainty estimates, such as confidence intervals, for key comparisons. (Figure 4, Table 8.) Q4) Please clarify whether throughput measurements include residual transfers and scheduling overhead, and report host memory usage. The methods use different batch sizes, so latency at the same load remains unclear. (Sections 4.4 and 5.4, Table 1.) Q5) Please provide the risk predictor architecture, checkpoint interval, auxiliary loss weight, and training data size to support reproducibility. (Sections 4.6 and 5.1.)

2 comments

Thank you for recognizing the motivation, repeatable precision recovery, and gains on agent tasks.

Q1. Risk prediction under mixed precision

To address your mixed-precision question, we use the same trained MLP and high-risk threshold to measure the damage from demoting a target region to 4 bits. Other regions stay at approximately 8 bits in All-HOT and vary in precision in Mixed.

MetricAll-HOTMixed
High-risk positive rate21.9%22.3%
AUPRC0.710.71
AUROC0.960.95
Spearman correlation0.610.60

AUPRC is unchanged at similar positive rates. AUROC and Spearman correlation each decline by 0.01, indicating stable risk ranking in the tested Mixed setting.


Q2. Task quality at matched GPU KV budgets

On Qwen3-8B, all methods target 28.3% of BF16 GPU KV bytes. For your comparison, we show accuracy or task success (%).

MethodMATH-500τ²-bench
KIVI58.607.00
PM-KVQ43.008.00
MiKV35.0016.00
DynamicKV61.0022.00

DynamicKV leads the strongest baseline by 2.4 percentage points on MATH-500 and 6 points on τ²-bench at this target budget.


Q3. AIME runs and uncertainty

On AIME 2025 with Qwen3-8B, all methods use the same 30 questions, prompts, and grading, with thinking enabled, temperature 0.6, and seeds 20261108 and 20261109.

We average correctness across seeds for each question, then use 10,000 question-level bootstrap resamples for approximate 95% confidence intervals. Each question contributes one averaged observation.

MethodSeedsMean accuracy (%)Approx. 95% CI (%)
KIVI250.00[33.33, 66.67]
PM-KVQ250.00[33.33, 66.67]
DynamicKV258.33[43.33, 73.33]

DynamicKV leads by 8.33 percentage points in mean accuracy.

With only 30 questions, the intervals remain wide. We interpret this result alongside the larger benchmarks, agent tasks, and fixed-budget ablations.


Q4. System overhead and matched-load latency

To address your overhead question, E2E timing includes prefill, decode, prediction, sorting, residual encoding, CPU–GPU transfers, scheduling, and synchronization waits. It excludes HTTP/IPC and vLLM V1 serving.

We choose Qwen3-8B for performance tests due to the RTX 4090's VRAM capacity limit.

We use Qwen3-8B on an RTX 4090 with a non-standard VRAM configuration (47.40 GiB visible to CUDA), with all methods evaluated on the same GPU.

At B4 with 32K input/512 output tokens, throughput and latency are medians of five independent runs (throughput: sample SD).

Throughput includes batch cleanup; request E2E ends after generation and transfer synchronization, before cleanup.

KVE2E tok/s (SD)TPOT (ms)Request E2E (s)GPU KV (GiB)Host RSS (GiB)
BF1650.26 (0.05)37.7840.7518.2811.716
FP857.06 (0.03)28.1135.899.1411.685
DynamicKV61.69 (0.04)27.9033.175.2259.891

GPU KV memory is measured after generation. Sampled peak host RSS includes DynamicKV's 4.855 GiB of pinned memory.

Relative to BF16, DynamicKV raises throughput by 22.75% and cuts TPOT and request E2E latency by 26.13% and 18.59%. DynamicKV also reduces request E2E latency by 7.58% versus FP8.

Under the same 48-request Poisson arrival trace per rate, all methods use a 40 GiB GPU budget and a B4 cap. Request E2E p95 latency includes queueing.

Arrival rate (req/s)BF16 (s)FP8 (s)DynamicKV (s)
0.0517856.2548.6342.01
0.0841490.5370.7156.99
0.11650118.7986.7278.94

DynamicKV lowers p95 versus BF16/FP8 by 37.05%/19.40% at 0.08414 req/s and 33.55%/8.98% at 0.11650 req/s.


Q5. Predictors and training data

  • The demand predictor is a single-layer GRUCell (input 5, hidden 64) with a separate hidden state per region.
  • The risk predictor is a 5→64→1 MLP with ReLU, sigmoid output, and 449 trainable parameters.

Decisions occur every 128 decode steps or at nonterminal event ends. Regions hold up to 128 tokens and physical KV blocks hold 16 tokens.

The GRU auxiliary Huber regression loss (λₐ = 1.0, δ = 1.0) is training-only. Allocation uses πd, the product of the demand classification and risk scores.

MATH sampling is stratified by category and difficulty (seed 42).

WorkloadSourceTrainValidation
Math reasoningMATH train split20060
Agentτ²-bench v1.0.114830
Function callingBFCL V419560

Thank you for your answer. It has resolved all of my questions about this article. Therefore, we decide to raise the review score to 4.

Reviewer_WrMsOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a KV-cache compression method that efficiently compresses KV states into low-precision representations and restores them with minimal accuracy loss.

DynamicKV considers both the potential accuracy degradation caused by compressing a particular KV-cache region and the likelihood that the region will be accessed in the future.

The authors show that quantization-induced accuracy degradation and attention demand are not necessarily proportional, and that both can be predicted using separately trained models.

Critical strengths

Separately modeling demand and risk is an interesting approach, and the ability to restore previously reduced precision allows DynamicKV to achieve a high compression ratio with relatively little accuracy degradation.

Critical weaknesses

DynamicKV requires workload-specific training of its MLP and GRU predictors. However, in real-world serving environments, there is no guarantee that incoming requests will match the workloads used during training.

The paper also lacks a comparison with FP8 KV caching. Although the A100 GPU used in the experiments does not natively support FP8, FP8 KV caching is still one of the most widely used practical baselines, and such a comparison would have been valuable.

Questions for authors
  1. Have the authors evaluated how well the trained MLP and GRU predictors generalize to workloads different from those used during training?

  2. Could the authors conduct an experiment comparing DynamicKV with FP8 KV caching on hardware with native FP8 support, to better understand the trade-offs among accuracy, memory usage, and throughput?

  3. Which serving engine and version were used for the aggregate throughput experiments? Have the authors also evaluated throughput with an 8K context length?

  4. Could the authors provide a kernel-level performance comparison of INT4, INT8, BF16, and DynamicKV's Base + Residual representation? It would also be helpful to analyze the overhead introduced by residual reconstruction and CPU-GPU transfers.

4 comments

Thank you for your constructive feedback.

Q1. Generalization across agent workloads

To address your question about generalization, we trained GRU and MLP only on τ²-bench (customer service) and tested without fine-tuning on BFCL V4 (function calls, search, and memory).

Train → TestGRU AUPRCMLP AUPRCTask success %
τ² → τ²0.93470.799222.00
τ² → BFCL V40.92140.705452.50
BFCL → BFCL V40.96560.718855.50

On BFCL V4, transfer lowers success by 3 percentage points. Both models retain predictive value (GRU/MLP positive-label rates 47.63%/38.28%). Broader generalization remains open.


Q2. Comparison with native FP8 KV caching

Quality.

Qwen3-8B on an FP8-capable RTX 4090 (accuracy/success in %).

KVAIME24τ²-bench
BF1643.3331.00
FP830.0020.00
DynamicKV30.0022.00

DynamicKV matches FP8 on AIME24 and improves τ²-bench success by 2 percentage points through selective 4-bit residual restoration. BF16 leads on both benchmarks.

Performance.

We choose Qwen3-8B for performance tests due to the RTX 4090's VRAM capacity limit.

We use an RTX 4090 with a non-standard VRAM configuration (47.40 GiB visible to CUDA); all methods use this GPU in Q2–Q4.

Table setup: Qwen3-8B BF16 weights replacing Qwen3.5-27B; FP8 E4M3 with FlashInfer paged attention; 32K input/512 output tokens; 8192-token prefill chunks.

For your comparison, timing values are five-run medians; KV memory is measured after generation.

BatchMethodKV GiBDecode tok/sE2E tok/s
4BF1618.281105.8950.26
4FP89.141142.3057.06
4DynamicKV5.225143.3561.69
8FP818.281207.2465.25
8DynamicKV10.450207.1571.05

DynamicKV uses 42.84% less GPU KV memory than FP8, with similar decode speed and 8.12%–8.89% higher E2E throughput.

At B4, decode and E2E gains over BF16 are 35.38% and 22.75%. E2E includes prefill, decode, policy, transfers, and scheduling.

Long decode.

B4, fixed output lengths; one full-length warmup and one timed run. Decode tok/s (including policy/transfer waits):

Input/outputBF16FP8DynamicKV
32K/8K99.68136.42138.80
8K/32K122.23157.23155.52

Q3. Engine details and 8K evaluation

For your reference, we use the vLLM 0.22.1rc1.dev300+g59401ac9f.cu129 Qwen3 executor with PyTorch 2.11.0+cu129, FlashInfer 0.6.12, and Triton 3.6.0.

Timing covers scheduling and execution, excluding HTTP/IPC and vLLM V1 serving. We use the official KIVI implementation.

E2E throughput (tok/s) at 8K input/512 output tokens

BatchBF16FP8DynamicKV
150.1851.4249.87
284.1788.6385.78
4133.77145.52139.37
8189.24213.95202.36

At B8 (medians), DynamicKV gains 6.93% E2E over BF16 and cuts KV memory from 9.562 to 2.793 GiB (70.8%). FP8 is faster. Other batches use single runs.

Matched-concurrency cached-prefix throughput (tok/s) under a 44 GiB GPU budget

ContextConcurrencyBF16DynamicKVGain
8K23476.82581.2721.91%
32K6124.45179.9044.55%

Peak throughput (tok/s), with concurrency in parentheses

ContextBF16FP8DynamicKV
8K476.82 (23)906.12 (46)785.26 (58)
32K124.45 (6)244.78 (12)273.27 (15)

At 8K/32K, DynamicKV reaches 1.65×/2.20× BF16 peak throughput and supports higher concurrency than FP8. Its peak trails FP8 at 8K and leads at 32K.


Q4. Kernel costs and residual overhead

Single-layer CUDA Graph decode timing (μs/call) uses identical Q/K/V inputs and includes KV append, tail handling, attention, and LSE.

INT8/INT4/DynamicKV also include full-region statistics.

KV/kernel8K/B132K/B18K/B432K/B4
BF16 paged FA239.05171.89159.55593.30
FP8 FlashInfer23.4056.6956.52315.27
INT8 affine UINT829.2578.0386.02370.91
INT4 base23.0849.5554.63210.24
DynamicKV base + residual28.0057.1157.87233.70
All residuals on GPU33.5095.18107.62431.03

At 32K/B4, DynamicKV is 2.54×/1.35×/1.59× as fast as BF16/FP8/INT8. The base+residual path adds 11.16% latency over INT4.

Separate profiling measures 201.79 μs for packed attention with reconstruction. Encoding a 128-token K/V residual region (36 layers, 8 KV heads) takes 0.377 ms with allocation and validation.

Regions (MiB)GPU→CPU msCPU→GPU ms
1 (4.781)0.6681.044
8 (38.250)4.3184.157
32 (153.000)16.97415.994

CPU→GPU includes allocation, transfer, and sync. GPU→CPU saves new residuals to host. E2E includes transfer waits.

Peak 32K/B4 host RSS is 9.891 GiB, including 4.855 GiB pinned.

Thank you for conducting these additional experiments. The results suggest that DynamicKV is competitive with FP8 KV caching, particularly in terms of memory efficiency and throughput. Most of my concerns have now been addressed.

As a final check, I would appreciate seeing the results for the reverse cross-workload setting (training on BFCL V4 and testing on τ²-bench without fine-tuning). These results would help me reassess my current review score.

Thank you for your follow-up. Following your suggestions, we added a new experiment, which trained the GRU and MLP predictors on BFCL V4 and tested them on τ²-bench without fine-tuning.

SourceTrain → TestGRU AUPRCMLP AUPRCTask Success (%)
Paperτ² → τ²0.93470.799222.00
New experimentBFCL V4 → τ²0.95570.687121.00

Task success remained comparable (21% vs. 22%). GRU AUPRC increased, while MLP AUPRC was lower. The positive-label rates were 41.22% for GRU and 30.95% for MLP, approximating their random-ranking AUPRC baselines. Both predictors achieved AUPRC scores substantially above these baselines.

Together with the previous τ² → BFCL V4 results, these findings support cross-workload transfer in both directions. We will include these results in the revised paper.

Thank you again for your feedback.

Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns, and I have updated my review score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.