- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
a KV-cache precision allocation method for autoregressive reasoning and agent workloads. Historical KV states are divided into regions, each retaining a 4-bit GPU base and a quantized residual in host memory. At decision boundaries, a GRU predicts next-interval attention demand and an MLP predicts damage from reducing a region’s precision. Their product determines which residuals are loaded under a proportional GPU memory budget, allowing regions to be repeatedly demoted and restored.
Critical strengths
- Earlier information can become important again, particularly during tool interactions. Preserving a residual makes later precision restoration plausible.
- The reported improvements on τ²-bench and BFCL suggest that reversible precision allocation may be useful for workflows that revisit earlier constraints and observations.
Critical weaknesses
- The experiments do not sufficiently establish the contribution over closely related methods. ThinKV combines thought-dependent precision with evolving retention decisions, and R-KV repeatedly selects tokens during reasoning using importance and redundancy. Neither is evaluated, to judge the unique advantage this work provides.
Questions for authors
- How does DynamicKV compare with ThinKV, R-KV, and QEvict at matched GPU KV-byte budgets?
- Relation to Polestar-Cache: Polestar-Cache also uses GPU-resident summaries, CPU-resident quantized residuals, and selective retrieval to refresh cached states [1]. Although it addresses representation staleness in diffusion LLMs, whereas DynamicKV addresses quantization damage in autoregressive inference, could you clarify the novelty of DynamicKV’s residual-storage and recovery mechanism relative to this design? Which aspects extend beyond adapting selective residual retrieval, and which contributions arise specifically from the learned demand and demotion-risk predictors?
- Do the throughput measurements include attention-statistic collection, prediction, sorting, residual encoding, and CPU–GPU transfers? What are the measured host-memory footprint, transfer volume, and throughput at a common batch size?
[1] Mingyu Lee, Akshat Ramachandran, Souvik Kundu, and Tushar Krishna. Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs, 2026. https://arxiv.org/abs/2607.14107
Thank you for highlighting precision restoration in agent workflows.
Q1. Matched GPU KV budgets
On Qwen3-8B, all methods target 28.3% of BF16 GPU KV bytes and use a 32,768-token output cap. For your comparison, we show accuracy or task success (%).
DynamicKV leads the best listed baseline by 11.0/43.5 percentage points on MATH-500/BFCL V4.
With no public code, we reimplemented ThinKV using tertile thresholds after trimodal calibration failed. Repetition exhausted context, so results may not reflect the original.
Q2. Polestar-Cache and ablations
Both methods use compact GPU storage, host residuals, and selective retrieval.
Polestar-Cache stores hidden-state deviations from cluster centers and retrieves them for sparse forward computation to refresh stale KV in diffusion LLMs.
DynamicKV reconstructs higher-precision historical KV as follows.
B=Q4(X),R=Q4(Q8(X)−B).
Q denotes quantization followed by dequantization. Attention uses B at low precision and B+R after restoration to approximate Q8. Regions can be demoted and restored repeatedly without KV recomputation.
For precision allocation, a GRU predicts next-interval attention demand π. An MLP estimates demotion risk d, supervised by single-region precision interventions. The controller ranks regions by πd per additional GPU byte.
Predictor ablation.
At the same 28.3% budget without residual refinement, MATH-500 accuracy is 40.00% with π, 36.60% with d, and 51.20% with πd.
With refinement, replacing previous-interval attention with learned π in the ranking improves AIME 2025 from 43.33% to 56.67% and MATH-500 from 36.60% to 61.00%.
Recovery ablation.
All rules use the Q4 base, B+R hot representation, host residuals, πd ranking, and the same budget, without an extra BF16 or Q8 copy.
Accuracy/success (%)
Frozen fixes the initial allocation. Demote-only and full DynamicKV start with the same high-precision regions. Allowing restoration after demotion improves all three scores.
Q3. End-to-end costs and dynamic workloads
E2E timing includes attention statistics, prediction, sorting, residual encoding and validation, both transfer directions, scheduling, and synchronization. Model loading, warmup, and tokenization precede timing.
We choose Qwen3-8B for performance tests due to the RTX 4090's VRAM capacity limit.
All methods use the same RTX 4090 with a non-standard VRAM configuration (47.40 GiB visible to CUDA). We use Qwen3-8B BF16 weights and vLLM 0.22.1rc1.dev300+g59401ac9f.cu129.
FP8: E4M3, FlashInfer paged attention. Throughput: five-run medians (sample SD) at 32K input/512 output tokens and B4.
Sampled peak RSS includes pinned memory. Transfer volumes sum residual copies across prefill and decode.
DynamicKV gains 22.75%/8.12% E2E throughput over BF16/FP8, with most of the gain over FP8 before the first token. Final GPU KV usage is 5.225 GiB, 71.42%/42.84% below BF16/FP8. At 8K/B2 it reaches 85.78 tok/s, between BF16 and FP8.
Selected decode overheads.
At 32K/B4 and 512 output tokens, 12 decode checkpoints (4 requests × 3) take cumulative CPU times of 5.719 ms for statistics finalization/readback, 6.386/1.247 ms for demand/risk, 2.436 ms for sorting/allocation, 3.379 ms for joint K/V encoding, and 2.923/4.519 ms for GPU→CPU/CPU→GPU submission.
Stages can overlap, and submission times measure CPU work. Single-layer attention takes 233.70 μs/call with reconstruction, tail handling, and full-region statistics.
Poisson arrivals.
All methods use the same 48-request traces, 32K input/512 output tokens per request, a 40 GiB GPU budget, and first-come, first-served scheduling capped at B4.
Throughput covers the full run. E2E p95 includes queueing and uses the middle 32 arrivals. Measurements exclude HTTP/IPC and vLLM V1 serving. Cells report tok/s and p95 seconds.
Against BF16/FP8, DynamicKV lowers p95 by 37.05%/19.40% at 0.08414 req/s. At 0.11650 req/s, it raises throughput by 14.20%/3.21% and lowers p95 by 33.55%/8.98%.
Long decode.
B4, fixed output lengths; one full-length warmup and one timed run. Decode tok/s (including policy/transfer waits):