acceptodds
Under review as a conference paper at ICLR 2027

ReVisKV: Converting Dynamic Visual Execution Sparsity into Reversible GPU KV Residency

Abstract

Vision-language models repeatedly access visual evidence during extended reasoning, making the visual key-value (KV) cache a persistent source of GPU memory pressure. Dynamic visual selection reduces attention computation, but retaining complete candidate KV on GPU leaves this storage cost unchanged. We introduce ReVisKV, which preserves complete visual KV in CPU memory for managed decoder layers and maintains only a small hot buffer on GPU, recalling missing entries on demand. Motivated by cross-layer drift in visual working sets, ReVisKV refreshes selections at multiple anchors, allowing offloading to begin early while preserving access to later-needed evidence. The hot buffer exploits short-term reuse across decoding steps to reduce transfers, and stage-shared cache decisions with asynchronous execution reduce management overhead. On Qwen3-VL-Thinking 4B, ReVisKV retains 99.5% of VisionPulse's average score across five benchmarks at a 10% selection upper bound, while using 62.5% less GPU visual-KV capacity. Compared with dense decoding, the optimized runtime delivers up to decode speedup and supports up to the batch size.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Multimodal reasoning, KV cache, Efficient inference, Memory optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AyBaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ReVisKV distinguishes sparse attention computation from sparse physical KV residency. Sparse visual attention selects vision tokens dynamically, reducing compute, but keeps complete set of KV in GPU memory. On the other hand, permanent pruning frees up GPU memory, but prevents later revisiting of discarded evidence. ReVisKV stores original-precision visual KV in pinned CPU memory for selected decoder layers and maintains bounded GPU hot buffers.

Critical strengths
  1. ReVisKV converts sparse visual attention into real GPU memory savings, while keeping sparsity reversible by pinning the complete visual KV on CPU memory. The formulation addresses a real difference: execution sparsity alone does not release candidate-cache storage. Immutable CPU backing and GPU-resident history give a coherent recovery mechanism without KV recomputation.
  2. On downstream benchmarks, ReVisKV achieves near-equal results as the dense model, even at extremely sparse token budgets.
  3. Method is well motivated. The proposed multi-anchor selection and temporal caching mechanisms are supported by empirical observations of cross-layer drift and cross-step overlap (Figure 1). Stage-shared metadata further reduces redundant cache-management operations.
  4. Comprehensive evaluations. The hot-buffer ablation isolates the benefits of temporal KV reuse, demonstrating reduced H2D traffic and decoding latency without additional GPU memory or changes in generated outputs. The evaluation also covers latency, throughput, batch scalability, and serving performance, while acknowledging performance variability at high batch sizes and limited TTFT improvements under low concurrency.
Critical weaknesses
  1. Results are not entirely promising. In Table 1, at both token budgets, ReVisKV only achieves second best in benchmark metric (S), consistently behind VisionPulse in both Avg and most of the 5 benchmarks. This is further evidenced by Table 2’s Qwen3-VL-Thinking-8B results.
  2. ReVisKV adopts the attention-based importance estimator from FastV (Eq. 1), which requires per-token attention probabilities at each anchor layer. However, IO-aware attention implementations such as FlashAttention and FlashInfer do not directly expose these probabilities, raising concerns about implementation compatibility and computational overhead. The authors should clarify (i) how the attention probabilities AA are obtained and whether the implementation is compatible with IO-optimized attention, and (ii) the additional latency introduced by anchor scoring. An anchor-count versus latency ablation would also be valuable to identify when the overhead of additional anchors (and other operations introduced by ReVisKV) outweighs the computational savings from sparse attention.
  3. The paper reports H2D traffic and aggregate decoding latency but does not isolate the overhead of cache lookup, KV recall, and data transfer. A latency breakdown comparing ReVisKV with dense and sparse attention baselines would help quantify these costs and assess the effectiveness of asynchronous execution.
  4. While the GPU (A800-80GB) is specified, CPU specifications, host memory bandwidth, and PCIe configuration are omitted. Being a algorithm-hardware codesign paper, these details are essential for identifying hardware bottlenecks and assessing the generalizability of the reported speedups.
  5. Although temporal KV reuse is central to ReVisKV, the paper does not report cache hit rates across workloads or buffer configurations. Moreover, latency results lack results from repeated runs and variance estimates.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Quality, attention compatibility, and the cost of reversible offloading

Thank you for identifying the quality, kernel compatibility and runtime costs that matter for evaluating reversible offloading.

Quality and GPU residency. On Qwen3-VL-Thinking 4B, the default layout uses 62.5% less GPU visual KV than VisionPulse, with five-task mean scores 0.43 and 0.59 points lower at the 10% and 5% upper bounds, respectively (Table 1). On 8B, the two-task means for ReVisKV and VisionPulse are 68.07 and 67.69 at 10%, and 67.46 and 68.81 at 5% (Table 2). With the same 5–10% selection bounds, shifting the first 4B anchor from layer 8 to 13 improves ReVisKV from 68.44 to 70.46 on MMVet and from 72.03% to 72.68% on RealWorldQA. The revised layout stays within 0.4 points of VisionPulse on both tasks, halves its GPU visual-KV allocation, and raises the largest completed batch from 40 to 56 at 8K and from 16 to 32 at 16K (Table 5). Separately, a control with the same anchor [17], selection rule, sparse kernels and preallocated KV reduces GPU visual KV from 8.98 to 4.94 GiB, with TPOT increasing from 39.56 to 41.55 ms. All eight fixed-length output sequences agree in each of three pairs (Appendix E).

Attention compatibility and anchor cost. In Hugging Face, each anchor reads the current query's probability row from eager attention. In vLLM, model attention uses FlashAttention and a separate Triton scorer reads the query and paged keys directly. The scorer normalizes across all valid text and visual keys. Head-averaged probabilities rank the visual IDs, and temperature-scaled visual mass sets the selection count (§3.1). FlashInfer compatibility remains unverified. At 8K/B16, one, two and three anchors take 1.84, 3.71 and 5.58 ms per step for GPU scoring plus Top-K. Their TPOTs are 72.71, 88.22 and 98.95 ms, compared with 119.39 ms for VisionPulse and 126.02 ms for Dense. The layouts also change sparse-layer coverage and recall traffic. In the separate fixed-retention study, three anchors achieve the highest sparse scores on both tasks (Table 4). At B1, the three-anchor setup takes 27.96 ms, compared with 25.32 ms for VisionPulse and 24.02 ms for Dense (§§4.4, 4.7 and Appendix E).

Lookup, transfer and asynchronous execution. Figure 8 profiles Dense, VisionPulse and ReVisKV at B1, B8 and B16. At B8, ReVisKV uses 2.39 ms per step for CPU slot lookup, 8.85 ms for GPU mapped gather, and 4.46 ms for sparse attention. Mapped gather reads pinned host memory directly, so its time includes the KV transfer. The CPU and GPU tracks can overlap. With the same recall volume and outputs, synchronous submission raises TPOT from 56.19 to 66.41 ms (+18.2%, Table 6). The same-layout control in Appendix E separates cache append from offloading. Going from GPU residency to CPU backing changes TPOT from 71.52 to 56.66 ms with dynamic append and from 44.81 to 47.01 ms with preallocated append. With preallocation, peak decode allocation falls from 20.40 to 14.78 GiB.

Host and link. Appendix E specifies an A800-80GB, a Xeon Platinum 8360Y with 16 visible vCPUs, one visible NUMA node, 94.3 GiB of guest RAM and PCIe Gen4 ×16. Over seven trials, median host Triad and contiguous pinned H2D bandwidths are 62.61 and 25.34 GB/s. Mapped gather sustains 14.97–16.02 GB/s for 64–256 misses per request with 4 KiB KV pairs. These are measurements on the host used for the direct comparisons.

Cache hits and run-to-run variation. Hit rate is the fraction of selected rows served from GPU in the managed layers. It is 77.78% at 8K/B8 and 77.20% at 16K/B8 (Appendix E). At 32K, retaining inactive entries in the same allocated GPU pool raises hits from 66.2% to 78.0%, reduces H2D by 34.9% and lowers TPOT by 9.5%, with identical generated outputs (§4.6, Fig. 5). Tables 5 and 6 report mean ± sample standard deviation across three unprofiled runs per configuration. For example, default ReVisKV at 8K/B8 records 56.19 ± 0.15 ms.

Reviewer_Hu55OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the large visual KV memory cost in VLMs during long reasoning tasks. ReVisKV keeps the full visual KV for "managed" decoder layers in pinned CPU memory, holds a bounded GPU hot buffer, and recalls misses over H2D. There're three design components: (i) multi-anchor refresh: anchors at a few specific layers, and reselect the working set from the full candidate set, so drift across depth doesn't force one early selection to serve all layers; (ii) an LRU hot buffer that exploits ~63% adjacent-step working-set reuse to make recall incremental; (iii) stage-shared cache decisions plus two Triton kernels and async submission so slot bookkeeping is done once per stage rather than per layer. Experiments show ReVisKV retains 99.5% of VisionPulse's 5-benchmark mean on Qwen3-VL-Thinking 4B at a 10% upper bound while using 62.5% less GPU visual-KV capacity.

Critical strengths
  1. Working-set characterization. The paper identifies a real gap: sparse visual attention does not imply sparse visual storage. Figure 1 supports this with measurements, separating drift across decoder depth from persistence across nearby decoding steps.
  2. Multi-anchor reversible offloading. Reversibility is empirically justified, since most ever-selected tokens reactivate and a nontrivial share return only after absences too long for any bounded buffer to cover. The idea also generalizes, as alternative selectors adapted to reversible decoding recover most of the dense score they lose under native pruning.
  3. Temporal reuse and stage-shared recall. The hot-buffer ablation holds pool, policy, and generated outputs fixed, cleanly isolating the benefit of reuse at constant memory. The supporting implementation is substantive, including stage-shared slot maps, Triton gather from pinned memory, and a working vLLM integration.
Critical weaknesses
  1. Several key terms are used before definition or are never formally defined. Like figure 1's metrics "Jaccard similarity," "selected-token overlap," "layer distance," and "working-set" are reported quantitatively but given no equations anywhere in the paper.
  2. No latency breakdown. Only aggregate TPOT is ever reported. The per-step stacked breakdown would make this paper more solid, like dense attention at prefix/anchors/anchor selection/tok-k/others at 2–3 batch sizes, plus an ablation table turning off stage-sharing, the CUDA graph, and async submission individually.
  3. Add comparison of ReVisKV vs. VisionPulse in terms of TPOT, H2D, peak memory, and max batch for at matched budget and matched quality.
Questions for authors

/

1 comment
AuthorsOpenReview ↗

Matched-budget results, runtime ablations, and metric definitions

Thank you for requesting a comparison at matched budgets and comparable quality, as well as clearer runtime evidence and definitions.

VisionPulse at the same selection bounds and comparable quality. On Qwen3-VL-Thinking 4B, both methods use 5–10% dynamic selection with τ=0.4\tau=0.4. On the full MMVet and RealWorldQA splits, VisionPulse [17] scores 70.09 and 72.42%, and ReVisKV [13,17,26] scores 70.46 and 72.68%. The system results in Table 5 use those same layouts.

Visual inputTPOT (ms, mean ± SD)Visual-KV H2D (MB/output token)GPU visual KV (GiB)Peak decode allocation (GiB)Largest completed batch
8K68.01 ± 0.43 → 64.33 ± 0.160 → 12.228.98 → 4.4920.40 → 15.9040 → 56
16K112.18 ± 0.18 → 104.26 ± 0.050 → 25.5117.96 → 8.9831.14 → 22.1616 → 32

Each pair reads VisionPulse → ReVisKV. TPOT and peak memory use the same A800-80GB, requests, dynamic KV append, and three B8 runs with 1,024 output tokens per request. At the shared 5–10% bounds, mean visual participation in sparse layers is 6.07% versus 6.73% at 8K and 6.35% versus 7.00% at 16K. The capacity tests use grouped prefill with 2,048 output tokens. Each reported maximum completed three runs. Separately, a preallocated-KV control with the same anchor [17], selection rule and sparse kernels reduces GPU visual KV from 8.98 to 4.94 GiB and raises TPOT from 39.56 to 41.55 ms. All eight fixed-length output sequences agree in each of three pairs (Appendix E).

Latency breakdown and independent ablations. Figure 8 compares Dense, VisionPulse and ReVisKV at 8K with B1, B8 and B16. It identifies full-context and anchor attention, scoring and Top-K, CPU lookup, mapped GPU recall, sparse attention and dynamic KV append on separate CPU and GPU tracks, which can overlap. At B8, ReVisKV spends 2.39 ms per step on CPU slot lookup and 8.85 ms on GPU gather, including the pinned-host reads. Table 6 reports three unprofiled repeats per variant at B8. Changing stage-shared decisions to per-layer decisions raises TPOT by 21.4% (56.19 to 68.21 ms). Disabling the attention CUDA graph raises TPOT to 57.67 ms (+2.6%), and synchronous submission raises it to 66.41 ms (+18.2%). Each change preserves outputs, selected counts, visual-KV recall bytes and allocated hot-buffer capacity.

Figure 1 definitions. Section 1 defines St,ℓS_{t,\ell} as the original visual-token IDs selected at layer ℓ\ell and decode step tt. Layer distance is d=ℓ′−ℓd=\ell'-\ell for ℓ<ℓ′\ell<\ell'. Panel (a) divides the intersection by the union of the two sets. Panel (b) divides the intersection by the size of the later layer's set. Panel (c) divides ∣St,a∩St+g,a∣|S_{t,a}\cap S_{t+g,a}| by ∣St+g,a∣|S_{t+g,a}| at anchor aa and step gap gg. Panel (d) uses g=1g=1 and weights anchor-step events equally within each dataset. In the runtime, a working set is the visual IDs an anchor selects for its managed stage (§3.1). Appendix D gives the aggregation.

Reviewer_bDutOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReVisKV, which preserves complete visual KV states in CPU memory and maintains bounded GPU hot buffers for selected entries. Multiple full-context anchors refresh visual selections across decoder layers with asynchronous execution reducing overhead. Experiments show the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses a practical limitation of dynamic visual selection: reducing active attention does not reduce the underlying GPU KV allocation. The motivation is clear and interesting.
  2. The characterization of cross-layer working-set drift and temporal reuse provides useful support for multi-anchor selection and incremental recall, which is beneficial for performance.
  3. The evaluation includes actual latency improvements. The vLLM integration provides additional evidence that the memory savings can improve serving responsiveness.
Critical weaknesses
  1. Quality comparisons include VisionPulse and HybridKV, but the main latency, throughput, and serving experiments primarily compare against Dense, lacking other methods.
  2. It would be better to conduct more comparisons with other methods, e.g., HoloV (NeurIPS 2025), to fully verify the effectiveness of the proposed method. Besides, the performance improvements of the proposed method over other methods are relatively minor.
Questions for authors

Please refer to the Weaknesses.

1 comment
AuthorsOpenReview ↗

ReVisKV gains over VisionPulse and trade-offs with HoloV

Thank you for asking whether the gains carry over to stronger baselines and to serving.

VisionPulse offline. On Qwen3-VL-Thinking 4B with shared 5–10% dynamic selection bounds, ReVisKV [13,17,26] stays within 0.4 points of VisionPulse on both full benchmarks (70.46 vs. 70.09 on MMVet and 72.68% vs. 72.42% on RealWorldQA). It halves GPU visual KV and lowers time per output token (TPOT) at B8 by 5.4% at 8K and 7.1% at 16K. The largest completed batch increases from 40 to 56 and from 16 to 32, respectively. The default [8,17,26] layout reaches batches of 72 at 8K and 40 at 16K, or 1.8× and 2.5× VisionPulse's capacity. At VisionPulse's best tested batch size for each input length, default ReVisKV raises throughput from 120.11 to 142.38 tokens/s at 8K/B32 and from 64.44 to 78.47 tokens/s at 16K/B16. Its peak observed throughput on the grid improves by 20.9% and 23.7%, respectively. Capacity tests use grouped prefill and 2,048 output tokens per request on one A800-80GB (§4.7, Table 5). Table 1 reports quality for the GPU-resident HybridKV reimplementation.

Serving. Under the same vLLM GPU KV budget of 58.35 GiB at client concurrency 96 and engine limit 64, ReVisKV reduces median request completion time by 18.4% (253.96 to 207.20 s) and median time to first token by 53.2% (136.73 to 64.04 s). Throughput rises by 5.4%, and preemptions fall from 24 to 16 per run across three paired runs. Median TPOT increases by 33.2%. Appendix F reports the second engine limit in Table 10 and a scheduler setting where VisionPulse finishes requests sooner.

HoloV. The Qwen3-VL adaptation in Table 5 retains 50% of visual tokens before prefill and uses entropy-gated feed-forward network (FFN) reinjection during decoding. At 8K/B8, HoloV and ReVisKV [13,17,26] each allocate 4.49 GiB for visual attention KV. ReVisKV preserves access to the original layer-specific KV. It scores 11.74 points higher on MMVet (70.46 vs. 58.72) and 1.96 percentage points higher on RealWorldQA (72.68% vs. 70.72%). HoloV has lower TPOT (57.68 ± 0.50 vs. 64.33 ± 0.16 ms), lower peak decode allocation (15.06 vs. 15.90 GiB), and a larger completed batch (64 vs. 56). HoloV completes all three B64 runs and exhausts GPU memory in both B72 runs (§4.7 and Appendix E).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.