- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
ReVisKV distinguishes sparse attention computation from sparse physical KV residency. Sparse visual attention selects vision tokens dynamically, reducing compute, but keeps complete set of KV in GPU memory. On the other hand, permanent pruning frees up GPU memory, but prevents later revisiting of discarded evidence. ReVisKV stores original-precision visual KV in pinned CPU memory for selected decoder layers and maintains bounded GPU hot buffers.
Critical strengths
- ReVisKV converts sparse visual attention into real GPU memory savings, while keeping sparsity reversible by pinning the complete visual KV on CPU memory. The formulation addresses a real difference: execution sparsity alone does not release candidate-cache storage. Immutable CPU backing and GPU-resident history give a coherent recovery mechanism without KV recomputation.
- On downstream benchmarks, ReVisKV achieves near-equal results as the dense model, even at extremely sparse token budgets.
- Method is well motivated. The proposed multi-anchor selection and temporal caching mechanisms are supported by empirical observations of cross-layer drift and cross-step overlap (Figure 1). Stage-shared metadata further reduces redundant cache-management operations.
- Comprehensive evaluations. The hot-buffer ablation isolates the benefits of temporal KV reuse, demonstrating reduced H2D traffic and decoding latency without additional GPU memory or changes in generated outputs. The evaluation also covers latency, throughput, batch scalability, and serving performance, while acknowledging performance variability at high batch sizes and limited TTFT improvements under low concurrency.
Critical weaknesses
- Results are not entirely promising. In Table 1, at both token budgets, ReVisKV only achieves second best in benchmark metric (S), consistently behind VisionPulse in both Avg and most of the 5 benchmarks. This is further evidenced by Table 2’s Qwen3-VL-Thinking-8B results.
- ReVisKV adopts the attention-based importance estimator from FastV (Eq. 1), which requires per-token attention probabilities at each anchor layer. However, IO-aware attention implementations such as FlashAttention and FlashInfer do not directly expose these probabilities, raising concerns about implementation compatibility and computational overhead. The authors should clarify (i) how the attention probabilities are obtained and whether the implementation is compatible with IO-optimized attention, and (ii) the additional latency introduced by anchor scoring. An anchor-count versus latency ablation would also be valuable to identify when the overhead of additional anchors (and other operations introduced by ReVisKV) outweighs the computational savings from sparse attention.
- The paper reports H2D traffic and aggregate decoding latency but does not isolate the overhead of cache lookup, KV recall, and data transfer. A latency breakdown comparing ReVisKV with dense and sparse attention baselines would help quantify these costs and assess the effectiveness of asynchronous execution.
- While the GPU (A800-80GB) is specified, CPU specifications, host memory bandwidth, and PCIe configuration are omitted. Being a algorithm-hardware codesign paper, these details are essential for identifying hardware bottlenecks and assessing the generalizability of the reported speedups.
- Although temporal KV reuse is central to ReVisKV, the paper does not report cache hit rates across workloads or buffer configurations. Moreover, latency results lack results from repeated runs and variance estimates.
Questions for authors
See weaknesses.
Quality, attention compatibility, and the cost of reversible offloading
Thank you for identifying the quality, kernel compatibility and runtime costs that matter for evaluating reversible offloading.
Quality and GPU residency. On Qwen3-VL-Thinking 4B, the default layout uses 62.5% less GPU visual KV than VisionPulse, with five-task mean scores 0.43 and 0.59 points lower at the 10% and 5% upper bounds, respectively (Table 1). On 8B, the two-task means for ReVisKV and VisionPulse are 68.07 and 67.69 at 10%, and 67.46 and 68.81 at 5% (Table 2). With the same 5–10% selection bounds, shifting the first 4B anchor from layer 8 to 13 improves ReVisKV from 68.44 to 70.46 on MMVet and from 72.03% to 72.68% on RealWorldQA. The revised layout stays within 0.4 points of VisionPulse on both tasks, halves its GPU visual-KV allocation, and raises the largest completed batch from 40 to 56 at 8K and from 16 to 32 at 16K (Table 5). Separately, a control with the same anchor [17], selection rule, sparse kernels and preallocated KV reduces GPU visual KV from 8.98 to 4.94 GiB, with TPOT increasing from 39.56 to 41.55 ms. All eight fixed-length output sequences agree in each of three pairs (Appendix E).
Attention compatibility and anchor cost. In Hugging Face, each anchor reads the current query's probability row from eager attention. In vLLM, model attention uses FlashAttention and a separate Triton scorer reads the query and paged keys directly. The scorer normalizes across all valid text and visual keys. Head-averaged probabilities rank the visual IDs, and temperature-scaled visual mass sets the selection count (§3.1). FlashInfer compatibility remains unverified. At 8K/B16, one, two and three anchors take 1.84, 3.71 and 5.58 ms per step for GPU scoring plus Top-K. Their TPOTs are 72.71, 88.22 and 98.95 ms, compared with 119.39 ms for VisionPulse and 126.02 ms for Dense. The layouts also change sparse-layer coverage and recall traffic. In the separate fixed-retention study, three anchors achieve the highest sparse scores on both tasks (Table 4). At B1, the three-anchor setup takes 27.96 ms, compared with 25.32 ms for VisionPulse and 24.02 ms for Dense (§§4.4, 4.7 and Appendix E).
Lookup, transfer and asynchronous execution. Figure 8 profiles Dense, VisionPulse and ReVisKV at B1, B8 and B16. At B8, ReVisKV uses 2.39 ms per step for CPU slot lookup, 8.85 ms for GPU mapped gather, and 4.46 ms for sparse attention. Mapped gather reads pinned host memory directly, so its time includes the KV transfer. The CPU and GPU tracks can overlap. With the same recall volume and outputs, synchronous submission raises TPOT from 56.19 to 66.41 ms (+18.2%, Table 6). The same-layout control in Appendix E separates cache append from offloading. Going from GPU residency to CPU backing changes TPOT from 71.52 to 56.66 ms with dynamic append and from 44.81 to 47.01 ms with preallocated append. With preallocation, peak decode allocation falls from 20.40 to 14.78 GiB.
Host and link. Appendix E specifies an A800-80GB, a Xeon Platinum 8360Y with 16 visible vCPUs, one visible NUMA node, 94.3 GiB of guest RAM and PCIe Gen4 ×16. Over seven trials, median host Triad and contiguous pinned H2D bandwidths are 62.61 and 25.34 GB/s. Mapped gather sustains 14.97–16.02 GB/s for 64–256 misses per request with 4 KiB KV pairs. These are measurements on the host used for the direct comparisons.
Cache hits and run-to-run variation. Hit rate is the fraction of selected rows served from GPU in the managed layers. It is 77.78% at 8K/B8 and 77.20% at 16K/B8 (Appendix E). At 32K, retaining inactive entries in the same allocated GPU pool raises hits from 66.2% to 78.0%, reduces H2D by 34.9% and lowers TPOT by 9.5%, with identical generated outputs (§4.6, Fig. 5). Tables 5 and 6 report mean ± sample standard deviation across three unprofiled runs per configuration. For example, default ReVisKV at 8K/B8 records 56.19 ± 0.15 ms.