- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Rivet, a vision KV caching method that allows the same image KV to be shared across requests with different text prefixes. It combines canonical MRoPE pinning to remove positional dependence and sink-preserving isolation to remove contextual dependence on preceding text. This produces request-independent vision KV that can be reused without per-request recomputation. The authors evaluate Rivet on Qwen3-VL models (2B/4B/8B) across three VQA benchmarks. Rivet achieves accuracy comparable to full recomputation and reports up to 7.4× TTFT speedup. The paper also analyzes accuracy degradation with long preceding text and introduces shifted text positions to address this issue.
Critical strengths
1. The approach to request-independent vision KV caching is interesting. I like the idea of making vision KV request-independent rather than correcting it for every request. The decomposition into positional and contextual dependencies is clear, and the proposed techniques address these issues in a relatively simple way. In particular, preserving the attention sink while isolating the remaining prefix is an interesting observation. The ablation results in Table 4 provide convincing evidence that this design is important for maintaining accuracy.
2. The accuracy and efficiency results are promising. Rivet achieves accuracy comparable to full recomputation across different model sizes and datasets while eliminating per-request vision KV recomputation. The latency improvements are also substantial in the evaluated setting. I appreciate the experiments on long preceding text, which reveal a limitation of canonical positioning and demonstrate how shifted text positions can recover accuracy
Critical weaknesses
1. I am not fully convinced that the serving evaluation demonstrates Rivet's advantage over standard prefix caching.
The main motivation is that the same image may appear after different text prefixes, preventing existing prefix caches from reusing its KV. However, Section 5.1 states that the accuracy experiments use a fixed system prompt followed by the image and question. In this setting, requests sharing an image would also share the prefix up to the image, so standard prefix caching should already be effective.
While this does not undermine the accuracy evaluation, the serving experiments in Figures 3 and 4 do not clearly specify whether the text before the image differs across requests. This distinction matters because Rivet's advantage over standard prefix caching depends on prefix diversity.
I would like to see a comparison with standard prefix caching, such as SGLang RadixAttention or vLLM APC, under workloads with different text prefixes. Without this comparison, it is difficult to assess the practical advantage of Rivet over existing serving systems.
2. Some details of the VLCache baseline implementation are unclear.
Section 3 describes VLCache as storing pre-RoPE KV, whereas Appendix A states that the evaluated implementation reuses post-RoPE KV without additional rotation. I would like to understand the reason for this implementation choice and whether it affects the comparison.
I am also confused by the KV materialization overhead attributed to VLCache. Section 5.3 states that VLCache can share the unrecomputed portion of its KV cache, but Appendix A describes a per-request copy of the cached vision KV during serving.
Is this copy necessary for selective recomputation, or could the unrecomputed entries be shared directly through page-table references?
Since KV materialization contributes noticeably to the baseline latency in Figure 4, this implementation choice could affect the reported speedup over VLCache.
Questions for authors
1. What is the exact request setup in Figures 3 and 4? Do requests sharing an image have different text prefixes before the image? How would Rivet compare with standard prefix caching under varying prefix diversity?
2. Could the authors clarify the VLCache implementation? Why does it reuse post-RoPE KV without additional rotation? Is per-request KV materialization necessary, or could the unrecomputed entries be shared directly?
3. Appendix B notes that shifted text positions can limit text-prefix caching. How much does this affect the practical benefit of Rivet for long-prefix requests?
Thank you for the insightful comments and the precise questions. We appreciate that the reviewer finds the approach to request-independent vision KV caching "interesting", the decomposition into positional and contextual dependencies "clear", and the accuracy and efficiency results "promising". Below we give a point-by-point response. Q1 and Q2 also answer Weaknesses 1 and 2. G1 and G2 refer to the general response, and revisions are highlighted in blue in the updated PDF.
Q1. Request setup of Figures 3 and 4, and prefix caching. The requests of a batch share the system prompt and the image and differ in the question, so the text before the image is identical. A prefix cache such as RadixAttention serves this setup equally well. G1 adds the comparison you ask for, with 1 and 64 distinct system prompts. With 64 prompts the prefix cache remains within 4% of full recomputation on 8B, and Rivet lowers the median TTFT by 35% against it (51.5 against 79.2 ms at 2 req/s). With one prompt the two are equal at 2 and 4 req/s, and the prefix cache is faster at 6 req/s (55.4 against 68.9 ms). The advantage of Rivet over RadixAttention is therefore limited to requests whose text before the image differs.
Q2. VLCache implementation. Section 3 describes the design of the VLCache paper, which stores KV before the rotation (its Section 4.2). Appendix A describes what we evaluated, which follows the authors' released proof of concept (nanovllm, Qwen2.5-VL). That code stores keys after the rotation, copies the stored rows into the blocks of each new request and does not rotate them again. The authors also released an SGLang implementation that follows the paper. We did not evaluate it. In the main experiments the text before the image is the short default system prompt, so the stored and the target positions nearly coincide and the missing rotation has little effect there. The revision states this in Section 3.
The per-request copy is not required by selective recomputation. It is a property of our baseline implementation. With unrecomputed entries shared through the page table, VLCache reaches 84 ms on 2B and 108 ms on 8B at B=16, against 82 and 91 ms for Rivet, so Rivet is 1.03$\times$ and 1.19$\times$ faster instead of 1.4$\times$. The revision reports both variants and corrects the claim (Section 5.3).
Q3. Shifted text positions and prefix caching. Shifted positions apply only above a length threshold. The shift depends only on the number of tokens before the image, so requests with equal prefix length share shifted text KV, and requests with different lengths do not. After the new accuracy results in G2, we recommend recomputation above the threshold, so shifted positions are no longer part of the common path and standard text-prefix caching is unaffected for the requests that Rivet serves.