acceptodds
Under review as a conference paper at ICLR 2027

Rivet: Request-Independent Vision KV Cache for Zero-Recomputation VLM Serving

Abstract

Vision-language model (VLM) serving engines redundantly compute and store an image's decoder key-value (KV) cache for every request, even when multiple requests share the same image. Existing solutions fail to fully eliminate redundancy: prefix cache methods reuse KV only when text and positions align exactly, while position-independent approaches still require per-request recomputation, preventing a single physical cache from serving multiple requests. We present Rivet, which builds a request-independent vision KV cache that a serving engine computes once and shares across requests with zero recomputation. For VLMs with multi-dimensional rotary position embeddings (MRoPE), Rivet removes the dependence of vision KV on the request with (1) canonical MRoPE pinning, which replaces the position-dependent offset with a fixed canonical offset, producing identical post-RoPE keys across requests, and (2) sink-preserving isolation that retains the attention sink while blocking context-specific preceding text, making vision-token values request-independent. Together, these techniques produce a vision-KV object that is identical for every request, so one physical copy serves an entire batch through page-table sharing. When a long text precedes the image, Rivet shifts the text positions instead of the image and preserves the accuracy of full recomputation. On Qwen3-VL at 2B, 4B, and 8B scales, Rivet achieves accuracy on par with or even surpassing partial-recomputation baselines such as VLCache and CacheBlend across VQAv2, ChartQA, and TextVQA, while delivering up to 7.4 time-to-first-token speedup over full recomputation and 1.4–2.3 over previous methods.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache Reuse, Position-Independent Cache, LLM Inference, Prefill

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bzi8OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rivet, a vision KV caching method that allows the same image KV to be shared across requests with different text prefixes. It combines canonical MRoPE pinning to remove positional dependence and sink-preserving isolation to remove contextual dependence on preceding text. This produces request-independent vision KV that can be reused without per-request recomputation. The authors evaluate Rivet on Qwen3-VL models (2B/4B/8B) across three VQA benchmarks. Rivet achieves accuracy comparable to full recomputation and reports up to 7.4× TTFT speedup. The paper also analyzes accuracy degradation with long preceding text and introduces shifted text positions to address this issue.

Critical strengths

1. The approach to request-independent vision KV caching is interesting. I like the idea of making vision KV request-independent rather than correcting it for every request. The decomposition into positional and contextual dependencies is clear, and the proposed techniques address these issues in a relatively simple way. In particular, preserving the attention sink while isolating the remaining prefix is an interesting observation. The ablation results in Table 4 provide convincing evidence that this design is important for maintaining accuracy.

2. The accuracy and efficiency results are promising. Rivet achieves accuracy comparable to full recomputation across different model sizes and datasets while eliminating per-request vision KV recomputation. The latency improvements are also substantial in the evaluated setting. I appreciate the experiments on long preceding text, which reveal a limitation of canonical positioning and demonstrate how shifted text positions can recover accuracy

Critical weaknesses

1. I am not fully convinced that the serving evaluation demonstrates Rivet's advantage over standard prefix caching.

The main motivation is that the same image may appear after different text prefixes, preventing existing prefix caches from reusing its KV. However, Section 5.1 states that the accuracy experiments use a fixed system prompt followed by the image and question. In this setting, requests sharing an image would also share the prefix up to the image, so standard prefix caching should already be effective.

While this does not undermine the accuracy evaluation, the serving experiments in Figures 3 and 4 do not clearly specify whether the text before the image differs across requests. This distinction matters because Rivet's advantage over standard prefix caching depends on prefix diversity.

I would like to see a comparison with standard prefix caching, such as SGLang RadixAttention or vLLM APC, under workloads with different text prefixes. Without this comparison, it is difficult to assess the practical advantage of Rivet over existing serving systems.

2. Some details of the VLCache baseline implementation are unclear.

Section 3 describes VLCache as storing pre-RoPE KV, whereas Appendix A states that the evaluated implementation reuses post-RoPE KV without additional rotation. I would like to understand the reason for this implementation choice and whether it affects the comparison.

I am also confused by the KV materialization overhead attributed to VLCache. Section 5.3 states that VLCache can share the unrecomputed portion of its KV cache, but Appendix A describes a per-request copy of the cached vision KV during serving.

Is this copy necessary for selective recomputation, or could the unrecomputed entries be shared directly through page-table references?

Since KV materialization contributes noticeably to the baseline latency in Figure 4, this implementation choice could affect the reported speedup over VLCache.

Questions for authors

1. What is the exact request setup in Figures 3 and 4? Do requests sharing an image have different text prefixes before the image? How would Rivet compare with standard prefix caching under varying prefix diversity?

2. Could the authors clarify the VLCache implementation? Why does it reuse post-RoPE KV without additional rotation? Is per-request KV materialization necessary, or could the unrecomputed entries be shared directly?

3. Appendix B notes that shifted text positions can limit text-prefix caching. How much does this affect the practical benefit of Rivet for long-prefix requests?

1 comment

Thank you for the insightful comments and the precise questions. We appreciate that the reviewer finds the approach to request-independent vision KV caching "interesting", the decomposition into positional and contextual dependencies "clear", and the accuracy and efficiency results "promising". Below we give a point-by-point response. Q1 and Q2 also answer Weaknesses 1 and 2. G1 and G2 refer to the general response, and revisions are highlighted in blue in the updated PDF.

Q1. Request setup of Figures 3 and 4, and prefix caching. The requests of a batch share the system prompt and the image and differ in the question, so the text before the image is identical. A prefix cache such as RadixAttention serves this setup equally well. G1 adds the comparison you ask for, with 1 and 64 distinct system prompts. With 64 prompts the prefix cache remains within 4% of full recomputation on 8B, and Rivet lowers the median TTFT by 35% against it (51.5 against 79.2 ms at 2 req/s). With one prompt the two are equal at 2 and 4 req/s, and the prefix cache is faster at 6 req/s (55.4 against 68.9 ms). The advantage of Rivet over RadixAttention is therefore limited to requests whose text before the image differs.

Q2. VLCache implementation. Section 3 describes the design of the VLCache paper, which stores KV before the rotation (its Section 4.2). Appendix A describes what we evaluated, which follows the authors' released proof of concept (nanovllm, Qwen2.5-VL). That code stores keys after the rotation, copies the stored rows into the blocks of each new request and does not rotate them again. The authors also released an SGLang implementation that follows the paper. We did not evaluate it. In the main experiments the text before the image is the short default system prompt, so the stored and the target positions nearly coincide and the missing rotation has little effect there. The revision states this in Section 3.

The per-request copy is not required by selective recomputation. It is a property of our baseline implementation. With unrecomputed entries shared through the page table, VLCache reaches 84 ms on 2B and 108 ms on 8B at B=16B=16, against 82 and 91 ms for Rivet, so Rivet is 1.03$\times$ and 1.19$\times$ faster instead of 1.4$\times$. The revision reports both variants and corrects the claim (Section 5.3).

Q3. Shifted text positions and prefix caching. Shifted positions apply only above a length threshold. The shift depends only on the number of tokens before the image, so requests with equal prefix length share shifted text KV, and requests with different lengths do not. After the new accuracy results in G2, we recommend recomputation above the threshold, so shifted positions are no longer part of the common path and standard text-prefix caching is unaffected for the requests that Rivet serves.

Reviewer_fpnPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Rivet addresses a fundamental problem with KV cache. Existing systems reuse KV cache if the prefix are exactly the same and aligned. As a result, KV cache of images interleaved within text, despite having the same token sequences, cannot be shared across requests if their preceding text differ. Rivet presents a new method to cache image KV tokens in a request-independent way to increase reusability.

Critical strengths
  1. Clear formulation and coherent cache construction. The paper separates positional and contextual dependencies in vision KV and addresses them through canonical MRoPE pinning and sink-preserving isolation. This provides a principled mechanism for constructing a reusable image-KV object while preserving the image’s internal spatial position differences.
  2. Practical integration addressing both computation and memory redundancy. Rivet implements physical sharing of one image-KV object across requests through page-table mapping. The integration demonstrates meaningful efficiency gains on warm shared-image workloads, reporting 1.4–2.3× lower TTFT than the implemented partial-recomputation baselines, without significant reduction in results.
  3. Informative ablations supporting the central design. The ablation study clearly demonstrates the importance of retaining attention sinks: retaining just one token raises accuracy from 50.6% to 89.6% on 2B and from 58.6% to 93.2% on 8B compared with canonical pinning plus full isolation. Comparing splicing against canonical recomputation also helps distinguish cache-reuse error from the effects of modifying the model.
Critical weaknesses
  1. Insufficient evaluation under diverse preceding contexts. The primary goal of Rivet is to enable cross-request vision KV reuse regardless of preceding contexts or positional assignments (Fig. 1). However, according to Sec. 5.1, the main accuracy benchmarks use a fixed “{System Prompt} {Image} {Question}” structure, where standard prefix caching (e.g., RadixAttention) already enables exact KV reuse for repeated images. While Fig. 5 evaluates varying preceding-text lengths, it is unclear how these prefixes are constructed or whether they contain semantically diverse, meaningful contexts. Varying prefix length alone does not establish robustness to different preceding contexts, especially since Rivet explicitly blocks vision tokens from attending to most preceding text. The authors should evaluate downstream accuracy under diverse, context-dependent workloads (e.g., multi-turn conversations or context-dependent VQA) to demonstrate that request-independent KV reuse preserves cross-modal reasoning capabilities. The evaluation therefore does not fully establish accuracy preservation in the diverse-context setting that motivates request-independent reuse.
  2. Insufficient serving baselines. The reported 7.4× speedup includes savings from image loading, preprocessing, and vision encoding, leaving Rivet’s incremental benefit over encoder caching unclear (Sec 5.3). The serving workload also does not specify whether preceding prefixes differ, so its advantage over exact-prefix caching remains unestablished. Comparisons against both mechanisms under identical- and diverse-prefix workloads are needed.
  3. Permissive evaluation for main experiments. App. A states that the evaluation replaces the official VQA scoring with “a prediction is correct when at least one lowercased, alphanumeric-normalized gold answer equals or occurs within the normalized prediction”. This may result in ambiguous and contradictory response, which can conceal meaningful differences in the outputs and ground truth. In Tables 3 and 4, Rivet exceeds the best baseline by only a small margin. Considering the efficiency gains, I have no problems with the small margin. However, I am uncertain whether the metrics reported reflect that under official evaluation, or a result of the permissive evaluation. I urge the authors to perform evaluation according to the official VQA guidelines (https://visualqa.org/evaluation.html).
  4. Mismatch between latency measurements and serving workload. The latency experiments use batches querying the same image, with warm GPU-resident cache hits. Furtheremore, store construction is excluded from cache hit timings. This set up demonstrates a favorable repeated-hit case, and does not follow that of real serving workloads. The evaluation does not show whether construction and cache residency costs are repaid under a realistic reuse distribution, or whether the memory savings improve sustainable concurrency or throughput, such as with cache eviction. Furthermore, using the representative 33.5MiB KV cache per image (8B model), storing the vision KV cache alone for 1000 images would require more than 32GiB, without accounting for model weights, activations and KV cache for prompts. Although the proposed store is feasible, the authors’ ideal assumption of GPU residency is not, and cannot treat loading of vision KV cache as free during serving.
  5. Inconsistent KV-cache materialization and memory accounting. Table 1 and Sec. 5.3 suggest that VLCache materializes vision KV for each request, incurring substantial copying overhead (30–60 ms). However, ther analysis in Sec. 5.3 assumes that VLCache physically shares un-recomputed KV entries and stores only recomputed entries separately, requiring just 1+rB copies rather than B. These appear inconsistent unless temporary buffers are used. The authors should clarify the actual implementation, buffer lifetimes, and peak memory consumption to ensure a fair comparison of latency and memory efficiency.
Questions for authors

Please see weaknesses.

2 comments

Thank you for the detailed and insightful review. We appreciate the recognition of the "clear formulation and coherent cache construction", of the "practical integration addressing both computation and memory redundancy", and of the "informative ablations supporting the central design". Below we give a point-by-point response. G1 and G2 refer to the general response, and revisions are highlighted in blue in the updated PDF.

1. Evaluation under diverse preceding contexts. Please see G2. The prefixes of Figure 5 are one short sentence followed by a single repeated word, so they vary the length and carry no content. G2 adds eight distinct system prompts, natural prose of 25 to 2,000 tokens, and a context-dependent task whose answer needs both the text and the image. Rivet matches full recomputation under the system prompts on both models. On 8B it loses 2 to 6 points with prose beyond 70 tokens and about 5 points on the context-dependent task, so the revision restricts the claim and recomputes requests above a length threshold.

2. Gain over encoder caching. The 4.5$\times$ and 7.4$\times$ factors are relative to full recomputation and include image loading, preprocessing and the vision encoder. The stage breakdown of Figure 4 gives the comparison with an ideal encoder cache: without those three stages, full recomputation at B=16B=16 needs 147 ms on 2B and 372 ms on 8B, against 82 and 91 ms for Rivet (1.8$\times$ and 4.1$\times$). G1 adds a measured encoder-cache baseline under online arrivals. There Rivet lowers the median TTFT against the encoder cache by 20% on 8B (51.5 against 64.5 ms at 2 req/s) and by 10% on 2B (29.4 against 32.6 ms at 4 req/s). The gain is smaller than in the stage breakdown because a prefill batch holds about one request at these loads and because TTFT also contains the text prefill and the first decoding step. The revision reports both.

3. Official VQA metric. We rescored the stored predictions of the main experiment (5,000 questions per setting) with the official metric, min⁡(1,#matches/3)\min(1, \#\text{matches}/3) averaged over the leave-one-out annotator subsets, after the official answer normalization.

Official score (%)VQAv2 2BVQAv2 4BVQAv2 8BTextVQA 2BTextVQA 4BTextVQA 8B
Full recomputation78.380.180.972.174.476.7
VLCache77.580.181.371.873.576.1
CacheBlend 0.0578.180.381.271.873.776.9
CacheBlend 0.1678.180.381.371.773.776.8
Rivet78.479.881.272.273.876.7

All methods score 8 to 9 points lower than under the binary match, and the comparison does not change. Rivet is within 0.6 points of full recomputation in all six settings. The 95% paired bootstrap interval of the difference excludes zero only for TextVQA 4B (−0.58-0.58, [−0.98,−0.17][-0.98, -0.17]). ChartQA already uses relaxed accuracy, its standard metric. The revision reports the official scores (Appendix A, Table 5).

4. GPU residency. We agree that 1,000 objects of 33.5 MiB do not fit beside the model. In G1 the store is a bounded LRU cache inside a fixed GPU byte budget that it shares with the KV pool, and a miss pays the full construction cost. With a 4 GB budget on 8B the store holds about 50 objects for about 120 distinct images, evicts 85 to 137 objects during a run and reaches a hit rate of 0.54 to 0.66. An encoder cache inside the same budget reaches 0.67 to 0.73 because its entries are about 4.5 times smaller (32 KiB against 144 KiB per vision token, including the DeepStack features). Rivet still equals or improves on the encoder cache at that budget (table in G1). The revision describes this bounded store (Sections 4.5 and 5.3, Appendix F).

5. VLCache materialization and memory accounting. The two statements were inconsistent, as you point out. Our VLCache and CacheBlend baselines copy the cached vision KV into each request's pages and then recompute the selected tokens, as the released VLCache proof of concept does. The latency of Figures 3 and 4 includes this copy (30 ms at B=16B=16), and the peak memory of that implementation is BB private objects in addition to the store. The 1+rB1+rB figure in Section 5.3 instead assumed that unrecomputed entries are shared, which our implementation does not do.

The revision reports both variants. If VLCache shares its unrecomputed entries through the page table, the copy disappears and its TTFT at B=16B=16 becomes 84 ms on 2B and 108 ms on 8B (measured total minus the measured copy stage), against 82 and 91 ms for Rivet. Rivet is then 1.03$\times$ and 1.19$\times$ faster and holds 1 object instead of 1+rB=1.81+rB=1.8. CacheBlend adapted to MRoPE re-rotates every key for the positions of the request, which writes a private copy, so its 1.7 to 2.3$\times$ gap remains. We revised the claim against VLCache to these numbers.

Thank you for the response and additional experiments. Most of my concerns are addressed and I have edited my score.

W1: Firstly, I suggest the authors state the 5 system prompts used for better clarity. The new experiments do support the reuse across different system prompts. However, I am concerned that the term "request-independent" remains too broad. The experiment varies the system prompt before the first image, leaving open whether the KV for a later image can be reused across histories that contain earlier images and dialogue. Under the author's claim of "request independence", the method should be tested on any arbitrary prefix, not just different system prompts. If the claim is that the same image KV can be reused regardless of the preceding context under Rivet’s modified computation, please establish that property explicitly, e.g., how the attention mask and position handling work when the prefix contains other images and dialogue.

W4: Since each setting uses one 400-request run, I advice the authros to frame the tail-latency gain as specific to the tested workloads, and the explanation based on admitting more requests only as a hypothesis.

Reviewer_3Y2KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Rivet enables image KV caches to be shared across requests with different text prefixes by fixing image positions and blocking attention to preceding text while retaining attention-sink tokens. This allows one physical cache to serve multiple requests without image-token recomputation. Experiments on Qwen3-VL show competitive accuracy and improvements in latency and memory use.

Critical strengths

The paper addresses a practical limitation of prefix caching when the same image appears after different prompts or conversation histories. The method requires no training, and the ablations support the importance of retaining attention sinks: removing all preceding context substantially hurts accuracy, while preserving the sink recovers most of the loss. The reported 1.4–2.3× TTFT improvement over partial-recomputation baselines is encouraging, and sharing one image KV object provides a useful memory benefit.

Critical weaknesses

The main concern is accuracy when preceding context matters for interpreting the image. Rivet changes the model’s computation, whereas the main accuracy evaluation uses a default system prompt. Prefix-length experiments help, but do not adequately test semantically different instructions or conversation histories. The serving results also focus on warm, GPU-resident caches and shared-image batches. They establish cache-hit efficiency, but leave the overall benefit under mixed hit rates and cache pressure unclear. Applicability is further limited by the single reusable vision-segment setting and the reduced text-prefix reuse introduced by the long-prefix position correction.

Questions for authors

No extra question for author

2 comments

Thank you for the constructive comments. We appreciate the recognition that the paper "addresses a practical limitation of prefix caching", that the method "requires no training", that "the ablations support the importance of retaining attention sinks", and that sharing one image KV object "provides a useful memory benefit". Below we give a point-by-point response. G1 and G2 refer to the general response, and revisions are highlighted in blue in the updated PDF.

Accuracy when the preceding context matters. G2 reports accuracy under eight semantically different system prompts, under natural prose of 25 to 2,000 tokens, and on a task whose answer combines the image with a sentence stated before it. Rivet matches full recomputation under the system prompts on both models and on the combined task on 2B. On 8B it loses 2 to 6 points with prose beyond 70 tokens and about 5 points on the combined task at every length. We restricted the claim to these measured conditions and added the length threshold as part of the method.

Conversation history. When the image arrives in the first turn, later turns append text after it, and Rivet and ordinary session caching apply unchanged. When the image arrives after earlier turns, the history is text before the image, which is the condition of G2 (ii) and (iii). Rivet then matches full recomputation only up to 70 to 100 tokens, so a server recomputes the image in that case. We did not run a multi-turn benchmark and state this scope in the revision.

Mixed hit rates and cache pressure. G1 replaces warm batches with open-loop arrivals, a Zipf popularity, a fixed byte budget with eviction and builds on a miss. With a hit rate of 0.70 on 8B, Rivet lowers the median TTFT by 37% against full recomputation and by 20% against an encoder cache at 2 and 4 req/s. Under a 4 GB budget with eviction (hit rate 0.54 to 0.66) it equals or improves on the encoder cache. The gain over the encoder cache disappears at 8 req/s and above, where decoding dominates, and with a single system prompt a prefix cache is equally fast.

Single reusable vision segment. This remains the scope of the submission. With several images in one request, isolated objects lose cross-image attention, and we state this in the deployment scope.

Shifted text positions and text-prefix reuse. The shift depends only on the number of tokens before the image, so requests with equal prefix length still share shifted text KV. Shifted positions apply only above the length threshold, and after G2 the recommended policy above the threshold is recomputation, so the common path keeps standard text positions and standard prefix caching for text.

Thanks for the detailed reply. We believe our initial rating remains appropriate and will keep it unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.