acceptodds
Under review as a conference paper at ICLR 2027

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

10 views

Abstract

Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23× parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8× gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04× prefill speedup. For online serving, it handles 1.8× the request throughput of target prefill on a single GPU, and at the target’s saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0× and 6.4× respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache transfer, cross-model KV cache, prefill acceleration, selective recomputation, LLM serving, model cascades, time to first token, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MXskOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important problem in multi-model LLM serving: reusing the KV cache produced by a smaller model when a larger model takes over a request. RaReCache first translates the source cache using a calibrated linear mapping and then selectively recomputes a subset of token positions in the target model. Its main contribution is a target-free selection criterion, rank disagreement, which measures the energy of mapped representations outside the dominant subspace of the calibration predictions.

Critical strengths
  • The problem is practically relevant. Model cascades and mid-conversation model switching can incur substantial repeated prefill costs, and selectively repairing a transferred cache is a plausible way to trade computation for accuracy.
  • The method is relatively simple and avoids a target-model forward pass for selection at inference time. The decomposition of the score into a full-prediction norm and a low-rank projection norm is straightforward, and the implementation optimizations address meaningful overheads.
Critical weaknesses
  • The score measures projection residual energy, but the paper does not establish that this reliably identifies mapping uncertainty or tokens with the greatest recomputation benefit.
  • The selector may primarily benefit from avoiding template tokens. Comparisons against content-only random selection and template-filtered attention/error selectors are needed to isolate its contribution.
Questions for authors
  • Does rank disagreement retain its advantage when all selectors exclude fixed template and instruction tokens?
  • How much performance is lost when using one shared calibration corpus across tasks or changing prompt templates?
  • Some figures have layout issues, e.g., Figure 2 extends below the bottom edge of the adjacent text block.
3 comments

We thank the reviewer MXsk for their appreciation of our work and insightful feedback. The following is our detailed response to their concerns and questions.

1. Does rank disagreement retain its advantage when all selectors exclude fixed template and instruction tokens?

We present the selector comparison using Qwen3-0.6B → 14B in Table 12. TI denotes template and instruction tokens. Rank disagreement retains a significant advantage over other selectors for longer context lengths.

Table 12. Selector comparison for Qwen3-0.6B→14B

DatasetρRandom w/o TIUniform w/o TIRank Disagreement
GSM8K (n=300)0.391.091.790.3
MATH-500 (n=500)0.372.271.875.4
LongBench-E 0-4k (n=467)0.544.847.658.8
LongBench-E 4-8k (n=470)0.546.345.752.9
LongBench-E 8k+ (n=337)0.538.644.952.7

Rank Disagreement consistently outperforms content-only random and uniform selection across the evaluated LongBench-E length buckets. On short prompts such as GSM8K and MATH-500, template and instruction (TI) tokens constitute approximately 45–60% of the input, so excluding them concentrates recomputation on question tokens and provides a strong baseline. Content-only uniform selection achieves the highest GSM8K accuracy, though rank disagreement’s is close with 90.3% accuracy. On MATH-500, rank disagreement outperforms random and uniform selection excluding TI by 3.2 and 3.6 pp respectively.

Overall, these results support rank disagreement as an effective selector without explicit template annotations, and with clear long-context benefits over content-only random and uniform selection.

2. How much performance is lost when using one shared calibration corpus across tasks or changing prompt templates?

To analyze generalization of the calibration map across benchmarks, we evaluated MATH-500 with a map calibrated on GSM8K train set, MATH-500 train set, and a mix of both. At 30% recomputation with rank disagreement, mixed calibration achieves 73.6% compared with 75.4% for the MATH-500 calibrated map, suffering an accuracy drop of 1.8 pp. The GSM8K map suffers a much higher accuracy drop of 6.6 pp. Therefore, including the target prompt distribution during calibration helps the map generalize, though generalization costs a small drop in accuracy over task-specific calibration.

Table 13. MATH-500 evaluation on Qwen3-0.6B→14B using shared calibration corpus

Recompute selectorρGSM8K-train mapMATH-train mapMixed map (GSM8K+MATH train)
Transfer026.261.458.0
Mapping (Rank Disagreement)0.3068.875.473.6
3.The score measures projection residual energy, but the paper does not establish that this reliably identifies mapping uncertainty or tokens with the greatest recomputation benefit.

We agree that projection residual energy alone does not establish mapping uncertainty or identify tokens with the greatest recomputation benefit. It measures the component of a token’s mapped KV representation outside the fitted calibration subspace and serves as a heuristic for prioritizing recomputation.

We provide empirical evidence to support this heuristic. Across multiple evaluated tasks and context lengths, the overall method retains 90–99% of target-model accuracy while recomputing fewer than 50% of tokens. These results show the heuristic is effective, and at a matched budget, it outperforms the alternatives.

4. Some figures have layout issues, e.g., Figure 2 extends below the bottom edge of the adjacent text block.

Thanks for pointing this out. We will fix the layout issue of Figure 2 in the updated manuscript.

Dear Reviewer,

We thank you again for your valuable feedback and insightful questions. Based on your suggestions, we have provided detailed experimental evaluations. We have also uploaded a revised PDF with the following changes:

  • We have included a comprehensive selector comparison excluding template and instruction tokens (Table 3, Section 4.3)
  • We demonstrated generalization across tasks using one shared calibration corpus for GSM8K and MATH-500 (Table 11, Appendix Section D.3) and LongBench-E (Table 1, Section 4.2)
  • We fixed the layout issue in Figure 2

Since the author-reviewer discussion phase ends on November 18 (AoE), we would like to kindly ask whether our responses have addressed your concerns. We are happy to engage in further discussions or provide additional clarifications if needed.

Thank you again for your time and thoughtful review.

Sincerely,

Authors Paper-43343

Thanks for the detailed responses and additional experiments, which substantively address my empirical concerns; I will maintain my positive score.

Reviewer_rqGbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache reuse between models of different sizes within the same model family. Existing linear mappings lose task accuracy when the capacity gap is large. RaReCache repairs the mapped cache by recomputing only selected token positions in the target model. Rank disagreement, the proposed selection metric, measures the squared energy of centered mapped outputs outside the dominant calibration-data subspace. Scores are summed across layers and KV tensors to select one shared set of positions. Scoring does not require full target-model prefill for the current input, although fitting the offline mapping still requires target-model calibration caches. The method is evaluated within the Qwen3 and Llama 3 families.

Critical strengths
  1. The problem is well motivated and practically relevant. Cross-model hand-offs are increasingly plausible in cascaded and agentic systems, while ordinary prefix caching does not solve reuse across models of different sizes.

  2. This paper shows concrete improvements on making existing model cache useful to a subsequent target model. A modest amount of target-side recomputation substantially improves transfer quality across large capacity gaps.

Critical weaknesses
  1. The main selector comparison uses one model pair and 300 GSM8K examples. Much of the observed difference comes from whether a selector spends its budget on template tokens. The paper should compare against random selection excluding templates, uniform selection within the content, and the centered prediction norm.

  2. The quality and speed results support a narrower use case than the paper claims. For example, in Table 5, MMLU-Redux retains only 77.9% of target accuracy at ρ=0.3; LongBench-E QA is not reported at that budget and retains 90.7% of target F1 even at ρ=0.5. Thus, 95–99% retention on three reasoning benchmarks cannot characterize all five benchmarks.

Questions for authors
  1. With the same mapping, model pair, and budget, please compare against random/uniform selection excluding template tokens and the r=0 norm score. Please include GSM8K and one other task, and explain why r=128 was chosen.

  2. How well does the method generalize, and how does it compare with a nonlinear mapper? Please report any results using one fixed mixed-domain calibration mapping. Under the same calibration budget, how do a nonlinear mapper and linear mapping plus recomputation compare in accuracy and cost?

  3. If the last prompt token is not selected for recomputation, how does RaReCache obtain the logits for the first output token? Is that position always recomputed, and is its cost included in the recomputation budget and reported TTFT? Please provide pseudocode for recomputation and cache update.

  4. As stated in Section 4.6, 30% recomputation retains 95-99% of target accuracy. I’m curious why MMLU-Redux in Table 5 does not support that range. Could the authors explain this?

5 comments

We thank the reviewer rqGb for their appreciation of our work and insightful feedback. The following is our detailed response to their concerns and questions.

1. The paper should compare against random selection excluding templates, uniform selection within the content, and the centered prediction norm.

We thank the reviewer for suggesting a comprehensive set of selectors. We present the selector comparison using Qwen3-0.6B → 14B in Table 9. TI denotes template and instruction tokens. Centered prediction norm is essentially rank disagreement with rank 0, or the squared distance of mapped KV from its calibration-set mean, summed over layers and K/V.

Table 9. Selector comparison for Qwen3-0.6B→14B

DatasetρRandom w/o TIUniform w/o TICentered pred. normRank Disagreement
GSM8K (n=300)0.391.091.776.090.3
MATH-500 (n=500)0.372.271.855.875.4
LongBench-E 0-4k (n=467)0.544.847.657.358.8
LongBench-E 4-8k (n=470)0.546.345.755.052.9
LongBench-E 8k+ (n=337)0.538.644.951.752.7

Rank Disagreement consistently outperforms content-only random and uniform selection across the evaluated LongBench-E length buckets. On short prompts such as GSM8K and MATH-500, template and instruction (TI) tokens constitute approximately 45–60% of the input, so excluding them concentrates recomputation on question tokens and provides a strong baseline. Content-only uniform selection achieves the highest GSM8K accuracy, though rank disagreement’s is close with 90.3% accuracy. On MATH-500, rank disagreement outperforms random and uniform selection excluding TI by 3.2 and 3.6 pp respectively. Centered prediction norm, on the other hand, performs substantially worse than rank disagreement on short prompts but comparably on LongBench-E. Overall, these results support rank disagreement as an effective selector without explicit template annotations, and with clear long-context benefits over content-only random and uniform selection.

2. Please include GSM8K and one other task, and explain why r=128 was chosen.

We thank the reviewer for suggesting this ablation. Rank r=128 achieves the highest accuracy on MMLU-Redux, while r=64 performs best on GSM8K (n=300). Across r=64–256, accuracy varies by at most 2.4 percentage points on GSM8K and 0.7 on MMLU-Redux. We use r=128 across benchmarks to avoid task-specific tuning, although validation-based task-specific rank selection is likely to further improve performance.

Table 10. Rank selection on GSM8K and MMLU-Redux

Probe rank r064128256
GSM8K (n=300)76.092.790.391.0
MMLU-Redux (n=5700)47.860.561.260.7
3. How well does the method generalize? Please report any results using one fixed mixed-domain calibration mapping. 

To analyze generalization of the calibration map across benchmarks, we evaluated MATH-500 with a map calibrated on GSM8K train set, MATH-500 train set, and a mix of both. Mixed calibration achieves 73.6% against 75.4% for task-specific mapping, incurring an accuracy drop of 1.8 pp. Therefore, including the target prompt distribution during calibration helps the map generalize, though generalization costs a small drop in accuracy over task-specific calibration. Moreover, without recomputation, the corresponding cross-task gap is substantially larger: 26.2% vs 61.4%. Thus, selective recomputation mitigates calibration mismatch, while including the target task distribution in a mixed calibration set further improves performance.

Notably, all 5 tasks across different context length buckets in LongBench also use a single calibration map fitted on a mix of the train sets of those tasks.

Table 11. MATH-500 evaluation on Qwen3-0.6B→14B using mixed-calibration mapping

Recompute selectorρGSM8K-train mapMATH-train mapMixed map (GSM8K+MATH train)
Transfer026.261.458.0
Mapping (Rank Disagreement)0.3068.875.473.6
4. Under the same calibration budget, how do a nonlinear mapper and linear mapping plus recomputation compare in accuracy and cost?

We agree this comparison is important. However, recomputation complements the mapper rather than replacing it: recomputed positions overwrite the mapped entries whatever the mapper is, so the same procedure applies on top of a nonlinear mapper. Moreover, we would also like to point out that our map is closed-form and gradient-free. It needs one forward pass over the calibration set, and has no optimization hyperparameters.

We will train an MLP mapper on exactly the same calibration rows as our calibration maps and provide a direct comparison between non-linear mapper and linear mapper with recompute in the camera-ready version, if accepted.

5. If the last prompt token is not selected for recomputation, how does RaReCache obtain the logits for the first output token? Is that position always recomputed, and is its cost included in the recomputation budget and reported TTFT? Please provide pseudocode for recomputation and cache update.

Yes, the last prompt token is always recomputed by the target, whether or not the selector picks it. Its cost is included in every latency and TTFT number we report. The selector picks round(ρT) positions. If the last token is not among them, it is added, so the recompute set is either round(ρT) or round(ρT) + 1 positions.

The first output token is predicted from the hidden state at the last prompt position, so that position has to pass through all of the target's layers. We therefore add position T−1 to the recompute set after selection. It then runs through the target like every other recomputed position, and the LM head is applied to its final hidden state.

Below is the Pseudocode for our method. We will add the pseudocode to the appendix.

Pseudocode for RaReCache

Input: prompt x[0..T-1], source model S, target model M with L layers,
       linear maps {W_l}, recompute budget ρ

# 1. Source prefill and mapping (all T positions)
src_kv      ← S.prefill(x)
(K̂_l, V̂_l) ← W_l(src_kv)                for l = 1..L    # mapped target cache

# 2. Selection (no target forward needed)
s ← RankDisagreement(src_kv)                             # one score per position
P ← top-k(s, k = round(ρT))
P ← sort(P ∪ {T−1})                                      # last token always included

# 3. Sparse recompute with in-place cache update
h ← M.embed(x[P])
for l = 1..L:
    q, k, v ← M.layer[l].qkv(h, positions = P)           # RoPE at true positions
    K̂_l[P] ← k ;  V̂_l[P] ← v                             # overwrite BEFORE attention
    a ← Attention(q, K̂_l, V̂_l, mask: query p sees keys j ≤ p)
    h ← M.layer[l].output_and_mlp(h, a)
logits ← M.lm_head(M.norm(h[row of T−1]))                # first output token
y_1    ← decode_rule(logits)

# 4. Generation continues normally from the updated cache
for t = 1, 2, ...:  y_{t+1} ← M.decode_step(y_t, cache = {K̂_l, V̂_l})
6. As stated in Section 4.6, 30% recomputation retains 95-99% of target accuracy. I’m curious why MMLU-Redux in Table 5 does not support that range. Could the authors explain this?

Thank you for this comment. The 95–99% target-accuracy retention at 30% recomputation applies to our evaluations of GSM8K, ARC-Easy, and ARC-Challenge, all of which use chain-of-thought (CoT) prompting. We observe that generating intermediate reasoning steps helps the model compensate for perturbations introduced by approximate KV reuse. MMLU-Redux was evaluated without CoT prompting and exhibits lower retention, as reported in Table 5. We will revise Section 4.6 and the abstract to explicitly clarify the retention rates obtained for individual benchmarks.

Dear Reviewer,

We thank you again for your detailed feedback and insightful questions. Based on your suggestions, we have provided detailed experimental evaluations. We have also uploaded a revised PDF with the following changes:

  • We have included a comprehensive selector comparison excluding template and instruction tokens (Table 3, Section 4.3)
  • Generalization across tasks using mixed-domain calibration mapping for GSM8K and MATH-500 (Table 11, Appendix Section D.3) and LongBench-E (Table 1, Section 4.2)
  • Pseudocode for our method (Appendix Section B)
  • Rank ablation on GSM8K, MMLU-Redux and MATH-500 (Table 12, Appendix Section D.4)
  • Revised the abstract to clarify that the 95-99% target accuracy is obtained on GSM8K and ARC.

Since the author-reviewer discussion phase ends on November 18 (AoE), we would like to kindly ask whether our responses have addressed your concerns. We are happy to engage in further discussions or provide additional clarifications if needed.

Thank you again for your time and thoughtful review.

Sincerely,

Authors Paper-43343

Reviewer_ovMHOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-model KV-cache reuse from a much smaller source model to a larger target. It builds on the linear mapping of Heo et al. and adds selective recomputation, where tokens are chosen by Rank Disagreement without running the target. For Qwen3-0.6B to 14B, recomputing 30% of positions improves GSM8K from 71.6% to 90.2% (14B alone: 95.1%). Other Qwen3 and Llama 3 pairs are also evaluated, and prefill speedups are reported with a PyTorch implementation on a single A100.

Critical strengths
  • The observation on very small sources is useful. With 0.6B, plain transfer loses more than 20 points on GSM8K, and the gap grows as the source gets smaller.
  • Partial recomputation seems to remove most of the dependence on source size. For example, 0.6B with 30% recomputation is close to 4B with no recomputation, which is a practically interesting result.
  • In the matched-budget comparison, Rank Disagreement clearly beats random, attention, and L2-error selection on GSM8K. The point that attention mostly goes to template tokens also explains why attention-based selection does not work here.
  • The implementation work is solid. The FP32 mapper is slower than native prefill, and BF16 plus block sharing brings it to 1.95× at T=2048.
Critical weaknesses
  1. Rank Disagreement is tested in one setting. All selector comparisons are on 300 GSM8K examples for Qwen3-0.6B to 14B. The CacheBlend baseline is an L2 oracle, not the original method. Also, since only 0.23% of selected tokens are in the template, part of the gain over random may come from simply skipping template tokens. Random over non-template tokens would be a fairer baseline.

  2. Long context looks weak in both quality and speed. On LongBench-E, even 50% recomputation stays well below native prefill (59.7 vs. 65.9 F1). The speedup also drops with length (1.95× at 2K, 1.78× at 8K, ρ=0.3), and at ρ=0.5 it is only 1.38× at 2K. Accuracy and latency are never measured together on long inputs.

  3. KV transfer is not included. The source KV is assumed to already be on the target GPU, so the comparison with full target prefill leaves out the transfer cost. This may be fine for GSM8K-length prompts, but for long contexts the transfer cost can matter. Runtime results are also only for 0.6B to 14B.

  4. Benchmarks and calibration. Llama is evaluated only on GSM8K (300 examples for 8B to 70B), and the long-context set is only the 0–4K QA bucket of LongBench-E. Harder reasoning such as MATH-500, and the full LongBench (16 tasks) with longer inputs, are missing. Since GSM8K prompts are short and fairly simple, it is hard to say whether the selector still finds the right tokens in harder or longer tasks. The map and the Rank Disagreement basis are also calibrated per benchmark, so generalization is unclear. The abstract's 95–99% retention at 30% holds only for GSM8K and ARC.

Questions for authors
  1. How does Rank Disagreement compare with random selection over non-template tokens and with the original CacheBlend? Does the advantage hold on MMLU-Redux, LongBench-E, and the Llama pairs, not only on GSM8K?
  2. Could the authors report results on MATH-500, including the selector comparison (Rank Disagreement vs. random and other selectors)?
  3. Could the authors evaluate on the full LongBench (16 tasks) or longer buckets, where long-context use actually matters?
  4. Including KV transfer from the source GPU, mapping, selection, and recomputation, what is the end-to-end latency compared with full target prefill at long inputs (e.g., 8K or longer) and at ρ=0.4–0.5? Could this also be shown for another pair, such as Llama 8B to 70B?
7 comments

We thank the reviewer ovMH for their appreciation of our work and insightful feedback. The following is our detailed response to their concerns and questions.

1. Comparison with Random Selection over Non-Template Tokens

Table 1 compares rank disagreement with random excluding template and instruction tokens (Random w/o TI) on GSM8K, MATH-500, and three context-length buckets of LongBench-E.

Table 1. Comparison against random excluding template and instruction tokens

DatasetρRandom w/o TIRank Disagreement
GSM8K (n=300)0.3091.090.3
MATH-500 (n=500)0.3072.275.4
LongBench-E 0–4k (n=467)0.5044.858.8
LongBench-E 4–8k (n=470)0.5046.352.9
LongBench-E 8k+ (n=337)0.5038.652.7

Rank disagreement demonstrates clear and significant improvements over random w/o TI on long-context tasks, exceeding accuracy by up to 14.1 pp on 8k+ contexts on LongBench. For MATH-500 rank disagreement beats random w/o TI by 3.2 pp, whereas for GSM8K, random w/o TI performs slightly better, achieving 91% accuracy compared with 90.3% for rank disagreement.

We note that template and instruction tokens constitute approximately 45-60% of the prompts for short-context tasks like GSM8K and MATH-500. Since this text is shared across calibration and test sets, its mapping may be easier to learn. Excluding these tokens concentrates the recomputation budget on a substantial fraction of the question tokens, explaining why random w/o TI can be effective for short-context tasks. On the other hand, the template and instruction tokens of LongBench-E constitute <1% of the prompt. Consequently, random selection excluding template and instruction tokens does not help significantly. More generally, the results show that rank disagreement is a reliable metric for selective recomputation irrespective of the context length.

2. CacheBlend baseline is an L2 oracle, not the original method

We respectfully clarify that our CacheBlend baseline is not an L2 oracle, but a direct adaptation of the original CacheBlend method to our cross-model setting. We will clarify this in the revised manuscript to prevent any misunderstanding.

In its original single-model context, CacheBlend reuses independently prefilled chunked KV caches. It selects which tokens to recompute by running the full context through the target model's first layer and comparing that newly generated target KV cache to the reused chunked KV cache. To faithfully evaluate this mechanism in our pipeline, we follow the exact same procedure: we prefill the context through the first layer of the target model and compute the L2 deviation between this Layer1 target cache and the Layer1 mapped cache from the source model.

Interestingly, our results demonstrate that this target-side, early-layer selector is less effective in a cross-model setting (Figure 8 in the paper). While CacheBlend's core assumption, that Layer1 deviation reliably predicts late-layer errors, holds true for missing cross-chunk attention, it does not hold for cross-model mapping errors.

3. Evaluation on MATH-500 including selector comparison (Rank Disagreement vs. random and other selectors)

As requested, we present MATH-500 with selector comparison results in the Table 2 below. Rank disagreement yields the best performance across all selectors, improving over random w/o TI by 3.2 pp and uniform w/o TI by 3.6 pp.

Table 2. Evaluation and selector comparison on MATH-500

TargetSourceTransfer(map)RandomRandom w/o TIUniform w/o TIRank Disagreement
82.248.061.466.672.271.875.4
4. Evaluation on LongBench (16 tasks) or longer buckets

As requested, we report the results for 5 English QA tasks (HotpotQA, 2WikiMQA, Qasper, TriviaQA, and MultiFieldQA-en) in Longbench across all context lengths. We use a single calibration map for all 5 QA tasks and context length buckets. Notably, RaReCache is effective across all context lengths, even though transfer accuracy without recomputation falls below the source model. In fact, retention improved with increasing context length retaining 89.2% target accuracy at context lengths 8k+.

Table 3. Mean F1 for LongBench for Qwen3-0.6B→14B, ρ=0.5.

Bucket (n, mean prompt tokens)TargetSourceMapperMapper + recompute (ρ=0.5)Retention
0-4k (467, 4.3k)65.534.228.956.986.8%
4-8k (470, 8.6k)60.130.726.952.988.0%
8k+ (337, 15.6k)59.132.225.552.789.2%
Standard (950, 9.2k)61.833.827.753.185.9%

The experiments on other LongBench tasks, including summarization and code generation, take significantly longer to run. We will include them in the camera-ready version, if accepted.

5. Map and the Rank Disagreement basis are also calibrated per benchmark, so generalization is unclear

To analyze generalization of the calibration map across benchmarks, we evaluated MATH-500 with a map calibrated on GSM8K train set, MATH-500 train set, and a mix of both.

At 30% recomputation with rank disagreement, the GSM8K-calibrated map achieves 68.8% accuracy, compared with 75.4% for the MATH-calibrated map suffering a 6.6 pp drop. Mixed calibration achieves 73.6%, reducing the gap to 1.8 pp. Therefore, including the target prompt distribution during calibration helps the map generalize, though generalization costs a small drop in accuracy over task-specific calibration. Moreover, without recomputation, the corresponding cross-task gap is substantially larger: 26.2% vs 61.4%. Thus, selective recomputation mitigates calibration mismatch, while including the target task distribution in a mixed calibration set further improves performance.

Table 4. MATH-500 evaluation on Qwen3-0.6B→14B using different calibration maps

Recompute selectorρGSM8K-train mapMATH-train mapMixed map (GSM8K+MATH train)
Transfer026.261.458.0
Mapping (Rank Disagreement)0.3068.875.473.6
6. Accuracy and latency are never measured together on long inputs.

We report accuracy and latency together on LongBench-E QA 0-4k bucket in Table 5 for ρ=0.4,0.5\rho=0.4,0.5. RaReCache achieves a speedup of 1.31× while retaining 91% of target accuracy for ρ=0.5\rho=0.5.

Table 5. Accuracy and speedup on LongBench 0-4K**

Recompute ρF1Retention %Speedup
0.453.681%1.56×
0.559.791%1.31×
7. KV transfer cost is not included, the source KV is assumed to already be on the target GPU; for long contexts the transfer cost can matter

We thank the reviewer for suggesting to include the cache transfer latency for end-to-end latency evaluation. Our experiments suggest that transfer latency barely changes the overall speedup, and in fact, becomes even less significant as context length increases.

When we transfer the KV between Qwen3-0.6B to 14B, each target model (14B) layer is mapped from 8 selected source model (0.6B) layers. Across all 40 target layers, these selections cover 25 of the 0.6B's 28 layers. Per token, that is 25 layers × 8 KV heads × 128 dims × 2 (keys and values) × 2 bytes = 100 KiB of memory. Compared to a prefill-decode disaggregated serving (DistServe[1], Splitwise[2]), where prefill cache is shipped from a different GPU, we transfer a smaller payload, given that a full 14B cache is 160 KiB/token.

We measure transfer latency across three communication links on the same machine: NVLink, PCIe and a 100 Gb/s ethernet network and report the updated speedup values with the added latency for the corresponding communication links in the Table 6 below.

Table 6. Speedup with KV transfer cost included (Qwen3-0.6B → 14B, ρ = 0.4, batch size 1)

Prompt lengthData sentNo transferNVLinkPCIeVia host memory100 Gb/s network
2K200 MiB1.53×1.53× (0.9 ms)1.48× (8.4 ms)1.42× (17 ms)1.42× (17 ms)
8K800 MiB1.43×1.42× (3.2 ms)1.38× (33 ms)1.34× (67 ms)1.34× (67 ms)
16K1.6 GiB1.22×1.22× (6.2 ms)1.19× (67 ms)1.16× (134 ms)1.16× (134 ms)

Our results show that including cache transfer barely changes the speedup over NVLink. For the worst case, speedup drops at most from 1.53× to 1.42× through a slower transfer path like a 100 Gb/s network at T = 2K. In fact, transfer matters less as prompts get longer. Cache transfer latency scales linearly with prompt length, whereas full target prefill computation grows super-linearly. Even across host memory, transfer accounts for at most 5% and 3.5% of full target prefill runtime at 2K and 32K tokens, respectively.

[1] Yinmin Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, OSDI '24.

[2] Pratyush Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting, ISCA '24.

8. End-to-end latency compared with full target prefill at long inputs (e.g., 8K or longer) and at ρ=0.4–0.5, including KV transfer from the source GPU, mapping, selection, and recomputation

As requested, we report the end-to-end latency (cache transfer + mapping + selection + recompute) at long inputs and ρ = 0.4-0.5 against a full target model prefill for both LLaMA and Qwen model pairs. All timings are in ms. for batch size 1. At 8K input length, RaReCache achieves 1.6× and 1.30× speedups for Llama 3 8B → 70B, and 1.42× and 1.19× for Qwen3-0.6B → 14B, at ρ = 0.4 and 0.5 respectively.

Table 7. Qwen3-0.6B → 14B (target on one A100; source on a second A100 linked by NVLink, cache sent over NVLink)

TρFull 14B prefillTransferMap + selectRecomputeEnd-to-endSpeedup
2K0.43240.9431682121.53×
2K0.53240.9431932371.37×
8K0.414153.21678259951.42×
8K0.514153.2167101711871.19×

Table 8. Llama 3 8B → 70B (70B split across two NVLinked A100s; 8B on a third A100, cache sent over PCIe to both, each receiving only the layers its half of the 70B reads)

TρFull 70B prefillTransferMap + selectRecomputeEnd-to-endSpeedup
2K0.41442201597249021.60×
2K0.51442201598219991.44×
8K0.4620179616328139761.56×
8K0.5620179616406847621.30×

The speedup of RaReCache over full target prefill is limited by partial recomputation at longer context lengths, and the cause lies in the attention kernel. While recomputing ρT positions scales down feed-forward computation by ρ, but each recomputed query still attends to all preceding keys. Our torch implementation computes this via an explicit mask over a full ρT × T = ρT² block, whereas native target prefill leverages causal FlashAttention over the lower triangle (T²/2). As sequence length T grows and attention dominates linear layers, the theoretical speedup ceiling asymptotically approaches T²/2 ÷ ρT² = 1/(2ρ). A custom causal attention kernel for scattered recompute positions would skip, for each recomputed token, the keys that come after it, just as FlashAttention does for full prefill. That would remove most of the extra attention-specific computation and recover much of the speedup lost at longer context. We leave this kernel to future work.

Comments on the paper

2 comments
Area_Chair_LiHWOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

The authors have posted responses to your reviews. The author-reviewer discussion closes on November 18 (AoE).

Please read the responses and reply here before then: let the authors know whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews is also welcome.

Thank you, Area Chair

Area_Chair_LiHWOpenReview ↗

Final reminder: discussion closes today (Nov 18 AoE)

The author-reviewer discussion closes today, November 18 (AoE).

Dear reviewers,

Thank you to those who have already responded. If you have not yet replied: Please read the authors' response and post a brief reply here before the deadline: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Your response is important for the final recommendation.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.